diff --git a/.claude/skills/slayer-models.md b/.claude/skills/slayer-models.md index 32fc7991..6b23bc33 100644 --- a/.claude/skills/slayer-models.md +++ b/.claude/skills/slayer-models.md @@ -43,7 +43,7 @@ measures: formula: "amount:sum / *:count" ``` -Aggregation is specified at query time with **colon syntax**: `"amount:sum"`, `"amount:avg"`, `"*:count"`. A bare-name reference like `{"formula": "aov"}` resolves to the saved `ModelMeasure` formula on the model. Built-in aggregations: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, `first`, `last`, `weighted_avg`, `median`, `percentile`, `stddev_samp`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar_samp`, `covar_pop`. The two-column ones (`corr`, `covar_samp`, `covar_pop`) take the second column as a named param: `price:corr(other=quantity)`. +Aggregation is specified at query time with **colon syntax**: `"amount:sum"`, `"amount:avg"`, `"*:count"`. A bare-name reference like `{"formula": "aov"}` resolves to the saved `ModelMeasure` formula on the model. Built-in aggregations: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, `count_distinct_approx`, `first`, `last`, `weighted_avg`, `median`, `percentile`, `stddev_samp`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar_samp`, `covar_pop`. `count_distinct_approx` is dialect-aware (native approximate-distinct where available, exact `COUNT(DISTINCT)` fallback otherwise). The two-column ones (`corr`, `covar_samp`, `covar_pop`) take the second column as a named param: `price:corr(other=quantity)`. ## Data Types diff --git a/.claude/skills/slayer-overview.md b/.claude/skills/slayer-overview.md index ba9a68d9..353955ab 100644 --- a/.claude/skills/slayer-overview.md +++ b/.claude/skills/slayer-overview.md @@ -23,16 +23,18 @@ SLayer is a lightweight, agent-first semantic layer. Instead of writing raw SQL, Query-backed models support two access patterns: **run by name** (`engine.execute("monthly_revenue", variables={...})` runs the stored backing query) and **as a source_model** (`{"source_model": "monthly_revenue", ...}` in another query). Variable precedence: runtime kwarg > stage > outer query > model defaults. +- **SessionPolicy** (Row-Level Security, DEV-1578 / DEV-1718) — immutable, agent-invisible forced filter passed at engine/local-client init. Carries one required `ruleset`: a `ColumnFilterRuleset` (filter every table that has the tenant column) or a `JoinFilterRuleset` (one anchor table holds the identifier; others reach it via explicit joins, with a whitelist for shared tables). `SlayerQueryEngine(storage, policy=SessionPolicy(ruleset=ColumnFilterRuleset(column="organization_uuid", value=...)))`. Silently scopes every query (base, joins, CTEs, sql-mode, query-backed, profiling) to one tenant at the final-SQL layer. No-filtering = `policy=None`; local-engine only (HTTP `policy=` raises). See [row-level-security.md](../../docs/concepts/row-level-security.md). + ## MCP Tools -Discovery: `list_datasources`, `models_summary`, `inspect_model` (with sample data) -Querying: `query` +Discovery: `list_datasources`, `models_summary`, `inspect` (point lookup by `reference` + required `entity_type`; the model path carries sample data; `reference` accepts a single id, a same-kind **list** for a batched lookup, or **`None`/omitted** for the whole **collection** at that kind — `entity_type="model"` lists all models grouped by datasource, `entity_type="datasource"` lists all datasources, subsuming `models_summary` / `list_datasources`). `inspect_model` is DEPRECATED — use `inspect`. +Querying: `query`, `recommend_root_model` (given `model.column` / `model.metric` items, introspects the join graph to recommend the query `source_model` + each item's join path from it; optional `root_hint` forces an intended root when it reaches every item; returns partial-root `coverage` when no single model reaches all items) Model editing: `create_model`, `edit_model`, `delete_model` Datasources: `create_datasource`, `list_datasources`, `describe_datasource` (includes table listing by default), `edit_datasource`, `delete_datasource`, `set_datasource_priority` Ingestion: `ingest_datasource_models` Schema drift: `validate_models` (read-only diff against live schema; surfaces `SchemaDriftError` cleanups) Memory write side: `save_memory`, `forget_memory` (per-entity learnings indexed by canonical entity strings — see [memories.md](../../docs/concepts/memories.md)) -Search: `search` (three-channel: entity-overlap BM25 over memories + tantivy full-text + optional dense embedding similarity, RRF-fused per kind so each output bucket — `memories` / `example_queries` / `entities` — has membership/order invariant under the other buckets' caps; embeddings require the `embedding_search` extra and degrade gracefully when unavailable; partitions query-bearing memories into `example_queries` — see [search.md](../../docs/concepts/search.md)) +Search: `search` (three-channel: entity-overlap BM25 over memory tags + tantivy full-text + optional dense embedding similarity, RRF-fused into a single flat `SearchResponse.results: List[SearchHit]` (DEV-1532) with a `kind` discriminator — `"memory"` / `"datasource"` / `"model"` / `"column"` / `"measure"` / `"aggregation"`; query-bearing memories are still memory hits, distinguished by `hit.query is not None`. Optional `cypher_filter` pre-narrows all three channels: full openCypher when `advanced_search` is installed, naive `MATCH (n:Label) RETURN n.id AS id` kind-filter otherwise. Embeddings also require the `advanced_search` extra and degrade gracefully when unavailable — see [search.md](../../docs/concepts/search.md)) ## Package Structure diff --git a/.claude/skills/slayer-query.md b/.claude/skills/slayer-query.md index e95a496e..77ab2bb2 100644 --- a/.claude/skills/slayer-query.md +++ b/.claude/skills/slayer-query.md @@ -22,7 +22,9 @@ A `SlayerQuery` is a JSON/dict object. The same shape works across the REST API, `order[].column` is the short alias (`count`, `revenue_sum`) — not the colon form. -**Dim-only queries deduplicate.** A query with no measures and at least one dimension or time-dimension auto-emits `GROUP BY ` and returns the distinct combinations. The `GROUP BY` is applied before `LIMIT`, so a row cap can't silently drop unique tuples. There is no opt-out — if you want the raw row stream, query the underlying table outside the semantic layer. +**Ordering by something you don't project.** `order` may name an undeclared column/aggregate/expression ("top-N by X, show only Y, Z"). Computed hidden, sorted on, and stripped from the result: an **aggregate** (`amount:sum`, `customers.revenue:sum`), an inline **transform** (`rank(amount:sum)`, `change(...)`, `cumsum`/`lag`/`lead`/`ntile`), an inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`), and a **windowed** aggregate (`amount:sum(window='90d')`, alone or inside a composite). A **raw row column** is orderable only in a raw-rows query (`distinct_dimension_values: false`); in a grouped/dedup query it's rejected (HTTP 400 — add it to `dimensions` or order by an aggregate of it). A **joined** row column is rejected — project it. Order expressions must use formula syntax for their operands, not the `name`s of measures declared in the same query: `{"column": "revenue:sum / cnt:sum"}` works, `{"column": "rev / cnt"}` is rejected. + +**Dim-only queries deduplicate.** A query with no measures and at least one dimension or time-dimension auto-emits `GROUP BY ` and returns the distinct combinations. The `GROUP BY` is applied before `LIMIT`, so a row cap can't silently drop unique tuples. To opt out, set `"distinct_dimension_values": false` on the query — emits raw rows (no top-level `GROUP BY`), with WHERE / ORDER BY / LIMIT applied as usual. Any measure reference in `measures` / `filters` / `order` raises `DistinctDimensionValuesError` in this mode. ## Measures — colon aggregation @@ -40,11 +42,15 @@ Each entry in `measures` is either a bare formula string or a `{"formula": ..., "last(revenue:sum)", "time_shift(revenue:sum, -1, 'year')", "lag(revenue:sum, 1)", - "rank(revenue:sum)" + "rank(revenue:sum)", + "round(revenue:sum, 2)", + "abs(revenue:sum - cost:sum)" ] ``` -Built-in aggregations: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, `first`, `last`, `weighted_avg`, `median`, `percentile`, `stddev_samp`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar_samp`, `covar_pop`. Two-column `corr`/`covar_samp`/`covar_pop` take the second column as a named param: `price:corr(other=quantity)`. `sum` and `avg` accept an optional trailing-window: `revenue:sum(window='30d')`. +Built-in aggregations: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, `count_distinct_approx`, `first`, `last`, `weighted_avg`, `median`, `percentile`, `stddev_samp`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar_samp`, `covar_pop`. `count_distinct_approx` is dialect-aware (native approximate-distinct where available, exact `COUNT(DISTINCT)` fallback otherwise). Two-column `corr`/`covar_samp`/`covar_pop` take the second column as a named param: `price:corr(other=quantity)`. `sum` and `avg` accept an optional trailing-window: `revenue:sum(window='30d')`. A time bound narrows which buckets come back, not which rows the window may reach — so `date_range` and an equivalent explicit filter (`created_at >= '2025-01-01'`) give identical windowed numbers. Only `<`/`<=`/`>`/`>=` against a time dimension's own column and a literal counts; other operators, non-time-dimension columns, bounds under `or`/`not`, and model-level `filters` all restrict the window's input as usual. Same rule for `time_shift`. + +For month-over-month / period-over-period growth use `change_pct(x)` (absolute delta: `change(x)`) — both are calendar-aware and partition-safe (the underlying self-join matches on all non-time dimensions, so per-group series reset cleanly). Reach for `time_shift` only when you need the shifted value itself as a term in custom arithmetic or at a different grain (`time_shift(revenue:sum, -1, 'year')` for year-over-year). `*:count` is always available — no column definition needed. `col:count` counts non-nulls. @@ -116,6 +122,20 @@ Reference measures from joined models with dotted syntax + colon aggregation: A dotted reference may target a *derived* column on the joined model (a column whose own `sql` is itself an expression). The engine recursively inlines the chain at query time — `"B.foo_normalized:sum"` where `B.foo_normalized.sql = "foo_raw / 100.0"` emits `SUM(B.foo_raw / 100.0)`. The same chaining works inside `Column.sql`, `filters`, and `dimensions`. When a filter names a *bare* local derived column whose SQL crosses a join (e.g. `Column(name="is_eu", sql="CASE WHEN customers.region = 'EU' THEN 1 ELSE 0 END")` referenced as `"filters": ["is_eu = 1"]`), the planner walks the column's chain and adds the joins the chain implies — no need to also list the column in `dimensions`. +## Picking the root model + +Not sure which model to use as `source_model` for a set of columns/metrics? Call `recommend_root_model` with the `model.column` / `model.metric` items you want; it introspects the join graph and returns the recommended root plus each item's join-qualified path from it (aggregation suffixes preserved), ready to drop into a query. + +```python +rec = client.recommend_root_model_sync(["customers.name", "products.category"]) +rec.root_model # "orders" +[ip.path for ip in rec.item_paths] # ["customers.name", "products.category"] +``` + +Pass `root_hint` (a bare model name or `.`) to force an intended root — useful when the host is a bridge model that owns none of the items but matches your grain. It's honored when it reaches every item; otherwise the auto-pick is used and `warnings` says why. + +MCP: `recommend_root_model(items, data_source=None, root_hint=None, format="markdown")`. If no single model reaches every item, `root_model` is `None` and `coverage` lists the best partial roots — a hint to split into a multi-stage `source_queries` query. + ## ModelExtension Extend a model inline with extra columns, named-formula measures, joins, or filters. The stored model is not modified: diff --git a/.dockerignore b/.dockerignore index 2dea5a51..4c42ab74 100644 --- a/.dockerignore +++ b/.dockerignore @@ -17,7 +17,6 @@ examples *.md !README.md !LICENSE -mkdocs.yml -.readthedocs.yaml +zensical.toml .dockerignore .gitignore diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 118a0707..fe093aee 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -1,5 +1,8 @@ name: CI +permissions: + contents: read + on: push: branches: [main] @@ -48,72 +51,169 @@ jobs: - name: Run integration tests timeout-minutes: 20 - run: poetry run pytest tests/ -v -m integration --timeout=180 - - integration-examples: + # DEV-1564: the MySQL / ClickHouse / SQL Server pytest integration + # suites have their own path-gated workflows under + # .github/workflows/integration-.yml. They require Docker + # + testcontainers + (for SQL Server) the ODBC Driver 18 — provisioned + # only in those workflows. Exclude them from the always-on integration + # job; they run when their respective dialect files change. + # + # DEV-1562: the live-Metabase e2e suite (`metabase_e2e` marker) has + # its own workflow at .github/workflows/pg-facade-e2e.yml — same + # reasoning, exclude here so this job isn't on the hook for booting + # a Metabase container. + run: | + poetry run pytest tests/ -v -m "integration and not metabase_e2e" --timeout=180 \ + --ignore=tests/integration/test_integration_mysql.py \ + --ignore=tests/integration/test_integration_clickhouse.py \ + --ignore=tests/integration/test_integration_sqlserver.py + + # DEV-1564: the previous `integration-examples` matrix (verify.py-based + # end-to-end checks for ClickHouse + MySQL) migrated into the per-dialect + # workflows so each dialect's CI lives in one place and is path-gated. + # SQL Server gained its first verify.py-based CI coverage there too. + + bigquery-example: + # Skip on forked-PR runs where the GCP secrets aren't available — the job + # would fail otherwise and we'd rather not nag external contributors. + if: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }} runs-on: ubuntu-latest - strategy: - fail-fast: false - matrix: - example: [clickhouse, mysql] - + needs: lint-and-test steps: - uses: actions/checkout@v4 + - name: Skip if GCP secrets are unset + id: gate + env: + GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }} + GCP_SA_KEY_B64: ${{ secrets.GCP_SA_KEY_B64 }} + run: | + if [[ -z "$GCP_PROJECT_ID" || -z "$GCP_SA_KEY_B64" ]]; then + echo "BigQuery secrets unset — skipping the example." + echo "skip=true" >> "$GITHUB_OUTPUT" + else + echo "skip=false" >> "$GITHUB_OUTPUT" + fi + - name: Set up Python + if: steps.gate.outputs.skip != 'true' uses: actions/setup-python@v5 with: python-version: "3.11" - - name: Install verify.py dependencies - run: pip install sqlalchemy - - - name: Make slayer_data writable for container user - working-directory: examples/${{ matrix.example }} - run: chmod -R 777 slayer_data + - name: Install Poetry + if: steps.gate.outputs.skip != 'true' + run: pip install poetry - - name: Build images - working-directory: examples/${{ matrix.example }} - run: docker compose build + - name: Install dependencies (with bigquery extra) + if: steps.gate.outputs.skip != 'true' + run: poetry install -E bigquery - - name: Start DB + seed - working-directory: examples/${{ matrix.example }} - run: docker compose up -d --wait --wait-timeout 180 seed + - name: Decode service-account JSON key + if: steps.gate.outputs.skip != 'true' + env: + GCP_SA_KEY_B64: ${{ secrets.GCP_SA_KEY_B64 }} + run: | + echo "$GCP_SA_KEY_B64" | base64 -d > "$RUNNER_TEMP/sa-key.json" + echo "GOOGLE_APPLICATION_CREDENTIALS=$RUNNER_TEMP/sa-key.json" >> "$GITHUB_ENV" - - name: Start slayer service - working-directory: examples/${{ matrix.example }} - run: docker compose up -d slayer + - name: Start SLayer server + if: steps.gate.outputs.skip != 'true' + env: + GCP_PROJECT_ID: ${{ secrets.GCP_PROJECT_ID }} + run: | + poetry run bash examples/bigquery/start.sh > /tmp/slayer-bq.log 2>&1 & + echo "SLAYER_PID=$!" >> "$GITHUB_ENV" - name: Wait for SLayer API to accept connections - working-directory: examples/${{ matrix.example }} + if: steps.gate.outputs.skip != 'true' run: | - for i in $(seq 1 120); do - if curl -sf http://localhost:5143/datasources >/dev/null; then + for i in $(seq 1 60); do + if curl -sf http://localhost:5143/health >/dev/null; then echo "SLayer API ready after ${i} attempts" exit 0 fi - # If the container died, fail fast and show why. - if [ "$(docker compose ps -q slayer | xargs -r docker inspect -f '{{.State.Running}}')" = "false" ]; then - echo "slayer container exited before becoming ready — logs:" >&2 - docker compose logs --no-color slayer - exit 1 - fi sleep 2 done echo "SLayer API never came up — logs:" >&2 - docker compose logs --no-color slayer + cat /tmp/slayer-bq.log exit 1 - name: Run verify.py + if: steps.gate.outputs.skip != 'true' timeout-minutes: 5 - run: python examples/${{ matrix.example }}/verify.py + run: poetry run python examples/bigquery/verify.py - - name: Dump all container logs - if: always() - working-directory: examples/${{ matrix.example }} - run: docker compose logs --no-color + - name: Dump server logs on failure + if: failure() && steps.gate.outputs.skip != 'true' + run: cat /tmp/slayer-bq.log + + - name: Stop SLayer server + if: always() && steps.gate.outputs.skip != 'true' + run: | + if [ -n "${SLAYER_PID:-}" ]; then + kill "$SLAYER_PID" 2>/dev/null || true + fi + rm -f "$RUNNER_TEMP/sa-key.json" + + snowflake-integration: + # Skip on forked-PR runs where the Snowflake secret isn't available — the + # job would fail otherwise and we'd rather not nag external contributors. + if: ${{ github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository }} + runs-on: ubuntu-latest + needs: lint-and-test + steps: + - uses: actions/checkout@v4 + + - name: Skip if Snowflake secret is unset + id: gate + env: + SNOWFLAKE_CONNECTIONS_TOML: ${{ secrets.SNOWFLAKE_CONNECTIONS_TOML }} + run: | + if [[ -z "$SNOWFLAKE_CONNECTIONS_TOML" ]]; then + echo "Snowflake secret unset — skipping the integration suite." + echo "skip=true" >> "$GITHUB_OUTPUT" + else + echo "skip=false" >> "$GITHUB_OUTPUT" + fi + + - name: Set up Python + if: steps.gate.outputs.skip != 'true' + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install Poetry + if: steps.gate.outputs.skip != 'true' + run: pip install poetry - - name: Tear down stack + - name: Install dependencies (with snowflake extra) + if: steps.gate.outputs.skip != 'true' + run: poetry install -E snowflake + + - name: Write ~/.snowflake/connections.toml from secret + if: steps.gate.outputs.skip != 'true' + env: + SNOWFLAKE_CONNECTIONS_TOML: ${{ secrets.SNOWFLAKE_CONNECTIONS_TOML }} + run: | + mkdir -p ~/.snowflake + # `printf '%s'` is literal — no backslash mangling, no extra trailing + # newline. `chmod 600` is required: snowflake-connector-python + # refuses to load a TOML that's world-/group-readable. + printf '%s' "$SNOWFLAKE_CONNECTIONS_TOML" > ~/.snowflake/connections.toml + chmod 600 ~/.snowflake/connections.toml + + - name: Run Snowflake integration tests + if: steps.gate.outputs.skip != 'true' + timeout-minutes: 15 + # SLAYER_SNOWFLAKE_CONNECTION names the profile in the TOML the + # integration suite should use. The default in the test file is + # `slayer_test` — match the section header in + # `secrets.SNOWFLAKE_CONNECTIONS_TOML`. + env: + SLAYER_SNOWFLAKE_CONNECTION: slayer_test + run: poetry run pytest tests/integration/test_integration_snowflake.py -v -m integration --timeout=300 + + - name: Wipe Snowflake credentials if: always() - working-directory: examples/${{ matrix.example }} - run: docker compose down -v + run: rm -f ~/.snowflake/connections.toml diff --git a/.github/workflows/integration-clickhouse.yml b/.github/workflows/integration-clickhouse.yml new file mode 100644 index 00000000..213354a5 --- /dev/null +++ b/.github/workflows/integration-clickhouse.yml @@ -0,0 +1,114 @@ +name: Integration - ClickHouse + +# DEV-1564: per-dialect CI for ClickHouse — pytest suite + verify.py +# end-to-end check. Path-gated to ClickHouse-specific files plus the +# shared SQL generator + dialect base. + +on: + pull_request: + branches: [main] + paths: + - 'slayer/sql/dialects/clickhouse.py' + - 'slayer/sql/dialects/base.py' + - 'slayer/sql/generator.py' + - 'examples/clickhouse/**' + - 'tests/integration/test_integration_clickhouse.py' + - '.github/workflows/integration-clickhouse.yml' + push: + branches: [main] + paths: + - 'slayer/sql/dialects/clickhouse.py' + - 'slayer/sql/dialects/base.py' + - 'slayer/sql/generator.py' + - 'examples/clickhouse/**' + - 'tests/integration/test_integration_clickhouse.py' + - '.github/workflows/integration-clickhouse.yml' + workflow_dispatch: + +jobs: + pytest: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install Poetry + run: pip install poetry + + - name: Install dependencies + run: poetry install -E all --with dev + + - name: Verify testcontainers[clickhouse] extra is importable + run: poetry run python -c "import testcontainers.clickhouse" + + - name: Run ClickHouse integration tests + timeout-minutes: 20 + run: | + poetry run pytest tests/integration/test_integration_clickhouse.py \ + -v -m integration --timeout=300 + + verify-example: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install verify.py dependencies + run: pip install sqlalchemy + + - name: Make slayer_data writable for container user + working-directory: examples/clickhouse + run: chmod -R 777 slayer_data + + - name: Build images + working-directory: examples/clickhouse + run: docker compose build + + - name: Start DB + seed + working-directory: examples/clickhouse + run: docker compose up -d --wait --wait-timeout 180 seed + + - name: Start slayer service + working-directory: examples/clickhouse + run: docker compose up -d slayer + + - name: Wait for SLayer API to accept connections + working-directory: examples/clickhouse + run: | + for i in $(seq 1 120); do + if curl -sf http://localhost:5143/datasources >/dev/null; then + echo "SLayer API ready after ${i} attempts" + exit 0 + fi + if [ "$(docker compose ps -q slayer | xargs -r docker inspect -f '{{.State.Running}}')" = "false" ]; then + echo "slayer container exited before becoming ready — logs:" >&2 + docker compose logs --no-color slayer + exit 1 + fi + sleep 2 + done + echo "SLayer API never came up — logs:" >&2 + docker compose logs --no-color slayer + exit 1 + + - name: Run verify.py + timeout-minutes: 5 + run: python examples/clickhouse/verify.py + + - name: Dump all container logs + if: always() + working-directory: examples/clickhouse + run: docker compose logs --no-color + + - name: Tear down stack + if: always() + working-directory: examples/clickhouse + run: docker compose down -v diff --git a/.github/workflows/integration-mysql.yml b/.github/workflows/integration-mysql.yml new file mode 100644 index 00000000..3a5c25c7 --- /dev/null +++ b/.github/workflows/integration-mysql.yml @@ -0,0 +1,118 @@ +name: Integration - MySQL + +# DEV-1564: per-dialect CI for MySQL — runs both the testcontainers-based +# pytest suite and the verify.py end-to-end check. Path-gated so it only +# fires on changes that could affect MySQL output (the dialect file, the +# shared SQL generator + dialect base, the MySQL example, or this file). + +on: + pull_request: + branches: [main] + paths: + - 'slayer/sql/dialects/mysql.py' + - 'slayer/sql/dialects/base.py' + - 'slayer/sql/generator.py' + - 'examples/mysql/**' + - 'tests/integration/test_integration_mysql.py' + - '.github/workflows/integration-mysql.yml' + push: + branches: [main] + paths: + - 'slayer/sql/dialects/mysql.py' + - 'slayer/sql/dialects/base.py' + - 'slayer/sql/generator.py' + - 'examples/mysql/**' + - 'tests/integration/test_integration_mysql.py' + - '.github/workflows/integration-mysql.yml' + workflow_dispatch: + +jobs: + pytest: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install Poetry + run: pip install poetry + + - name: Install dependencies + run: poetry install -E all --with dev + + - name: Verify testcontainers[mysql] extra is importable + # DEV-1564: surface a missing extra as a CI failure rather than as a + # silent `importorskip` skip — the suite's value is gone if the dep + # is absent. + run: poetry run python -c "import testcontainers.mysql" + + - name: Run MySQL integration tests + timeout-minutes: 20 + run: | + poetry run pytest tests/integration/test_integration_mysql.py \ + -v -m integration --timeout=300 + + verify-example: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install verify.py dependencies + run: pip install sqlalchemy + + - name: Make slayer_data writable for container user + working-directory: examples/mysql + run: chmod -R 777 slayer_data + + - name: Build images + working-directory: examples/mysql + run: docker compose build + + - name: Start DB + seed + working-directory: examples/mysql + run: docker compose up -d --wait --wait-timeout 180 seed + + - name: Start slayer service + working-directory: examples/mysql + run: docker compose up -d slayer + + - name: Wait for SLayer API to accept connections + working-directory: examples/mysql + run: | + for i in $(seq 1 120); do + if curl -sf http://localhost:5143/datasources >/dev/null; then + echo "SLayer API ready after ${i} attempts" + exit 0 + fi + if [ "$(docker compose ps -q slayer | xargs -r docker inspect -f '{{.State.Running}}')" = "false" ]; then + echo "slayer container exited before becoming ready — logs:" >&2 + docker compose logs --no-color slayer + exit 1 + fi + sleep 2 + done + echo "SLayer API never came up — logs:" >&2 + docker compose logs --no-color slayer + exit 1 + + - name: Run verify.py + timeout-minutes: 5 + run: python examples/mysql/verify.py + + - name: Dump all container logs + if: always() + working-directory: examples/mysql + run: docker compose logs --no-color + + - name: Tear down stack + if: always() + working-directory: examples/mysql + run: docker compose down -v diff --git a/.github/workflows/integration-sqlserver.yml b/.github/workflows/integration-sqlserver.yml new file mode 100644 index 00000000..4a3d9444 --- /dev/null +++ b/.github/workflows/integration-sqlserver.yml @@ -0,0 +1,145 @@ +name: Integration - SQL Server + +# DEV-1564: per-dialect CI for SQL Server — pytest suite + verify.py +# end-to-end check. Path-gated to T-SQL dialect file, the SQL Server +# example, the shared SQL generator + dialect base, and this file. +# +# SQL Server is the only Tier-1 dialect that had no CI before this +# workflow existed. + +on: + pull_request: + branches: [main] + paths: + - 'slayer/sql/dialects/tsql.py' + - 'slayer/sql/dialects/base.py' + - 'slayer/sql/generator.py' + - 'examples/sqlserver/**' + - 'tests/integration/test_integration_sqlserver.py' + - '.github/workflows/integration-sqlserver.yml' + push: + branches: [main] + paths: + - 'slayer/sql/dialects/tsql.py' + - 'slayer/sql/dialects/base.py' + - 'slayer/sql/generator.py' + - 'examples/sqlserver/**' + - 'tests/integration/test_integration_sqlserver.py' + - '.github/workflows/integration-sqlserver.yml' + workflow_dispatch: + +jobs: + pytest: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install ODBC Driver 18 for SQL Server + # The pytest suite uses pyodbc IN-PROCESS on the runner host (unlike + # verify-example, where pyodbc lives only inside Docker containers + # built from examples/sqlserver/Dockerfile). Microsoft's apt repo + # is the canonical install path on Ubuntu — derive the version from + # /etc/os-release so this keeps working when `ubuntu-latest` rolls + # from 24.04 → 26.04 etc. + run: | + UBUNTU_VERSION="$(. /etc/os-release && echo "$VERSION_ID")" + echo "Installing msodbcsql18 for Ubuntu ${UBUNTU_VERSION}" + curl -sSL https://packages.microsoft.com/keys/microsoft.asc \ + | sudo tee /etc/apt/trusted.gpg.d/microsoft.asc > /dev/null + curl -sSL "https://packages.microsoft.com/config/ubuntu/${UBUNTU_VERSION}/prod.list" \ + | sudo tee /etc/apt/sources.list.d/mssql-release.list > /dev/null + sudo apt-get update + sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 unixodbc-dev + + - name: Install Poetry + run: pip install poetry + + - name: Install dependencies + run: poetry install -E all --with dev + + - name: Verify testcontainers[mssql] extra is importable + run: poetry run python -c "import testcontainers.mssql" + + - name: Verify ODBC Driver 18 is visible to pyodbc + run: | + poetry run python -c "import pyodbc; \ + drivers = pyodbc.drivers(); \ + assert 'ODBC Driver 18 for SQL Server' in drivers, \ + f'Missing ODBC Driver 18 — installed: {drivers!r}'; \ + print('ODBC drivers:', drivers)" + + - name: Run SQL Server integration tests + timeout-minutes: 25 + run: | + poetry run pytest tests/integration/test_integration_sqlserver.py \ + -v -m integration --timeout=300 + + verify-example: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install verify.py dependencies + # verify.py talks HTTP to the slayer container; the ODBC driver lives + # inside the container (built from examples/sqlserver/Dockerfile) so + # the runner host doesn't need msodbcsql18 here. + run: pip install sqlalchemy + + - name: Make slayer_data writable for container user + working-directory: examples/sqlserver + run: chmod -R 777 slayer_data + + - name: Build images + working-directory: examples/sqlserver + run: docker compose build + + - name: Start DB + seed + working-directory: examples/sqlserver + run: docker compose up -d --wait --wait-timeout 300 seed + + - name: Start slayer service + working-directory: examples/sqlserver + run: docker compose up -d slayer + + - name: Wait for SLayer API to accept connections + working-directory: examples/sqlserver + run: | + for i in $(seq 1 120); do + if curl -sf http://localhost:5143/datasources >/dev/null; then + echo "SLayer API ready after ${i} attempts" + exit 0 + fi + if [ "$(docker compose ps -q slayer | xargs -r docker inspect -f '{{.State.Running}}')" = "false" ]; then + echo "slayer container exited before becoming ready — logs:" >&2 + docker compose logs --no-color slayer + exit 1 + fi + sleep 2 + done + echo "SLayer API never came up — logs:" >&2 + docker compose logs --no-color slayer + exit 1 + + - name: Run verify.py + timeout-minutes: 5 + run: python examples/sqlserver/verify.py + + - name: Dump all container logs + if: always() + working-directory: examples/sqlserver + run: docker compose logs --no-color + + - name: Tear down stack + if: always() + working-directory: examples/sqlserver + run: docker compose down -v diff --git a/.github/workflows/notify-docs.yml b/.github/workflows/notify-docs.yml new file mode 100644 index 00000000..bac00aaf --- /dev/null +++ b/.github/workflows/notify-docs.yml @@ -0,0 +1,43 @@ +name: Notify docs site of release + +# When a GitHub Release is published, tell the combined product-docs site +# (MotleyAI/motley-docs) to pull this release's docs + navigation and +# republish https://docs.motley.ai/slayer. +# + +on: + release: + types: [published] + # Allow re-triggering a sync by hand (e.g. after fixing the docs repo). + workflow_dispatch: + inputs: + ref: + description: "SLayer git ref to sync (tag/branch)" + required: true + default: main + +jobs: + notify: + runs-on: ubuntu-latest + permissions: + contents: read + steps: + - name: Mint installation token + id: app-token + uses: actions/create-github-app-token@v2 + with: + app-id: ${{ vars.MOTLEY_DOCS_APP_ID }} + private-key: ${{ secrets.MOTLEY_DOCS_APP_PRIVATE_KEY }} + owner: MotleyAI + repositories: motley-docs + permission-contents: write + permission-metadata: read + + - name: Trigger motley-docs sync + uses: peter-evans/repository-dispatch@ff45666b9427631e3450c54a1bcbee4d9ff4d7c0 # v3 + with: + token: ${{ steps.app-token.outputs.token }} + repository: MotleyAI/motley-docs + event-type: slayer-release + client-payload: | + {"ref": "${{ github.event.release.tag_name || inputs.ref }}"} diff --git a/.github/workflows/pg-facade-e2e.yml b/.github/workflows/pg-facade-e2e.yml new file mode 100644 index 00000000..2ed4f71e --- /dev/null +++ b/.github/workflows/pg-facade-e2e.yml @@ -0,0 +1,89 @@ +name: PG Facade Live-Metabase E2E + +permissions: + contents: read + +on: + push: + branches: [main] + paths: + - 'slayer/pg_facade/**' + - 'slayer/facade/**' + - 'slayer/demo/**' + - 'tests/integration/test_metabase_e2e.py' + - 'tests/integration/conftest_metabase.py' + - 'tests/integration/_pg_serve_helpers.py' + - 'tests/integration/conftest.py' + - 'pyproject.toml' + - 'poetry.lock' + - '.github/workflows/pg-facade-e2e.yml' + pull_request: + branches: [main] + paths: + - 'slayer/pg_facade/**' + - 'slayer/facade/**' + - 'slayer/demo/**' + - 'tests/integration/test_metabase_e2e.py' + - 'tests/integration/conftest_metabase.py' + - 'tests/integration/_pg_serve_helpers.py' + - 'tests/integration/conftest.py' + - 'pyproject.toml' + - 'poetry.lock' + - '.github/workflows/pg-facade-e2e.yml' + +jobs: + metabase-e2e: + runs-on: ubuntu-latest + timeout-minutes: 25 + + steps: + - uses: actions/checkout@v4 + + - name: Set up Python 3.11 + uses: actions/setup-python@v5 + with: + python-version: "3.11" + + - name: Install Poetry + run: pip install poetry + + - name: Install dependencies + run: poetry install -E all --with dev + + - name: Install jafgen (for demo data generation) + run: poetry run pip install git+https://github.com/rossbowen/jaffle-shop-generator.git@09557a1118b000071f8171aa97d54d5029bf0f0b + + - name: Cache Metabase image + id: mb_cache + uses: actions/cache@v4 + with: + path: /tmp/metabase-image.tar + key: metabase-image-v0.62.1.5-${{ runner.os }}-${{ runner.arch }} + + - name: Load Metabase image from cache + if: steps.mb_cache.outputs.cache-hit == 'true' + run: docker load -i /tmp/metabase-image.tar + + - name: Pull and save Metabase image + if: steps.mb_cache.outputs.cache-hit != 'true' + run: | + docker pull metabase/metabase:v0.62.1.5 + docker save metabase/metabase:v0.62.1.5 -o /tmp/metabase-image.tar + + - name: Run live-Metabase e2e suite + timeout-minutes: 15 + run: poetry run pytest -m metabase_e2e tests/integration/test_metabase_e2e.py -v --timeout=300 + + - name: Dump Metabase container logs on failure + if: failure() + run: | + if [ -f /tmp/slayer-metabase-e2e-container.log ]; then + echo "=== /tmp/slayer-metabase-e2e-container.log ===" + cat /tmp/slayer-metabase-e2e-container.log + else + echo "No fixture-side log dump found — falling back to docker logs." + for cid in $(docker ps -aq --filter ancestor=metabase/metabase:v0.62.1.5); do + echo "=== container $cid ===" + docker logs --tail=400 "$cid" || true + done + fi diff --git a/.gitignore b/.gitignore index eb539976..828667e9 100644 --- a/.gitignore +++ b/.gitignore @@ -219,3 +219,9 @@ docs/examples/jaffle_data/jaffle-data/ docs/examples/jaffle_data/jaffle_shop.duckdb docs/examples/jaffle_data/demo/ docs/examples/jaffle_data/slayer_models/ + +# Auto-generated by the dbt MetricFlow demo (clone + DuckDB + converted models) +docs/examples/11_dbt_metricflow/.cache/ + +# Auto-generated by the OSI import demo (DuckDB + converted models) +docs/examples/13_osi_import/.cache/ diff --git a/.readthedocs.yaml b/.readthedocs.yaml deleted file mode 100644 index b64948be..00000000 --- a/.readthedocs.yaml +++ /dev/null @@ -1,13 +0,0 @@ -version: 2 - -build: - os: ubuntu-22.04 - tools: - python: "3.12" - -mkdocs: - configuration: mkdocs.yml - -python: - install: - - requirements: docs/requirements.txt diff --git a/CLAUDE.md b/CLAUDE.md index 58e8fdc4..c9a59e03 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,260 +1,104 @@ # CLAUDE.md -This file provides guidance to Claude Code when working with code in this repository. +Guidance for Claude Code when working in this repository. ## What is SLayer? -SLayer (Semantic Layer) is a lightweight, open-source (MIT) semantic layer for AI agents, built by MotleyAI. Instead of writing raw SQL, agents describe what data they want — measures, dimensions, filters — and SLayer generates and executes the query. +SLayer (Semantic Layer) is a lightweight, open-source (MIT) semantic layer for AI agents, +built by MotleyAI. Instead of writing raw SQL, agents describe what data they want — +measures, dimensions, filters — and SLayer generates and executes the query. -Default API port: **5143**. +Server ports: REST API 5143, Flight SQL 5144, Postgres facade 5145. -When generating SLayer query examples or answering questions about SLayer syntax and capabilities, always read the documentation files in `docs/` first (especially `docs/concepts/queries.md`, `docs/concepts/formulas.md`, `docs/concepts/models.md`, and `docs/examples/`) to understand the current syntax and features. +When writing SLayer query examples or answering questions about syntax and capabilities, +read `docs/` first — especially `docs/concepts/queries.md`, `docs/concepts/formulas.md`, +`docs/concepts/models.md`, and `docs/examples/`. -## Common Commands - -```bash -# Install with all extras -poetry install -E all - -# Run unit tests (excludes integration tests) -poetry run pytest - -# Run SQLite integration tests -poetry run pytest tests/integration/test_integration.py -m integration - -# Run Postgres integration tests (auto-spawns temp Postgres via pytest-postgresql) -poetry run pytest tests/integration/test_integration_postgres.py -m integration - -# Run DuckDB integration tests (no Docker, runs in-process) -poetry run pytest tests/integration/test_integration_duckdb.py -m integration +## Layout -# Run a specific test file -poetry run pytest tests/test_sql_generator.py -v +`slayer/core` domain models & errors; `slayer/engine` query engine; `slayer/sql` SQL +generation + dialects; `slayer/storage` YAML/SQLite backends + migrations; `slayer/api` +REST; `slayer/mcp` MCP server; `slayer/flight` / `slayer/pg_facade` / `slayer/facade` +BI wire protocols; `slayer/memories` + `slayer/search` agent memory & semantic search; +`slayer/cli.py` CLI. -# Start API server (uses platform default storage path, override with --storage) -poetry run slayer serve - -# Start MCP server -poetry run slayer mcp +## Common Commands -# Lint -poetry run ruff check slayer/ tests/ +```bash +poetry install -E all # install with all extras +poetry run pytest -m "not integration" # unit tests (excludes integration) +poetry run pytest tests/integration/ -m integration # all integration tests +poetry run pytest tests/test_sql_generator.py -v # one file +poetry run slayer serve # REST API server +poetry run slayer mcp # MCP server +poetry run ruff check slayer/ tests/ # lint ``` +All CLI commands accept `--storage` (YAML dir or `.db` file); defaults to the platform +data dir, override with `$SLAYER_STORAGE`. + ## Key Conventions - Python 3.11+, Pydantic v2 for all models +- NEVER use dataclasses — use Pydantic classes instead - Use `poetry run` for all Python commands - Use keyword arguments for functions with more than 1 parameter - Imports at the top of files -- SQL generation uses sqlglot AST building (not string concatenation) -- **Two reference modes** (DEV-1369): SLayer has exactly two expression layers and the rules differ by design. **Mode A (SQL)** covers `Column.sql`, `Column.filter`, and `SlayerModel.filters` — sqlglot-parsed free SQL accepting any function call (`json_extract`, `coalesce`, `CASE WHEN`, …), bare names referencing the underlying table, and `__`-delimited join paths (`customers__regions.name` — `__` between hops, single dot before the leaf). **Mode B (DSL)** covers `ModelMeasure.formula`, `SlayerQuery.measures`, `SlayerQuery.filters`, and every other query field — Python-AST DSL accepting only `Column` / `ModelMeasure` references, single-dot dotted paths through joins, aggregation colon syntax (`revenue:sum`, `*:count`), transform calls, and arithmetic/boolean ops. DSL mode rejects raw SQL function calls, `__` in user input, and bare names that don't resolve. The internal carve-out: `Column.name` accepts `__` because `_query_as_model` flattens joined columns into virtual-model columns like `stores__name`. Single source of truth: [docs/concepts/references.md](docs/concepts/references.md). Predicate-promotion (DEV-1336) is removed — a query filter naming a `Column` whose `sql` contains a window function now raises with a suggestion to use a rank-family transform (`rank() <= N`, etc.). DEV-1378 closed two implementation gaps: (1) Mode A model filters now route through `parse_sql_predicate` at enrichment time (previously they hit the DSL parser and `Column.filter`/`SlayerModel.filters` containing arbitrary SQL functions raised at runtime); (2) `SlayerQuery.filters` (Mode B) accepts a small lowercase allowlist of string-hygiene scalars — `lower`, `upper`, `trim`, `replace`, `substr`, `instr`, `length`, `concat`, `like` — plus the SQL `||` concat operator. DEV-1484 wired up the last two (the reference docs had promised them but the typed pipeline rejected both): `||` is desugared to a `concat(...)` scalar call via a `|`/BitOr rewrite in `slayer/engine/syntax.py`, and `like(value, pattern)` is a 2-arg scalar that emits the SQL `LIKE` operator (sqlglot `exp.Like`) — wrap it in `not (...)` for `NOT LIKE`. -- Models, columns, measures (formulas), and aggregations have an optional `meta: Dict[str, Any]` field for arbitrary user-defined JSON metadata. Persisted in storage, editable via MCP (`edit_model`), HTTP API, and CLI. `inspect_model` renders `meta` for any entity that has it set; the column is auto-pruned when no entity in the section uses meta. -- **Schema versioning**: `SlayerModel`, `SlayerQuery`, and `DatasourceConfig` carry a `version: int` (currently `7` for `SlayerModel`, `3` for `SlayerQuery`, `1` for `DatasourceConfig`). On load, older versions are upgraded via the converter chain in `slayer/storage/migrations.py` before Pydantic validates the dict. Saves always emit the current version. The hook is on the Pydantic class itself (`@model_validator(mode="before")`), so every storage backend — YAML, SQLite, third-party backends, plus MCP/API/dbt entry points — gets migrations automatically without backend changes. The v1→v2 converter (in `slayer/storage/v2_migration.py`) merges v1 `dimensions`+`measures` into v2 `columns`, repurposes `measures` to hold `ModelMeasure` formulas, and renames `SlayerQuery.fields`→`measures`. The v2→v3 converter (in `slayer/storage/v3_migration.py`) drops the legacy `dry_run`/`explain` fields from `SlayerQuery` (they are now engine kwargs only — `engine.execute(query, dry_run=..., explain=...)`) and walks `SlayerModel.source_queries` entries through the SlayerQuery chain. The v3→v4 converter (in `slayer/storage/v4_migration.py`) requires non-empty `data_source` on table-backed SlayerModel dicts (`sql_table` or `sql` mode); query-backed models (`source_queries` set) are exempt because their `data_source` is filled by `engine._validate_and_populate_cache` from the resolved virtual model before save. The v4 converter also ships layout migrators that move legacy `models/.yaml` flat files into `models//.yaml` and rebuild the SQLite `models` table with a composite `(data_source, name)` primary key. The v4→v5 converter (in `slayer/storage/v5_migration.py`, DEV-1361) coarse-renames `Column.type` legacy values to the new sqlglot-aligned vocabulary (`string→TEXT`, `number→DOUBLE`, `time→TIMESTAMP`, etc.) and strips the dead aggregation pseudo-types (`count`/`sum`/...). The v5→v6 converter (in `slayer/storage/v6_migration.py`, DEV-1375) is a no-op forward — v6 introduces a single new optional field, `Column.sampled: Optional[str]`, that caches the per-column sample-value snapshot consumed by [`search`](docs/concepts/search.md) and `inspect_model`. The v6→v7 converter (in `slayer/storage/v7_migration.py`, DEV-1480) is also a no-op forward — v7 adds two more `Column` fields: `sampled_values: Optional[List[str]]` (the structured top-50-by-frequency list, paired with the existing text `sampled`) and `distinct_count: Optional[int]` (true cardinality at profile time, surfaced via a secondary `count_distinct` query on overflow). Storage backends additionally introspect each model's datasource on first load and refine `DOUBLE → INT` for base columns whose live SQL type is integer (`slayer/storage/type_refinement.py`); the refined model is written back so subsequent loads skip both steps. `SlayerQuery` v3 sets `extra="forbid"` so unknown fields raise. See [docs/concepts/models.md](docs/concepts/models.md#schema-versioning). -- **DataType / CAST emission** (v5, DEV-1361): `DataType` values match sqlglot's `exp.DataType.Type` byte-for-byte: `TEXT`, `INT`, `DOUBLE`, `BOOLEAN`, `DATE`, `TIMESTAMP`. Auto-ingestion narrows integer DB types (INTEGER/BIGINT/SERIAL/INT8…/UINT8…) to `INT`, and floats/numerics (FLOAT/DOUBLE/DECIMAL with scale>0) to `DOUBLE`; NUMERIC(p,0) and DECIMAL(p,0) are integer-shaped and narrow to `INT`. The SQL generator wraps **non-bare** `Column.sql` (function calls, arithmetic, CASE WHEN) in `CAST(... AS )` driven by `Column.type`; bare identifiers and `sql=None` are emitted unchanged. `TEXT` is a no-op (skipped, since `CAST AS TEXT` is cosmetic and doesn't unwrap SQLite's JSON-quoted strings anyway). `ModelMeasure.type` (also `Optional[DataType]`) declares the formula's result type; when set, the aggregation expression is wrapped in an outer CAST. Lenient `before`-validators on `Column.type` and `ModelMeasure.type` absorb legacy lowercase agent input (`"string"` → `TEXT`, `"number"` → `DOUBLE`, etc.) and silently drop dropped pseudo-types. `slayer storage migrate-types --dry-run [--data-source X]` runs the storage refinement step explicitly for batch / inspectable usage. Schema-drift detection (`data_type_bucket`) keeps `INT` and `DOUBLE` in the same `"number"` bucket so a `DOUBLE`-typed persisted column does not flag as drift against an `INT` live column. -- **Datasource-scoped storage** (v4, DEV-1330): Models are keyed by `(data_source, name)`, not bare `name`. Two datasources can share a table name without collision. `storage.get_model(name, data_source=None)` and `storage.delete_model(name, data_source=None)` resolve bare names by: (1) returning the unique match if exactly one model has that name, (2) walking `storage.get_datasource_priority()` to pick the first datasource in the priority list that has the name, (3) raising `slayer.core.errors.AmbiguousModelError` otherwise. The priority list is configured via `storage.set_datasource_priority(["db_a", ...])`, the MCP `set_datasource_priority` tool, the REST `PUT /datasources/priority`, or the `slayer datasources priority` CLI subcommand. `engine.execute(query, data_source=...)` passes a hint that wins over the priority list. Joins resolve targets within the parent model's `data_source` only — cross-datasource joins are not auto-mirrored. A sibling Linear issue ([DEV-1342](https://linear.app/motley-ai/issue/DEV-1342)) tracks whether to add `datasource.model_name` dot syntax inside query strings. -- **Source modes**: a `SlayerModel` has exactly one source: `sql_table` (physical table), `sql` (explicit SQL subquery), or `source_queries` (query-backed: `List[SlayerQuery]`). Validators enforce mutual exclusivity, reject empty `source_queries=[]`, require `name` on every non-final stage, and reject duplicate stage names. `SlayerModel.source_queries` entries given as dicts are auto-parsed into `SlayerQuery` instances by a Pydantic before-validator. -- **Stages form a DAG, not just a chain**: any stage in `source_queries` or a runtime query list may use a named sibling as `source_model` or as `joins.target_model`. The **runtime list path** (`engine.execute(query=[...])`, CLI `slayer query @file.json`, MCP `query_nested`) **auto-sorts** the input via `SlayerQueryEngine._topologically_order_queries` — a hand-rolled Kahn's algorithm (no `graphlib` dep) that reorders so every stage appears after the siblings it references. The last entry of the input stays last as the entry point / DAG root, so the result returned doesn't depend on sort order. Validations performed up front: missing `name` on a non-final entry; duplicate names; self-references; the root being referenced by any other stage (root must be the dependency sink); cycles. Unreachable utility sub-queries are accepted — they flow through the sort but are silently dropped from the emitted SQL since nothing resolves them. Stored `SlayerModel.source_queries` retains strict-order semantics (relies on `_scope_named_queries_to_prior` plus the `_forbidden_sibling_refs_var` ContextVar in `_resolve_model_inner` / `_resolve_join_target`) — YAML-defined source_queries are read top-to-bottom and the strict-order error pins typos at resolve time. The runtime list's reference extractor walks `source_model` (string sibling refs), `ModelExtension.source_name`, and `joins[].target_model` from both dict and typed shapes; formula-only references resolve correctly because the explicit join they require is what the extractor picks up. -- **Query-backed models**: `SlayerModel.query_variables: Dict[str, Any]` provides defaults for `{var}` placeholders in `source_queries`. `engine.create_model_from_query(query, name, variables=..., save=True)` saves a query as a query-backed model and populates the `columns` + `backing_query_sql` cache. `engine.save_model(model)` is the engine-side save helper that runs source-mode validation + cache refresh + persistence; user-supplied `columns` and `backing_query_sql` on a query-backed model are rejected at save with a clear error. The cache is refreshed only on save paths (`engine.save_model` / `create_model_from_query(save=True)`); `engine.execute` never writes to storage, even on stale or empty caches (closes #74 — `tests/test_query_backed_models.py::test_execute_never_writes_to_storage` pins this). -- **Run-by-name execution**: `engine.execute(str, variables=..., dry_run=..., explain=...)` and `execute_sync(str, ...)` run the stored backing query for a query-backed model. Errors surface as `Model '' not found` or `Model '' is not query-backed; pass a SlayerQuery with source_model=''.`. Variable precedence (highest first): runtime kwarg > stage > outer query > model defaults. The `variables=` kwarg works uniformly for str, dict, SlayerQuery, and list inputs. Runtime kwargs are merged into the available variable set; extra keys not referenced by any `{var}` placeholder simply remain unused. `dry_run`/`explain` are engine kwargs (not query fields) and apply to every input shape. Surfaced via REST `POST /query` with `{"name": "...", "variables": {...}}`, MCP `query` tool with `variables=`, CLI `slayer query --variables k=v`. -- **Unified columns** (v2): `SlayerModel.columns: List[Column]` replaces v1's separate `dimensions` and `measures`. A `Column` carries name, sql, type (`DataType`), `primary_key`, `description`, `label`, `hidden`, `format`, `allowed_aggregations` (whitelist), `filter` (CASE-WHEN at aggregation time), `meta`. What a column is "used as" (group-by dim vs aggregation source) is decided per query. -- **Measures are named formulas**: `SlayerModel.measures: List[ModelMeasure]` is a library of saved formulas of shape `{formula, name, label, description}`. Same shape as the inline `SlayerQuery.measures` entries. Queries can reference them by bare name (`{formula: "aov"}`) or expand them inline. -- **Aggregations are query-time**: specified via **colon syntax** in formulas — `"revenue:sum"`, `"*:count"`, `"price:weighted_avg(weight=quantity)"`, `"price:corr(other=quantity)"`. Built-in aggregations: sum, avg, min, max, count, count_distinct, first, last, weighted_avg, median, percentile, stddev_samp, stddev_pop, var_samp, var_pop, corr, covar_samp, covar_pop. Custom aggregations defined at model level in `aggregations` list. -- **`*:count`** for COUNT(*) — `*` means "all rows", `count` is just a regular aggregation. `col:count` = COUNT(col) for non-nulls. -- Columns can have `allowed_aggregations` whitelist — validated at model creation and query time. Primary-key columns are always restricted to `count`/`count_distinct` regardless of type. Default eligibility per data type lives in `slayer/core/enums.py:DEFAULT_AGGREGATIONS_BY_TYPE`. -- Auto-ingestion emits one `Column` per non-joined column. PK columns get `primary_key=True`. Columns named "count" rename to "count_col" to avoid clashing with `*:count`. -- **Idempotent auto-ingestion** (DEV-1356): `slayer ingest` / `ingest_datasource_models` MCP tool / `POST /ingest` REST endpoint are idempotent by default. Re-runs are additive only — new columns/joins/tables are appended; existing column metadata (`description`, `label`, `format`, `meta`, `allowed_aggregations`) is never overwritten. `sql`-mode and query-backed models are skipped silently. The return shape is `IdempotentIngestResult(additions, to_delete, errors)` where `to_delete` is the verbatim `validate_models` output (so type drift surfaces in the same call). Implemented in `slayer/engine/ingestion.py:ingest_datasource_idempotent`. -- **Schema-drift validation** (DEV-1356): `engine.validate_models(data_source=None)` returns the minimal list of deletes (`EditModelDelete` / `WholeModelDelete`) needed for SQL generation to remain valid against the live schema. Read-only — never writes to storage. Surfaced as MCP tool `validate_models`, REST `POST /validate-models`, and CLI `slayer validate-models [--datasource X]`. Compares persisted columns/types/joins to live introspection via SQLAlchemy `Inspector` (sql_table mode) or trial-execute cursor metadata (sql mode). Type comparison uses coarse buckets (`number`/`string`/`boolean`/`temporal`); INTEGER↔FLOAT and DATE↔TIMESTAMP collapse. PK drops do not cascade. Cascade walking stays within the parent datasource. FK introspection limitations: ClickHouse, BigQuery, Snowflake don't expose FK metadata via `Inspector` — joins on those backends must be defined manually. -- **`SchemaDriftError`** (DEV-1356): when `engine.execute()` raises a DBAPI error, the engine attempts to attribute it via `validate_models` against the touched models' datasources. If drift is found, raises `SchemaDriftError(models, to_delete, original)` (with `original` as `__cause__`). Healthy queries pay zero overhead. REST translates to HTTP 422 with `{"error": "schema_drift", "models": [...], "to_delete": [...], "original": "..."}`. -- **`apply_drift_deletes`** (DEV-1356): `await engine.apply_drift_deletes(deletes)` applies each entry via `engine.edit_model_remove` / `engine.delete_model_by_name` and returns `ApplyDriftResult(applied, errors, residual)`. Per-entry failures are captured; processing continues. Surfaced **only** via `slayer validate-models --force-clean [--yes]` — destructive auto-application is opt-in at the CLI layer. Not exposed via MCP or REST. -- Queries support `measures` (renamed from `fields` in v2) — list of `{"formula": "...", "name": "...", "label": "..."}` parsed by `slayer/core/formula.py`. `label` is an optional human-readable display name (also supported on `ColumnRef` and `TimeDimension`). -- **Dim-only queries deduplicate**: a `SlayerQuery` with no measures and at least one dimension/time-dimension auto-emits `GROUP BY ` and returns the distinct combinations (Cube.js semantics). The `GROUP BY` is applied **before** `LIMIT`, so a row cap can never silently drop unique tuples that surface past row N. The rule is unconditional — no opt-out flag. Implemented in `SQLGenerator._generate_base` as `dim_only_dedup = bool(group_by_columns) and not enriched.measures`, OR'd into `needs_group_by` alongside `has_aggregation` / `cross_model_measures` / `skip_isolated`. With aggregating measures present the aggregation `GROUP BY` is already mandatory, so the dim-only term is idempotent. -- **Result column naming**: `revenue:sum` → `orders.revenue_sum` (colon becomes underscore). `*:count` → `orders._count` — the `*` is dropped but the underscore is kept as a leading marker so the alias never collides with a user-defined column literally named `count`. When converting queries to models (`create_model_from_query`), the same colon-to-underscore mapping applies. An explicit `name` on the measure spec overrides the canonical form for **both** simple aggregations and arithmetic/transform formulas — `{"formula": "amount:sum", "name": "rev"}` surfaces as `orders.rev`. This matters most for inner stages of multi-stage `source_queries`, where downstream stages reference inner-stage outputs by the chosen name. **DEV-1443**: when a query measure is renamed via `name`, filters and ORDER BY entries in the same node may reference EITHER the raw colon form (`amount:sum > 100`) OR the user alias (`rev > 100`) — both resolve to the user alias, and a colon-form filter classifies as HAVING. Renaming never changes the legal filter/order form. Two enrichment-time validations on renames: (1) a query measure `name` colliding with a source column on the source model is rejected (the alias-form filter would otherwise silently bind to the source column instead of the aggregate); (2) a rename whose canonical alias (e.g. `revenue_sum`) literally shadows a source column on the same model is also rejected — the colon-form filter would otherwise be ambiguous between the renamed aggregate and the source column. **DEV-1448**: the user-supplied `name` on a *cross-model* aggregated measure (`{"formula": "customers.revenue:sum", "name": "cust_rev"}`) is now honored for the projection alias and the downstream-stage virtual model column. Only the canonical **leaf** of the dotted path is swapped to the user name; the hop path is preserved — same dot-syntax shape as every other multi-hop caller-facing key. So `{"formula": "customers.revenue:sum", "name": "cust_rev"}` surfaces as `orders.customers.cust_rev`, and `{"formula": "customers.regions.population:sum", "name": "region_pop"}` surfaces as `orders.customers.regions.region_pop`. Two caller-facing surfaces: (a) the top-level result-column key uses the hop-preserved form (`orders.customers.cust_rev`); (b) the downstream-stage virtual model column built by `_query_as_model` uses the BARE user name (`cust_rev`) — a special-case in the cross-model loop short-circuits the `__`-flattening of `_alias_to_short` whenever `cm.name` is a bare identifier, so a stage-2 reference like `cust_rev:max` resolves directly. The inner `CrossModelMeasure.measure` (used to build the CTE aggregate expression) intentionally retains the canonical form — only the outer handle is renamed. The canonical-collision guard from DEV-1443 was lifted into a pre-pass so it fires for both local AND cross-model renames symmetrically. Covers `customers.*:count` and `customers.col:count_distinct` cross-model variants. Same-stage **filter** referencing a renamed cross-model measure (`filters=["cust_rev > 100"]` or the colon form `"customers.revenue:sum > 100"`) is NOT auto-resolved — both raise at strict resolution; the SQL generator has no path to route the bare user alias to the cross-model CTE's output column for filter classification, so admitting the filter would emit broken SQL (`WHERE orders.cust_rev > 100` against a column that doesn't exist on the base table). Filter remap stays DEV-1445 territory. **ORDER BY** referencing the bare user alias (`order=[{"column": "cust_rev"}]`) DOES resolve via `SQLGenerator._resolve_order_column`'s `alias_lookup[cm.name] = cm.alias` mapping and emits `ORDER BY "orders.customers.cust_rev"` against the cross-model CTE's output column. **Cross-model parametric result keys** (DEV-1450): a cross-model parametric aggregate keeps its kwarg-signature suffix in the result key — `customers.revenue:percentile(p=0.5)` → `orders.customers.revenue_percentile_p_0_5`, and `p=0.5` / `p=0.95` on the same target column yield two DISTINCT keys. This diverges from legacy, which dropped the suffix so the two variants collided (a bug). Plain `customers.revenue:sum` → `orders.customers.revenue_sum` in both. Workaround until DEV-1445 lands: restructure as a multi-stage `source_queries` so the cross-model measure becomes a local measure in the downstream stage. See companion tickets DEV-1445 (cross-model filter remap) and DEV-1446 (transform-wrapped inner-ref dedup). -- **Response attributes**: `SlayerResponse.attributes` is a `ResponseAttributes` with `.dimensions` and `.measures` dicts, each mapping column alias → `FieldMetadata(label, format)`. Split by type so consumers can distinguish dimension metadata from measure metadata. -- Available formula transforms: cumsum, time_shift, change, change_pct, rank, percent_rank, dense_rank, ntile, first (FIRST_VALUE window ASC), last (FIRST_VALUE window DESC), lag, lead, consecutive_periods. time_shift uses a self-join CTE where the shifted sub-query has the time column expression offset by INTERVAL (calendar-based, gap-safe). change and change_pct are desugared at enrichment time into a hidden time_shift + arithmetic expression. lag/lead use LAG/LEAD window functions directly (more efficient but produce NULLs at edges). Non-transform SQL function calls (`nullif`, `coalesce`, `ln`, `sqrt`, etc.) may also wrap aggregated refs inside arithmetic expressions, e.g. `"*:count / nullif(revenue:max, 0)"` — the call passes through to emitted SQL while the inner refs resolve to their measure aliases -- **Rank-family transforms** (DEV-1353): `rank`, `percent_rank`, `dense_rank`, and `ntile` are timeless window-function transforms emitted as `RANK() / PERCENT_RANK() / DENSE_RANK() / NTILE(n) OVER (... ORDER BY DESC)`. They default to **no `PARTITION BY`** (rank across the entire result set, unlike cumsum/lag/lead which auto-partition by query dimensions), and accept an optional `partition_by=col` or `partition_by=[col1, col2]` kwarg to opt into per-partition ranking; the columns referenced must be query dimensions or time dimensions. `ntile` additionally requires `n=`. Standard SQL across SQLite (≥3.25), Postgres, DuckDB, MySQL, and ClickHouse — no UDFs needed. -- Filters can reference computed field names or contain inline transform expressions (e.g., `"change(revenue:sum) > 0"`, `"last(change(revenue:sum)) < 0"`). These are auto-extracted as hidden fields and applied as post-filters on the outer query -- **Window functions in filters**: filter strings and `ModelMeasure.formula` cannot contain raw `OVER (...)` SQL — SLayer's formula parser is Python-AST-based and rejects with an actionable error pointing at the `rank()` / `first()` / `last()` / `lag()` / `lead()` transforms. Filtering on a `Column` whose `sql` contains a window function is also rejected (DEV-1369; the prior auto-promotion escape hatch from DEV-1336 is removed). For top-N use the inline `rank() <= N` transform (or `dense_rank` / `percent_rank` / `ntile(n=)`); for non-standard window expressions, factor them into an earlier stage of a multi-stage `source_queries` model. -- **Bare same-model derived refs in `Column.sql`** (DEV-1410): A bare identifier inside a `Column.sql` that names a sibling **derived** column on the host model is inlined parenthesised, identical to the qualified form (`A.foo` and bare `foo` produce the same SQL). Inlining is scope-guarded — refs inside a nested scope (sub-query, set-op branch, CTE, `VALUES`) are left alone because they belong to the inner rowset. Cycles in the derived-ref graph (`c1.sql = "c2 + 1"`, `c2.sql = "c1 - 1"`) raise `slayer.core.errors.ColumnCycleError` at `storage.save_model` time, before the model reaches a query. `ColumnCycleError` subclasses both `SlayerError` and `ValueError` so existing `except ValueError` call sites keep working. `StorageBackend.save_model` is a template method that runs the cycle validator before delegating to the backend's `_save_model_impl`; the migration write-back at `_migrate_and_refine_on_load` passes `_validate=False` so legacy cyclic data remains loadable. Save-time validation stays within the model's `data_source`; unresolved join targets are silently skipped (best-effort) — the compile-time guard in `slayer/engine/column_expansion.py` remains authoritative. -- Filters support `{variable}` placeholders substituted from `query.variables: Dict[str, Any]`. Values must be str/number, inserted as-is. `{{`/`}}` for literal braces. Undefined variables raise errors. -- Models can have explicit `joins` to other models (LEFT JOINs). Cross-model measures use dotted syntax with colon aggregation (`customers.revenue:sum`) and multi-hop (`customers.regions.name`). Joins are auto-resolved by walking the join graph. Transforms work on cross-model measures (`cumsum(customers.revenue:sum)`) -- **Filtered-local isolation** (DEV-1503): a host aggregate whose `Column.filter` references a joined table (`loss_payment_amt:sum` where `loss_payment_amt` has `filter="loss_payment.has_flag = 1"`) is hoisted out of the host `_base` SELECT into its own `_cm_*` CTE — same isolation mechanism as forward cross-model aggregates, host-rooted variant. Without isolation, two such measures whose filter joins are different INNER joins on different tables would intersect in the host base to only the rows present in BOTH targets, silently corrupting both aggregates. The planner trigger is structural: cross-model planner fires whenever `AggregateKey.source.path` is non-empty OR `column_filter_key.referenced_join_paths` is non-empty. The plan carries `cte_root_model = host.name` as the host-rooted disambiguator (vs forward / re-rooted, which target-root). An AGGREGATE-phase host filter referencing the isolated aggregate (`loss_payment_amt:sum > 1000`) routes to an outer WHERE on the combined non-aggregating SELECT — not HAVING-into-the-CTE, which would surface host rows as NULL instead of dropping them. Mixed filters (`loss_payment_amt:sum > 1000 AND total_amount:sum > 10`) promote the non-isolated operand into `_base` as a hidden aux slot so the outer WHERE can reference both. See [docs/architecture/cross-model-aggregates.md](docs/architecture/cross-model-aggregates.md#strategy-3-filtered-local-isolation-dev-1503). -- **Path-based table aliases**: Joined tables use `__`-delimited path aliases in SQL to disambiguate diamond joins. In queries, dots denote paths (`customers.regions.name`); in model SQL definitions, `__` denotes the table alias (`customers__regions.name`). For diamond joins (same table reached via different paths, e.g., `orders → customers → regions` AND `orders → warehouses → regions`), each path gets a unique alias (`customers__regions` vs `warehouses__regions`). Auto-ingestion creates only direct joins (one per FK on the source table); multi-hop paths are resolved at query time by walking each intermediate model's own joins -- `SlayerQuery.source_model` accepts a model name, inline `SlayerModel`, or `ModelExtension` (extends a model with extra `columns`/`measures` formulas/`joins`). `create_model_from_query()` saves a query as a permanent model -- Models can have `filters` (always-applied WHERE conditions, e.g., `"deleted_at IS NULL"`) -- **Core principle**: adding a measure/field must never affect result cardinality or other fields' values — achieved via CTEs, sub-queries, and correct JOIN dimensions -- Functions needing time ordering: single time_dimensions entry is used automatically; with 2+ time dimensions, `main_time_dimension` disambiguates (or model's `default_time_dimension` if among query's time dims); with none, falls back to model default -- SlayerModel has optional `default_time_dimension` field for time-dependent formula resolution -- SQLite dialect uses STRFTIME instead of DATE_TRUNC (handled automatically by sqlglot) -- See "Database Support" section below for dialect tiers and testing expectations -- Result column keys use `model_name.column_name` format (e.g., `"orders._count"` for `*:count`, `"orders.revenue_sum"` for `revenue:sum`). For multi-hop joined dimensions, the full path is included: `"orders.customers.regions.name"` -- Datasource configs support `${ENV_VAR}` references resolved at read time -- Integration tests are marked with `@pytest.mark.integration` and skip when DB is unavailable -- NEVER use dataclasses, if you want to use dataclasses, use Pydantic classes instead. - -- **Memories + semantic search** — an agent-memory layer indexed by canonical entity strings. - - **Write side**: `save_memory(learning, linked_entities, id=None)` and `forget_memory(id)`, exposed via MCP, REST (`POST /memories`, `DELETE /memories/{id}`), CLI (`slayer memory {save,forget}`), and `SlayerClient`. `linked_entities` is either a list of entity strings (resolved strictly; `memory:` accepted) or an inline `SlayerQuery` / dict (entities auto-extracted; the query is persisted on the memory). Optional `id` (DEV-1428) pins a user-controlled canonical memory id; duplicate id → unconditional upsert, `created_at` preserved. - - **Read side**: a single `search(entities, query, question, datasource=None, max_memories=5, max_example_queries=2, max_entities=5)` tool. Surfaces: MCP, REST (`POST /search`), CLI (`slayer search …`), `SlayerClient.search()`. DEV-1428: search is **lenient** — unresolved `entities` / `query` tokens become warnings rather than raising; stale memory entity tags are filtered out at retrieval time (belt) before BM25 ranks AND before `matched_entities` is surfaced; for `example_queries` hits whose attached `Memory.query` no longer resolves, a `example_query memory:: attached query has stale references; re-save to clean.` warning is emitted (the query is not rewritten). - - **Canonical entity form** is ``, `.`, `..`, or — DEV-1428 — `memory:` (cross-memory references). Aggregation suffixes are stripped (`revenue:sum` → `..revenue`); `*:count` collapses to the source model; multi-hop paths keep only the leaf. Resolver: `slayer/memories/resolver.py` (the `memory:` branch runs at the top of `resolve_entity`, before `_strip_agg_suffix`, so `memory:abc` parses as the memory branch). Memory ids are non-empty strings (DEV-1428) — pure-digit auto-allocated by the storage layer (`"1"`, `"2"`, ...), or user-supplied (`"kb.policy.42"`); forbidden charset: `:`, `/`, `?`, `#`, whitespace, ASCII control. Bare names never resolve to memories (the `memory:` prefix is mandatory). `delete_memory` cascades to the matching embedding row AND strips every `memory:` reference to it from every other memory's `entities` (DEV-1428 cascade layer 1). - - **Cascade-on-delete** (DEV-1428). `delete_model` / `delete_datasource` / `forget_memory` / `edit_model_remove` strip dangling refs from every memory's `entities` list. Match predicate splits by kind: `.[.]` matches exactly OR as a strict dotted-path descendant (`mydb.orders` strips both `mydb.orders` and `mydb.orders.amount`; `mydb.orders_archive` is NOT touched); `memory:` is exact-match only (`memory:42` does not strip `memory:421`). Memories with zero entities after the strip are kept (learning text stands alone). Memory embedded-text (`render_memory_text_for_embedding`) is `learning` only — tags excluded — so cascade-strip writes don't change the embedding content hash and no embedding refresh fires. Storage-layer concern, lives in `StorageBackend.strip_dangling_entities_from_memories`; cascade writes go through `_save_memory_row` directly (bypassing `MemoryService.save_memory`). Ingest-time cleanup (`slayer ingest` / `--ingest-on-startup`) re-walks each memory's refs and strips ones that resolve to a definitive "not found"; transient lookup failures keep the ref intact. Stale `Memory.query` on example-queries memories surfaces as `IngestionError(model_name="memory:")` rather than a rewrite. - - **Search runs three channels**, fused via Reciprocal Rank Fusion (`k=60`): - - **BM25** over each memory's stored entity tags (`rank_bm25.BM25Plus`). - - **Tantivy** in-memory full-text index, built fresh per call over memories ∪ non-hidden entities (datasources / models / columns / named measures / aggregations), using the `en_stem` analyzer. - - **Embeddings** (optional `embedding_search` pip extra) — dense cosine over a persistent `embeddings` sidecar keyed by `(canonical_id, embedding_model_name)`. Model from `SLAYER_EMBEDDING_MODEL` (default `openai/text-embedding-3-small`), dispatched via litellm. When the extra is missing, no API key is set, or the corpus is empty, the channel contributes nothing and emits one warning into `SearchResponse.warnings`. - - BM25 contributes to memory ranking only; entity hits are RRF-fused across tantivy + embeddings. Memory hits are partitioned by `Memory.query is None` into `memories` (learning-only) and `example_queries` (query-bearing), each with its own cap. Each output bucket is ranked independently of the others — varying one `max_X` cap cannot reorder or move items in/out of any other bucket. The in-memory tantivy index is built with `writer(num_threads=1)` so doc-id tiebreaks on equal BM25 scores are deterministic. Empty-input fallback returns the newest memories per bucket with a warning. - - **Indexed text** is rendered by `slayer/search/render.py`. Hidden models / columns are excluded; `meta` is never indexed. Named children (columns, measures, aggregations, join targets) are referenced by name + kind only (each child has its own indexed doc). - - **`datasource` filter**: all surfaces accept optional `datasource: Optional[str] = None`. When set, every channel pre-filters its corpus to canonical ids rooted at that datasource (exact name or strict dotted-path descendant); memories surface iff at least one of their `entities` is rooted there. Unknown datasource → `ValueError` (HTTP 400 on REST). - - **Embedding refresh** runs inline on `slayer ingest`, `edit_model`, `save_memory`, and `--ingest-on-startup`. Each per-datasource ingest pass refreshes the datasource doc, every visible model + its visible children, and every memory whose canonical entities are rooted at the datasource. Content-hash skips the litellm call when nothing has changed; the hot path issues one batched read + one batched write per refresh, independent of subtree size. Per-entity failures are non-fatal; per-memory failures surface as `IngestionError(model_name="memory:", …)` in `IdempotentIngestResult.errors`. - - **Embedding storage**: `SQLiteStorage` writes embeddings into the main `.db`; `YAMLStorage` uses a sidecar `/embeddings.db` so the YAML store stays git-diffable. Both go through `slayer/storage/sidecar_embedding_store.py`. Cascade-delete on a `canonical_id` matches exactly or as a strict dotted-path descendant — never as a character prefix. - - **Sample-value snapshots** are cached on `Column.sampled` (text), `Column.sampled_values` (structured top-50 list for categorical columns, DEV-1480), and `Column.distinct_count` (true cardinality for categorical columns, DEV-1480). Refreshed on `slayer ingest` (table-backed models only), on `slayer search refresh-samples`, on `edit_model` (column edits → that column; model-level changes to `filters` / `sql` / `source_queries` → every column), and lazily on `inspect_model` cache miss (best-effort write-back). Categorical columns are ordered by count desc with alphabetical tie-break; the structured list is the consumer-facing way to compare predicate literals against actual stored values (text-split on `sampled` is ambiguous for values containing commas, e.g. `"R$ 1,000–3,000"`). Cache validity for categorical columns requires `sampled_values is not None` (v6 → v7 upgrades re-profile on next `inspect_model`). sql-mode and query-backed models do not yet have sample-value coverage. - - `inspect_model` auto-renders a `Learnings` section showing only learning-only memories (`query is None`); query-bearing memories surface only via `search` in the `example_queries` bucket. - - See [docs/concepts/memories.md](docs/concepts/memories.md) and [docs/concepts/search.md](docs/concepts/search.md). - -## Async Architecture - -- **Engine is async-first**: `SlayerQueryEngine.execute()` is `async`. Use `execute_sync()` for CLI/notebooks/scripts. -- **Storage backends are async**: All `StorageBackend` methods are `async def`. YAMLStorage uses sync I/O inside async (fast local files). SQLiteStorage uses `asyncio.to_thread`. Future Postgres storage can use true async (asyncpg). -- **SQL client**: Uses native async drivers for Postgres (`asyncpg`) and MySQL (`aiomysql`). Falls back to `asyncio.to_thread` for SQLite, DuckDB, ClickHouse. Connection pools are cached per `SlayerSQLClient` instance. -- **Tests use `pytest-asyncio`** with `asyncio_mode = "auto"` — test functions can be `async def` and `await` directly. -- **Sync wrappers**: `run_sync()` in `async_utils.py` bridges async→sync for CLI and MCP tools. Handles both "no event loop" and "inside Jupyter" cases. -- **Client mirrors engine union** (DEV-1437): every `SlayerClient` query entry point — `query`, `query_sync`, `sql`, `sql_sync`, `explain`, `explain_sync`, `query_df` — accepts the same input shapes as `engine.execute`: `SlayerQuery | dict | list[SlayerQuery | dict] | str`. The list form is the multi-stage DAG (`{"queries": [...]}` body in HTTP mode, same shape as MCP `query_nested`); `str` is run-by-name. `SlayerClient._build_query_body` is the single source of truth for the HTTP body shape; the local-engine path defers all validation to `engine.execute_sync` / `engine.execute`. `variables=` and `data_source=` kwarg forwarding is tracked separately as DEV-1438. - -## Flight SQL - -- Port **5144** by default (one above the REST API's 5143). `slayer flight-serve [--host HOST] [--port PORT] [--storage PATH] [--token T] [--tls-cert C] [--tls-key K] [--demo]`. Wire-compatible with the upstream Apache `flight-sql-jdbc-driver` v18.3.0 — same JAR the dbt Semantic Layer connectors use. Lives in `slayer/flight/`. -- **Loopback no-token fallback** (auth.py): non-loopback binds without a `--token` (or `$SLAYER_FLIGHT_TOKEN`) are refused at startup. With `--demo` and no explicit `--host` or `--token`, the effective host defaults to `127.0.0.1` so the no-token fallback applies cleanly. -- **Stateless server**: the prepared-statement `handle` and Flight `Ticket.ticket` both carry the **original UTF-8 SQL bytes** (the ticket wraps them in `TicketStatementQuery` for ticket-shape conformance). `ActionClosePreparedStatementRequest` is a no-op — nothing to free. -- **Path A vs Path B** (the "LIMIT 0 two-round-trip" story): the JDBC driver always routes `executeQuery` through the prepared-statement triplet. The translator/handler chain runs three times per BI query — once on `CreatePreparedStatement`, once on `get_flight_info(CommandPreparedStatementQuery)`, once on `do_get`. Database round-trips stay at two (`LIMIT 0` for schema validation, then full). -- **Catalog convention**: dotted form end-to-end — `customers.regions.name`. Same form in `INFORMATION_SCHEMA.*`, in the BI-tool projection list, in `WHERE`, and in the SLayer DSL. No `__` → `.` rewrite step in the translator. -- **`Any` wrapping** (server.py / handlers.py): the Apache JDBC driver wraps every `do_action` body AND expects every `do_action` response body to be `google.protobuf.Any`-wrapped (`type_url` = the action class's full name); the pyarrow-flight Python client sends raw bytes. `_parse_action_body` accepts both; response always sends an `Any`. Don't strip the wrapper. -- **JDBC `token=X` is Phase 2** — the Apache driver pre-handshakes the bearer token. SLayer's middleware validates headers per RPC, not via handshake, so JDBC clients using `token=X` get `UNIMPLEMENTED` during handshake. The pyarrow Python client works because it sets per-call `Authorization` headers. `tests/integration/test_integration_flight.py::test_auth_positive` is `xfail(strict=True)` so a future handshake-handler implementation auto-promotes to PASSED. -- **JVM `--add-opens` for Arrow on Java 17+**: the upstream `flight-sql-jdbc-driver` reflectively pokes `java.nio.Buffer.address`, blocked by strict module access on Java 17+. The JayDeBeAPI integration tests pre-start JPype's JVM with `--add-opens=java.base/java.nio=ALL-UNNAMED` (+ `java.lang` + `java.util`) — see `tests/integration/conftest.py:_ensure_jvm_started_for_arrow`. Document this for DBeaver users. -- **Wire-capture story**: `tests/flight/fixtures/CAPTURE-FINDINGS.md` is the canonical record of what the upstream JDBC driver emits during a real session; `capture-latest.jsonl` holds the JSONL trace. Refresh by running `poetry run python tests/flight/capture_dbt_jdbc.py` (requires Java + Maven Central access for the JAR). -- **Test fixtures**: `jdbc_jar` auto-downloads + caches the JAR into `tests/.cache/`; `jaydebeapi_connect` is a connect factory; `capture_stub` boots a recording-only Flight stub. Java-free integration coverage is in `tests/integration/test_integration_flight_pyarrow_client.py`. -- **Wire schema is catalog-declared in Phase 1**: derived from `QueryResult.projection_types` (`Column.type` for dims, `ModelMeasure.type` for measures). The `LIMIT 0` engine call still runs for validation. A `ModelMeasure` with a wrong/absent `type` surfaces as `ArrowTypeError` over the wire — tighten by setting `ModelMeasure.type`. Phase 2 issue: drive the wire schema from the actual LIMIT-0 execution. - -## Shared facade layer (`slayer/facade/`) - -- **DEV-1486 refactor**: the facade-agnostic half of the Flight SQL facade was extracted into `slayer/facade/` so the Postgres facade can reuse it without an Arrow dependency. Moved: `translator.py` (the SQL→`SlayerQuery` pipeline), `catalog.py` (renamed `Flight*`→`Facade*`: `FacadeCatalog`/`FacadeTable`/`FacadeMetric`/`FacadeDimension`/`FacadeSchema`, `build_catalog`), `info_schema.py`, `probe_queries.py`, plus new `rows.py` (`RowBatch` — pyarrow-free typed columns + row dicts) and `datatypes.py` (`SUPPORTED_DATATYPES`, `datatype_to_jdbc`). `slayer/flight/{translator,catalog,info_schema,probe_queries}.py` are now thin **shims**: they re-export the shared symbols under the historical `Flight*` names and re-wrap `RowBatch`→`pa.Table` at the edge (`slayer/flight/types.py:row_batch_to_arrow`), so existing Flight imports/tests are unchanged. -- **`translate(sql, catalog, *, dialect=None, probe_matcher=None, catalog_matchers=())`**: `dialect` is passed to the sqlglot parser ONLY (it does not widen the query surface). `probe_matcher` overrides the default Flight probe whitelist (the pg facade injects a datasource-aware one). `catalog_matchers` are extra canned-table matchers tried after INFORMATION_SCHEMA (the pg facade injects `match_pg_catalog`). `NoOpResult` carries a facade-neutral `command_tag` (`BEGIN`/`COMMIT`/`SET`/…); Flight ignores it, the pg facade uses it to drive transaction state. New `PgCatalogResult`. `QueryResult.facade_table` (was `flight_table`). -- **Aggregate-SQL → metric mapping** (DEV-1486 decision 21, shared/facade-agnostic — Flight gains it too): `SUM(col)`/`AVG`/`MIN`/`MAX`/`COUNT(col)`/`COUNT(*)`/`COUNT(DISTINCT col)` in a projection map to a query measure (`col:sum`, `*:count`, `col:count_distinct`), validated by looking the colon-form formula up in the catalog's pre-expanded metrics (so eligibility = the same `_eligible_aggregations` rules). Also covers `ORDER BY ` and `HAVING ` (emitted as a colon-form filter). Aggregating over a saved `ModelMeasure` or a non-column expression raises with a pointer to **DEV-1493** (multi-stage rewrite). In practice the mapping is **base-column only**: a joined-column aggregate (`COUNT(customers.region)`) resolves to a cross-model metric whose *projection* is a pre-existing unsupported path (`SlayerQuery` measure names can't contain dots — DEV-1448 territory), so it fails identically to projecting that metric by name. - -## Postgres facade (`slayer/pg_facade/`) - -- **DEV-1486**: a hand-rolled, read-only Postgres wire-protocol v3 server (no new runtime dep) on port **5145** (REST 5143 + Flight 5144 + 2). Lets BI tools with a Postgres connector (Metabase/Superset/Tableau/Power BI/Looker), `psql`, `asyncpg`, `psycopg` connect to SLayer as if it were Postgres. `slayer pg-serve [--host H] [--port 5145] [--storage P] [--token T] [--tls-cert C] [--tls-key K] [--demo]`; env `$SLAYER_PG_TOKEN`. Reuses the shared `slayer/facade/` translator with `dialect="postgres"`. -- **Multi-datasource routing**: the client's startup `database` parameter scopes the connection to ONE SLayer datasource; that datasource's models appear under PG schema `public`. `current_database()` → datasource name; `current_schema()` → `public`. Unknown/missing `database` → `FATAL: database "" does not exist` (SQLSTATE `3D000`) at startup. Cross-datasource queries are not supported. -- **Auth**: `AuthenticationCleartextPassword` + optional `--token` (loopback-no-token fallback, identical bind rules to Flight — `slayer/pg_facade/auth.py`). TLS upgrade when `--tls-cert`/`--tls-key` set; otherwise reply `N` to `SSLRequest`. `verify_password` is constant-time; `None` token accepts any non-empty password, empty is always rejected. -- **Wire format**: simple-query (`Q`) results are always text; extended-query (`Parse`/`Bind`/`Describe`/`Execute`/`Sync`) honours the per-column result-format codes from `Bind` — both **text and binary** encoders exist for the 6 types (asyncpg, the integration-test client, requests binary). Binary follows `integer_datetimes=on`: int8 big-endian, float8 IEEE, date int32 days-since-2000-01-01, timestamp int64 microseconds-since-2000-01-01. `value_to_text` emits Postgres-format floats (`NaN`/`Infinity`/`-Infinity`) and space-separated timestamps. Type→OID: `TEXT→25`, `INT→20`, `DOUBLE→701`, `BOOLEAN→16`, `DATE→1082`, `TIMESTAMP→1114`; unknown→text. **Only these built-in OIDs are ever emitted**, so asyncpg never triggers its `pg_type` introspection path. -- **Bound parameters** are **literal-substituted**: each value is decoded (text/binary per the `Bind` format codes) and rendered as a safe quoted SQL literal, then `$N` is replaced in the SQL before translation. `ParameterDescription` infers the parameter count from the `$N` placeholders (asyncpg leaves `Parse` parameter OIDs empty), defaulting unspecified params to text so they arrive text-encoded. -- **Transaction state machine** (per connection, `slayer/pg_facade/connection.py`): `I`/`T`/`E` reported on every `ReadyForQuery`. `BEGIN`/`START TRANSACTION`→`T`, `COMMIT`/`ROLLBACK`/`END`→`I`, an error while in `T`→`E` (then `25P02` until tx end). A **single simple-query `Q` message may carry multiple semicolon-separated statements** — each gets its own `CommandComplete`, with exactly one final `ReadyForQuery`. `max_rows` on `Execute` is ignored (no `PortalSuspended`). Empty query → `EmptyQueryResponse`. `FunctionCall`/`CopyData`/`CopyDone`/`CopyFail` → `0A000`. -- **`pg_catalog`** (`slayer/pg_facade/pg_catalog.py`): Phase 1 covers `pg_namespace`, `pg_class`, `pg_attribute`, `pg_type`, `pg_proc`, `pg_settings`. `WHERE` is ignored (returns all rows; client filters); both `pg_catalog.X` and bare `X` resolve. OIDs are deterministic via `zlib.crc32` over a namespaced `.[.]` string (NOT builtin `hash()`, which is per-process salted), with a collision check at build time. **Datasource-aware probes** (`slayer/pg_facade/probes.py`): `version()`→PostgreSQL string, `current_database()`→datasource, `current_schema()`→`public`, `SHOW `→row, `current_setting('jit')`→`off`, `set_config('jit',…)`→value; falls back to the shared `match_probe` for `SELECT 1` / `SELECT NULL WHERE 1=0`. -- **New `[pg_facade]` extra** (empty — pure stdlib; keeps the `pip install motley-slayer[pg_facade]` install path consistent). Integration tests (`tests/integration/test_integration_pg_facade.py`) drive a live demo server with `asyncpg`. See [docs/interfaces/pg-facade.md](docs/interfaces/pg-facade.md). - -## CLI - -- All commands accept `--storage` (directory for YAML, `.db` file for SQLite). Defaults to platform-appropriate path (`~/.local/share/slayer` on Linux, `~/Library/Application Support/slayer` on macOS, `%LOCALAPPDATA%\slayer` on Windows). Override with `$SLAYER_STORAGE` env var. -- `slayer query` supports `--dry-run` (preview SQL) and `--explain` (execution plan, dialect-aware). -- `slayer datasources create-inline` supports `--password-stdin` for secure credential input. -- `slayer datasources test` verifies connectivity. -- `slayer datasources create demo [--ingest]` spins up the bundled Jaffle Shop DuckDB (idempotent). `slayer serve --demo` and `slayer mcp --demo` do the same at server startup. Requires the `duckdb` extra and `jafgen` (git-only install); missing deps trigger a clean install-hint message. Lives in `slayer/demo/jaffle_shop.py`. -- `slayer serve --ingest-on-startup` and `slayer mcp --ingest-on-startup` (DEV-1392) — opt-in boot-time idempotent auto-ingestion across every configured datasource, sync-before-listen (uvicorn/mcp.run don't start until ingest finishes). Continue-on-failure: per-datasource errors are friendly-formatted to stderr and never abort startup; `storage.list_datasources()` raising is the only thing that prevents the server from starting. `to_delete` drift entries are printed but **never auto-applied** — destructive cleanup stays gated behind `slayer validate-models --force-clean [--yes]`. Composes freely with `--demo` (demo first, then the ingest pass over every datasource including the freshly-created demo). Also exposed via `SLAYER_INGEST_ON_STARTUP=1` env var (flag wins when both set) and the `ingest_on_startup=True` kwarg on `create_app` / `create_mcp_server`. All output goes to stderr — `slayer mcp` stdio JSON-RPC remains protocol-safe. Orchestrator: `slayer/engine/ingestion.py::ingest_all_datasources_idempotent`. **Memory embeddings** (DEV-1416): each per-datasource pass also re-embeds every memory whose canonical entities are rooted at the datasource, so a stale `embeddings.db` is repaired by the next `--ingest-on-startup` without extra steps. See [docs/concepts/ingestion.md](docs/concepts/ingestion.md#ingesting-at-startup). -- `slayer validate-models [--datasource X] [--force-clean] [--yes]` (DEV-1356) — read-only diff against live schemas; with `--force-clean`, prompts to apply each delete via `engine.apply_drift_deletes`. See [docs/concepts/schema-drift.md](docs/concepts/schema-drift.md). -- `slayer storage migrate-types [--data-source X] [--dry-run]` (DEV-1361) — refine `DOUBLE → INT` on base columns whose live SQL type is integer for every persisted model, then write the refined v5 dict back. Hard-fails if a datasource is unreachable. The same refinement runs transparently inside `storage.get_model` on first load; this CLI is a batch / inspectable alternative. -- `slayer search [--entity ENT ...] [--query JSON_OR_@FILE] [--question TEXT] [--datasource DS] [--max-memories N] [--max-example-queries N] [--max-entities N] [--format json|text]` (DEV-1375 / DEV-1386 / DEV-1409) — up to three-channel semantic search over memories + canonical entities (BM25 over memory entity tags + tantivy full-text + optional dense embedding similarity). `--datasource` scopes the corpus to one datasource (entity hits + memories pre-filtered). See [docs/concepts/search.md](docs/concepts/search.md). -- `slayer search refresh-samples [--data-source X] [--model M ...]` (DEV-1375) — re-profile and persist `Column.sampled` for table-backed models. Best-effort: per-column failures are reported but don't abort. -- MCP `query()` tool has a `format` parameter: `"markdown"` (default), `"json"`, or `"csv"`. -- **`query_nested` MCP tool**: companion to `query` for the multi-stage DAG shape that `engine.execute(query=list[...])` already supports. Takes `queries: List[Dict[str, Any]]` plus the usual `variables` / `show_sql` / `dry_run` / `explain` / `format` knobs. Earlier entries are named sub-queries that later entries reference via `source_model: ""` or `joins.target_model`; the engine auto-sorts the list (Kahn's algorithm), so order doesn't matter. The single-stage `query` tool is unchanged — keep using it whenever the typed per-field schema fits, since it surfaces a richer signature to agents. -- **REST `POST /query` accepts both shapes**: body is `Union[QueryRequest, QueryListRequest]` (FastAPI auto-discriminates by the presence of `queries`). Single-query body is unchanged; multi-stage body is `{"queries": [{...}, ...], "variables": {...}, "dry_run": ..., "explain": ...}` — mirrors `query_nested` and `engine.execute(query=list[...])`. `QueryRequest.source_model` accepts a string (stored model name) or a dict (inline `ModelExtension` / `SlayerModel`) for both single and list shapes. The CLI route is `slayer query @file.json` — the CLI parser also accepts both a single object and a top-level list. +- SQL generation uses sqlglot AST building, not string concatenation +- Async-first: engine and storage methods are async; `execute_sync()` / `run_sync()` bridge for CLI/scripts +- Core principle: adding a measure/field must never change result cardinality or other fields' values +- Two expression layers — Mode A: free SQL in `Column.sql` / model `filters` (`__`-delimited join paths); Mode B: Python-AST DSL in formulas and query fields (dotted paths, colon aggregations, scalar-allowlist functions only). Rules: `docs/concepts/references.md` +- Aggregations are query-time, colon syntax: `revenue:sum`, `*:count` for COUNT(*), `price:percentile(p=0.9)` +- Result column keys are `model.column`: `revenue:sum` → `orders.revenue_sum`, `*:count` → `orders._count`; joined dimensions keep the full path (`orders.customers.regions.name`) +- Dots denote join paths in queries (`customers.regions.name`); `__` denotes path aliases in model SQL (`customers__regions.name`) +- Models are keyed by `(data_source, name)`; joins resolve within the parent model's datasource +- Models/queries/datasource configs carry a `version` field; storage migrations run automatically on load (`slayer/storage/migrations.py`) +- Filters support `{variable}` placeholders from `query.variables`; datasource configs support `${ENV_VAR}` ## Database Support -SLayer uses sqlglot for dialect-aware SQL generation. Databases are supported at two tiers: - -**Tier 1 — fully tested** (integration tests + Docker examples, must not regress): -- **SQLite** — integration tests in `tests/integration/test_integration.py`, embedded example -- **Postgres** — integration tests in `tests/integration/test_integration_postgres.py`, Docker example -- **DuckDB** — integration tests in `tests/integration/test_integration_duckdb.py` (no Docker, runs in-process) -- **MySQL** — Docker example with `verify.py` -- **ClickHouse** — Docker example with `verify.py` - -**Tier 2 — code-covered** (unit tests for SQL generation, no live instance verification): -- Snowflake, BigQuery, Redshift, Trino/Presto, Databricks/Spark, MS SQL Server, Oracle - -Dialect mapping lives in `query_engine.py:_dialect_for_type()`. Dialect-specific SQL lives in `generator.py` — mainly `_build_date_trunc` (SQLite branch), `_build_time_offset_expr` (date arithmetic for shifted CTEs), `_build_median`, `_build_percentile`, and `_build_stat_agg` (stddev/var/corr). Calendar-based time shifts use timestamp offset inside DATE_TRUNC with simple equality joins (no per-dialect join logic). All other SQL differences are handled by sqlglot transpilation. When adding a new dialect: add it to `_dialect_for_type`, add a `_build_time_offset_expr` branch if it doesn't use Postgres-style `INTERVAL`, and add parameterized tests in `TestMultiDialectGeneration`. - -**Aggregation caveats:** -- **SQLite**: `median`, `percentile_cont`, `percentile_disc`, `stddev_samp`, `stddev_pop`, `var_samp` (also aliased as `variance`), `var_pop` (also aliased as `variance_pop`), `corr`, `covar_samp`, `covar_pop` are provided via Python aggregate UDFs registered on every new connection (`slayer/sql/sqlite_udfs.py`); SQLite has no native equivalent. Scalar UDFs `ln`, `log10`, `log2`, `exp`, `sqrt`, `pow`, `power` are also registered there; `log2` overrides SQLite ≥3.35's silent-NULL built-in to keep the strict math-domain-error semantics. The 2-arg `log(B, X)` UDF (returns log_B(X) — base first, value second) is registered on **every** SQLite version, including ≥3.35 where it overrides the built-in's silent-NULL behaviour to match Postgres's strict error semantics. Same B-first arg order in both. -- **ClickHouse**: `percentile` emits the parametric `quantile(p)(x)` syntax; `median` uses native `median(x)`. `stddev_samp`/`stddev_pop`/`var_samp`/`var_pop`/`corr` are native (sqlglot transpiles to dialect-appropriate spelling). -- **MySQL**: `median`, `percentile`, `corr`, `covar_samp`, `covar_pop` are not supported — MySQL has no native function and no Python-UDF mechanism. The generator raises `NotImplementedError` at SQL generation time. Use MariaDB or compute client-side. `stddev_samp`/`stddev_pop`/`var_samp`/`var_pop` are native on MySQL. -- **Postgres / DuckDB**: native `PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY x)` (DuckDB via sqlglot's `QUANTILE_CONT` translation). `STDDEV_SAMP`/`STDDEV_POP`/`VAR_SAMP`/`VAR_POP`/`CORR`/`COVAR_SAMP`/`COVAR_POP` are native on both. - -**In-memory SQLite caveat:** `sqlite:///:memory:` (and equivalent URI variants — `sqlite://`, `sqlite:///file::memory:?…`, `mode=memory`) works across `await` calls on a single `SlayerSQLClient` because the client owns a per-instance `StaticPool` engine with `check_same_thread=False`. Two separate `SlayerSQLClient` instances on `:memory:` are isolated from each other. Use a file path or `mode=memory&cache=shared` URI form to share state across clients. File-backed SQLite is unaffected — it routes through the module-level engine cache as before. - -**SQLite JSON extraction:** `json_extract(col, '$.path')` in `Column.sql` (or any expression `SQLGenerator` parses on SQLite) is preserved as the function-call form, not rewritten to `col -> '$.path'`. The `->` operator in SQLite returns the JSON-quoted form (e.g. `'"Owned"'` with literal quotes), which silently breaks equality / CASE WHEN matches against bare-string literals; the function form returns the unquoted scalar. Implemented via `slayer/sql/sqlite_dialect.py::rewrite_sqlite_json_extract`, applied uniformly through `SQLGenerator._parse`. Use `->>` (`exp.JSONExtractScalar`) directly if you specifically want the dialect operator — SLayer leaves it untouched. - -**`log10` / `log2` literal preservation:** A user-written `log10(x)` or `log2(x)` in `Column.sql` / `ModelMeasure.formula` / filters is emitted verbatim as `log10(x)` / `log2(x)`, not canonicalised to `LOG(10, x)` / `LOG(2, x)`. sqlglot's default behaviour normalises both into a generic `Log(base, expression)` AST and re-emits as `LOG(base, x)`, which is correct numerically but breaks formula-text round-tripping for benchmark agents reading `inspect_model.last_sql` and trips dialects without a 2-arg `LOG`. Implemented via `SQLGenerator._rewrite_log_aliases`, applied through `_parse`. Allowlists in `slayer/sql/generator.py` (`_LOG10_NATIVE_DIALECTS`, `_LOG2_NATIVE_DIALECTS`) cover every supported backend except Oracle (no `LOG10` / `LOG2`) and T-SQL (no `LOG2`); those fall through to the canonical 2-arg form. Other 2-arg calls — `log(3, x)`, `log(some_col, x)` — always emit as `LOG(B, X)`. +- Tier 1 (integration-tested, must not regress): SQLite, Postgres, DuckDB, MySQL, ClickHouse, SQL Server, BigQuery, Snowflake +- Tier 2 (unit-tested SQL generation only): Redshift, Trino/Presto, Databricks/Spark, Oracle +- Per-dialect emission lives in `slayer/sql/dialects/`; tiers and caveats in `docs/database-support.md` ## Testing -**Important**: Always use `poetry run` to run tests — this ensures the correct Poetry-managed virtualenv is used (not the system or conda Python). +Always use `poetry run` (correct Poetry-managed virtualenv). Integration tests are marked +`@pytest.mark.integration` and skip when their DB is unavailable; shared fixtures in +`tests/conftest.py`. ```bash -# Run ALL tests (unit + integration) -poetry run pytest tests/ -m "integration or not integration" -v - -# Run unit tests only (default, excludes integration) -poetry run pytest - -# Run all integration tests -poetry run pytest tests/integration/ -m integration - -# Run specific integration suite -poetry run pytest tests/integration/test_integration.py -m integration # SQLite -poetry run pytest tests/integration/test_integration_postgres.py -m integration # Postgres -poetry run pytest tests/integration/test_integration_duckdb.py -m integration # DuckDB +poetry run pytest -m "not integration" # unit only +poetry run pytest tests/integration/ -m integration # integration +poetry run pytest tests/ -m "integration or not integration" # everything +poetry run pytest -m metabase_e2e tests/integration/test_metabase_e2e.py # live Metabase e2e (needs Docker) ``` -- Unit tests: `tests/test_models.py`, `test_sql_generator.py`, `test_storage.py`, `test_sqlite_storage.py`, `test_mcp_server.py` -- Integration tests (SQLite): `tests/integration/test_integration.py` -- Integration tests (Postgres): `tests/integration/test_integration_postgres.py` — uses pytest-postgresql (auto-spawns temp Postgres) -- Integration tests (DuckDB): `tests/integration/test_integration_duckdb.py` — uses duckdb directly (no Docker) -- Shared fixtures in `tests/conftest.py` - ## Linting -**ALWAYS run the linter at the end of every task and fix any issues before finishing.** - -```bash -poetry run ruff check slayer/ tests/ -``` +ALWAYS run the linter at the end of every task and fix any issues before finishing: -To auto-fix fixable issues: ```bash -poetry run ruff check --fix slayer/ tests/ +poetry run ruff check slayer/ tests/ # check +poetry run ruff check --fix slayer/ tests/ # auto-fix ``` ## Documentation Requirements -**ALWAYS update documentation when making API or user-facing changes.** Check and update ALL of these locations: +ALWAYS update documentation when making API or user-facing changes: + +- `docs/` — concept docs, getting-started, reference, configuration +- `.claude/skills/` — slayer-query.md, slayer-models.md, slayer-overview.md +- When renaming a field or changing a response shape, grep all docs and skills for the old name -1. **`CLAUDE.md`** — Key Conventions, Async Architecture, CLI, Database Support sections -2. **`docs/`** — concept docs (`models.md`, `queries.md`, `formulas.md`, `ingestion.md`), getting-started guides, reference docs -3. **`.claude/skills/`** — `slayer-query.md`, `slayer-models.md`, `slayer-overview.md` -4. **`docs/configuration/`** — datasources, storage backends +Every page under `docs/` must be linked from the `nav` block in `zensical.toml` (repo +root) — add or update the entry in the same commit as the page. Otherwise the page is +still published, but as an orphan users cannot reach through site navigation. +Intentional exceptions: `docs/CLAUDE.md` and `docs/api_gaps.md`. -When renaming a field, adding a parameter, or changing response structure, **grep all docs and skills** for the old name and update every occurrence. +## Design Decisions -**Every page under `docs/` must be linked from `mkdocs.yml`.** When adding a new page, immediately add a corresponding entry to the `nav:` block in `mkdocs.yml` — otherwise it ships as an orphan file the published site can't reach. When renaming, moving, or deleting a page, update `mkdocs.yml` in the same commit. `docs/` and `mkdocs.yml` must stay in sync. The only intentional exceptions are `docs/CLAUDE.md` (doc-authoring style rules for Claude) and `docs/api_gaps.md` (internal punch list) — these are deliberately unlinked. +`DECISIONS.md` (repo root) is the append-only dated log of design decisions and their +rationale, with issue refs. Consult it before changing established behavior. When you +make an important design decision in a session, append one entry at the bottom in the +existing format. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e5115518..d1ba76f5 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -138,16 +138,16 @@ docs/ Preview locally: ```bash -pip install mkdocs-material mkdocs-jupyter mkdocs-section-index -python3 -m mkdocs serve -a localhost:8000 +pip install zensical +zensical serve # http://localhost:8000, live reload ``` **Style rule:** Use JSON/dict syntax in all docs and examples — not Python class constructors. Write `{"name": "status"}` not `ColumnRef(name="status")`. This keeps examples portable across Python, REST API, and MCP. (See `docs/CLAUDE.md` for details.) -**Plugins we use:** +**Notes:** -- `mkdocs-jupyter` — renders `.ipynb` notebooks as pages -- `mkdocs-section-index` — makes section headers clickable (links to `index.md` or first untitled entry) +- Section headers are clickable via the `navigation.indexes` theme feature (attach an `index.md` as the first entry of a section) +- Notebooks (`.ipynb`) are listed by their `docs/`-relative path; Zensical has no Jupyter plugin, so they don't render inline — the docs sync rewrites them to GitHub links, and local `zensical serve` shows them as plain links. ## Pull Requests diff --git a/DECISIONS.md b/DECISIONS.md new file mode 100644 index 00000000..23924b46 --- /dev/null +++ b/DECISIONS.md @@ -0,0 +1,87 @@ +# Design Decisions + +Append-only log of SLayer design decisions: what was decided and why, with issue refs. +Flat chronological, newest at the bottom. Never rewrite or reorder existing entries. + +Entry format: `- YYYY-MM-DD — (DEV-xxxx / #NNN)` + +For agents: when a session makes an important design decision, append ONE entry at the +bottom in this format. Compress to 1–3 lines — the decision and its why, not +implementation detail. Include issue refs when known. + +--- + +- 2026-03-31 — Time-dimension resolution for time-ordered transforms: a single query time dimension is used automatically; with 2+, `main_time_dimension` disambiguates; with none, the model's `default_time_dimension` is the fallback. +- 2026-04-09 — Filters may reference computed fields or embed inline transform expressions (`change(revenue:sum) > 0`); these are auto-extracted as hidden fields and applied as post-filters on an outer query. +- 2026-04-17 — Async-first architecture: `engine.execute()` and all `StorageBackend` methods are async; sync entry points bridge via `execute_sync`/`run_sync` (handles no-loop and Jupyter cases); native async DB drivers where available, `asyncio.to_thread` otherwise. +- 2026-04-17 — Path-based `__` table aliases for joined tables so diamond joins (same table via different paths) get distinct aliases; auto-ingestion creates only direct FK joins, multi-hop paths are resolved at query time by walking each model's joins. +- 2026-04-17 — `SlayerResponse.attributes` splits per-alias metadata into separate `dimensions` and `measures` dicts so consumers can tell the two kinds apart. +- 2026-04-29 — v2 model schema: one `columns` list replaces separate dimensions/measures (role decided per query); `measures` repurposed as a library of named formulas; aggregations moved to query time via colon syntax (`revenue:sum`, `*:count` with `*` meaning all rows); per-column `allowed_aggregations` whitelist, PK columns always restricted to count/count_distinct; auto-ingested columns named `count` renamed `count_col` to avoid clashing with `*:count`. +- 2026-05-02 — A model has exactly one source mode — `sql_table`, `sql`, or `source_queries` — enforced by validators (mutual exclusivity, non-empty stages, named non-final stages, no duplicate stage names). +- 2026-05-03 — Query-backed models cache `columns` + `backing_query_sql`, refreshed only on save paths; `execute` never writes to storage even on a stale cache (#74). Run-by-name: `execute("")` runs the stored backing query; variable precedence is runtime kwarg > stage > outer query > model defaults; `dry_run`/`explain` are engine kwargs, not query fields. +- 2026-05-04 — Optional `meta: Dict[str, Any]` on models, columns, measures, and aggregations for arbitrary user JSON; persisted, editable via all surfaces, rendered by inspect. +- 2026-05-04 — Datasource-scoped storage (DEV-1330): models keyed by `(data_source, name)` so two datasources can share a table name; bare-name lookups resolve by uniqueness, then a configurable datasource priority list, else `AmbiguousModelError`; joins never cross datasources (dotted datasource syntax tracked as DEV-1342). +- 2026-05-05 — Transform set: cumsum, time_shift, change/change_pct, rank family, first/last, lag/lead, consecutive_periods. time_shift is a calendar-based self-join CTE (gap-safe); change/change_pct desugar into hidden time_shift + arithmetic; lag/lead use LAG/LEAD directly (faster, NULL edges). Rank family (DEV-1353) defaults to no PARTITION BY, opt-in `partition_by=`; standard SQL, no UDFs. +- 2026-05-05 — SQLite quirks owned in the dialect: missing stats/math functions provided as Python UDFs registered per connection; user-written `log10`/`log2` emitted verbatim (not canonicalized to 2-arg LOG) to keep formula text round-trippable; `:memory:` databases get a per-client StaticPool engine so state survives across awaits (separate clients isolated). +- 2026-05-06 — Column types aligned byte-for-byte with sqlglot's type vocabulary (`TEXT`/`INT`/`DOUBLE`/`BOOLEAN`/`DATE`/`TIMESTAMP`) (DEV-1361); non-bare `Column.sql` expressions are CAST-wrapped from `Column.type` (TEXT is a no-op); lenient validators absorb legacy lowercase names; live-schema refinement narrows DOUBLE→INT on load and via `slayer storage migrate-types`. +- 2026-05-06 — Schema-drift family (DEV-1356): re-ingestion is idempotent and additive-only (user metadata never overwritten); read-only `validate_models` computes the minimal deletes vs the live schema using coarse type buckets (INT↔FLOAT, DATE↔TIMESTAMP collapse); DBAPI errors are auto-attributed and raise `SchemaDriftError` (REST 422); destructive cleanup (`apply_drift_deletes`) is CLI-only behind `validate-models --force-clean`. ClickHouse/BigQuery expose no FK metadata so joins there are manual; Snowflake's declarative FKs do surface. +- 2026-05-08 — Raw `OVER (...)` SQL is rejected in filters and formulas with a pointer to the rank/first/last/lag/lead transforms; non-standard window expressions belong in an earlier multi-stage stage (DEV-1369). +- 2026-05-14 — Multi-stage queries form a DAG, not a chain: runtime lists auto-topo-sort (hand-rolled Kahn), the last entry stays the root/sink, unreachable stages are dropped; stored `source_queries` keep strict order so typos fail at resolve time. Surfaced via MCP `query_nested` and a dual-shape REST `POST /query`; the single-stage `query` tool stays for its richer typed schema. +- 2026-05-14 — Bare sibling derived-column refs in `Column.sql` inline identically to qualified refs, scope-guarded (nested scopes untouched); derived-ref cycles raise `ColumnCycleError` at save time via a template-method validator on `StorageBackend.save_model` (DEV-1410). +- 2026-05-14 — Flight SQL server on port 5144, wire-compatible with the upstream Apache JDBC driver (DEV-1390); stateless — the original SQL bytes travel in the prepared-statement handle/ticket; dotted catalog form end-to-end (no `__` rewrite); wire schema is catalog-declared in Phase 1. +- 2026-05-14 — Search stack: three channels fused by Reciprocal Rank Fusion — BM25 over memory entity tags, tantivy full-text (fresh per call, deterministic single-thread writer), optional embeddings via the `advanced_search` extra (litellm; degrades to a warning, never a hard dep); `datasource` filter scopes every channel; embeddings refresh inline on ingest/edit/save with content-hash skip; YAML storage keeps embeddings in a sidecar `.db` so the store stays git-diffable. +- 2026-05-14 — Opt-in boot-time idempotent ingestion (`--ingest-on-startup`, DEV-1392): sync-before-listen, continue-on-failure per datasource, drift deletes printed but never auto-applied, all output to stderr for MCP stdio safety. +- 2026-05-18 — Memory model (DEV-1428): user-suppliable canonical ids (`memory:`, duplicate id = upsert); canonical entity form `[.[.]]` with aggregation suffixes stripped; search is lenient (unresolved refs warn, stale tags filtered); deletes cascade-strip dangling refs from all memories — only `learning` is embedded, so tag-strips don't trigger re-embedding. +- 2026-05-19 — `SlayerClient` mirrors the engine input union on every query entry point: SlayerQuery | dict | list (multi-stage) | str (run-by-name) (DEV-1437). +- 2026-05-20 — Result naming (DEV-1443/DEV-1448): colon → underscore (`orders.revenue_sum`); `*:count` → `orders._count` (leading underscore avoids colliding with a user column named `count`); an explicit measure `name` overrides the canonical alias and filters/ORDER BY accept either form; cross-model renames swap only the path leaf (`orders.customers.cust_rev`); renames shadowing source columns are rejected; same-stage filters on renamed cross-model measures still rejected (remap is DEV-1445). +- 2026-05-27 — Schema versioning: models/queries/datasource configs carry `version`; the converter chain runs in a Pydantic before-validator, so every storage backend and entry point migrates automatically and saves always emit the current version. Chain highlights: v2 columns merge, v3 dry_run/explain → engine kwargs, v4 datasource-scoped layout, v5 sqlglot type rename, v6/v7 sample-cache fields. +- 2026-05-28 — Facade layer extracted to `slayer/facade/` (Arrow-free) so Flight and the Postgres facade share the SQL→SlayerQuery translator, catalog, and info-schema; aggregate-SQL projections (SUM/COUNT/HAVING/ORDER BY forms) map to colon-form metrics validated against catalog eligibility (DEV-1486). +- 2026-05-28 — Postgres wire facade (DEV-1486): hand-rolled read-only PG protocol v3 server on port 5145 so PG-connector BI tools, psql, and asyncpg connect natively; pure stdlib, ships with the base install; the startup `database` param scopes a connection to one datasource (as schema `public`); only built-in type OIDs are emitted so clients never introspect `pg_type`; bound params are decoded and literal-substituted before translation. +- 2026-06-05 — Unified search read side: one `search()` returning a single flat RRF-ranked list of memories + entities; BM25 docs carry implicit self-references so canonical ids surface themselves (DEV-1513); optional `cypher_filter` graph pre-filter — full openCypher with the extra installed, naive label-only fallback without (DEV-1464/DEV-1532). +- 2026-06-07 — SQLite declared types are affinity hints, not constraints, so INT ingestion is probe-verified over a value sample and widened to DOUBLE/TEXT on evidence — prevents silent CAST truncation (DEV-1538); runs at ingest, idempotent re-ingest, and validate_models. Related: `json_extract` calls are preserved as function form because SQLite's `->` returns JSON-quoted strings that break equality matches. +- 2026-06-09 — Dim-only queries deduplicate: no measures + ≥1 dimension auto-emits GROUP BY over all dims, applied before LIMIT (Cube.js semantics); per-query opt-out `distinct_dimension_values=False` emits raw rows and rejects measure refs in filters/order (DEV-1543); honored per stage in DAGs. +- 2026-06-10 — Per-dialect SQL emission consolidated into `slayer/sql/dialects/` — `SqlDialect` base, one file per Tier-1 dialect, data-shaped Tier-2 table (DEV-1542); sqlglot transpilation handles everything not explicitly overridden; new dialects register + get strategy tests + join the multi-dialect matrix. +- 2026-06-10 — Snowflake supported (DEV-1551): TOML `connection_name` auth or inline creds, no Docker (tests skip without creds); runtime quirks isolated in the dialect file. +- 2026-06-10 — BigQuery rejects dots in column aliases, so the universal `model.column` alias is mangled to `model___column` at emission and reversed on result keys via symmetric dialect hooks (identity for all other dialects); `___` deliberately distinct from the `__` path-flattening separator. +- 2026-06-10 — Compact-by-default agent surfaces (DEV-1549): search/models_summary/inspect_model return descriptions and counts by default, verbose is opt-in; memories gained an optional ≤500-char `description` that feeds both compact previews and the embedding text. +- 2026-06-15 — Dedicated per-dialect CI workflows for MySQL/ClickHouse/SQL Server (DEV-1564): path-gated, testcontainers pytest job + docker-compose verify job, excluded from main CI. Live-Metabase e2e suite (DEV-1562) drives a real Metabase container against pg-serve; runs only on facade-touching PRs. +- 2026-06-18 — `TimeGranularity.WEEK` is Monday/ISO-8601; `WEEK_SUNDAY` added for Metabase compatibility (DEV-1572), implemented once as the dialect-independent shift `WEEK(col + 1 day) − 1 day` so correctness tracks WEEK; BigQuery overrides with native `WEEK(SUNDAY)`. +- 2026-06-18 — The facade translator recognizes Metabase's LEFT-JOIN-subquery MBQL shape and maps join-alias refs to cross-model dotted form (DEV-1565); an existing model join is matched by (target, join pairs), otherwise an inline ModelExtension is built with a warning; Phase 1 = one single-hop equality join. +- 2026-06-22 — Two reference modes formalized (DEV-1369): Mode A free SQL (`Column.sql`, `Column.filter`, model filters — any function, `__` paths) vs Mode B Python-AST DSL (formulas, query fields — refs, dots, colon aggs, transforms only); predicate auto-promotion removed in favor of rank-family suggestions (DEV-1336); one canonical case-insensitive `SCALAR_PASSTHROUGH` allowlist for every Mode B surface, replacing three ad hoc lists (DEV-1576/DEV-1378) — extend that set only. Canonical doc: docs/concepts/references.md. +- 2026-06-22 — `aclose()` disposal on engine and SQL client; `execute_sync` disposes in `finally` so pooled connections don't leak when `asyncio.run` closes its loop; clients survive for `:memory:` pinned connections; dispose failures log, never raise. +- 2026-06-23 — `inspect(reference, entity_type)` single-entity point lookup replaces `inspect_model` (kept as deprecated passthrough) (DEV-1588); `entity_type` is required to disambiguate canonical collisions; later extended to homogeneous-kind batches with per-id error isolation (DEV-1612) and `reference=None` collection views over models/datasources (DEV-1667). Escalation ladder: models_summary < inspect compact < inspect full. +- 2026-06-24 — `count_distinct_approx` aggregation (DEV-1595): emits each DB's native approximate-distinct, falls back to exact COUNT(DISTINCT) where none exists — exact is more accurate, so the fallback is safe. +- 2026-06-30 — psql interactive UX: `\d`/`\du`/`\l` served by stubbed `pg_am`/`pg_roles`/`pg_database` catalog relations with an `extra_relations` override hook so embedders can project real tenant rows without forking; `SELECT *` expands to all non-hidden columns in pure browse mode only (mixed with aggregates still rejects — better guidance); measures never auto-included under `*`. +- 2026-07-02 — `recommend_root_model` (DEV-1626): a pure join-graph primitive recommends the query root minimizing total hops over the mentioned models; a feasible `root_hint` overrides the auto-pick (supports bridge models owning no items); when nothing reaches everything, returns a Pareto frontier of partial roots. +- 2026-07-03 — Row-level security (DEV-1578): engine-level `SessionPolicy` (frozen Pydantic) silently scopes every query by wrapping each physical table ref in a filtered subquery — a pure sqlglot transform applied post-generation so dry_run/explain/execute see identical SQL; scope-aware (CTEs skipped); anything unconfirmable fails closed; Python-API/engine-global only in Phase 1. +- 2026-07-06 — RLS join constraints (DEV-1627): tables without the tenant column reach it via join paths stated explicitly in the policy as string hops (never auto-discovered), compiled to correlated EXISTS semi-joins (cardinality-safe); a join-ruled table is scoped only by its join rules; any policy with join rules must include a blocking column rule as backstop; ClickHouse correlated subqueries require ≥25.4 (version-probed, fail closed). +- 2026-07-06 — Opt-in per-engine in-memory query cache (DEV-1587), Python API only: key = sha256(final SQL + connection fingerprint); TTL plus Cube-style refresh keys scanned only by `refresh()` (user expression verbatim, not auto-MAX-wrapped); re-exec re-prepares from the original input with identity-guarded commits; no single-flight, unbounded, deep-copied responses. +- 2026-07-06 — OSI import (DEV-1643): direct OSI→SLayer conversion, deliberately not via the dbt intermediate (structural mismatch on measure-less metrics and join inference); live introspection provides types, OSI overlays semantics; metrics anchor via the shared min-hops-root primitive; ambiguous COUNT(*) grain clean-fails, never guessed. `ModelJoin` gained optional `description`/`meta` with no version bump (additive). +- 2026-07-11 — MCP help-seed runs only for a real storage backend and is best-effort — a seed failure warns instead of aborting server construction (DEV-1669). +- 2026-07-15 — Sample-value snapshots (DEV-1480/DEV-1516/DEV-1615): profiling is lazy (first inspect, search column hits, edit_model, explicit refresh) and never runs at ingest — per-column full scans would dominate ingest wall-clock; one scan caps at top-50-by-frequency with `distinct_count=None` on overflow (no second scan); `ensure_column_sample_fresh` is the single refresh path; structured `sampled_values` exists because comma-splitting the text form is ambiguous. +- 2026-07-16 — SLayer owns its reserved-keyword set (`slayer/sql/reserved_keywords.py`) because sqlglot's per-dialect sets are mostly empty (DEV-1686): installed into every dialect generator for AST emission, plus token-level pre-quoting before re-parsing generated SQL; common type-ish words (`date`, `name`, `count`) deliberately excluded; extend that one set only. +- 2026-07-20 — Hoisted hidden-transform names are qualified by the owning measure's field name and collision-checked against every projected name (DEV-1692) — formula-local counters alone produced duplicate CTEs and silently wrong values; any new hoisting path must derive names from `field_name`, never a formula-local counter. +- 2026-07-20 — Demo datasource ships curated semantic enrichment (labels, formats, saved measures), additive-only and idempotent so user edits survive re-runs; jafgen invoked via `sys.executable` so the demo works under pipx/uv. +- 2026-07-22 — Case-colliding ids rejected in the YAML backend only (#249): ids are filenames there and case variants alias on macOS/Windows; SQLite keeps case variants as distinct rows (so such stores can't export to YAML without renames); backends opt in via the `_ids_collide_as_filenames` flag + shared collision helpers; case-variant reads/deletes are exact-directory-entry checked (not-found / no-op, never the wrong file). +- 2026-07-31 — RLS policy restructured around one required `ruleset` (DEV-1718 / #260), superseding the `data_filters` rule list of DEV-1578/DEV-1627: a `SessionPolicy` carries exactly one `ColumnFilterRuleset` or `JoinFilterRuleset`, discriminated on an explicit `kind` (no inference — a kind-less dict fails to discriminate), and no-filtering is `policy=None` rather than an empty policy. The join ruleset hoists the tenant anchor (`table`/`column`/`value`) up from the rules, so nested rules carry only `target_table` + `join_path` and classification becomes fully structural and DB-free (`has_column` is never probed); an explicit `whitelist` replaces the mandatory blocking-column backstop, so a table that is neither the anchor, a join target, nor whitelisted fails closed. Join paths may be written from either endpoint and are normalized target-first, so the correlated EXISTS always lands its terminal predicate on the anchor. +- 2026-08-01 — Scope-closure validator (DEV-1705, DEV-1703 Stage 1): `slayer/sql/scope_check.py::assert_scope_closed` walks every sqlglot scope and flags a *provable* out-of-scope reference — a table qualifier not bound in that scope's FROM/JOINs (C1) or a cross-scope reference naming a column an inner scope does not project (C2, with a plain/`REPLACE` star exporting every name and `* EXCEPT (c)` dropping `c`). Deliberately sound-on-corpus: unqualified/ambiguous refs and physical-table column names are unverifiable and never flagged (zero false positives), so it can gate every currently-passing statement. Runs on **post-mangle, pre-RLS** generator output: dialect alias mangling (`.`→`___`) runs first, so BigQuery/T-SQL names carry no dotted output columns — pre-mangle those dotted refs parse as `table.column` (false unbound leaks) and trigger BigQuery's `TypeError`; mangling is identity for non-mangling dialects. RLS's correlated `_rls_src` EXISTS is applied downstream by the engine and whitelisted only via `allow_rls_correlation=True`. The generator terminals call `maybe_validate_scopes` env-gated by `SLAYER_VALIDATE_SCOPES`; the test harness sets it suite-wide (autouse) so a scope leak fails at generation time. A validator misfire is a validator bug to fix, never silenced. BigQuery `TypeError` on parse is a bounded, reported skip owned by Stage 9 (DEV-1713). +- 2026-08-02 — DEV-1703 Stage 2 (DEV-1706): `ScopeFrame` + single resolver + minimal `AliasAllocator` (`slayer/sql/scope.py`, `slayer/sql/naming.py`). `ScopeFrame.resolve(ref, consumer=None)` anchors a ref at the scope root or a `__`-path alias and REGISTERS every join it crosses into the scope's ordered `join_paths` in the same call (Law 1 — discovery is a side effect of rendering, never a separate step), with the Law-2 materialise branch built and unit-proven ahead of its first e2e consumer (Stage 4). The host base SELECT builds a host `ScopeFrame` and resolves column-ref aggregation kwargs (`weighted_avg(weight=)`, `corr(other=)`) through it: the resolve base-pulls the crossed LEFT JOIN and the resolved expression is embedded verbatim as a typed `ResolvedAggKwarg(kind="expr")`, deleting the `agg_kwarg_canonical_str` emission round-trip that collapsed a derived kwarg to a bare (non-existent) column name (DEV-1527 local half; cross-model remainder is Stage 4). Pulled forward because they are hard dependencies of the reserved-word fix landing here: DEV-1686 FROM/JOIN alias quoting (Identifier-node aliases + `prequote_reserved_identifiers` at every `_parse`/`_parse_predicate` re-parse) and DEV-1645 mixed-case *identifier* quoting (Flavor B — `_quote_mixed_case_identifiers` / `_to_ident` / `_to_table`); DEV-1645 ORDER-BY placement policies remain Stage 8. DEV-1539 predicate outer-parens land as a comparison-operand wrap in `_build_arithmetic_for_filter` (multi-term operands parenthesised) — derived-column filters were already precedence-safe via their type CAST. Of the four host-base join collectors, the two AGGREGATE-phase ones are folded into the resolver here — `_collect_aggregate_source_join_paths` (DEV-1502 derived sources) and `_collect_column_filter_join_paths` (DEV-1494 Column.filter) now register through the host scope in `_resolve_agg_inputs_via_scope`. The two ROW-phase ones (`_collect_filter_join_paths` WHERE-filter ValueKey trees + the join-collection half of `_expand_derived_row_dims`) are an order-coupled pair (derived-dims must register before WHERE-filters) and fold in together in Stage 4 (DEV-1708), where filter routing through the scope is reworked; the host-rooted placeholder's LIMIT-1 filter discovery (D-J) moves with them. +- 2026-08-02 — Unified cross-model reroot (DEV-1707, DEV-1703 Stage 3): one pure `slayer.core.keys.reroot_aggregate_key(key, *, target_path)` re-anchors ALL embedded references of an `AggregateKey` (source, positional args, kwarg values, `column_filter_key`) symmetrically when a cross-model aggregate renders in its target scope — replacing the three scattered per-field strip implementations (`_local_agg_formula`/`_reroot_col_kwarg` in `cross_model_planner.py`; the inline `_reroot_kwarg`/`local_args` and `_reroot_having` blocks in `generator.py`) that had diverged into two semantics. Unified on the planner's prefix-strip-with-residual (`('customers','regions')` under target `('customers',)` → `('regions',)`; exact match → local); the generator's old exact-match is subsumed. Non-matching paths pass through unchanged (function is total, never raises). `column_filter_key` is owner-anchored (its `canonical_sql`/`referenced_join_paths` are relative to the model owning the filtered column) so it is invariant under reroot and copied through unchanged — a rerooted filtered cross-model aggregate still reads local-source + non-empty filter paths, matching the DEV-1503 trigger. Closes DEV-1476 (c) (args now strip symmetrically with kwargs) and (d-cross) (path-bearing `ColumnSqlKey` time args become target-local before `_resolve_explicit_time_col`). A time arg left with a *residual* path (a hop past the target) stays a loud gap: the derived-column case raises `NotImplementedError` pointing at DEV-1526 (Stage 4 — the isolated CTE does not yet pull the deeper join), the bare-column case is caught by the `SLAYER_VALIDATE_SCOPES` scope-closure validator. +- 2026-08-02 — Cross-model / isolation CTE renderer on `ScopeFrame` (DEV-1708, DEV-1703 Stage 4). The forward `_cm_*` CTE renderer (`generator.py::_render_cross_model_cte`) and its routed-filter renderer now build a per-CTE `ScopeFrame` rooted at the target relation and route every expression through it (Law 1): the rerooted aggregate source, positional args, column-ref kwargs, `Column.filter`, shared-grain dimensions, target-model filters, and routed host WHERE/HAVING filters all `resolve()`/register their crossed joins into the CTE's single ordered `join_paths` set — the ad-hoc `_add_cte_join_paths` closure is deleted and the CTE FROM is built from that set. Closes **DEV-1526** (source `Column.sql` crossing a further join), **DEV-1527** cross-model remainder (a column-ref kwarg naming a derived target column now expands through the scope and is embedded as a typed `ResolvedAggKwarg(kind="expr")`, not a bare non-existent column), and the WHERE/HAVING routed-filter derived-ref gap. A routed-filter **pre-pass** (`_register_routed_filter_joins`) walks the full `ValueKey` tree (nested arithmetic/boolean/IN operands + aggregate leaves' source/args/kwargs/`column_filter`) before the FROM is built, so a HAVING (rendered later, after the ranked-subquery rn maps exist) still contributes its joins. **Law 2** (DEV-1702 B2, forward variant): when the CTE contains a first/last ranked subquery whose SOURCE value crosses a join, the crossing value is materialised as a `_val_` projection INSIDE the subquery (the outer `MAX(CASE WHEN _last_rn = 1 THEN … END)` would otherwise reference a table bound only inside the subquery); the routed-HAVING variant binds the SAME alias via `FirstLastRenderState.value_alias_by_sql`. A generation-wide `AliasAllocator` is installed by `generate_from_planned` (save/restore) so inline forward CTEs and the host base share `_val_` naming; recursive rerooted sub-generations get their own. **DEV-1701** host-side fix: a joined derived TIME dimension whose `Column.sql` crosses a further join expands with `is_root=False` (host-path alias `customers_v2__regions`, not bare `regions`) in the shared `_raw_time_col_expr_for_planned`, fixing host base and CTE from one helper — deliberately touching host-base rendering (nominally out of the issue's scope) because the e2e query is otherwise invalid and the suite-wide scope validator rejects it. **Null-safe grain join-back** (Codex F2): the combined-SELECT `LEFT JOIN _cm_* ON` uses `SqlDialect.build_null_safe_eq` — base `NullSafeEQ` → `IS NOT DISTINCT FROM` (Postgres/DuckDB/Snowflake/BigQuery/Trino/Presto/Databricks/Spark/ClickHouse), MySQL `<=>`, SQLite `IS` override (native form needs ≥3.39), and an expanded `a = b OR (a IS NULL AND b IS NULL)` for T-SQL/Oracle/Redshift — so NULL dimension values and nullable truncated time grains join back instead of dropping. **Decision (user-approved):** a PLAIN derived (non-time) dimension used as cross-model shared grain now raises `NotImplementedError` (planner, gated on `not hidden` so filter-only derived refs are unaffected) instead of silently CROSS-JOIN-broadcasting the global aggregate across groups — full support rides with DEV-1495-b1 (Stage 8/9). Out of scope, deferred to Stage 5 (DEV-1531/1709): the DEV-1702-B2 filtered-local (host-rooted) variant, whose value materialisation lives in `_build_first_last_base_select`. +- 2026-08-02 — first/last explicit-time completion (DEV-1710, DEV-1703 Stage 6): a first/last explicit ranking-time arg (`amount:last(customers.signup_at)`) now discovers its crossed join through the host `ScopeFrame` like every other input, not through a bespoke collector. Three sites are unified on one arg-selection contract, `SQLGenerator._explicit_time_arg_of(key)` (first positional arg iff it is a `ColumnKey`/`ColumnSqlKey`, else `None` — first/last never takes a leading non-column positional): the raise-gate in `_build_first_last_base_select` (was a divergent `any(isinstance(a,(ColumnKey,ColumnSqlKey)))` scan-all-args, so a scalar-first/col-later shape slipped the "requires a ranking time column" raise and then crashed building the `ROW_NUMBER` map — Codex F1), the new discovery sub-pass, and the render seam. `_resolve_agg_inputs_via_scope` gains sub-pass 4 (position 7): for each LOCAL first/last it `scope.resolve(arg)` register-only, so the crossed LEFT JOIN base-pulls as a Law-1 side effect (bare single-hop, bare multi-hop — every prefix registered — and local derived args whose `Column.sql` reaches a joined table); a path-bearing `ColumnSqlKey` (the DEV-1526 residual) is skipped here, not anchored. `_resolve_explicit_time_col` renders through a throwaway host-rooted `ScopeFrame` when a bundle is present (its `join_paths` discarded — discovery is owned by the base pass; same throwaway pattern as `_resolve_agg_kwargs_for_key`), which also gains DEV-1686 reserved-word qualifier quoting for free; the early returns/raises (None for no explicit arg, `NotImplementedError` DEV-1526 before any resolve, `ValueError` for a not-found derived column before any resolve) are preserved in order, and the pre-existing `bundle=None` fallback (bare-ident f-string / verbatim emit) stays verbatim because the `_build_agg_render_spec_from_planned` unit pins and the two direct-call guard tests invoke it bundle-less. Safe because `synth.time_column` is re-parsed downstream (interpolated into a `ROW_NUMBER() OVER (... ORDER BY {tc} ...)` string that `_parse` re-emits), so resolver normalization (`date()`→`DATE()`) washes out. The `Phase.AGGREGATE` arm of `_collect_joined_paths_for_base` is deleted and its signature narrowed to `(base_render_order, slots_by_id)` — it now collects only ROW dimension paths. Closes DEV-1476 fully (all four acceptance reprose green: local no-TD, cross-model bare, cross-model derived, plus the Stage-A local cases). Deliberately NOT done under Option A: the residual-hop `NotImplementedError` is kept (narrowed, owned by DEV-1526/Stage 4) rather than removed, and Stage 5 arg-isolation is not pulled forward. Out of scope and flagged separately: a derived time column whose `Column.sql` references ANOTHER derived column (nested inlining) — `expand_derived_refs_sync` does not recursively inline bare sibling-derived refs, and this fails identically for a plain dimension, so it is a general column-expansion limitation, not a time-arg concern. +- 2026-08-03 — Widened Law-3 isolation trigger (DEV-1709, DEV-1703 Stage 5): a LOCAL aggregate isolates into a host-rooted `_cm_*` CTE when ANY explicit input crosses a join — source `Column.sql` (dotted / `__` / sibling derived chains), `Column.filter` (the pre-existing DEV-1503 half), positional args incl. the explicit first/last time arg (D2), and kwargs (column refs, user template-fragment strings, and non-overridden model-default `AggregationParam.sql` fragments — an unparseable fragment contributes nothing, parity with the filter scan's fallback, a documented D1 carve-out). Non-filter kinds are computed plan-time by `slayer/engine/aggregate_input_paths.py::compute_aggregate_input_join_paths` (crossing info recomputed from the bundle, never cached on keys — DEV-1703 Q3); the recursion flag is renamed `disable_dev1503_isolation` → `disable_host_rooted_isolation` and gates ONLY the host-rooted half. Headline consequence: the top-level host base only ever contains purely-local aggregates, so a measure-pulled 1:N join can no longer multiply the rows sibling measures see (sibling protection — pinned by executed DuckDB values); the crossing measure itself keeps multiply-per-match semantics inside its CTE (F1). Aggregate-phase filters referencing a newly-isolated aggregate route to the combined-SELECT outer WHERE (never HAVING-into-the-CTE); host ROW filters (local and pathed) propagate into the host-rooted sub-plan (F4); composite crossing leaves isolate individually per interned `AggregateKey` slot (identical keys share one CTE, distinct keys get distinct CTEs — merging is DEV-1688/`may_inline` territory). Inside the CTE's sub-render, `_build_first_last_base_select` gains the Law-2 materialisation (closing DEV-1531 and DEV-1702-B1): every crossing input expression the ranked outer scope consumes — aggregate SOURCE and column-ref KWARG values (also closing the DEV-1527/DEV-1476 first/last kwarg deferral) — is projected inside the ranked subquery as a `_val_` whose body is the RESOLVED value (qualified + `Column.type` inner CAST for non-bare expressions, so `SUM(CAST(x AS t))` semantics survive materialisation and same-sql-different-type aggregates keep distinct `_val`s); alias maps are keyed by that resolved text end-to-end (host path, Stage-4 CTE path, HAVING/composite consumers). `_validate_aggregate_kwarg_paths` is relaxed for LOCAL sources (structurally-crossing kwargs are now supported inputs; the cross-model path-mismatch rejection survives). Deferred with a strict-xfail + follow-up ticket: an IMPLICITLY-resolved crossing time column (model `default_time_dimension` pointing at a crossing derived column) does not trigger — D2 covers explicit args only, and plan-time duplication of the render-time time-fallback resolution was judged not worth the drift risk. +- 2026-08-03 — time_shift CTEs on `ScopeFrame` (DEV-1711, DEV-1703 Stage 7): the shifted CTE resolves every partition key and the shift-axis time expression through a per-slot `ScopeFrame`, so its FROM pulls exactly the joins they cross. This closes **DEV-1474** (cross-model partitions like `change(order_total:sum)` by `stores.name`) and makes the sjoin grain uniformly *every projected dimension* — joined, derived, and secondary-time alike — joined back **null-safe** so NULL dimension / NULL time-bucket groups keep their prior-period value instead of dropping. A joined-column ROW filter now pulls its join into the shifted CTE instead of raising. +- 2026-08-03 — Full naming module (DEV-1713, DEV-1703 Stage 9): `slayer/sql/naming.py` becomes the single owner of every alias / result-key decision — `result_key()` (dotted FINAL-stage keys, hops via `path`, dot-free `leaf`), `result_key_from_alias()` (an already-canonical relative alias that may embed hop dots, e.g. a cross-model measure `customers.revenue_sum`), `flat_name()` (the `__`-joined INNER-stage downstream bind names), the relocated BigQuery/T-SQL `encode_alias`/`decode_alias` mangling bijection (was `slayer/sql/dialects/_alias_mangle.py`, now deleted), the relocated DEV-1645 mixed-case identifier-quoting policy (`quote_mixed_case_identifiers`/`maybe_quote_ident`, generator keeps thin delegators), and the DEV-1692 `assert_unique_cte_names` per-`WITH`-scope collision belt. The legacy flatteners (`_alias_to_short`, `_alias_to_short_local`, `_flatten_dotted`, `_cte_name_from_alias`, the stage-wrapper strip) all delegate to `flat_name` (byte-identical) so the two forms can't drift while the legacy stack lives (deleted in Stage 11). +- 2026-08-03 — D3 dotted joined-dimension result keys (DEV-1495 bug 1, DEV-1713): a joined DERIVED dimension (`ColumnSqlKey` with a non-empty path) now projects and returns under the DOTTED key `orders.customers.rev_x2` — matching cross-model measures and the documented result-key contract — not the flat `orders.customers__revenue`. The generator's `_full_alias_for_slot` and `response_meta._slot_result_keys` both route the three ROW key shapes (`ColumnKey`/`ColumnSqlKey`/`TimeTruncKey`) through `result_key`, so the SQL alias and the response key cannot diverge. The planner's flat `declared_name`/`StageColumn.name` (the downstream-bind contract) is untouched; only FINAL-stage public keys change. **Deliberate breaking change** for any consumer adapted to the buggy flat form. ORDER BY on a projected joined dimension follows the same dotted alias (plain-path resolver routed through `_full_alias_for_slot`) so the sort key names a real projected column. +- 2026-08-03 — Bare named-measure aliasing (DEV-1713): a query measure that is a bare identifier resolving to a saved `ModelMeasure` surfaces under the measure NAME (`orders.rev_total`), not the formula-derived canonical (`orders.revenue_sum`) that `expand_model_measures` would otherwise leave; explicit query `name` still wins, and the canonical alias is retained for DEV-1443 colon-form filter/ORDER-BY resolution. A self-qualified reference (`orders.rev_total`) normalizes to the bare form (`strip_source_model_prefix`) and behaves identically. +- 2026-08-03 — DEV-1692 time_shift de-collision (DEV-1713): the fix has two halves, both in the TYPED pipeline's `_emit_time_shift_ctes_for_planned` (the hoisted placeholder name `_time_shift_inner` repeats across arithmetic-wrapped shifts). A per-generation `AliasAllocator` (reserve every deterministic CTE name, allocate the `shifted_`/`sjoin_`/`step`/`cp_` families around them) makes CTE NAMES unique, AND the hidden slot's projected value alias is allocated uniquely (`_time_shift_inner`, `_time_shift_inner_2`, …) so downstream arithmetic (resolved by slot id) reads each shift's own value instead of collapsing both onto the first — the value corruption the duplicate CTE name had masked. User-facing (`public_aliases`) shift aliases are already unique and left untouched. +- 2026-08-03 — BigQuery scope-validator carve-out removed (DEV-1713, closing the DEV-1705 inherited item): with BigQuery naming/mangling finalized, no real BigQuery generator output makes sqlglot raise `TypeError` on parse (the dotted-alias shapes are collapsed to `___` before validation, and the stale calendar-time_shift INTERVAL round-trip bug no longer reproduces). The `_SQLGLOT_TYPEERROR_DIALECTS` skip-set is deleted from `scope_check.py` and all three test harnesses; BigQuery output is now scope-validated and CTE-collision-checked like every other dialect, and a parse `TypeError` propagates for every dialect with no exception. Verified empirically by running the full BigQuery test surface with the skip-set emptied — only the self-referential monkeypatch test (which forced a TypeError) reacted; zero real residual, so no follow-up ticket. +- 2026-08-03 — CTE-name collision detection stays case-sensitive for now (DEV-1713 Codex review, deferred to DEV-1726): `AliasAllocator` / `assert_unique_cte_names` compare CTE names by exact string, but generated CTE names are emitted unquoted and case-fold on Postgres/Snowflake/Redshift — so two user measure aliases differing only in case, both generating CTEs, can still collide there. Deferred rather than fixed in Stage 9 because it is pre-existing (pre-DEV-1713 the names weren't deduped at all), an edge case, and the correct fix is dialect-aware (fold only for case-folding dialects; a blanket case-insensitive dedup would wrongly merge genuinely-distinct names on case-sensitive BigQuery / quoted SQLite). Tracked in DEV-1726. +- 2026-08-03 — Merge resolution, Stage 4 × Stage 9 (DEV-1708 × DEV-1713): the user-approved derived-shared-grain raise (DEV-1708) wins over Stage 9's combined test vehicle — D3's dotted final-stage keys do NOT by themselves make a plain derived joined dim legal as cross-model shared grain (the CTE join-back rendering is still unbuilt; full support remains DEV-1495). The two Stage-9 Codex-F6 alias/key-agreement tests split their derived-dim and cross-model-aggregate coverage into separate queries. +- 2026-08-03 — Dialect-aware CTE-name case-folding (DEV-1726): `AliasAllocator` gains `folds_case` (resolved via `naming.dialect_folds_case`, threaded by the single `SQLGenerator._new_allocator` factory — the only construction site, test-pinned) and `assert_unique_cte_names` folds per dialect. Comparison-only folding with `str.lower()` (not `casefold`; sqlglot `normalize_identifier` parity) — allocated names keep original case, so output changes only when a genuine fold-collision forces a `_2` walk. Fold set = every registry dialect EXCEPT ClickHouse; unknown strings stay exact. Issue-text corrections (GoogleSQL docs + sqlglot + empirics): BigQuery FOLDS (CTE names are query aliases, case-insensitive — only real table names are CS) and SQLite/DuckDB fold even quoted names; MySQL/T-SQL fold deliberately despite config-dependence (folding is rename-only-safe, not folding leaves the bug live on majority configs). The belt folds regardless of quoting — over-strict by design on allocator-sanitized output, never a general SQL validator. Public result keys are reserved, never allocator-minted, so folding can never rename them. +- 2026-08-03 — Order-only hidden slots + plan-time order/partition validations (DEV-1712, DEV-1703 Stage 8). An ORDER BY ref that is not a declared dimension/measure is classified at **plan time** in `stage_planner.plan_query` (right after `_bucket_slots`): an **aggregate** (local or cross-model) always materialises hidden and orders — never rejected; a **local row column** is allowed only in a raw-rows query (`distinct_dimension_values=False`, no grouping) and emits a SPLIT `orders.` reference in the generator's `_apply_order_limit_from_planned` (the old `NotImplementedError` becomes a defensive internal assertion — the plan-time pass guarantees only that shape reaches it); a **grouped** local row column raises `ValueError` (not in GROUP BY — add it to dims or order by an aggregate of it); a **joined** row column raises `UnresolvableOrderColumnError`; an inline **transform/composite** (`change(amount:sum)`, only reachable as `raw_formula` — composite arithmetic is unexpressible via `OrderItem.column`, Pydantic rejects it) raises `ValueError` pointing at "declare it as a measure" — full support deferred to **DEV-1733** with strict-xfail future tests + a matching worktree. The grouping predicate is planner-semantic: `bool(agg_slots) or (dims/tds present and distinct_dimension_values)` — a hidden order aggregate that induces grouping counts, so a row column in that same query is correctly rejected. **Hidden cross-model aggregate trim** (DEV-1495 bug 2): an order-only CMA gets `hidden=True`/`public_alias=None` from the planner; the generator's combined-projection loop skips it (`public_aliases=[]`) **only when there is no transform chain** — a hidden CMA feeding a `cumsum(...)` step must stay projected for the step CTE to consume (the transform outer-wrap does the public-vs-hidden trim there). Its ORDER BY term is CTE-qualified (`_cm_*.""`) since the bare alias is no longer projected. The DEV-1495-bug-1 malformed-alias half (`orders.customers._sum`) was already fixed by Stage 9's naming module — only the projection leak remained. **partition_by grain guard** (DEV-1497): a pre-intern pass (`planning.rewrite_rank_partition_keys`, mirroring `lower_sugar_transforms`' identity-preserving rebuild) validates every rank-family (`rank`/`dense_rank`/`percent_rank`/`ntile`) `partition_by` key resolves to a query dimension/time-dimension by **exact ValueKey membership** — the typed binder resolves `partition_by` to a ValueKey before validation, so the legacy string-matching ambiguity can't arise. A time-dimension **source column** is rewritten to its `TimeTruncKey` so `PARTITION BY` uses the truncated bucket (not the raw timestamp, which had silently widened the GROUP BY grain and emitted a duplicate alias); a non-dimension raises `ValueError` naming the transform + column + available dims (the legacy `enrichment._resolve_rank_partition` message, restored). The 7 DEV-1645 Flavor-A ORDER-BY unit pins (split-not-composite for unprojected sort keys; `UnresolvableOrderColumnError` for joined sort keys) were made green by porting main's legacy `_OrderColRef`/`_order_split_sql`/`_resolve_order_column` fix (lost in the Stage-0 merge) into the legacy generator — throwaway parity, deleted with the legacy stack in Stage 11 — so `tests/parity_xfails.py` is now empty (the DEV-1485 end-state). Deliberate divergence from main: the typed pipeline rejects a grouped raw-row order at plan time (HTTP 400) instead of emitting SQL the database rejects at execution. A follow-up review sweep (PR #274) extended the same joined-ORDER-BY host-local guard into the legacy `_resolve_order_column` and rejected an unprojected sort key in the CTE-wrapped `_apply_pagination_to_sql`; added a partition ambiguity guard for a time column carried at two granularities. +- 2026-08-04 — Duration-windowed measures on the typed pipeline (DEV-1714, DEV-1703 Stage 10): `revenue:sum(window='90d')` is reimplemented as a plan-time `WindowedAggregatePlan` (symmetric with `CrossModelAggregatePlan`) plus a host-rooted `_wm___` range-join CTE rendered as a `ScopeFrame` client, closing DEV-1496 (all pinned strict-xfails promote). The `window` kwarg is a globally reserved aggregation-kwarg name (legacy parity — enrichment pops it unconditionally) and triggers the plan. The CTE's `_src` subquery self-selects host rows — dims → `_w_dim_`, other time dims date-trunc'd → `_w_td_` (grain-preserving), the raw window time column → `_w_time`, the value → `_w_value` (CASE-wrapped by `Column.filter`) — discovers its joins through a host `ScopeFrame` (Law 1, replacing the legacy regex `_window_referenced_aliases` scanner: isolates from unrelated query joins, keeps filter-referenced joins incl. multi-hop), and range-joins to `_base` on the grain (`_src._w_time >= bucket_end − window` / `< bucket_end`; per-unit `INTERVAL` via the DEV-1716 dialect strategy, SQLite `DATETIME` modifiers). It LEFT-JOINs back to `_base` null-safe on the grain, reusing the `_cm_*` orchestration in `_render_with_cross_model_plans` so windowed and cross-model measures coexist in one query. The compact-duration parser moved to `slayer/core/window_duration.py` (dependency-free) so the ENGINE planner validates durations at plan time without importing the SQL layer; the plan carries the parsed `(amount, unit)` parts so the renderer never re-parses. A filter referencing a windowed measure reclassifies to `Phase.POST` (outer WHERE on the joined-back column, never HAVING on the plain base aggregate). Scope is **sum/avg local measures only**: eight plan-time guards (precedence G1→G8→G3→G4→G5→G7→G6→G2, running on the ORIGINAL value-key trees before hidden-slot interning so transform/composite win over hidden) raise loudly on non-sum/avg, no-time-dim, cross-model, transform (input or sibling), arithmetic/composite, hidden filter-only, mixed windowed+plain filter, and malformed/empty/non-string duration — the DEV-1504 shapes stay guarded, never silently degraded. `_src` row-filter semantics are exact legacy parity: model + WHERE-phase filters apply inside `_src`, only the typed `date_range` is stripped (the trailing window must reach rows before the range start); an explicit raw-time-column filter still truncates the window near the boundary — a documented inconsistency tracked as DEV-1732. Windowed CAST follows the base path (casts the inferred slot type, matching plain aggregates — not legacy explicit-type-only, whose distinction the typed pipeline no longer has). NULL-dimension groups get a NULL windowed value (the plain `=` inside the CTE never matches NULL — the same F1-style documented cardinality decision as the cross-model CTEs). An ORDER-BY-only windowed reference is not a reachable hidden shape — `OrderItem` coercion drops the window kwarg from the column name before the planner sees it. Post-merge with DEV-1712 Stage 8: the plan-time order-by validation and the windowed-plan build sit side by side after `_bucket_slots`; a selected windowed measure ordered-by is an `AggregateKey` so the Stage-8 pass lets it through, and the windowed slot resolves in the combined ORDER BY via its bare projected `_wm_` alias (never a dangling `_base.` ref). +- 2026-08-04 — Render a derived cross-model shared grain (DEV-1728, closes DEV-1495-b1): the DEV-1708 `derived_shared_grain_not_implemented` gate is DELETED (both the planner raise in `_compute_shared_grain_slots` and the generator backstop). A plain derived (non-time) joined dimension used as a cross-model shared grain now renders through the same forward `_cm_*` CTE path a base column and a TimeTrunc-derived grain already used: `_compute_shared_grain_slots`' `ColumnSqlKey` branch mirrors the `ColumnKey` branch exactly (append the slot; the `not s.hidden` guard is dropped because the generator's `base_projection_ids` intersection already excludes hidden filter-only refs), and `_render_cross_model_cte` expands the derived `Column.sql` rooted at the target, groups by it under the DOTTED host alias (`orders.customers.rev_x2` — the naming half DEV-1713 fixed, which is what unblocked this), and joins back null-safe. A plain derived grain is CAST to its declared type to match the host base's `_wrap_cast_for_type` (bare-column / TEXT grains skip the cast identically on both sides), so the join-back compares identically-typed values instead of silently dropping groups on an INT-vs-float mismatch. **Law 2 for grains:** a first/last aggregate grouped by a CROSSING derived grain materialises that grain as a `_val_` projection inside the ranked subquery (outer `SELECT`/`GROUP BY` reference the alias; `PARTITION BY` keeps the raw expression where the join is bound) — the same treatment the crossing first/last SOURCE value already got, and this also fixes a confirmed live bug where a crossing derived TIME grain + first/last emitted invalid SQL (the ranked subquery re-exported only `target.*`, leaking an unbound `regions.opened_at` into the outer `DATE_TRUNC`). The CTE now reserves the target model's physical column names on the shared allocator so a minted `_val_` can never shadow a `target.*` column (Codex F6; previously a latent gap on the DEV-1709 source materialisation too). A target-LOCAL derived grain needs no materialisation (its refs are re-exported by `target.*`). **Out of scope (unchanged):** an intermediate-hop shared grain (a grain on a middle hop of a multi-hop aggregate target path) still raises the pre-existing 7b.12 `NotImplementedError` — it hits base columns identically, so lifting it is a separate feature, not part of removing the derived-grain gate. This supersedes the 2026-08-02 DEV-1708 user-approved raise and the 2026-08-03 Stage-4×Stage-9 merge-resolution note (which deferred full support to DEV-1495): the CTE join-back rendering that was "still unbuilt" there is now built. The re-rooted and filtered-local sibling paths were probed and already handle a derived grain correctly (regression-guarded). +- 2026-08-04 — Frame bounds vs population filters (DEV-1732). A ROW-phase filter conjunct that compares a **non-hidden query time dimension's raw column** against a **temporal literal** using `<`/`<=`/`>`/`>=` (either operand order), or a `BetweenKey` over such a column, is a **FRAME bound**: it narrows which buckets are returned, not which rows a CTE may read. Frame bounds are therefore stripped from every CTE that must reach outside the visible frame, so the two spellings of one intent agree — `date_range=[A,B]` and `filters=["created_at >= A and created_at <= B"]` now produce identical windowed numbers, where before the explicit spelling silently truncated the trailing window at the boundary (Stage 10 pinned that truncation as documented legacy parity; this ticket inverts the pin). **Both** bounds are stripped, not just the lower one: `date_range` strips a single `BetweenKey` node — i.e. both — so equivalence demands the same, accepting that a mid-bucket frame end lets the last bucket read rows past the stated end exactly as `date_range` already did (pinned by VALUE on SQLite + DuckDB, not just SQL shape). The strippable set is **every** non-hidden time dimension's raw column, not just the window axis — precisely the set for which a `date_range` spelling exists; matching is by ValueKey identity, so derived (`ColumnSqlKey`) temporal columns are covered for free. Hidden `TimeTruncKey` slots are excluded and the exclusion is **load-bearing**: `_build_windowed_plans` skips hidden row slots, so a hidden time axis is never equality-joined into `_src` and stripping its bound would leave it wholly unconstrained (over-count) where keeping it merely preserves prior behaviour. A **temporal literal** is a bare `LiteralKey` holding a non-`None` `str` — a deliberate whitelist of one shape, mirroring `BetweenKey`'s endpoints, rather than "contains no column ref" (which would admit dynamic expressions, and would strip `created_at < None`, turning an empty result into the full population, and `created_at >= 5`). A top-level `and` is **split** (n-ary flat, nested `and` recursed; survivors rebuilt in order) so `"created_at >= X and status = 'paid'"` keeps constraining `_src` to paid rows while reaching back before X; `or`/`not` are never descended into — no sound split exists, and keeping the predicate whole preserves prior numbers. NOT frame bounds: `==`/`!=`/`in`/`is` (equality on a raw timestamp means "this instant", never a range), a non-literal RHS, a `ScalarCallKey` LHS, and a time column that is not a query time dimension (no `date_range` spelling exists for it, so dropping it would over-count against every other measure — the core cardinality principle). **Mode-A `SlayerModel.filters` are exempt entirely** (deliberate, not an oversight): a model filter defines which rows EXIST rather than which frame the query looks at, there is no model-level `date_range` to be inconsistent with, and analysing arbitrary dialect SQL with `__` join-path aliases would make a silent mis-strip possible — so a time-scoped model still clips the window. **No escape hatch** (no opt-out kwarg, no legacy flag): one intent, one meaning; genuine population clipping goes in an inner stage of a multi-stage query. Accepted risk: a caller who depended on the old truncation sees numbers move with no error. Implementation: `slayer/core/time_bounds.py` (dependency-free so planner and generator share it) holds the analysis; `plan_query` computes `PlannedQuery.frame_bound_columns` **once** and partitions filters into `WindowedAggregatePlan.where_filter_ids` + `src_filter_rewrites` (residuals); the generator's `_effective_src_filters` materialises that view once and feeds the SAME list to both join discovery and rendering, so the two cannot disagree about the CTE's contents. Note that stripping can never orphan a join — the strippable columns are exactly the time dimensions `_src` always projects (`_w_time` / `_w_td_`), so their joins are independently required. The `date_range` filter-id skip is kept alongside the helper as a redundant floor, making a Stage-10 regression structurally impossible. +- 2026-08-04 — Scope amendment: DEV-1732's frame-bound rule also fixes the **`time_shift` shifted CTE** (`_shifted_where_part`), which carried the identical asymmetry — `BetweenKey` omitted, every other ROW filter propagated — so the earliest visible bucket's shifted value was correct under `date_range` and NULL under the explicit spelling. The `isinstance(..., BetweenKey)` special case is subsumed by `strip_frame_bounds` (a `date_range`'s column is always a query time dimension's raw column, so the helper returns `None` for it — same behaviour, one rule), and the join-path scan now runs on the residual. Deliberately widened past the issue's title after weighing it: the rule is a semantic decision, not a `_wm_` detail, and shipping it honoured in only one of the two places it applies would leave the next reader to re-derive the analysis. Codex flagged the widening as unrequested scope during plan review; the scope was amended explicitly rather than implicitly, which was Codex's own prescribed remedy. This is a user-visible behaviour change for anyone who wrote an explicit bound and relied on the truncated shifted CTE. +- 2026-08-04 — Order-only transform / composite / windowed ORDER BY targets (DEV-1733, DEV-1703 Stage 8 follow-up). Stage 8 rejected an ORDER BY ref bound to a **transform** or **composite** slot with an actionable `ValueError`; that raise is deleted. The full contract for an undeclared order target is now: local aggregate, cross-model aggregate, **transform** (`rank`/`cumsum`/`lag`/`lead`/`ntile`/`change`/`change_pct`), **composite** (`revenue:sum / cnt:sum`, `abs(a:sum)`, `change(a:sum) / 2`) and **windowed** (`a:sum(window='90d')`, bare or inside a composite) are all supported as HIDDEN order targets, stripped from response columns + `StageSchema`. They reach that via TWO materialisation paths, not one (detail under **Materialisation** below): a standalone target materialises a hidden slot that an outer wrap then trims, whereas a composite whose operands include a `_cm_` / `_wm_` value stays **inline** in the combined ORDER BY — the combined path has no outer trim wrap, so a materialised hidden column there would leak as a public result column. Row-column and joined-column shapes keep their Stage-8 behaviour verbatim. **Entry point** (D1): `_coerce_order_column` emits an `_expr_pending` placeholder `ColumnRef` when the canonical name fails `ColumnRef` validation AND the string is a *formula candidate* (contains `:` or matches the func-style call pattern), with `raw_formula` carrying the original; `_order_formula_candidate` is the single predicate shared with `_capture_raw_formula` so the two validators cannot drift. The boundary is deliberate: `order=[{"column": "rev / cnt"}]` — a composite over declared measure **aliases** — is not a candidate and keeps its pre-existing Pydantic error, because alias references inside expressions are unsupported everywhere in SLayer (a measure `{"formula": "rev / id:count"}` fails the same way). The planner routes on the sentinel **AND** a non-empty `raw_formula`, so a model with a genuine `_expr_pending` column, or a hand-built/deserialized `OrderItem`, still resolves normally. **Silent-drop fix**: `_iter_slot_deps` yields a composite's operands but never the composite itself, so `find_by_key` returned `None` in `plan_query` and the ORDER BY entry was *silently discarded* — `order=[{"column": "change(a:sum)"}]` ran unsorted with no error. `ProjectionPlanner.plan` now interns the top-level `ArithmeticKey`/`ScalarCallKey` of an **order** spec as a hidden slot; filters keep the operands-only walk (their top-level composite is rendered inline into WHERE/HAVING). **Materialisation** (D4) is per-path and not uniform: the transform path already emits a step CTE for unmaterialised composites; the no-transform path adds the composite's own slot id to `base_render_order` so it renders in the base SELECT and `_build_outer_trim_wrap_sql` trims it; the cross-model/windowed combined path keeps DEV-1503's **inline** `outer_composite_order_expressions` term, because that path has no outer trim wrap and a materialised hidden column would leak as a public result column. A composite with a `_cm_`/`_wm_` operand is therefore excluded from base materialisation (`_composite_has_remote_operand`) and from the `_add_local_aux_slots` promotion — otherwise it renders in `_base` from a **plain** aggregate while the CTE sits joined but unused, silently substituting a non-rolling value. `_apply_order_limit_from_planned`'s hidden branch dispatches on an explicit `(AggregateKey, ArithmeticKey, ScalarCallKey, TransformKey)` tuple rather than "any hidden slot with an alias", so a hidden ROW slot still hits the split-emission / invariant branches. **Windowed** (DEV-1714 Stage 10 x DEV-1733): `_guard_windowed_measures` returns `dict[key, hidden]` and gains an order-vk pass registering order-only windowed keys as hidden plans; `WindowedAggregatePlan.hidden`/`public_alias` (present but unused since Stage 10) are now populated, the combined SELECT trims a hidden windowed column under the same `plan.hidden and not transform_layers` predicate the hidden-CMA branch uses, and `hidden_cma_order_ref` was generalised to `hidden_cte_order_refs` — checked **before** the `cma_slot_ids` gate, since a windowed slot is not a cross-model slot and would otherwise fall through to a bare alias the SELECT no longer emits. G5 is relaxed for **order** vks only (deliberate asymmetry: a windowed composite is legal in `order`, still 400 in `measures`, since projecting it surfaces the rolling value's NULLs as user-visible results — DEV-1504); G4 (windowed + any transform) and cross-model/non-sum-avg windowed guards are untouched. This fixed a live silent-wrong-answer: an order-only windowed ref rendered a **plain** `SUM` and ordered by it (the pre-existing planner comment asserting the shape was unreachable was wrong — `OrderItem.raw_formula` preserves the `window=` kwarg). **Hidden-alias uniqueness** (D5) moved from the renderer to the PLAN: `ValueRegistry` uniquifies a hidden slot's `declared_name` (`_2`/`_3`/…) against every taken name, and `ProjectionPlanner.plan` pre-reserves all declared public/canonical names before the first intern so the outcome is intern-order independent (a hidden dep of measure *i* must not claim a name measure *j>i* declares publicly — public names are never renamed). This fixed a second live silent-wrong-answer: two hidden transform slots of the same op both took `_cumsum_inner`, so `cumsum(a:sum) + cumsum(b:sum)` projected two step-CTE columns under one alias and computed `cumsum(a) + cumsum(a)` (52/79 came back as 100/150). DEV-1692 had fixed this class inside the `time_shift`/`consecutive_periods` emitters only; owning it at intern time covers every renderer at once. Two DEV-1501-era tests pinning the old contract were inverted, not deleted: `test_order_arithmetic_walks_to_aggregate` (the arithmetic root is now slotted) and `test_hidden_composite_order_rejected_at_input_validation` (the composite string is now accepted). Out of scope, still guarded: windowed composites in `measures` and windowed + transform (DEV-1504); `time_shift` combined with a cross-model aggregate (DEV-1450 stage 7b.15e), which rejects the declared-measure form too. diff --git a/README.md b/README.md index 09d823a8..303dd60a 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,20 @@ [![PyPI](https://img.shields.io/pypi/v/motley-slayer?label=PyPI)](https://pypi.org/project/motley-slayer/) [![Python](https://img.shields.io/pypi/pyversions/motley-slayer)](https://pypi.org/project/motley-slayer/) -[![Docs](https://img.shields.io/badge/docs-readthedocs-blue)](https://motley-slayer.readthedocs.io/) +[![Docs](https://img.shields.io/badge/docs-docs.motley.ai-blue)](https://docs.motley.ai/slayer/) [![License](https://img.shields.io/github/license/MotleyAI/slayer)](LICENSE) [![GitHub stars](https://img.shields.io/github/stars/MotleyAI/slayer?style=social)](https://github.com/MotleyAI/slayer/stargazers) [![Discord](https://img.shields.io/badge/Discord-join-5865F2?logo=discord&logoColor=white)](https://discord.gg/egWxMctHCA) -**SLayer** is a semantic layer that lets AI agents query your database, manage data models, and learn from the data. +**SLayer** is a lightweight semantic layer and query engine. + +Define fields and metrics you need in data models, link your context, and query semantically; SLayer generates and runs the SQL across any database, for any surface: AI agents, dashboards, notebooks. Python-embeddable or standalone (CLI, MCP, API server). + +### What you can do with SLayer + +- **Allow your team to self-serve analytics** — model your metrics once and let anyone (or their AI agents, over MCP) ask questions, with answers grounded in your definitions and business context instead of the LLM's guesses. +- **Embed on-demand analytics into your app** — turn agent-generated query specs into safe, executed SQL, with row-level security so each user sees only what they're allowed to. +- **Load data from SQL databases to Python semantically** — point SLayer at them, and you don't have to build any SQL-translation logic. It generates and translates SQL across Postgres, MySQL, Snowflake, BigQuery, and more; returns clean dataframes. Import the library and use it in-process. > If you find SLayer useful, a ⭐ helps others discover it! > Questions, ideas, or feedback? [Join our Discord](https://discord.gg/egWxMctHCA). @@ -18,20 +26,20 @@ ## How it works -SLayer sits between your database and AI agents (or internal tools, dashboards, scripts). It allows to: +SLayer sits between your databases and whatever consumes the data — AI agents, internal tools, dashboards, scripts. It lets you: -- Auto-create data models from the database schema (warm start) -- Query using a [structured API](https://motley-slayer.readthedocs.io/en/latest/concepts/queries/) of measures, dimensions, and filters -- Edit models at runtime or create new ones and use them immediately -- Specify the desired aggregations [at query time, not in the models](https://motley-slayer.readthedocs.io/en/latest/examples/07_aggregations/aggregations/) -- Save and retrieve natural-language memories about the data and queries -- Run itself in-process, as a Python module or serverless via CLI +- Auto-generate data models from your database schema (warm start) +- Query through a [structured API](https://docs.motley.ai/slayer/concepts/queries/) of measures, dimensions, and filters +- Choose aggregations [at query time, not in the models](https://docs.motley.ai/slayer/examples/07_aggregations/aggregations/) +- Create or edit models at runtime and use them immediately — by hand, from your app, or by an agent +- Save and retrieve natural-language memories about your data and queries +- Run in-process as a Python library, or standalone via CLI, MCP, or API server -SLayer naturally evolves when the agent uses it. For example, if a query requires a new measure, the agent will update the models and will use it in other contexts. +Because models are editable at runtime, your semantic layer can grow with use: when a query needs a new measure, you (or an agent) add it once and reuse it everywhere. -SLayer compiles queries into the correct SQL for your database, handling joins, aggregations, time-based calculations, and dialect differences. Its DSL is very expressive, [supporting](https://motley-slayer.readthedocs.io/en/latest/examples/04_time/time/) queries like _"month-on-month % increase in total revenue, compared to the previous year"_, [queries-as-models](https://motley-slayer.readthedocs.io/en/latest/examples/06_multistage_queries/multistage_queries/) and much more. +SLayer compiles queries into the correct SQL for your database, handling joins, aggregations, time-based calculations, and dialect differences. Its DSL is very expressive, [supporting](https://docs.motley.ai/slayer/examples/04_time/time/) queries like _"month-on-month % increase in total revenue, compared to the previous year"_, [queries-as-models](https://docs.motley.ai/slayer/examples/06_multistage_queries/multistage_queries/) and much more. -SLayer exposes [MCP](https://github.com/MotleyAI/slayer?tab=readme-ov-file#mcp-server), [REST API](https://github.com/MotleyAI/slayer?tab=readme-ov-file#rest-api), [CLI](https://github.com/MotleyAI/slayer?tab=readme-ov-file#cli), [Python](https://github.com/MotleyAI/slayer?tab=readme-ov-file#python-client), [Flight SQL](https://motley-slayer.readthedocs.io/en/latest/interfaces/flight-sql/) (JDBC, BI-tool compatible), and a [Postgres facade](https://motley-slayer.readthedocs.io/en/latest/interfaces/pg-facade/) (point any BI dashboard's Postgres connector at SLayer) interfaces and [supports](https://motley-slayer.readthedocs.io/en/latest/configuration/datasources/#supported-database-types) most popular databases. +SLayer exposes [MCP](https://github.com/MotleyAI/slayer?tab=readme-ov-file#mcp-server), [REST API](https://github.com/MotleyAI/slayer?tab=readme-ov-file#rest-api), [CLI](https://github.com/MotleyAI/slayer?tab=readme-ov-file#cli), [Python](https://github.com/MotleyAI/slayer?tab=readme-ov-file#python-client), [Flight SQL](https://docs.motley.ai/slayer/interfaces/flight-sql/) (JDBC, BI-tool compatible), and a [Postgres facade](https://docs.motley.ai/slayer/interfaces/pg-facade/) (point any BI dashboard's Postgres connector at SLayer) interfaces and [supports](https://docs.motley.ai/slayer/configuration/datasources/#supported-database-types) most popular databases. ### Example @@ -58,7 +66,7 @@ claude mcp add slayer_demo -- slayer mcp --demo ``` ### Using your own data -Set up your datasource, substituting the correct database, username, hostname, and db_name. +Set up your datasource, substituting the correct database, username, hostname, and db_name. ```bash slayer datasources create 'postgresql://user:${DB_PASSWORD}@hostname/db_name' @@ -74,7 +82,7 @@ claude mcp add slayer -- slayer mcp --ingest-on-startup Now SLayer MCP will be visible in Claude Code next time you start it. Make sure to launch Claude Code from a shell where `DB_PASSWORD` is exported — the MCP subprocess inherits its environment from the launching process. -Read more on how to get started with [MCP](https://motley-slayer.readthedocs.io/en/latest/getting-started/mcp/), [CLI](https://motley-slayer.readthedocs.io/en/latest/getting-started/cli/), [REST API](https://motley-slayer.readthedocs.io/en/latest/getting-started/rest-api/), [Python](https://motley-slayer.readthedocs.io/en/latest/getting-started/python/) in the docs. +Read more on how to get started with [MCP](https://docs.motley.ai/slayer/getting-started/mcp/), [CLI](https://docs.motley.ai/slayer/getting-started/cli/), [REST API](https://docs.motley.ai/slayer/getting-started/rest-api/), [Python](https://docs.motley.ai/slayer/getting-started/python/) in the docs. ### Known limitations @@ -105,7 +113,7 @@ claude mcp add slayer-remote --transport sse --url http://localhost:5143/mcp/sse SLayer **does not expose credentials** to consumers once created. -Both transports expose the same tools, allowing to inspect, create and update datasources and models and run queries. More info in the [docs](https://motley-slayer.readthedocs.io/en/latest/reference/mcp/). +Both transports expose the same tools, allowing to inspect, create and update datasources and models and run queries. More info in the [docs](https://docs.motley.ai/slayer/reference/mcp/). ### CLI @@ -123,7 +131,7 @@ slayer query '{"source_model": "orders", "measures": ["*:count"], "dimensions": slayer query @query.json --format json ``` -These commands do not depend on a running server. See more in the [docs](https://motley-slayer.readthedocs.io/en/latest/reference/cli/). +These commands do not depend on a running server. See more in the [docs](https://docs.motley.ai/slayer/reference/cli/). ### Python Client @@ -151,7 +159,7 @@ df = client.query_df(query) print(df) ``` -See more in the [docs](https://motley-slayer.readthedocs.io/en/latest/reference/python-client/). +See more in the [docs](https://docs.motley.ai/slayer/reference/python-client/). ### REST API @@ -168,21 +176,33 @@ curl http://localhost:5143/models curl http://localhost:5143/datasources/my_postgres ``` -See more in the [docs](https://motley-slayer.readthedocs.io/en/latest/reference/rest-api/). +See more in the [docs](https://docs.motley.ai/slayer/reference/rest-api/). ### BI Dashboards View your SLayer models from any BI tool — no Java or custom driver needed. Start the Postgres facade and point a dashboard's **PostgreSQL** connector at it: ```bash -# Start SLayer speaking the Postgres wire protocol (Jaffle Shop demo) -poetry run slayer pg-serve --demo # listens on 127.0.0.1:5145 - -# e.g. Metabase: Add database -> PostgreSQL -# host=host.docker.internal port=5145 database=jaffle_shop (user/password: anything) +# Start SLayer speaking the Postgres wire protocol (Jaffle Shop demo). +# Containerized BI tools connect over the network, so bind all interfaces +# (non-loopback binds require a token). +slayer pg-serve --demo --host 0.0.0.0 --token pick-a-secret + +# Run the BI tool with host.docker.internal mapped to the Docker host +# (built into Docker Desktop; the flag makes it work on Linux too). The +# volume keeps Metabase's settings/dashboards across container re-creates. +docker run -d -p 3000:3000 --name metabase \ + --add-host=host.docker.internal:host-gateway \ + -e MB_DB_FILE=/metabase.data/metabase.db \ + -v metabase-data:/metabase.data \ + metabase/metabase + +# Metabase: Add database -> PostgreSQL +# host=host.docker.internal port=5145 database=jaffle_shop +# user=anything password=pick-a-secret SSL=off ``` -The connection's `database` selects the SLayer datasource; its models appear as tables under schema `public`. There's also an [Arrow Flight SQL](https://motley-slayer.readthedocs.io/en/latest/interfaces/flight-sql/) facade for JDBC clients. See the [Postgres facade docs](https://motley-slayer.readthedocs.io/en/latest/interfaces/pg-facade/) for auth, TLS, and the supported SQL surface. +The connection's `database` selects the SLayer datasource; its models appear as tables under schema `public`. There's also an [Arrow Flight SQL](https://docs.motley.ai/slayer/interfaces/flight-sql/) facade for JDBC clients. See the [Postgres facade docs](https://docs.motley.ai/slayer/interfaces/pg-facade/) for auth, TLS, and the supported SQL surface. @@ -248,7 +268,7 @@ The `measures` parameter on a query specifies what data columns to return. Aggre } ``` -Available functions: `cumsum`, `time_shift`, `change`, `lag`, and more – see [docs](https://motley-slayer.readthedocs.io/en/latest/concepts/formulas/). Formulas support arbitrary nesting — e.g., `change(cumsum(revenue:sum))` or `cumsum(revenue:sum) / *:count`. +Available functions: `cumsum`, `time_shift`, `change`, `lag`, and more – see [docs](https://docs.motley.ai/slayer/concepts/formulas/). Formulas support arbitrary nesting — e.g., `change(cumsum(revenue:sum))` or `cumsum(revenue:sum) / *:count`. ## Filters @@ -265,7 +285,7 @@ Filters use simple formula strings — no verbose JSON objects: } ``` -Filters support a variety of operators, composition, pattern matching. Transforms & computed columns can also be used for filtering. See [docs](https://motley-slayer.readthedocs.io/en/latest/concepts/queries/#filters) for more. +Filters support a variety of operators, composition, pattern matching. Transforms & computed columns can also be used for filtering. See [docs](https://docs.motley.ai/slayer/concepts/queries/#filters) for more. ## Auto-Ingestion @@ -320,7 +340,7 @@ password: ${DB_PASSWORD} Environment variable references (`${VAR}`) are resolved at read time. -See more in the [docs](https://motley-slayer.readthedocs.io/en/latest/configuration/datasources/). +See more in the [docs](https://docs.motley.ai/slayer/configuration/datasources/). ## Storage Backends @@ -331,7 +351,7 @@ SLayer ships with two storage backends: SLayer allows easily implementing your own storage backends, which is useful for features such as tenant isolation. -See the [documentation page for storage backends](https://motley-slayer.readthedocs.io/en/latest/configuration/storage/) for more. +See the [documentation page for storage backends](https://docs.motley.ai/slayer/configuration/storage/) for more. ## Roadmap diff --git a/docs/api_gaps.md b/docs/api_gaps.md index 431d0373..585edab7 100644 --- a/docs/api_gaps.md +++ b/docs/api_gaps.md @@ -27,4 +27,4 @@ Features available in at least one API but missing in at least one other. | Health check endpoint | Y | — | — | `GET /health` | | Start REST server | — | Y | — | `slayer serve` | | Start MCP server | — | Y | — | `slayer mcp` | -| Conceptual help | — | Y | Y | `help()` MCP tool / `slayer help [TOPIC]` subcommand; identical content from `slayer/help/topics/*.md` | \ No newline at end of file +| Conceptual help | Y | Y | Y | Seeded `memory:help.*` memories (from `slayer/memories/help_content/*.md`); read via `inspect(entity_type="memory")` / `slayer inspect memory:help.intro --type memory`, or `search` | \ No newline at end of file diff --git a/docs/architecture/cross-model-aggregates.md b/docs/architecture/cross-model-aggregates.md index 151aed04..34b07932 100644 --- a/docs/architecture/cross-model-aggregates.md +++ b/docs/architecture/cross-model-aggregates.md @@ -63,6 +63,26 @@ goes through host-filter classification. `shared_grain_slots` is the set of host dimension/time-dimension slots reachable from the target, used to LEFT JOIN the CTE back without changing cardinality. +### Rerooting the aggregate's embedded references (`reroot_aggregate_key`) + +When the forward `_cm_*` CTE (`_render_cross_model_cte`), its HAVING route, and +the re-rooted-plan formula (`_local_agg_formula`) render a cross-model aggregate +in its target scope, every reference embedded in the `AggregateKey` — the +`source`, positional `args` (e.g. the `first`/`last` explicit time arg), keyword +`kwargs` values (e.g. `weighted_avg(weight=…)`), and `column_filter_key` — must +be re-anchored from the query root's coordinate system to the target's. This is +one symmetric transform, `slayer.core.keys.reroot_aggregate_key(key, *, +target_path)` (DEV-1707), which prefix-strips `target_path` off each ref's join +path and keeps the residual (`('customers','regions')` under target +`('customers',)` → `('regions',)`; an exact match → local). `column_filter_key` +is owner-anchored (stamped against the model that owns the filtered column) and +therefore invariant under reroot — it is carried through unchanged. A time arg +left with a *residual* path after reroot (a hop past the target) is a +[DEV-1526](https://linear.app/motley-ai/issue/DEV-1526) Stage-4 gap: the isolated +CTE does not yet pull that deeper join, so `_resolve_explicit_time_col` raises +for the derived-column case and the scope-closure validator catches the +bare-column case. + ## Strategy 2: re-rooting (the deviation) `IsolatedCteCrossModelPlanner` alone is insufficient. When the host query carries @@ -117,41 +137,91 @@ path for a permutation" the redesign set out to eliminate. It works and is tested, but it is the place a future reviewer should look first when reasoning about cross-model behavior. -## Strategy 3: filtered-local isolation (DEV-1503) - -A **cross-model-FILTERED local measure** is a host aggregate whose `Column.filter` -references a joined table — `loss_payment_amt:sum` where `loss_payment_amt` has -`filter="loss_payment.has_flag = 1"`. The aggregate's `source.path` is empty -(it's a local column), but its `column_filter_key.referenced_join_paths` is -non-empty, so emitting it inline in the host base SELECT would pull the -filter-target join into the host's FROM. With **two** such measures whose -filter targets are different INNER joins, the host base would intersect to -only the rows present in BOTH targets — silently corrupting both aggregates. +## Strategy 3: host-rooted isolation — any crossing input (DEV-1503, widened by DEV-1709) + +A LOCAL aggregate (empty `source.path`) isolates into a **host-rooted** CTE +when **any** of its inputs crosses a join (Law 3, DEV-1703 D1/D2): + +- its `Column.filter` references a joined table — the original DEV-1503 + case (`loss_payment_amt:sum` with `filter="loss_payment.has_flag = 1"`), + read from the bind-time `column_filter_key.referenced_join_paths`; +- its **source `Column.sql`** crosses a join (`region_pay` with + `sql="customers__regions.payment_amount"`, single-dot forms, and sibling + derived chains); +- a **positional arg** crosses — including the explicit first/last time arg + (`amount:last(customers.signup_at)` and derived variants); +- a **kwarg** crosses — a column ref (`weighted_avg(weight=customers.w)` or + a crossing derived column), a user-supplied template-fragment string, or + a non-overridden model-default `AggregationParam.sql` fragment. + +The non-filter kinds are computed plan-time by +`slayer/engine/aggregate_input_paths.py::compute_aggregate_input_join_paths` +(the same parse → derived-expansion → root-scope-walk pipeline the filter +scan uses; an unparseable template fragment contributes nothing — parity +with the filter scan's defensive fallback). Without isolation, a crossing +input emitted inline in the host base SELECT would pull its join into the +host's FROM: two measures whose filter targets are different INNER joins +would intersect the base to rows present in BOTH targets, and any 1:N +crossing join would **multiply the host rows seen by sibling measures** — +the sibling-protection guarantee is the point of Law 3. The crossing +measure itself keeps multiply-per-match semantics inside its CTE (F1 +decision — 1:N semantics unchanged, only the scope moved). The trigger predicate is structural: -`agg_path` non-empty (forward cross-model) **OR** -`column_filter_key.referenced_join_paths` non-empty (filtered-local). Both -route through `IsolatedCteCrossModelPlanner.plan`; the filtered-local branch -calls `_plan_filtered_local`, which builds a **host-rooted** nested -`PlannedQuery` (same `source_model`, same dims/TDs, only the filtered measure -as the single aggregate) and attaches it via the same -`rerooted_plan` / `rerooted_grain_pairs` / `rerooted_agg_slot_id` slots the -re-rooted path uses. The plan carries `cte_root_model = host_model.name` as -the disambiguator the renderer reads; `_render_rerooted_cross_model_cte` -short-circuits the source-model swap when `cte_root_model` is set. +`agg_path` non-empty (forward cross-model, target-rooted) **OR** any +crossing input (host-rooted). Both route through +`IsolatedCteCrossModelPlanner.plan`; the host-rooted branch +calls `_plan_filtered_local`, which rebuilds the measure's formula text +via `_local_agg_formula` (round-trip-tested for every input shape) into a +**host-rooted** nested `PlannedQuery` (same `source_model`, same dims/TDs, +only the crossing measure as the single aggregate) and attaches it via the +same `rerooted_plan` / `rerooted_grain_pairs` / `rerooted_agg_slot_id` +slots the re-rooted path uses. The plan carries +`cte_root_model = host_model.name` as the disambiguator the renderer +reads; `_render_rerooted_cross_model_cte` short-circuits the source-model +swap when `cte_root_model` is set. Isolation is strictly +per-`AggregateKey`-slot: identical keys intern to one slot and share one +CTE; distinct keys get distinct CTEs (cross-CTE merging is DEV-1688 / +`may_inline` territory). ```mermaid flowchart TB - detect["column_filter_key.referenced_join_paths non-empty?"] + detect["any crossing input?\n(filter / source sql / arg / kwarg)"] detect -->|yes| build["_plan_filtered_local builds host-rooted SlayerQuery"] build --> replan["subplan_builder(rerooted_query, bundle)"] replan --> attach["attach with cte_root_model = host.name"] attach --> gen["generator: _render_rerooted_cross_model_cte (host-rooted branch)"] ``` -`subplan_builder` always passes `disable_dev1503_isolation=True` so the -recursive `plan_query` call inside the sub-plan does NOT re-trigger isolation -on the same measure. +`subplan_builder` always passes `disable_host_rooted_isolation=True` +(DEV-1709 rename of `disable_dev1503_isolation`) so the recursive +`plan_query` call inside the sub-plan does NOT re-trigger isolation on the +same measure — inside the CTE the crossing inputs render inline +(base-pull), which is legal there because the CTE is the aggregate's own +scope. The flag never affects target-rooted isolation. + +### Composite lowering (F3) + +In an AGGREGATE-phase composite (`a:sum + b:sum`, `coalesce(a:sum, 0)`), +each **crossing leaf** isolates individually (the leaves are hidden +aggregate slots that traverse the same trigger loop); local leaves stay in +`_base`; the composite expression renders only in the combined SELECT via +the leaves' projected aliases. This holds for projected composites, +filter-only composites (routed to the outer WHERE), and order-only +aggregate refs. + +### Law 2 inside the ranked scope + +The first/last ranked subquery re-exports only `source_relation.*` plus +rank/`_td`/`_dim` columns, so any crossing expression the outer SELECT +consumes — an aggregate SOURCE or a column-ref KWARG value — is +materialised as a `_val_` projection inside the subquery +(`_build_first_last_base_select`, mirroring the Stage-4 CTE path). The +projection is the **resolved** value (qualified, with the `Column.type` +inner CAST for non-bare expressions), so `SUM(CAST(x AS t))` semantics are +preserved and same-sql-different-type aggregates keep distinct +materialisations; HAVING and composite consumers bind to the same alias +via `FirstLastRenderState.value_alias_by_sql` (keyed by resolved text). ### Filter routing for filtered-local @@ -191,6 +261,67 @@ public projection trims them out. aggregated column renders as `SUM(CASE WHEN THEN END)`. See [SQL generation](sql-generation.md). +### The forward CTE is a `ScopeFrame` (DEV-1708, DEV-1703 Stage 4) + +`_render_cross_model_cte` builds one `ScopeFrame` rooted at the target relation +and routes **every** expression it renders — the rerooted aggregate source, +positional args, column-ref kwargs, `Column.filter`, shared-grain dimensions, +target-model filters, and routed host WHERE/HAVING filters — through +`resolve()` (Law 1). Each `resolve` anchors the ref at the target and registers +the joins it crosses into the CTE's single ordered `join_paths` set, from which +the CTE `FROM` is built. Discovery can no longer be forgotten per carrier: a +cross-model aggregate whose target column's `Column.sql` crosses a *further* +join (`customers.deep_pop:sum` where `deep_pop` is `regions.population`) now +pulls that `LEFT JOIN regions` into the `_cm_*` CTE, and a parametric-agg +column-ref kwarg naming a derived target column expands through the scope +instead of emitting a bare, non-existent column. Routed WHERE/HAVING filters +register their joins in a **pre-pass** that walks the full `ValueKey` tree +(nested arithmetic/boolean/IN operands + aggregate leaves' source/args/kwargs/ +`column_filter`) before the `FROM` is built, so a HAVING — rendered later, once +the ranked-subquery rank columns exist — still contributes its joins. + +**First/last value materialization (Law 2).** When the CTE wraps its rows in a +`ROW_NUMBER`-ranked subquery and the first/last **source value** crosses a join, +the crossing value is materialized as a `_val_` projection *inside* the +subquery and the outer `MAX(CASE WHEN _last_rn = 1 THEN _val_ END)` references +the alias — a raw crossing ref there is bound only inside the subquery. A HAVING +on the same aggregate binds the same alias. The **shared grain** obeys the same +law (DEV-1728): a crossing derived grain is materialized as a `_val_` +projection inside the subquery, the outer `SELECT`/`GROUP BY` reference the +alias, and `PARTITION BY` keeps the raw expression (evaluated where the join is +bound). `generate_from_planned` installs one generation-wide `AliasAllocator` +(save/restore) so inline forward CTEs, grain projections, and the host base +never collide on `_val_`; the CTE reserves the target's physical column names +so a minted alias never shadows a `target.*` column. + +### Derived shared-grain rendering (DEV-1728) + +A cross-model aggregate can be grouped by a joined **derived** dimension +(`{"dimensions": ["customers.rev_x2"], "measures": ["customers.revenue:sum"]}`, +where `rev_x2` is a `Column.sql` on `customers`). The grain loop expands the +derived column's `Column.sql` rooted at the target relation, adds it to the CTE +`SELECT` + `GROUP BY` under the **dotted** host alias +(`orders.customers.rev_x2` — the same alias the host base projects since +DEV-1713), and joins back null-safe. A derived expression that crosses a +*further* join (`deep_pop` = `regions.population`) pulls that join into the CTE +`FROM` via the same `ScopeFrame` machinery the derived-time-dimension grain uses; +a plain derived grain is wrapped in the same `CAST(... AS )` the host base +applies so the join-back compares identically-typed values. Only an +**intermediate-hop** grain (a grain on a middle hop of a multi-hop aggregate +target path) is still unrendered — it raises the same 7b.12 `NotImplementedError` +a base column on an intermediate hop does. + +### Null-safe grain join-back + +The combined-SELECT `LEFT JOIN _cm_* ON` grain equality uses a dialect-aware +null-safe predicate (`SqlDialect.build_null_safe_eq`): `IS NOT DISTINCT FROM` +on Postgres/DuckDB/Snowflake/BigQuery/Trino/Presto/Databricks/Spark/ClickHouse, +`<=>` on MySQL, bare `IS` on SQLite (the native form needs SQLite ≥3.39), and +the expanded `a = b OR (a IS NULL AND b IS NULL)` on T-SQL/Oracle/Redshift. A +plain `=` would yield `NULL` for `NULL = NULL`, so a NULL dimension value or a +nullable truncated time grain would drop its joined-back aggregate; the +null-safe form retains it. + ## Known limitations (documented, not blocking) - A host-local filter on a **no-dimension** cross-model-agg query is applied @@ -205,8 +336,17 @@ aggregated column renders as `SUM(CASE WHEN THEN END)`. See collision bug; the new path keeps the suffix. This violates **P10** for this one combination and is tested structurally, not by parity. See [the deviations list](index.md#deviations-from-the-plan). -- `weighted_avg(weight=qty)` / `corr(other=qty)` cross-model semantics (a - host-local weight column evaluated inside the target CTE) are not supported. +- A cross-model parametric-agg kwarg naming a **target** column + (`customers.revenue:weighted_avg(weight=customers.qty)`) is supported and + expands through the CTE `ScopeFrame` (DEV-1708). The kwarg must be + **relation-qualified** — a bare `weight=qty` resolves against the host by DSL + rule and raises at bind time. A *host-local* weight column evaluated inside + the target CTE remains unsupported. +- A **shared-grain dimension on an intermediate hop** of a multi-hop aggregate + target path (base or derived) raises the 7b.12 `NotImplementedError` — use the + terminal-target path or pull the dimension to the host base. (A plain derived + grain on the *terminal* target path is fully rendered — see + [Derived shared-grain rendering](#derived-shared-grain-rendering-dev-1728).) ## Design rationale diff --git a/docs/architecture/sql-generation.md b/docs/architecture/sql-generation.md index 2de7f6ff..b94f9840 100644 --- a/docs/architecture/sql-generation.md +++ b/docs/architecture/sql-generation.md @@ -97,6 +97,51 @@ forward-path CTE renders (FROM bare target, grouped at the forward dims). `SUM(CASE WHEN THEN END)`. See [Cross-model aggregates](cross-model-aggregates.md). +The same renderer also emits one `_wm_*` CTE per `WindowedAggregatePlan` — a +duration-windowed measure (`revenue:sum(window='90d')`). Unlike `_cm_*` (rooted +at the join target), a `_wm_*` CTE is **host-rooted**: an inner `_src` subquery +self-selects the host rows (dimensions → `_w_dim_`, other time dims → +`_w_td_`, the raw window time column → `_w_time`, the value → `_w_value`) +with its joins discovered through a host `ScopeFrame`, and +`FROM _base LEFT JOIN _src` +pairs the grain equalities with a trailing `INTERVAL` range predicate +(`_src._w_time >= bucket_end − window` / `< bucket_end`). The result groups at +the query grain and joins back to `_base` null-safe, exactly like a `_cm_*` CTE, +so windowed and cross-model measures coexist in one query. Windowed-measure +filters route to the combined-SELECT outer `WHERE` (`Phase.POST`). `sum`/`avg` +local measures only; other shapes raise at plan time (`_guard_windowed_measures` +in `stage_planner.py`). + +### Frame bounds vs population filters (DEV-1732) + +`_src` inherits the host's ROW-phase filters **minus their frame bounds** — a +relational comparison (`<`, `<=`, `>`, `>=`) between a non-hidden time +dimension's raw column and a temporal literal. Without that, the trailing window +cannot reach rows before the earliest visible bucket and that bucket +under-counts; with it, `date_range` and the explicit spelling of the same intent +produce identical numbers. + +`slayer/core/time_bounds.py` owns the analysis (dependency-free, so planner and +generator share it). `stage_planner.plan_query` computes the strippable column +set once into `PlannedQuery.frame_bound_columns` and partitions the filters into +`WindowedAggregatePlan.where_filter_ids` (applied) plus `src_filter_rewrites` +(applied as a residual — a top-level `and` is split so only its frame-bound +conjuncts drop; `or`/`not` are never descended into). The generator's +`_effective_src_filters` materialises that view **once** and feeds the same list +to both join discovery and rendering, so the two cannot disagree about what the +CTE contains. + +Hidden `TimeTruncKey` slots are excluded from `frame_bound_columns` on purpose: +`_build_windowed_plans` skips hidden row slots, so a hidden time axis is never +equality-joined into `_src` and stripping its bound would leave it +unconstrained. Mode-A `SlayerModel.filters` are exempt entirely — they define +which rows exist, not which frame the query looks at. + +The `time_shift` shifted CTE (`_shifted_where_part`) applies the same rule, +reading the same `frame_bound_columns`; its former +`isinstance(..., BetweenKey)` special case is subsumed, since a `date_range`'s +`BetweenKey` column is always a query time dimension's raw column. + ## Mode-A filter inlining and join discovery (DEV-1494) A column-level `Column.filter` on an aggregated measure becomes a CASE-WHEN @@ -153,14 +198,20 @@ call chain just before `_build_from_and_joins`: All three collectors restrict to **local** aggregate sources (empty `AggregateKey.source.path`); cross-model aggregates own their own join -discovery inside the per-plan `_cm_*` CTE for the `Column.filter` side -(DEV-1494 / DEV-1503). The symmetric source-`Column.sql` discovery inside -the `_cm_*` CTE is a known gap (DEV-1526) — a cross-model aggregate whose -target column's `Column.sql` crosses a further join does not yet have that -join pulled into the CTE FROM. All three host-side collectors feed the -shared `needed_join_paths` list, so repeated paths surfaced by different -sources dedupe naturally via `_build_from_and_joins`'s `emitted_aliases` -guard. +discovery inside the per-plan `_cm_*` CTE — for the `Column.filter` side +(DEV-1494 / DEV-1503) and, since Stage 4 (DEV-1708 closed DEV-1526), for a +target column's `Column.sql` that crosses a further join. All three +host-side collectors feed the shared `needed_join_paths` list, so repeated +paths surfaced by different sources dedupe naturally via +`_build_from_and_joins`'s `emitted_aliases` guard. + +Since Stage 5 (DEV-1709, widened Law-3 trigger), a LOCAL aggregate with +any crossing input — source `Column.sql`, `Column.filter`, positional +args, kwargs — never renders in the top-level host base at all: it +isolates into a host-rooted `_cm_*` CTE, and the discovery above runs +inside that CTE's sub-render (see +[cross-model-aggregates.md](cross-model-aggregates.md#strategy-3-host-rooted-isolation--any-crossing-input-dev-1503-widened-by-dev-1709)). +The host base only ever contains purely-local aggregates. ## Result-key contract (P10) diff --git a/docs/concepts/formulas.md b/docs/concepts/formulas.md index 92be99c6..d59c3708 100644 --- a/docs/concepts/formulas.md +++ b/docs/concepts/formulas.md @@ -62,6 +62,70 @@ Units can be combined in descending or practical order, for example `'1y2m3w5d6h7min8s'`, `'90d'`, `'6h'`, or `'15min'`. Quote the duration value inside the formula. +Windowed measures need exactly one resolvable time dimension (a single +`time_dimensions` entry, or `main_time_dimension` to disambiguate). Filtering on +a windowed measure (`{"formula": "revenue:sum(window='90d') > 100"}`) applies +after aggregation, and the windowed measure must also be selected. + +A windowed measure may also be used purely as an **order** target without being +selected — `{"order": [{"column": "revenue:sum(window='90d')", "direction": "desc"}]}` +ranks by the rolling value and keeps it out of the result. That works both for a +bare windowed measure and for one inside an order-only composite +(`{"column": "revenue:sum(window='90d') / cnt:sum"}`). + +Note the deliberate asymmetry: a windowed measure inside a composite is allowed +in `order` but not yet in `measures`. Ordering needs only a single scalar +comparison, whereas projecting the composite surfaces the rolling value's NULLs +(a grain bucket with no matching source rows yields NULL) as user-visible +result values — settling those semantics is part of the follow-up below. + +#### Time bounds do not clip the window + +A trailing window has to read rows from *before* the earliest bucket you asked +for — otherwise that bucket silently under-counts. So a **time bound narrows +which buckets come back, not which rows the window may reach**. These two +queries return identical numbers: + +```json +{"time_dimensions": [{"dimension": "created_at", "granularity": "month", + "date_range": ["2025-01-01", "2025-12-31"]}]} +``` +```json +{"time_dimensions": [{"dimension": "created_at", "granularity": "month"}], + "filters": ["created_at >= '2025-01-01' and created_at <= '2025-12-31'"]} +``` + +A bound counts as a *frame* bound when it compares a **time dimension's own +column** against a **literal** using `<`, `<=`, `>`, or `>=`. Everything else is +an ordinary row filter and does restrict the window's input, including: + +- other operators on that column — `created_at == '2025-01-01'`, `IN (…)`, + `IS NOT NULL`; +- a bound on a time column that is not one of the query's time dimensions; +- a comparison against another column rather than a literal; +- a bound wrapped in `or` or `not`, which cannot be separated out safely; +- `filters` declared on the **model** — those define which rows exist at all, so + a model scoped to `created_at >= '2024-01-01'` does clip the window there. + +Mixed filters are split, so only the time part is set aside: +`"created_at >= '2025-01-01' and status = 'paid'"` restricts the window's input +to paid rows while still reaching back before January. + +The same rule applies to [`time_shift`](#transform-functions) — the earliest +visible bucket still gets its prior-period value under either spelling. + +If you genuinely want to clip the underlying rows, apply the bound in an inner +stage of a multi-stage query so the windowed stage never sees the raw column. + +The following windowed-measure shapes raise a clear error rather than returning +wrong numbers, and are planned follow-ups: a windowed aggregation other than +`sum`/`avg`; a cross-model windowed measure (`customers.revenue:sum(window=…)`); +a windowed measure combined with a transform (`cumsum`, `time_shift`, …) in any +position; a windowed measure nested in an arithmetic/composite expression in +`measures` (`{"formula": "revenue:sum(window='90d') / 2"}`); or one compared +against a plain aggregate inside one filter +(`revenue:sum(window='90d') > 100 and revenue:sum > 50`). + --- ## Field Formulas @@ -131,12 +195,12 @@ Functions apply window operations to measures: | Function | Description | SQL Generated | |----------|-------------|---------------| | `cumsum(x)` | Running total over time | `SUM(x) OVER (PARTITION BY dims ORDER BY time)` | -| `time_shift(x, n)` | Value N periods back/ahead | Self-join CTE with INTERVAL offset | +| `time_shift(x, n)` | Value N time buckets back/ahead (calendar-aware) | Self-join CTE with INTERVAL offset | | `time_shift(x, offset, gran)` | Value from a different time bucket | Self-join CTE with INTERVAL offset | | `lag(x, n)` | Value N rows back (window function) | `LAG(x, n) OVER (PARTITION BY dims ORDER BY time)` | | `lead(x, n)` | Value N rows ahead (window function) | `LEAD(x, n) OVER (PARTITION BY dims ORDER BY time)` | -| `change(x)` | Difference from previous period | Desugars to `x - time_shift(x, -1)` | -| `change_pct(x)` | Percentage change from previous | Desugars to `(x - ts) / ts` where `ts = time_shift(x, -1)` | +| `change(x)` | Period-over-period difference (partition-safe, resets per group) | Desugars to `x - time_shift(x, -1)` | +| `change_pct(x)` | Period-over-period % change, e.g. month-over-month growth (partition-safe, resets per group; NULL when the prior period's value is 0 or missing) | Desugars to `CASE WHEN ts != 0 THEN (x - ts) / ts END` where `ts = time_shift(x, -1)` | | `consecutive_periods(predicate)` | Current trailing run length where predicate is true | Staged window CTEs with reset groups | | `rank(x[, partition_by=...])` | Ranking by value (descending) | `RANK() OVER ([PARTITION BY ...] ORDER BY x DESC)` | | `percent_rank(x[, partition_by=...])` | Relative rank in [0, 1] (descending) | `PERCENT_RANK() OVER ([PARTITION BY ...] ORDER BY x DESC)` | @@ -155,6 +219,14 @@ total per status, not one running total across the whole result set. `time_shift` uses a **self-join CTE** with an INTERVAL-shifted time column. `change` and `change_pct` are desugared into a hidden `time_shift` + arithmetic expression at query enrichment time. The shifted sub-query applies the time offset everywhere (WHERE, GROUP BY, SELECT), so it can reach outside the current result set — no edge NULLs when the database has the data, and correct handling of gaps in time series. +The self-join matches on **every projected dimension as well as the shifted time column** — plain columns, joined columns (`stores.name`), derived columns, and any secondary time dimension all take part in the join grain (e.g. `ON base.month IS NOT DISTINCT FROM shifted.month AND base.store IS NOT DISTINCT FROM shifted.store`). So these transforms are partition-safe: each group's series is compared only against itself, and per-group series reset cleanly. One store's first month is never diffed against another store's last month. The grain match is **null-safe** (`IS NOT DISTINCT FROM`, or the dialect equivalent), so a group with a NULL dimension value — for example rows with no matching row across a LEFT join — still lines up against its own prior period instead of dropping to a NULL shifted value. + +**Intent recipes:** + +- Month-over-month / period-over-period growth → `change_pct(revenue:sum)` with a `time_dimensions` entry at the desired granularity. Prefer this over hand-building the ratio from `time_shift`. +- Absolute period-over-period delta → `change(revenue:sum)`. +- Comparing against a *different* grain than the query's (e.g. year-over-year on a monthly series), or using the shifted value as a term in custom arithmetic → `time_shift(revenue:sum, -1, 'year')`. + `lag(x, n)` and `lead(x, n)` use SQL `LAG`/`LEAD` window functions directly. They are more efficient but have two trade-offs: - **Edge NULLs**: the first/last N rows always return NULL since window functions can only see rows within the current result set. @@ -222,7 +294,7 @@ Combine with a filter to get "top N": **Ranking within a partition (`partition_by=`):** -To rank within groups instead of across the whole result set, pass `partition_by=` referencing one or more **query dimensions** (or time dimensions). The columns must already be grouped on — partitioning by a column that's not a dimension errors at enrichment time. +To rank within groups instead of across the whole result set, pass `partition_by=` referencing one or more **query dimensions** (or time dimensions). The columns must already be grouped on — partitioning by a column that's not a dimension errors at plan time (HTTP 400). Naming a query time-dimension partitions by its truncated bucket, not the raw timestamp. ```json { @@ -277,8 +349,20 @@ Inside `Column.sql`, `ModelMeasure.formula`, or any `Aggregation.formula`, you c | `exp(x)` | 1 | `e^x` | | `sqrt(x)` | 1 | Square root | | `pow(x, n)` / `power(x, n)` | 2 | `x^n`. Both spellings are accepted (sqlglot may emit either depending on origin dialect). | +| `round(x[, ndigits])` | 1–2 | Round to `ndigits` decimal places (default 0). `ndigits` must be an integer literal. | +| `abs(x)` | 1 | Absolute value. | + +Unlike the other scalar functions above — which pass through only when embedded in a larger `Column.sql` or arithmetic expression — `round` and `abs` are also valid as the **top-level** form of a query measure or `ModelMeasure.formula`: + +```python +{"formula": "round(revenue:sum, 2)"} # round an aggregate +{"formula": "abs(revenue:sum - cost:sum)"} # absolute difference +{"formula": "round(revenue:sum / *:count, 2)"} +``` + +On Postgres, 2-argument `round` over a floating-point value is automatically cast to `numeric` so it executes (Postgres has no `round(double precision, integer)` overload). SQLite and DuckDB round `DOUBLE` natively. -These are native on Postgres / DuckDB / MySQL / ClickHouse. SQLite doesn't have most of them in the standard build, so SLayer registers Python implementations on every connection (see `slayer/sql/sqlite_udfs.py`). NULL inputs always return NULL. Math-domain errors (`ln(0)`, `sqrt(-1)`, `pow(0, -1)`) propagate as `sqlite3.OperationalError` — matching Postgres's strict semantics rather than SQLite ≥3.35's silent-NULL built-in `log()`. +These are native on Postgres / DuckDB / MySQL / ClickHouse. SQLite doesn't have most of them in the standard build, so SLayer registers Python implementations on every connection (see `slayer/sql/dialects/sqlite.py`). NULL inputs always return NULL. Math-domain errors (`ln(0)`, `sqrt(-1)`, `pow(0, -1)`) propagate as `sqlite3.OperationalError` — matching Postgres's strict semantics rather than SQLite ≥3.35's silent-NULL built-in `log()`. The 2-arg `log(B, X)` UDF is registered on **every** SQLite version, including ≥3.35 where it overrides the built-in's silent-NULL behaviour to match Postgres's strict error semantics. `ln`, `log10`, and `log2` also always register; the `log2` UDF overrides SQLite ≥3.35's silent-NULL built-in to keep the same strict semantics. diff --git a/docs/concepts/ingestion.md b/docs/concepts/ingestion.md index 59525831..09cff708 100644 --- a/docs/concepts/ingestion.md +++ b/docs/concepts/ingestion.md @@ -51,6 +51,23 @@ FK columns from referenced tables are excluded from the source model to avoid re All models use `sql_table` (the source table) plus `joins` (direct FK joins only, storing source/target column pairs). Multi-hop JOINs are resolved dynamically at query time by walking the join graph. +### SQLite affinity probing + +SQLite's declared column types are affinity hints, not strict constraints: a column declared `INTEGER` can store `INTEGER`, `REAL`, `TEXT`, or `BLOB` values per row. To prevent silent truncation downstream (a column declared `INTEGER` but actually storing `0.99` would cast to `0` and break `AVG`/`SUM` results), SLayer runs an additional value-level probe on SQLite ingestion for every column the inspector reports as `INTEGER`-affinity. + +The probe samples up to **`PROBE_SCAN_CAP + 1` rows** (100,001 by default; configurable via `slayer.sql.sqlite_introspect.PROBE_SCAN_CAP`). The `+1` lets the probe detect saturation — if 100,001 rows come back, there's at least one row past the cap, and the probe declines to certify INT. It decides per column: + +- **DOUBLE** when any row's storage class is `REAL`, or any integer-storage value fails `ROUND(col) = col`, or every distinct TEXT value coerces to a finite `float()`. +- **TEXT** when any row holds a `BLOB`, or any TEXT value is non-coercible / non-finite, or the distinct-text sample saturates the 1,000-distinct-value cap. +- **INT** when the entire sample is integer-shaped and the sample isn't saturated. +- The SA-derived `INT` is kept (probe returns `None`) when the column is empty, all-NULL, the row sample saturates without enough evidence, or the probe itself errors. The probe logs a `WARNING` in the saturated / error cases. + +The probe is **idempotent re-ingest aware**: a persisted `Column.type = INT` is widened to `DOUBLE` / `TEXT` on the next `slayer ingest` if the live storage classes disagree. `Column.format = NumberFormat(INTEGER)` (the auto-ingested default) is flipped to `FLOAT` for `DOUBLE` or cleared for `TEXT`; user-set custom formats (currency, custom precision) are preserved verbatim with an `INFO` log noting the type change. The CLI prints `Updated: (widened: )` so the change is visible. + +Non-SQLite datasources (Postgres, MySQL, DuckDB, ClickHouse, SQL Server) skip the probe entirely — their type systems are strict and this class of bug doesn't exist. + +Already-persisted v7 SQLite models with the wrong `INT` type are **not** auto-repaired on `storage.get_model()` load (running a full table scan per column on every load would be too expensive). Re-ingest is the auto-heal path: `slayer ingest` or `slayer serve --ingest-on-startup`. The DEV-1361 DOUBLE → INT narrowing on legacy-dict migration is also gated on the probe on SQLite — it only fires when the probe positively certifies INT. + ## Usage ### CLI @@ -157,7 +174,7 @@ Each per-datasource pass refreshes embeddings for the datasource doc, every visible model + its visible children, **and every memory whose canonical entities are rooted at the datasource** (DEV-1416). A stale `embeddings.db` (created without an `OPENAI_API_KEY`, or after a manual -`memories.yaml` edit) is therefore repaired by the next +`memories/.md` edit) is therefore repaired by the next `--ingest-on-startup` with no extra step. Per-memory embed failures surface as `IngestionError(model_name="memory:", …)` in the result's `errors` list. @@ -259,15 +276,17 @@ Ingest-on-startup: N/M datasources ingested (K failed: name1, name2) - **Existing `sql_table`-mode model** → append new columns and joins from the live schema. Existing columns and joins are **never** mutated — `description`, `label`, `format`, `meta`, and `allowed_aggregations` are preserved verbatim. - **Existing `sql`-mode or query-backed model with the matching name** → skipped silently; those are user-authored. +With the default YAML storage, two live tables whose quoted names differ only by letter case (`"Orders"` vs `orders`) cannot both be persisted — model names collide as filenames on macOS / Windows, so the save is rejected (`IdCollisionError`). The first table wins; the second surfaces as a per-model entry in `IdempotentIngestResult.errors` (or a per-model message on the CLI / MCP paths) without aborting the rest of the ingest. SQLite storage persists both. + After the additive pass, `validate_models` runs against the in-scope models and the result is merged into the response (`IdempotentIngestResult.to_delete`). Type-bucket drift on existing columns surfaces there — apply via `slayer validate-models --force-clean`, then re-ingest to pick up the new live type. See [Schema Drift](schema-drift.md) for the full diff / cascade contract. ### Search side effects After validation, every ingest also refreshes the search corpus for the touched datasource: -- **Sample values** (`Column.sampled`) — re-profiled for every non-hidden, non-PK column on every table-backed model in the datasource. The cached snapshot is consumed by the tantivy search index and by `inspect_model`. See [Search](search.md#sample-value-cache). -- **Embedding rows** — when the `embedding_search` extra is installed and a usable provider API key is in the environment, the embedding refresh re-runs for the datasource doc plus every visible model + its visible children. `SLAYER_EMBEDDING_MODEL` is an *optional* override of the default (`openai/text-embedding-3-small`); setting it is not required. The SHA256 `content_hash` on each row means re-ingests are cheap when nothing changed. See [Search](search.md#channel-3--dense-embedding-similarity). +- **Sample values** (`Column.sampled`) — **not** re-profiled at ingest. A per-column full-table scan would dominate ingest wall-clock on wide datasources, so samples are populated lazily on the first `inspect` of a column (or explicitly via `slayer search refresh-samples`). See [Search](search.md#sample-value-cache). +- **Embedding rows** — when the `advanced_search` extra is installed and a usable provider API key is in the environment, the embedding refresh re-runs for the datasource doc plus every visible model + its visible children. `SLAYER_EMBEDDING_MODEL` is an *optional* override of the default (`openai/text-embedding-3-small`); setting it is not required. The SHA256 `content_hash` on each row means re-ingests are cheap when nothing changed. See [Search](search.md#channel-3--dense-embedding-similarity). -Both refreshes are best-effort: per-entity runtime failures land in `IdempotentIngestResult.errors` as friendly strings, never aborting ingestion. When the `embedding_search` extra is not installed or no API key is configured for the active embedding model, the embedding pass is silently skipped — the user-visible signal lives on the next `search` response. +Both refreshes are best-effort: per-entity runtime failures land in `IdempotentIngestResult.errors` as friendly strings, never aborting ingestion. When the `advanced_search` extra is not installed or no API key is configured for the active embedding model, the embedding pass is silently skipped — the user-visible signal lives on the next `search` response. `include_tables` / `exclude_tables` constrain the additive pass plus the `sql_table`-mode subset of validation: a `sql_table`-mode model whose table is excluded is left out of both. `sql`-mode and query-backed models in the same datasource are still passed through `validate_models` regardless of the table filter — they are not tied to a specific table name. Run `validate_models` directly (no `--include`/`--exclude`) to validate only those modes. diff --git a/docs/concepts/memories.md b/docs/concepts/memories.md index f6892fe4..437e8070 100644 --- a/docs/concepts/memories.md +++ b/docs/concepts/memories.md @@ -12,14 +12,17 @@ canonical entity matches via tantivy full-text — see the search docs). A memory has two flavours: - **Learning** — a memory with no attached query. Surfaces in - `inspect_model` and in the `memories` list of `search`. + `inspect_model` and as a `kind="memory"` hit in `search` with + `hit.query is None`. - **Query-bearing** — a memory whose `query` field carries a - `SlayerQuery`. Surfaces only in the `example_queries` list of - `search` (capped independently from `memories` so bulky examples - cannot crowd out small notes). + `SlayerQuery`. Surfaces only via `search`, as a `kind="memory"` hit + with `hit.query is not None`. Not rendered in `inspect_model`. -The split is implicit: pass an entity list to `save_memory` to record -a learning; pass a `SlayerQuery` and the memory carries that query. +Both flavours land in the same flat `SearchResponse.results` list +(DEV-1532); callers split on `hit.query` rather than on separate +buckets. The split is implicit at save time: pass an entity list to +`save_memory` to record a learning; pass a `SlayerQuery` and the +memory carries that query. ## The canonical entity form @@ -57,7 +60,7 @@ Memory retrieval is part of [`search`](search.md) (one tool covers both memories and canonical entity discovery). This page covers only the write side. -### `save_memory(learning, linked_entities, id=None)` +### `save_memory(learning, linked_entities, id=None, description=None)` Persist a memory. `linked_entities` accepts either form: @@ -74,12 +77,22 @@ int-shaped id (`"1"`, `"2"`, ...). Supply a string for a stable user-controlled id (`"kb.policy.42"`) — useful for knowledge-base ingestion pipelines. Charset excludes `:`, `/`, `?`, `#`, whitespace, and ASCII control characters. Duplicate id → unconditional **upsert**, -`created_at` preserved. +`created_at` preserved. In the default YAML storage, an id that differs +only by letter case from an existing one (`X` vs `x`) raises +`IdCollisionError` — ids are filenames there, and case variants collide +on macOS / Windows. + +`description` is optional (≤ 500 chars). When set, `search(compact=True)` +and `inspect_model(compact=True)` surface this short preview instead of +the full `learning` body — the token-saving counterpart of the verbose +body. Empty / whitespace-only `description` is normalized to `None` at +save time. When unset, compact rendering falls back to the first +non-empty paragraph of `learning` (capped at 500 chars). Returns `memory_id` (a non-empty string), the canonical entities stored, and any non-fatal warnings. -**Embedding side effect.** When the `embedding_search` extra is +**Embedding side effect.** When the `advanced_search` extra is installed and `SLAYER_EMBEDDING_MODEL` resolves to a configured provider, `save_memory` also embeds the new memory inline so it participates in the embedding-similarity search channel right away. @@ -125,9 +138,12 @@ list (exact-match only — `memory:42` never strips `memory:421`). 1. **Plan the query.** Decide the source model and the columns / measures you intend to use. 2. **Call `search` first.** Pass the entities you're considering (and/or - the draft query, and/or a free-text `question`). Read the returned - `memories` and `example_queries` — they may flag pitfalls you'd - otherwise hit (NULL handling, units, deprecated columns, etc.). + the draft query, and/or a free-text `question`). Walk + `response.results`: hits with `kind="memory"` are prior notes (a + `hit.query is not None` marks a saved example query); other kinds + surface canonical entities matched by the full-text / embedding + channels. Memory hits may flag pitfalls you'd otherwise hit (NULL + handling, units, deprecated columns, etc.). 3. **Issue the actual query** via the `query` tool. 4. **Save what you learn.** When you discover a non-obvious quirk (encoding, NULL semantics, business rule), call `save_memory` @@ -139,9 +155,9 @@ list (exact-match only — `memory:42` never strips `memory:421`). every memory **whose `query` is `None`** and whose stored entity set overlaps the model's own entity set (the model itself, every column, every named measure, every custom aggregation). Query-bearing memories -appear only via `search` (in the `example_queries` bucket). The -section is auto-pruned when there are no matches — no header is -emitted in that case. +appear only via `search` (as `kind="memory"` hits with +`hit.query is not None`). The section is auto-pruned when there are +no matches — no header is emitted in that case. ## Surfaces @@ -160,8 +176,12 @@ For retrieval, see [`search`](search.md) (MCP `search`, REST `POST ## Storage layout -YAML uses a single `memories.yaml` file alongside the model and -datasource folders. SQLite uses a `memories` table plus a +YAML stores one Markdown file per memory at `memories/.md` — YAML +frontmatter for the structured fields (`description`, `entities`, +`query`, `created_at`, `version`) and the Markdown body as the +`learning`. The id is the filename (not repeated in the frontmatter). +A legacy flat `memories.yaml` is migrated into per-file `.md` on first +open (and then deleted). SQLite uses a `memories` table plus a `memory_entities` index table for the entity-overlap filter. IDs are non-empty strings (DEV-1428). The auto-allocator walks diff --git a/docs/concepts/models.md b/docs/concepts/models.md index c76db918..6831a6cc 100644 --- a/docs/concepts/models.md +++ b/docs/concepts/models.md @@ -22,7 +22,7 @@ A query then asks for `revenue:sum` (aggregate the `revenue` column), `aov` (the | Field | Type | Required | Description | |-------|------|----------|-------------| -| `name` | string | Yes | Unique model name | +| `name` | string | Yes | Unique model name. In the default YAML storage, names differing only by letter case (`Orders` vs `orders`) are rejected at save time (`IdCollisionError`) — they would collide as filenames on macOS / Windows | | `sql_table` | string | One of | A physical database table (e.g. `public.orders`) | | `sql` | string | these | A SQL subquery to use as the source | | `source_queries` | list[SlayerQuery] | three | Saved query stages — makes the model **query-backed** | @@ -69,9 +69,9 @@ A column is the unit of structure on the model. The same column entry can serve | `allowed_aggregations` | list[str] | No | — | Whitelist (must be a subset of the type-default eligibility set, or a custom aggregation defined on this model) | | `filter` | string | No | — | SQL condition applied inside `CASE WHEN` at aggregation time. See [Filtered columns](#filtered-columns) | | `meta` | dict | No | — | Arbitrary JSON metadata | -| `sampled` | string | No | — | Cached sample-value text snapshot (top-20 by frequency joined, or `top20 ... (N distinct)` on overflow, or `min .. max` for numeric/temporal); populated by `slayer ingest` and friends | +| `sampled` | string | No | — | Cached sample-value text snapshot (top-20 by frequency joined, or `top20 ... (50+ distinct)` on overflow, or `min .. max` for numeric/temporal); populated lazily on the first `inspect` of the column (or via `slayer search refresh-samples`), not at ingest time | | `sampled_values` | list[str] | No | — | Structured top-50-by-frequency list (categorical only); the unambiguous counterpart to `sampled` for consumers that need to compare predicate literals against stored values. `None` for numeric/temporal columns | -| `distinct_count` | int | No | — | True total cardinality at profile time (categorical only). Set via a secondary `count_distinct` query when overflow is detected, so it's exact rather than capped. `None` for numeric/temporal columns | +| `distinct_count` | int | No | — | Exact distinct count when ≤ 50 (categorical only). `None` on overflow (> 50 distinct — one scan only, no secondary `count_distinct` query) and for numeric/temporal columns | ### Data types @@ -89,12 +89,12 @@ A column with no explicit `allowed_aggregations` whitelist gets a default set ba | Type | Default eligible aggregations | |------|-------------------------------| -| `number` | sum, avg, min, max, count, count_distinct, median, weighted_avg, percentile, first, last, stddev_samp, stddev_pop, var_samp, var_pop, corr, covar_samp, covar_pop | -| `string` | count, count_distinct, first, last, min, max | -| `boolean` | count, count_distinct, sum, min, max, first, last | -| `date` / `time` | count, count_distinct, first, last, min, max | +| `number` | sum, avg, min, max, count, count_distinct, count_distinct_approx, median, weighted_avg, percentile, first, last, stddev_samp, stddev_pop, var_samp, var_pop, corr, covar_samp, covar_pop | +| `string` | count, count_distinct, count_distinct_approx, first, last, min, max | +| `boolean` | count, count_distinct, count_distinct_approx, sum, min, max, first, last | +| `date` / `time` | count, count_distinct, count_distinct_approx, first, last, min, max | -Primary-key columns are always restricted to `count` / `count_distinct` regardless of type. When `allowed_aggregations` is set, every entry must already be eligible under the type-default map (or be a custom aggregation defined on this model); violations are caught at model construction time, so query-time validation is a single membership check. +`count_distinct_approx` is dialect-aware: it emits the database-native approximate-distinct function where one exists and falls back to an exact `COUNT(DISTINCT)` where it does not (Postgres / SQLite / MySQL). Primary-key columns are always restricted to `count` / `count_distinct` / `count_distinct_approx` regardless of type. When `allowed_aggregations` is set, every entry must already be eligible under the type-default map (or be a custom aggregation defined on this model); violations are caught at model construction time, so query-time validation is a single membership check. ### Filtered columns diff --git a/docs/concepts/queries.md b/docs/concepts/queries.md index ad6212e1..52863906 100644 --- a/docs/concepts/queries.md +++ b/docs/concepts/queries.md @@ -21,6 +21,8 @@ A `SlayerQuery` specifies what data to retrieve from a model. You can pass a single query or a **list of queries** to `execute()`. When passing a list, earlier queries are named sub-queries that later queries can reference. The last query in the list is the main one whose results are returned. See [Query Lists](#query-lists) for examples. +A query carries no tenant scoping of its own. To force every query through an engine to one tenant's rows — joins and sub-queries included — configure a policy at engine construction; see [Row-Level Security](row-level-security.md). + ## Dimensions Each entry in `dimensions` is either a bare string (the canonical short form for a column without a custom label) or a `ColumnRef` dict with `name` and optional `label`. Both styles support dotted paths for joined models, auto-resolved via the join graph. @@ -51,6 +53,33 @@ A query with no measures and at least one dimension or time-dimension returns th Emits `SELECT orders.status FROM orders GROUP BY orders.status LIMIT 100`. +### Raw rows (`distinct_dimension_values`) + +Set `distinct_dimension_values: false` on a query to opt out of the auto-dedup and project raw rows instead. No top-level `GROUP BY`; the usual `WHERE` / `ORDER BY` / `LIMIT` / `OFFSET` still apply. + +```json +{ + "source_model": "orders", + "dimensions": ["status", "amount"], + "filters": ["amount > 100"], + "order": [{"column": "amount", "direction": "desc"}], + "limit": 100, + "distinct_dimension_values": false +} +``` + +Emits roughly `SELECT orders.status, orders.amount FROM orders WHERE orders.amount > 100 ORDER BY orders.amount DESC LIMIT 100` — one row per source row. + +**Rules** when `distinct_dimension_values=False`: + +- `measures` must be empty — `DistinctDimensionValuesError` otherwise. +- At least one of `dimensions` / `time_dimensions` must be non-empty (nothing to project otherwise). +- Filters / order items must not reference any measure — neither colon-form (`amount:sum > 100`, `*:count > 0`), transform calls (`rank(amount:sum) <= 5`), nor a bare saved-`ModelMeasure` name. + +**Time dimensions** are allowed: each one emits its `DATE_TRUNC` truncation as a projected column without aggregating. For raw column values (no truncation), put the time column in `dimensions` instead. + +**Multi-stage**: the flag is per-stage in DAG queries — an inner stage with `false` produces a flat raw-row sub-query that the outer stage can aggregate over. + ## TimeDimension A time dimension with a required granularity and an optional date range. Supports an optional `label` for human-readable output. To use a time column without truncation, add it as a regular dimension instead. @@ -64,16 +93,58 @@ A time dimension with a required granularity and an optional date range. Support } ``` -**Granularities**: `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year` +**Granularities**: `second`, `minute`, `hour`, `day`, `week`, `week_sunday`, `month`, `quarter`, `year` + +`week` is Monday-anchored (ISO-8601); `week_sunday` is Sunday-anchored (weeks start Sunday, end Saturday) for tools that use Sunday weeks. Both are model granularities you set on a `TimeDimension` — `week_sunday` is the SLayer value, not a wire keyword sent by a BI tool. + +`date_range` and an equivalent explicit filter (`"created_at >= '2024-01-01' and created_at <= '2024-12-31'"`) are interchangeable — including for trailing-window measures and `time_shift`, which still read rows from before the range so the earliest bucket isn't short-changed. See [Time bounds do not clip the window](formulas.md#time-bounds-do-not-clip-the-window) for exactly which predicates count as a time bound. ## OrderItem +A sort specification: `column` is the short alias (`status`, `revenue_sum`, `*:count`), `direction` is `asc` or `desc`. + ```json {"column": "*:count", "direction": "desc"} ``` Via MCP: `{"column": "*:count", "direction": "desc"}` +### Ordering by something you don't project + +`order` may reference a column or aggregate that is **not** declared as a dimension/measure — the classic "top-N by metric X, display only Y, Z" pattern: + +```json +{"source_model": "orders", "dimensions": ["status"], "measures": [{"formula": "*:count"}], + "order": [{"column": "amount:sum", "direction": "desc"}], "limit": 10} +``` + +The `amount:sum` aggregate is computed as a hidden column, sorted on, and **stripped from the result** — the response projects only `status` and `_count`. This works for local aggregates, cross-model aggregates (`customers.revenue:sum`), and inner-stage columns re-aggregated in a later DAG stage (`customers__revenue_sum:max`). + +What each shape of an *undeclared* order target does: + +| Order target | Behavior | +| --- | --- | +| An aggregate (`amount:sum`, `customers.revenue:sum`) | Computed hidden, sorted on, stripped from the result. Always allowed. | +| An inline **transform** (`rank(amount:sum)`, `cumsum(...)`, `change(...)`, `lag`/`lead`/`ntile`) | Computed hidden, sorted on, stripped. | +| An inline **composite** (`revenue:sum / cnt:sum`, `abs(amount:sum)`, `change(amount:sum) / 2`) | Computed hidden, sorted on, stripped. | +| A **windowed** aggregate (`amount:sum(window='90d')`), alone or inside a composite | Computed hidden in its own rolling-window CTE, sorted on, stripped. | +| A raw row column, in a **raw-rows** query (`distinct_dimension_values: false`, no measures) | Sorted on directly (`ORDER BY orders.created_at`). | +| A raw row column, in an **aggregated / dedup** query | Rejected (HTTP 400): it isn't in the `GROUP BY`. Add it to `dimensions`, or order by an aggregate of it (`created_at:max`). | +| A **joined** row column (`customers.regions.name`) not projected | Rejected (HTTP 400): project it (add to `dimensions`) or order by a projected field. | + +Transform and composite order targets accept the full formula syntax, so +`{"column": "revenue:sum / cnt:sum"}` and `{"column": "change(revenue:sum)"}` both +work without declaring a measure. One limit: the operands must be written as +formulas, not as the *names* of measures you declared in the same query — +`{"column": "rev / cnt"}` is rejected at validation, because referencing a +declared measure by its alias inside an expression is not supported anywhere in +SLayer. Write `{"column": "revenue:sum / cnt:sum"}` instead. + +A windowed measure inside a **declared** composite measure +(`{"formula": "revenue:sum(window='90d') / cnt:sum"}`), and any combination of a +windowed measure with a transform, are still rejected — see +[formulas](formulas.md#windowed-sum-and-average). + ## Response Query results are returned as a `SlayerResponse`: @@ -294,6 +365,38 @@ MCP equivalent: `query(source_model="", variables={...}, dry_run=True/Fal --- +## Choosing a root model + +When you know the columns and metrics you want but not which model to use as `source_model`, `recommend_root_model` introspects the join graph and picks it for you. Give it the `model.column` / `model.metric` items (aggregation suffixes allowed) and it returns the recommended root plus each item's join-qualified reference path from that root — ready to paste into a query. + +```python +rec = engine.recommend_root_model_sync(["customers.name", "products.category"]) +rec.root_model # "orders" (the bridge model that reaches both) +{ip.input_item: ip.path for ip in rec.item_paths} +# {"customers.name": "customers.name", "products.category": "products.category"} +``` + +A root is valid when every requested item is reachable from it over the join graph — LEFT joins are directional (source → target), INNER joins traverse both ways. Among valid roots, the one with the fewest total join hops wins. Root-owned items come back as a bare leaf (`status`); joined items as a dotted path (`customers.regions.name`); aggregation suffixes are preserved (`revenue:sum`). + +When no single model reaches everything, `root_model` is `None`, `reachable` is `False`, and `coverage` lists the best partial roots (each with its reachable / unreachable items) so you can split the request into a multi-stage [`source_queries`](models.md#query-backed-models) query. + +### Forcing a root with `root_hint` + +Sometimes you already know the host you want — often a **bridge** model that owns none of the requested items but matches the grain you're building on. Pass `root_hint` (a bare model name or `.`) to force it: + +```python +rec = engine.recommend_root_model_sync( + ["customers.name", "regions.name"], root_hint="orders" +) +rec.root_model # "orders" (honored — it reaches both, overriding the closer auto-pick) +``` + +When the hint reaches every item it's honored outright, overriding the fewest-hops pick. When it can't reach everything, the auto-pick is used instead and `warnings` explains which owning models the hint missed and which root was chosen. If no model reaches everything (`reachable` is `False`), the hint's own row is included in `coverage` too, so you can see exactly what it reaches. `root_hint` is resolved after the datasource is fixed from the items, so it names a model *within* that datasource — it can't choose the datasource. A hint that isn't a model in the resolved datasource raises. + +Surfaces: MCP `recommend_root_model(items, data_source=None, root_hint=None, format="markdown")`, REST `POST /recommend-root-model` (`{"items": [...], "data_source": null, "root_hint": null}`), CLI `slayer recommend-root-model ITEM... [--data-source X] [--root-hint M] [--format json|text]`, and `SlayerClient.recommend_root_model(_sync)`. The optional `data_source` scopes name resolution to one datasource; all items must resolve to a single datasource. + +--- + ## Examples ### Count by status diff --git a/docs/concepts/query-cache.md b/docs/concepts/query-cache.md new file mode 100644 index 00000000..df6e6c8a --- /dev/null +++ b/docs/concepts/query-cache.md @@ -0,0 +1,144 @@ +# Query Cache + +SLayer has an optional, in-memory, per-query result cache **local to a single +`SlayerQueryEngine` instance**. It is opt-in per call via `cache=True`, modelled +on [Cube's in-memory cache](https://cube.dev/docs/product/caching). This is a +**Python-API-only** feature — there is no REST / MCP / CLI / Flight / pg-facade +surface, and no `SlayerClient` plumbing. + +Two engines with different connection settings keep separate caches, so a cached +result is never served across datasource identities. + +> **See it run:** the [Query Cache notebook](../examples/12_query_cache/query_cache_nb.ipynb) +> walks through miss → hit, staleness, `refresh()`, TTL, and eviction on a tiny live database. + +## Enabling the cache + +Construct the engine with a `CacheConfig`, then pass `cache=True` per call: + +```python +from slayer.engine.cache import CacheConfig +from slayer.engine.query_engine import SlayerQueryEngine + +engine = SlayerQueryEngine( + storage, + cache_config=CacheConfig( + ttl_seconds=300, + refresh_keys=[("public.orders", "MAX(updated_at)")], + ), +) + +# Any input shape works — SlayerQuery / dict, a multi-stage list, or a +# run-by-name string. Each funnels into one final SQL that is cached. +resp = await engine.execute({"source_model": "orders", + "measures": [{"formula": "amount:sum"}]}, + cache=True) +``` + +`cache_config` defaults to an empty `CacheConfig()` when not supplied. +Reassigning `engine.cache_config = CacheConfig(...)` **clears the cache**. + +`cache=True` is ignored (no caching, no error) when `dry_run` or `explain` is +set. + +### `CacheConfig` + +| Field | Meaning | +| --- | --- | +| `ttl_seconds: Optional[float]` | Wall-clock age bound. `None` (default) means no time-based expiry. | +| `refresh_keys: List[Tuple[str, str]]` | Cube-style `(physical_table, select_expression)` pairs. The same table may repeat with different expressions. | + +An empty/default `CacheConfig` caches indefinitely with no automatic staleness — +only `evict` / `clear_cache` / an explicit re-execution change an entry. + +## Cache key + +The key is `sha256(final_generated_sql + "|" + connection_string + "|" + +runtime_fingerprint)`. Variables are already substituted into the SQL before the +key is computed, so different variable sets produce different keys automatically. +The datasource identity is the engine's SQL-client fingerprint (not the bare +datasource name), so a config edit under the same name never serves the wrong +rows. + +Because the key needs the SQL, the resolve → enrich → SQL-generation pipeline +runs on **every** cached call (cheap, no DB hit). A cache hit skips only the DB +execution and result decode. + +## Staleness + +Each entry has two independent staleness signals. + +**TTL** is checked lazily on read: if an entry's age exceeds `ttl_seconds`, the +read is treated as a miss and the entry is re-executed synchronously. + +**Refresh keys** are acted on only by an explicit `engine.refresh()`. For each +entry, the *applicable* refresh keys are those whose table is referenced by the +entry's SQL. A baseline value per applicable key is captured at cache-write time +(scanned **before** the data query, so cached data always reflects a state at +least as new as the baseline). `refresh()` re-evaluates each applicable key and +re-executes the entry if any value differs. + +Refresh-key sensitivity is your choice of expression: + +- `MAX(updated_at)` misses in-place updates that don't move the column, backfills + below the current max, and inserts with old timestamps. +- A `COUNT(*)`-bearing expression additionally catches deletes. +- A hash / concatenation (e.g. `MAX(updated_at) || '|' || COUNT(*)`) catches more. + +`ttl_seconds` is the time-based backstop. SLayer evaluates each +`select_expression` verbatim as `SELECT FROM ` — it is +**not** wrapped in `MAX(...)`. + +## Management methods + +```python +await engine.refresh() # -> RefreshResult (+ engine.refresh_sync()) +await engine.evict(query) # -> bool (+ engine.evict_sync()) +engine.clear_cache() # drop all entries +engine.cache_size # -> int (live entry count) +``` + +`evict(query, variables=None, data_source=None)` accepts the same input union as +`execute`, recomputes the SQL + datasource key (no DB execution), and removes +that one entry. `refresh()` returns a `RefreshResult`: + +```python +class RefreshResult: + refreshed: List[str] # re-run because a refresh-key value moved + expired_refreshed: List[str] # re-run because the TTL lapsed + unchanged: List[str] + errors: List[RefreshError] # continue-on-failure diagnostics +``` + +`refresh()` re-prepares each stale entry from its **original input** through the +full `execute` pipeline (not the frozen SQL), so `whole_periods_only` boundaries +re-snap and model/schema edits are picked up. If the freshly-prepared SQL differs +(a new period, an edited model), the entry is re-keyed under the new SQL hash. +`refresh()` is continue-on-failure: a failed refresh-key scan or re-execution +leaves the existing entry unchanged and records a `RefreshError`. + +`refresh()` scans and re-executes each entry against the **datasource identity +it was cached under** (its SQL-client fingerprint), reusing the client created +at write time — so a `datasource`-priority change or a same-name connection edit +between caching and `refresh()` can never migrate a table-backed entry or read +from a different database. One residual: a run-by-name **query-backed** model +resolves its inner `source_queries` metadata through the current datasource +priority, so if the same backing-model name lives in two datasources and the +priority changes between caching and `refresh()`, the inner SQL may be shaped +from the other datasource. After repointing or re-prioritising a datasource +while a cache is live, call `clear_cache()`. + +The synchronous wrappers `execute_sync(..., cache=True, data_source=...)`, +`refresh_sync()`, and `evict_sync(...)` are available for CLI / notebook / +script use. + +## Limitations + +- **In-memory, per-process.** The cache lives on the engine instance; it is not + persisted across restarts and is not shared across engines or processes. +- **No request coalescing.** Concurrent identical misses both execute + (last-writer-wins on store). +- **Unbounded.** There is no LRU or size cap; manage memory with `evict` / + `clear_cache`. +- **Python API only.** No REST / MCP / CLI / Flight / pg-facade / `SlayerClient` + surface. diff --git a/docs/concepts/row-level-security.md b/docs/concepts/row-level-security.md new file mode 100644 index 00000000..d152cd41 --- /dev/null +++ b/docs/concepts/row-level-security.md @@ -0,0 +1,250 @@ +# Row-Level Security (Forced Filter) + +SLayer can scope every query a session runs to a single tenant, so an agent +only ever sees that tenant's rows — across joins, CTEs, sql-mode sub-queries, +query-backed stages, and profiling/sample data. The scoping is **immutable +engine state**: the agent cannot read it, override it, or disable it through +any query field. + +A policy carries exactly one **ruleset**, one of two kinds: + +- A **`ColumnFilterRuleset`** — "every physical table that has column `C` is + filtered to `C = ` (or `C IN (...)`)". This fits the common shape + where the same tenant column (e.g. `organization_uuid`) is present on every + table. +- A **`JoinFilterRuleset`** — the tenant identifier lives on **one** anchor + table; every other table reaches it through an explicit join, and a + `whitelist` names the shared tables that need no filtering. Any table that is + neither the anchor, a join target, nor whitelisted **fails closed**. + +For a runnable walkthrough on the Jaffle Shop demo, see the +[Row-Level Security notebook](../examples/10_row_level_security/row_level_security_nb.ipynb). + +## Configuring a policy + +A policy is set once, at engine (or local-engine client) construction: + +```python +from slayer.core.policy import SessionPolicy, ColumnFilterRuleset +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.storage.yaml_storage import YAMLStorage + +storage = YAMLStorage(base_dir="./slayer_data") # your configured backend + +policy = SessionPolicy( + ruleset=ColumnFilterRuleset(column="organization_uuid", value="7ef3ab6c-...."), +) +engine = SlayerQueryEngine(storage=storage, policy=policy) +``` + +Every query the engine runs is now tenant-scoped, with no model or query +changes: + +```python +resp = await engine.execute({ + "source_model": "orders", + "measures": [{"formula": "*:count"}], +}) +# -> count of THIS org's orders only; a join to customers/regions is +# org-scoped on every side too. +``` + +The same `policy=` argument works on the local-engine client: + +```python +from slayer.client import SlayerClient + +client = SlayerClient(storage=storage, policy=policy) +df = client.query_df({"source_model": "orders", "measures": [{"formula": "*:count"}]}) +``` + +`ruleset` is **required** — the "no filtering" case is simply `policy=None` +(the default). A bare `SessionPolicy()` raises rather than silently building a +no-op. + +## Column ruleset + +### Operator: scalar vs list + +The `value` shape selects the operator: + +```python +# Single tenant -> column = value +ColumnFilterRuleset(column="organization_uuid", value="7ef3...") + +# Several tenants in one session -> column IN (...) +ColumnFilterRuleset(column="organization_uuid", value=["7ef3...", "a1b2..."]) +``` + +### Tables that lack the column: `on_unapplicable` + +A table that **confirms it does not have** the column is handled by +`on_unapplicable`: + +- `"block"` (default) — fail the whole query, naming the table. Use this when + every table is expected to carry the tenant column; a table that doesn't is + a leak you want surfaced. +- `"pass"` — leave that table unfiltered (it is treated as shared/global data). + +```python +# Allow column-less (shared) tables through unfiltered instead of failing: +SessionPolicy(ruleset=ColumnFilterRuleset( + column="organization_uuid", value="7ef3...", on_unapplicable="pass")) +``` + +A table whose column presence **cannot be confirmed** (an introspection error) +always fails closed — the query is blocked regardless of `on_unapplicable`. +This is a deliberate security control: SLayer never emits an unscoped query on +a table it could not verify. + +## Join ruleset + +When the tenant column lives on **only one** table, use a `JoinFilterRuleset`. +It names the anchor `table` + `column` + `value` that hold the identifier; +every other table either reaches the anchor through an explicit `JoinFilterRule` +or is listed in the `whitelist`. + +```python +from slayer.core.policy import ( + SessionPolicy, JoinFilterRuleset, JoinFilterRule, +) + +policy = SessionPolicy(ruleset=JoinFilterRuleset( + # The tenant identifier lives on customers.organization_uuid. + table="customers", + column="organization_uuid", + value="7ef3...", + joins=[ + # orders lacks the column -> reach it via orders.customer_id = customers.id + JoinFilterRule( + target_table="orders", + join_path=["orders.customer_id = customers.id"], + ), + # line_items reaches it multihop: line_items -> orders -> customers + JoinFilterRule( + target_table="line_items", + join_path=[ + "line_items.order_id = orders.id", + "orders.customer_id = customers.id", + ], + ), + ], + # exchange_rates is shared reference data — emit it unfiltered. + whitelist=["exchange_rates"], +)) +``` + +Each table is scoped as follows: + +- **The anchor** (`customers`) is filtered directly, exactly like a column + ruleset: `WHERE organization_uuid = '7ef3...'`. +- **A join target** (`orders`, `line_items`) is scoped by a correlated + `EXISTS` semi-join along its `join_path` — cardinality-safe (it never + multiplies rows) and `LEFT JOIN`-preserving. `orders` becomes: + + ```sql + FROM (SELECT * FROM orders AS _rls_src + WHERE EXISTS ( + SELECT 1 FROM customers AS _rls_j0 + WHERE _rls_j0.id = _rls_src.customer_id + AND _rls_j0.organization_uuid = '7ef3...' + )) AS orders + ``` + +- **A whitelisted table** (`exchange_rates`) is emitted untouched. +- **Anything else** — a table that is not the anchor, not a join target, and + not whitelisted — **fails closed** (raises), so a table the operator forgot + can never leak. + +### The join path + +Each hop is a string `"from_table.from_column = to_table.to_column"` in +**physical DB table/column names** (not model names; tables optionally +schema/catalog-qualified). A path connects the target table to the anchor at +its two endpoints, and may be written in **either** direction — target-first or +anchor-first: + +```python +# these two are equivalent +JoinFilterRule(target_table="orders", join_path=["orders.customer_id = customers.id"]) +JoinFilterRule(target_table="orders", join_path=["customers.id = orders.customer_id"]) +``` + +`value` selects the operator exactly like a column ruleset (scalar → `=`, +non-empty list → `IN`). A target schema-qualified as `public.orders` matches +only the same-schema table; a bare `orders` matches the table in any schema +(case-insensitive). + +### Trust model + +The policy author is trusted; the agent is not. SLayer emits the anchor +`column`, the join-path table/column names, and the `whitelist` entries +**verbatim** — it does not introspect them. So: + +- A **bare** table name matches the table in **any** schema. When your tenant + data spans more than one schema, schema-qualify the anchor `table`, every + `target_table`, every hop table, and every `whitelist` entry + (`public.orders`, `public.customers`, …). A mismatched path can only + over-filter / mis-scope (the terminal tenant predicate is always emitted) — + it cannot mass-leak — but it can silently return the wrong rows. +- The anchor's `column` is trusted to exist; a typo surfaces as a database + error at execution, not a silent pass. +- Intermediate hop tables are part of the enforcement path, not agent input, + so they are not classified against the whitelist. The `whitelist` governs + only which tables an agent's query may read **directly**, unfiltered. + +### ClickHouse + +Correlated subqueries are experimental on ClickHouse and require **server +≥ 25.4**. When a join target is scoped on ClickHouse, SLayer probes the server +version once per datasource, attaches +`SETTINGS allow_experimental_correlated_subqueries = 1`, and logs a warning. +An older (or undeterminable) server version **fails closed** — the query is +blocked rather than run unscoped. + +## How it works + +The filter is applied at the final-SQL layer: each physical-table reference is +wrapped in place (a filtered sub-query for a column filter / the anchor, or a +correlated-`EXISTS` sub-query for a join target), preserving its alias. + +```sql +-- before +FROM orders +LEFT JOIN customers c ON c.id = orders.customer_id + +-- after (ColumnFilterRuleset on organization_uuid = '7ef3...') +FROM (SELECT * FROM orders WHERE organization_uuid = '7ef3...') AS orders +LEFT JOIN (SELECT * FROM customers WHERE organization_uuid = '7ef3...') AS c + ON c.id = orders.customer_id +``` + +Wrapping the table (rather than appending to the outer `WHERE`) preserves +`LEFT JOIN` semantics. Filter values are always emitted as bound literals, so +the rewrite is injection-safe. Previewing a query with `dry_run=True` returns +exactly the SQL that would execute, including the wraps. + +## Scope and limits + +- The policy is **engine-global** — it applies to whatever datasource a query + targets. Per-model / per-datasource scoping is a future addition. +- It is enforced in the **local engine** only. Passing `policy=` to a + `SlayerClient` in HTTP mode raises — server-side policy is not yet available. +- Join paths are **explicit** (authored in the policy), never auto-discovered + from model joins or BFS-resolved. Alternate-column-per-table overrides, + composite-key hops (one column pair per hop), auto/BFS join resolution, + multiple join paths to one target (diamond), and a multi-column + `ColumnFilterRuleset` are out of scope. +- The wrapper preserves each table's original alias (or the bare table name if + no alias was written). SLayer-generated SQL references columns by table + alias, so this is transparent. A hand-written `sql`-mode model that + *schema-qualifies its own column references* (`SELECT public.orders.id ...`) + is the one shape that won't resolve against the wrapped alias — such a query + errors rather than executing. It fails safe (it cannot leak another tenant's + rows); reference columns by table name (`orders.id`) instead. +- Cross-catalog (three-part `catalog.schema.table`) references — e.g. a + BigQuery query spanning two projects — cannot be confirmed by a + `ColumnFilterRuleset`'s schema-only column probe, so under that ruleset they + **fail closed** (the query is blocked). Single-catalog usage (the table's + catalog matches the connection's own) is unaffected. Catalog-aware + introspection is a future addition. diff --git a/docs/concepts/schema-drift.md b/docs/concepts/schema-drift.md index b1c4b1f1..04f722d7 100644 --- a/docs/concepts/schema-drift.md +++ b/docs/concepts/schema-drift.md @@ -130,6 +130,26 @@ models and the result is merged into `IdempotentIngestResult.to_delete`. `include_tables` / `exclude_tables` constrain *both* the additive pass and the validator — excluded tables are not touched in either direction. +## SQLite affinity-probe drift + +On SQLite, `validate_models` additionally runs the value-level affinity +probe (see [Ingestion → SQLite affinity probing](ingestion.md#sqlite-affinity-probing)) +against every persisted base column with `type = INT`. When the probe +disagrees with the persisted type — the live storage class is `REAL`, +non-coercible `TEXT`, `BLOB`, or otherwise non-integral numeric evidence — +`validate_models` emits an `EditModelDelete` removing the column with +reason `"SQLite affinity probe widened .from INT to +; re-run slayer ingest to recreate with the correct type."` + +The probe drop is merged into the model's diff state **before** the +cascade fixed-point walk, so dependent measures, derived columns, joins, +and filters that reference the widened column are cascade-dropped in the +same pass. Pair with `--force-clean` to apply the deletes; the next +`slayer ingest` recreates the column with the correct type. + +Probe failures and saturated samples produce no drift entry (a `WARNING` +is logged instead). Non-SQLite datasources skip the probe entirely. + ## FK-introspection limitations Some dialects (ClickHouse, BigQuery, Snowflake) don't expose foreign-key diff --git a/docs/concepts/search.md b/docs/concepts/search.md index a8ab21b6..a87a10a3 100644 --- a/docs/concepts/search.md +++ b/docs/concepts/search.md @@ -6,8 +6,16 @@ aggregations) using up to three parallel retrieval channels merged by Reciprocal Rank Fusion. It is the **only** retrieval surface — there is no separate recall tool. +!!! tip "`search` vs `inspect`" + `search` is the *fusion / ranking* surface — use it to discover entities + or to pull an entity back **in context** with its tagged memories. When + you already know the exact entity and just want its definition, use the + [`inspect`](../reference/mcp.md) tool instead: `inspect(reference, + entity_type)` returns the rendered detail for **exactly one** entity — + no ranking, no `cypher_filter`, and no bundled memories. + A third channel (dense embeddings via litellm) is gated behind the -optional `embedding_search` extra. When the extra is not installed or +optional `advanced_search` extra. When the extra is not installed or no provider API key is configured, the embedding channel emits a warning into `SearchResponse.warnings` and search degrades gracefully via tantivy + BM25 alone. @@ -20,7 +28,7 @@ supplied. ## The three retrieval channels -### Channel 1 — entity-overlap BM25 over memories +### Channel 1 — entity-overlap BM25 with implicit self-references Inputs are resolved to canonical entity strings (``, `.`, or `..` — see @@ -28,6 +36,40 @@ or `..` — see each memory's stored entity tags via `BM25Plus`. Memories with zero overlap are excluded. +**Implicit self-references (DEV-1513).** Channel 1 contributes to BOTH +the memory ranking AND the entity ranking via a single unifying model: +every doc is conceptually tagged with an implicit reference to itself. + +- A memory `M` is treated as having effective tags `M.entities ∪ {memory:}`, so an `entities=["memory:"]` ref surfaces the named memory itself at the top of the memory BM25 ranking. +- An entity `E` is treated as having a single tag `{}`, so an `entities=[".."]` ref surfaces the named entity at the top of the entities bucket. + +Concretely: + +```json +{ + "call": {"entities": ["mydb.orders.amount"], "max_results": 5}, + "response": { + "results": [{"id": "mydb.orders.amount", "kind": "column", "score": 0.0}] + } +} +``` + +```json +{ + "call": {"entities": ["memory:42"], "max_results": 5}, + "response": { + "results": [{"id": "42", "kind": "memory", "matched_entities": ["memory:42"]}] + } +} +``` + +Filter rules for the new entity surfacing: + +- `memory:` refs contribute to the memory portion of the ranking only — they surface as `kind="memory"` hits. +- Refs not rooted at `datasource` (when set) drop with a warning `entity '' is not rooted at datasource ''; dropped from entities bucket.` The memory side fires the symmetric `memory: is not rooted at datasource ''; dropped.` when the named memory has no entities rooted at the requested datasource. +- Refs on a hidden model or hidden column drop from the entities portion with `entity '' is on a hidden model/column; dropped from entities bucket.` BM25 over original memory tags is unaffected — memories tagged with that canonical still surface. +- An explicitly-named `memory:` whose attached `Memory.query` has stale references emits the standard stale-query warning (the user explicitly asked for that memory; they deserve to know the query is broken). + Activated when `entities` and/or `query` is supplied to `search`. ### Channel 2 — tantivy full-text over memories ∪ entities @@ -62,7 +104,7 @@ top-k cosine similarities are computed with numpy. Activated when **all of the following** hold: - `question` is supplied; -- the `embedding_search` extra is installed (`pip install motley-slayer[embedding_search]`); +- the `advanced_search` extra is installed (`pip install motley-slayer[advanced_search]`); - at least one embedding row exists for the active model name; - the query-embedding call succeeds. @@ -108,23 +150,22 @@ Memory rankings from every active channel are fused via RRF (`k = 60`): score(d) = Σ_r 1 / (k + rank_r(d)) ``` -Entity rankings from channels 2 and 3 are RRF-fused the same way. -Channel 1 contributes to the memory ranking only (it operates on -memory entity tags, not on entity docs). +Entity rankings from channels 1, 2, and 3 are RRF-fused the same way. +Channel 1's entity ranking is the user-supplied canonical refs in +supplied order (DEV-1513); channels 2 and 3 contribute fuzzy hits. -### Per-bucket ranking invariance (DEV-1414) +### Ranking stability (DEV-1414) Each channel produces a **full per-kind ranking** — channel 2 runs as two kind-filtered tantivy queries (one over memory docs only, one over entity docs only), and channel 3 partitions the embedding corpus by -`entity_kind` and ranks each side independently. There is no shared -candidate-pool budget across kinds, so for a fixed -`(question, datasource, max_X)` the membership and order of the -returned `X` bucket (`memories` / `example_queries` / `entities`) is a -pure function of the corpus + question + that one cap. Varying the -other two caps cannot move an id in or out of the returned list nor -reorder it. The `max_*` caps are pure post-fusion slice operations on -the three independent ranked lists. +`entity_kind` and ranks each side independently. The per-kind rankings +are RRF-fused into a single flat list before the `max_results` cap is +applied. Because the fusion is deterministic, the relative order of any +subset of the flat list is stable with respect to the corpus and +question — changing only `max_results` never reorders existing entries +nor causes an entry to appear or disappear unless the cap boundary +moves past it. ## Tool surface @@ -134,9 +175,9 @@ search( query: Optional[Union[SlayerQuery, dict]] = None, question: Optional[str] = None, datasource: Optional[str] = None, - max_memories: int = 5, - max_example_queries: int = 2, - max_entities: int = 5, + max_results: int = 10, + cypher_filter: Optional[str] = None, + compact: bool = True, ) -> SearchResponse ``` @@ -144,25 +185,53 @@ search( |---|---| | MCP | `search(entities=[...], question="...")` tool | | REST | `POST /search` with `SearchRequest` body | -| CLI | `slayer search --entity --question "..." [--format json]` | +| CLI | `slayer search --entity --question "..." [--format json] [--verbose]` | | Python client | `await client.search(entities=[...], question="...")` | +### `compact=True` default (0.7.3) + +Compact-by-default trims per-hit payloads so a normal `search` call +returns ~10× less text than the pre-0.7.3 verbose shape. The trim is +symmetric across hit kinds: + +- **Memory hits** — `SearchHit.description` holds `Memory.description` + when the user set it on `save_memory`, otherwise the first non-empty + paragraph of `Memory.learning` capped at 500 chars. `SearchHit.text` + is empty (`""`). `query` is preserved for example-query memories. +- **Entity hits** (datasource / model / column / measure / aggregation) + — `SearchHit.description` holds the entity's structured + `description` field (`None` when the entity has none). `SearchHit.text` + is empty. +- `matched_entities` is identical across modes. + +Pass `compact=False` (CLI: `--verbose`) to restore the full per-hit +render — useful when drilling into a single memory: + +```json +{"entities": ["memory:42"], "max_results": 1, "compact": false} +``` + ### `datasource` filter (DEV-1409) All four surfaces accept an optional `datasource: Optional[str] = None` argument. When set, every channel pre-filters its corpus to that one datasource: -- **Entity hits** (channels 2 and 3) include only docs whose +- **Entity hits** (channels 1, 2, and 3) include only docs whose `canonical_id` is rooted at the requested datasource — exact name match (``) or strict dotted-path descendant (`.`, `..`). Character-prefix matches do NOT qualify, so `datasource="prod"` excludes a sibling datasource named `prod_v2`. + Channel 1 (DEV-1513) drops a user-supplied `entities=` ref that + isn't rooted at the requested datasource with a warning rather than + silently surfacing it. - **Memory hits** (channels 1, 2, 3, and the recency fallback) include only memories whose `entities` list has at least one entry rooted at the requested datasource. A memory that references both `prod.*` and `staging.*` surfaces from each datasource when each is filtered - independently; an untagged memory drops out under any filter. + independently; an untagged memory drops out under any filter. A + user-supplied `entities=["memory:"]` ref whose memory was + filtered out emits a symmetric warning. - BM25 and tantivy IDF statistics reflect the filtered subset only — pre-filter, not post-filter. The embedding cosine corpus (channel 3) is filtered before the numpy matrix is built, so cosine scores are @@ -178,49 +247,119 @@ already enforces (DEV-1405). Datasource names cannot contain `.` (rejected by `DatasourceConfig.name` + `SlayerModel.data_source` validators), so the prefix match is unambiguous. +### `cypher_filter` graph pre-filter (DEV-1464) + +All four surfaces accept an optional `cypher_filter: Optional[str] = None` +argument. When set, an openCypher `MATCH … RETURN … AS id` query is run +against an ephemeral in-memory property graph (LadybugDB) built from +the current storage state. The returned canonical IDs become a **hard +allowlist** that pre-filters all three channels before any ranking: + +- Only memories whose `memory:` is in the returned set are ranked by + channels 1 / 2 / 3. +- Only entity docs whose `canonical_id` is in the returned set are ranked + by channels 2 and 3. +- If the query returns an empty set, a warning is emitted and an empty + response is returned immediately (no channels fire). + +**Naive fallback (no `advanced_search` required)**. When the +`advanced_search` extra is not installed, a simple subset of Cypher is +supported without LadybugDB. The naive parser accepts only: + +```cypher +MATCH (var:Label1:Label2:...) RETURN var.id AS id +``` + +Labels must be one or more of `Memory`, `Datasource`, `Model`, +`Column` (or its alias `ModelColumn`), `Measure`, `Aggregation` +(case-insensitive). The colon-separated +multi-label form is a union — it returns hits whose kind matches any of +the listed labels. Any other Cypher (WHERE clauses, relationships, +multiple MATCH clauses, etc.) raises `SlayerError` with a hint to +install the `advanced_search` extra. + +**Full `advanced_search` path**: When the `advanced_search` extra is +installed, the full openCypher query runs against the property graph +(see graph schema below). Complex filters, relationship traversals, and +property conditions are all supported. + +**Query safety**: the Cypher statement must be: +- A single statement (no semicolons). +- Read-only (no `CREATE` / `MERGE` / `DELETE` / `SET` / `DROP` / `CALL`). +- Returns exactly one column aliased `id` (e.g. `RETURN n.id AS id`). + +**Graph schema** (nodes and relationships in the ephemeral graph): + +| Node table | Properties | +|---|---| +| `Memory` | `id` (`memory:` form), `learning` | +| `Datasource` | `id`, `name` | +| `Model` | `id` (`.`), `name`, `description` | +| `ModelColumn` | `id` (`..`), `name`, `data_type`, `description` | +| `Measure` | `id` (`..`), `name`, `description` | +| `Aggregation` | `id` (`..`), `name` | + +Note: the column node table is named `ModelColumn` (not `Column`) because `Column` is a reserved keyword in LadybugDB ≥ 0.15. + +| Relationship | From → To | +|---|---| +| `MENTIONS` | Memory → {Datasource, Model, ModelColumn, Measure, Aggregation, Memory} | +| `CONTAINS` | Datasource → Model, Model → {ModelColumn, Measure, Aggregation} | +| `JOINS` | Model → Model | + +Hidden models and hidden columns are excluded from the graph. The graph is +rebuilt automatically when the storage fingerprint changes (file mtime for +YAML and SQLite). + +**Example** — surface only memories that mention the `orders` model: + +```cypher +MATCH (m:Memory)-[:MENTIONS]->(n:Model {id: 'shop.orders'}) +RETURN m.id AS id +``` + +**Example** — surface columns in the `shop` datasource: + +```cypher +MATCH (d:Datasource {id: 'shop'})-[:CONTAINS*1..3]->(c:ModelColumn) +RETURN c.id AS id +``` + +**Multi-label union**: `MATCH (n:Memory:ModelColumn)` returns nodes from both +`Memory` AND `ModelColumn` tables (LadybugDB union semantics). + ### Behaviour matrix | `entities`/`query` | `question` | Result | |---|---|---| -| set | set | All eligible channels run. Memories RRF-fused (channels 1 + 2 + 3); entities RRF-fused (channels 2 + 3). Channel 3 is skipped with a warning when the `embedding_search` extra is missing. Query-bearing memories partitioned out to `example_queries`. | -| set | unset/empty | Channel 1 only. Memories partitioned by `query` presence; no entity hits. | -| unset/empty | set | Channels 2 and 3 (when eligible). Memories RRF-fused; entities RRF-fused. | -| unset/empty | unset/empty | Recency fallback: newest `max_memories` learning-only memories + newest `max_example_queries` query-bearing memories, with a warning. | +| set | set | All eligible channels run. Memories and entities are RRF-fused across all active channels. Channel 3 is skipped with a warning when the `advanced_search` extra is missing. | +| set | unset/empty | Channel 1 only. Memory hits ranked by entity-tag overlap; entity hits = the named refs themselves (DEV-1513). | +| unset/empty | set | Channels 2 and 3 (when eligible). Memories and entities RRF-fused. | +| unset/empty | unset/empty | Recency fallback: newest memories (any kind) capped at `max_results`, with a warning. | ### Response shape -Memories are partitioned by `Memory.query is None`: learning-only -memories land in `memories`, query-bearing memories in -`example_queries`. The two lists are capped independently so a few -bulky example queries cannot crowd out small learning-only notes. +All hits — memories (both learning-only and query-bearing) and entities +(datasources, models, columns, measures, aggregations) — are returned +as a single flat ranked `results` list capped at `max_results`. +Query-bearing memories have `query` set; learning-only memories have +`query=None`; entity hits have `kind` set to their entity type. ```python -class MemoryHit(BaseModel): - id: str # memory id (forget_memory(id=hit.id) works) - score: float # RRF-fused (or single-channel raw) - text: str # full indexed text (no truncation) - matched_entities: List[str] # canonical entities that channel-1 input - # overlapped with the memory's tags; - # stale tags are filtered before this is - # computed (DEV-1428 lazy GC). - -class ExampleQueryHit(BaseModel): - id: str # memory id - score: float # RRF-fused - text: str # full indexed text - matched_entities: List[str] - query: SlayerQuery # always set on this hit type - -class EntityHit(BaseModel): - id: str # canonical entity string - kind: str # "datasource"|"model"|"column"|"measure"|"aggregation" - score: float # raw tantivy BM25 - text: str # full indexed text (no truncation) +class SearchHit(BaseModel): + id: str # memory id OR canonical entity string + kind: str # "memory"|"datasource"|"model"|"column"|"measure"|"aggregation" + score: float # RRF-fused score + text: str # full indexed text under compact=False; "" under compact=True + description: Optional[str] # compact preview (always populated for + # entity hits with descriptions; populated + # for memory hits in compact mode) + matched_entities: List[str] # channel-1 overlap (memory hits only; + # stale tags filtered, DEV-1428) + query: Optional[SlayerQuery] # set on query-bearing memory hits class SearchResponse(BaseModel): - memories: List[MemoryHit] # learning-only (query is None) - example_queries: List[ExampleQueryHit] # query-bearing - entities: List[EntityHit] + results: List[SearchHit] resolved_input_entities: List[str] # echo of the resolver output warnings: List[str] ``` @@ -238,7 +377,7 @@ search proceeds against whatever did resolve. Examples: - A stale entity tag inside a saved memory does not contribute to channel-1 BM25 ranking, and is excluded from any hit's `matched_entities` list. -- An `example_queries` hit whose attached `Memory.query` references a +- A query-bearing memory hit whose attached `Memory.query` references a vanished column gets the warning `example_query memory:: attached query has stale references (...); re-save to clean.` but is still surfaced with its stored query intact. @@ -254,34 +393,96 @@ sample-value fields: - `Column.sampled` — a formatted text snapshot. For categorical columns, the top-20 most-common values comma-joined; for high-cardinality - categoricals (> 50 distinct), the top-20 plus a `... (N distinct)` - suffix carrying the true total. For numeric / temporal columns, the - `min .. max` range. + categoricals (> 50 distinct), the top-20 plus a `... (50+ distinct)` + overflow marker (the exact total is not computed — one scan only). For + numeric / temporal columns, the `min .. max` range. - `Column.sampled_values` (DEV-1480) — the structured top-50-by-frequency list for categorical columns. Stays `None` for numeric / temporal columns. Consumers comparing predicate literals against actual stored values should read this field directly — text-split on `sampled` is ambiguous for values that themselves contain commas (e.g. `"R$ 1,000–3,000"`). -- `Column.distinct_count` (DEV-1480) — the true total cardinality at - profile time. Set for every profiled categorical column (computed via a - secondary `count_distinct` query when overflow is detected so the count - is exact, not capped). Stays `None` for numeric / temporal columns. +- `Column.distinct_count` (DEV-1480) — the exact distinct count when the + column has ≤ 50 distinct values. On overflow (> 50 distinct) it is + `None`: profiling uses a **single** full-table scan for the top-50 by + frequency and does not fire a second `count_distinct` scan for the exact + total. Also `None` for numeric / temporal columns. -All three are populated: +Profiling is **not** run at ingest time — a per-column full-table scan +would dominate ingest wall-clock on wide datasources. Samples are +populated **lazily, on the first `inspect` of a column** (synchronous, +per-column), and additionally: -- on every `slayer ingest` / `ingest_datasource_models` MCP call / - `POST /ingest` for every table-backed model in the touched datasource; - on `slayer search refresh-samples [--data-source X] [--model M ...]`; - on `edit_model` (column edits → that column; model-level filter / sql / source-query body change → every column); - lazily on `inspect_model` when the cached value is missing (write-back - best-effort). Cache validity for categorical columns requires - `sampled_values is not None` — v6 (legacy `sampled` only) models - re-profile on next call so the structured field gets populated. + best-effort); +- lazily inside `search()` itself for any column hit whose persisted + `sampled_values` is stale (DEV-1516). The post-fusion column-hit hook + groups hits by `(data_source, model_name)` — refreshes within a model + serialise (the storage write is a model-level read-modify-write); + refreshes across different models run concurrently via + `asyncio.gather`. When `search()` is constructed without an engine + (storage-only contexts), the hook is a silent no-op; +- lazily on the single-entity `inspect` point-lookup (DEV-1615) — an + `entity_type="column"` read at `compact=False` back-fills the column's + sample before rendering, so `inspect` matches the `inspect_model` / + `search` reads it replaced. `compact=True` stays description-only and + DB-free (no refresh); engine-guarded (no-op without an engine). + +Cache validity for categorical columns requires `sampled_values is not None` — +v6 (legacy `sampled` only) models re-profile on the next `inspect_model`, +`search()` column hit, or `inspect` (column, `compact=False`) so the +structured field gets populated. As of DEV-1615 the shared +`ensure_column_sample_fresh` helper also back-fills genuinely-unsampled +numeric / temporal columns (min/max range), not just categorical ones. sql-mode and query-backed models are silently skipped in v1. +### How sample values surface in search results + +The per-column doc rendered by `slayer/search/render.py:render_column_text` +prefers the structured `sampled_values` list (full top-50) over the +20-truncated `sampled` text. When `sampled_values` is populated: + +```text +Column: warehouse.orders.status +Type: TEXT +Description: Order status. +Sample values: ["paid", "refunded", "cancelled", "pending", …] ← JSON-encoded, top 50 +``` + +The list is rendered as a JSON array (not comma-joined) so values that +themselves contain commas — `"R$ 1,000–3,000"`, locale-formatted numbers, +multi-clause labels — survive unambiguously to the consumer. This is why +DEV-1480 introduced the structured `sampled_values` field in the first +place; comma-joining it back to a flat string would re-introduce the +exact ambiguity it was meant to solve. + +When `sampled_values` is `None` (numeric / temporal columns, or legacy +v6 data), the renderer falls back to the persisted `sampled` text. For +overflow categorical columns `sampled_values` is populated (top-50) and +the text carries a `... (50+ distinct)` marker; `distinct_count` stays +`None`, so no `Distinct count` line is emitted. An empty `sampled_values=[]` list is +authoritative-empty: the line is skipped entirely (no fallback to stale +`sampled`). + +This same text feeds both the per-column search index doc AND +`EntityHit.text` returned by `search()` — single renderer, single +source of truth. `inspect_model`'s markdown `## Columns` table is the +**all-columns-at-once** surface and continues to show the 20-truncated +`sampled` text per column for readability on wide models. JSON +`inspect_model` output already carries the full `sampled_values` list. + +**Known limitation.** The refresh hook runs **after** RRF fusion, on the +top-K hits being returned. Ranking (BM25 / tantivy / embeddings) still +operates on whatever the corpus held at index-build time. A query whose +only match against a column is a newly-revealed value in positions 21-50 +may still fail to surface that column. The text the agent sees IS +refreshed; tantivy / embeddings will catch up on the next +`slayer ingest` content-hash pass. + ## Index design notes - The tantivy index is built **fresh on every search call** in v1 (no @@ -322,7 +523,7 @@ sql-mode and query-backed models are silently skipped in v1. `delete_datasource("orders")` does not touch a sibling datasource named `orders_archive`; `delete_model("orders", "customers")` does not touch a sibling `customers_v2`. -- Optional pip extra: `pip install motley-slayer[embedding_search]` +- Optional pip extra: `pip install motley-slayer[advanced_search]` installs `litellm` + `numpy`. When omitted, the embedding channel emits a one-line warning and contributes nothing. - **Storage shape**: embeddings are stored as JSON lists of floats — diff --git a/docs/concepts/terminology.md b/docs/concepts/terminology.md index a78d83f0..8950b173 100644 --- a/docs/concepts/terminology.md +++ b/docs/concepts/terminology.md @@ -14,7 +14,7 @@ Key terms used throughout SLayer documentation and code. **Measure (named formula)** — A saved formula stored on a model (`SlayerModel.measures: List[ModelMeasure]`). Shape `{formula, name, label, description}` — same as a query's inline `measures` entry. Queries reference saved measures by bare name in any formula context (`{"formula": "aov"}`). -**Aggregation** — How a column is rolled up. Built-in aggregations: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, `first`, `last`, `weighted_avg`, `median`, `percentile`, `stddev_samp`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar_samp`, `covar_pop`. Custom aggregations can be defined at model level. Applied at query time via colon syntax: `revenue:sum`, `*:count`, `price:weighted_avg(weight=quantity)`, `price:corr(other=quantity)`. +**Aggregation** — How a column is rolled up. Built-in aggregations: `sum`, `avg`, `min`, `max`, `count`, `count_distinct`, `count_distinct_approx`, `first`, `last`, `weighted_avg`, `median`, `percentile`, `stddev_samp`, `stddev_pop`, `var_samp`, `var_pop`, `corr`, `covar_samp`, `covar_pop`. Custom aggregations can be defined at model level. Applied at query time via colon syntax: `revenue:sum`, `*:count`, `price:weighted_avg(weight=quantity)`, `price:corr(other=quantity)`. **Join** — A LEFT JOIN relationship between two models. Defined by a target model name and join key pairs (from the model's own foreign keys). Each model only stores direct joins — multi-hop paths like `customers.regions.name` are resolved at query time by walking each intermediate model's own joins. @@ -38,7 +38,7 @@ Key terms used throughout SLayer documentation and code. **Time dimension** — A dimension of type `time` or `date`, used for time-based grouping. When specified in `time_dimensions`, SLayer truncates it to the given granularity (e.g., monthly buckets). The same column can also be used as a regular dimension (without truncation). -**Granularity** — The level of time truncation applied to a time dimension, to determine the size of each time bucket (one row's time span) in the result, or as an argument in time related functions. Available granularities: `second`, `minute`, `hour`, `day`, `week`, `month`, `quarter`, `year`. +**Granularity** — The level of time truncation applied to a time dimension, to determine the size of each time bucket (one row's time span) in the result, or as an argument in time related functions. Available granularities: `second`, `minute`, `hour`, `day`, `week`, `week_sunday`, `month`, `quarter`, `year`. (`week` is Monday-anchored; `week_sunday` is Sunday-anchored.) **Time bucket** — A single unit of the granularity. If granularity is `month`, each time bucket is one calendar month (e.g., January 2024, February 2024). Each time bucket becomes one row in the query result. @@ -52,7 +52,7 @@ Key terms used throughout SLayer documentation and code. **Self-join vs window-function transforms:** -- `time_shift`, `change`, and `change_pct` use self-join CTEs — they can reach outside the current result set (no edge NULLs) and handle gaps in data correctly. `time_shift(revenue, -1, 'year')` (with granularity) joins on calendar date arithmetic for comparisons like year-over-year. +- `time_shift`, `change`, and `change_pct` use self-join CTEs — they can reach outside the current result set (no edge NULLs) and handle gaps in data correctly. The join matches on all non-time dimensions as well as the shifted time column, so grouped comparisons are partition-safe and reset per group. `time_shift(revenue, -1, 'year')` (with granularity) joins on calendar date arithmetic for comparisons like year-over-year. - `lag(revenue, 1)` / `lead(revenue, 1)` use SQL `LAG`/`LEAD` window functions directly — more efficient, but produce NULLs at the edges and are sensitive to gaps in data. **Nesting** — Formulas can be nested: `change(cumsum(revenue))` applies `change` to the result of `cumsum`. Each level of nesting generates an additional CTE layer in the SQL. diff --git a/docs/configuration/datasources.md b/docs/configuration/datasources.md index 049fc841..33f424da 100644 --- a/docs/configuration/datasources.md +++ b/docs/configuration/datasources.md @@ -24,6 +24,11 @@ type: postgres connection_string: postgresql://user:pass@host:5432/dbname ``` +When configuring individual connection fields, enter credentials exactly as issued. SLayer URL-encodes +reserved characters when building the connection string. When supplying `connection_string` directly, +percent-encode reserved characters within credential components yourself; do not encode the URL's +structural delimiters. + ## Environment Variables Use `${VAR_NAME}` references for credentials — resolved at read time from the process environment: @@ -55,6 +60,7 @@ These databases are verified by integration tests and runnable Docker examples. | `mysql` / `mariadb` | `motley-slayer[mysql]` | `mysql+pymysql://user:pass@localhost:3306/db` | | `clickhouse` | `motley-slayer[clickhouse]` | `clickhouse+http://user:pass@localhost:8123/db` | | `duckdb` | `motley-slayer[duckdb]` | `duckdb:///path/to/db.duckdb` | +| `snowflake` | `motley-slayer[snowflake]` | `snowflake://?connection_name=default` (TOML-driven) or `snowflake://user:pw@account/db/schema?warehouse=wh&role=role` (inline). See [Snowflake](#snowflake) below. | #### Additional support @@ -62,16 +68,82 @@ SQL generation is covered by unit tests, but not verified against live instances | Type | SQLAlchemy Driver | Install | |------|-------------------|---------| -| `snowflake` | `snowflake-sqlalchemy` | `pip install snowflake-sqlalchemy` | | `bigquery` | `sqlalchemy-bigquery` | `pip install sqlalchemy-bigquery` | | `redshift` | `sqlalchemy-redshift` + `redshift_connector` | `pip install sqlalchemy-redshift redshift-connector` | | `trino` / `presto` / `athena` | `trino` or `PyAthena` | `pip install trino` or `pip install PyAthena` | | `databricks` / `spark` | `databricks-sql-connector` | `pip install databricks-sql-connector` | | `oracle` | `oracledb` | `pip install oracledb` | -| `mssql` / `sqlserver` / `tsql` | `pyodbc` or `pymssql` | `pip install pyodbc` or `pip install pymssql` | +| `mssql` / `sqlserver` / `tsql` | `pyodbc` (auto-generated strings) or `pymssql` (manual `connection_string` only) | `pip install pyodbc` or `pip install pymssql` | + +!!! warning "SQL Server — requires SQL Server 2022+" + SLayer uses `DATETRUNC` for time-dimension queries, which was introduced in SQL Server 2022 (version 16.0). + SQL Server 2019 and earlier will return an error on time-dimension queries. + The Docker example uses `mcr.microsoft.com/mssql/server:2022-latest`. + +!!! warning "SQL Server — TrustServerCertificate" + Auto-generated SQL Server connection strings include `TrustServerCertificate=yes`, which disables + TLS certificate validation. This is correct for local development and Docker environments that use + self-signed certificates, but **must not be used in production** — it allows a man-in-the-middle + attack on the database connection. For production, supply a `connection_string` field directly with + a valid CA certificate chain, or configure your SQL Server instance with a certificate signed by a + trusted CA and omit `TrustServerCertificate`. !!! note - Snowflake, BigQuery, ClickHouse, and similar analytical warehouses typically don't have foreign keys, so auto-ingestion won't discover joins. Define joins manually in your model YAML. + BigQuery, ClickHouse, and similar analytical warehouses typically don't have foreign keys, so auto-ingestion won't discover joins. Define joins manually in your model YAML. Snowflake is an exception — it stores declarative (non-enforced) FK constraints AND exposes them via the Inspector, so auto-ingestion discovers joins like Postgres / MySQL / SQLite. + +### Snowflake + +The recommended path is the named-connection form, which delegates auth to +`snowflake.connector.connect(connection_name=...)` reading +`~/.snowflake/connections.toml`. This is the only path that supports key-pair, +OAuth, SSO, and MFA. + +```toml +# ~/.snowflake/connections.toml +[default] +account = "jp13593" # Snowflake account identifier, NOT a hostname +user = "YOUR_USER" +password = "YOUR_PASSWORD" +warehouse = "COMPUTE_WH" +database = "SLAYER_DEMO" +schema = "PUBLIC" +``` + +```yaml +# datasources/sf.yaml +name: sf +type: snowflake +connection_name: default +``` + +Or the inline form (host stores the Snowflake account identifier): + +```yaml +name: sf +type: snowflake +host: jp13593 +username: YOUR_USER +password: YOUR_PASSWORD +database: SLAYER_DEMO +schema_name: PUBLIC +warehouse: COMPUTE_WH +role: PUBLIC +``` + +Both forms flow through a shared engine factory that wires a per-connection +`USE WAREHOUSE / USE ROLE / USE DATABASE / USE SCHEMA` listener when those +typed fields are set. The `connection_name` profile's defaults are overridden +by anything you set on the DatasourceConfig. + +Statement-level timeout is enforced via +`ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = N` on the connection. + +!!! warning "Snowflake identifier casing" + Snowflake stores unquoted identifiers in uppercase but resolves them + case-insensitively. sqlglot's snowflake dialect emits bare lowercase + identifiers, which resolve correctly against uppercase storage. + Mixed-case names like `"Revenue"` get double-quoted by sqlglot and + become case-sensitive — they must match the stored case exactly. !!! tip If your database isn't listed but is supported by sqlglot, it may already work — SLayer falls back to Postgres-style SQL by default. Try it and [open an issue](https://github.com/MotleyAI/slayer/issues) if you hit a problem. @@ -88,6 +160,7 @@ SQL generation is covered by unit tests, but not verified against live instances | `username` | string | No | Database username | | `password` | string | No | Database password | | `connection_string` | string | No | Full connection string (alternative to individual fields) | +| `credentials_json` | string | No | Credentials JSON (used for BigQuery service accounts) | | `schema_name` | string | No | Default schema name | !!! note diff --git a/docs/configuration/storage.md b/docs/configuration/storage.md index 6b3d5d5c..b8e5eb58 100644 --- a/docs/configuration/storage.md +++ b/docs/configuration/storage.md @@ -56,15 +56,19 @@ slayer_data/ my_postgres.yaml other_db.yaml priority.yaml # datasource priority list (optional) - memories.yaml # agent memories (optional) + memories/ # agent memories, one .md file each (optional) + 1.md + help.intro.md embeddings.db # SQLite sidecar for embedding rows (DEV-1405) ``` **Layout note:** Models live under `models//.yaml` so two datasources sharing a table name don't collide. Opening a `YAMLStorage` on a legacy flat directory migrates `models/.yaml` files into the nested layout automatically. If a flat file has an empty `data_source` and exactly one datasource is registered, the migrator auto-fills it; otherwise it hard-fails so the user can edit `data_source` by hand before reopening. +**Name collisions:** Because names become filenames here, two ids differing only by letter case (`Orders` vs `orders`) would address the same file on macOS / Windows. Saving a datasource, model, or memory whose id differs only by case from an existing one therefore raises `IdCollisionError` — on every platform, so a YAML store created on Linux stays portable. Re-saving the exact same id is still a normal upsert. On the read side, lookups compare the exact filename, so `get_model("Orders")` when only `orders.yaml` exists returns "not found" instead of the wrong model (and a delete is a no-op). The layout migrations apply the same check up front and refuse to run — legacy files untouched — if migrating would collide. This is YAML-specific: SQLite keys are case-sensitive and store case-variant ids distinctly (which also means such a SQLite store cannot be exported to the YAML layout without renaming). + **Embeddings sidecar (DEV-1405):** Embedding rows used by the optional dense-search channel live in a SQLite file at `/embeddings.db`, **not** in `embeddings.yaml`. Embeddings are derived artifacts (regeneratable by `slayer ingest` / `--ingest-on-startup`), not user-authored config, so the diffable-in-git property that drives the YAML choice for models doesn't apply. A pre-DEV-1405 `embeddings.yaml` or `counters.yaml` is silently renamed to `.yaml.legacy` on first open and ignored thereafter; re-run `slayer ingest` to repopulate `embeddings.db`. The schema is identical to the `SQLiteStorage` embedding table — both backends delegate to a shared `SidecarEmbeddingStore` helper. -**Memory id allocation (DEV-1405):** The next memory id is derived from `memories.yaml` itself — `last_row.id + 1` — rather than a separate counter file. Ids of deleted memories may be reused by future saves; cascade-on-delete in `delete_memory` already removes the matching embedding row. +**Memory id allocation (DEV-1658):** The next int-shaped memory id is derived by scanning the `memories/` directory of per-id `.md` files — `max(int-shaped id) + 1` — rather than a separate counter file. Non-int ids (e.g. `help.intro`, `kb.policy.42`) are ignored by the allocator. Ids of deleted memories may be reused by future saves; cascade-on-delete in `delete_memory` already removes the matching embedding row. ### SQLiteStorage @@ -107,21 +111,25 @@ from slayer.storage.base import StorageBackend from slayer.core.models import SlayerModel, DatasourceConfig class MyCustomStorage(StorageBackend): - # Models are keyed by (data_source, name). - def save_model(self, model: SlayerModel) -> None: ... - def _list_all_model_identities(self) -> list[tuple[str, str]]: ... - def get_model(self, name: str, data_source: str | None = None) -> SlayerModel | None: ... - def delete_model(self, name: str, data_source: str | None = None) -> bool: ... + # Models are keyed by (data_source, name). ``save_model`` is a template + # method on the base class (it runs shared validation); backends + # implement only ``_save_model_impl``. + async def _save_model_impl(self, model: SlayerModel) -> None: ... + async def _list_all_model_identities(self) -> list[tuple[str, str]]: ... + async def get_model(self, name: str, data_source: str | None = None) -> SlayerModel | None: ... + async def _delete_model_row(self, *, data_source: str, name: str) -> bool: ... # ``StorageBackend`` provides default implementations of ``list_models`` # and ``resolve_model_identity`` on top of ``_list_all_model_identities``. - def save_datasource(self, datasource: DatasourceConfig) -> None: ... - def get_datasource(self, name: str) -> DatasourceConfig | None: ... - def list_datasources(self) -> list[str]: ... - def delete_datasource(self, name: str) -> bool: ... + async def save_datasource(self, datasource: DatasourceConfig) -> None: ... + async def get_datasource(self, name: str) -> DatasourceConfig | None: ... + async def list_datasources(self) -> list[str]: ... + async def _delete_datasource_row(self, name: str) -> bool: ... ``` +If your backend stores ids as filenames (or another case-insensitive keyspace), set the class attribute `_ids_collide_as_filenames = True` — the base save templates then reject ids differing only by case — and call `await self.check_datasource_id_collision(datasource.name)` at the top of your `save_datasource`. + Register it for URI-based resolution: ```python diff --git a/docs/database-support.md b/docs/database-support.md index 5aa66cfa..e70256f6 100644 --- a/docs/database-support.md +++ b/docs/database-support.md @@ -5,27 +5,43 @@ SQL generation. Databases are supported at two tiers. ## Tier 1 — fully tested -Integration tests and/or Docker examples; must not regress. - -| Engine | Coverage | -|---|---| -| **SQLite** | Integration tests in `tests/integration/test_integration.py`; embedded example. | -| **Postgres** | Integration tests in `tests/integration/test_integration_postgres.py`; Docker example. | -| **DuckDB** | Integration tests in `tests/integration/test_integration_duckdb.py` (in-process, no Docker). | -| **MySQL** | Docker example with `verify.py`. | -| **ClickHouse** | Docker example with `verify.py`. | +Live-instance integration tests must not regress. Where Docker images exist, +the suites spin up the engine via `testcontainers`; the cloud-only engines +(BigQuery, Snowflake) skip cleanly when credentials aren't available and run +against the live service in CI when they are. + +| Engine | Live test | Docker example | +|---|---|---| +| **SQLite** | `tests/integration/test_integration.py` (in-process) | `examples/embedded/` | +| **Postgres** | `tests/integration/test_integration_postgres.py` (pytest-postgresql, spawned temp instance) | `examples/postgres/` | +| **DuckDB** | `tests/integration/test_integration_duckdb.py` (in-process) | `examples/embedded/` (DuckDB mode) | +| **MySQL** | `tests/integration/test_integration_mysql.py` (`testcontainers[mysql]`) | `examples/mysql/` | +| **ClickHouse** | `tests/integration/test_integration_clickhouse.py` (`testcontainers[clickhouse]`) | `examples/clickhouse/` | +| **SQL Server** | `tests/integration/test_integration_sqlserver.py` (`testcontainers`, `msodbcsql18` + `unixodbc-dev` on the runner) | `examples/sqlserver/` | +| **Snowflake** | `tests/integration/test_integration_snowflake.py` (skips without `~/.snowflake/connections.toml`; profile name overridable via `$SLAYER_SNOWFLAKE_CONNECTION`) | `examples/snowflake/` (no Docker) | +| **BigQuery** | `examples/bigquery/verify.py` driven by CI against `bigquery-public-data.thelook_ecommerce` (gated on `GCP_PROJECT_ID` / `GCP_SA_KEY_B64` repo secrets) | `examples/bigquery/` (no Docker — managed service) | + +BigQuery does not yet have a pytest-style integration suite; its CI coverage +runs the example's `verify.py` directly via `.github/workflows/ci.yml`. That +exercises auto-ingestion, basic projection, joins, time-grain dimensions, and +the cardinality / sum-of-grouped-equals-total invariants — enough to catch +emitted-SQL regressions, but the verify-script tier is shallower than the +testcontainers suites. ## Tier 2 — code-covered Unit tests for SQL generation; no live-instance verification. -Snowflake, BigQuery, Redshift, Trino/Presto, Databricks/Spark, -MS SQL Server, Oracle. +Redshift, Trino/Presto (Athena uses the Presto dialect), Databricks/Spark, +Oracle. ## Aggregation support Most aggregations (`sum`, `avg`, `min`, `max`, `count`, `count_distinct`, -`first`, `last`, `weighted_avg`) work on every supported database. +`count_distinct_approx`, `first`, `last`, `weighted_avg`) work on every +supported database. `count_distinct_approx` is dialect-aware (see +[below](#count_distinct_approx-by-dialect)) but always available — it falls +back to an exact `COUNT(DISTINCT)` where there's no native function. `median`, `percentile`, the variance/stddev family (`stddev_samp`, `stddev_pop`, `var_samp`, `var_pop`), and the paired statistics (`corr`, `covar_samp`, `covar_pop`) need dialect-specific handling @@ -37,14 +53,34 @@ because no standard syntax works everywhere: | DuckDB | yes | yes | yes | yes | sqlglot rewrites ordered-set percentiles to `QUANTILE_CONT`. Native `STDDEV_*`/`VAR_*`/`CORR`/`COVAR_*` (sqlglot may emit `VARIANCE` for `var_samp`). | | SQLite | yes | yes | yes | yes | Python aggregate UDFs registered on every connection — see "SQLite caveats" below. | | ClickHouse | yes | yes | yes | yes | Native `median(x)`, parametric `quantile(p)(x)`, native `stddev_*`/`var_*`/`corr`/`covar*` (camelCase variants emitted by sqlglot for `var_samp`). | +| Snowflake | yes | yes | yes | yes | Native `MEDIAN`, `PERCENTILE_CONT(p) WITHIN GROUP`, `STDDEV_*`/`VAR_*`/`CORR`/`COVAR_*`. `LOG10` native; no native `LOG2` (falls through to `LOG(2, x)`). | | MySQL | **no** | **no** | yes | **no** | No native `MEDIAN`/`PERCENTILE_CONT`/`CORR`/`COVAR_*` and no Python-UDF mechanism — SLayer raises `NotImplementedError` for those. `STDDEV_SAMP`/`STDDEV_POP`/`VAR_SAMP`/`VAR_POP` are native on MySQL. Use MariaDB or compute the unsupported aggregations client-side. | +| SQL Server (T-SQL) | **no** | **no** | yes | yes (decomposed) | `MEDIAN` doesn't exist and T-SQL's `PERCENTILE_CONT` is window-only (no `WITHIN GROUP` aggregate form) — SLayer raises `NotImplementedError`. Native `STDEV`/`STDEVP`/`VAR`/`VARP` (slayer renames the canonical `STDDEV_*`/`VAR_*` names at emit time). `CORR`/`COVAR_*` use the same variance-decomposition formula as MySQL (`cov(x,y) = (var(x+y) − var(x) − var(y)) / 2`, `corr = cov / (stddev(x) · stddev(y))`). | +| BigQuery | **no** | **no** | yes | yes | BigQuery has no `MEDIAN` aggregate, and its `PERCENTILE_CONT` is analytic-only (no `WITHIN GROUP` syntax) — the base class emit `PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY x)` fails at runtime. If you need percentile on BigQuery, define a custom `Aggregation` using `APPROX_QUANTILES(x, 100)[OFFSET(N)]`. Native `STDDEV_SAMP`/`STDDEV_POP`/`VAR_SAMP`/`VAR_POP`/`CORR`/`COVAR_SAMP`/`COVAR_POP` (sqlglot may emit `VARIANCE` for `var_samp`). | + +### `count_distinct_approx` by dialect + +`count_distinct_approx` emits each database's native approximate-distinct +function where one exists, and falls back to an **exact** `COUNT(DISTINCT)` +where it does not. The fallback is exact (more accurate, never approximate), +so results are always at least as precise as requested. The per-dialect +mapping lives in `SqlDialect.build_approx_count_distinct`. + +| Engine | Emitted SQL | +|---|---| +| DuckDB / Spark / Databricks | `approx_count_distinct(x)` | +| ClickHouse | `uniq(x)` | +| BigQuery / Snowflake / SQL Server (T-SQL) / Oracle | `APPROX_COUNT_DISTINCT(x)` | +| Trino / Presto | `approx_distinct(x)` | +| Redshift | `APPROXIMATE COUNT(DISTINCT x)` | +| Postgres / SQLite / MySQL | `COUNT(DISTINCT x)` (exact fallback) | ### SQLite caveats SQLite has a much smaller built-in math/stat catalog than the other supported engines. SLayer registers Python aggregate and scalar UDFs on every new SQLite connection via SQLAlchemy's `connect` event (see -`slayer/sql/sqlite_udfs.py`). +`slayer/sql/dialects/sqlite.py`). **Aggregate UDFs:** @@ -78,7 +114,12 @@ than SQLite ≥3.35's silent-NULL built-in `log()`. These are registered automatically as long as connections go through `SlayerSQLClient` (which uses the cached SQLAlchemy engine). If you open a SQLite connection directly outside SLayer, the UDFs will not be available — -call `register_sqlite_udfs(connection)` manually if you need them. +import and call the registration helper manually if you need them: + +```python +from slayer.sql.dialects.sqlite import register_sqlite_udfs +register_sqlite_udfs(connection) +``` ### MySQL caveats @@ -98,6 +139,96 @@ If you need percentiles on MySQL, the recommended options are: - Define a custom `Aggregation` on the model with whatever `GROUP_CONCAT`- based or windowed expression suits your data shape and group sizes. +### SQL Server (T-SQL) caveats + +T-SQL has `STDEV`/`STDEVP`/`VAR`/`VARP` (not `STDDEV_SAMP`/`STDDEV_POP`/ +`VAR_SAMP`/`VAR_POP`); sqlglot's tsql transpiler emits incorrect names like +`VAR_SAMP` and `VARIANCE_POP`, so the T-SQL dialect overrides the canonical +spellings via `Anonymous` rewrites in `slayer/sql/dialects/tsql.py`. + +`CORR`/`COVAR_SAMP`/`COVAR_POP` are derived from variance: +`cov(x, y) = (var(x + y) − var(x) − var(y)) / 2`, +`corr = cov / (stddev(x) · stddev(y))`. The decomposition is shared with +MySQL via `_build_covar_decomposition` in `slayer/sql/dialects/base.py`. + +`MEDIAN` doesn't exist, and `PERCENTILE_CONT` in T-SQL is a window function +only — there is no `WITHIN GROUP` aggregate form. SLayer raises +`NotImplementedError` for both at SQL generation time. Use the windowed form +as a custom `Aggregation` if you need it, or compute client-side. + +Other T-SQL specifics surfaced by the dialect: + +- `DATETRUNC(unit, col)` for time-grain dimensions (SQL Server 2022+ — + earlier versions don't have `DATETRUNC` and aren't supported). +- `DATETRUNC(iso_week, col)` for Monday-aligned week truncation — + `@@DATEFIRST`-independent so the bucketing is deterministic. +- `DATEADD(unit, n, col)` for time-shift arithmetic — T-SQL has no + `INTERVAL` literal. +- Bracketed `[ident]` quoting — `.` SLayer aliases get + mangled to `___` at emit and decoded back on result-row + keys (mirror of the BigQuery `___` mangling; see DEV-1571). +- Native `LOG10`, no native `LOG2` (`log2(x)` falls through to the + canonical 2-arg `LOG(2, x)` form). + +### Snowflake caveats + +Snowflake is a fully managed cloud warehouse — no Docker, no local instance. +The integration suite skips by default unless `~/.snowflake/connections.toml` +contains a profile named `slayer_test` (override with +`$SLAYER_SNOWFLAKE_CONNECTION`). See [Datasources → +Snowflake](configuration/datasources.md#snowflake) for connection setup. + +- **`LIMIT 0` type probes still compile.** SLayer infers column types via + `LIMIT 0` wrapper queries. Snowflake compiles those — consuming a small + amount of warehouse compute — even though no rows are returned. A future + `DESCRIBE QUERY`-based probe would skip this; not yet implemented. +- **Identifier casing.** Snowflake stores unquoted identifiers in uppercase + but resolves them case-insensitively. sqlglot's snowflake dialect emits + bare lowercase identifiers, which therefore resolve correctly against + uppercase storage. **Mixed-case** names like `"Revenue"` get double-quoted + by sqlglot and become case-sensitive — they must match the stored case + exactly. +- **Declarative FK constraints are surfaced.** Unlike ClickHouse / BigQuery, + Snowflake exposes its (non-enforced) FK metadata via the Inspector. Auto- + ingestion discovers joins like Postgres / MySQL / SQLite. +- **No native LOG2.** `log2(x)` in a `Column.sql` falls through to the + canonical 2-arg `LOG(2, x)` form. `LOG10` and the rest of the math / + statistical functions are native. + +### BigQuery caveats + +BigQuery is a fully managed cloud warehouse — no Docker, no local instance. +CI runs the example's `verify.py` against `bigquery-public-data.thelook_ecommerce`, +gated on `GCP_PROJECT_ID` and `GCP_SA_KEY_B64` repo secrets (forks without +them skip cleanly). Auth via Google Application Default Credentials +(`$GOOGLE_APPLICATION_CREDENTIALS` pointing at a service-account JSON key, +plus `$GCP_PROJECT_ID` for billing). The `bigquery://` driver requires the +`sqlalchemy-bigquery` extra. + +- **No FK introspection.** BigQuery exposes no foreign-key metadata via + `INFORMATION_SCHEMA`, so auto-ingestion cannot discover joins. Hand-declare + `ModelJoin`s on the model. +- **Dotted alias mangling.** BigQuery rejects column names containing `.` + (output schema names must match `[A-Za-z_][A-Za-z0-9_]*`), so SLayer + rewrites `.` aliases (`orders._count`, + `orders.products.category`) to `___` at emit time and + reverses the mapping on result rows. The triple-underscore separator is + distinct from `__` (used by `_query_as_model` for cross-model leaf + flattening), so the two encodings never collide. In `Column.sql`, + fully-qualified table paths must be backticked per-segment + (`` `project`.`dataset`.`table` ``) — a single backticked dotted path of + word-only segments (`` `my_dataset.my_table` ``) would false-positive + mangle. +- **No `MEDIAN` aggregate; `PERCENTILE_CONT` is analytic-only.** Both + raise at SQL generation time (sqlglot doesn't transpile the base class's + `PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY x)` to BigQuery's analytic + form). Use a custom `Aggregation` with `APPROX_QUANTILES(x, 100)[OFFSET(N)]` + when you need it. +- **No native EXPLAIN.** BigQuery has no SQL-level `EXPLAIN`. The + `BigqueryDialect.explain_prefix` is `None`, so `engine.execute(..., + explain=True)` returns the dry-run SQL unchanged rather than an execution + plan. + ## Adding a new dialect 1. Add the mapping to `slayer/engine/query_engine.py:_dialect_for_type()`. diff --git a/docs/dbt/dbt_import.md b/docs/dbt/dbt_import.md index 2fb4c0d2..cba512e3 100644 --- a/docs/dbt/dbt_import.md +++ b/docs/dbt/dbt_import.md @@ -136,17 +136,53 @@ Nothing to add — the underlying measure is already directly queryable. All three fold into a `ModelMeasure` on the source semantic model. Inputs are referenced by **bare ModelMeasure name**, so the formula parser resolves them locally: -- **Derived**: `formula: "metric_a + metric_b"` -- **Ratio**: `formula: "numerator / denominator"` -- **Cumulative (unbounded)**: `formula: "cumsum(measure_name)"` - -#### Unconverted metrics - -Some dbt metrics cannot be expressed as a `ModelMeasure`. They are reported in `ConversionResult.unconverted_metrics` and printed with an `UNCONVERTED` tag. Categories: - -- **Cumulative with window or grain_to_date**: SLayer's `cumsum` is unbounded. -- **Conversion metrics**: entity-based sequential event tracking is not supported. -- **Transform-name shadowing**: a dbt measure or metric named after a SLayer transform (`cumsum`, `lag`, `lead`, `change`, `change_pct`, `time_shift`, `rank`, `percent_rank`, `dense_rank`, `ntile`, `first`, `last`) is rejected — using it bare in a formula would shadow the transform. +- **Derived**: `formula: "metric_a + metric_b"`. An `offset_window` on a single-aggregate input (a measure or simple metric) lowers to a `time_shift`: a `1 month` offset becomes `time_shift(metric_a, -1, 'month')`. +- **Ratio**: `formula: "numerator / nullif(denominator, 0)"` — the denominator is NULL-guarded to prevent divide-by-zero. +- **Cumulative (unbounded)**: `formula: "cumsum(measure_name)"`. + +#### Supported mappings + +Every legal dbt construct that reaches the importer is either represented exactly or [failed cleanly](#clean-fail-and-unsupported). Represented exactly: + +| dbt construct | SLayer representation | +| --- | --- | +| Measure `agg: sum/avg/min/max/count/count_distinct/median` | `ModelMeasure` `col:` | +| Measure `agg: percentile` (continuous) | `col:percentile(p=)` | +| Measure `agg: count_distinct_approx` | `col:count_distinct_approx` (dialect-aware) | +| Measure `agg: sum_boolean` | `Column.sql = "CASE WHEN () THEN 1 ELSE 0 END"`, type `INT`, `col:sum` | +| Metric-level / per-input `filter` | pushed down into a leaf `Column.filter` (CASE-inside-aggregate) | +| Filter as string **or** list (`WhereFilterIntersection`) | AND-joined into one filter | +| Ratio metric | `num / nullif(den, 0)` | +| Derived metric | `ModelMeasure` formula over inputs | +| Derived input `offset_window` (single aggregate) | `time_shift(input, -, '')` (plural grains normalized) | +| Unbounded cumulative | `cumsum(measure)` | +| `config.meta`, semantic-model `label` | carried onto the corresponding entity's `meta` | + +#### Clean-fail and unsupported + +Constructs that cannot be expressed exactly are **failed cleanly** — never converted to approximate or wrong SQL. Each is routed to the conversion report with a category, severity, and documented workaround, and the raw construct is stashed into the owning entity's `meta` so nothing is silently lost. + +| dbt construct | Why | Workaround | +| --- | --- | --- | +| Cumulative `window` (rolling) | Query-grain-dependent re-aggregation | Use `cumsum(measure)` for an unbounded total | +| Cumulative `grain_to_date` | Reset-at-grain can't bake into a saved measure | `cumsum(measure)` + put the grain dimension in the query | +| Cumulative `period_agg` ≠ `first` | Only the default running total is exact | Use the default `period_agg` | +| Derived input `offset_to_grain` | No truncate-to-grain shift transform | Use `cumsum(...)` + grain dimension | +| `offset_window` on a ratio/derived input | Multi-aggregate offset isn't exactly expressible | Restructure as a multi-stage `source_queries` model | +| Non-standard granularity (e.g. `fortnight`) | Not a SLayer granularity | Use day/week/month/quarter/year | +| `non_additive_dimension` (semi-additive) | Not exactly expressible | `balance:last(\n", " \n", " \n", + " \n", " \n", " \n", - " \n", " \n", " \n", " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", - " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", " \n", - " \n", - " \n", - " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", " \n", " \n", "
orders.stores.nameorders.ordered_atorders.revenueorders.avg_orderorders.median_order
0Brooklyn1037680.1212.6642117.282025-10-01127902.0011.209641
1Philadelphia802124.2412.7680037.42Brooklyn2025-12-01125207.3311.404256
2Chicago438820.6813.1932507.43Brooklyn2025-11-01124262.0111.571097
3San Francisco352668.9213.2120387.52Brooklyn2026-01-01122842.4911.069883
4Brooklyn2025-09-01122738.5311.402688
...............
104Brooklyn2023-12-0115949.4010.961787
105San Francisco2025-02-0114331.8511.347466
106New Orleans2026-06-0111825.8111.241264
107New Orleans55789.7412.7606917.282025-12-0111686.4510.341991
108Philadelphia2023-06-0111037.3210.461915
\n", + "

109 rows × 4 columns

\n", "" ], "text/plain": [ - " orders.stores.name orders.revenue orders.avg_order orders.median_order\n", - "0 Brooklyn 1037680.12 12.664211 7.28\n", - "1 Philadelphia 802124.24 12.768003 7.42\n", - "2 Chicago 438820.68 13.193250 7.43\n", - "3 San Francisco 352668.92 13.212038 7.52\n", - "4 New Orleans 55789.74 12.760691 7.28" + " orders.stores.name orders.ordered_at orders.revenue orders.avg_order\n", + "0 Brooklyn 2025-10-01 127902.00 11.209641\n", + "1 Brooklyn 2025-12-01 125207.33 11.404256\n", + "2 Brooklyn 2025-11-01 124262.01 11.571097\n", + "3 Brooklyn 2026-01-01 122842.49 11.069883\n", + "4 Brooklyn 2025-09-01 122738.53 11.402688\n", + ".. ... ... ... ...\n", + "104 Brooklyn 2023-12-01 15949.40 10.961787\n", + "105 San Francisco 2025-02-01 14331.85 11.347466\n", + "106 New Orleans 2026-06-01 11825.81 11.241264\n", + "107 New Orleans 2025-12-01 11686.45 10.341991\n", + "108 Philadelphia 2023-06-01 11037.32 10.461915\n", + "\n", + "[109 rows x 4 columns]" ] }, "execution_count": 3, @@ -246,9 +379,9 @@ " 'measures': [\n", " {'formula': 'order_total:sum', 'name': 'revenue'},\n", " {'formula': 'order_total:avg', 'name': 'avg_order'},\n", - " {'formula': 'order_total:median', 'name': 'median_order'},\n", " ],\n", " 'dimensions': ['stores.name'],\n", + " 'time_dimensions': [{'dimension': 'ordered_at', 'granularity': 'month'}],\n", " 'order': [{'column': 'revenue', 'direction': 'desc'}],\n", "})\n", "\n", @@ -272,7 +405,15 @@ "cell_type": "code", "execution_count": 4, "id": "bd771578", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-09T20:07:00.667405Z", + "iopub.status.busy": "2026-06-09T20:07:00.666886Z", + "iopub.status.idle": "2026-06-09T20:07:00.924255Z", + "shell.execute_reply": "2026-06-09T20:07:00.922456Z" + }, + "scrolled": true + }, "outputs": [ { "data": { @@ -305,83 +446,83 @@ " \n", " \n", " 0\n", - " Philadelphia\n", - " 2023-06-01\n", - " 6498.32\n", - " 15.660223\n", - " NaN\n", + " Brooklyn\n", + " 2024-01-01\n", + " 40397.72\n", + " 1.532868\n", + " None\n", " \n", " \n", " 1\n", - " Chicago\n", - " 2025-02-01\n", - " 12728.15\n", - " 2.163515\n", - " NaN\n", + " Philadelphia\n", + " 2023-07-01\n", + " 24685.00\n", + " 1.236503\n", + " None\n", " \n", " \n", " 2\n", " San Francisco\n", - " 2025-06-01\n", - " 15231.01\n", - " 0.886053\n", - " NaN\n", + " 2025-03-01\n", + " 31591.94\n", + " 1.204317\n", + " None\n", " \n", " \n", " 3\n", - " Chicago\n", - " 2025-06-01\n", - " 19681.54\n", - " 0.816735\n", - " NaN\n", + " New Orleans\n", + " 2026-01-01\n", + " 25075.42\n", + " 1.145683\n", + " None\n", " \n", " \n", " 4\n", - " Brooklyn\n", - " 2024-06-01\n", - " 24083.25\n", - " 0.815521\n", - " NaN\n", + " San Francisco\n", + " 2025-07-01\n", + " 64800.28\n", + " 0.605787\n", + " None\n", " \n", " \n", " 5\n", - " San Francisco\n", - " 2025-07-01\n", - " 27545.57\n", - " 0.808519\n", - " NaN\n", + " Brooklyn\n", + " 2024-07-01\n", + " 95375.10\n", + " 0.486821\n", + " None\n", " \n", " \n", " 6\n", - " Philadelphia\n", - " 2024-06-01\n", - " 19436.72\n", - " 0.789620\n", - " 1.991038\n", + " Chicago\n", + " 2025-07-01\n", + " 73926.68\n", + " 0.472685\n", + " None\n", " \n", " \n", " 7\n", " Philadelphia\n", - " 2023-07-01\n", - " 11601.89\n", - " 0.785368\n", - " NaN\n", + " 2023-09-01\n", + " 40046.75\n", + " 0.402552\n", + " None\n", " \n", " \n", " 8\n", - " Philadelphia\n", - " 2025-07-01\n", - " 32888.09\n", - " 0.784384\n", - " 0.140402\n", + " Brooklyn\n", + " 2024-03-01\n", + " 54936.87\n", + " 0.367116\n", + " None\n", " \n", " \n", " 9\n", - " Brooklyn\n", - " 2025-06-01\n", - " 32618.51\n", - " 0.668387\n", - " 0.354406\n", + " New Orleans\n", + " 2026-03-01\n", + " 32700.66\n", + " 0.357876\n", + " None\n", " \n", " \n", "\n", @@ -389,28 +530,28 @@ ], "text/plain": [ " orders.stores.name orders.ordered_at orders.revenue orders.mom_growth \\\n", - "0 Philadelphia 2023-06-01 6498.32 15.660223 \n", - "1 Chicago 2025-02-01 12728.15 2.163515 \n", - "2 San Francisco 2025-06-01 15231.01 0.886053 \n", - "3 Chicago 2025-06-01 19681.54 0.816735 \n", - "4 Brooklyn 2024-06-01 24083.25 0.815521 \n", - "5 San Francisco 2025-07-01 27545.57 0.808519 \n", - "6 Philadelphia 2024-06-01 19436.72 0.789620 \n", - "7 Philadelphia 2023-07-01 11601.89 0.785368 \n", - "8 Philadelphia 2025-07-01 32888.09 0.784384 \n", - "9 Brooklyn 2025-06-01 32618.51 0.668387 \n", + "0 Brooklyn 2024-01-01 40397.72 1.532868 \n", + "1 Philadelphia 2023-07-01 24685.00 1.236503 \n", + "2 San Francisco 2025-03-01 31591.94 1.204317 \n", + "3 New Orleans 2026-01-01 25075.42 1.145683 \n", + "4 San Francisco 2025-07-01 64800.28 0.605787 \n", + "5 Brooklyn 2024-07-01 95375.10 0.486821 \n", + "6 Chicago 2025-07-01 73926.68 0.472685 \n", + "7 Philadelphia 2023-09-01 40046.75 0.402552 \n", + "8 Brooklyn 2024-03-01 54936.87 0.367116 \n", + "9 New Orleans 2026-03-01 32700.66 0.357876 \n", "\n", - " orders.yoy_growth \n", - "0 NaN \n", - "1 NaN \n", - "2 NaN \n", - "3 NaN \n", - "4 NaN \n", - "5 NaN \n", - "6 1.991038 \n", - "7 NaN \n", - "8 0.140402 \n", - "9 0.354406 " + " orders.yoy_growth \n", + "0 None \n", + "1 None \n", + "2 None \n", + "3 None \n", + "4 None \n", + "5 None \n", + "6 None \n", + "7 None \n", + "8 None \n", + "9 None " ] }, "metadata": {}, @@ -433,6 +574,7 @@ " 'filters': ['change_pct(order_total:sum) > 0'],\n", " 'order': [{'column': 'mom_growth', 'direction': 'desc'}],\n", " 'limit': 10,\n", + " 'whole_periods_only': True,\n", "}\n", "result = engine.execute_sync(query=hero)\n", "assert len(result.data) > 0, 'hero query returned no rows'\n", @@ -444,7 +586,14 @@ "cell_type": "code", "execution_count": 5, "id": "d0b5263d", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-09T20:07:00.927729Z", + "iopub.status.busy": "2026-06-09T20:07:00.927317Z", + "iopub.status.idle": "2026-06-09T20:07:00.933217Z", + "shell.execute_reply": "2026-06-09T20:07:00.931491Z" + } + }, "outputs": [ { "name": "stdout", @@ -467,6 +616,8 @@ "FROM orders AS orders\n", "LEFT JOIN stores AS stores\n", " ON orders.store_id = stores.id\n", + "WHERE\n", + " orders.ordered_at <= '2026-05-31'\n", "GROUP BY\n", " stores.name,\n", " DATE_TRUNC('MONTH', orders.ordered_at)\n", @@ -479,6 +630,10 @@ "FROM orders AS orders\n", "LEFT JOIN stores AS stores\n", " ON orders.store_id = stores.id\n", + "WHERE\n", + " (\n", + " orders.ordered_at + INTERVAL '1' MONTH\n", + " ) <= '2026-05-31'\n", "GROUP BY\n", " stores.name,\n", " DATE_TRUNC('MONTH', CAST(orders.ordered_at + INTERVAL '1' MONTH AS TIMESTAMP))\n", @@ -497,6 +652,10 @@ "FROM orders AS orders\n", "LEFT JOIN stores AS stores\n", " ON orders.store_id = stores.id\n", + "WHERE\n", + " (\n", + " orders.ordered_at + INTERVAL '1' YEAR\n", + " ) <= '2026-05-31'\n", "GROUP BY\n", " stores.name,\n", " DATE_TRUNC('MONTH', CAST(orders.ordered_at + INTERVAL '1' YEAR AS TIMESTAMP))\n", @@ -515,6 +674,10 @@ "FROM orders AS orders\n", "LEFT JOIN stores AS stores\n", " ON orders.store_id = stores.id\n", + "WHERE\n", + " (\n", + " orders.ordered_at + INTERVAL '1' MONTH\n", + " ) <= '2026-05-31'\n", "GROUP BY\n", " stores.name,\n", " DATE_TRUNC('MONTH', CAST(orders.ordered_at + INTERVAL '1' MONTH AS TIMESTAMP))\n", @@ -580,18 +743,25 @@ "cell_type": "code", "execution_count": 6, "id": "07e3be87", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-09T20:07:00.936888Z", + "iopub.status.busy": "2026-06-09T20:07:00.936475Z", + "iopub.status.idle": "2026-06-09T20:07:01.079933Z", + "shell.execute_reply": "2026-06-09T20:07:01.078067Z" + } + }, "outputs": [ { "data": { "text/markdown": [ "| monthly_store_revenue.stores__name | monthly_store_revenue.order_total_sum_avg |\n", "| --- | --- |\n", - "| Brooklyn | 34.6K |\n", - "| Chicago | 25.8K |\n", - "| San Francisco | 22.0K |\n", - "| Philadelphia | 21.7K |\n", - "| New Orleans | 9298 |" + "| Brooklyn | 92.6K |\n", + "| Chicago | 68.6K |\n", + "| San Francisco | 61.1K |\n", + "| Philadelphia | 60.2K |\n", + "| New Orleans | 24.5K |" ], "text/plain": [ "" @@ -645,8 +815,22 @@ "cell_type": "code", "execution_count": 7, "id": "921e2cf1", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-09T20:07:01.083741Z", + "iopub.status.busy": "2026-06-09T20:07:01.083277Z", + "iopub.status.idle": "2026-06-09T20:07:40.896412Z", + "shell.execute_reply": "2026-06-09T20:07:40.894987Z" + } + }, "outputs": [ + { + "name": "stderr", + "output_type": "stream", + "text": [ + "\u001b[92m22:07:01 - LiteLLM:WARNING\u001b[0m: get_model_cost_map.py:271 - LiteLLM: Failed to fetch remote model cost map from https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json: Using SOCKS proxy, but the 'socksio' package is not installed. Make sure to install httpx using `pip install httpx[socks]`.. Falling back to local backup.\n" + ] + }, { "name": "stdout", "output_type": "stream", @@ -682,7 +866,14 @@ "cell_type": "code", "execution_count": 8, "id": "ee6873a6", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-09T20:07:40.898907Z", + "iopub.status.busy": "2026-06-09T20:07:40.898615Z", + "iopub.status.idle": "2026-06-09T20:07:41.266415Z", + "shell.execute_reply": "2026-06-09T20:07:41.264865Z" + } + }, "outputs": [ { "name": "stdout", @@ -719,7 +910,7 @@ "id": "6ab6cf82", "metadata": {}, "source": [ - "## Find — one tool, three channels\n", + "## Find — one tool, three retrieval channels, and Cypher filtering\n", "\n", "`search` retrieves both memories and canonical entities, merging three channels via Reciprocal Rank Fusion:\n", "\n", @@ -734,111 +925,283 @@ "cell_type": "code", "execution_count": 9, "id": "9c9057d0", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-09T20:07:41.269681Z", + "iopub.status.busy": "2026-06-09T20:07:41.269259Z", + "iopub.status.idle": "2026-06-09T20:07:42.044573Z", + "shell.execute_reply": "2026-06-09T20:07:42.042848Z" + } + }, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "Memories:\n", - " [lightning.brooklyn_pos] score=0.033\n", - " -> Brooklyn switched to a new POS system in late 2024. Order totals before 2025-01-01 are from the lega...\n" - ] + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
kindidscoretext
0memorylightning.brooklyn_pos0.032787Brooklyn switched to a new POS system in late ...
1memorylightning.top_customers0.016129Top-5 customers by lifetime spend - known-good...
\n", + "
" + ], + "text/plain": [ + " kind id score \\\n", + "0 memory lightning.brooklyn_pos 0.032787 \n", + "1 memory lightning.top_customers 0.016129 \n", + "\n", + " text \n", + "0 Brooklyn switched to a new POS system in late ... \n", + "1 Top-5 customers by lifetime spend - known-good... " + ] + }, + "execution_count": 9, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "# You can search using a text query\n", - "\n", + "# Search with a natural-language question. cypher_filter scopes the graph to\n", + "# memory nodes, so the *search call* does the narrowing — not Python.\n", "resp = run_sync(client.search(\n", " question='What should I know before comparing Brooklyn revenue to other stores?',\n", - " max_memories=3,\n", - " max_example_queries=2,\n", - " max_entities=0,\n", + " cypher_filter='MATCH (m:Memory) RETURN m.id AS id',\n", + " max_results=10,\n", "))\n", - "print('Memories:')\n", - "for hit in resp.memories:\n", - " print(f' [{hit.id}] score={hit.score:.3f}')\n", - " print(f' -> {hit.text[:100]}...')\n", - "assert any(m.id == 'lightning.brooklyn_pos' for m in resp.memories), \\\n", - " 'Brooklyn memory must surface in resp.memories for this question'" + "assert any(h.id == 'lightning.brooklyn_pos' for h in resp.results), 'Brooklyn memory must surface'\n", + "pd.DataFrame([h.model_dump() for h in resp.results])[['kind', 'id', 'score', 'text']]" ] }, { "cell_type": "code", "execution_count": 10, "id": "27ca5496", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-09T20:07:42.047419Z", + "iopub.status.busy": "2026-06-09T20:07:42.047143Z", + "iopub.status.idle": "2026-06-09T20:07:42.113470Z", + "shell.execute_reply": "2026-06-09T20:07:42.112418Z" + } + }, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "Memories anchored to jaffle_shop.orders.order_total:\n", - " [lightning.brooklyn_pos] matched_entities=['jaffle_shop.orders.order_total']\n", - " -> Brooklyn switched to a new POS system in late 2024. Order totals before 2025-01-01 are from the lega...\n" - ] + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
kindidscorematched_entities
0memorylightning.brooklyn_pos0.016393[jaffle_shop.orders.order_total]
1memorylightning.top_customers0.016129[jaffle_shop.orders.order_total]
\n", + "
" + ], + "text/plain": [ + " kind id score matched_entities\n", + "0 memory lightning.brooklyn_pos 0.016393 [jaffle_shop.orders.order_total]\n", + "1 memory lightning.top_customers 0.016129 [jaffle_shop.orders.order_total]" + ] + }, + "execution_count": 10, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "# And/or entity references\n", - "\n", + "# Anchor on a specific column instead of free text; the same graph filter applies.\n", "resp = run_sync(client.search(\n", " entities=['jaffle_shop.orders.order_total'],\n", - " max_memories=3,\n", - " max_entities=0,\n", + " cypher_filter='MATCH (m:Memory) RETURN m.id AS id',\n", + " max_results=10,\n", "))\n", - "print('Memories anchored to jaffle_shop.orders.order_total:')\n", - "for hit in resp.memories:\n", - " print(f' [{hit.id}] matched_entities={hit.matched_entities}')\n", - " print(f' -> {hit.text[:100]}...')\n", - "assert any(m.id == 'lightning.brooklyn_pos' for m in resp.memories), \\\n", - " 'Brooklyn memory must surface via the order_total entity tag'" + "assert any(h.id == 'lightning.brooklyn_pos' for h in resp.results), 'Brooklyn memory must surface via order_total'\n", + "pd.DataFrame([h.model_dump() for h in resp.results])[['kind', 'id', 'score', 'matched_entities']]" ] }, { "cell_type": "code", "execution_count": 11, "id": "48660096", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-09T20:07:42.116290Z", + "iopub.status.busy": "2026-06-09T20:07:42.116075Z", + "iopub.status.idle": "2026-06-09T20:07:42.519678Z", + "shell.execute_reply": "2026-06-09T20:07:42.518133Z" + } + }, "outputs": [ { - "name": "stdout", - "output_type": "stream", - "text": [ - "Example queries:\n", - " [lightning.top_customers] score=0.033\n", - " -> Top-5 customers by lifetime spend - known-good query pattern used in the weekly CRM-ops dashboard.\n", - "T...\n", - "\n", - "Canonical entities:\n", - " [column] jaffle_shop.orders.customer_id (score=0.016)\n", - " [model] jaffle_shop.customers (score=0.016)\n", - " [model] jaffle_shop.orders (score=0.016)\n", - " [column] jaffle_shop.customers.name (score=0.016)\n", - " [column] jaffle_shop.customers.id (score=0.015)\n" - ] + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
kindidscoretext
0memorylightning.top_customers0.032787Top-5 customers by lifetime spend - known-good...
1memorylightning.brooklyn_pos0.032258Brooklyn switched to a new POS system in late ...
2modeljaffle_shop.customers0.016129Model: jaffle_shop.customers\\nsql_table: custo...
3modeljaffle_shop.orders0.015873Model: jaffle_shop.orders\\nsql_table: orders\\n...
4modeljaffle_shop.tweets0.015152Model: jaffle_shop.tweets\\nsql_table: tweets\\n...
\n", + "
" + ], + "text/plain": [ + " kind id score \\\n", + "0 memory lightning.top_customers 0.032787 \n", + "1 memory lightning.brooklyn_pos 0.032258 \n", + "2 model jaffle_shop.customers 0.016129 \n", + "3 model jaffle_shop.orders 0.015873 \n", + "4 model jaffle_shop.tweets 0.015152 \n", + "\n", + " text \n", + "0 Top-5 customers by lifetime spend - known-good... \n", + "1 Brooklyn switched to a new POS system in late ... \n", + "2 Model: jaffle_shop.customers\\nsql_table: custo... \n", + "3 Model: jaffle_shop.orders\\nsql_table: orders\\n... \n", + "4 Model: jaffle_shop.tweets\\nsql_table: tweets\\n... " + ] + }, + "execution_count": 11, + "metadata": {}, + "output_type": "execute_result" } ], "source": [ - "# And specifically look for memories with queries\n", - "\n", + "# cypher_filter is a graph pre-filter. A multi-label match scopes the search to\n", + "# memories AND model entities in one call (columns, measures, … are dropped).\n", "resp = run_sync(client.search(\n", " question='How have analysts queried customer lifetime spend before?',\n", - " max_memories=0,\n", - " max_example_queries=2,\n", - " max_entities=5,\n", + " cypher_filter='MATCH (n:Memory:Model) RETURN n.id AS id',\n", + " max_results=20,\n", "))\n", - "print('Example queries:')\n", - "for eq in resp.example_queries:\n", - " print(f' [{eq.id}] score={eq.score:.3f}')\n", - " print(f' -> {eq.text[:100]}...')\n", - "print()\n", - "print('Canonical entities:')\n", - "for ent in resp.entities:\n", - " print(f' [{ent.kind}] {ent.id} (score={ent.score:.3f})')\n", - "assert any(eq.id == 'lightning.top_customers' for eq in resp.example_queries), \\\n", - " 'top-customers example query must surface in resp.example_queries'\n", - "assert len(resp.entities) > 0, 'expected at least one canonical entity hit'" + "example_queries = [h for h in resp.results if h.query is not None]\n", + "entities = [h for h in resp.results if h.kind != 'memory']\n", + "assert any(h.id == 'lightning.top_customers' for h in example_queries), 'top-customers query must surface'\n", + "assert entities, 'expected at least one entity hit'\n", + "pd.DataFrame([h.model_dump() for h in resp.results])[['kind', 'id', 'score', 'text']]" ] }, { @@ -855,9 +1218,8 @@ "* Auto-ingestion\n", "* Postgres / MySQL/ DuckDB / ClickHouse / SQLite\n", "* Multistage queries, named measures, custom aggregations,\n", - "* Memories with embedding + entity + full text search.\n", + "* Memories with embedding + entity + full text search, and Cypher filtering\n", "\n", - "**Coming next:** proper knowledge graph support, so memories and entities form an explicit graph the agent can traverse.\n", "\n", "### Try it in Claude Code\n", "\n", @@ -876,7 +1238,14 @@ "cell_type": "code", "execution_count": 12, "id": "8e2c6030", - "metadata": {}, + "metadata": { + "execution": { + "iopub.execute_input": "2026-06-09T20:07:42.522580Z", + "iopub.status.busy": "2026-06-09T20:07:42.522222Z", + "iopub.status.idle": "2026-06-09T20:07:42.557441Z", + "shell.execute_reply": "2026-06-09T20:07:42.555704Z" + } + }, "outputs": [ { "name": "stdout", @@ -918,7 +1287,7 @@ "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", - "version": "3.13.12" + "version": "3.12.3" } }, "nbformat": 4, diff --git a/docs/examples/09_lightning_talk/slayer_models/.gitignore b/docs/examples/09_lightning_talk/slayer_models/.gitignore index 52455640..0c0103d5 100644 --- a/docs/examples/09_lightning_talk/slayer_models/.gitignore +++ b/docs/examples/09_lightning_talk/slayer_models/.gitignore @@ -2,5 +2,5 @@ # Only the committed models/ snapshot is portable across machines. datasources/ embeddings.db -memories.yaml -memories.yaml.lock +memories/ +memories.lock diff --git a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/customers.yaml b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/customers.yaml index 9e1f014c..386184aa 100644 --- a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/customers.yaml +++ b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/customers.yaml @@ -1,4 +1,4 @@ -version: 6 +version: 7 name: customers sql_table: customers query_variables: {} @@ -8,14 +8,23 @@ columns: sql: id type: TEXT primary_key: true + label: Customer ID hidden: false - name: name sql: name type: TEXT primary_key: false + description: Customer name. + label: Customer hidden: false -measures: [] +measures: +- formula: id:count_distinct + name: customer_count + label: Customers + description: Number of distinct customers. + type: INT aggregations: [] joins: [] filters: [] +description: "Customers of the Jaffle Shop \u2014 one row per person." hidden: false diff --git a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/items.yaml b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/items.yaml index 4639cfe5..a54061be 100644 --- a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/items.yaml +++ b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/items.yaml @@ -1,4 +1,4 @@ -version: 6 +version: 7 name: items sql_table: items query_variables: {} @@ -8,18 +8,28 @@ columns: sql: id type: TEXT primary_key: true + label: Item ID hidden: false - name: order_id sql: order_id type: TEXT primary_key: false + description: Order this line belongs to. + label: Order hidden: false - name: sku sql: sku type: TEXT primary_key: false + description: Product sold on this line. + label: Product hidden: false -measures: [] +measures: +- formula: id:count + name: units_sold + label: Units Sold + description: Number of item units sold. + type: INT aggregations: [] joins: - target_model: orders @@ -33,4 +43,5 @@ joins: - sku join_type: left filters: [] +description: "Order line items \u2014 one row per unit sold on an order." hidden: false diff --git a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/orders.yaml b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/orders.yaml index 12d23930..8c059245 100644 --- a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/orders.yaml +++ b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/orders.yaml @@ -1,4 +1,4 @@ -version: 6 +version: 7 name: orders sql_table: orders query_variables: {} @@ -8,45 +8,110 @@ columns: sql: id type: TEXT primary_key: true + label: Order ID hidden: false - name: customer_id sql: customer_id type: TEXT primary_key: false + description: Customer who placed the order. + label: Customer ID hidden: false - name: ordered_at sql: ordered_at type: DATE primary_key: false + description: When the order was placed. + label: Order Date hidden: false - name: store_id sql: store_id type: TEXT primary_key: false + description: Store where the order was placed. + label: Store ID hidden: false - name: subtotal sql: subtotal type: DOUBLE primary_key: false + description: Pre-tax order amount, in dollars. + label: Net Sales hidden: false format: - type: float + type: currency + precision: 2 + symbol: $ - name: tax_paid sql: tax_paid type: DOUBLE primary_key: false + description: Tax charged on the order, in dollars. + label: Tax Paid hidden: false format: - type: float + type: currency + precision: 2 + symbol: $ - name: order_total sql: order_total type: DOUBLE primary_key: false + description: Final order amount including tax, in dollars. + label: Order Total hidden: false format: - type: float -measures: [] -aggregations: [] + type: currency + precision: 2 + symbol: $ +measures: +- formula: order_total:sum + name: total_revenue + label: Total Revenue + description: Gross sales including tax, in dollars. + type: DOUBLE +- formula: subtotal:sum + name: net_sales + label: Net Sales (pre-tax) + description: Sales before tax, in dollars. + type: DOUBLE +- formula: tax_paid:sum + name: tax_collected + label: Tax Collected + description: Total tax collected, in dollars. + type: DOUBLE +- formula: id:count + name: order_count + label: Orders + description: Number of orders. + type: INT +- formula: customer_id:count_distinct + name: unique_customers + label: Unique Customers + description: Distinct customers who ordered. + type: INT +- formula: order_total:sum / nullif(id:count, 0) + name: avg_order_value + label: Average Order Value + description: Revenue per order, in dollars. + type: DOUBLE +- formula: tax_paid:sum / nullif(subtotal:sum, 0) + name: effective_tax_rate + label: Effective Tax Rate + description: Tax collected as a share of net sales. + type: DOUBLE +- formula: order_total:weighted_avg + name: sales_weighted_aov + label: Sales-Weighted Avg Order + description: Average order total weighted by subtotal (larger orders weigh more), + in dollars. + type: DOUBLE +aggregations: +- name: weighted_avg + params: + - name: weight + sql: subtotal + description: Weighted average defaulting the weight to order subtotal. joins: - target_model: customers join_pairs: @@ -60,4 +125,7 @@ joins: join_type: left filters: [] default_time_dimension: ordered_at +description: "Customer orders \u2014 one row per order. Monetary amounts are in dollars.\ + \ The fact table at the center of the demo: joins to customers and stores, and is\ + \ referenced by items." hidden: false diff --git a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/products.yaml b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/products.yaml index b507cdfc..9f3988e2 100644 --- a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/products.yaml +++ b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/products.yaml @@ -1,4 +1,4 @@ -version: 6 +version: 7 name: products sql_table: products query_variables: {} @@ -8,31 +8,54 @@ columns: sql: sku type: TEXT primary_key: true + description: Product identifier. + label: SKU hidden: false - name: name sql: name type: TEXT primary_key: false + description: Product name. + label: Product hidden: false - name: type sql: type type: TEXT primary_key: false + description: Product category (jaffle or beverage). + label: Category hidden: false - name: price sql: price type: DOUBLE primary_key: false + description: List price, in dollars. + label: Price hidden: false format: - type: float + type: currency + precision: 2 + symbol: $ - name: description sql: description type: TEXT primary_key: false + description: Product description. + label: Description hidden: false -measures: [] +measures: +- formula: sku:count_distinct + name: product_count + label: Products + description: Number of distinct products. + type: INT +- formula: price:avg + name: avg_price + label: Average Price + description: Average list price, in dollars. + type: DOUBLE aggregations: [] joins: [] filters: [] +description: "Menu items sold at the Jaffle Shop \u2014 jaffles and beverages." hidden: false diff --git a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/stores.yaml b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/stores.yaml index 66646696..6165f734 100644 --- a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/stores.yaml +++ b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/stores.yaml @@ -1,4 +1,4 @@ -version: 6 +version: 7 name: stores sql_table: stores query_variables: {} @@ -8,26 +8,40 @@ columns: sql: id type: TEXT primary_key: true + label: Store ID hidden: false - name: name sql: name type: TEXT primary_key: false + description: Store name. + label: Store hidden: false - name: opened_at sql: opened_at type: DATE primary_key: false + description: When the store opened. + label: Opened hidden: false - name: tax_rate sql: tax_rate type: DOUBLE primary_key: false + description: Local sales-tax rate. + label: Tax Rate hidden: false format: - type: float -measures: [] + type: percent + precision: 1 +measures: +- formula: id:count_distinct + name: store_count + label: Stores + description: Number of stores. + type: INT aggregations: [] joins: [] filters: [] +description: Physical Jaffle Shop locations. hidden: false diff --git a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/supplies.yaml b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/supplies.yaml index 9ce95e0a..1b35a87f 100644 --- a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/supplies.yaml +++ b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/supplies.yaml @@ -1,4 +1,4 @@ -version: 6 +version: 7 name: supplies sql_table: supplies query_variables: {} @@ -8,30 +8,51 @@ columns: sql: id type: TEXT primary_key: true + label: Supply ID hidden: false - name: name sql: name type: TEXT primary_key: false + description: Supply name. + label: Supply hidden: false - name: cost sql: cost type: DOUBLE primary_key: false + description: Cost per unit, in dollars. + label: Unit Cost hidden: false format: - type: float + type: currency + precision: 2 + symbol: $ - name: perishable sql: perishable type: TEXT primary_key: false + description: Whether the supply is perishable. + label: Perishable hidden: false - name: sku sql: sku type: TEXT primary_key: true + description: Product this supply is used for. + label: Product hidden: false -measures: [] +measures: +- formula: cost:sum + name: total_supply_cost + label: Total Supply Cost + description: Total supply cost, in dollars. + type: DOUBLE +- formula: cost:avg + name: avg_unit_cost + label: Avg Unit Cost + description: Average supply unit cost, in dollars. + type: DOUBLE aggregations: [] joins: - target_model: products @@ -40,4 +61,6 @@ joins: - sku join_type: left filters: [] +description: Supplies (ingredients and packaging) used per product, with unit costs + in dollars. hidden: false diff --git a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/tweets.yaml b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/tweets.yaml index 9b55ad12..6056df54 100644 --- a/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/tweets.yaml +++ b/docs/examples/09_lightning_talk/slayer_models/models/jaffle_shop/tweets.yaml @@ -1,4 +1,4 @@ -version: 6 +version: 7 name: tweets sql_table: tweets query_variables: {} @@ -8,23 +8,35 @@ columns: sql: id type: TEXT primary_key: true + label: Tweet ID hidden: false - name: user_id sql: user_id type: TEXT primary_key: false + description: Customer who tweeted. + label: Customer ID hidden: false - name: tweeted_at sql: tweeted_at type: DATE primary_key: false + description: When the tweet was posted. + label: Tweet Date hidden: false - name: content sql: content type: TEXT primary_key: false + description: Tweet text. + label: Tweet hidden: false -measures: [] +measures: +- formula: id:count + name: tweet_count + label: Tweets + description: Number of tweets. + type: INT aggregations: [] joins: - target_model: customers @@ -34,4 +46,5 @@ joins: join_type: left filters: [] default_time_dimension: tweeted_at +description: Synthetic customer tweets mentioning the Jaffle Shop. hidden: false diff --git a/docs/examples/10_row_level_security/row_level_security_nb.ipynb b/docs/examples/10_row_level_security/row_level_security_nb.ipynb new file mode 100644 index 00000000..e2d961eb --- /dev/null +++ b/docs/examples/10_row_level_security/row_level_security_nb.ipynb @@ -0,0 +1,638 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "cee16d33", + "metadata": {}, + "source": [ + "# Row-Level Security (forced filter)\n", + "\n", + "**TL;DR:** Hand `SlayerQueryEngine` (or the local-mode `SlayerClient`) a\n", + "`SessionPolicy`, and every query it runs is silently scoped to one tenant —\n", + "base tables, joins, sub-queries, and sample data included. The agent issuing\n", + "queries cannot read, override, or disable the policy.\n", + "\n", + "This notebook tells a **franchise** story on the Jaffle Shop data: a session\n", + "that belongs to one store's operator must only ever see that store's data.\n", + "`orders` carries a `store_id`. A `SessionPolicy` holds exactly one ruleset, so\n", + "this notebook demonstrates the two kinds in **separate** engines: first a\n", + "**column ruleset** that scopes every table with a `store_id`, then a separate\n", + "policy with a **join ruleset** that reaches the column-less `customers` table\n", + "through an explicit join.\n", + "\n", + "See also: [Row-Level Security (concept)](../../concepts/row-level-security.md)\n", + "\n", + "**Prerequisites:** `pip install motley-slayer`" + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "6c97b7e3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:47.376779Z", + "iopub.status.busy": "2026-07-03T11:00:47.376383Z", + "iopub.status.idle": "2026-07-03T11:00:47.639702Z", + "shell.execute_reply": "2026-07-03T11:00:47.639343Z" + } + }, + "outputs": [], + "source": [ + "import os\n", + "import sys\n", + "\n", + "sys.path.insert(0, os.path.join(os.getcwd(), \"..\", \"..\", \"..\"))\n", + "sys.path.insert(0, os.path.join(os.getcwd(), \"..\", \"jaffle_data\"))\n", + "\n", + "from setup_jaffle import ensure_jaffle_shop\n", + "\n", + "engine, storage, models = ensure_jaffle_shop()" + ] + }, + { + "cell_type": "markdown", + "id": "cd8c4dde", + "metadata": {}, + "source": [ + "## The data, unscoped\n", + "\n", + "With no policy, the engine sees every store. Let's list orders per store and\n", + "pick the busiest one to scope our session to." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "abc694e3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:47.646151Z", + "iopub.status.busy": "2026-07-03T11:00:47.645765Z", + "iopub.status.idle": "2026-07-03T11:00:47.742414Z", + "shell.execute_reply": "2026-07-03T11:00:47.742070Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Store Orders\n", + "--------------------------\n", + "Brooklyn 255,149\n", + "Philadelphia 193,538\n", + "Chicago 105,684\n", + "San Francisco 93,920\n", + "New Orleans 15,809\n", + "\n", + "Scoping the session to: Brooklyn (bd80252e-cb73-43d4-af52-2bacfab2176b)\n" + ] + } + ], + "source": [ + "by_store = engine.execute_sync(\n", + " query={\n", + " \"source_model\": \"orders\",\n", + " \"measures\": [\"*:count\"],\n", + " \"dimensions\": [\"stores.name\", \"store_id\"],\n", + " \"order\": [{\"column\": \"_count\", \"direction\": \"desc\"}],\n", + " }\n", + ")\n", + "\n", + "print(f\"{'Store':<16}{'Orders':>10}\")\n", + "print(\"-\" * 26)\n", + "for row in by_store.data:\n", + " print(f\"{row['orders.stores.name']:<16}{row['orders._count']:>10,}\")\n", + "\n", + "# Scope this session to the busiest store (a franchise operator's view).\n", + "busiest = max(by_store.data, key=lambda r: r[\"orders._count\"])\n", + "STORE_ID = busiest[\"orders.store_id\"]\n", + "STORE_NAME = busiest[\"orders.stores.name\"]\n", + "TOTAL_ORDERS = sum(r[\"orders._count\"] for r in by_store.data)\n", + "print(f\"\\nScoping the session to: {STORE_NAME} ({STORE_ID})\")" + ] + }, + { + "cell_type": "markdown", + "id": "42b6644e", + "metadata": {}, + "source": [ + "## Configure the policy\n", + "\n", + "A `SessionPolicy` carries one **ruleset**. Here it is a `ColumnFilterRuleset`,\n", + "which means *\"every table that has a `store_id` column is filtered to this\n", + "store.\"* The value is a scalar, so the operator is `=` (a list would mean\n", + "`IN (...)`).\n", + "\n", + "`on_unapplicable=\"pass\"` says: a table that **lacks** `store_id` (the shared\n", + "product catalog, the customer list) is left unfiltered rather than blocked —\n", + "those tables aren't store-specific. We'll see the stricter `\"block\"` mode\n", + "further down.\n", + "\n", + "The policy is set **once**, at engine construction. There is no query field\n", + "that can change it. (`ruleset` is required — the no-filtering case is simply\n", + "`policy=None`.)" + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "2fc23f20", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:47.743456Z", + "iopub.status.busy": "2026-07-03T11:00:47.743355Z", + "iopub.status.idle": "2026-07-03T11:00:47.745325Z", + "shell.execute_reply": "2026-07-03T11:00:47.745065Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Scoped engine ready.\n" + ] + } + ], + "source": [ + "from slayer.core.policy import SessionPolicy, ColumnFilterRuleset\n", + "from slayer.engine.query_engine import SlayerQueryEngine\n", + "\n", + "policy = SessionPolicy(\n", + " ruleset=ColumnFilterRuleset(\n", + " column=\"store_id\", value=STORE_ID, on_unapplicable=\"pass\"\n", + " )\n", + ")\n", + "store_engine = SlayerQueryEngine(storage=storage, policy=policy)\n", + "print(\"Scoped engine ready.\")" + ] + }, + { + "cell_type": "markdown", + "id": "fb817453", + "metadata": {}, + "source": [ + "## Same query, now tenant-scoped\n", + "\n", + "The query is identical to one you'd run unscoped — no `store_id` filter, no\n", + "model changes. The engine injects the scoping for us." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "a5471b77", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:47.746272Z", + "iopub.status.busy": "2026-07-03T11:00:47.746203Z", + "iopub.status.idle": "2026-07-03T11:00:47.973669Z", + "shell.execute_reply": "2026-07-03T11:00:47.973310Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "All stores: 664,100 orders\n", + "Brooklyn only: 255,149 orders\n" + ] + } + ], + "source": [ + "all_orders = engine.execute_sync(\n", + " query={\"source_model\": \"orders\", \"measures\": [\"*:count\"]}\n", + ")\n", + "scoped_orders = store_engine.execute_sync(\n", + " query={\"source_model\": \"orders\", \"measures\": [\"*:count\"]}\n", + ")\n", + "\n", + "print(f\"All stores: {all_orders.data[0]['orders._count']:>10,} orders\")\n", + "print(f\"{STORE_NAME} only: {scoped_orders.data[0]['orders._count']:>10,} orders\")" + ] + }, + { + "cell_type": "markdown", + "id": "d2465478", + "metadata": {}, + "source": [ + "## Preview the rewritten SQL\n", + "\n", + "`dry_run=True` returns exactly the SQL that would execute — including the\n", + "per-table wrap. Each physical `orders` reference becomes a filtered sub-query,\n", + "with the original alias preserved. The literal is bound, so it's injection-safe." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "b0c1e2bd", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:47.975055Z", + "iopub.status.busy": "2026-07-03T11:00:47.974931Z", + "iopub.status.idle": "2026-07-03T11:00:47.989655Z", + "shell.execute_reply": "2026-07-03T11:00:47.989373Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SELECT SUM(orders.order_total) AS \"orders.order_total_sum\" FROM (SELECT * FROM orders WHERE store_id = 'bd80252e-cb73-43d4-af52-2bacfab2176b') AS orders\n" + ] + } + ], + "source": [ + "preview = store_engine.execute_sync(\n", + " query={\"source_model\": \"orders\", \"measures\": [\"order_total:sum\"]},\n", + " dry_run=True,\n", + ")\n", + "print(preview.sql)" + ] + }, + { + "cell_type": "markdown", + "id": "d3a754ba", + "metadata": {}, + "source": [ + "## Joins stay scoped on every side\n", + "\n", + "Cross-tenant leakage usually hides in joins. Here we rank customers by revenue\n", + "— a query that joins `orders` to `customers`. Under the policy, the `orders`\n", + "side is wrapped (it has `store_id`); `customers` is a shared table (no\n", + "`store_id`, so it passes through). The result only ever reflects **this\n", + "store's** orders." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "f97c71e9", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:47.990928Z", + "iopub.status.busy": "2026-07-03T11:00:47.990859Z", + "iopub.status.idle": "2026-07-03T11:00:48.052278Z", + "shell.execute_reply": "2026-07-03T11:00:48.051656Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Top customers at Brooklyn (this store's revenue only):\n", + " Cheyenne Briggs $11,761.36\n", + " Daniel Schneider $11,601.20\n", + " Scott Jackson $10,842.00\n", + " Tony Mendoza $10,811.84\n", + " Michael Schultz $10,621.52\n" + ] + } + ], + "source": [ + "top_customers = store_engine.execute_sync(\n", + " query={\n", + " \"source_model\": \"orders\",\n", + " \"measures\": [\"order_total:sum\"],\n", + " \"dimensions\": [\"customers.name\"],\n", + " \"order\": [{\"column\": \"order_total_sum\", \"direction\": \"desc\"}],\n", + " \"limit\": 5,\n", + " }\n", + ")\n", + "\n", + "print(f\"Top customers at {STORE_NAME} (this store's revenue only):\")\n", + "for row in top_customers.data:\n", + " print(f\" {row['orders.customers.name']:<22} ${row['orders.order_total_sum']:>9,.2f}\")" + ] + }, + { + "cell_type": "markdown", + "id": "48c21f3a", + "metadata": {}, + "source": [ + "## `pass` vs `block`: the tables without the column\n", + "\n", + "`store_id` lives only on `orders`. What happens when a query touches a table\n", + "that **doesn't** have it — like the shared `products` catalog?\n", + "\n", + "- With `on_unapplicable=\"pass\"` (our policy), that table flows through\n", + " unfiltered — products aren't store-specific, so this is correct." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "5d23e483", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:48.053555Z", + "iopub.status.busy": "2026-07-03T11:00:48.053477Z", + "iopub.status.idle": "2026-07-03T11:00:48.070001Z", + "shell.execute_reply": "2026-07-03T11:00:48.069358Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Products (shared catalog, passed through): 10\n" + ] + } + ], + "source": [ + "products = store_engine.execute_sync(\n", + " query={\"source_model\": \"products\", \"measures\": [\"*:count\"]}\n", + ")\n", + "print(f\"Products (shared catalog, passed through): {products.data[0]['products._count']}\")" + ] + }, + { + "cell_type": "markdown", + "id": "ea792ad7", + "metadata": {}, + "source": [ + "- With the default `on_unapplicable=\"block\"`, a table that confirms it lacks\n", + " the tenant column **fails the whole query**. Use this when *every* table is\n", + " expected to carry the tenant column (true RLS-on-every-table), so a missing\n", + " column surfaces as an error instead of a silent leak." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "e659b274", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:48.071501Z", + "iopub.status.busy": "2026-07-03T11:00:48.071378Z", + "iopub.status.idle": "2026-07-03T11:00:48.087352Z", + "shell.execute_reply": "2026-07-03T11:00:48.086962Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Blocked: Forced filter rule on column 'store_id' requires column 'store_id' on table 'products', which does not have it.\n", + " table='products' column='store_id'\n" + ] + } + ], + "source": [ + "from slayer.core.errors import ForcedFilterError\n", + "\n", + "strict_engine = SlayerQueryEngine(\n", + " storage=storage,\n", + " policy=SessionPolicy(\n", + " ruleset=ColumnFilterRuleset(column=\"store_id\", value=STORE_ID) # block (default)\n", + " ),\n", + ")\n", + "\n", + "try:\n", + " strict_engine.execute_sync(\n", + " query={\"source_model\": \"products\", \"measures\": [\"*:count\"]}\n", + " )\n", + "except ForcedFilterError as exc:\n", + " print(f\"Blocked: {exc}\")\n", + " print(f\" table={exc.table!r} column={exc.column!r}\")" + ] + }, + { + "cell_type": "markdown", + "id": "27914a8d", + "metadata": {}, + "source": [ + "A table whose column presence **cannot be confirmed** at all (an\n", + "introspection error) always fails closed, regardless of `on_unapplicable` —\n", + "SLayer never emits an unscoped query against a table it couldn't verify." + ] + }, + { + "cell_type": "markdown", + "id": "bd0b5c33", + "metadata": {}, + "source": [ + "## Join ruleset: scope a table that *lacks* the column\n", + "\n", + "`store_id` lives only on `orders`. Above, `customers` was left to pass\n", + "through unfiltered. But sometimes you want the column-less table scoped\n", + "too — here, to **only the customers who have ordered at this store**.\n", + "\n", + "A `JoinFilterRuleset` names the one anchor table that holds the identifier\n", + "(`orders.store_id`), and each `JoinFilterRule` reaches it through a join\n", + "**stated explicitly in the policy** (physical table/column names), scoping the\n", + "target with a correlated `EXISTS` semi-join — cardinality-safe and\n", + "`LEFT JOIN`-preserving. Shared tables go in the `whitelist`; any table that is\n", + "neither the anchor, a join target, nor whitelisted **fails closed**, so a table\n", + "nobody accounted for can't leak." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "id": "63c082fe", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:48.088090Z", + "iopub.status.busy": "2026-07-03T11:00:48.088022Z", + "iopub.status.idle": "2026-07-03T11:00:48.117220Z", + "shell.execute_reply": "2026-07-03T11:00:48.116422Z" + } + }, + "outputs": [], + "source": [ + "from slayer.core.policy import JoinFilterRuleset, JoinFilterRule\n", + "\n", + "join_policy = SessionPolicy(\n", + " ruleset=JoinFilterRuleset(\n", + " # The identifier lives on orders.store_id — the anchor, filtered directly.\n", + " table=\"orders\",\n", + " column=\"store_id\",\n", + " value=STORE_ID,\n", + " joins=[\n", + " # customers has no store_id -> reach it via customers.id = orders.customer_id\n", + " JoinFilterRule(\n", + " target_table=\"customers\",\n", + " join_path=[\"customers.id = orders.customer_id\"],\n", + " ),\n", + " ],\n", + " # shared catalogs need no scoping — emit them unfiltered.\n", + " whitelist=[\"products\", \"stores\"],\n", + " )\n", + ")\n", + "join_engine = SlayerQueryEngine(storage=storage, policy=join_policy)\n", + "\n", + "all_customers = engine.execute_sync(\n", + " query={\"source_model\": \"customers\", \"measures\": [\"*:count\"]}\n", + ").data[0][\"customers._count\"]\n", + "store_customers = join_engine.execute_sync(\n", + " query={\"source_model\": \"customers\", \"measures\": [\"*:count\"]}\n", + ").data[0][\"customers._count\"]\n", + "\n", + "print(f\"Customers, all stores: {all_customers:>6,}\")\n", + "print(f\"Customers who bought at {STORE_NAME}: {store_customers:>6,}\")" + ] + }, + { + "cell_type": "markdown", + "id": "a337af09", + "metadata": {}, + "source": [ + "The `customers` table is now wrapped in the correlated semi-join — it\n", + "keeps only rows that have a matching order at this store. `orders` (the\n", + "anchor, which carries `store_id`) is filtered directly wherever it appears.\n", + "\n", + "Classification is **structural**: a table is the anchor, a join target, a\n", + "whitelisted pass-through, or an error. Nothing is probed against the database.\n", + "(On ClickHouse, correlated subqueries need server >= 25.4 — SLayer probes the\n", + "version and fails closed on older servers.)" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "acb870fa", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:48.118767Z", + "iopub.status.busy": "2026-07-03T11:00:48.118612Z", + "iopub.status.idle": "2026-07-03T11:00:48.143784Z", + "shell.execute_reply": "2026-07-03T11:00:48.143003Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SELECT COUNT(*) AS \"customers._count\" FROM (SELECT * FROM customers AS _rls_src WHERE EXISTS(SELECT 1 FROM orders AS _rls_j0 WHERE _rls_j0.customer_id = _rls_src.id AND _rls_j0.store_id = 'bd80252e-cb73-43d4-af52-2bacfab2176b')) AS customers\n" + ] + } + ], + "source": [ + "preview = join_engine.execute_sync(\n", + " query={\"source_model\": \"customers\", \"measures\": [\"*:count\"]},\n", + " dry_run=True,\n", + ")\n", + "print(preview.sql)" + ] + }, + { + "cell_type": "markdown", + "id": "3994d61f", + "metadata": {}, + "source": [ + "## Immutable and agent-invisible\n", + "\n", + "The policy is frozen engine state. An agent can't widen its own scope by\n", + "mutating the policy or by adding a contradictory filter — the wrap is applied\n", + "to the final SQL after the agent's query has been generated." + ] + }, + { + "cell_type": "code", + "execution_count": 11, + "id": "8e046f1e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-03T11:00:48.145267Z", + "iopub.status.busy": "2026-07-03T11:00:48.145134Z", + "iopub.status.idle": "2026-07-03T11:00:48.181987Z", + "shell.execute_reply": "2026-07-03T11:00:48.181592Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Mutating the policy is rejected: ValidationError\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Agent asked for a different store -> rows returned: 0\n" + ] + } + ], + "source": [ + "# The policy object itself is immutable.\n", + "try:\n", + " store_engine.policy.ruleset = None\n", + "except Exception as exc:\n", + " print(f\"Mutating the policy is rejected: {type(exc).__name__}\")\n", + "\n", + "# Even if the agent asks for another store explicitly, it still only sees\n", + "# this store's rows — the forced filter is ANDed onto the final SQL.\n", + "other = next(\n", + " (r[\"orders.store_id\"] for r in by_store.data if r[\"orders.store_id\"] != STORE_ID),\n", + " None,\n", + ")\n", + "if other:\n", + " sneaky = store_engine.execute_sync(\n", + " query={\n", + " \"source_model\": \"orders\",\n", + " \"measures\": [\"*:count\"],\n", + " \"filters\": [f\"store_id = '{other}'\"],\n", + " }\n", + " )\n", + " print(f\"Agent asked for a different store -> rows returned: {sneaky.data[0]['orders._count']}\")" + ] + }, + { + "cell_type": "markdown", + "id": "18f0c6fe", + "metadata": {}, + "source": [ + "## Summary\n", + "\n", + "| Aspect | Behaviour |\n", + "|---|---|\n", + "| **Where** | `SlayerQueryEngine(storage, policy=...)` / `SlayerClient(storage=..., policy=...)` |\n", + "| **`ColumnFilterRuleset`** | wraps every physical table that has the tenant column in a filtered sub-query |\n", + "| **`JoinFilterRuleset`** | one anchor table holds the identifier; others reach it via an explicit join path + correlated `EXISTS`; a `whitelist` passes shared tables; anything else fails closed |\n", + "| **Operator** | scalar value -> `=`, list value -> `IN (...)` |\n", + "| **`on_unapplicable=\"pass\"`** | (column ruleset) tables lacking the column are left unfiltered (shared data) |\n", + "| **`on_unapplicable=\"block\"`** | (column ruleset) tables lacking the column fail the query |\n", + "| **Unconfirmable / unlisted table** | always fails closed (security control) |\n", + "| **Scope** | base, joins, CTEs, sql-mode, query-backed stages, profiling/sample data |\n", + "| **Mutability** | immutable; set only at engine/client init; the agent can't read or change it |\n", + "\n", + "Forced filters are the tightest useful slice of tenant isolation. Per-model\n", + "scoping and server-side (REST/MCP) policy are future additions — see\n", + "[Row-Level Security](../../concepts/row-level-security.md)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} \ No newline at end of file diff --git a/docs/examples/11_dbt_metricflow/dbt_metricflow.md b/docs/examples/11_dbt_metricflow/dbt_metricflow.md new file mode 100644 index 00000000..837d8d32 --- /dev/null +++ b/docs/examples/11_dbt_metricflow/dbt_metricflow.md @@ -0,0 +1,35 @@ +# From dbt MetricFlow to SLayer + +SLayer can ingest a dbt [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow) semantic layer — its `semantic_models` and `metrics` — and turn it into queryable SLayer models. This worked example runs the real [dbt-labs ACME Insurance benchmark](https://github.com/dbt-labs/semantic-layer-llm-benchmarking) through the converter end-to-end and answers two of its questions, checking each answer against the benchmark's gold SQL. + +The companion notebook ([`dbt_metricflow_nb.ipynb`](dbt_metricflow_nb.ipynb)) is self-contained — everything it generates lands in a gitignored `.cache/` directory next to it: + +1. **Clone** the dbt project at a pinned commit (a shallow `git` fetch of a few MB; reused on later runs). +2. **Load** its CSV data into a local DuckDB file. +3. **Convert** the dbt MetricFlow definitions into SLayer models with [`DbtToSlayerConverter`](../../dbt/dbt_import.md). +4. **Query** the converted models with hand-written SLayer queries — and verify against gold SQL. + +## What the conversion produces + +Each dbt **semantic model** becomes a SLayer model; each dbt **metric** folds into a `ModelMeasure` formula on its source model. The second query showcases the metric types this conversion handles: + +- `loss_payment_amount` and `loss_reserve_amount` are **simple metrics with a filter** (`has_loss_payment = 1` / `has_loss_reserve = 1`). The converter pushes the filter down so each becomes a filtered aggregate. +- `total_loss_amount` is a **derived metric** — `loss_payment_amount + loss_reserve_amount` — expressed as a formula over the two filtered metrics. + +## The two queries + +| Question | SLayer query | Verified against | +|----------|--------------|------------------| +| How many claims do we have? | `{"source_model": "claim", "measures": ["*:count"]}` | `SELECT COUNT(*) FROM claim` | +| Total loss by claim number | `total_loss_amount` grouped by `claim.company_claim_number` | the benchmark's multi-join gold SQL | + +The claim-number grouping reaches across a join that the converter inferred from the dbt entities — no manual SQL join is written. Both answers match the gold SQL exactly. + +## Gold checks run up front + +SLayer opens the DuckDB file through a read-write engine, and DuckDB will not let a second raw connection share the file under a different configuration. The notebook therefore runs every gold SQL query **before** any SLayer query touches the file, caches the expected numbers, and compares afterwards. + +## Further reading + +- [Importing dbt Semantic Layer definitions](../../dbt/dbt_import.md) — the full conversion reference, including what is converted exactly and what fails cleanly. +- [SLayer vs dbt](../../dbt/slayer_vs_dbt.md) — how the two semantic layers compare. diff --git a/docs/examples/11_dbt_metricflow/dbt_metricflow_nb.ipynb b/docs/examples/11_dbt_metricflow/dbt_metricflow_nb.ipynb new file mode 100644 index 00000000..23c07c86 --- /dev/null +++ b/docs/examples/11_dbt_metricflow/dbt_metricflow_nb.ipynb @@ -0,0 +1,445 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "19052d0a", + "metadata": {}, + "source": [ + "# From dbt MetricFlow to SLayer\n", + "\n", + "**TL;DR:** SLayer can ingest a dbt [MetricFlow](https://docs.getdbt.com/docs/build/about-metricflow) semantic layer — `semantic_models` + `metrics` — and turn it into queryable SLayer models. This notebook runs the real [dbt-labs ACME Insurance benchmark](https://github.com/dbt-labs/semantic-layer-llm-benchmarking) through the converter end-to-end and answers two of its questions, checking each answer against the benchmark's gold SQL.\n", + "\n", + "It runs as three explicit steps (everything lands in a gitignored `.cache/` next to this notebook):\n", + "\n", + "1. **Get the data** — clone the dbt project at a pinned commit and load its CSVs into a local DuckDB file.\n", + "2. **Convert** — run the dbt MetricFlow definitions through `convert_dbt_to_slayer` (its own cell below).\n", + "3. **Query** — point a SLayer client at the converted models and verify against gold SQL.\n", + "\n", + "> Requires network access on first run (a shallow `git` fetch of ~a few MB). Re-runs are instant from the cache." + ] + }, + { + "cell_type": "markdown", + "id": "f472b929", + "metadata": {}, + "source": [ + "## Step 1 — Get the data\n", + "\n", + "Clone the pinned dbt project and load its ACME Insurance CSVs into DuckDB." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "354742bb", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-01T15:51:11.158504Z", + "iopub.status.busy": "2026-07-01T15:51:11.158396Z", + "iopub.status.idle": "2026-07-01T15:51:11.598035Z", + "shell.execute_reply": "2026-07-01T15:51:11.597368Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Cloned dbt project @ e4bdee5baeaa and loaded ACME CSVs into DuckDB.\n" + ] + } + ], + "source": [ + "import os\n", + "import sys\n", + "\n", + "import pandas as pd\n", + "\n", + "# The setup helper lives next to this notebook.\n", + "sys.path.insert(0, os.getcwd())\n", + "\n", + "from setup_metricflow import (\n", + " ensure_dbt_data,\n", + " convert_dbt_to_slayer,\n", + " fetch_gold,\n", + " DB_PATH,\n", + " MODELS_DIR,\n", + " DBT_PIN_SHA,\n", + ")\n", + "from slayer.client.slayer_client import SlayerClient\n", + "from slayer.storage.yaml_storage import YAMLStorage\n", + "\n", + "dbt_project = ensure_dbt_data()\n", + "print(f\"Cloned dbt project @ {DBT_PIN_SHA[:12]} and loaded ACME CSVs into DuckDB.\")" + ] + }, + { + "cell_type": "markdown", + "id": "b618f6ed", + "metadata": {}, + "source": [ + "## Step 2 — Convert dbt MetricFlow to SLayer\n", + "\n", + "This is the heart of the demo. `convert_dbt_to_slayer` parses the project's `semantic_models` and `metrics` and runs [`DbtToSlayerConverter`](../../dbt/dbt_import.md): each dbt **semantic model** becomes a SLayer model, and each dbt **metric** folds into a `ModelMeasure` formula on its source model.\n", + "\n", + "Two of those metrics drive Query 2 below:\n", + "\n", + "- `loss_payment_amount` and `loss_reserve_amount` are **simple metrics with a filter** (`has_loss_payment = 1` / `has_loss_reserve = 1`) — the converter pushes each filter down into a filtered aggregate.\n", + "- `total_loss_amount` is a **derived metric** (`loss_payment_amount + loss_reserve_amount`) — expressed as a formula over the two filtered metrics." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "3b715d90", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-01T15:51:11.599498Z", + "iopub.status.busy": "2026-07-01T15:51:11.599300Z", + "iopub.status.idle": "2026-07-01T15:51:12.033133Z", + "shell.execute_reply": "2026-07-01T15:51:12.032736Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Converted 28 SLayer models (0 metrics could not be converted).\n" + ] + } + ], + "source": [ + "result = convert_dbt_to_slayer(\n", + " dbt_project_path=dbt_project,\n", + " models_dir=MODELS_DIR,\n", + " db_path=DB_PATH,\n", + ")\n", + "print(\n", + " f\"Converted {len(result.models)} SLayer models \"\n", + " f\"({len(result.unconverted_metrics)} metrics could not be converted).\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "id": "eea04734", + "metadata": {}, + "source": [ + "## Step 3 — Query the converted models\n", + "\n", + "Point a SLayer client at the freshly converted models." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "b3a793bb", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-01T15:51:12.034957Z", + "iopub.status.busy": "2026-07-01T15:51:12.034862Z", + "iopub.status.idle": "2026-07-01T15:51:12.037017Z", + "shell.execute_reply": "2026-07-01T15:51:12.036469Z" + } + }, + "outputs": [], + "source": [ + "client = SlayerClient(storage=YAMLStorage(base_dir=str(MODELS_DIR)))" + ] + }, + { + "cell_type": "markdown", + "id": "ee42778a", + "metadata": {}, + "source": [ + "### Reference answers (gold SQL)\n", + "\n", + "The benchmark ships a gold SQL query for every question. We run them **first**, up front, and stash the expected numbers.\n", + "\n", + "Why up front? SLayer opens the DuckDB file through a read-write engine, and DuckDB won't let a second raw connection share the file under a different configuration. So all gold queries run before any SLayer query touches the file." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "c7db07ef", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-01T15:51:12.038697Z", + "iopub.status.busy": "2026-07-01T15:51:12.038531Z", + "iopub.status.idle": "2026-07-01T15:51:12.069230Z", + "shell.execute_reply": "2026-07-01T15:51:12.068858Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gold — how many claims: [{'NoOfClaims': 2}]\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
Company_Claim_NumberLossAmount
0123127012200
1123127024400
\n", + "
" + ], + "text/plain": [ + " Company_Claim_Number LossAmount\n", + "0 12312701 2200\n", + "1 12312702 4400" + ] + }, + "execution_count": 4, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "GOLD_CLAIMS_SQL = \"SELECT COUNT(*) AS NoOfClaims FROM claim\"\n", + "\n", + "GOLD_LOSS_SQL = \"\"\"\n", + "SELECT\n", + " company_claim_number,\n", + " (ca_lp.claim_amount + ca_lr.claim_amount) AS LossAmount\n", + "FROM Claim\n", + " INNER JOIN claim_amount ca_lp ON claim.claim_identifier = ca_lp.claim_identifier\n", + " INNER JOIN loss_payment ON ca_lp.claim_amount_identifier = loss_payment.claim_amount_identifier\n", + " INNER JOIN claim_amount ca_lr ON claim.claim_identifier = ca_lr.claim_identifier\n", + " INNER JOIN loss_reserve ON ca_lr.claim_amount_identifier = loss_reserve.claim_amount_identifier\n", + "ORDER BY company_claim_number\n", + "\"\"\"\n", + "\n", + "gold_claims = fetch_gold(DB_PATH, GOLD_CLAIMS_SQL)\n", + "gold_loss = fetch_gold(DB_PATH, GOLD_LOSS_SQL)\n", + "\n", + "print(\"gold — how many claims:\", gold_claims)\n", + "pd.DataFrame(gold_loss)" + ] + }, + { + "cell_type": "markdown", + "id": "8cfa7e9c", + "metadata": {}, + "source": [ + "## Query 1 — \"How many claims do we have?\"\n", + "\n", + "A plain row count. In SLayer, `*:count` is `COUNT(*)` and is always available without defining a measure." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "bdfd4fdc", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-01T15:51:12.070184Z", + "iopub.status.busy": "2026-07-01T15:51:12.070112Z", + "iopub.status.idle": "2026-07-01T15:51:12.226669Z", + "shell.execute_reply": "2026-07-01T15:51:12.226192Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SLayer: [{'claim._count': 2}]\n", + "OK — SLayer and gold agree: 2 claims\n" + ] + } + ], + "source": [ + "q1 = {\"source_model\": \"claim\", \"measures\": [\"*:count\"]}\n", + "slayer_claims = client.query_sync(q1).data\n", + "print(\"SLayer:\", slayer_claims)\n", + "\n", + "# Compare the scalar answer to gold (compare values; column names differ by design).\n", + "slayer_count = next(iter(slayer_claims[0].values()))\n", + "gold_count = next(iter(gold_claims[0].values()))\n", + "assert slayer_count == gold_count, f\"{slayer_count} != {gold_count}\"\n", + "print(f\"OK — SLayer and gold agree: {slayer_count} claims\")" + ] + }, + { + "cell_type": "markdown", + "id": "40d3af32", + "metadata": {}, + "source": [ + "## Query 2 — \"Total loss by claim number\"\n", + "\n", + "This is the interesting one. We ask for the **derived metric** `total_loss_amount` (= `loss_payment_amount + loss_reserve_amount`, each a *filtered* simple metric) grouped by claim number. `claim.company_claim_number` reaches the claim number through the join the converter inferred from the dbt entities — no manual SQL join needed.\n", + "\n", + "We rename the measure to `LossAmount` via the `name` field so the output column reads cleanly." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "e77f7f33", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-01T15:51:12.227664Z", + "iopub.status.busy": "2026-07-01T15:51:12.227565Z", + "iopub.status.idle": "2026-07-01T15:51:12.411298Z", + "shell.execute_reply": "2026-07-01T15:51:12.410865Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
claim_amount.claim.company_claim_numberclaim_amount.LossAmount
0123127012200
1123127024400
\n", + "
" + ], + "text/plain": [ + " claim_amount.claim.company_claim_number claim_amount.LossAmount\n", + "0 12312701 2200\n", + "1 12312702 4400" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "OK — SLayer and gold agree on 2 claim(s)\n" + ] + } + ], + "source": [ + "q2 = {\n", + " \"source_model\": \"claim_amount\",\n", + " \"measures\": [{\"formula\": \"total_loss_amount\", \"name\": \"LossAmount\"}],\n", + " \"dimensions\": [\"claim.company_claim_number\"],\n", + " \"order\": [{\"column\": \"claim.company_claim_number\"}],\n", + "}\n", + "slayer_loss = client.query_sync(q2).data\n", + "display(pd.DataFrame(slayer_loss))\n", + "\n", + "# Compare (claim_number, loss) pairs to gold, by value.\n", + "slayer_pairs = [(r[\"claim_amount.claim.company_claim_number\"], r[\"claim_amount.LossAmount\"]) for r in slayer_loss]\n", + "gold_pairs = [(g[\"Company_Claim_Number\"], g[\"LossAmount\"]) for g in gold_loss]\n", + "assert slayer_pairs == gold_pairs, f\"{slayer_pairs} != {gold_pairs}\"\n", + "print(f\"OK — SLayer and gold agree on {len(slayer_pairs)} claim(s)\")" + ] + }, + { + "cell_type": "markdown", + "id": "bac6fe92", + "metadata": {}, + "source": [ + "## Recap\n", + "\n", + "Starting from a dbt MetricFlow project we did not write, SLayer:\n", + "\n", + "- converted the `semantic_models` and `metrics` into queryable models,\n", + "- resolved a join from dbt entities so a measure on one model could group by a column on another,\n", + "- expressed a **derived metric over two filtered metrics** as a single formula,\n", + "\n", + "and the answers matched the benchmark's gold SQL exactly.\n", + "\n", + "### Further reading\n", + "\n", + "- [Importing dbt Semantic Layer definitions](../../dbt/dbt_import.md) — the full conversion reference.\n", + "- [SLayer vs dbt](../../dbt/slayer_vs_dbt.md) — how the two semantic layers compare.\n", + "- [dbt-labs/semantic-layer-llm-benchmarking](https://github.com/dbt-labs/semantic-layer-llm-benchmarking) — the source dbt project.\n", + "\n", + "The same converted models also back an LLM benchmark, where a model generates these SLayer queries from the natural-language questions instead of us hand-writing them." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/examples/11_dbt_metricflow/setup_metricflow.py b/docs/examples/11_dbt_metricflow/setup_metricflow.py new file mode 100644 index 00000000..5cc1f47a --- /dev/null +++ b/docs/examples/11_dbt_metricflow/setup_metricflow.py @@ -0,0 +1,329 @@ +"""Setup helper for the dbt MetricFlow -> SLayer demo notebook. + +Self-bootstrapping: clones the dbt-labs ACME Insurance benchmark project at a +**pinned commit**, loads its CSV data into a local DuckDB file, and converts the +dbt MetricFlow definitions (``semantic_models`` + ``metrics``) into SLayer models +via :class:`~slayer.dbt.converter.DbtToSlayerConverter`. Everything generated +lives under a gitignored ``.cache/`` next to this file, so nothing generated is +committed and re-runs are instant. + +The dbt project is pinned to an exact commit (``DBT_PIN_SHA``) rather than a +branch tip: the upstream branch is mutable, and the notebook asserts hard-coded +gold numbers, so a moving branch could silently invalidate them. Fetching the +pinned SHA keeps the demo reproducible regardless of upstream movement. + +Gold-query helper note: SLayer opens the DuckDB file through SQLAlchemy with a +read-write engine that DuckDB will not let a second raw connection share under a +different configuration. So :func:`fetch_gold` must be called **before** any +SLayer query touches the file — the notebook precomputes all gold answers up +front for exactly this reason. + +Returns from :func:`ensure_metricflow_demo`: ``(client, db_path, result)``. +""" + +import logging +import shutil +import subprocess +import time +from pathlib import Path +from typing import List + +import duckdb + +from slayer.async_utils import run_sync +from slayer.client.slayer_client import SlayerClient +from slayer.core.models import DatasourceConfig +from slayer.dbt.converter import ConversionResult, DbtToSlayerConverter +from slayer.dbt.parser import parse_dbt_project +from slayer.storage.yaml_storage import YAMLStorage + +logger = logging.getLogger(__name__) + +DBT_REPO_URL = "https://github.com/dbt-labs/semantic-layer-llm-benchmarking.git" +# Pinned commit on the ``refresh-2025-additional-models`` branch (the branch that +# ships the 3 bridge models + the derived/filtered MetricFlow metrics this demo +# showcases). Update this SHA *and* the notebook's gold numbers together if the +# upstream data ever changes. +DBT_PIN_SHA = "e4bdee5baeaa9b0ecb8345315c4adfffbeb2f0d1" +DATASOURCE_NAME = "acme_duckdb" + +_THIS_DIR = Path(__file__).resolve().parent +CACHE_DIR = _THIS_DIR / ".cache" +DBT_CHECKOUT = CACHE_DIR / "semantic-layer-llm-benchmarking" +DB_PATH = CACHE_DIR / "acme.duckdb" +MODELS_DIR = CACHE_DIR / "slayer_models" +_COMPLETE_MARKER = CACHE_DIR / ".complete" +_CSV_SUBDIR = "ACME_Insurance/data" + + +class MetricFlowDemoError(RuntimeError): + """Raised when the demo cannot bootstrap (e.g. clone failed). Recognizable + so the integration-test skip guard can distinguish a network failure from a + genuine conversion bug.""" + + +# Substrings (matched case-insensitively against git's stderr) that mark a fetch +# failure as a transient network/server hiccup — a 5xx/429 from GitHub, DNS, or a +# dropped connection — rather than a deterministic error (bad SHA, auth). These +# are worth retrying, and worth skipping (not failing) the integration test on. +_TRANSIENT_GIT_SIGNATURES = ( + "the requested url returned error: 500", + "the requested url returned error: 502", + "the requested url returned error: 503", + "the requested url returned error: 504", + "the requested url returned error: 429", + "error: 429", + "could not resolve host", + "failed to connect", + "connection reset", + "connection timed out", + "timed out", + "empty reply from server", + "recv failure", + "send failure", + "gnutls_handshake", + "ssl_read", + "early eof", + "rpc failed", + "remote end hung up", + "unexpectedly closed", +) + +# Retry knobs for the shallow fetch. Small, bounded backoff: enough to ride out a +# brief GitHub blip without slowing CI when the very first attempt succeeds. +_FETCH_RETRIES = 3 +_FETCH_BACKOFF_SECONDS = 2.0 + + +def _is_transient_git_error(text: str) -> bool: + """True if ``text`` (a git stderr string) looks like a transient network or + server failure rather than a deterministic one. Used both to decide whether a + failed fetch is worth retrying and whether the integration test should skip + instead of fail.""" + lowered = text.lower() + return any(sig in lowered for sig in _TRANSIENT_GIT_SIGNATURES) + + +def _git(*args: str, cwd: Path) -> str: + """Run a git command, returning stripped stdout. Raises on failure.""" + result = subprocess.run( + ["git", *args], + cwd=str(cwd), + capture_output=True, + text=True, + check=True, + ) + return result.stdout.strip() + + +def _fetch_pinned_commit(tmp: Path) -> None: + """Shallow-fetch the pinned SHA into ``tmp``, retrying on transient network + errors (GitHub 5xx/429, DNS, dropped connections). Deterministic failures + (bad SHA, missing ``git``) raise on the first attempt.""" + for attempt in range(1, _FETCH_RETRIES + 1): + try: + _git("fetch", "--depth", "1", "origin", DBT_PIN_SHA, cwd=tmp) + return + except subprocess.CalledProcessError as exc: + stderr = exc.stderr or "" + if attempt < _FETCH_RETRIES and _is_transient_git_error(stderr): + logger.warning( + "Transient git fetch failure (attempt %d/%d), retrying: %s", + attempt, + _FETCH_RETRIES, + stderr.strip(), + ) + time.sleep(_FETCH_BACKOFF_SECONDS * attempt) + continue + raise + + +def _checkout_is_valid(checkout: Path) -> bool: + """True iff ``checkout`` is a git repo whose HEAD is the pinned commit.""" + if not (checkout / ".git").exists(): + return False + try: + return _git("rev-parse", "HEAD", cwd=checkout) == DBT_PIN_SHA + except (subprocess.CalledProcessError, OSError): + # OSError covers a missing `git` binary; treat as "not valid" so the + # caller falls through to the clone path, which raises the recognized + # MetricFlowDemoError instead of a raw FileNotFoundError. + return False + + +def clone_dbt_project() -> Path: + """Fetch the dbt project at the pinned commit into ``DBT_CHECKOUT``. + + Idempotent: if a valid checkout (HEAD == pinned SHA) already exists, it is + reused. Otherwise the project is fetched into a temp dir and atomically + renamed into place, so a partial/failed clone never leaves a directory that + later looks like a usable cache. + """ + if _checkout_is_valid(DBT_CHECKOUT): + logger.info("Reusing cached dbt checkout at %s", DBT_CHECKOUT) + return DBT_CHECKOUT + + # Drop any stale/partial checkout before re-fetching. + if DBT_CHECKOUT.exists(): + shutil.rmtree(DBT_CHECKOUT) + + CACHE_DIR.mkdir(parents=True, exist_ok=True) + tmp = CACHE_DIR / "_clone_tmp" + if tmp.exists(): + shutil.rmtree(tmp) + tmp.mkdir(parents=True) + + try: + # Shallow-fetch the exact pinned commit. GitHub allows fetching an + # unadvertised SHA, so we never depend on the branch tip. The fetch is + # retried on transient network failures (GitHub 5xx/429, DNS, resets). + _git("init", "-q", cwd=tmp) + _git("remote", "add", "origin", DBT_REPO_URL, cwd=tmp) + _fetch_pinned_commit(tmp) + _git("checkout", "-q", "FETCH_HEAD", cwd=tmp) + head = _git("rev-parse", "HEAD", cwd=tmp) + if head != DBT_PIN_SHA: + raise MetricFlowDemoError( + f"Fetched commit {head} != pinned {DBT_PIN_SHA}; update the pin." + ) + except (subprocess.CalledProcessError, OSError) as exc: + # OSError covers a missing `git` binary (FileNotFoundError); both map to + # the structured demo error the notebook / test skip-guard recognise. + shutil.rmtree(tmp, ignore_errors=True) + detail = getattr(exc, "stderr", None) or exc + raise MetricFlowDemoError( + f"Failed to clone {DBT_REPO_URL} @ {DBT_PIN_SHA}: {detail}" + ) from exc + + tmp.rename(DBT_CHECKOUT) + logger.info("Cloned dbt project @ %s into %s", DBT_PIN_SHA, DBT_CHECKOUT) + return DBT_CHECKOUT + + +def load_csvs_into_duckdb(csv_dir: Path, db_path: Path) -> List[str]: + """Load every ACME Insurance CSV into its own DuckDB table. + + Table name = CSV filename stem (e.g. ``Claim.csv`` -> ``Claim``). Overwrites + any existing database file. Returns the created table names. + """ + if db_path.exists(): + db_path.unlink() + + conn = duckdb.connect(str(db_path)) + try: + csv_files = sorted(csv_dir.glob("*.csv")) + if not csv_files: + raise MetricFlowDemoError(f"No CSV files found in {csv_dir}") + for csv_file in csv_files: + # Bind the path as a parameter so an apostrophe in the checkout path + # can't break the SQL string. + conn.execute( + f'CREATE TABLE "{csv_file.stem}" AS ' + "SELECT * FROM read_csv_auto(?, header=true)", + [str(csv_file)], + ) + # FireClaim.Premium is all-NULL in the CSV, so DuckDB infers VARCHAR; + # coerce to DOUBLE so the numeric measure converts/queries correctly. + tables = {row[0] for row in conn.execute("SHOW TABLES").fetchall()} + if "FireClaim" in tables: + conn.execute("ALTER TABLE FireClaim ALTER Premium TYPE DOUBLE") + return sorted(tables) + finally: + conn.close() + + +def convert_dbt_to_slayer( + dbt_project_path: Path, models_dir: Path, db_path: Path +) -> ConversionResult: + """Convert the dbt MetricFlow project into SLayer models and persist them. + + Parses ``/models`` for ``semantic_models`` + ``metrics``, + runs :class:`DbtToSlayerConverter`, and saves each model plus a DuckDB + datasource config into a fresh ``YAMLStorage`` rooted at ``models_dir``. + """ + # Quieten the converter's benign "foreign entity '…' has no matching primary + # entity" notices so the notebook output stays focused on the demo. + logging.getLogger("slayer.dbt").setLevel(logging.ERROR) + + if models_dir.exists(): + shutil.rmtree(models_dir) + + project = parse_dbt_project(str(dbt_project_path / "models")) + result = DbtToSlayerConverter( + project=project, data_source=DATASOURCE_NAME + ).convert() + + storage = YAMLStorage(base_dir=str(models_dir)) + for model in result.models: + run_sync(storage.save_model(model)) + run_sync( + storage.save_datasource( + DatasourceConfig( + name=DATASOURCE_NAME, + type="duckdb", + database=str(db_path.resolve()), + ) + ) + ) + return result + + +def fetch_gold(db_path: Path, sql: str) -> List[dict]: + """Run a raw gold SQL query against the DuckDB file and return rows as dicts. + + MUST be called before any SLayer query opens ``db_path``: SLayer holds a + read-write engine on the file that a second raw connection cannot share, so + the notebook precomputes all gold answers up front. + """ + conn = duckdb.connect(str(db_path), read_only=True) + try: + cur = conn.execute(sql) + columns = [c[0] for c in cur.description] + return [dict(zip(columns, row)) for row in cur.fetchall()] + finally: + conn.close() + + +def ensure_dbt_data() -> Path: + """Clone the pinned dbt project and load its CSVs into DuckDB; return the + dbt checkout path. + + Idempotent: a completed prior run (valid checkout + completeness marker + + DuckDB file) is reused without touching the network. A partial prior run is + rebuilt from scratch. The dbt -> SLayer conversion is a **separate**, + always-run step (:func:`convert_dbt_to_slayer`) so the notebook can show it + as its own explicit cell. + """ + reuse = ( + _COMPLETE_MARKER.exists() + and _checkout_is_valid(DBT_CHECKOUT) + and DB_PATH.exists() + ) + if reuse: + logger.info("Reusing cached dbt checkout + DuckDB under %s", CACHE_DIR) + return DBT_CHECKOUT + + if _COMPLETE_MARKER.exists(): + _COMPLETE_MARKER.unlink() + dbt_path = clone_dbt_project() + csv_dir = dbt_path / _CSV_SUBDIR + if not csv_dir.exists(): + raise MetricFlowDemoError(f"CSV data dir not found: {csv_dir}") + load_csvs_into_duckdb(csv_dir=csv_dir, db_path=DB_PATH) + _COMPLETE_MARKER.touch() + return dbt_path + + +def ensure_metricflow_demo() -> "tuple[SlayerClient, Path, ConversionResult]": + """One-shot convenience: ensure data, convert, and build a ready client. + + Returns ``(client, db_path, result)``. Notebooks that want to show the + conversion as an explicit step call :func:`ensure_dbt_data` and + :func:`convert_dbt_to_slayer` separately instead. + """ + dbt_path = ensure_dbt_data() + result = convert_dbt_to_slayer( + dbt_project_path=dbt_path, models_dir=MODELS_DIR, db_path=DB_PATH + ) + client = SlayerClient(storage=YAMLStorage(base_dir=str(MODELS_DIR))) + return client, DB_PATH, result diff --git a/docs/examples/12_query_cache/query_cache_nb.ipynb b/docs/examples/12_query_cache/query_cache_nb.ipynb new file mode 100644 index 00000000..85a191ce --- /dev/null +++ b/docs/examples/12_query_cache/query_cache_nb.ipynb @@ -0,0 +1,405 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c26ac79e", + "metadata": {}, + "source": [ + "# Query Cache — Mechanics\n", + "\n", + "SLayer can cache query results **in memory, per `SlayerQueryEngine` instance**, opt-in per call\n", + "via `cache=True`. It is modelled on [Cube's in-memory cache](https://cube.dev/docs/product/caching):\n", + "a cache hit skips the database round-trip entirely, and staleness is governed by an optional\n", + "**time-to-live** and an optional set of Cube-style **refresh keys** that you scan explicitly with\n", + "`engine.refresh()`.\n", + "\n", + "This notebook builds a tiny SQLite shop from scratch (so we can freely mutate it) and walks through:\n", + "\n", + "1. opt-in caching (miss → hit),\n", + "2. how a hit serves the *last* result even after the data changes,\n", + "3. refresh keys invalidating on a real change,\n", + "4. TTL time-based expiry,\n", + "5. managing the cache (`evict` / `clear_cache` / `cache_size`).\n", + "\n", + "> The cache is a **Python-API-only** feature — there is no REST / MCP / CLI surface. See the\n", + "> [Query Cache concept doc](../../concepts/query-cache.md) for the full reference." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "e19fe2dd", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-06T09:14:42.184444Z", + "iopub.status.busy": "2026-07-06T09:14:42.184004Z", + "iopub.status.idle": "2026-07-06T09:14:42.410281Z", + "shell.execute_reply": "2026-07-06T09:14:42.409881Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "workspace: /tmp/tmp8r4a7ds1\n" + ] + } + ], + "source": [ + "import sqlite3\n", + "import tempfile\n", + "import time\n", + "from pathlib import Path\n", + "\n", + "from slayer.async_utils import run_sync\n", + "from slayer.core.enums import DataType\n", + "from slayer.core.models import Column, DatasourceConfig, SlayerModel\n", + "from slayer.engine.cache import CacheConfig\n", + "from slayer.engine.query_engine import SlayerQueryEngine\n", + "from slayer.storage.yaml_storage import YAMLStorage\n", + "\n", + "# A throwaway workspace: a SQLite \"shop\" database + a YAML model store.\n", + "work = Path(tempfile.mkdtemp())\n", + "db_path = work / \"shop.db\"\n", + "\n", + "conn = sqlite3.connect(db_path)\n", + "conn.executescript(\n", + " \"\"\"\n", + " CREATE TABLE orders (\n", + " id INTEGER PRIMARY KEY,\n", + " status TEXT,\n", + " amount REAL,\n", + " updated_at TEXT\n", + " );\n", + " INSERT INTO orders VALUES\n", + " (1, 'completed', 100.0, '2026-01-01'),\n", + " (2, 'pending', 50.0, '2026-01-02'),\n", + " (3, 'completed', 200.0, '2026-01-03');\n", + " \"\"\"\n", + ")\n", + "conn.commit()\n", + "conn.close()\n", + "\n", + "def mutate(sql: str) -> None:\n", + " \"\"\"Apply a write directly to the underlying table (simulating an ETL job).\"\"\"\n", + " c = sqlite3.connect(db_path)\n", + " c.execute(sql)\n", + " c.commit()\n", + " c.close()\n", + "\n", + "# Register the datasource + a simple `orders` model. Storage methods are async;\n", + "# run_sync bridges them for a notebook/script.\n", + "storage = YAMLStorage(base_dir=str(work / \"store\"))\n", + "run_sync(storage.save_datasource(\n", + " DatasourceConfig(name=\"shop\", type=\"sqlite\", database=str(db_path))\n", + "))\n", + "run_sync(storage.save_model(SlayerModel(\n", + " name=\"orders\",\n", + " sql_table=\"orders\",\n", + " data_source=\"shop\",\n", + " columns=[\n", + " Column(name=\"id\", sql=\"id\", type=DataType.INT, primary_key=True),\n", + " Column(name=\"status\", sql=\"status\", type=DataType.TEXT),\n", + " Column(name=\"amount\", sql=\"amount\", type=DataType.DOUBLE),\n", + " Column(name=\"updated_at\", sql=\"updated_at\", type=DataType.TIMESTAMP),\n", + " ],\n", + ")))\n", + "print(\"workspace:\", work)" + ] + }, + { + "cell_type": "markdown", + "id": "97b80f1f", + "metadata": {}, + "source": [ + "## 1. Opt-in caching\n", + "\n", + "Build an engine with a `CacheConfig`. We give it two **refresh keys** on the `orders` table —\n", + "`MAX(updated_at)` and `COUNT(*)` — which we'll use in step 3. For now there is no TTL, so entries\n", + "only change when we refresh, evict, or clear them.\n", + "\n", + "The query total is `100 + 50 + 200 = 350`." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "155ff8eb", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-06T09:14:42.411363Z", + "iopub.status.busy": "2026-07-06T09:14:42.411283Z", + "iopub.status.idle": "2026-07-06T09:14:42.433873Z", + "shell.execute_reply": "2026-07-06T09:14:42.433429Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "first : [{'orders.amount_sum': 350.0}]\n", + "second: [{'orders.amount_sum': 350.0}]\n", + "cache_size: 1\n" + ] + } + ], + "source": [ + "engine = SlayerQueryEngine(\n", + " storage=storage,\n", + " cache_config=CacheConfig(\n", + " ttl_seconds=None, # no time-based expiry (yet — see step 4)\n", + " refresh_keys=[\n", + " (\"orders\", \"MAX(updated_at)\"),\n", + " (\"orders\", \"COUNT(*)\"),\n", + " ],\n", + " ),\n", + ")\n", + "\n", + "query = {\"source_model\": \"orders\", \"measures\": [\"amount:sum\"]}\n", + "\n", + "first = engine.execute_sync(query, cache=True) # miss: runs SQL, stores the result\n", + "second = engine.execute_sync(query, cache=True) # hit: served from memory, no DB round-trip\n", + "\n", + "print(\"first :\", first.data)\n", + "print(\"second:\", second.data)\n", + "print(\"cache_size:\", engine.cache_size)" + ] + }, + { + "cell_type": "markdown", + "id": "6ac61215", + "metadata": {}, + "source": [ + "## 2. A hit serves the *last* result\n", + "\n", + "Now an ETL job inserts a new completed order worth `1000`. The true total becomes `1350`.\n", + "\n", + "A `cache=True` call still returns the cached `350` — that is the whole point of a cache. Passing\n", + "`cache=False` bypasses it and shows the fresh value." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "7f3bb82e", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-06T09:14:42.434840Z", + "iopub.status.busy": "2026-07-06T09:14:42.434766Z", + "iopub.status.idle": "2026-07-06T09:14:42.444105Z", + "shell.execute_reply": "2026-07-06T09:14:42.443706Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "cached (cache=True) : [{'orders.amount_sum': 350.0}]\n", + "bypass (cache=False): [{'orders.amount_sum': 1350.0}]\n" + ] + } + ], + "source": [ + "mutate(\"INSERT INTO orders VALUES (4, 'completed', 1000.0, '2026-02-01')\")\n", + "\n", + "print(\"cached (cache=True) :\", engine.execute_sync(query, cache=True).data)\n", + "print(\"bypass (cache=False):\", engine.execute_sync(query, cache=False).data)" + ] + }, + { + "cell_type": "markdown", + "id": "63ea2707", + "metadata": {}, + "source": [ + "## 3. Refresh keys — invalidate on a real change\n", + "\n", + "`engine.refresh()` scans each entry's applicable refresh keys and re-executes the entry when a\n", + "value has **moved**. Our insert bumped both `MAX(updated_at)` and `COUNT(*)`, so the entry is\n", + "re-run and lands in the `refreshed` bucket. Afterwards the cached value reflects the new total.\n", + "\n", + "Refresh-key sensitivity is your choice of expression: `MAX(updated_at)` catches new-timestamped\n", + "inserts; a `COUNT(*)` key additionally catches **deletes** that leave the max untouched." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "95ed7b8b", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-06T09:14:42.444950Z", + "iopub.status.busy": "2026-07-06T09:14:42.444878Z", + "iopub.status.idle": "2026-07-06T09:14:42.456445Z", + "shell.execute_reply": "2026-07-06T09:14:42.456051Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "refreshed : 1\n", + "expired_refreshed: 0\n", + "unchanged : 0\n", + "errors : []\n", + "cached after refresh: [{'orders.amount_sum': 1350.0}]\n" + ] + } + ], + "source": [ + "result = engine.refresh_sync()\n", + "print(\"refreshed :\", len(result.refreshed))\n", + "print(\"expired_refreshed:\", len(result.expired_refreshed))\n", + "print(\"unchanged :\", len(result.unchanged))\n", + "print(\"errors :\", result.errors)\n", + "print(\"cached after refresh:\", engine.execute_sync(query, cache=True).data)" + ] + }, + { + "cell_type": "markdown", + "id": "392412dc", + "metadata": {}, + "source": [ + "## 4. TTL — time-based expiry\n", + "\n", + "A `ttl_seconds` bounds an entry's wall-clock age. It is checked **lazily on read**: once an entry\n", + "is older than the TTL, the next `cache=True` read treats it as a miss and re-executes. Here we use\n", + "a 1-second TTL and sleep past it." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "1d2c33a3", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-06T09:14:42.457440Z", + "iopub.status.busy": "2026-07-06T09:14:42.457364Z", + "iopub.status.idle": "2026-07-06T09:14:43.572061Z", + "shell.execute_reply": "2026-07-06T09:14:43.571673Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "populate: [{'orders.amount_sum': 1350.0}]\n", + "within TTL (stale): [{'orders.amount_sum': 1350.0}]\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "after TTL (fresh): [{'orders.amount_sum': 1375.0}]\n" + ] + } + ], + "source": [ + "ttl_engine = SlayerQueryEngine(\n", + " storage=storage,\n", + " cache_config=CacheConfig(ttl_seconds=1.0),\n", + ")\n", + "\n", + "print(\"populate:\", ttl_engine.execute_sync(query, cache=True).data)\n", + "mutate(\"INSERT INTO orders VALUES (5, 'pending', 25.0, '2026-02-02')\")\n", + "\n", + "print(\"within TTL (stale):\", ttl_engine.execute_sync(query, cache=True).data)\n", + "time.sleep(1.1) # let the entry age past ttl_seconds\n", + "print(\"after TTL (fresh):\", ttl_engine.execute_sync(query, cache=True).data)" + ] + }, + { + "cell_type": "markdown", + "id": "f0db73e4", + "metadata": {}, + "source": [ + "## 5. Managing the cache\n", + "\n", + "- `evict(query)` removes one entry (recomputing its key without touching the database).\n", + "- `clear_cache()` drops everything.\n", + "- `cache_size` reports the live entry count." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "32060a6c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-06T09:14:43.573536Z", + "iopub.status.busy": "2026-07-06T09:14:43.573426Z", + "iopub.status.idle": "2026-07-06T09:14:43.585072Z", + "shell.execute_reply": "2026-07-06T09:14:43.584751Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "size before evict: 1\n", + "evict returned : True\n", + "size after evict : 0\n", + "size after re-cache: 1\n", + "size after clear : 0\n" + ] + } + ], + "source": [ + "print(\"size before evict:\", engine.cache_size)\n", + "print(\"evict returned :\", engine.evict_sync(query))\n", + "print(\"size after evict :\", engine.cache_size)\n", + "\n", + "engine.execute_sync(query, cache=True)\n", + "print(\"size after re-cache:\", engine.cache_size)\n", + "engine.clear_cache()\n", + "print(\"size after clear :\", engine.cache_size)" + ] + }, + { + "cell_type": "markdown", + "id": "9aa9bd9a", + "metadata": {}, + "source": [ + "## Recap\n", + "\n", + "- `execute(query, cache=True)` is opt-in and per-call; `dry_run` / `explain` are never cached.\n", + "- The cache is **per engine instance** — two engines (e.g. different tenants / connection settings)\n", + " never share cached rows.\n", + "- Staleness has two independent signals: **TTL** (lazy on read) and **refresh keys** (scanned by\n", + " `engine.refresh()`).\n", + "- Everything shown here has a `*_sync` wrapper (`execute_sync`, `refresh_sync`, `evict_sync`) and an\n", + " `async` twin.\n", + "\n", + "For the full reference — cache key, table detection, the refresh-key trade-off, and non-goals —\n", + "see the [Query Cache concept doc](../../concepts/query-cache.md)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/examples/13_osi_import/osi_import.md b/docs/examples/13_osi_import/osi_import.md new file mode 100644 index 00000000..ce59deb5 --- /dev/null +++ b/docs/examples/13_osi_import/osi_import.md @@ -0,0 +1,72 @@ +# From OSI to SLayer + +SLayer ingests an [OSI](https://open-semantic-interchange.org/) (Open Semantic Interchange) config — its `datasets`, `relationships`, and `metrics` — and turns it into queryable SLayer models, filling in column types from live introspection of your database. Here is the whole path, from install to querying it through an agent. + +## 1. Install SLayer + +With [uv](https://docs.astral.sh/uv/): + +```bash +uv tool install 'motley-slayer[advanced_search]' +``` + +DuckDB and SQLite work out of the box; for other databases add the driver extra (e.g. `motley-slayer[postgres,advanced_search]`). + +## 2. Point SLayer at a storage folder + +SLayer keeps its datasources and models in one folder. Set it once, and every command uses it: + +```bash +export SLAYER_STORAGE=~/slayer-data +``` + +## 3. Register the datasource + +OSI carries no column types, so the importer reads them live — the datasource has to exist first: + +```bash +slayer datasources create duckdb:///path/to/shop.duckdb --name shop_osi +``` + +## 4. Import the OSI config + +```bash +slayer import-osi shop.osi.yaml --datasource shop_osi +``` + +Each dataset becomes a model, each relationship a join, each metric a measure. See [Importing OSI configs](../../osi/osi_import.md) for exactly what converts and what fails cleanly. + +## 5. Connect it to Claude Code + +Register SLayer as an MCP server; Claude Code spawns it on demand and calls its tools: + +```bash +claude mcp add slayer -- slayer mcp --storage "$SLAYER_STORAGE" +``` + +Now ask your agent to explore — it calls `models_summary`, `inspect`, `search`, and `query` against your imported models. + +## Try it — the notebooks + +Two self-contained, **fully offline** notebooks build a tiny retail DuckDB and run the whole flow end to end: + +- [`osi_import_nb.ipynb`](osi_import_nb.ipynb) — the **library** path: import with `OsiToSlayerConverter`, query with the `SlayerClient`. +- [`osi_import_agent_nb.ipynb`](osi_import_agent_nb.ipynb) — the **agent** path: the CLI commands above, then the MCP tools an agent calls. + +Both import [`shop.osi.yaml`](shop.osi.yaml) and check every answer against gold SQL. + +## What the import produces + +Each OSI **dataset** becomes a SLayer model, with real column *types* from live introspection of the datasource (OSI carries no type hints). Each **relationship** becomes a LEFT join, and each **metric** folds into a `ModelMeasure` formula on the model the converter picks as its anchor. OSI `ai_context` and `custom_extensions` are carried onto the SLayer entities as descriptions and `meta`, so an agent reading the model sees them. + +## The five queries + +| Question | SLayer query | OSI feature shown | +|----------|--------------|-------------------| +| Total order value | `total_amount` | simple metric (`SUM(amount)` → `amount:sum`) | +| Total by region | `total_amount` by `customers.regions.name` | multi-hop join inferred from relationships | +| Average order value | `aov` | derived metric (`SUM(amount) / COUNT(*)`) | +| Amount per distinct customer | `cust_reach` | cross-dataset metric through a join | +| Amount plus region population | `rev_plus_pop` | multi-hop metric with sub-query isolation | + +The region grouping and the last two metrics reach across joins the converter inferred from the OSI relationships — no manual SQL join is written. Every answer matches the gold SQL exactly. diff --git a/docs/examples/13_osi_import/osi_import_agent_nb.ipynb b/docs/examples/13_osi_import/osi_import_agent_nb.ipynb new file mode 100644 index 00000000..163fc1aa --- /dev/null +++ b/docs/examples/13_osi_import/osi_import_agent_nb.ipynb @@ -0,0 +1,584 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "d02e034d", + "metadata": {}, + "source": [ + "# OSI Import, the agent way (CLI + MCP)\n", + "\n", + "**TL;DR:** the [companion notebook](osi_import_nb.ipynb) imported an OSI config with the Python library. This one walks the *same* demo the way an **AI agent wired to SLayer** experiences it — ingest the config with two **command-line** commands, then explore and query the result through SLayer's **MCP tools** (the exact tool calls Claude would make).\n", + "\n", + "Steps (everything lands in a gitignored `.cache/`):\n", + "\n", + "1. **Build the data & reference answers** — the same tiny retail DuckDB; gold SQL computed up front.\n", + "2. **Ingest via the CLI** — `slayer datasources create`, then `slayer import-osi`.\n", + "3. **Connect the MCP server** and explore — `models_summary`, `inspect`, `search`.\n", + "4. **Query via the MCP `query` tool** — verified against gold.\n", + "\n", + "Fully offline — no network needed." + ] + }, + { + "cell_type": "markdown", + "id": "1f06c3e5", + "metadata": {}, + "source": [ + "## Configure SLayer as an MCP server in Claude\n", + "\n", + "In real use you register SLayer once with your agent; Claude then spawns it on demand and calls its tools. With [uv](https://docs.astral.sh/uv/) installed:\n", + "\n", + "```bash\n", + "claude mcp add slayer -- uvx --from 'motley-slayer[advanced_search]' slayer mcp --storage /path/to/store\n", + "```\n", + "\n", + "or, after a permanent install (`uv tool install 'motley-slayer[advanced_search]'`):\n", + "\n", + "```bash\n", + "claude mcp add slayer -- slayer mcp --storage /path/to/store\n", + "```\n", + "\n", + "Point `--storage` at the folder holding your datasource + models — here, this demo's `.cache/slayer_models`. Agents that take a JSON config use the equivalent:\n", + "\n", + "```json\n", + "{\n", + " \"mcpServers\": {\n", + " \"slayer\": {\n", + " \"command\": \"uvx\",\n", + " \"args\": [\"--from\", \"motley-slayer[advanced_search]\", \"slayer\", \"mcp\", \"--storage\", \"/path/to/store\"]\n", + " }\n", + " }\n", + "}\n", + "```\n", + "\n", + "`slayer mcp` speaks JSON-RPC over **stdio**, so there's no HTTP URL to curl. This notebook drives the same tools **in-process** via `create_mcp_server(...).call_tool(name, arguments)` — the identical `tools/call` request an stdio client issues — so you can watch each call run. Full guide: [MCP Setup — AI Agents](../../getting-started/mcp.md)." + ] + }, + { + "cell_type": "markdown", + "id": "c6524889", + "metadata": {}, + "source": [ + "## Step 1 — Build the data & reference answers\n", + "\n", + "The same retail DuckDB as the companion notebook. We compute the gold answers **first**, before the importer or the MCP engine opens the file (DuckDB won't share a read-write file across connections). The gold SQL lives in `setup_osi.compute_gold`, shared by both notebooks." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "a1d3faf8", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:36:09.438291Z", + "iopub.status.busy": "2026-07-15T12:36:09.438225Z", + "iopub.status.idle": "2026-07-15T12:36:09.923922Z", + "shell.execute_reply": "2026-07-15T12:36:09.923503Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "reference answers: {'total': 1400.0, 'aov': 233.33333333333334, 'cust_reach': 466.6666666666667, 'rev_plus_pop': 4400.0}\n" + ] + } + ], + "source": [ + "import json\n", + "import os\n", + "import warnings\n", + "import shutil\n", + "import subprocess\n", + "import sys\n", + "\n", + "import pandas as pd\n", + "\n", + "# FastMCP emits a benign PydanticJsonSchemaWarning when it builds the\n", + "# tool JSON schemas; silence it so the tool listing stays clean.\n", + "warnings.filterwarnings(\"ignore\", message=\".*not JSON serializable.*\")\n", + "\n", + "# The setup helper lives next to this notebook.\n", + "sys.path.insert(0, os.getcwd())\n", + "\n", + "from setup_osi import (\n", + " build_shop_duckdb,\n", + " compute_gold,\n", + " OSI_CONFIG,\n", + " DB_PATH,\n", + " MODELS_DIR,\n", + " DATASOURCE_NAME,\n", + ")\n", + "\n", + "build_shop_duckdb(DB_PATH)\n", + "GOLD = compute_gold(DB_PATH)\n", + "print(\"reference answers:\", {k: GOLD[k] for k in (\"total\", \"aov\", \"cust_reach\", \"rev_plus_pop\")})" + ] + }, + { + "cell_type": "markdown", + "id": "401c0198", + "metadata": {}, + "source": [ + "## Step 2 — Ingest the OSI config with the CLI\n", + "\n", + "Two commands — exactly what you'd type in a shell (or an agent would run for you):\n", + "\n", + "1. **Register the datasource.** `import-osi` reads real column *types* by introspecting the database live, so the datasource has to exist first — this is the separate command that must run before the import.\n", + "2. **Import the OSI config** into that datasource.\n", + "\n", + "We point both at this demo's storage folder with the `SLAYER_STORAGE` environment variable — the same folder the MCP server reads in Step 3." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "2381ccfa", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:36:09.925014Z", + "iopub.status.busy": "2026-07-15T12:36:09.924872Z", + "iopub.status.idle": "2026-07-15T12:36:10.990754Z", + "shell.execute_reply": "2026-07-15T12:36:10.990456Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "$ slayer datasources create duckdb:///.cache/shop.duckdb --name shop_osi -y\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Created datasource 'shop_osi' (duckdb).\n", + "\n", + "$ slayer import-osi shop.osi.yaml --datasource shop_osi\n" + ] + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Imported model: orders (8 columns, 7 measures)\n", + "Imported model: customers (4 columns, 0 measures)\n", + "Imported model: products (3 columns, 0 measures)\n", + "Imported model: regions (3 columns, 0 measures)\n", + "\n", + "Done: 4 models, 0 unconverted, 0 dropped\n", + "\n" + ] + } + ], + "source": [ + "env = {**os.environ, \"SLAYER_STORAGE\": str(MODELS_DIR)}\n", + "shutil.rmtree(MODELS_DIR, ignore_errors=True) # start from a clean store\n", + "\n", + "\n", + "def slayer(*args):\n", + " \"\"\"Run a `slayer` CLI command against the demo storage; echo it like a shell.\"\"\"\n", + " print(\"$ slayer\", *args)\n", + " result = subprocess.run([\"slayer\", *map(str, args)], capture_output=True, text=True, env=env)\n", + " print(result.stdout, end=\"\")\n", + " if result.returncode != 0:\n", + " print(result.stderr)\n", + " raise RuntimeError(f\"slayer exited {result.returncode}\")\n", + " print()\n", + "\n", + "\n", + "slayer(\"datasources\", \"create\", f\"duckdb:///{os.path.relpath(DB_PATH)}\", \"--name\", DATASOURCE_NAME, \"-y\")\n", + "slayer(\"import-osi\", os.path.relpath(OSI_CONFIG), \"--datasource\", DATASOURCE_NAME)" + ] + }, + { + "cell_type": "markdown", + "id": "ff74d0b9", + "metadata": {}, + "source": [ + "## Step 3 — Connect the MCP server\n", + "\n", + "The models exist now. We spin up SLayer's MCP server **in-process** against the same storage the CLI just wrote, and list its tools — the same set Claude sees over stdio." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "d27f8c72", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:36:10.991576Z", + "iopub.status.busy": "2026-07-15T12:36:10.991497Z", + "iopub.status.idle": "2026-07-15T12:36:15.583476Z", + "shell.execute_reply": "2026-07-15T12:36:15.583210Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SLayer MCP exposes 21 tools:\n", + "query, query_nested, models_summary, inspect_model, inspect, create_model, edit_model, create_datasource, list_datasources, describe_datasource, edit_datasource, delete_model, validate_models, recommend_root_model, delete_datasource, ingest_datasource_models, set_datasource_priority, get_datasource_priority, save_memory, forget_memory, search\n" + ] + } + ], + "source": [ + "from slayer.mcp.server import create_mcp_server\n", + "from slayer.storage.yaml_storage import YAMLStorage\n", + "\n", + "mcp = create_mcp_server(storage=YAMLStorage(base_dir=str(MODELS_DIR)))\n", + "tools = await mcp.list_tools()\n", + "print(f\"SLayer MCP exposes {len(tools)} tools:\")\n", + "print(\", \".join(t.name for t in tools))\n", + "\n", + "\n", + "async def call(name, **arguments):\n", + " \"\"\"Invoke an MCP tool and return its text response (what an agent receives).\"\"\"\n", + " content, _ = await mcp.call_tool(name=name, arguments=arguments)\n", + " return content[0].text" + ] + }, + { + "cell_type": "markdown", + "id": "4390385e", + "metadata": {}, + "source": [ + "## Step 4 — Explore like an agent\n", + "\n", + "Before querying, an agent orients itself: `models_summary` for the lay of the land, then `inspect` to drill into a model — and a single column, where the OSI `ai_context` shows up as descriptions and synonyms." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "16aa0c9c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:36:15.584636Z", + "iopub.status.busy": "2026-07-15T12:36:15.584549Z", + "iopub.status.idle": "2026-07-15T12:36:15.593693Z", + "shell.execute_reply": "2026-07-15T12:36:15.593471Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "# Datasource: `shop_osi` — 4 model(s)\n", + "\n", + "## `customers`\n", + "Columns: 4\n", + "Measures: \n", + "Joins to: `regions`\n", + "\n", + "## `orders`\n", + "Order line items\n", + "One row per order.\n", + "Synonyms: sales, purchases\n", + "Columns: 7\n", + "Measures: total_amount, order_count, aov, revenue_line, cust_reach, rev_plus_pop, bridge_metric\n", + "Joins to: `customers`, `products`\n", + "\n", + "## `products`\n", + "Columns: 3\n", + "Measures: \n", + "Joins to: _(none)_\n", + "\n", + "## `regions`\n", + "Columns: 3\n", + "Measures: \n", + "Joins to: _(none)_\n" + ] + } + ], + "source": [ + "print(await call(\"models_summary\", datasource_name=DATASOURCE_NAME))" + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "57c7444a", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:36:15.594623Z", + "iopub.status.busy": "2026-07-15T12:36:15.594532Z", + "iopub.status.idle": "2026-07-15T12:36:15.605871Z", + "shell.execute_reply": "2026-07-15T12:36:15.605619Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "# `orders`\n", + "Order line items\n", + "One row per order.\n", + "Synonyms: sales, purchases\n", + "Columns: order_id, customer_id, product_id, amount, quantity, ordered_at, status\n", + "Measures: total_amount, order_count, aov, revenue_line, cust_reach, rev_plus_pop, bridge_metric\n", + "Aggregations: _(none)_\n", + "Joins to: customers, products\n" + ] + } + ], + "source": [ + "print(await call(\"inspect\", reference=\"orders\", entity_type=\"model\"))" + ] + }, + { + "cell_type": "markdown", + "id": "5cdc21ef", + "metadata": {}, + "source": [ + "The OSI `ai_context` and `custom_extensions` rode along into the imported model. Drill into the `amount` column to see them — description, synonyms, label, and a live sample range:" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "b447be07", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:36:15.606869Z", + "iopub.status.busy": "2026-07-15T12:36:15.606799Z", + "iopub.status.idle": "2026-07-15T12:36:15.684084Z", + "shell.execute_reply": "2026-07-15T12:36:15.683565Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Column: shop_osi.orders.amount\n", + "Type: DOUBLE\n", + "Description: Gross order value in USD.\n", + "Synonyms: revenue, gross\n", + "Label: Order amount\n", + "Format: type= precision=None symbol=None\n", + "SQL: amount\n", + "Sample values: 100.0 .. 400.0\n" + ] + } + ], + "source": [ + "print(await call(\"inspect\", reference=\"orders.amount\", entity_type=\"column\", compact=False))" + ] + }, + { + "cell_type": "markdown", + "id": "7eee0e12", + "metadata": {}, + "source": [ + "### Semantic discovery with `search`\n", + "\n", + "An agent that doesn't know the measure name can `search` for it. Scoping to the datasource keeps hits on this shop. (The embedding channel is off here — no API key — so this runs on BM25 + full-text alone, which still finds it.)" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "fc0ba435", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:36:15.685577Z", + "iopub.status.busy": "2026-07-15T12:36:15.685456Z", + "iopub.status.idle": "2026-07-15T12:36:15.865785Z", + "shell.execute_reply": "2026-07-15T12:36:15.865116Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + " measure shop_osi.orders.aov (score 0.016)\n", + " column shop_osi.orders.amount (score 0.016)\n", + " measure shop_osi.orders.total_amount (score 0.016)\n", + " column shop_osi.orders.ordered_at (score 0.016)\n", + " measure shop_osi.orders.order_count (score 0.015)\n", + "\n", + "OK — search surfaced the `aov` measure\n" + ] + } + ], + "source": [ + "hits = json.loads(await call(\"search\", question=\"average order value\", datasource=DATASOURCE_NAME, max_results=5))\n", + "for h in hits[\"results\"]:\n", + " print(f\" {h['kind']:<8} {h.get('id') or h.get('canonical_id')} (score {h['score']:.3f})\")\n", + "\n", + "found = [h.get(\"id\") or h.get(\"canonical_id\") for h in hits[\"results\"]]\n", + "assert any(str(x).endswith(\"aov\") for x in found), found\n", + "print(\"\\nOK — search surfaced the `aov` measure\")" + ] + }, + { + "cell_type": "markdown", + "id": "0b986d41", + "metadata": {}, + "source": [ + "## Step 5 — Query via the MCP `query` tool\n", + "\n", + "Now the agent issues `query` calls — the same JSON-RPC an LLM emits — and we check each answer against the gold values from Step 1. `format=\"json\"` returns the rows followed by a human-readable attributes block; `raw_decode` reads just the leading JSON." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "d1cc284c", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:36:15.866990Z", + "iopub.status.busy": "2026-07-15T12:36:15.866881Z", + "iopub.status.idle": "2026-07-15T12:36:15.974241Z", + "shell.execute_reply": "2026-07-15T12:36:15.973405Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "total_amount: [{'orders.total_amount': 1400.0}] gold: 1400.0\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
orders.customers.regions.nameorders.total_amount
0North750.0
1South650.0
\n", + "
" + ], + "text/plain": [ + " orders.customers.regions.name orders.total_amount\n", + "0 North 750.0\n", + "1 South 650.0" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "cust_reach: [{'orders.cust_reach': 466.6666666666667}] gold: 466.6666666666667\n", + "\n", + "OK — every MCP query matches gold\n" + ] + } + ], + "source": [ + "async def query_rows(**arguments):\n", + " text = await call(\"query\", format=\"json\", **arguments)\n", + " rows, _ = json.JSONDecoder().raw_decode(text.strip())\n", + " return rows\n", + "\n", + "\n", + "# Q1 — a simple metric\n", + "rows = await query_rows(source_model=\"orders\", measures=[{\"formula\": \"total_amount\"}])\n", + "print(\"total_amount:\", rows, \" gold:\", GOLD[\"total\"])\n", + "assert abs(rows[0][\"orders.total_amount\"] - GOLD[\"total\"]) < 1e-9\n", + "\n", + "# Q2 — grouped by a multi-hop joined dimension\n", + "rows = await query_rows(\n", + " source_model=\"orders\",\n", + " measures=[{\"formula\": \"total_amount\"}],\n", + " dimensions=[\"customers.regions.name\"],\n", + " order=[{\"column\": \"customers.regions.name\"}],\n", + ")\n", + "display(pd.DataFrame(rows))\n", + "pairs = [(r[\"orders.customers.regions.name\"], r[\"orders.total_amount\"]) for r in rows]\n", + "assert pairs == [(g[\"region\"], g[\"amount\"]) for g in GOLD[\"by_region\"]]\n", + "\n", + "# Q3 — a cross-dataset metric\n", + "rows = await query_rows(source_model=\"orders\", measures=[{\"formula\": \"cust_reach\"}])\n", + "print(\"cust_reach: \", rows, \" gold:\", GOLD[\"cust_reach\"])\n", + "assert abs(rows[0][\"orders.cust_reach\"] - GOLD[\"cust_reach\"]) < 1e-9\n", + "\n", + "print(\"\\nOK — every MCP query matches gold\")" + ] + }, + { + "cell_type": "markdown", + "id": "141a9a6a", + "metadata": {}, + "source": [ + "## Recap\n", + "\n", + "The same OSI import as the [library notebook](osi_import_nb.ipynb), driven the way an agent lives it:\n", + "\n", + "- **two CLI commands** — `slayer datasources create` (the datasource the importer introspects) then `slayer import-osi`,\n", + "- **MCP tools** — `models_summary` / `inspect` / `search` to explore (OSI `ai_context` surfacing as descriptions + synonyms), then `query` to answer questions,\n", + "- every answer checked against the shared gold SQL in `setup_osi.compute_gold`.\n", + "\n", + "### Further reading\n", + "\n", + "- [MCP Setup — AI Agents](../../getting-started/mcp.md) — registering SLayer with Claude Code and other agents.\n", + "- [Introspecting a datasource via MCP](../08_mcp_introspect/mcp_introspect_nb.ipynb) — a deeper tour of the introspection tools.\n", + "- [Importing OSI configs](../../osi/osi_import.md) — the full conversion reference." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/examples/13_osi_import/osi_import_nb.ipynb b/docs/examples/13_osi_import/osi_import_nb.ipynb new file mode 100644 index 00000000..bab9065b --- /dev/null +++ b/docs/examples/13_osi_import/osi_import_nb.ipynb @@ -0,0 +1,625 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "id": "c1974b60", + "metadata": {}, + "source": [ + "# From OSI to SLayer\n", + "\n", + "**TL;DR:** SLayer can ingest an [OSI](https://open-semantic-interchange.org/) (Open Semantic Interchange) config — its `datasets`, `relationships`, and `metrics` — and turn it into queryable SLayer models. This notebook imports a small retail OSI config end-to-end and answers five questions, checking each answer against gold SQL.\n", + "\n", + "It runs as four explicit steps (everything lands in a gitignored `.cache/` next to this notebook):\n", + "\n", + "1. **Build the data** — create a tiny retail DuckDB (orders, customers, products, regions) with deterministic rows.\n", + "2. **Reference answers** — run the gold SQL *up front*, before the importer touches the file.\n", + "3. **Import** — convert `shop.osi.yaml` into SLayer models with `OsiToSlayerConverter`.\n", + "4. **Query** — point a SLayer client at the imported models and verify against gold.\n", + "\n", + "> Unlike the [dbt MetricFlow demo](../11_dbt_metricflow/dbt_metricflow_nb.ipynb), this one is **fully offline** — no network access is needed at any point.\n", + "\n", + "The committed OSI config ([`shop.osi.yaml`](shop.osi.yaml)) exercises every OSI feature the importer understands: `ai_context` metadata, `custom_extensions`, relationships that become joins, and simple / derived / cross-dataset / multi-hop metrics." + ] + }, + { + "cell_type": "markdown", + "id": "5d3d80db", + "metadata": {}, + "source": [ + "## Step 1 — Build the demo database\n", + "\n", + "Create the retail DuckDB the OSI config binds to. OSI carries no column *types* — the importer reads them by **live introspection** of this database — so it has to exist first." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "id": "ee1f7f14", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:30.314738Z", + "iopub.status.busy": "2026-07-15T12:34:30.314611Z", + "iopub.status.idle": "2026-07-15T12:34:30.770478Z", + "shell.execute_reply": "2026-07-15T12:34:30.770093Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Built retail DuckDB 'shop.duckdb' with orders, customers, products, regions.\n" + ] + } + ], + "source": [ + "import os\n", + "import sys\n", + "\n", + "import pandas as pd\n", + "\n", + "# The setup helper lives next to this notebook.\n", + "sys.path.insert(0, os.getcwd())\n", + "\n", + "from setup_osi import (\n", + " build_shop_duckdb,\n", + " convert_osi_to_slayer,\n", + " compute_gold,\n", + " OSI_CONFIG,\n", + " DB_PATH,\n", + " MODELS_DIR,\n", + " DATASOURCE_NAME,\n", + ")\n", + "from slayer.async_utils import run_sync\n", + "from slayer.client.slayer_client import SlayerClient\n", + "from slayer.storage.yaml_storage import YAMLStorage\n", + "\n", + "db_path = build_shop_duckdb(DB_PATH)\n", + "print(f\"Built retail DuckDB '{db_path.name}' with orders, customers, products, regions.\")" + ] + }, + { + "cell_type": "markdown", + "id": "ea6aa32d", + "metadata": {}, + "source": [ + "## Step 2 — Reference answers (gold SQL)\n", + "\n", + "We run the gold SQL **first**, up front, and stash the expected numbers.\n", + "\n", + "Why up front? The importer introspects the datasource live, opening a read-write SQLAlchemy engine on the DuckDB file — and DuckDB won't let a second raw connection share the file under a different configuration. So every gold query runs *before* the import (Step 3) touches the file." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "id": "b0c5e635", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:30.771593Z", + "iopub.status.busy": "2026-07-15T12:34:30.771434Z", + "iopub.status.idle": "2026-07-15T12:34:30.812558Z", + "shell.execute_reply": "2026-07-15T12:34:30.812208Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "gold total amount : 1400.0\n", + "gold aov : 233.33333333333334\n", + "gold cust_reach : 466.6666666666667\n", + "gold rev_plus_pop : 4400.0\n" + ] + }, + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
regionamount
0North750.0
1South650.0
\n", + "
" + ], + "text/plain": [ + " region amount\n", + "0 North 750.0\n", + "1 South 650.0" + ] + }, + "execution_count": 2, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "# The reference answers live in setup_osi.compute_gold — shared with the\n", + "# agent-workflow notebook so both demos check against identical numbers.\n", + "GOLD = compute_gold(DB_PATH)\n", + "\n", + "print(\"gold total amount :\", GOLD[\"total\"])\n", + "print(\"gold aov :\", GOLD[\"aov\"])\n", + "print(\"gold cust_reach :\", GOLD[\"cust_reach\"])\n", + "print(\"gold rev_plus_pop :\", GOLD[\"rev_plus_pop\"])\n", + "pd.DataFrame(GOLD[\"by_region\"])" + ] + }, + { + "cell_type": "markdown", + "id": "7e047a82", + "metadata": {}, + "source": [ + "## Step 3 — Import the OSI config\n", + "\n", + "This is the heart of the demo. `convert_osi_to_slayer` registers the DuckDB as a SLayer datasource, parses `shop.osi.yaml`, and runs [`OsiToSlayerConverter`](../../osi/osi_import.md):\n", + "\n", + "- each OSI **dataset** becomes a SLayer model (column *types* from live introspection),\n", + "- each **relationship** becomes a LEFT join,\n", + "- each **metric** folds into a `ModelMeasure` formula on the model the converter picks as its anchor.\n", + "\n", + "The converter reports anything it can't express cleanly — here, everything converts." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "id": "a70cbdf6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:30.813594Z", + "iopub.status.busy": "2026-07-15T12:34:30.813511Z", + "iopub.status.idle": "2026-07-15T12:34:30.997351Z", + "shell.execute_reply": "2026-07-15T12:34:30.996967Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Imported models: ['orders', 'customers', 'products', 'regions']\n", + "0 metrics unconverted, 0 items dropped\n", + "\n", + "No conversion issues.\n" + ] + } + ], + "source": [ + "result = convert_osi_to_slayer(OSI_CONFIG, MODELS_DIR, DB_PATH)\n", + "\n", + "print(\"Imported models:\", [m.name for m in result.models])\n", + "unconverted, dropped = result.tally()\n", + "print(f\"{unconverted} metrics unconverted, {dropped} items dropped\")\n", + "print()\n", + "print(result.render_report())" + ] + }, + { + "cell_type": "markdown", + "id": "2f8114b3", + "metadata": {}, + "source": [ + "## Step 4 — The imported metadata survived\n", + "\n", + "OSI's `ai_context` and `custom_extensions` don't just decorate the source file — the importer carries them onto the SLayer entities so agents can read them. Below, the `orders.amount` column keeps its label, its `ai_context` instructions as a description, and both blocks in `meta`; the `orders -> customers` join keeps its relationship `ai_context` as a description." + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "id": "d04aac72", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:30.998576Z", + "iopub.status.busy": "2026-07-15T12:34:30.998500Z", + "iopub.status.idle": "2026-07-15T12:34:31.007527Z", + "shell.execute_reply": "2026-07-15T12:34:31.007153Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "orders.amount.label : Order amount\n", + "orders.amount.description : Gross order value in USD.\n", + "Synonyms: revenue, gross\n", + "orders.amount.meta : {'osi_ai_context': {'instructions': 'Gross order value in USD.', 'synonyms': ['revenue', 'gross']}, 'osi_custom_extensions': [{'vendor_name': 'SNOWFLAKE', 'data': '{\"unit\": \"usd\"}'}]}\n", + "\n", + "join orders -> customers: 'Each order has one customer.'\n", + "join orders -> products: None\n", + "\n", + "measures on orders: ['total_amount', 'order_count', 'aov', 'revenue_line', 'cust_reach', 'rev_plus_pop', 'bridge_metric']\n" + ] + } + ], + "source": [ + "storage = YAMLStorage(base_dir=str(MODELS_DIR))\n", + "orders = run_sync(storage.get_model(\"orders\", data_source=DATASOURCE_NAME))\n", + "\n", + "amount = next(c for c in orders.columns if c.name == \"amount\")\n", + "print(\"orders.amount.label :\", amount.label)\n", + "print(\"orders.amount.description :\", amount.description)\n", + "print(\"orders.amount.meta :\", amount.meta)\n", + "print()\n", + "for j in orders.joins:\n", + " print(f\"join orders -> {j.target_model}: {j.description!r}\")\n", + "print()\n", + "print(\"measures on orders:\", [m.name for m in orders.measures])" + ] + }, + { + "cell_type": "markdown", + "id": "a2ae1298", + "metadata": {}, + "source": [ + "## Query the imported models\n", + "\n", + "Point a SLayer client at the freshly imported models. Every query below is plain SLayer JSON — the OSI origin is now invisible." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "id": "46cc2d91", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:31.008671Z", + "iopub.status.busy": "2026-07-15T12:34:31.008592Z", + "iopub.status.idle": "2026-07-15T12:34:31.010818Z", + "shell.execute_reply": "2026-07-15T12:34:31.010445Z" + } + }, + "outputs": [], + "source": [ + "client = SlayerClient(storage=YAMLStorage(base_dir=str(MODELS_DIR)))\n", + "\n", + "\n", + "def approx(a, b, tol=1e-9):\n", + " return abs(a - b) < tol\n", + "\n", + "\n", + "def show(got, exp):\n", + " \"\"\"Print a SLayer scalar answer next to its gold value.\"\"\"\n", + " print(\"SLayer:\", got, \" gold:\", exp)" + ] + }, + { + "cell_type": "markdown", + "id": "bfaf8fea", + "metadata": {}, + "source": [ + "### Query 1 — total order value\n", + "\n", + "A simple metric: `total_amount` is `SUM(amount)`, imported from the OSI `SUM(amount)` expression as the SLayer formula `amount:sum`." + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "id": "3b478ce6", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:31.011818Z", + "iopub.status.busy": "2026-07-15T12:34:31.011737Z", + "iopub.status.idle": "2026-07-15T12:34:31.044635Z", + "shell.execute_reply": "2026-07-15T12:34:31.044206Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SLayer: [{'orders.total_amount': 1400.0}]\n", + "OK — SLayer and gold agree: 1400.0\n" + ] + } + ], + "source": [ + "q1 = {\"source_model\": \"orders\", \"measures\": [\"total_amount\"]}\n", + "rows = client.query_sync(q1).data\n", + "print(\"SLayer:\", rows)\n", + "\n", + "assert approx(rows[0][\"orders.total_amount\"], GOLD[\"total\"])\n", + "print(f\"OK — SLayer and gold agree: {GOLD['total']}\")" + ] + }, + { + "cell_type": "markdown", + "id": "68a026f7", + "metadata": {}, + "source": [ + "### Query 2 — total by region (multi-hop join)\n", + "\n", + "`total_amount` grouped by `customers.regions.name` walks two joins the converter inferred from OSI relationships — `orders -> customers -> regions` — with no SQL written by hand." + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "id": "041ba315", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:31.045637Z", + "iopub.status.busy": "2026-07-15T12:34:31.045564Z", + "iopub.status.idle": "2026-07-15T12:34:31.075211Z", + "shell.execute_reply": "2026-07-15T12:34:31.074939Z" + } + }, + "outputs": [ + { + "data": { + "text/html": [ + "
\n", + "\n", + "\n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + " \n", + "
orders.customers.regions.nameorders.total_amount
0North750.0
1South650.0
\n", + "
" + ], + "text/plain": [ + " orders.customers.regions.name orders.total_amount\n", + "0 North 750.0\n", + "1 South 650.0" + ] + }, + "metadata": {}, + "output_type": "display_data" + }, + { + "name": "stdout", + "output_type": "stream", + "text": [ + "OK — SLayer and gold agree on 2 region(s)\n" + ] + } + ], + "source": [ + "q2 = {\n", + " \"source_model\": \"orders\",\n", + " \"measures\": [\"total_amount\"],\n", + " \"dimensions\": [\"customers.regions.name\"],\n", + " \"order\": [{\"column\": \"customers.regions.name\"}],\n", + "}\n", + "rows = client.query_sync(q2).data\n", + "display(pd.DataFrame(rows))\n", + "\n", + "slayer_pairs = [(r[\"orders.customers.regions.name\"], r[\"orders.total_amount\"]) for r in rows]\n", + "gold_pairs = [(g[\"region\"], g[\"amount\"]) for g in GOLD[\"by_region\"]]\n", + "assert slayer_pairs == gold_pairs, f\"{slayer_pairs} != {gold_pairs}\"\n", + "print(f\"OK — SLayer and gold agree on {len(slayer_pairs)} region(s)\")" + ] + }, + { + "cell_type": "markdown", + "id": "4c6f2751", + "metadata": {}, + "source": [ + "### Query 3 — average order value (derived metric)\n", + "\n", + "`aov` came from the OSI expression `(SUM(amount)) / (COUNT(*))` — a **derived** metric over two aggregates, imported as the formula `amount:sum / *:count`." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "id": "9e8675ff", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:31.076455Z", + "iopub.status.busy": "2026-07-15T12:34:31.076379Z", + "iopub.status.idle": "2026-07-15T12:34:31.130919Z", + "shell.execute_reply": "2026-07-15T12:34:31.130564Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SLayer: 233.33333333333334 gold: 233.33333333333334\n", + "OK — average order value matches gold\n" + ] + } + ], + "source": [ + "q3 = {\"source_model\": \"orders\", \"measures\": [\"aov\"]}\n", + "rows = client.query_sync(q3).data\n", + "show(rows[0][\"orders.aov\"], GOLD[\"aov\"])\n", + "\n", + "assert approx(rows[0][\"orders.aov\"], GOLD[\"aov\"])\n", + "print(\"OK — average order value matches gold\")" + ] + }, + { + "cell_type": "markdown", + "id": "f577eed9", + "metadata": {}, + "source": [ + "### Query 4 — amount per distinct customer (cross-dataset metric)\n", + "\n", + "`cust_reach` = `SUM(amount) / COUNT(DISTINCT customers.customer_id)`. The metric references a column on **another** dataset (`customers`); the converter anchored it on `orders` and emits the cross-model reference through the inferred join." + ] + }, + { + "cell_type": "code", + "execution_count": 9, + "id": "cc971b57", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:31.131913Z", + "iopub.status.busy": "2026-07-15T12:34:31.131835Z", + "iopub.status.idle": "2026-07-15T12:34:31.179876Z", + "shell.execute_reply": "2026-07-15T12:34:31.179575Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SLayer: 466.6666666666667 gold: 466.6666666666667\n", + "OK — amount per distinct customer matches gold\n" + ] + } + ], + "source": [ + "q4 = {\"source_model\": \"orders\", \"measures\": [\"cust_reach\"]}\n", + "rows = client.query_sync(q4).data\n", + "show(rows[0][\"orders.cust_reach\"], GOLD[\"cust_reach\"])\n", + "\n", + "assert approx(rows[0][\"orders.cust_reach\"], GOLD[\"cust_reach\"])\n", + "print(\"OK — amount per distinct customer matches gold\")" + ] + }, + { + "cell_type": "markdown", + "id": "736ffab2", + "metadata": {}, + "source": [ + "### Query 5 — amount plus region population (multi-hop metric)\n", + "\n", + "`rev_plus_pop` = `SUM(amount) + SUM(regions.population)` reaches `regions` two joins away (`orders -> customers -> regions`).\n", + "\n", + "The subtle part: SLayer aggregates `regions.population` **at the regions grain** (each distinct region counted once), so adding this measure can't multiply the order rows. A naive five-way join would instead add a region's population *once per order* in that region and badly over-count. Our gold mirrors SLayer's isolation — summing population over the distinct regions the orders reach — and the two agree." + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "id": "81c2a213", + "metadata": { + "execution": { + "iopub.execute_input": "2026-07-15T12:34:31.180890Z", + "iopub.status.busy": "2026-07-15T12:34:31.180813Z", + "iopub.status.idle": "2026-07-15T12:34:31.231345Z", + "shell.execute_reply": "2026-07-15T12:34:31.230982Z" + } + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "SLayer: 4400.0 gold: 4400.0\n", + "OK — SLayer isolates the joined aggregate; both agree at 4400.0\n" + ] + } + ], + "source": [ + "q5 = {\"source_model\": \"orders\", \"measures\": [\"rev_plus_pop\"]}\n", + "rows = client.query_sync(q5).data\n", + "show(rows[0][\"orders.rev_plus_pop\"], GOLD[\"rev_plus_pop\"])\n", + "\n", + "assert approx(rows[0][\"orders.rev_plus_pop\"], GOLD[\"rev_plus_pop\"])\n", + "print(\"OK — SLayer isolates the joined aggregate; both agree at\", GOLD[\"rev_plus_pop\"])" + ] + }, + { + "cell_type": "markdown", + "id": "0db0c3d7", + "metadata": {}, + "source": [ + "## Recap\n", + "\n", + "Starting from an OSI config we did not write, SLayer:\n", + "\n", + "- imported each `dataset` as a queryable model, with real column types from live introspection,\n", + "- turned OSI `relationships` into joins, so measures and dimensions reach across them (including multi-hop),\n", + "- folded simple, derived, cross-dataset, and multi-hop `metrics` into `ModelMeasure` formulas,\n", + "- and preserved `ai_context` / `custom_extensions` as descriptions and `meta` for agents to read.\n", + "\n", + "Every answer matched gold SQL — including the multi-hop measure, where SLayer's sub-query isolation kept the added aggregate from fanning out the order rows.\n", + "\n", + "### Further reading\n", + "\n", + "- [Importing OSI configs](../../osi/osi_import.md) — the full conversion reference, including exactly what converts and what fails cleanly.\n", + "- The one-liner this notebook mirrors: `slayer import-osi shop.osi.yaml --datasource shop_osi` (against a registered datasource)." + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.3" + } + }, + "nbformat": 4, + "nbformat_minor": 5 +} diff --git a/docs/examples/13_osi_import/setup_osi.py b/docs/examples/13_osi_import/setup_osi.py new file mode 100644 index 00000000..19ad9bc2 --- /dev/null +++ b/docs/examples/13_osi_import/setup_osi.py @@ -0,0 +1,225 @@ +"""Setup helper for the OSI -> SLayer demo notebook. + +Self-contained and **fully offline** (no network, unlike the dbt MetricFlow +demo's git clone): builds a tiny retail DuckDB with deterministic rows, registers +it as a SLayer datasource, and converts the committed ``shop.osi.yaml`` OSI +config into queryable SLayer models via +:class:`~slayer.osi.converter.OsiToSlayerConverter`. + +Everything generated lives under a gitignored ``.cache/`` next to this file, so +nothing generated is committed and re-runs rebuild it cheaply. + +The OSI importer takes column *types* from **live introspection** of the +datasource (OSI carries no type hints), so the datasource must be saved and +reachable *before* :func:`convert_osi_to_slayer` runs. + +Gold-query helper note: SLayer opens the DuckDB file through SQLAlchemy with a +read-write engine that DuckDB will not let a second raw connection share under a +different configuration. So :func:`fetch_gold` must be called **before** any +SLayer query touches the file — the notebook precomputes all gold answers up +front for exactly this reason. + +Returns from :func:`ensure_osi_demo`: ``(client, db_path, result)``. +""" + +import logging +import shutil +from pathlib import Path +from typing import List + +import duckdb + +from slayer.async_utils import run_sync +from slayer.client.slayer_client import SlayerClient +from slayer.core.models import DatasourceConfig +from slayer.ingest_report import ConversionResult +from slayer.osi.converter import OsiToSlayerConverter +from slayer.osi.parser import parse_osi_path +from slayer.sql import engine_factory +from slayer.storage.yaml_storage import YAMLStorage + +logger = logging.getLogger(__name__) + +DATASOURCE_NAME = "shop_osi" + +_THIS_DIR = Path(__file__).resolve().parent +OSI_CONFIG = _THIS_DIR / "shop.osi.yaml" +CACHE_DIR = _THIS_DIR / ".cache" +DB_PATH = CACHE_DIR / "shop.duckdb" +MODELS_DIR = CACHE_DIR / "slayer_models" + +# DuckDB DDL for the four tables the OSI config binds to. Mirrors the schema in +# tests/test_cli_import_osi.py, with DuckDB-native types (DOUBLE / VARCHAR). +_SCHEMA = [ + "CREATE TABLE orders (order_id INTEGER PRIMARY KEY, customer_id INTEGER, " + "product_id INTEGER, amount DOUBLE, quantity INTEGER, ordered_at DATE, status VARCHAR)", + "CREATE TABLE customers (customer_id INTEGER PRIMARY KEY, region_id INTEGER, " + "name VARCHAR, segment VARCHAR)", + "CREATE TABLE products (product_id INTEGER PRIMARY KEY, category VARCHAR, price DOUBLE)", + "CREATE TABLE regions (region_id INTEGER PRIMARY KEY, name VARCHAR, population INTEGER)", +] + +# Deterministic rows so every gold number below is exact. +_REGIONS = [ + (1, "North", 1000), + (2, "South", 2000), +] +_CUSTOMERS = [ + (1, 1, "Alice", "consumer"), + (2, 1, "Bob", "business"), + (3, 2, "Carol", "consumer"), +] +_PRODUCTS = [ + (1, "Beverages", 5.0), + (2, "Bakery", 3.0), +] +_ORDERS = [ + # order_id, customer_id, product_id, amount, quantity, ordered_at, status + (1, 1, 1, 100.0, 2, "2024-01-01", "completed"), + (2, 1, 2, 200.0, 1, "2024-01-05", "completed"), + (3, 2, 1, 300.0, 3, "2024-02-01", "completed"), + (4, 2, 2, 150.0, 1, "2024-02-10", "pending"), + (5, 3, 1, 250.0, 5, "2024-03-01", "completed"), + (6, 3, 2, 400.0, 2, "2024-03-15", "completed"), +] + + +def build_shop_duckdb(db_path: Path = DB_PATH) -> Path: + """Create the retail DuckDB (four tables + deterministic rows). + + Overwrites any existing file so a re-run always starts from a clean, known + dataset. Returns the database path. + """ + db_path.parent.mkdir(parents=True, exist_ok=True) + if db_path.exists(): + db_path.unlink() + + conn = duckdb.connect(str(db_path)) + try: + for ddl in _SCHEMA: + conn.execute(ddl) + conn.executemany("INSERT INTO regions VALUES (?, ?, ?)", _REGIONS) + conn.executemany("INSERT INTO customers VALUES (?, ?, ?, ?)", _CUSTOMERS) + conn.executemany("INSERT INTO products VALUES (?, ?, ?)", _PRODUCTS) + conn.executemany( + "INSERT INTO orders VALUES (?, ?, ?, ?, ?, ?, ?)", _ORDERS + ) + finally: + conn.close() + logger.info("Built retail DuckDB at %s", db_path) + return db_path + + +def convert_osi_to_slayer( + osi_path: Path = OSI_CONFIG, + models_dir: Path = MODELS_DIR, + db_path: Path = DB_PATH, +) -> ConversionResult: + """Convert the OSI config into SLayer models and persist them. + + Saves a DuckDB ``DatasourceConfig`` into a fresh ``YAMLStorage`` rooted at + ``models_dir`` (the importer introspects it live for column types), parses + the OSI documents, runs :class:`OsiToSlayerConverter`, and saves each model. + """ + # Quieten the converter's benign clean-fail notices so the notebook output + # stays focused on the demo. + logging.getLogger("slayer.osi").setLevel(logging.ERROR) + + if models_dir.exists(): + shutil.rmtree(models_dir) + + storage = YAMLStorage(base_dir=str(models_dir)) + ds = DatasourceConfig( + name=DATASOURCE_NAME, type="duckdb", database=str(db_path.resolve()) + ) + run_sync(storage.save_datasource(ds)) + + documents = parse_osi_path(osi_path) + sa_engine = engine_factory.get_engine(ds.resolve_env_vars()) + result = OsiToSlayerConverter( + documents=documents, + data_source=DATASOURCE_NAME, + sa_engine=sa_engine, + dialect="ANSI_SQL", + target_dialect="duckdb", + ).convert() + + for model in result.models: + run_sync(storage.save_model(model)) + return result + + +def fetch_gold(db_path: Path, sql: str) -> List[dict]: + """Run a raw gold SQL query against the DuckDB file and return rows as dicts. + + MUST be called before any SLayer query opens ``db_path``: SLayer holds a + read-write engine on the file that a second raw connection cannot share, so + the notebook precomputes all gold answers up front. + """ + conn = duckdb.connect(str(db_path), read_only=True) + try: + cur = conn.execute(sql) + columns = [c[0] for c in cur.description] + return [dict(zip(columns, row)) for row in cur.fetchall()] + finally: + conn.close() + + +# --- Reference ("gold") answers, shared by both demo notebooks --------------- +# Hand-written SQL whose results are the trusted numbers every SLayer query in +# the notebooks is checked against. `rev_plus_pop` mirrors SLayer's sub-query +# isolation: the joined `regions.population` is summed at the *regions* grain +# (distinct regions the orders reach), not once per order. + +_GOLD_BY_REGION_SQL = """ + SELECT r.name AS region, SUM(o.amount) AS amount + FROM orders o + JOIN customers c ON o.customer_id = c.customer_id + JOIN regions r ON c.region_id = r.region_id + GROUP BY r.name + ORDER BY r.name +""" + +_GOLD_REV_PLUS_POP_SQL = """ + WITH reached AS ( + SELECT DISTINCT c.region_id + FROM orders o JOIN customers c ON o.customer_id = c.customer_id + ) + SELECT (SELECT SUM(amount) FROM orders) + + (SELECT SUM(population) FROM regions + WHERE region_id IN (SELECT region_id FROM reached)) AS value +""" + + +def compute_gold(db_path: Path = DB_PATH) -> dict: + """Compute all reference answers up front (before SLayer opens ``db_path``). + + Returns a dict with keys ``total``, ``by_region``, ``aov``, ``cust_reach``, + ``rev_plus_pop`` — the expected values both notebooks assert their SLayer / + MCP query results against. + """ + return { + "total": fetch_gold(db_path, "SELECT SUM(amount) AS v FROM orders")[0]["v"], + "by_region": fetch_gold(db_path, _GOLD_BY_REGION_SQL), + "aov": fetch_gold( + db_path, "SELECT SUM(amount) * 1.0 / COUNT(*) AS v FROM orders" + )[0]["v"], + "cust_reach": fetch_gold( + db_path, + "SELECT SUM(amount) * 1.0 / COUNT(DISTINCT customer_id) AS v FROM orders", + )[0]["v"], + "rev_plus_pop": fetch_gold(db_path, _GOLD_REV_PLUS_POP_SQL)[0]["value"], + } + + +def ensure_osi_demo() -> "tuple[SlayerClient, Path, ConversionResult]": + """One-shot convenience: build data, convert OSI, and return a ready client. + + Returns ``(client, db_path, result)``. Notebooks that want to show the build + and conversion as explicit steps call :func:`build_shop_duckdb` and + :func:`convert_osi_to_slayer` separately instead. + """ + build_shop_duckdb(DB_PATH) + result = convert_osi_to_slayer(OSI_CONFIG, MODELS_DIR, DB_PATH) + client = SlayerClient(storage=YAMLStorage(base_dir=str(MODELS_DIR))) + return client, DB_PATH, result diff --git a/docs/examples/13_osi_import/shop.osi.yaml b/docs/examples/13_osi_import/shop.osi.yaml new file mode 100644 index 00000000..ebe8c4d9 --- /dev/null +++ b/docs/examples/13_osi_import/shop.osi.yaml @@ -0,0 +1,147 @@ +# A worked OSI (Open Semantic Interchange) config for the SLayer import demo. +# +# It describes a small retail shop over four tables (orders, customers, +# products, regions) and exercises every OSI feature the importer understands: +# * ai_context on the model, fields, joins, and metrics +# * custom_extensions carried into column meta +# * relationships that become SLayer joins (incl. a multi-hop chain) +# * simple, derived, cross-dataset, and multi-hop metrics +# +# Sources are BARE table names so the importer's live introspection resolves +# them directly against the demo DuckDB the notebook builds. +version: "0.2.0.dev0" + +semantic_model: + - name: shop + description: Retail shop semantic model + ai_context: + instructions: "Use for order, customer, and product analytics." + synonyms: + - "store" + - "retail" + + datasets: + - name: orders + source: orders + primary_key: [order_id] + description: Order line items + ai_context: + instructions: "One row per order." + synonyms: + - "sales" + - "purchases" + fields: + - name: order_id + expression: {dialects: [{dialect: ANSI_SQL, expression: order_id}]} + description: Order id + - name: customer_id + expression: {dialects: [{dialect: ANSI_SQL, expression: customer_id}]} + - name: product_id + expression: {dialects: [{dialect: ANSI_SQL, expression: product_id}]} + - name: amount + expression: {dialects: [{dialect: ANSI_SQL, expression: amount}]} + label: Order amount + ai_context: + instructions: "Gross order value in USD." + synonyms: + - "revenue" + - "gross" + custom_extensions: + - vendor_name: SNOWFLAKE + data: '{"unit": "usd"}' + - name: quantity + expression: {dialects: [{dialect: ANSI_SQL, expression: quantity}]} + - name: ordered_at + expression: {dialects: [{dialect: ANSI_SQL, expression: ordered_at}]} + dimension: {is_time: true} + - name: status + expression: {dialects: [{dialect: ANSI_SQL, expression: status}]} + + - name: customers + source: customers + primary_key: [customer_id] + fields: + - name: customer_id + expression: {dialects: [{dialect: ANSI_SQL, expression: customer_id}]} + - name: region_id + expression: {dialects: [{dialect: ANSI_SQL, expression: region_id}]} + - name: name + expression: {dialects: [{dialect: ANSI_SQL, expression: name}]} + - name: segment + expression: {dialects: [{dialect: ANSI_SQL, expression: segment}]} + + - name: products + source: products + primary_key: [product_id] + fields: + - name: product_id + expression: {dialects: [{dialect: ANSI_SQL, expression: product_id}]} + - name: category + expression: {dialects: [{dialect: ANSI_SQL, expression: category}]} + - name: price + expression: {dialects: [{dialect: ANSI_SQL, expression: price}]} + + - name: regions + source: regions + primary_key: [region_id] + fields: + - name: region_id + expression: {dialects: [{dialect: ANSI_SQL, expression: region_id}]} + - name: name + expression: {dialects: [{dialect: ANSI_SQL, expression: name}]} + - name: population + expression: {dialects: [{dialect: ANSI_SQL, expression: population}]} + + relationships: + - name: orders_to_customers + from: orders + to: customers + from_columns: [customer_id] + to_columns: [customer_id] + ai_context: + instructions: "Each order has one customer." + - name: orders_to_products + from: orders + to: products + from_columns: [product_id] + to_columns: [product_id] + - name: customers_to_regions + from: customers + to: regions + from_columns: [region_id] + to_columns: [region_id] + + metrics: + - name: total_amount + description: Total order amount + ai_context: + instructions: "Sum of gross order value." + synonyms: + - "gmv" + expression: {dialects: [{dialect: ANSI_SQL, expression: SUM(amount)}]} + - name: order_count + expression: {dialects: [{dialect: ANSI_SQL, expression: COUNT(*)}]} + - name: aov + description: Average order value + expression: {dialects: [{dialect: ANSI_SQL, expression: (SUM(amount)) / (COUNT(*))}]} + - name: revenue_line + description: Quantity-weighted revenue (materialized operand) + expression: {dialects: [{dialect: ANSI_SQL, expression: SUM(quantity * amount)}]} + - name: cust_reach + description: Amount per distinct customer (cross-dataset, anchor = orders) + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(amount) / COUNT(DISTINCT customers.customer_id) + - name: rev_plus_pop + description: Amount plus region population (multi-hop orders->customers->regions) + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(amount) + SUM(regions.population) + - name: bridge_metric + description: References products + customers only; must anchor on orders (bridge) + expression: + dialects: + - dialect: ANSI_SQL + expression: SUM(products.price) + COUNT(DISTINCT customers.customer_id) diff --git a/docs/getting-started/cli.md b/docs/getting-started/cli.md index 69471c15..cd57f118 100644 --- a/docs/getting-started/cli.md +++ b/docs/getting-started/cli.md @@ -23,7 +23,13 @@ slayer datasources create demo --ingest slayer query '{"source_model": "orders", "measures": ["*:count"]}' ``` -This generates ~2 years of synthetic coffee-shop data into a local DuckDB file under your storage directory and ingests the models (`customers`, `orders`, `items`, `products`, `stores`, `supplies`, `tweets`). Re-running is idempotent — the DuckDB is reused if it already exists. Override the years with `--years N`; the default is kept small so `slayer serve --demo` / `slayer mcp --demo` finish quickly enough to fit inside MCP-client startup timeouts. Only the first four bundled stores open within the first two years — bump `--years` to 4+ if you want all six. +This generates ~2 years of synthetic coffee-shop data into a local DuckDB file under your storage directory and ingests the models (`customers`, `orders`, `items`, `products`, `stores`, `supplies`, `tweets`). The demo models come pre-enriched with a curated semantic layer: column labels and descriptions, currency/percent formats, ready-made measures (`total_revenue`, `avg_order_value`, `effective_tax_rate`, `unique_customers`, …), and a custom-aggregation example (`weighted_avg` defaulting its weight to `subtotal`) — so `models_summary` / `inspect` output is informative out of the box and there are saved measures to query by name: + +```bash +slayer query '{"source_model": "orders", "measures": [{"formula": "total_revenue"}, {"formula": "avg_order_value"}], "dimensions": ["stores.name"]}' +``` + +Re-running is idempotent — the DuckDB is reused if it already exists, and the enrichment is additive-only (your edits to labels, descriptions, or measures are never overwritten). Override the years with `--years N`; the default is kept small so `slayer serve --demo` / `slayer mcp --demo` finish quickly enough to fit inside MCP-client startup timeouts. Only the first four bundled stores open within the first two years — bump `--years` to 4+ if you want all six. Pre-populate once and `--demo` is instant afterwards: @@ -130,6 +136,37 @@ orders._count If you see "Model 'orders' not found", check that `slayer ingest` ran successfully and that `--storage` points to the right location. +## Search & memories + +`slayer search` runs semantic retrieval over memories and canonical entities (models, columns, named measures, custom aggregations). Three channels run in parallel — BM25 over memory entity tags, Tantivy full-text, and (with `motley-slayer[advanced_search]` plus a provider API key) dense embeddings — and are RRF-fused into a single ranked list. + +```bash +# Entity-driven +slayer search --entity jaffle_shop.orders.order_total + +# Question-driven +slayer search --question "What stores are in jaffle_shop?" + +# Graph-narrow with cypher_filter (naive form, always available) +slayer search --question "Brooklyn POS" \ + --cypher-filter 'MATCH (n:Memory) RETURN n.id AS id' +``` + +`--cypher-filter` accepts full openCypher when `advanced_search` is installed (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges). Without the extra, only the naive `MATCH (n:Label) RETURN n.id AS id` kind-filter form is accepted; richer Cypher raises with an install hint. Without the extra (or a provider API key) the embedding channel emits a single warning into `SearchResponse.warnings` and search degrades to BM25 + Tantivy. + +Persist a note with `slayer memory save` so the next session inherits it: + +```bash +slayer memory save \ + --learning "orders.is_returned in {0,1,NULL}; treat NULL as not returned" \ + --entities jaffle_shop.orders.is_returned \ + --id kb.returns.null-handling + +slayer memory forget kb.returns.null-handling # cascade-strips memory: refs +``` + +See [Search](../concepts/search.md), [Memories](../concepts/memories.md), and the [CLI Reference](../reference/cli.md#slayer-search) for the full signature. + ## Start a server (optional) If you also want a REST API or MCP endpoint: diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md index abed3877..3be7b99d 100644 --- a/docs/getting-started/index.md +++ b/docs/getting-started/index.md @@ -10,6 +10,7 @@ SLayer is a semantic layer that sits between your database and whatever consumes | Query from the terminal or scripts | **CLI** | [CLI Setup](cli.md) | | Build an app that queries data (any language) | **REST API** | [REST API Setup](rest-api.md) | | Use SLayer as a Python library | **Python SDK** | [Python Setup](python.md) | +| Find prior context (memories) and discover models / columns | **All four** | [Search](../concepts/search.md) · [Memories](../concepts/memories.md) | All four interfaces use the same query language and the same models — pick the one that fits your workflow. You can use multiple interfaces simultaneously (e.g., MCP for your agent + REST API for your dashboard). diff --git a/docs/getting-started/mcp.md b/docs/getting-started/mcp.md index 85e28a80..632e4521 100644 --- a/docs/getting-started/mcp.md +++ b/docs/getting-started/mcp.md @@ -13,17 +13,17 @@ Install [uv](https://docs.astral.sh/uv/getting-started/installation/) — the fa Register SLayer as an MCP server — Claude Code will spawn it automatically when needed: ```bash -claude mcp add slayer -- uvx --from 'motley-slayer[embedding_search]' slayer mcp --ingest-on-startup +claude mcp add slayer -- uvx --from 'motley-slayer[advanced_search]' slayer mcp --ingest-on-startup ``` `--ingest-on-startup` walks every configured datasource on boot and runs idempotent auto-ingestion before the MCP channel opens, so models are available on the agent's first tool call. Drop it to defer ingestion to a manual `ingest_datasource_models` call. -The `embedding_search` extra enables semantic search over models and memories. When it's installed **and** a provider API key is in the environment (`OPENAI_API_KEY` by default; override the embedding model with `SLAYER_EMBEDDING_MODEL=voyage/voyage-3` + `VOYAGE_API_KEY`, etc.), the boot-time ingest pass also refreshes per-entity embeddings — hash-skipped, so steady-state boots make zero embedding API calls. Without the extra (or without a provider key), search and ingest still work; the embedding channel is silently disabled. +The `advanced_search` extra enables semantic search over models and memories. When it's installed **and** a provider API key is in the environment (`OPENAI_API_KEY` by default; override the embedding model with `SLAYER_EMBEDDING_MODEL=voyage/voyage-3` + `VOYAGE_API_KEY`, etc.), the boot-time ingest pass also refreshes per-entity embeddings — hash-skipped, so steady-state boots make zero embedding API calls. Without the extra (or without a provider key), search and ingest still work; the embedding channel is silently disabled. For databases other than SQLite, add the driver extra alongside (see [full list](../configuration/datasources.md#database-drivers)): ```bash -claude mcp add slayer -- uvx --from 'motley-slayer[postgres,embedding_search]' slayer mcp --ingest-on-startup +claude mcp add slayer -- uvx --from 'motley-slayer[postgres,advanced_search]' slayer mcp --ingest-on-startup ``` ### Other agents (JSON config) @@ -35,7 +35,7 @@ Most MCP-compatible agents accept a JSON server configuration. Add this to your "mcpServers": { "slayer": { "command": "uvx", - "args": ["--from", "motley-slayer[postgres,embedding_search]", "slayer", "mcp", "--ingest-on-startup"], + "args": ["--from", "motley-slayer[postgres,advanced_search]", "slayer", "mcp", "--ingest-on-startup"], "env": { "OPENAI_API_KEY": "sk-..." } @@ -44,7 +44,7 @@ Most MCP-compatible agents accept a JSON server configuration. Add this to your } ``` -Replace `postgres` with your database driver, or use `motley-slayer[all]` for all supported databases (every driver plus `embedding_search`). +Replace `postgres` with your database driver, or use `motley-slayer[all]` for all supported databases (every driver plus `advanced_search`). ### Remote / shared server @@ -99,11 +99,30 @@ The agent should call `list_datasources` and then `models_summary(datasource_nam 2. Models have been ingested (via `slayer mcp --ingest-on-startup`, `ingest_datasource_models`, or `create_datasource` with auto-ingest) 3. Environment variables referenced in the datasource config are set +## Search & memories + +Once a datasource is ingested, the agent has access to two extra tools beyond `query`: + +- `search` — three-channel retrieval (BM25 over memory tags + Tantivy full-text + optional dense embeddings) fused into a single ranked list of memory hits and canonical entity discovery hits. +- `save_memory` / `forget_memory` — persist or remove free-form notes tagged with canonical entities (e.g. `mydb.orders.amount`) so the next session inherits the context. + +Try it conversationally: + +> "Save a memory: 'orders.is_returned is in {0,1,NULL}; treat NULL as not returned' linked to mydb.orders.is_returned" +> +> "Search for what we know about orders.is_returned" + +The agent calls `save_memory` then `search` and gets back a `SearchResponse` with `results: [...]` — each hit carries a `kind` discriminator (`"memory"` for prior notes, `"column"` / `"model"` / etc. for entity discovery hits) and a `score`. + +The same `search` call also accepts `cypher_filter` for graph-shaped narrowing (full openCypher with `advanced_search` installed, naive `MATCH (n:Label) RETURN n.id AS id` kind-filter without). The embedding channel needs `motley-slayer[advanced_search]` plus a provider API key — without those, search degrades to BM25 + Tantivy with a single warning in `SearchResponse.warnings`. + +See [Search](../concepts/search.md), [Memories](../concepts/memories.md), and the [MCP Reference](../reference/mcp.md#memories-semantic-search) for full signatures. + ## Alternative: permanent install If you prefer a traditional install instead of `uvx`: ```bash -uv tool install 'motley-slayer[postgres,embedding_search]' +uv tool install 'motley-slayer[postgres,advanced_search]' claude mcp add slayer -- slayer mcp --ingest-on-startup ``` diff --git a/docs/getting-started/python.md b/docs/getting-started/python.md index 4d3fad15..6904720c 100644 --- a/docs/getting-started/python.md +++ b/docs/getting-started/python.md @@ -75,6 +75,22 @@ result.sql # generated SQL (when dry_run or explain is set) result.attributes # ResponseAttributes with .dimensions and .measures dicts ``` +### Tenant scoping (row-level security) + +Pass a `policy=` to the engine (or the local-mode client) to silently scope +every query to one tenant — joins, sub-queries, and sample data included. The +agent cannot read, override, or disable it: + +```python +from slayer.core.policy import SessionPolicy, ColumnFilterRuleset + +engine = SlayerQueryEngine(storage=storage, policy=SessionPolicy( + ruleset=ColumnFilterRuleset(column="organization_uuid", value="7ef3..."), +)) +``` + +See [Row-Level Security](../concepts/row-level-security.md) for the full model. + ## Remote mode (client → server) Connect to a running SLayer server: @@ -144,6 +160,37 @@ result = engine.execute_sync(query={"source_model": "orders", "measures": ["*:co print(f"{result.row_count} row(s), columns: {result.columns}") ``` +## Search & memories + +`SlayerClient.search`, `save_memory`, and `forget_memory` are the single retrieval surface for prior notes **and** canonical entity discovery. All three are async; wrap them with `run_sync` for synchronous use. Local mode (`storage=`) goes through `SearchService` / `MemoryService` directly; remote mode (`url=`) POSTs to `/search` / `/memories`. + +```python +from slayer.async_utils import run_sync + +# Save a learning so the next session inherits it +run_sync(client.save_memory( + learning="orders.is_returned in {0,1,NULL}; treat NULL as not returned", + linked_entities=["mydb.orders.is_returned"], + id="kb.returns.null-handling", # optional; auto-allocated if omitted +)) + +# Three retrieval channels (BM25 + Tantivy + optional dense embeddings) +# fused into one flat ranked list: +resp = run_sync(client.search( + question="What should I know about returns?", + max_results=10, +)) + +for hit in resp.results: + # kind: "memory" | "datasource" | "model" | "column" | "measure" | "aggregation" + # hit.query is not None marks a saved example query + print(hit.kind, hit.id, round(hit.score, 3), hit.text[:80]) +``` + +`client.search` also accepts `cypher_filter` for graph-shaped narrowing — full openCypher with the `advanced_search` extra (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges), naive `MATCH (n:Label) RETURN n.id AS id` kind-filter otherwise. Without `advanced_search` (or a provider API key) the dense-embedding channel emits a single warning into `SearchResponse.warnings` and search degrades to BM25 + Tantivy. Column hits embed the structured `sampled_values` snapshot (top 50 by frequency, JSON-encoded; overflow columns are marked `50+ distinct` in the text snapshot); stale profiles are refreshed lazily inside `search()`. + +See [Search](../concepts/search.md), [Memories](../concepts/memories.md), and the [Python Client Reference](../reference/python-client.md#memories-semantic-search) for the full signature. + ## Embedded REST / MCP servers If you're mounting SLayer's REST or MCP surface inside your own process and want models freshly ingested by the time the server starts handling requests, pass `ingest_on_startup=True` to the constructor: diff --git a/docs/getting-started/rest-api.md b/docs/getting-started/rest-api.md index 5421e8c7..f196d284 100644 --- a/docs/getting-started/rest-api.md +++ b/docs/getting-started/rest-api.md @@ -127,6 +127,39 @@ curl http://localhost:5143/models If `/models` returns an empty list, restart with `slayer serve --ingest-on-startup` or run `slayer ingest --datasource my_pg`. +## Search & memories + +`POST /search` returns memories and canonical entity discovery hits in a single flat ranked list, fused across up to three channels (BM25 over memory entity tags + Tantivy full-text + optional dense embeddings via `motley-slayer[advanced_search]` plus a provider API key). + +```bash +# Question-driven +curl -X POST http://localhost:5143/search \ + -H "Content-Type: application/json" \ + -d '{"question": "What should I know about returns?", "max_results": 10}' + +# Entity-driven; cypher_filter narrows to memories only (naive form, always available) +curl -X POST http://localhost:5143/search \ + -H "Content-Type: application/json" \ + -d '{ + "entities": ["mydb.orders.is_returned"], + "cypher_filter": "MATCH (n:Memory) RETURN n.id AS id" + }' + +# Persist a learning so the next session inherits it +curl -X POST http://localhost:5143/memories \ + -H "Content-Type: application/json" \ + -d '{ + "learning": "orders.is_returned in {0,1,NULL}; treat NULL as not returned", + "linked_entities": ["mydb.orders.is_returned"], + "id": "kb.returns.null-handling" + }' + +# Delete (cascade-strips memory: refs) +curl -X DELETE http://localhost:5143/memories/kb.returns.null-handling +``` + +Each hit in the response carries a `kind` discriminator (`"memory"` for prior notes, `"datasource"` / `"model"` / `"column"` / `"measure"` / `"aggregation"` for entity discovery hits) and a `score`. `cypher_filter` accepts full openCypher when `advanced_search` is installed (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges); without the extra, only the naive `MATCH (n:Label) RETURN n.id AS id` kind-filter form is accepted — anything richer returns HTTP 400. See [Search](../concepts/search.md), [Memories](../concepts/memories.md), and the [REST API Reference](../reference/rest-api.md#memories-semantic-search). + ## Using from other languages SLayer is just HTTP + JSON — use any HTTP client: diff --git a/docs/index.md b/docs/index.md index a4ff34be..dca11ca9 100644 --- a/docs/index.md +++ b/docs/index.md @@ -57,6 +57,7 @@ One query, and SLayer handles: - **Cross-model measures** — Query measures from [joined models](examples/05_joined_measures/joined_measures.md) with dot syntax: `"customers.score:avg"`. Joins are auto-resolved by walking the model graph ([tutorial](examples/05_joins/joins.md)). - **[Multistage queries](examples/06_multistage_queries/multistage_queries.md)** — Use one query as the source for another, or save any query as a permanent model. - **Runtime model editing** — Add measures, dimensions, and joins through any interface. No rebuild, no restart. +- **[Memories + semantic search](concepts/search.md)** — Persist free-form learnings tagged with canonical entities (`..`) and retrieve them alongside model / column discovery hits via a single `search` call. Three retrieval channels (BM25 over memory tags + Tantivy full-text + optional dense embeddings) are RRF-fused into one flat ranked list. Optional graph pre-filter via `cypher_filter` ([memories concept](concepts/memories.md)). - **Broad database support** — Integration-tested against Postgres, MySQL, ClickHouse, DuckDB, and SQLite. Others via sqlglot. ## Get started diff --git a/docs/interfaces/cli.md b/docs/interfaces/cli.md index 90929215..6d0b9d70 100644 --- a/docs/interfaces/cli.md +++ b/docs/interfaces/cli.md @@ -25,7 +25,7 @@ slayer serve --storage slayer.db | `--host` | `0.0.0.0` | Bind address | | `--port` | `5143` | Port number | | `--storage` | [platform default](../configuration/storage.md) | Storage path (directory for YAML, `.db` file for SQLite) | -| `--demo` | off | Spin up the bundled Jaffle Shop DuckDB datasource and ingest its models on startup. Idempotent; requires the `duckdb` extra and `jafgen`. | +| `--demo` | off | Spin up the bundled Jaffle Shop DuckDB datasource and ingest its models on startup. Idempotent; `duckdb` and `jafgen` ship as core dependencies, so no extra install is needed. | | `--ingest-on-startup` | off | Walk every configured datasource and run idempotent auto-ingestion before the port opens. Per-datasource errors are logged to stderr and never abort startup. Also enabled by `SLAYER_INGEST_ON_STARTUP=1`. | ### `slayer mcp` @@ -45,7 +45,7 @@ For MCP over HTTP (SSE), use `slayer serve` instead — it exposes MCP at `/mcp/ | Flag | Default | Description | |------|---------|-------------| | `--storage` | [platform default](../configuration/storage.md) | Storage path (directory for YAML, `.db` file for SQLite) | -| `--demo` | off | Spin up the bundled Jaffle Shop DuckDB datasource and ingest its models on startup. Idempotent; requires the `duckdb` extra and `jafgen`. | +| `--demo` | off | Spin up the bundled Jaffle Shop DuckDB datasource and ingest its models on startup. Idempotent; `duckdb` and `jafgen` ship as core dependencies, so no extra install is needed. | | `--ingest-on-startup` | off | Walk every configured datasource and run idempotent auto-ingestion before stdio JSON-RPC starts. Per-datasource errors are logged to stderr and never abort startup. Also enabled by `SLAYER_INGEST_ON_STARTUP=1`. | ### `slayer query` @@ -124,6 +124,21 @@ slayer import-dbt ./my_dbt_project --datasource my_postgres --include-hidden-mod | `dbt_project_path` | Yes | Path to the dbt project root (or a models directory) | | `--datasource` | Yes | SLayer datasource name for the imported models | | `--include-hidden-models` | No | Also import regular dbt models (those not wrapped by a `semantic_model`) as hidden SLayer models via SQL introspection. Requires the `dbt` extra. | + +### `slayer import-osi` + +Import OSI (Open Semantic Interchange) configs into SLayer. See [Importing OSI configs](../osi/osi_import.md). + +```bash +slayer import-osi ./osi_configs --datasource my_postgres +slayer import-osi ./model.yaml --datasource my_postgres --dialect SNOWFLAKE +``` + +| Flag | Required | Description | +|------|----------|-------------| +| `osi_path` | Yes | Path to an OSI file or directory (`.yaml`/`.yml`/`.json`) | +| `--datasource` | Yes | SLayer datasource name (must be reachable — types come from live introspection) | +| `--dialect` | No | OSI expression dialect to read (default `ANSI_SQL`); falls back to another SQL dialect when absent | | `--storage` | No | Storage path | ### `slayer models` @@ -137,6 +152,20 @@ slayer models create model.yaml slayer models delete orders ``` +### `slayer inspect` + +Point-lookup of an entity by reference + kind — no ranking, no bundled memories (use `slayer search` for an entity *in context*). Pass two or more references for a same-kind batch (DEV-1612). + +```bash +slayer inspect jaffle_shop.orders --type model +slayer inspect jaffle_shop.orders.order_total --type column --no-compact +slayer inspect memory:42 --type memory --no-compact +slayer inspect jaffle_shop.orders --type model --format json +slayer inspect jaffle_shop.orders.order_total jaffle_shop.orders.order_id --type column --no-compact # batch +``` + +`--type` is required (`datasource` / `model` / `column` / `measure` / `aggregation` / `memory`) and applies to every reference. The compact default is a schema skeleton for `--type model` (column / measure / aggregation names + joins, zero DB calls) and description-only for the other kinds; `--no-compact` returns the full render. Passing multiple references returns one `## ` block per reference, in input order (a JSON array under `--format json`), with per-reference error isolation. See the [CLI reference](../reference/cli.md#slayer-inspect) for all flags. + ### `slayer datasources` Manage datasources. @@ -146,24 +175,97 @@ slayer datasources list slayer datasources show my_postgres # credentials masked ``` -### `slayer help` +### `slayer search` + +Run semantic search over memories and canonical entities. Three retrieval channels run in parallel — BM25 over memory entity tags, Tantivy full-text over memories ∪ entities, and (with `motley-slayer[advanced_search]` plus a provider API key) dense embeddings — and are RRF-fused into a single ranked list. See [Search](../concepts/search.md). + +```bash +# Entity-driven +slayer search --entity jaffle_shop.orders.order_total + +# Question-driven +slayer search --question "What stores are in jaffle_shop?" + +# Inline query / from a file (auto-extracts canonical entities) +slayer search --query '{"source_model": "orders", "measures": ["order_total:sum"]}' +slayer search --query @draft_query.json + +# Narrow to one datasource +slayer search --question "lifetime spend" --datasource jaffle_shop + +# Graph-narrow with cypher_filter (naive form, always available) +slayer search --question "Brooklyn POS" --cypher-filter 'MATCH (n:Memory) RETURN n.id AS id' + +# JSON output for piping +slayer search --question "lifetime spend" --format json +``` + +| Flag | Default | Description | +|------|---------|-------------| +| `--entity ENT` (repeatable) | | Canonical entity string (``, `.`, `..`, `memory:`). | +| `--query JSON_OR_@FILE` | | Inline SLayer query (or `@path.json`); entities auto-extracted. | +| `--question TEXT` | | Free-text question. Drives Tantivy + embeddings. | +| `--datasource DS` | | Pre-narrow every channel to ids rooted at the named datasource. | +| `--cypher-filter CYPHER` | | Graph pre-filter. Full openCypher with the `advanced_search` extra (LadybugDB property graph: `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes; `MENTIONS` / `CONTAINS` / `JOINS` edges). Without the extra, only the naive `MATCH (n:Label1:Label2…) RETURN n.id AS id` form is accepted; richer Cypher raises with an install hint. | +| `--max-results N` | `10` | Cap applied after RRF fusion and the `cypher_filter` allowlist. | +| `--format` | `text` | `text` (newline-grouped human output) or `json` (full `SearchResponse`). | + +#### `slayer search refresh-samples` + +Re-profile and persist `Column.sampled` / `sampled_values` / `distinct_count` for table-backed models. Per-column failures are reported but do not abort. + +```bash +slayer search refresh-samples +slayer search refresh-samples --data-source jaffle_shop +slayer search refresh-samples --data-source jaffle_shop --model orders --model customers +``` + +### `slayer memory` + +Manage the agent-memory layer. See [Memories](../concepts/memories.md). + +```bash +# Save a learning (--entities is a single comma-separated string; --query is mutually exclusive) +slayer memory save \ + --learning "orders.is_returned in {0,1,NULL}; treat NULL as not returned" \ + --entities mydb.orders.is_returned \ + --id kb.returns.null-handling + +slayer memory save \ + --learning "Top customers by lifetime spend" \ + --query @top_customers.json \ + --id kb.top-customers + +slayer memory forget kb.returns.null-handling +``` + +| Subcommand | Flag | Description | +|------------|------|-------------| +| `save` | `--learning TEXT` (required) | The free-form note. | +| `save` | `--entities ENT,ENT,…` | Comma-separated canonical entity strings. Mutually exclusive with `--query`; one of the two is required. | +| `save` | `--query JSON_OR_@FILE` | Inline SLayer query (or `@path.json`). Entities auto-extracted and the query is persisted on the memory. | +| `save` | `--id ID` | User-pinned canonical memory id. Forbidden charset: `:`, `/`, `?`, `#`, whitespace, ASCII control. Omit to auto-allocate (`max(int-shaped id) + 1`). Duplicate id → unconditional upsert; `created_at` preserved. | +| `forget` | `` (positional) | Memory id. Cascade-strips every `memory:` reference to it from every other memory's `entities` list. | + +### Conceptual help -Show SLayer's conceptual help — the same content the MCP `help()` tool returns. -Intended to complement the schema/reference pages: it covers how concepts -compose (query evaluation order, transform trade-offs, cross-model measures, -the three meanings of "last") rather than restating field-by-field schemas. +SLayer's conceptual help ships as a predefined set of **help memories** +(`memory:help.intro` … `memory:help.workflow`) — read them with `inspect`, or +find the relevant one with `search`: ```bash -slayer help # intro (core entities, query shape, key invariants) -slayer help queries # deep dive on query anatomy -slayer help transforms # cumsum, time_shift, lag/lead trade-offs -slayer help --help # argparse-level help lists every topic +slayer inspect memory:help.intro --type memory # overview + the query shape +slayer inspect memory:help.transforms --type memory # cumsum, time_shift, lag/lead trade-offs +slayer search --question "how do transforms work" # surface the relevant topic ``` -Topics: `queries`, `formulas`, `aggregations`, `transforms`, `time`, `filters`, -`joins`, `models`, `extending`, `workflow`. Content lives in -`slayer/help/topics/*.md` and is discovered dynamically — dropping a new `.md` -in that directory adds a topic with no Python changes. See the corresponding -concept docs for full treatments: [queries](../concepts/queries.md), -[formulas](../concepts/formulas.md), [models](../concepts/models.md), -[ingestion](../concepts/ingestion.md). +`memory:help.intro` lists the deep-dive topics (`memory:help.queries`, +`memory:help.formulas`, `memory:help.aggregations`, `memory:help.transforms`, +`memory:help.time`, `memory:help.filters`, `memory:help.joins`, +`memory:help.models`, `memory:help.extending`, `memory:help.workflow`). The +topics complement the schema/reference pages: they cover how concepts compose +(query evaluation order, transform trade-offs, cross-model measures, the three +meanings of "last") rather than restating field-by-field schemas. See the +corresponding concept docs for full treatments: +[queries](../concepts/queries.md), [formulas](../concepts/formulas.md), +[models](../concepts/models.md), [ingestion](../concepts/ingestion.md). diff --git a/docs/interfaces/mcp.md b/docs/interfaces/mcp.md index 453b431d..64ee57d0 100644 --- a/docs/interfaces/mcp.md +++ b/docs/interfaces/mcp.md @@ -75,8 +75,9 @@ claude mcp list | Tool | Description | |------|-------------| -| `models_summary` | Brief summary of all non-hidden models in a datasource: each model's name, description, a table of its **columns** and **measures** (named formulas), and the list of models it joins to. The Markdown form (default) shows just `name` + `description` per column; the JSON form (`format="json"`) additionally includes the column `type`. Neither form includes distinct values, sample data, or joined-model field expansion — call `inspect_model` for those. Params: `datasource_name`, `format` (default `"markdown"`; also `"json"`). | -| `inspect_model` | Complete view of a single model: metadata with row count (and a `**meta:**` bullet when the model has `meta` set), any model-level or column-level filters, **columns table** (with a `sampled` column — distinct values for string/boolean columns, `min .. max` for number/date/time columns — and a `meta` cell when set), **measures table** of named formulas (with `formula`, `label`, `description`, `meta`), custom aggregations (with `meta`), joins, all fields reachable via joins (default depth 5), and a sample-data table. Every Markdown table auto-prunes all-empty columns (so the `meta` column is hidden when no entity has meta) and collapses to a comma-separated backticked list when only one column remains. Params: `model_name`, `num_rows` (default 3), `show_sql` (default false — include SQL for the sample-data query, the custom-SQL block, model-level filters, the cached backing-query SQL, and aggregation formulas/param SQL), `format` (default `"markdown"`; also `"json"`), `sections` (subset of `["columns", "measures", "aggregations", "joins", "reachable_fields", "samples"]` — default `None`/`[]` renders all six; the first four collapse to a one-line backticked CSV of names when omitted, `reachable_fields`/`samples` are dropped entirely, unknown names emit a footer warning. A non-empty list of *only* unknown names resolves to no sections — "all six" is reserved for `None`/`[]` so a typo can't silently trigger the full payload), `descriptions_max_chars` (when set, truncate each description longer than this with the suffix `"... [truncated]"` (prefixed by a space); applies to model, columns, measures, and aggregations; must be `>= 0`), `reachable_fields_depth` (max BFS depth in path segments — default 5, allowed range `[0, 20]`; ignored when `reachable_fields` is not in `sections`). When any section is trimmed, a quoted-Markdown footer lists what was shown / names-only / omitted with a hint on how to re-call. JSON output mirrors this with `
_names` siblings and top-level `omitted_sections`, `names_only_sections`, `unknown_sections` arrays. | +| `models_summary` | Brief summary of all non-hidden models in a datasource: each model's name, description, a table of its **columns** and **measures** (named formulas), and the list of models it joins to. The Markdown form (default) shows just `name` + `description` per column; the JSON form (`format="json"`) additionally includes the column `type`. Neither form includes distinct values or sample data — call `inspect_model` for those. For multi-hop discovery (fields reachable via joins from a given model), use the `search` tool with `cypher_filter` for graph queries. Params: `datasource_name`, `format` (default `"markdown"`; also `"json"`). | +| `inspect` | Single-entity point lookup: the rendered detail for **exactly one** entity by `reference` + required `entity_type` (`datasource`/`model`/`column`/`measure`/`aggregation`/`memory`). No fusion/ranking/cypher and no bundled memories — use `search` for an entity *in context*. `reference` accepts canonical ids, bare names, join paths (`orders.customers.region` → owning model), and `memory:`; normalised via the shared resolver (normalised id echoed in the JSON shape). `entity_type` settles the 3-part canonical collision (column vs measure vs aggregation sharing a name) and asserts the kind (mismatch → detailed error). Renders hidden entities. Params: `reference`, `entity_type`, `compact` (default true), `format` (`"markdown"`/`"json"`), and the model-only `num_rows`/`show_sql`/`sections`/`descriptions_max_chars` (used for `entity_type="model"`; `descriptions_max_chars` applies to every kind; others ignored-with-warning for non-model kinds, `show_sql` a silent no-op for column/measure/aggregation). **`compact=True`** is description-only for leaf/datasource/memory, and a cheap **schema skeleton** (column/measure/aggregation **names** + join targets, zero DB calls) for `model`. **`compact=False`** reuses the full `inspect_model` rendering for `model`, and renders a per-model skeleton per visible model (sorted by name; `models: [...]` in JSON) for `datasource`. JSON `text` is present iff non-empty (omitted under `compact=True`). Also on REST `POST /inspect`, CLI `slayer inspect`, and `SlayerClient.inspect`/`inspect_sync`. **Batch (DEV-1612):** `reference` also accepts a **list** of ids (homogeneous kind, one `entity_type` for all). A single `str` stays byte-for-byte; a list returns one block per id in input order, each echoing its resolved canonical id (`## ` headers in markdown, a JSON array under `format="json"`). One-element lists are still framed; per-id errors are isolated; an empty list raises. | +| `inspect_model` | **DEPRECATED — use `inspect`.** Complete view of a single model: metadata with row count (and a `**meta:**` bullet when the model has `meta` set), any model-level or column-level filters, **columns table** (with a `sampled` column — distinct values for string/boolean columns, `min .. max` for number/date/time columns — and a `meta` cell when set), **measures table** of named formulas (with `formula`, `label`, `description`, `meta`), custom aggregations (with `meta`), direct joins, and a sample-data table. Every Markdown table auto-prunes all-empty columns (so the `meta` column is hidden when no entity has meta) and collapses to a comma-separated backticked list when only one column remains. Params: `model_name`, `num_rows` (default 3), `show_sql` (default false — include SQL for the sample-data query, the custom-SQL block, model-level filters, the cached backing-query SQL, and aggregation formulas/param SQL), `format` (default `"markdown"`; also `"json"`), `sections` (subset of `["columns", "measures", "aggregations", "joins", "samples", "learnings"]` — default `None`/`[]` renders all six; the first four collapse to a one-line backticked CSV of names when omitted, `samples`/`learnings` are dropped entirely, unknown names emit a footer warning. A non-empty list of *only* unknown names resolves to no sections — "all six" is reserved for `None`/`[]` so a typo can't silently trigger the full payload), `descriptions_max_chars` (when set, truncate each description longer than this with the suffix `"... [truncated]"` (prefixed by a space); applies to model, columns, measures, and aggregations; must be `>= 0`). When any section is trimmed, a quoted-Markdown footer lists what was shown / names-only / omitted with a hint on how to re-call. JSON output mirrors this with `
_names` siblings and top-level `omitted_sections`, `names_only_sections`, `unknown_sections` arrays. For multi-hop reachability use the `search` tool. | | `create_model` | Create a model from a table/SQL definition or from a query. Pass `sql_table`/`sql` with `columns` (and optional named-formula `measures`) for table-based, or pass `query` (a SLayer query dict) to save it as a query-backed model whose `columns` + `backing_query_sql` are populated by a save-time dry-run. | | `edit_model` | Edit an existing model in one call. Upserts `columns`, `measures` (named formulas), `aggregations`, `joins` (pass the new entries; existing names are updated, new ones are added). Also accepts `description`, `data_source`, `default_time_dimension`, `sql_table`/`sql`/`source_queries`, `query_variables`, `hidden`, `meta`, `add_filters`/`remove_filters`, and `remove: {"columns": [...], "measures": [...], "aggregations": [...], "joins": [...]}` for entity removal. | | `delete_model` | Delete a model entirely. | @@ -86,6 +87,7 @@ claude mcp list | Tool | Description | |------|-------------| | `query` | Execute a semantic query. See [Queries](../concepts/queries.md) for format. | +| `query_nested` | Execute a multi-stage DAG of named sub-queries that reference one another via `source_model` or `joins.target_model`. Companion to `query`; the engine auto-sorts the list, so order doesn't matter. Params: `queries: List[Dict[str, Any]]`, plus `variables` / `show_sql` / `dry_run` / `explain` / `format` mirroring `query`. | **`query` parameters:** @@ -111,26 +113,56 @@ claude mcp list |------|-------------| | `ingest_datasource_models` | Auto-generate models from DB schema with rollup joins. Params: `datasource_name`, `include_tables`, `schema_name`. | -### Conceptual Help +### Memories + semantic search + +Memories are free-form notes the agent saves against canonical entity strings (``, `.`, `..`, or `memory:`). `search` is the only retrieval surface and returns memories **and** entity discovery hits in one flat ranked list. See [Memories](../concepts/memories.md) and [Search](../concepts/search.md). | Tool | Description | |------|-------------| -| `help` | Return SLayer concept explanations that complement the schema-focused tool docstrings. Call without arguments for the intro; pass `topic="..."` for a deep dive. The tool description lists every available topic — no exploratory call needed. | - -Available topics and what they cover (content lives in `slayer/help/topics/*.md`, discovered dynamically): - -| Topic | Covers | -|-------|--------| -| `queries` | Anatomy of a [query](../concepts/queries.md); evaluation order; dimensions vs [time dimensions](../concepts/queries.md#timedimension) on the same column; `main_time_dimension` disambiguation | -| `formulas` | The [formula mini-language](../concepts/formulas.md) shared by `measures` and `filters`; colon syntax; arithmetic; nesting | -| `aggregations` | Built-in and [custom aggregations](../examples/07_aggregations/aggregations.md); `first`/`last` time-column resolution; `allowed_aggregations` | -| `transforms` | `cumsum`, `time_shift`, `change`, `lag`, the rank family (`rank`/`percent_rank`/`dense_rank`/`ntile`, optional `partition_by=`), `last()` — trade-offs and nesting ([time post](../examples/04_time/time.md)) | -| `time` | Granularities, `date_range`, `whole_periods_only`, the three meanings of "last" | -| `filters` | Operators; auto-routing to HAVING / post-filter; filtered measures; [model-level filters](../concepts/models.md#model-filters) | -| `joins` | Dot syntax and the `__` alias convention; cross-model measures and diamond joins ([joins post](../examples/05_joins/joins.md), [joined measures](../examples/05_joined_measures/joined_measures.md)) | -| `models` | Source modes (`sql_table`, `sql`, `source_queries`); query-backed models, `query_variables`, cached `backing_query_sql`; result column naming; `default_time_dimension`; hidden models ([models ref](../concepts/models.md)) | -| `extending` | `ModelExtension`, query lists, `create_model_from_query` (with `variables=`), run-by-name via `query` tool ([multistage post](../examples/06_multistage_queries/multistage_queries.md)) | -| `workflow` | Tool-chaining playbook, query-iteration tips, common-error decoder | +| `search` | Up to three-channel retrieval over memories and canonical entities (datasource / model / column / measure / aggregation), RRF-fused (k=60) into a single ranked list of `SearchHit` objects. | +| `save_memory` | Persist a free-form `learning` tagged with canonical entities or an inline `SlayerQuery`. | +| `forget_memory` | Delete a memory by id. Cascade-strips every other memory's `memory:` ref to it. | + +**`search` parameters:** + +| Param | Type | Description | +|-------|------|-------------| +| `entities` | list[str] | Canonical entity strings (`mydb.orders.amount`, `memory:42`, …) or aggregated colon forms (`revenue:sum` — the suffix is stripped). Drives the BM25 channel. Unresolved tokens emit warnings, not errors. | +| `query` | dict \| SlayerQuery | Inline query; its `source_model`, dimensions, measures, time dims, and filters are walked for canonical entities. | +| `question` | str | Free-text question. Drives the Tantivy full-text channel and (when available) the dense-embedding channel. | +| `datasource` | str | When set, every channel pre-filters to canonical ids rooted at that datasource. Unknown name → error. | +| `cypher_filter` | str | Graph pre-filter applied to all three channels. Full openCypher when `advanced_search` is installed (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges; read-only — `CREATE`, `MERGE`, `DELETE`, `SET`, `REMOVE`, `DROP`, `CALL` are rejected). Without the extra, only the naive form `MATCH (n:Label1:Label2…) RETURN n.id AS id` is accepted as a label/kind filter; richer Cypher raises with an install hint. | +| `max_results` | int | Cap applied **after** RRF fusion and the `cypher_filter` allowlist. Default `10`. | + +`SearchResponse.results` is a flat list of `SearchHit { kind, id, score, text, matched_entities, query }`. `kind` is one of `"memory"`, `"datasource"`, `"model"`, `"column"`, `"measure"`, `"aggregation"`. For memory hits, `id` is the raw memory id (suitable for `forget_memory`); `hit.query is not None` marks a saved example query. Column hits embed the structured `sampled_values` snapshot (top 50 by frequency, JSON-encoded; overflow columns are marked `50+ distinct` in the text snapshot); stale column profiles are refreshed lazily inside `search()`. + +**`save_memory(learning, linked_entities, id=None)`** — `linked_entities` accepts canonical entity strings (strict resolution; `memory:` valid for cross-memory refs) **or** an inline `SlayerQuery` dict (the entities are auto-extracted and the query is persisted on the memory). Optional `id` pins a user-controlled canonical memory id; forbidden charset: `:`, `/`, `?`, `#`, whitespace, ASCII control. Omit `id` to let the allocator assign the next int-shaped id (`max(int-shaped id) + 1`). Duplicate id → unconditional upsert; `created_at` preserved. + +**`forget_memory(id)`** — removes the memory, drops the matching embedding row, and strips every `memory:` ref to it from every other memory's `entities` list (exact-match only — `memory:42` does not strip `memory:421`). + +### Conceptual help (help memories) + +SLayer's conceptual help is not a tool — it ships as a predefined set of **help +memories** seeded into storage on server startup. Read the entry point with +`inspect(reference="memory:help.intro", entity_type="memory")`; it lists the +deep-dive topics, each of which you inspect the same way. `search(question="…")` +also surfaces the relevant topic. The server instructions point new agents at +`memory:help.intro`. + +Available topics and what they cover (content lives in `slayer/memories/help_content/*.md`, seeded as `memory:help.`): + +| Topic id | Covers | +|----------|--------| +| `memory:help.queries` | Anatomy of a [query](../concepts/queries.md); evaluation order; dimensions vs [time dimensions](../concepts/queries.md#timedimension) on the same column; `main_time_dimension` disambiguation | +| `memory:help.formulas` | The [formula mini-language](../concepts/formulas.md) shared by `measures` and `filters`; colon syntax; arithmetic; nesting | +| `memory:help.aggregations` | Built-in and [custom aggregations](../examples/07_aggregations/aggregations.md); `first`/`last` time-column resolution; `allowed_aggregations` | +| `memory:help.transforms` | `cumsum`, `time_shift`, `change`, `lag`, the rank family (`rank`/`percent_rank`/`dense_rank`/`ntile`, optional `partition_by=`), `last()` — trade-offs and nesting ([time post](../examples/04_time/time.md)) | +| `memory:help.time` | Granularities, `date_range`, `whole_periods_only`, the three meanings of "last" | +| `memory:help.filters` | Operators; auto-routing to HAVING / post-filter; filtered measures; [model-level filters](../concepts/models.md#model-filters) | +| `memory:help.joins` | Dot syntax and the `__` alias convention; cross-model measures and diamond joins ([joins post](../examples/05_joins/joins.md), [joined measures](../examples/05_joined_measures/joined_measures.md)) | +| `memory:help.models` | Source modes (`sql_table`, `sql`, `source_queries`); query-backed models, `query_variables`, cached `backing_query_sql`; result column naming; `default_time_dimension`; hidden models ([models ref](../concepts/models.md)) | +| `memory:help.extending` | `ModelExtension`, query lists, `create_model_from_query` (with `variables=`), run-by-name via `query` tool ([multistage post](../examples/06_multistage_queries/multistage_queries.md)) | +| `memory:help.workflow` | Tool-chaining playbook, query-iteration tips, common-error decoder | ## Typical Agent Workflows diff --git a/docs/interfaces/pg-facade.md b/docs/interfaces/pg-facade.md index 06532283..a6b48f3a 100644 --- a/docs/interfaces/pg-facade.md +++ b/docs/interfaces/pg-facade.md @@ -18,10 +18,6 @@ slayer pg-serve --demo # Production-ish — non-loopback bind requires a password token slayer pg-serve --host 0.0.0.0 --token "$(pass slayer-token)" - -# TLS-enabled -slayer pg-serve --host 0.0.0.0 --token TOK \ - --tls-cert /etc/ssl/slayer.crt --tls-key /etc/ssl/slayer.key ``` Flags: @@ -31,7 +27,6 @@ Flags: | `--host HOST` | Bind address. Default `0.0.0.0`. With `--demo` and no token, defaults to `127.0.0.1` for the loopback fallback. | | `--port PORT` | Default `5145`. | | `--token T` | Password token. Falls back to `$SLAYER_PG_TOKEN`. Required for non-loopback binds. | -| `--tls-cert C` / `--tls-key K` | TLS certificate + key pair (must be supplied together). | | `--demo` | Generate + ingest the bundled Jaffle Shop dataset before starting. | | `--storage PATH` | Storage path (same as the REST + MCP servers). | @@ -60,20 +55,50 @@ Any tool with a PostgreSQL connector works. End-to-end with the bundled demo and ```bash # 1. Start SLayer speaking Postgres, with the Jaffle Shop demo preloaded. -slayer pg-serve --demo # listens on 127.0.0.1:5145 +# The BI tool connects over the network (e.g. from a Docker container), so +# bind all interfaces — a non-loopback bind requires a token. +slayer pg-serve --demo --host 0.0.0.0 --token pick-a-secret # 2. Run Metabase (any BI tool works — Superset, Tableau, Power BI, Grafana, …). -docker run -d -p 3000:3000 --name metabase metabase/metabase +# --add-host makes `host.docker.internal` resolve to the Docker host on +# every platform (built into Docker Desktop; required on Linux, Docker ≥ 20.10). +# The volume keeps Metabase's own settings/dashboards across container re-creates. +docker run -d -p 3000:3000 --name metabase \ + --add-host=host.docker.internal:host-gateway \ + -e MB_DB_FILE=/metabase.data/metabase.db \ + -v metabase-data:/metabase.data \ + metabase/metabase ``` In Metabase: **Admin → Databases → Add database → PostgreSQL** and fill in: | Field | Value | |---|---| -| Host | `host.docker.internal` (or your host's IP) | +| Host | `host.docker.internal` | | Port | `5145` | | Database name | the SLayer **datasource** (e.g. `jaffle_shop`) | -| Username / Password | anything when no `--token` is set; otherwise the token as the password | +| Username | anything non-empty (ignored) | +| Password | the `--token` value (`pick-a-secret`) | +| SSL | off | + +Or as a single JDBC connection string: + +```text +jdbc:postgresql://host.docker.internal:5145/jaffle_shop?user=metabase&password=pick-a-secret&sslmode=disable +``` + +> **Connection refused / name not resolving?** Two common causes: +> +> 1. The server was started without `--host 0.0.0.0` — the default demo bind is +> `127.0.0.1`, which containers cannot reach. +> 2. The BI container runs on Linux Docker without the `--add-host` mapping — +> `host.docker.internal` only exists out of the box on Docker Desktop. Either +> re-create the container with the flag (compose: `extra_hosts: +> ["host.docker.internal:host-gateway"]`), or use the container's default +> gateway IP as Host instead — find it with +> `docker exec ip route | awk '/default/ {print $3}'` +> (typically `172.17.0.1` on the default bridge network, but it differs per +> compose network and daemon config, so don't hard-code it). Metabase introspects the schema (via `INFORMATION_SCHEMA` + `pg_catalog`), lists each SLayer model as a table under schema `public`, and lets you build questions/dashboards @@ -90,7 +115,8 @@ against them. Project named metrics (`revenue_sum`) or write `SUM(amount)` / at startup. * With a token, the server requests a cleartext password (`AuthenticationCleartextPassword`); the client's password must equal the token. - Combine with TLS (or a loopback bind) so the password is not sent in the clear. + Use a loopback bind (or a trusted network) so the password is not sent in the clear. + Let us know if you would like us to support TLS. ## SQL Surface @@ -112,6 +138,122 @@ Postgres-specific predicates that aren't valid SLayer DSL (`ILIKE`, `::cast`, re `ANY`/`ALL`) parse but are rejected at execution — use the standard comparison / `IN` / `BETWEEN` forms. +### `CAST( AS )` in projection + +A projection of the shape `CAST( AS )` (and the equivalent `col::type` +sugar) is accepted when the inner expression is a bare or qualified column reference +**and** the (source, target) pair is in the allowlist below. The engine still executes +the bare column — the cast is a pure wire-layer type rewrite. The projected column's +Postgres OID is overridden to match the casted type. + +Common BI shapes covered: `SELECT CAST(ordered_at AS TIMESTAMP) FROM orders` (DATE +column promoted for a TIMESTAMP-aware client), `SELECT CAST(amount AS TEXT) AS s +FROM orders` (stringification), `SELECT CAST(customers.region AS TEXT) FROM orders` +(joined column). + +Out of scope: `CAST` around aggregates (`CAST(SUM(amount) AS DOUBLE)`), `TRY_CAST`, +and `CAST` around expressions that aren't a bare column (`CAST(SUBSTRING(...) AS T)`). +`CAST` wrapping a `DATE_TRUNC(...)` continues to route through the time-grain unwrap. + +`CAST(...)` in `ORDER BY` and `GROUP BY` has two layers of admission: + +1. **Unaliased canonical-form** (e.g. `ORDER BY CAST(c AS T)` repeating the + projection's CAST verbatim): **never admitted.** The translator raises + `ORDER BY column '...' is not in the projection list` / the GROUP BY + strict-on-extras error. Workaround: alias and reference the alias. +2. **Aliased reference** (`SELECT CAST(c AS T) AS x ... ORDER BY x` / + `... GROUP BY x`): admitted **only** when the `(source, target)` pair + preserves sort/group semantics under the bare-column engine projection. + +Pairs that **fail** the aliased-reference admission and raise +`ORDER BY on CAST projection '...' with lossy pair X→T is unsupported` +(symmetric message for GROUP BY): + +| Path | Lossy pairs | +|----------|--------------------------------------------------------------------------| +| ORDER BY | `X → TEXT` for every non-text `X` (lex sort ≠ engine's natural sort). `TEXT → TEXT` is identity and stays admitted. | +| GROUP BY | `TIMESTAMP → DATE` (many-to-one rollup); `INT → DOUBLE` (IEEE 754 collapse beyond ±2^53) | + +Every other admitted pair — identity (`X → X`), `DATE → TIMESTAMP`, +`TIMESTAMP → DATE` for ORDER BY, `INT → DOUBLE` — preserves the casted +semantics under the bare-column engine projection, so the alias path stays +open. + +```sql +-- Always rejected (canonical form): +SELECT CAST(delivered_at AS TIMESTAMP) FROM orders +ORDER BY CAST(delivered_at AS TIMESTAMP); + +-- Aliased reference, safe pair → works: +SELECT CAST(delivered_at AS TIMESTAMP) AS dt FROM orders +ORDER BY dt; + +-- Aliased reference, lossy pair → rejected: +SELECT CAST(id AS TEXT) AS s FROM orders ORDER BY s; +SELECT CAST(ordered_at AS DATE) AS d, COUNT(*) FROM orders GROUP BY d; +``` + +The wire-type override still applies in the safe-pair case — `dt` is +wire-typed `TIMESTAMP` even though the engine sorts the underlying `DATE`. +A future ticket can lift the remaining restrictions by pushing the CAST +into the engine SQL. + +Admitted (source, target) coercions: + +| Source type | Admitted target types | +|---------------|------------------------------| +| `DATE` | `DATE`, `TIMESTAMP`, `TEXT` | +| `TIMESTAMP` | `TIMESTAMP`, `DATE`, `TEXT` | +| `INT` | `INT`, `DOUBLE`, `TEXT` | +| `DOUBLE` | `DOUBLE`, `TEXT` | +| `BOOLEAN` | `BOOLEAN`, `TEXT` | +| `TEXT` | `TEXT` | +| *(unknown)* | `TEXT` | + +Pairs outside the allowlist (e.g. `CAST(name AS INT)`, `CAST(amount AS BOOLEAN)`) +raise `Unsupported CAST: cannot project column as (...). Admitted +coercions: see docs/interfaces/pg-facade.md.` Unsupported target types (`UUID`, +`JSON`, `ARRAY`, `STRUCT`, …) raise the standard `Unsupported projection +expression` error. + +#### CAST coarse-OID mapping + +CAST is a **coarse wire-OID hint**, not a precision-preserving conversion. +The SLayer engine projects the bare column unchanged; the pg-facade encoder +is OID-driven, so the wire bytes always match the OID we advertise. Some +PostgreSQL types the user can write in a CAST don't have a one-to-one +SLayer equivalent — those collapse onto the nearest broader SLayer type: + +| User wrote in `CAST(... AS X)` | SLayer maps to | Wire OID advertised | +|---|---|---| +| `INTEGER` / `INT` (pre-existing) | `DataType.INT` | 20 (`int8`) — not 23 (`int4`) | +| `SMALLINT` | `DataType.INT` | 20 (`int8`) — not 21 (`int2`) | +| `TINYINT` / `MEDIUMINT` (non-Postgres widths) | `DataType.INT` | 20 (`int8`) | +| `BIGINT` | `DataType.INT` | 20 (`int8`) ✓ exact match | +| `DECIMAL` / `NUMERIC` | `DataType.DOUBLE` | 701 (`float8`) — not 1700 (`numeric`) | +| `FLOAT` / `REAL` / `DOUBLE` | `DataType.DOUBLE` | 701 (`float8`) ✓ | +| `TIMESTAMPTZ` / `TIMESTAMP WITH TIME ZONE` | `DataType.TIMESTAMP` | 1114 (`timestamp`, no TZ) — not 1184 (`timestamptz`) | +| `TIMESTAMP` / `DATETIME` | `DataType.TIMESTAMP` | 1114 (`timestamp`) ✓ | + +What this means in practice: + +- The wire bytes the client receives are always consistent with the OID we + advertise (the encoder picks the binary/text form from the OID). There is + no value corruption. +- The OID is potentially broader than what the user typed. A client that + asked for `NUMERIC` and got `float8` sees a float on the wire and decodes + it correctly as a float — but loses the "exact precision" expectation. + A client that asked for `TIMESTAMPTZ` sees naive `timestamp` bytes — and + loses TZ-aware decoding semantics. +- Callers needing exact `NUMERIC` precision, narrow integer wire widths, or + TZ-aware timestamps must compute upstream (or wait for SLayer to model + those types natively). + +`DOUBLE → INT` is intentionally excluded: Python's `int()` truncates toward zero +while Postgres rounds half-to-even, so silently admitting the pair would diverge from +`psql` semantics. Pre-aggregate or pre-round on your side when an integer-typed result +is required. + ## Introspection * `INFORMATION_SCHEMA.METRICS` / `DIMENSIONS` / `SCHEMATA` / `TABLES` / `COLUMNS`. @@ -130,8 +272,40 @@ per column — `asyncpg` (which requests binary results) and `psql` (text) both ## Install -The facade is pure-stdlib; the extra exists only to keep the install path consistent: +The facade is pure-stdlib — no extra is needed. It ships with the base install: ```bash -pip install "motley-slayer[pg_facade]" +pip install motley-slayer ``` + +## Testing your changes + +For wire-level / translator changes, the unit suite under `tests/test_pg_facade*.py` covers +each component in isolation. Behaviour at the *interaction boundary* with a real BI client +is covered by the live-Metabase end-to-end suite (DEV-1562): + +```bash +poetry run pytest -m metabase_e2e tests/integration/test_metabase_e2e.py -v +``` + +The suite needs Docker; it boots `metabase/metabase:v0.62.1.5` alongside two +token-protected pg-serve processes (per-session random tokens, both bound on `0.0.0.0` +so the container reaches them via `host.docker.internal`; the second backs the L.2 / L.3 +bad-password tests) and drives ~62 cases through the real `pgjdbc` protocol — bootstrap + +sync, MBQL aggregations and time-grain breakouts, native-SQL probes, wire-format +round-trips, transactions, concurrency, and error envelopes. Skips cleanly when Docker is +unavailable. + +Known limitations (each tracked by a strict-`xfail` against a Linear ticket — the day the +referenced gap is fixed, the test XPASSes and CI flips red, prompting a lift): +LEFT JOIN-with-subquery projection (DEV-1565), CAST(col AS type) projection (DEV-1566), +catalog fingerprint measure leak (DEV-1567), MBQL aggregation-ordinal refs in HAVING / +ORDER BY (DEV-1568) and per-connection `SET` state (DEV-1569). + +Metabase week breakouts emit a Sunday-anchored week wrapper +(`DATE_TRUNC('week', col + INTERVAL '1 day') - INTERVAL '1 day'`); the translator maps this to +SLayer's `week_sunday` granularity (DEV-1572), so week breakouts bucket the way Metabase asks. + +CI fires automatically on PRs touching `slayer/pg_facade/`, `slayer/facade/`, +`slayer/demo/`, the +e2e test files, or `pyproject.toml` / `poetry.lock`. diff --git a/docs/interfaces/python-client.md b/docs/interfaces/python-client.md index 33b398b1..8ef069e6 100644 --- a/docs/interfaces/python-client.md +++ b/docs/interfaces/python-client.md @@ -87,6 +87,78 @@ datasources = client.list_datasources() client.create_datasource({"name": "mydb", "type": "postgres", ...}) ``` +### Inspect + +`inspect` / `inspect_sync` is a point lookup (DEV-1588): the rendered detail for **exactly one** entity by `reference` + required `entity_type`. No fusion / ranking / bundled memories — use `search` for an entity *in context*. Same arguments as the MCP `inspect` tool and `POST /inspect`; returns the rendered string. **DEV-1612:** `reference` also accepts a **list** — a homogeneous-kind batch (one `entity_type` for every id), returning one block per id in input order with per-id error isolation. + +```python +# Compact default: schema skeleton for a model; description-only for other kinds. +print(client.inspect_sync(reference="mydb.orders", entity_type="model")) + +# Full render of one column (compact=False). +print(client.inspect_sync( + reference="mydb.orders.amount", entity_type="column", compact=False, +)) + +# Batch: several columns of the same kind in one round-trip (DEV-1612). +print(client.inspect_sync( + reference=["mydb.orders.amount", "mydb.orders.customer_id"], + entity_type="column", compact=False, +)) + +# async form: await client.inspect(reference="mydb.orders", entity_type="model") +``` + +`entity_type` is required (`datasource` / `model` / `column` / `measure` / `aggregation` / `memory`) and asserts the resolved kind. The model-only `num_rows` / `show_sql` / `sections` apply for `entity_type="model"`; `descriptions_max_chars` applies to every kind. `format="json"` returns a JSON string instead of Markdown. + +### Memories + Semantic Search + +`SlayerClient` exposes the same single retrieval surface as MCP / REST. All three are async; wrap them with `run_sync` for synchronous use. Local mode goes through `SearchService` / `MemoryService` directly; remote mode POSTs to `/search` and `/memories`. See [Search](../concepts/search.md) and [Memories](../concepts/memories.md). + +```python +from slayer.async_utils import run_sync + +# Save a learning +run_sync(client.save_memory( + learning="orders.is_returned in {0,1,NULL}; treat NULL as not returned", + linked_entities=["mydb.orders.is_returned"], + id="kb.returns.null-handling", # optional; auto-allocated if omitted +)) + +# Search — single flat ranked list with kind discriminator +resp = run_sync(client.search( + question="What should I know about returns?", + max_results=10, +)) + +for hit in resp.results: + if hit.kind == "memory": + kind = "example_query" if hit.query is not None else "learning" + print(f"[{kind}] {hit.id} score={hit.score:.3f} {hit.text[:80]}") + else: + print(f"[{hit.kind}] {hit.id} score={hit.score:.3f}") + +# Forget by id (cascade-strips memory: refs from other memories) +run_sync(client.forget_memory("kb.returns.null-handling")) +``` + +`client.search` signature (keyword-only): + +```python +async def search( + self, + *, + entities: Optional[List[str]] = None, + query: Optional[Union[SlayerQuery, Dict[str, Any]]] = None, + question: Optional[str] = None, + datasource: Optional[str] = None, + max_results: int = 10, + cypher_filter: Optional[str] = None, +) -> SearchResponse: ... +``` + +`cypher_filter` accepts full openCypher when the `advanced_search` extra is installed (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges; mutation clauses rejected). Without the extra, only the naive form `MATCH (n:Label1:Label2…) RETURN n.id AS id` is accepted as a label/kind filter — anything richer raises with an install hint. Column hits embed the structured `sampled_values` snapshot (top 50 by frequency, JSON-encoded; overflow columns are marked `50+ distinct` in the text snapshot); stale profiles are refreshed lazily inside `search()`. + ## Direct Engine Access For maximum control, use the query engine directly: diff --git a/docs/interfaces/rest-api.md b/docs/interfaces/rest-api.md index 08810ba1..5d189c09 100644 --- a/docs/interfaces/rest-api.md +++ b/docs/interfaces/rest-api.md @@ -124,7 +124,7 @@ DELETE /datasources/{name} # Delete a datasource # List datasources curl http://localhost:5143/datasources -# Get datasource (password/connection_string shown as ***) +# Get datasource (credential fields shown as ***) curl http://localhost:5143/datasources/my_postgres ``` @@ -148,3 +148,89 @@ Response: "models": ["orders", "customers", "products"] } ``` + +### Inspect (single-entity point lookup) + +`POST /inspect` returns the rendered detail for **exactly one** entity by `reference` + required `entity_type` (DEV-1588). Unlike `POST /search`, there is no fusion / ranking / `cypher_filter` and no bundled memories — use `/search` when you want an entity surfaced *in context*. + +**`POST /inspect` body:** + +| Field | Type | Description | +|-------|------|-------------| +| `reference` | str \| array[str] | Canonical id (`mydb.orders.amount`), bare name, join path (`orders.customers.region` → resolved to the owning model), or `memory:`. Normalised via the shared resolver; the normalised id is echoed in the result. **A list is a homogeneous-kind batch** (DEV-1612): one `entity_type` for every id; `result` is one block per id in input order (a `## ` header per block in markdown, a JSON-array string under `format="json"`). Per-id resolution errors are isolated. An empty list returns HTTP 400; a non-string member returns HTTP 422. | +| `entity_type` | str | **Required.** One of `datasource`, `model`, `column`, `measure`, `aggregation`, `memory`. Disambiguates the 3-part canonical collision (a name shared by, e.g., a column and an aggregation) and asserts the kind — a mismatch returns HTTP 400. | +| `compact` | bool | Default `true`. Description-only for column / measure / aggregation / datasource / memory; a cheap schema **skeleton** (column / measure / aggregation names + join targets, zero DB calls) for `model`. `false` returns the full render, and a per-model skeleton for each visible model for `datasource`. | +| `format` | str | `"markdown"` (default) or `"json"`. | +| `num_rows` | int | Sample-data rows for `entity_type="model"`. Ignored (with a warning) for other kinds. Default `3`. | +| `show_sql` | bool | Include generated SQL for `entity_type="model"`. Ignored (with a warning) for datasource / memory; a silent no-op for column / measure / aggregation. | +| `sections` | array[str] | Section subset for `entity_type="model"` (`columns` / `measures` / `aggregations` / `joins` / `samples` / `learnings`). Ignored (with a warning) for other kinds. | +| `descriptions_max_chars` | int | Truncate every description field to this many characters (must be `>= 0`). Applies to every kind. | + +Renders hidden entities (deliberate escape hatch). Unknown fields are rejected (HTTP 422); a bad `entity_type` / `format`, a negative `descriptions_max_chars`, or an unresolvable reference returns HTTP 400. + +```bash +curl -X POST http://localhost:5143/inspect \ + -H "Content-Type: application/json" \ + -d '{"reference": "jaffle_shop.orders", "entity_type": "model"}' +``` + +The response is always `{"result": }` — the rendered Markdown (or, with `format="json"`, a JSON string). + +### Memories + Semantic Search + +`POST /search` is the single retrieval surface. It returns memories **and** entity discovery hits in one flat list, fused across up to three channels (BM25 over memory entity tags + Tantivy full-text + optional dense embeddings via `motley-slayer[advanced_search]` plus a provider API key) via Reciprocal Rank Fusion (`k=60`). See [Search](../concepts/search.md) and [Memories](../concepts/memories.md). + +``` +POST /search # Run a semantic search +POST /memories # Persist a memory +DELETE /memories/{id} # Delete a memory (cascade-strips memory: refs) +``` + +**`POST /search` body:** + +| Field | Type | Description | +|-------|------|-------------| +| `entities` | array[str] | Canonical entity strings (`mydb.orders.amount`, `memory:42`, …). Aggregation suffixes are stripped (`revenue:sum` → `mydb.orders.revenue`). Drives the BM25 channel. Unresolved tokens emit warnings rather than errors. | +| `query` | object | Inline SLayer query; its `source_model` / dimensions / measures / time dims / filters are walked for canonical entities. | +| `question` | str | Free-text question. Drives the Tantivy channel and (when available) the embedding channel. | +| `datasource` | str | Pre-narrows every channel to ids rooted at the named datasource. Unknown name → HTTP 400. | +| `cypher_filter` | str | Graph pre-filter applied to all three channels. Full openCypher with the `advanced_search` extra (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges; read-only — mutation clauses rejected). Without the extra, only the naive form `MATCH (n:Label1:Label2…) RETURN n.id AS id` is accepted as a label/kind filter; richer Cypher returns HTTP 400 with an install hint. | +| `max_results` | int | Applied **after** RRF fusion and the `cypher_filter` allowlist. Default `10`. | + +```bash +curl -X POST http://localhost:5143/search \ + -H "Content-Type: application/json" \ + -d '{ + "question": "What should I know about returns?", + "max_results": 10 + }' +``` + +Response (`SearchResponse`): + +```json +{ + "results": [ + {"kind": "memory", "id": "42", "score": 0.13, "text": "orders.is_returned in {0,1,NULL}; treat NULL as not returned", "matched_entities": [], "query": null}, + {"kind": "column", "id": "mydb.orders.is_returned", "score": 0.11, "text": "...\nSample values: [\"0\", \"1\"]", "matched_entities": [], "query": null} + ], + "resolved_input_entities": [], + "warnings": [] +} +``` + +`kind` is one of `"memory"`, `"datasource"`, `"model"`, `"column"`, `"measure"`, `"aggregation"`. For memory hits, `id` is the raw memory id (suitable for `DELETE /memories/{id}`); `query` carries the saved `SlayerQuery` when the memory is query-bearing. Column hits embed the structured `sampled_values` (top 50 by frequency, JSON-encoded; overflow columns are marked `50+ distinct` in the text snapshot); stale profiles are refreshed lazily inside `/search`. + +**`POST /memories` body:** + +```json +{ + "learning": "orders.is_returned in {0,1,NULL}; treat NULL as not returned", + "linked_entities": ["mydb.orders.is_returned"], + "id": "kb.returns.null-handling" +} +``` + +`linked_entities` accepts either an array of canonical entity strings (strict resolution; `memory:` valid for cross-memory refs) **or** an inline `SlayerQuery` dict — the entities are auto-extracted and the query is persisted on the memory. `id` is optional; omit to auto-allocate (`max(int-shaped id) + 1`). Forbidden charset on user-supplied ids: `:`, `/`, `?`, `#`, whitespace, ASCII control. Duplicate id → unconditional upsert; `created_at` preserved. + +**`DELETE /memories/{id}`** removes the memory, drops the matching embedding row, and strips every `memory:` reference to it from every other memory's `entities` list (exact-match only). diff --git a/docs/osi/osi_import.md b/docs/osi/osi_import.md new file mode 100644 index 00000000..47c19bae --- /dev/null +++ b/docs/osi/osi_import.md @@ -0,0 +1,50 @@ +# Importing OSI (Open Semantic Interchange) Configs + +SLayer can import [Open Semantic Interchange](https://github.com/open-semantic-interchange/OSI) (OSI) semantic-model configs and convert them into SLayer models. OSI is a vendor-neutral YAML/JSON standard for semantic models, datasets, relationships, and metrics. + +## Quick Start + +```bash +slayer import-osi ./osi_configs --datasource my_postgres --storage ./slayer_data +``` + +This reads every `.yaml`/`.yml`/`.json` file in the path (a single file or a directory), converts each OSI dataset into a SLayer model, and saves the models to storage. A **reachable datasource is required** — column data types come from live table introspection (OSI carries no column types), so the importer connects to `--datasource` and overlays OSI's semantic metadata on top. + +Spec versions `1.0`, `0.1.0`, `0.1.1`, and `0.2.0.dev0` are all accepted (they are structurally identical); an unknown version is warned about but still parsed. + +## What Gets Converted + +| OSI construct | SLayer target | +|---|---| +| `semantic_model[].datasets[]` | one `SlayerModel` each (name = dataset name) | +| dataset `source` (`db.schema.table`) | `sql_table` + `schema`; a query source → `sql` mode | +| dataset `fields[]` | `Column`s (real types from introspection) | +| field `expression` (bare) | overlaid onto the introspected column | +| field `expression` (derived, e.g. `UPPER(x)`) | a derived `Column` with `sql` set | +| field `dimension.is_time` | column typed temporal; sets `default_time_dimension` | +| dataset `primary_key` | `Column.primary_key = true` | +| `relationships[]` (`from` → `to`) | a LEFT `ModelJoin` on the `from` model | +| `metrics[]` (raw SQL aggregation) | a `ModelMeasure` formula | +| `ai_context` (instructions + synonyms) | entity `description` + `meta["osi_ai_context"]` | +| `unique_keys` / `custom_extensions` | model/column `meta` | + +### Metrics + +An OSI metric holds a raw SQL aggregation expression. SLayer parses it into colon-syntax formulas: + +- `SUM(amount)` → `amount:sum`, `COUNT(*)` → `*:count`, `COUNT(DISTINCT id)` → `id:count_distinct` +- arithmetic + constants + scalar functions pass through: `SUM(a) / NULLIF(COUNT(*), 0)` → `a:sum / nullif(*:count, 0)` +- a non-bare aggregate operand is materialized as a hidden derived column: `SUM(quantity * amount)` → a hidden column `quantity * amount` plus `:sum` +- `PERCENTILE_CONT(0.9) WITHIN GROUP (ORDER BY x)` → `x:percentile(p=0.9)` (`0.5` → `x:median`) + +A metric that references columns from more than one dataset is attached to an **anchor** model — the model that reaches every referenced dataset over the relationship-derived joins (chosen via the same logic as [`recommend_root_model`](../concepts/queries.md#choosing-a-root-model)). Cross-dataset columns are emitted as join-qualified dotted refs (`customers.regions.population:sum`). + +## Dialect Selection + +OSI expressions are multi-dialect. `--dialect` (default `ANSI_SQL`) picks which one to read; when it is absent, the importer falls back to another SQL-compatible dialect (`SNOWFLAKE`, `DATABRICKS`). An expression available only in a non-SQL dialect (`MDX`, `MAQL`, `TABLEAU`) is clean-failed to the report. + +## Clean-Fail Report + +Anything that cannot be expressed exactly is reported (never silently dropped) — each entry lists the entity, a reason, and any workaround. Examples: a metric with a top-level `CASE` (outside any aggregate) or a window function, a relationship with mismatched key lengths or an unknown target, a dataset whose table cannot be introspected, an illegal name (containing `.`/`:`), a metric whose referenced datasets are not connected by any join path, or an orphan `COUNT(*)` (a column-less metric in a semantic model with no unique fact table, so its grain is ambiguous). The CLI prints a grouped report and a `models / unconverted / dropped` tally at the end. + +A query source is introspected live (a `LIMIT 0` / cursor-metadata probe) for real column types, the same way table sources are — no connection-less heuristics. diff --git a/docs/reference/cli.md b/docs/reference/cli.md index 97b55c40..010d3f26 100644 --- a/docs/reference/cli.md +++ b/docs/reference/cli.md @@ -109,6 +109,21 @@ slayer import-dbt ./my_dbt_project --datasource my_postgres --include-hidden-mod | `dbt_project_path` | Yes | Path to the dbt project root (or a models directory) | | `--datasource` | Yes | SLayer datasource name for the imported models | | `--include-hidden-models` | No | Also import regular dbt models (those not wrapped by a `semantic_model`) as hidden SLayer models via SQL introspection. Requires the `dbt` extra (`pip install 'motley-slayer[dbt]'`). See [dbt Import](../dbt/dbt_import.md#regular-dbt-models-hidden-import). | + +### `slayer import-osi` + +Import OSI (Open Semantic Interchange) configs into SLayer. See [Importing OSI configs](../osi/osi_import.md). + +```bash +slayer import-osi ./osi_configs --datasource my_postgres +slayer import-osi ./model.yaml --datasource my_postgres --dialect SNOWFLAKE +``` + +| Flag | Required | Description | +|------|----------|-------------| +| `osi_path` | Yes | Path to an OSI file or directory (`.yaml`/`.yml`/`.json`) | +| `--datasource` | Yes | SLayer datasource name (must be reachable — column types come from live introspection) | +| `--dialect` | No | OSI expression dialect to read (default `ANSI_SQL`); falls back to another SQL dialect when the requested one is absent | | `--storage` | No | Storage path | ### `slayer models` @@ -157,6 +172,133 @@ slayer datasources create demo --ingest # bundled Jaffle Shop demo | `-y`, `--yes` | No | Overwrite existing datasource / colliding models without prompting | | `--storage` | No | Storage path | -The demo path generates a DuckDB at `/demo/jaffle_shop.duckdb` and is idempotent — re-running reuses the existing file. `duckdb` and `jafgen` are core dependencies of `motley-slayer`, so the demo works after a single `pip install motley-slayer` with no extras needed. +The demo path generates a DuckDB at `/demo/jaffle_shop.duckdb` and is idempotent — re-running reuses the existing file. Ingested demo models are enriched with curated column labels/descriptions, currency and percent formats, saved measures (e.g. `orders.total_revenue`, `orders.avg_order_value`, `orders.effective_tax_rate`), and a `weighted_avg` custom-aggregation example on `orders`. The enrichment is additive-only: labels/descriptions are filled only where unset and existing measures are never overwritten, so user edits survive re-runs. `duckdb` and `jafgen` are core dependencies of `motley-slayer`, so the demo works after a single `pip install motley-slayer` with no extras needed. If a datasource with the same name already exists, or (with `--ingest`) any generated model name collides with a stored model, SLayer prompts for confirmation. Use `--yes` for non-interactive use. + +### `slayer inspect` + +Point-lookup of an entity by reference and kind — no ranking, no bundled +memories (use `slayer search` for an entity *in context*). Pass **two or more** +references to inspect several entities of the **same kind** in one call +(DEV-1612): the output is one `## ` block per reference, in input +order (a JSON array under `--format json`), with per-reference error isolation. +**Omit the reference entirely** (DEV-1667) to list the whole **collection** at +`--type` — supported for `model` and `datasource` only. `--type model` lists all +models grouped by datasource (a terse one line per model by default; `--no-compact` +gives the full per-model tables); `--type datasource` lists all datasources. + +```bash +slayer inspect jaffle_shop.orders --type model +slayer inspect jaffle_shop.orders.order_total --type column --no-compact +slayer inspect jaffle_shop.orders.customers.region --type column --no-compact # join path → owning model +slayer inspect memory:42 --type memory --no-compact +slayer inspect jaffle_shop.orders --type model --format json +slayer inspect jaffle_shop.orders.order_total jaffle_shop.orders.order_id --type column --no-compact # batch +slayer inspect --type model # collection: all models, grouped by datasource +slayer inspect --type datasource # collection: all datasources +``` + +| Flag | Required | Description | +|------|----------|-------------| +| `reference` | No | Zero or more entity references: canonical id, bare name, join path (resolved to the owning model), or `memory:`. Two or more → a same-kind batch. **Omit entirely** to list the whole collection at `--type` (`model` or `datasource` only). | +| `--type` | Yes | Entity kind: `datasource`, `model`, `column`, `measure`, `aggregation`, or `memory`. Disambiguates same-named entities and asserts the kind. | +| `--no-compact` | No | Return the full render. The compact default is description-only for column/measure/aggregation/datasource/memory, and a cheap **schema skeleton** (column/measure/aggregation names + join targets, zero DB calls) for `--type model`; `--no-compact` on a datasource renders a per-model skeleton for each visible model. | +| `--format` | No | `markdown` (default) or `json`. | +| `--num-rows` | No | (model only) Sample-data rows. Ignored with a warning for other kinds. | +| `--show-sql` | No | (model only) Include generated SQL. No-op for column/measure/aggregation; warned for datasource/memory. | +| `--section` | No | (model only, repeatable) Restrict to a section subset. Ignored with a warning for other kinds. | +| `--descriptions-max-chars` | No | Truncate description fields. Applies to every kind. | +| `--storage` | No | Storage path. | + +### `slayer search` + +Run semantic search over memories and canonical entities (datasources, models, columns, named measures, custom aggregations). Three retrieval channels run in parallel — BM25 over memory entity tags, Tantivy full-text over memories ∪ entities, and (with the `advanced_search` extra plus a provider API key) dense embeddings — and are RRF-fused into a single ranked list. See [Search](../concepts/search.md). + +```bash +# Entity-driven +slayer search --entity jaffle_shop.orders.order_total + +# Question-driven +slayer search --question "What stores are in jaffle_shop?" + +# Query-driven (auto-extracts the entities the query references) +slayer search --query @draft_query.json + +# Inline query JSON +slayer search --query '{"source_model": "orders", "measures": ["order_total:sum"]}' + +# Narrow to one datasource +slayer search --question "lifetime spend" --datasource jaffle_shop + +# Graph-narrow with cypher_filter (naive form, always available) +slayer search --question "Brooklyn POS" --cypher-filter 'MATCH (n:Memory) RETURN n.id AS id' + +# Graph-narrow with cypher_filter (full openCypher; requires the advanced_search extra) +slayer search --question "store rev" --cypher-filter \ + "MATCH (d:Datasource {name: 'jaffle_shop'})-[:CONTAINS]->(m:Model)-[:CONTAINS]->(c:ModelColumn) RETURN c.id AS id" + +# JSON output for piping +slayer search --question "lifetime spend" --format json +``` + +| Flag | Default | Description | +|------|---------|-------------| +| `--entity ENT` (repeatable) | | Canonical entity string (``, `.`, `..`, `memory:`). Pass multiple times to combine. Drives the BM25 channel. | +| `--query JSON_OR_@FILE` | | Inline SLayer query (or `@path.json`). Entities are auto-extracted from `source_model`, dimensions, measures, time dims, and filters. | +| `--question TEXT` | | Free-text question. Drives Tantivy + embeddings. | +| `--datasource DS` | | Pre-narrow every channel to ids rooted at the named datasource. Unknown name raises. | +| `--cypher-filter CYPHER` | | Pre-narrow all three channels via a graph query. Full openCypher with `advanced_search` (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges). Without the extra, only the naive `MATCH (n:Label1:Label2…) RETURN n.id AS id` form is accepted; anything richer raises with an install hint. | +| `--max-results N` | `10` | Cap applied after RRF fusion and the `cypher_filter` allowlist. | +| `--format` | `text` | `text` (newline-grouped human output) or `json` (full `SearchResponse`). | + +In `text` mode each result row prints only `kind`, `id`, `score`, and a one-line preview of the hit's `text`. The full column snapshot — the top-50 `sampled_values`, JSON-encoded inside the hit's `text` — is returned in full only under `--format json`; columns with more than 50 distinct values surface only that top 50, with no exact total. (The `50+ distinct` overflow marker is a property of `Column.sampled` and shows in `slayer inspect` / `inspect_model` output, not in search results.) Memory hits with `query is not None` are saved example queries. Unresolved input entities surface as warnings rather than errors. + +### `slayer search refresh-samples` + +Re-profile and persist `Column.sampled` / `sampled_values` / `distinct_count` for table-backed models. Per-column failures are reported but do not abort. + +```bash +slayer search refresh-samples +slayer search refresh-samples --data-source jaffle_shop +slayer search refresh-samples --data-source jaffle_shop --model orders --model customers +``` + +| Flag | Default | Description | +|------|---------|-------------| +| `--data-source X` | all | Limit the refresh to one datasource. | +| `--model M` | all | Repeatable; limit to specific models. | + +### `slayer memory` + +Manage the agent-memory layer. See [Memories](../concepts/memories.md). + +```bash +# Save a learning (--entities is a single comma-separated string) +slayer memory save \ + --learning "orders.is_returned in {0,1,NULL}; treat NULL as not returned" \ + --entities jaffle_shop.orders.is_returned + +# Save with a pinned id and multiple entities +slayer memory save \ + --learning "Brooklyn POS changed late 2024" \ + --entities jaffle_shop.orders.order_total,jaffle_shop.stores.name \ + --id kb.brooklyn-pos + +# Save with an inline query (mutually exclusive with --entities) +slayer memory save \ + --learning "Top customers by lifetime spend" \ + --query @top_customers.json \ + --id kb.top-customers + +# Forget by id +slayer memory forget kb.brooklyn-pos +``` + +| Subcommand | Flag | Description | +|------------|------|-------------| +| `save` | `--learning TEXT` (required) | The free-form note. | +| `save` | `--entities ENT,ENT,…` | Comma-separated canonical entity strings. `memory:` is valid for cross-memory refs. Mutually exclusive with `--query`; one of the two is required. | +| `save` | `--query JSON_OR_@FILE` | Inline SLayer query (or `@path.json`). Entities are auto-extracted and the query is persisted on the memory. Mutually exclusive with `--entities`. | +| `save` | `--id ID` | User-pinned canonical memory id. Forbidden charset: `:`, `/`, `?`, `#`, whitespace, ASCII control. Omit to auto-allocate (`max(int-shaped id) + 1`). Duplicate id → unconditional upsert; `created_at` preserved. | +| `forget` | `` (positional) | Memory id. Cascade-strips every `memory:` reference to it from every other memory's `entities` list. | diff --git a/docs/reference/mcp.md b/docs/reference/mcp.md index fd23a53e..74d6eb19 100644 --- a/docs/reference/mcp.md +++ b/docs/reference/mcp.md @@ -80,8 +80,9 @@ claude mcp list | Tool | Description | |------|-------------| -| `models_summary` | Brief summary of all non-hidden models in a datasource: each model's name, description, a table of its **columns** and **measures** (named formulas), and the list of models it joins to. The Markdown form (default) shows just `name` + `description` per column; the JSON form (`format="json"`) additionally includes the column `type`. Neither form includes distinct values, sample data, or joined-model field expansion — call `inspect_model` for those. Params: `datasource_name`, `format` (default `"markdown"`; also `"json"`). | -| `inspect_model` | Complete view of a single model: metadata with row count (and a `**meta:**` bullet when the model has `meta` set), any model-level or column-level filters, **columns table** (with a `sampled` column — distinct values for string/boolean columns, `min .. max` for number/date/time columns — and a `meta` cell when set), **measures table** of named formulas (with `formula`, `label`, `description`, `meta`), custom aggregations (with `meta`), joins, all fields reachable via joins (default depth 5), and a sample-data table. Every Markdown table auto-prunes all-empty columns (so the `meta` column is hidden when no entity has meta) and collapses to a comma-separated backticked list when only one column remains. Params: `model_name`, `num_rows` (default 3), `show_sql` (default false — include SQL for the sample-data query, the custom-SQL block, model-level filters, the cached backing-query SQL, and aggregation formulas/param SQL), `format` (default `"markdown"`; also `"json"`), `sections` (subset of `["columns", "measures", "aggregations", "joins", "reachable_fields", "samples"]` — default `None`/`[]` renders all six; sections in the first four collapse to a one-line backticked CSV of names when omitted, `reachable_fields`/`samples` are dropped entirely, unknown names emit a footer warning. A non-empty list of *only* unknown names resolves to no sections — "all six" is reserved for `None`/`[]` so a typo can't silently trigger the full payload), `descriptions_max_chars` (when set, truncate each description longer than this with the suffix `"... [truncated]"` (prefixed by a space); applies to model, columns, measures, and aggregations; must be `>= 0`), `reachable_fields_depth` (max BFS depth in path segments — default 5, allowed range `[0, 20]`; ignored when `reachable_fields` is not in `sections`). When any section is trimmed, a quoted-Markdown footer at the end of the response lists what was shown / names-only / omitted, with a hint on how to fetch more. The JSON form mirrors this: trimmed sections appear as `
_names: [...]` siblings, fully omitted ones are absent, and top-level `omitted_sections`, `names_only_sections`, `unknown_sections` arrays are added when non-empty. | +| `models_summary` | Brief summary of all non-hidden models in a datasource: each model's name, description, a table of its **columns** and **measures** (named formulas), and the list of models it joins to. The Markdown form (default) shows just `name` + `description` per column; the JSON form (`format="json"`) additionally includes the column `type`. Neither form includes distinct values or sample data — call `inspect_model` for those. For multi-hop discovery (fields reachable via joins from a given model), use the `search` tool with `cypher_filter` for graph queries. Params: `datasource_name`, `format` (default `"markdown"`; also `"json"`). | +| `inspect` | Single-entity point lookup: returns the rendered detail for **exactly one** entity by `reference` + required `entity_type` (`datasource`/`model`/`column`/`measure`/`aggregation`/`memory`). No fusion/ranking/cypher and no bundled memories — use `search` instead when you want an entity surfaced *in context*. `reference` accepts canonical ids, bare names, join paths (`orders.customers.region` → resolved to the owning model), and `memory:`; it is normalised via the shared resolver and the normalised id is echoed in the JSON shape. `entity_type` disambiguates the 3-part canonical collision (a name shared by, e.g., a column and an aggregation) and asserts the resolved kind (mismatch → detailed error). Renders hidden entities. Params: `reference`, `entity_type`, `compact` (default true), `format` (`"markdown"` default, `"json"`), and the model-only `num_rows`/`show_sql`/`sections`/`descriptions_max_chars` (applied for `entity_type="model"`; `descriptions_max_chars` applies to every kind; the others are ignored with a warning for non-model kinds, `show_sql` a silent no-op for column/measure/aggregation). **`compact=True`** renders description-only for leaf/datasource/memory kinds, and a cheap **schema skeleton** (column/measure/aggregation **names** + join targets, **zero DB calls**) for the `model` kind — markdown always emits the four `Columns:`/`Measures:`/`Aggregations:`/`Joins to:` lines (`_(none)_` when empty); JSON always carries `column_names`/`measure_names`/`aggregation_names`/`joins_to`. **`compact=False`** reuses the full `inspect_model` rendering (sample rows, SQL, sections) for the `model` kind, and renders a per-model skeleton for each visible model (sorted by name; `models: [...]` in JSON) for the `datasource` kind. JSON `text` is present **iff non-empty** (omitted under `compact=True` for every kind). Three-tier escalation: `models_summary(compact)` (column count) < `inspect(model, compact=True)` (column names) < `inspect(model, compact=False)` (full). **Batch (DEV-1612):** `reference` also accepts a **list** of ids — a homogeneous-kind batch (one `entity_type` for all). A single `str` keeps single-id output byte-for-byte; a list returns one block per id in input order, each echoing its resolved canonical id (a `## ` header per block in markdown, a JSON array under `format="json"`). A one-element list is still batch-framed. Per-id resolution errors are isolated (in JSON a failed id is a `{"reference", "error"}` object). **Collection (DEV-1667):** omit `reference` (or pass `null`/`[]`) to list a whole kind. `entity_type="model"` lists all models grouped by datasource — `compact=True` (default) is a terse one-liner per model (`- \`name\` (N cols; joins: ...)` under a `# Datasource: \`\` — model(s)` header); `compact=False` is the full `models_summary` block per datasource. `entity_type="datasource"` lists all datasources (`compact=True` == `list_datasources`; `compact=False` adds descriptions + per-model skeletons). Only `model`/`datasource` support the collection view; other kinds raise. This subsumes `models_summary` / `list_datasources` (kept as thin aliases). JSON collection form is `{"entity_type", "collection": true, "datasources": [...], "warnings": []}`. | +| `inspect_model` | **DEPRECATED — use `inspect`.** Complete view of a single model: metadata with row count (and a `**meta:**` bullet when the model has `meta` set), any model-level or column-level filters, **columns table** (with a `sampled` column — distinct values for string/boolean columns, `min .. max` for number/date/time columns — and a `meta` cell when set), **measures table** of named formulas (with `formula`, `label`, `description`, `meta`), custom aggregations (with `meta`), direct joins, and a sample-data table. Every Markdown table auto-prunes all-empty columns (so the `meta` column is hidden when no entity has meta) and collapses to a comma-separated backticked list when only one column remains. Params: `model_name`, `num_rows` (default 3), `show_sql` (default false — include SQL for the sample-data query, the custom-SQL block, model-level filters, the cached backing-query SQL, and aggregation formulas/param SQL), `format` (default `"markdown"`; also `"json"`), `sections` (subset of `["columns", "measures", "aggregations", "joins", "samples", "learnings"]` — default `None`/`[]` renders all six; sections in the first four collapse to a one-line backticked CSV of names when omitted, `samples`/`learnings` are dropped entirely, unknown names emit a footer warning. A non-empty list of *only* unknown names resolves to no sections — "all six" is reserved for `None`/`[]` so a typo can't silently trigger the full payload), `descriptions_max_chars` (when set, truncate each description longer than this with the suffix `"... [truncated]"` (prefixed by a space); applies to model, columns, measures, and aggregations; must be `>= 0`). When any section is trimmed, a quoted-Markdown footer at the end of the response lists what was shown / names-only / omitted, with a hint on how to fetch more. The JSON form mirrors this: trimmed sections appear as `
_names: [...]` siblings, fully omitted ones are absent, and top-level `omitted_sections`, `names_only_sections`, `unknown_sections` arrays are added when non-empty. For multi-hop reachability use the `search` tool. | | `create_model` | Create a model from a table/SQL definition or from a query. Pass `sql_table`/`sql` with `columns` (and optional named-formula `measures`) for table-based, or pass `query` (a SLayer query dict) to save it as a query-backed model whose `columns` + `backing_query_sql` are populated by a save-time dry-run. | | `edit_model` | Edit an existing model in one call. Supports upsert for columns, measures, aggregations, and joins (create if new, update if existing). Also manages scalar metadata and filters. See params below. | | `delete_model` | Delete a model entirely. | @@ -91,6 +92,7 @@ claude mcp list | Tool | Description | |------|-------------| | `query` | Execute a semantic query. See [Queries](../concepts/queries.md) for format. | +| `query_nested` | Execute a multi-stage DAG of named sub-queries that can reference one another via `source_model` or `joins.target_model`. Companion to `query`; the engine auto-sorts the list (Kahn's algorithm), so order doesn't matter. Params: `queries: List[Dict[str, Any]]`, plus `variables` / `show_sql` / `dry_run` / `explain` / `format` mirroring `query`. See [Multistage Queries](../examples/06_multistage_queries/multistage_queries.md). | **`query` parameters:** @@ -105,11 +107,77 @@ claude mcp list | `limit` | int | Max rows | | `offset` | int | Skip rows | | `whole_periods_only` | bool | Snap date filters to time bucket boundaries, exclude the current incomplete time bucket | +| `distinct_dimension_values` | bool | Default `true` — auto-dedup dim-only queries (`GROUP BY `). Set `false` to emit raw rows (no top-level `GROUP BY`); rejects any measure reference in `measures` / `filters` / `order`. | | `show_sql` | bool | Include the generated SQL in the response for debugging | | `dry_run` | bool | Generate and return the SQL without executing it | | `explain` | bool | Run EXPLAIN ANALYZE and return the query plan | | `format` | string | Output format: `"markdown"` (default, compact), `"json"` (structured), or `"csv"` (most compact). Case-insensitive | +### Memories + semantic search + +Memories are free-form notes the agent saves against canonical entity strings (``, `.`, `..`, or `memory:`). `search` is the only retrieval surface and returns memories **and** entity discovery hits in one flat list. See [Memories](../concepts/memories.md) and [Search](../concepts/search.md). + +| Tool | Description | +|------|-------------| +| `search` | Up to three-channel retrieval over memories and canonical entities (datasource / model / column / measure / aggregation), RRF-fused (k=60) into a single ranked list. | +| `save_memory` | Persist a free-form `learning` tagged with canonical entities or an inline `SlayerQuery`. | +| `forget_memory` | Delete a memory by id. Cascade-strips every other memory's `memory:` ref to it. | + +**`search` parameters:** + +| Param | Type | Description | +|-------|------|-------------| +| `entities` | list[str] | Canonical entity strings (`mydb.orders.amount`, `memory:42`, …) or aggregated colon forms (`revenue:sum` — the suffix is stripped). Drives the BM25 channel. Unresolved tokens emit warnings, not errors. | +| `query` | dict \| SlayerQuery | Inline query; its `source_model`, dimensions, measures, time dims, and filters are walked for canonical entities. | +| `question` | str | Free-text question. Drives the Tantivy full-text channel and (when available) the dense-embedding channel. | +| `datasource` | str | When set, every channel pre-filters to canonical ids rooted at that datasource. Unknown name → error. | +| `cypher_filter` | str | Graph pre-filter applied to all three channels. Full openCypher when the `advanced_search` extra is installed (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges; read-only — `CREATE`, `MERGE`, `DELETE`, `SET`, `REMOVE`, `DROP`, `CALL` are rejected). Without the extra, only the naive form `MATCH (n:Label1:Label2…) RETURN n.id AS id` is accepted as a label/kind filter; anything richer raises with an install hint. | +| `max_results` | int | Cap applied **after** RRF fusion and after the `cypher_filter` narrowing, so it counts surviving items only. Default `10`. | + +**Response shape (`SearchResponse`):** + +```json +{ + "results": [ + {"kind": "memory", "id": "42", "score": 0.13, "text": "...", "matched_entities": ["mydb.orders.amount"], "query": null}, + {"kind": "column", "id": "mydb.orders.amount", "score": 0.11, "text": "...", "matched_entities": [], "query": null}, + {"kind": "model", "id": "mydb.orders", "score": 0.09, "text": "...", "matched_entities": [], "query": null} + ], + "resolved_input_entities": ["mydb.orders.amount"], + "warnings": [] +} +``` + +`kind` is one of `"memory"`, `"datasource"`, `"model"`, `"column"`, `"measure"`, `"aggregation"`. For memory hits, `id` is the raw memory id (suitable for `forget_memory`); `hit.query is not None` marks a saved example query. Column hits carry the column's structured sample-value snapshot — the top 50 `sampled_values` are rendered as a JSON array (so values containing commas survive); overflow columns (> 50 distinct) are marked `50+ distinct` in the text snapshot. `SearchService` refreshes any column hit whose profile is stale on the fly. + +**`save_memory` parameters:** + +| Param | Type | Description | +|-------|------|-------------| +| `learning` | str | The free-form note. | +| `linked_entities` | list[str] \| dict (SlayerQuery) | Canonical entity strings (strict resolution; `memory:` is valid for cross-memory refs) **or** an inline `SlayerQuery` dict whose entities are auto-extracted and which is persisted on the memory. | +| `id` | str (optional) | User-pinned canonical memory id. Forbidden charset: `:`, `/`, `?`, `#`, whitespace, ASCII control. Omit to let the allocator assign the next int-shaped id (`max(int-shaped id) + 1`, never less than `"1"`). Duplicate id → unconditional upsert; `created_at` is preserved. | + +**`forget_memory` parameters:** + +| Param | Type | Description | +|-------|------|-------------| +| `id` | str | Memory id. Cascade strips every `memory:` ref to it from every other memory's `entities` list and drops the matching embedding row. | + +**Cypher filter examples.** Naive form (always available): + +``` +MATCH (n:Memory) RETURN n.id AS id # memory hits only +MATCH (n:Column:Measure) RETURN n.id AS id # column + named-measure hits only +``` + +Full openCypher (requires `advanced_search`): + +``` +MATCH (d:Datasource {name: 'mydb'})-[:CONTAINS]->(m:Model)-[:CONTAINS]->(c:ModelColumn) +RETURN c.id AS id +``` + ## Typical Agent Workflows ### Connect and explore a new database diff --git a/docs/reference/python-client.md b/docs/reference/python-client.md index 33b398b1..18759733 100644 --- a/docs/reference/python-client.md +++ b/docs/reference/python-client.md @@ -66,6 +66,15 @@ client.query_sync([ # Run-by-name (query-backed model) client.query_sync("rev_by_region") + +# Raw rows — opt out of the dim-only auto-dedup. Per-stage in DAG queries. +client.query_sync({ + "source_model": "orders", + "dimensions": ["status", "amount"], + "filters": ["amount > 100"], + "limit": 100, + "distinct_dimension_values": False, +}) ``` ### Other Methods @@ -87,6 +96,98 @@ datasources = client.list_datasources() client.create_datasource({"name": "mydb", "type": "postgres", ...}) ``` +### Inspect + +`inspect` / `inspect_sync` is a point lookup (DEV-1588): the rendered detail for **exactly one** entity by `reference` + required `entity_type`. No fusion / ranking / bundled memories — use `search` for an entity *in context*. Same arguments as the MCP `inspect` tool and `POST /inspect`; returns the rendered string. **DEV-1612:** `reference` also accepts a **list** — a homogeneous-kind batch (one `entity_type` for every id), returning one block per id in input order with per-id error isolation. **DEV-1667:** `reference=None` (or `[]`) is the **collection** view — lists the whole kind (`model` grouped by datasource, or `datasource`); other kinds raise. Subsumes `models_summary` / `list_datasources`. + +```python +# Compact default: schema skeleton for a model (column / measure / aggregation +# names + joins, zero DB calls); description-only for the other kinds. +print(client.inspect_sync(reference="mydb.orders", entity_type="model")) + +# Collection: all models grouped by datasource (one terse line per model). +print(client.inspect_sync(reference=None, entity_type="model")) + +# Full render of one column (compact=False); join paths resolve to the owner. +print(client.inspect_sync( + reference="mydb.orders.customers.region", entity_type="column", + compact=False, +)) + +# Batch: several same-kind columns in one round-trip (DEV-1612). +print(client.inspect_sync( + reference=["mydb.orders.amount", "mydb.orders.customer_id"], + entity_type="column", compact=False, +)) + +# async form: await client.inspect(reference="mydb.orders", entity_type="model") +``` + +`entity_type` is required (`datasource` / `model` / `column` / `measure` / `aggregation` / `memory`) and asserts the resolved kind. The model-only `num_rows` / `show_sql` / `sections` apply for `entity_type="model"`; `descriptions_max_chars` applies to every kind. `format="json"` returns a JSON string instead of Markdown. + +### Memories + Semantic Search + +`SlayerClient` exposes the same single retrieval surface as MCP / REST. All three are async (`run_sync` wraps them for synchronous use); local mode (`storage=`) goes through `SearchService` / `MemoryService` directly, remote mode (`url=`) POSTs to `/search` and `/memories`. See [Search](../concepts/search.md) and [Memories](../concepts/memories.md). + +```python +from slayer.async_utils import run_sync + +# Save a learning +run_sync(client.save_memory( + learning="orders.is_returned in {0,1,NULL}; treat NULL as not returned", + linked_entities=["mydb.orders.is_returned"], + id="kb.returns.null-handling", # optional; auto-allocated if omitted +)) + +# Save a query-bearing memory — pass a SlayerQuery / dict for linked_entities +run_sync(client.save_memory( + learning="Top customers by lifetime spend", + linked_entities={ + "source_model": "orders", + "measures": [{"formula": "amount:sum", "name": "lifetime_spend"}], + "dimensions": ["customers.name"], + "order": [{"column": "lifetime_spend", "direction": "desc"}], + "limit": 5, + }, + id="kb.top-customers", +)) + +# Search +resp = run_sync(client.search( + question="What should I know before comparing Brooklyn revenue to other stores?", + max_results=10, +)) + +for hit in resp.results: + if hit.kind == "memory": + kind = "example_query" if hit.query is not None else "learning" + print(f"[{kind}] {hit.id} score={hit.score:.3f} {hit.text[:80]}") + else: + print(f"[{hit.kind}] {hit.id} score={hit.score:.3f}") + +# Forget by id (cascade-strips memory: refs from other memories) +run_sync(client.forget_memory("kb.returns.null-handling")) +``` + +`client.search` signature (keyword-only): + +```python +async def search( + self, + *, + entities: Optional[List[str]] = None, + query: Optional[Union[SlayerQuery, Dict[str, Any]]] = None, + question: Optional[str] = None, + datasource: Optional[str] = None, + max_results: int = 10, + cypher_filter: Optional[str] = None, +) -> SearchResponse: ... +``` + +`cypher_filter` accepts full openCypher when the `advanced_search` extra is installed (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges; mutation clauses rejected). Without the extra, only the naive form `MATCH (n:Label1:Label2…) RETURN n.id AS id` is accepted as a label/kind filter — anything richer raises with an install hint. + +`SearchResponse` carries a single flat ranked list. Each `SearchHit` has `kind` (`"memory"` / `"datasource"` / `"model"` / `"column"` / `"measure"` / `"aggregation"`), `id`, `score`, `text`, `matched_entities`, and `query` (the attached `SlayerQuery` for query-bearing memories, else `None`). Unresolved input tokens land in `SearchResponse.warnings` instead of raising. Column hits include the structured `sampled_values` snapshot (top 50 by frequency, JSON-encoded; overflow columns are marked `50+ distinct` in the text snapshot); stale column profiles are refreshed lazily inside `search()`. + ## Direct Engine Access For maximum control, use the query engine directly: diff --git a/docs/reference/rest-api.md b/docs/reference/rest-api.md index 62080125..1ee62bb4 100644 --- a/docs/reference/rest-api.md +++ b/docs/reference/rest-api.md @@ -53,6 +53,13 @@ Response: } ``` +The body accepts the same fields as a `SlayerQuery`, plus `dry_run`, `explain`, and `variables`. Notable optional fields: + +- `whole_periods_only` (bool) — snap date filters to bucket boundaries. +- `distinct_dimension_values` (bool, default `true`) — set `false` to emit raw rows (no top-level `GROUP BY`); rejects any measure reference in `measures` / `filters` / `order`. + +Multi-stage DAG bodies use `{"queries": [...]}` — each stage in the list honours its own `distinct_dimension_values`. + ### Models ``` @@ -89,7 +96,7 @@ DELETE /datasources/{name} # Delete a datasource # List datasources curl http://localhost:5143/datasources -# Get datasource (password/connection_string shown as ***) +# Get datasource (credential fields shown as ***) curl http://localhost:5143/datasources/my_postgres ``` @@ -113,3 +120,108 @@ Response: "models": ["orders", "customers", "products"] } ``` + +### Inspect (single-entity point lookup) + +`POST /inspect` returns the rendered detail for **exactly one** entity by `reference` + required `entity_type` (DEV-1588). Unlike `POST /search`, there is no fusion / ranking / `cypher_filter` and no bundled memories — use `/search` when you want an entity surfaced *in context*. + +**`POST /inspect` body:** + +| Field | Type | Description | +|-------|------|-------------| +| `reference` | str \| array[str] \| null | Canonical id (`mydb.orders.amount`), bare name, join path (`orders.customers.region` → resolved to the owning model), or `memory:`. Normalised via the shared resolver; the normalised id is echoed in the result. **A list is a homogeneous-kind batch** (DEV-1612): one `entity_type` for every id; `result` is one block per id in input order (a `## ` header per block in markdown, a JSON-array string under `format="json"`). Per-id resolution errors are isolated; a non-string member returns HTTP 422. **`null` / omitted (or `[]`) is the collection view** (DEV-1667): lists the whole kind at `entity_type` — `model` (all models grouped by datasource) or `datasource` (all datasources); other kinds return HTTP 400. `compact` toggles verbosity; the JSON `result` is a `{"entity_type", "collection": true, "datasources": [...], "warnings": []}` envelope string. Subsumes `models_summary` / `list_datasources`. | +| `entity_type` | str | **Required.** One of `datasource`, `model`, `column`, `measure`, `aggregation`, `memory`. Disambiguates the 3-part canonical collision (a name shared by, e.g., a column and an aggregation) and asserts the kind — a mismatch returns HTTP 400. | +| `compact` | bool | Default `true`. Description-only for column / measure / aggregation / datasource / memory; a cheap schema **skeleton** (column / measure / aggregation names + join targets, zero DB calls) for `model`. `false` returns the full render, and a per-model skeleton for each visible model for `datasource`. | +| `format` | str | `"markdown"` (default) or `"json"`. | +| `num_rows` | int | Sample-data rows for `entity_type="model"`. Ignored (with a warning) for other kinds. Default `3`. | +| `show_sql` | bool | Include generated SQL for `entity_type="model"`. Ignored (with a warning) for datasource / memory; a silent no-op for column / measure / aggregation. | +| `sections` | array[str] | Section subset for `entity_type="model"` (`columns` / `measures` / `aggregations` / `joins` / `samples` / `learnings`). Ignored (with a warning) for other kinds. | +| `descriptions_max_chars` | int | Truncate every description field to this many characters (must be `>= 0`). Applies to every kind. | + +Renders hidden entities (deliberate escape hatch). Unknown fields are rejected (HTTP 422); a bad `entity_type` / `format`, a negative `descriptions_max_chars`, or an unresolvable reference returns HTTP 400. + +```bash +curl -X POST http://localhost:5143/inspect \ + -H "Content-Type: application/json" \ + -d '{"reference": "jaffle_shop.orders", "entity_type": "model"}' +``` + +The response is always `{"result": }` — the rendered Markdown (or, with `format="json"`, a JSON string): + +```json +{"result": "# `orders`\nOne row per placed order.\nColumns: id, order_total, ...\nMeasures: ...\nAggregations: _(none)_\nJoins to: stores"} +``` + +### Memories + Semantic Search + +`POST /search` is the single retrieval surface. It returns memories **and** entity discovery hits in one flat list, fused across up to three channels (BM25 over memory entity tags, Tantivy full-text, and — when `motley-slayer[advanced_search]` is installed and a provider API key is set — dense embeddings) via Reciprocal Rank Fusion (`k=60`). See [Search](../concepts/search.md) and [Memories](../concepts/memories.md). + +``` +POST /search # Run a semantic search +POST /memories # Persist a memory +DELETE /memories/{id} # Delete a memory (cascade-strips memory: refs) +``` + +**`POST /search` body:** + +| Field | Type | Description | +|-------|------|-------------| +| `entities` | array[str] | Canonical entity strings (`mydb.orders.amount`, `memory:42`, …). Aggregation suffixes are stripped (`revenue:sum` → `mydb.orders.revenue`). Drives the BM25 channel. Unresolved tokens emit warnings rather than errors. | +| `query` | object | Inline SLayer query; its `source_model` / dimensions / measures / time dims / filters are walked for canonical entities. | +| `question` | str | Free-text question. Drives the Tantivy channel and (when available) the embedding channel. | +| `datasource` | str | Pre-narrows every channel to ids rooted at the named datasource. Unknown name → HTTP 400. | +| `cypher_filter` | str | Graph pre-filter applied to all three channels. Full openCypher when `advanced_search` is installed (LadybugDB property graph with `Memory` / `Datasource` / `Model` / `ModelColumn` / `Measure` / `Aggregation` nodes and `MENTIONS` / `CONTAINS` / `JOINS` edges; read-only — mutation clauses rejected). Without the extra, only the naive form `MATCH (n:Label1:Label2…) RETURN n.id AS id` is accepted as a label/kind filter; anything richer returns HTTP 400 with an install hint. | +| `max_results` | int | Applied **after** RRF fusion and after the `cypher_filter` allowlist. Default `10`. | + +```bash +curl -X POST http://localhost:5143/search \ + -H "Content-Type: application/json" \ + -d '{ + "question": "What should I know before comparing Brooklyn revenue to other stores?", + "max_results": 10 + }' +``` + +Response (`SearchResponse`): + +```json +{ + "results": [ + {"kind": "memory", "id": "42", "score": 0.13, "text": "Brooklyn switched POS in late 2024 …", "matched_entities": [], "query": null}, + {"kind": "column", "id": "jaffle_shop.orders.order_total", "score": 0.11, "text": "...\nSample values: [\"100.00\", \"42.50\", …]", "matched_entities": [], "query": null}, + {"kind": "model", "id": "jaffle_shop.stores", "score": 0.09, "text": "...", "matched_entities": [], "query": null} + ], + "resolved_input_entities": [], + "warnings": [] +} +``` + +`kind` is one of `"memory"`, `"datasource"`, `"model"`, `"column"`, `"measure"`, `"aggregation"`. For memory hits, `id` is the raw memory id (suitable for `DELETE /memories/{id}`); `query` carries the saved `SlayerQuery` when the memory is query-bearing. Column hits embed the structured `sampled_values` (top 50 by frequency, JSON-encoded; overflow columns are marked `50+ distinct` in the text snapshot); stale profiles are refreshed lazily inside `/search`. + +**`POST /memories` body:** + +```json +{ + "learning": "orders.is_returned in {0,1,NULL}; treat NULL as not returned", + "linked_entities": ["mydb.orders.is_returned"], + "id": "kb.returns.null-handling" +} +``` + +`linked_entities` accepts either an array of canonical entity strings (strict resolution; `memory:` valid for cross-memory refs) **or** an inline `SlayerQuery` dict — the entities are auto-extracted and the query is persisted on the memory. `id` is optional; omit to auto-allocate (`max(int-shaped id) + 1`). Forbidden charset on user-supplied ids: `:`, `/`, `?`, `#`, whitespace, ASCII control. Duplicate id → unconditional upsert; `created_at` preserved. + +Response: + +```json +{"memory_id": "kb.returns.null-handling", "entities": ["mydb.orders.is_returned"], "warnings": []} +``` + +**`DELETE /memories/{id}`:** + +```bash +curl -X DELETE http://localhost:5143/memories/kb.returns.null-handling +``` + +Removes the memory and the matching embedding row, and strips every `memory:` reference to it from every other memory's `entities` list (exact-match only — `memory:42` does not strip `memory:421`). + +A second cleanup pass runs on `slayer ingest` / `--ingest-on-startup` (best-effort, transient failures keep the ref intact). diff --git a/docs/requirements.txt b/docs/requirements.txt deleted file mode 100644 index ba36cf52..00000000 --- a/docs/requirements.txt +++ /dev/null @@ -1,2 +0,0 @@ -mkdocs-jupyter>=0.24.0 -mkdocs-section-index>=0.3.0 diff --git a/examples/bigquery/README.md b/examples/bigquery/README.md new file mode 100644 index 00000000..69ca5907 --- /dev/null +++ b/examples/bigquery/README.md @@ -0,0 +1,94 @@ +# BigQuery Example — `bigquery-public-data.thelook_ecommerce` + +Run SLayer against a real BigQuery dataset. No Docker — BigQuery is a managed +service, so the example just points SLayer at the public dataset and starts the +API server. + +## Prerequisites + +- A Google Cloud project (any project; it's only used for billing the BQ jobs). +- A service account in that project with the `roles/bigquery.jobUser` role. + - The dataset (`bigquery-public-data.thelook_ecommerce`) is world-readable; + no extra grant is needed on it. +- A JSON key file for that service account. + +## Configure auth + +SLayer reads BigQuery credentials via Google Application Default Credentials, +the same as every other Google client library. Set two env vars: + +```bash +export GCP_PROJECT_ID="your-billing-project-id" +export GOOGLE_APPLICATION_CREDENTIALS="/absolute/path/to/sa-key.json" +``` + +The datasource YAML (`slayer_data/datasources/thelook.yaml`) interpolates +`$GCP_PROJECT_ID` into the connection string; the `google-cloud-bigquery` +client auto-picks up `$GOOGLE_APPLICATION_CREDENTIALS` at query time. + +> **Never commit your JSON key.** The `.gitignore` in this directory blocks +> `*.json` to make accidents loud. + +## Install the BigQuery extra + +```bash +poetry install -E bigquery +# or +pip install "motley-slayer[bigquery]" +``` + +This pulls in `sqlalchemy-bigquery` + `google-cloud-bigquery`. + +## Start the server + +```bash +cd examples/bigquery +./start.sh +``` + +## Verify + +In another terminal: + +```bash +python examples/bigquery/verify.py +``` + +The verify script runs identity-style checks (sum-of-grouped == ungrouped +total, min ≤ avg ≤ max, etc.) rather than hardcoded row counts — the public +dataset isn't strictly frozen. + +## Models + +Four hand-authored models map the thelook tables: + +| Model | Backing table | +|---------------|--------------------------------------------------------| +| `orders` | `bigquery-public-data.thelook_ecommerce.orders` | +| `order_items` | `bigquery-public-data.thelook_ecommerce.order_items` | +| `products` | `bigquery-public-data.thelook_ecommerce.products` | +| `users` | `bigquery-public-data.thelook_ecommerce.users` | + +Joins are declared explicitly (BigQuery has no FK metadata, so `slayer ingest` +auto-resolution cannot derive them). + +## Why not `slayer ingest`? + +`sqlalchemy-bigquery`'s Inspector lets you introspect schemas inside your +billing project. Cross-project public datasets aren't introspectable that way +— the project in the connection URL is fixed. For your own datasets in your +billing project, `slayer ingest` works fine; see [docs/concepts/ingestion.md]. + +## Try a query + +```bash +curl -X POST http://localhost:5143/query \ + -H "Content-Type: application/json" \ + -d '{ + "source_model": "order_items", + "measures": ["*:count", "sale_price:sum"], + "dimensions": ["products.category"], + "order": [{"column": "count", "direction": "desc"}], + "limit": 5 + }' +``` diff --git a/examples/bigquery/slayer_data/datasources/thelook.yaml b/examples/bigquery/slayer_data/datasources/thelook.yaml new file mode 100644 index 00000000..4d92a421 --- /dev/null +++ b/examples/bigquery/slayer_data/datasources/thelook.yaml @@ -0,0 +1,7 @@ +name: thelook +type: bigquery +connection_string: bigquery://${GCP_PROJECT_ID} +description: | + Read-only access to bigquery-public-data.thelook_ecommerce. Jobs run in the + GCP billing project named by $GCP_PROJECT_ID; credentials picked up from + $GOOGLE_APPLICATION_CREDENTIALS via Google Application Default Credentials. diff --git a/examples/bigquery/slayer_data/models/thelook/order_items.yaml b/examples/bigquery/slayer_data/models/thelook/order_items.yaml new file mode 100644 index 00000000..7a665421 --- /dev/null +++ b/examples/bigquery/slayer_data/models/thelook/order_items.yaml @@ -0,0 +1,62 @@ +version: 7 +name: order_items +sql_table: bigquery-public-data.thelook_ecommerce.order_items +query_variables: {} +data_source: thelook +columns: +- name: id + sql: id + type: INT + primary_key: true + hidden: false +- name: order_id + sql: order_id + type: INT + primary_key: false + hidden: false +- name: user_id + sql: user_id + type: INT + primary_key: false + hidden: false +- name: product_id + sql: product_id + type: INT + primary_key: false + hidden: false +- name: status + sql: status + type: TEXT + primary_key: false + hidden: false +- name: sale_price + sql: sale_price + type: DOUBLE + primary_key: false + hidden: false +- name: created_at + sql: created_at + type: TIMESTAMP + primary_key: false + hidden: false +measures: [] +aggregations: [] +joins: +- target_model: orders + join_pairs: + - - order_id + - order_id + join_type: left +- target_model: products + join_pairs: + - - product_id + - id + join_type: left +- target_model: users + join_pairs: + - - user_id + - id + join_type: left +filters: [] +default_time_dimension: created_at +hidden: false diff --git a/examples/bigquery/slayer_data/models/thelook/orders.yaml b/examples/bigquery/slayer_data/models/thelook/orders.yaml new file mode 100644 index 00000000..dfe498d3 --- /dev/null +++ b/examples/bigquery/slayer_data/models/thelook/orders.yaml @@ -0,0 +1,57 @@ +version: 7 +name: orders +sql_table: bigquery-public-data.thelook_ecommerce.orders +query_variables: {} +data_source: thelook +columns: +- name: order_id + sql: order_id + type: INT + primary_key: true + hidden: false +- name: user_id + sql: user_id + type: INT + primary_key: false + hidden: false +- name: status + sql: status + type: TEXT + primary_key: false + hidden: false +- name: gender + sql: gender + type: TEXT + primary_key: false + hidden: false +- name: created_at + sql: created_at + type: TIMESTAMP + primary_key: false + hidden: false +- name: shipped_at + sql: shipped_at + type: TIMESTAMP + primary_key: false + hidden: false +- name: delivered_at + sql: delivered_at + type: TIMESTAMP + primary_key: false + hidden: false +- name: num_of_item + sql: num_of_item + type: INT + primary_key: false + hidden: false +measures: [] +aggregations: [] +joins: +- target_model: users + join_pairs: + - - user_id + - id + join_type: left +filters: [] +default_time_dimension: created_at +hidden: false diff --git a/examples/bigquery/slayer_data/models/thelook/products.yaml b/examples/bigquery/slayer_data/models/thelook/products.yaml new file mode 100644 index 00000000..654dce2f --- /dev/null +++ b/examples/bigquery/slayer_data/models/thelook/products.yaml @@ -0,0 +1,46 @@ +version: 7 +name: products +sql_table: bigquery-public-data.thelook_ecommerce.products +query_variables: {} +data_source: thelook +columns: +- name: id + sql: id + type: INT + primary_key: true + hidden: false +- name: cost + sql: cost + type: DOUBLE + primary_key: false + hidden: false +- name: category + sql: category + type: TEXT + primary_key: false + hidden: false +- name: brand + sql: brand + type: TEXT + primary_key: false + hidden: false +- name: retail_price + sql: retail_price + type: DOUBLE + primary_key: false + hidden: false +- name: department + sql: department + type: TEXT + primary_key: false + hidden: false +- name: sku + sql: sku + type: TEXT + primary_key: false + hidden: false +measures: [] +aggregations: [] +joins: [] +filters: [] +hidden: false diff --git a/examples/bigquery/slayer_data/models/thelook/users.yaml b/examples/bigquery/slayer_data/models/thelook/users.yaml new file mode 100644 index 00000000..bb3544a8 --- /dev/null +++ b/examples/bigquery/slayer_data/models/thelook/users.yaml @@ -0,0 +1,47 @@ +version: 7 +name: users +sql_table: bigquery-public-data.thelook_ecommerce.users +query_variables: {} +data_source: thelook +columns: +- name: id + sql: id + type: INT + primary_key: true + hidden: false +- name: age + sql: age + type: INT + primary_key: false + hidden: false +- name: gender + sql: gender + type: TEXT + primary_key: false + hidden: false +- name: country + sql: country + type: TEXT + primary_key: false + hidden: false +- name: city + sql: city + type: TEXT + primary_key: false + hidden: false +- name: traffic_source + sql: traffic_source + type: TEXT + primary_key: false + hidden: false +- name: created_at + sql: created_at + type: TIMESTAMP + primary_key: false + hidden: false +measures: [] +aggregations: [] +joins: [] +filters: [] +default_time_dimension: created_at +hidden: false diff --git a/examples/bigquery/start.sh b/examples/bigquery/start.sh new file mode 100755 index 00000000..bf7f03ed --- /dev/null +++ b/examples/bigquery/start.sh @@ -0,0 +1,22 @@ +#!/usr/bin/env bash +# Boot SLayer against bigquery-public-data.thelook_ecommerce. +# +# Requires: +# GCP_PROJECT_ID Billing project where BQ jobs run. +# GOOGLE_APPLICATION_CREDENTIALS Path to service-account JSON key. +# +# The JSON key file should grant the SA the roles/bigquery.jobUser role on +# $GCP_PROJECT_ID. The public dataset itself needs no extra grant — it is +# world-readable. +set -euo pipefail + +: "${GCP_PROJECT_ID:?GCP_PROJECT_ID is required (your billing project)}" +: "${GOOGLE_APPLICATION_CREDENTIALS:?GOOGLE_APPLICATION_CREDENTIALS must point at a service-account JSON key}" + +if [[ ! -f "$GOOGLE_APPLICATION_CREDENTIALS" ]]; then + echo "GOOGLE_APPLICATION_CREDENTIALS=$GOOGLE_APPLICATION_CREDENTIALS does not exist" >&2 + exit 1 +fi + +HERE="$(cd "$(dirname "$0")" && pwd)" +exec slayer serve --host 0.0.0.0 --port 5143 --storage "$HERE/slayer_data" diff --git a/examples/bigquery/verify.py b/examples/bigquery/verify.py new file mode 100644 index 00000000..b078987e --- /dev/null +++ b/examples/bigquery/verify.py @@ -0,0 +1,235 @@ +"""Verification script for the BigQuery example. + +Run after starting the SLayer API server (see README.md): + + python examples/bigquery/verify.py + +Targets the read-only public dataset ``bigquery-public-data.thelook_ecommerce``. +Assertions avoid hardcoded row counts (the public dataset isn't strictly +frozen) — instead they prove semantic correctness: + - models load, queries succeed, joins work end-to-end + - sum of grouped counts equals the ungrouped total (cardinality invariant) + - aggregates return numbers in plausible ranges + - time dimension truncation works against TIMESTAMP columns +""" + +import json +import os +import sys +import urllib.error +import urllib.request + + +BASE_URL = os.environ.get("SLAYER_URL", "http://localhost:5143") + +# Repeated literals (hoisted per SonarCloud python:S1192) — these strings +# would otherwise show up nine and eight times respectively. +QUERY_PATH = "/query" +COUNT_MEASURE = "*:count" + +_passed = 0 +_failed = 0 + + +def api(*, method, path, body=None): + data = json.dumps(body).encode() if body else None + req = urllib.request.Request( + f"{BASE_URL}{path}", + data=data, + headers={"Content-Type": "application/json"} if data else {}, + method=method, + ) + try: + with urllib.request.urlopen(req) as resp: + return json.loads(resp.read()) + except urllib.error.HTTPError as e: + body_text = e.read().decode("utf-8", errors="replace") + print(f" HTTP {e.code} on {method} {path}", file=sys.stderr) + if body: + print(f" request body: {json.dumps(body)}", file=sys.stderr) + print(f" response body: {body_text}", file=sys.stderr) + raise + + +def check(*, name, condition): + global _passed, _failed + if condition: + print(f" PASS: {name}") + _passed += 1 + else: + print(f" FAIL: {name}") + _failed += 1 + + +def summary(): + print(f"\n{'=' * 40}") + print(f"Results: {_passed} passed, {_failed} failed") + if _failed: + sys.exit(1) + print("All checks passed!") + + +def main(): + print("API health:") + try: + health = api(method="GET", path="/health") + check(name="health endpoint", condition=health.get("status") == "ok") + except Exception as e: + print(f" FAIL: cannot reach {BASE_URL} — {e}", file=sys.stderr) + print("\nStart the server first: ./start.sh", file=sys.stderr) + sys.exit(1) + + print("\nModels:") + models = api(method="GET", path="/models") + names = {m["name"] for m in models} + for expected in ("orders", "order_items", "products", "users"): + check(name=f"{expected} model present", condition=expected in names) + + print("\nDatasource:") + datasources = api(method="GET", path="/datasources") + check( + name="thelook datasource registered", + condition=any(d["name"] == "thelook" for d in datasources), + ) + + # --- Baseline counts --------------------------------------------------- + print("\nBaseline counts:") + total_orders = api( + method="POST", path=QUERY_PATH, + body={"source_model": "orders", "measures": [COUNT_MEASURE]}, + )["data"][0]["orders._count"] + check(name="orders count > 0", condition=total_orders > 0) + print(f" (orders._count = {total_orders})") + + total_users = api( + method="POST", path=QUERY_PATH, + body={"source_model": "users", "measures": [COUNT_MEASURE]}, + )["data"][0]["users._count"] + check(name="users count > 0", condition=total_users > 0) + + total_products = api( + method="POST", path=QUERY_PATH, + body={"source_model": "products", "measures": [COUNT_MEASURE]}, + )["data"][0]["products._count"] + # thelook_ecommerce.products has roughly 29k rows; allow a wide band. + check( + name="products count in [1k, 1M]", + condition=1_000 < total_products < 1_000_000, + ) + + # --- Cardinality invariant (the SLayer "adding a field can't change + # cardinality" principle, exercised at the wire layer) ---------------- + print("\nCardinality invariant (sum-of-grouped == total):") + by_status = api( + method="POST", path=QUERY_PATH, + body={ + "source_model": "orders", + "measures": [COUNT_MEASURE], + "dimensions": ["status"], + }, + )["data"] + summed = sum(row["orders._count"] for row in by_status) + check( + name=f"sum(orders by status) == total ({summed} == {total_orders})", + condition=summed == total_orders, + ) + statuses = {row["orders.status"] for row in by_status} + # thelook statuses (stable over years): Complete, Processing, Shipped, + # Cancelled, Returned. Require at least three of them to be present so + # the check survives a future status rename. + expected_subset = {"Complete", "Processing", "Shipped", "Cancelled", "Returned"} + check( + name=f"order statuses include >= 3 of {sorted(expected_subset)}", + condition=len(statuses & expected_subset) >= 3, + ) + + # --- Joined query: order_items → products (category rollup) ---------- + print("\nJoin: order_items by product category:") + by_category = api( + method="POST", path=QUERY_PATH, + body={ + "source_model": "order_items", + "measures": [COUNT_MEASURE], + "dimensions": ["products.category"], + }, + )["data"] + check(name="by-category rows present", condition=len(by_category) > 0) + total_items = api( + method="POST", path=QUERY_PATH, + body={"source_model": "order_items", "measures": [COUNT_MEASURE]}, + )["data"][0]["order_items._count"] + summed_cat = sum(row["order_items._count"] for row in by_category) + check( + name=f"sum(items by category) == total ({summed_cat} == {total_items})", + condition=summed_cat == total_items, + ) + + # --- Transitive join: order_items → users → country ------------------- + print("\nTransitive join: order_items by user country:") + by_country = api( + method="POST", path=QUERY_PATH, + body={ + "source_model": "order_items", + "measures": [COUNT_MEASURE], + "dimensions": ["users.country"], + "order": [{"column": "count", "direction": "desc"}], + "limit": 5, + }, + )["data"] + check(name="top-5 countries returned", condition=len(by_country) == 5) + check( + name="countries are non-empty strings", + condition=all(row.get("order_items.users.country") for row in by_country), + ) + + # --- Aggregates on a numeric column ---------------------------------- + print("\nAggregates on order_items.sale_price:") + aggs = api( + method="POST", path=QUERY_PATH, + body={ + "source_model": "order_items", + "measures": [ + "sale_price:sum", + "sale_price:avg", + "sale_price:min", + "sale_price:max", + ], + }, + )["data"][0] + s = aggs["order_items.sale_price_sum"] + a = aggs["order_items.sale_price_avg"] + mn = aggs["order_items.sale_price_min"] + mx = aggs["order_items.sale_price_max"] + check(name="sum > 0", condition=s > 0) + check(name="min >= 0", condition=mn >= 0) + check(name="max > min", condition=mx > mn) + check(name="avg between min and max", condition=mn <= a <= mx) + + # --- Time dimension (BigQuery DATE_TRUNC on TIMESTAMP) --------------- + print("\nTime dimension (month bucket on orders.created_at):") + by_month = api( + method="POST", path=QUERY_PATH, + body={ + "source_model": "orders", + "measures": [COUNT_MEASURE], + "time_dimensions": [ + {"dimension": "created_at", "granularity": "month"}, + ], + "order": [{"column": "created_at", "direction": "asc"}], + "limit": 6, + }, + )["data"] + check(name="month-bucket rows returned", condition=len(by_month) >= 1) + if by_month: + first_bucket = by_month[0].get("orders.created_at") + # BQ emits TIMESTAMP truncated to month; SLayer surfaces it as ISO 8601. + check( + name="first bucket parseable", + condition=first_bucket is not None and ("T" in str(first_bucket) or " " in str(first_bucket)), + ) + + summary() + + +if __name__ == "__main__": + main() diff --git a/examples/mysql/verify.py b/examples/mysql/verify.py index 286ad3a9..da7cbe6b 100644 --- a/examples/mysql/verify.py +++ b/examples/mysql/verify.py @@ -14,6 +14,7 @@ from verify_common import ( run_common_checks, check_rollup, + check_corr_covar, check_stddev_var, check, summary, @@ -35,9 +36,9 @@ check("4 models without rollup", len(models) == 4) # MySQL has native STDDEV_SAMP/STDDEV_POP/VAR_SAMP/VAR_POP. DEV-1317 smoke. - # corr / covar_samp / covar_pop are NOT supported on MySQL — SLayer - # raises NotImplementedError there, so we deliberately don't call - # check_corr_covar() from this script. Use MariaDB for those. check_stddev_var() + # MySQL corr/covar_samp/covar_pop now use a variance-decomposition formula. + check_corr_covar() + summary() diff --git a/examples/seed.py b/examples/seed.py index a1fc79b7..4dac380f 100644 --- a/examples/seed.py +++ b/examples/seed.py @@ -52,6 +52,73 @@ ); """ +# T-SQL (SQL Server): TEXT is deprecated — use NVARCHAR; TIMESTAMP is a binary +# rowversion type — use DATETIME2 instead. +CREATE_SQL_TSQL = """ +CREATE TABLE regions ( + id INTEGER PRIMARY KEY, + name NVARCHAR(255) NOT NULL +); + +CREATE TABLE customers ( + id INTEGER PRIMARY KEY, + name NVARCHAR(255) NOT NULL, + email NVARCHAR(255) NOT NULL, + region_id INTEGER REFERENCES regions(id) +); + +CREATE TABLE products ( + id INTEGER PRIMARY KEY, + name NVARCHAR(255) NOT NULL, + category NVARCHAR(255) NOT NULL, + price NUMERIC(10,2) NOT NULL +); + +CREATE TABLE orders ( + id INTEGER PRIMARY KEY, + customer_id INTEGER REFERENCES customers(id), + product_id INTEGER REFERENCES products(id), + quantity INTEGER NOT NULL, + status NVARCHAR(50) NOT NULL, + created_at DATETIME2 NOT NULL +); +""" + +# Snowflake (DEV-1551): NUMBER(38,0) is Snowflake's canonical integer type; +# NUMBER(p,s)/DECIMAL(p,s) for fixed-point. TIMESTAMP_NTZ avoids the +# session-time-zone gotcha of bare TIMESTAMP. FK constraints are declarative +# only (not enforced) — but the Inspector still surfaces them, so +# auto-ingestion discovers joins. +CREATE_SQL_SNOWFLAKE = """ +CREATE TABLE regions ( + id NUMBER(38,0) PRIMARY KEY, + name TEXT NOT NULL +); + +CREATE TABLE customers ( + id NUMBER(38,0) PRIMARY KEY, + name TEXT NOT NULL, + email TEXT NOT NULL, + region_id NUMBER(38,0) REFERENCES regions(id) +); + +CREATE TABLE products ( + id NUMBER(38,0) PRIMARY KEY, + name TEXT NOT NULL, + category TEXT NOT NULL, + price NUMBER(10,2) NOT NULL +); + +CREATE TABLE orders ( + id NUMBER(38,0) PRIMARY KEY, + customer_id NUMBER(38,0) REFERENCES customers(id), + product_id NUMBER(38,0) REFERENCES products(id), + quantity NUMBER(38,0) NOT NULL, + status TEXT NOT NULL, + created_at TIMESTAMP_NTZ NOT NULL +); +""" + # ClickHouse uses MergeTree engine, no PRIMARY KEY constraint, no REFERENCES CREATE_SQL_CLICKHOUSE = """ CREATE TABLE regions ( @@ -86,8 +153,13 @@ def _get_create_sql(connection_string: str) -> str: """Return dialect-appropriate CREATE TABLE SQL.""" - if "clickhouse" in connection_string.lower(): + cs = connection_string.lower() + if "clickhouse" in cs: return CREATE_SQL_CLICKHOUSE + if "mssql" in cs or "sqlserver" in cs: + return CREATE_SQL_TSQL + if cs.startswith("snowflake://") or "snowflakecomputing" in cs: + return CREATE_SQL_SNOWFLAKE return CREATE_SQL_STANDARD @@ -208,8 +280,30 @@ def _get_create_sql(connection_string: str) -> str: def seed(connection_string: str) -> None: - """Create tables and insert seed data.""" - engine = sa.create_engine(connection_string) + """Create tables and insert seed data. + + Goes through ``engine_factory.get_engine`` (not raw + ``sa.create_engine``) so dialect runtime hooks fire — in particular + Snowflake's ``creator=`` bridge for the + ``snowflake://?connection_name=`` sentinel URL and the SQLite + UDF registration listener. + """ + from slayer.core.models import DatasourceConfig + from slayer.sql import engine_factory + + # Build a minimal DatasourceConfig from the URL so the dialect + # strategy class can route correctly. + from urllib.parse import urlparse + parsed = urlparse(connection_string) + ds_type = parsed.scheme.split("+", 1)[0].lower() + if ds_type == "postgresql": + ds_type = "postgres" + ds = DatasourceConfig( + name="seed_target", + type=ds_type, + connection_string=connection_string, + ) + engine = engine_factory.get_engine(ds) create_sql = _get_create_sql(connection_string) diff --git a/examples/snowflake/README.md b/examples/snowflake/README.md new file mode 100644 index 00000000..36d6714b --- /dev/null +++ b/examples/snowflake/README.md @@ -0,0 +1,95 @@ +# Snowflake example + +Snowflake is a Tier 1 dialect — full integration test coverage plus this verify +script. Unlike Postgres / MySQL / SQL Server / ClickHouse there is no +`docker-compose.yml`: Snowflake doesn't ship a free local image. You'll need a +real account. + +## 1. Install the extra + +```bash +pip install 'motley-slayer[snowflake]' +``` + +The extra pulls in `snowflake-connector-python` and `snowflake-sqlalchemy`. + +## 2. Configure a connection + +Edit `~/.snowflake/connections.toml`: + +```toml +[default] +account = "jp13593" # Snowflake account identifier (NOT a hostname) +user = "YOUR_USER" +password = "YOUR_PASSWORD" +warehouse = "COMPUTE_WH" +database = "SLAYER_DEMO" +schema = "PUBLIC" +``` + +Key-pair, OAuth, SSO, and MFA are all supported via the connector's standard +TOML keys — see [Snowflake docs](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-connect#using-connection-parameters). + +## 3. Seed the demo schema + +```bash +python ../seed.py "snowflake://?connection_name=default" +``` + +This drops + recreates the four canonical tables (`regions`, `customers`, +`products`, `orders`) and inserts the standard fixture dataset. + +## 4. Register the datasource and ingest + +```bash +slayer datasources create "snowflake://?connection_name=default" --name sf --ingest +``` + +Auto-ingestion walks the schema and creates one `SlayerModel` per table. +**Snowflake exposes declarative `FOREIGN KEY` constraints via its Inspector, +so join models are discovered automatically** — no manual `joins:` editing +required. + +## 5. Verify + +```bash +python verify.py +``` + +`verify.py` runs the same battery used by the other Tier 1 examples: +auto-ingestion + rollup joins + column-type assertions + aggregation matrix +(`median`, `percentile`, `stddev_samp/pop`, `var_samp/pop`, `corr`, `covar_samp/pop`). +Every aggregation is native on Snowflake; no formula fallbacks. + +## Connection forms + +The `connection_name=` URL is the recommended path — auth credentials stay in +`connections.toml`, and the connector handles key-pair / OAuth / SSO / MFA +transparently. An inline form is also supported: + +```yaml +# datasources/sf.yaml +name: sf +type: snowflake +host: jp13593 # Snowflake "account" goes in `host` +username: YOUR_USER +password: YOUR_PASSWORD +database: SLAYER_DEMO +schema_name: PUBLIC +warehouse: COMPUTE_WH +role: PUBLIC +``` + +Both forms flow through `engine_factory.get_engine`, which also wires a +per-connection `USE WAREHOUSE / USE ROLE / USE DATABASE / USE SCHEMA` listener +when those fields are set. + +## Known limitations + +- **`LIMIT 0` type probing compiles Snowflake queries** and consumes a small + amount of warehouse compute. SLayer doesn't yet use `DESCRIBE QUERY` for the + probe. +- **Identifier casing:** SLayer relies on Snowflake's case-insensitive + resolution of unquoted identifiers. Mixed-case names (`"Revenue"`) get + double-quoted by sqlglot and become case-sensitive — they must match the + stored case exactly. diff --git a/examples/snowflake/verify.py b/examples/snowflake/verify.py new file mode 100644 index 00000000..9b8866a1 --- /dev/null +++ b/examples/snowflake/verify.py @@ -0,0 +1,80 @@ +"""Verification script for the Snowflake example (DEV-1551). + +Run after seeding + ingesting: + python examples/seed.py "snowflake://?connection_name=default" + slayer datasources create "snowflake://?connection_name=default" --name sf --ingest + python examples/snowflake/verify.py + +Snowflake exposes declarative FK constraints via the Inspector, so the +rollup joins ARE generated (unlike ClickHouse / BigQuery). Every aggregation +in the matrix is native — no formula fallbacks like MySQL / SQL Server. +""" + +import os +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) +from verify_common import ( + check, + check_column_types, + check_corr_covar, + check_median_percentile, + check_rollup, + check_stddev_var, + run_common_checks, + summary, +) + +if __name__ == "__main__": + models = run_common_checks() + check("4 models", len(models) == 4) + # Snowflake DOES expose declarative FK constraints — rollup joins should + # auto-discover. Behaviour matches Postgres / MySQL / SQLite, NOT + # ClickHouse / BigQuery. + check_rollup(expect_rollup=True) + # NUMBER(38,0) → INT; NUMBER(10,2) → DOUBLE; TEXT → TEXT; + # TIMESTAMP_NTZ → TIMESTAMP. + check_column_types( + model_name="orders", + expected_types={ + "id": "INT", + "customer_id": "INT", + "product_id": "INT", + "quantity": "INT", + "status": "TEXT", + "created_at": "TIMESTAMP", + }, + ) + check_column_types( + model_name="customers", + expected_types={ + "id": "INT", + "name": "TEXT", + "email": "TEXT", + "region_id": "INT", + }, + ) + check_column_types( + model_name="products", + expected_types={ + "id": "INT", + "name": "TEXT", + "category": "TEXT", + "price": "DOUBLE", + }, + ) + check_column_types( + model_name="regions", + expected_types={ + "id": "INT", + "name": "TEXT", + }, + ) + # Snowflake has native MEDIAN and PERCENTILE_CONT WITHIN GROUP. + check_median_percentile() + # Native STDDEV_SAMP / STDDEV_POP / VAR_SAMP / VAR_POP — no SqlDialect + # variance-decomposition formula fallback. + check_stddev_var() + # Native CORR / COVAR_SAMP / COVAR_POP — no formula fallback. + check_corr_covar() + summary() diff --git a/examples/sqlserver/CLAUDE.md b/examples/sqlserver/CLAUDE.md new file mode 100644 index 00000000..c47f9d96 --- /dev/null +++ b/examples/sqlserver/CLAUDE.md @@ -0,0 +1,23 @@ +# SQL Server Example + +This example uses **SQL Server 2022** (`mcr.microsoft.com/mssql/server:2022-latest`). + +## Important: SQL Server 2022 required + +`DATETRUNC` was introduced in SQL Server 2022. Earlier versions (2019 and older) do not have +this function and will error on time-dimension queries. The Docker image tag +`mcr.microsoft.com/mssql/server:2022-latest` is the only supported tag for this example. + +## ODBC driver dependency + +The seed and SLayer containers use a custom `Dockerfile` (in this directory) that installs +`msodbcsql18` via the Microsoft apt repository. The driver version is pinned to 18 because +pyodbc's connection string includes `ODBC+Driver+18+for+SQL+Server`. + +## Running + +```bash +cd examples/sqlserver +docker compose up -d +python verify.py +``` diff --git a/examples/sqlserver/Dockerfile b/examples/sqlserver/Dockerfile new file mode 100644 index 00000000..135edd8e --- /dev/null +++ b/examples/sqlserver/Dockerfile @@ -0,0 +1,34 @@ +FROM python:3.14-slim-bookworm + +WORKDIR /app + +# Install msodbcsql18 driver (OS-level dependency for pyodbc) +RUN apt-get update && apt-get install -y --no-install-recommends \ + curl \ + gnupg \ + unixodbc-dev \ + && curl -fsSL https://packages.microsoft.com/keys/microsoft.asc \ + | gpg --dearmor -o /usr/share/keyrings/microsoft-prod.gpg \ + && curl -fsSL https://packages.microsoft.com/config/debian/12/prod.list \ + > /etc/apt/sources.list.d/mssql-release.list \ + && apt-get update \ + && ACCEPT_EULA=Y apt-get install -y --no-install-recommends msodbcsql18 \ + && rm -rf /var/lib/apt/lists/* + +# Install Python dependencies +COPY pyproject.toml poetry.lock README.md LICENSE ./ +RUN pip install --no-cache-dir poetry && \ + poetry config virtualenvs.create false && \ + poetry install -E all --no-root --no-interaction --no-ansi && \ + pip uninstall -y poetry + +# Copy application code and install project +COPY slayer/ slayer/ +RUN pip install --no-deps . && \ + useradd --create-home slayer +USER slayer + +ENV SLAYER_STORAGE=/data +EXPOSE 5143 + +CMD ["slayer", "serve", "--host", "0.0.0.0", "--port", "5143", "--storage", "/data"] diff --git a/examples/sqlserver/README.md b/examples/sqlserver/README.md new file mode 100644 index 00000000..c50d28d7 --- /dev/null +++ b/examples/sqlserver/README.md @@ -0,0 +1,31 @@ +# SLayer + SQL Server Example + +Runs SLayer against a SQL Server 2022 database using Docker Compose. + +## Prerequisites + +- Docker and Docker Compose +- Python 3.11+ + +## Quick start + +```bash +cd examples/sqlserver +docker compose up -d +# Wait ~30 s for SQL Server to be ready and the seed to complete, then: +python verify.py +``` + +## What it does + +1. Starts a SQL Server 2022 container +2. Creates the `slayer_demo` database +3. Seeds it with the shared e-commerce dataset (regions, customers, products, orders) +4. Starts a SLayer API server on port 5143 + +## Notes + +- SQL Server 2022 is required — `DATETRUNC` (used for time-dimension truncation) was added in 2022. +- `median` and `percentile` are not supported on T-SQL; SLayer raises `NotImplementedError` for those. +- `corr`, `covar_samp`, and `covar_pop` use a variance-decomposition formula (no native T-SQL equivalent). +- The `Dockerfile` in this directory extends the standard SLayer image with `msodbcsql18` (Microsoft ODBC Driver 18). diff --git a/examples/sqlserver/docker-compose.yml b/examples/sqlserver/docker-compose.yml new file mode 100644 index 00000000..4e19899e --- /dev/null +++ b/examples/sqlserver/docker-compose.yml @@ -0,0 +1,57 @@ +services: + sqlserver: + image: mcr.microsoft.com/mssql/server:2022-latest + environment: + ACCEPT_EULA: "Y" + MSSQL_SA_PASSWORD: "YourStrong@Passw0rd" + MSSQL_PID: "Developer" + ports: + - "1433:1433" + healthcheck: + test: + - "CMD-SHELL" + - > + /opt/mssql-tools18/bin/sqlcmd + -S localhost -U sa -P 'YourStrong@Passw0rd' + -Q 'SELECT 1' -No || exit 1 + interval: 5s + timeout: 10s + retries: 30 + start_period: 30s + + createdb: + image: mcr.microsoft.com/mssql/server:2022-latest + command: > + /opt/mssql-tools18/bin/sqlcmd + -S sqlserver -U sa -P 'YourStrong@Passw0rd' + -Q "IF DB_ID(N'slayer_demo') IS NULL CREATE DATABASE slayer_demo;" -No + depends_on: + sqlserver: + condition: service_healthy + + seed: + build: + context: ../.. + dockerfile: examples/sqlserver/Dockerfile + command: > + python /examples/seed.py + "mssql+pyodbc://sa:YourStrong%40Passw0rd@sqlserver:1433/slayer_demo?driver=ODBC+Driver+18+for+SQL+Server&TrustServerCertificate=yes" + volumes: + - ../seed.py:/examples/seed.py:ro + depends_on: + createdb: + condition: service_completed_successfully + + slayer: + build: + context: ../.. + dockerfile: examples/sqlserver/Dockerfile + command: sh /examples/start.sh + ports: + - "5143:5143" + volumes: + - ./start.sh:/examples/start.sh:ro + - ./slayer_data:/data + depends_on: + seed: + condition: service_completed_successfully diff --git a/examples/sqlserver/slayer_data/.gitignore b/examples/sqlserver/slayer_data/.gitignore new file mode 100644 index 00000000..d6b7ef32 --- /dev/null +++ b/examples/sqlserver/slayer_data/.gitignore @@ -0,0 +1,2 @@ +* +!.gitignore diff --git a/examples/sqlserver/start.sh b/examples/sqlserver/start.sh new file mode 100644 index 00000000..3c0756f6 --- /dev/null +++ b/examples/sqlserver/start.sh @@ -0,0 +1,21 @@ +#!/bin/sh +# Ingest models from SQL Server and start the SLayer API server. + +python -c " +from slayer.async_utils import run_sync +from slayer.core.models import DatasourceConfig +from slayer.engine.ingestion import ingest_datasource_idempotent +from slayer.storage.yaml_storage import YAMLStorage + +storage = YAMLStorage(base_dir='/data') +ds = DatasourceConfig( + name='demo', type='mssql', + host='sqlserver', port=1433, + database='slayer_demo', username='sa', password='YourStrong@Passw0rd', +) +run_sync(storage.save_datasource(ds)) +result = run_sync(ingest_datasource_idempotent(datasource=ds, storage=storage)) +print(f'Ingested {len(result.additions)} models') +" + +exec slayer serve --host 0.0.0.0 --port 5143 --storage /data diff --git a/examples/sqlserver/verify.py b/examples/sqlserver/verify.py new file mode 100644 index 00000000..75383f72 --- /dev/null +++ b/examples/sqlserver/verify.py @@ -0,0 +1,56 @@ +"""Verification script for the SQL Server Docker example. + +Run after `docker compose up -d`: + python examples/sqlserver/verify.py + +SQL Server 2022 supports STDEV/STDEVP/VAR/VARP natively; corr/covar_samp/ +covar_pop use a variance-decomposition formula (no native function on T-SQL). +median/percentile are not supported on T-SQL and raise NotImplementedError. +""" + +import os +import sys + +sys.path.insert(0, os.path.join(os.path.dirname(__file__), "..")) +from verify_common import ( + run_common_checks, + check_rollup, + check_stddev_var, + check_corr_covar, + check_column_types, + summary, +) + +if __name__ == "__main__": + models = run_common_checks() + check_rollup(expect_rollup=True) + + check_column_types( + model_name="orders", + expected_types={ + "id": "INT", + "customer_id": "INT", + "product_id": "INT", + "quantity": "INT", + "status": "TEXT", + "created_at": "TIMESTAMP", + }, + ) + check_column_types( + model_name="products", + expected_types={ + "id": "INT", + "name": "TEXT", + "category": "TEXT", + "price": "DOUBLE", + }, + ) + + # T-SQL uses STDEV/STDEVP/VAR/VARP (not stddev_samp etc.) — verified via + # the SQL generator; the API response is the same regardless of dialect. + check_stddev_var() + + # corr/covar_samp/covar_pop via variance-decomposition formula. + check_corr_covar() + + summary() diff --git a/examples/verify_common.py b/examples/verify_common.py index e9dc8ff4..f0dc43be 100644 --- a/examples/verify_common.py +++ b/examples/verify_common.py @@ -62,7 +62,7 @@ def check_column_types(model_name, expected_types): """Assert /models/{name} returns the expected DataType strings. expected_types: dict mapping column name to DataType .value string - (e.g. "number", "string", "time", "date"). Columns absent + (e.g. "DOUBLE", "TEXT", "TIMESTAMP", "DATE"). Columns absent from the dict are ignored — different dialects expose different column sets, and this helper is a positive-coverage check, not an exhaustive schema comparison. @@ -317,9 +317,9 @@ def check_stddev_var(measure="quantity"): def check_corr_covar(measure="quantity", other="customer_id"): """2-arg stat aggregates: corr, covar_samp, covar_pop. - Do NOT call from MySQL examples — SLayer raises ``NotImplementedError`` - for these on MySQL (no native function, no Python-UDF mechanism). - Use MariaDB or compute client-side as a workaround. + Safe to call for all Tier-1 dialects including MySQL and T-SQL (SQL Server): + those use a variance-decomposition formula instead of native functions. + MariaDB and all others use native CORR/COVAR_*. """ print("\nCorrelation / covariance:") diff --git a/mkdocs.yml b/mkdocs.yml deleted file mode 100644 index e418201a..00000000 --- a/mkdocs.yml +++ /dev/null @@ -1,104 +0,0 @@ -site_name: SLayer -site_description: A lightweight semantic layer for AI agents -site_url: https://motley-slayer.readthedocs.io -repo_url: https://github.com/motleyai/slayer -repo_name: motleyai/slayer -edit_uri: edit/main/docs/ - -theme: - name: readthedocs - logo: https://raw.githubusercontent.com/MotleyAI/slayer/refs/heads/main/docs/SLayer.png - -nav: - - Home: index.md - - Getting Started: - - getting-started/index.md - - MCP (AI Agents): getting-started/mcp.md - - CLI (Terminal): getting-started/cli.md - - REST API (Any Language): getting-started/rest-api.md - - Python SDK: getting-started/python.md - - Concepts: - - Terminology: concepts/terminology.md - - Models: concepts/models.md - - Queries: concepts/queries.md - - Formulas: concepts/formulas.md - - References (SQL vs DSL): concepts/references.md - - Auto-Ingestion: concepts/ingestion.md - - Schema Drift: concepts/schema-drift.md - - Memories: concepts/memories.md - - Search: concepts/search.md - - Reference: - - MCP Server: reference/mcp.md - - REST API: reference/rest-api.md - - Python Client: reference/python-client.md - - CLI: reference/cli.md - - Database Support: database-support.md - - Wire Protocols (BI Tools): - - Flight SQL: interfaces/flight-sql.md - - Postgres Facade: interfaces/pg-facade.md - - dbt: - - SLayer vs dbt: dbt/slayer_vs_dbt.md - - Importing dbt definitions: dbt/dbt_import.md - - Tutorials: - - Make it dynamic: examples/01_dynamic/dynamic.md - - SQL vs DSL: - - examples/02_sql_vs_dsl/sql_vs_dsl.md - - Notebook: examples/02_sql_vs_dsl/sql_vs_dsl_nb.ipynb - - Auto-Ingestion: - - examples/03_auto_ingest/auto_ingest.md - - Notebook: examples/03_auto_ingest/auto_ingest_nb.ipynb - - Time Dimensions: - - examples/04_time/time.md - - Notebook: examples/04_time/time_nb.ipynb - - Joins: - - examples/05_joins/joins.md - - Notebook: examples/05_joins/joins_nb.ipynb - - Joined Measures: - - examples/05_joined_measures/joined_measures.md - - Notebook: examples/05_joined_measures/joined_measures_nb.ipynb - - Multistage Queries: - - examples/06_multistage_queries/multistage_queries.md - - Notebook: examples/06_multistage_queries/multistage_queries_nb.ipynb - - Aggregations: - - examples/07_aggregations/aggregations.md - - Notebook: examples/07_aggregations/aggregations_nb.ipynb - - Lightning Talk: - - examples/09_lightning_talk/lightning_talk.md - - Notebook: examples/09_lightning_talk/lightning_talk_nb.ipynb - - Schema Drift (worked example): examples/schema-drift.md - - Configuration: - - Datasources: configuration/datasources.md - - Storage Backends: configuration/storage.md - - Architecture: - - Overview: architecture/index.md - - Typed keys: architecture/typed-keys.md - - Scopes & source bundle: architecture/scopes-and-bundle.md - - Slack normalization: architecture/slack-normalization.md - - Parsing: architecture/parsing.md - - Binding: architecture/binding.md - - Planning: architecture/planning.md - - Cross-model aggregates: architecture/cross-model-aggregates.md - - Stage planning: architecture/stage-planning.md - - SQL generation: architecture/sql-generation.md - - Errors & warnings: architecture/errors-and-warnings.md - - Engine orchestration: architecture/engine-orchestration.md - - Development: development.md - -plugins: - - search - - section-index - - mkdocs-jupyter: - include_source: true - ignore_h1_titles: true - -markdown_extensions: - - admonition - - pymdownx.details - - pymdownx.superfences - - pymdownx.tabbed: - alternate_style: true - - pymdownx.highlight: - anchor_linenums: true - - tables - - toc: - permalink: true diff --git a/poetry.lock b/poetry.lock index 541e880e..238961f3 100644 --- a/poetry.lock +++ b/poetry.lock @@ -1,4 +1,4 @@ -# This file is automatically @generated by Poetry 2.3.4 and should not be changed by hand. +# This file is automatically @generated by Poetry 2.4.1 and should not be changed by hand. [[package]] name = "agate" @@ -32,7 +32,7 @@ description = "Happy Eyeballs for asyncio" optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "aiohappyeyeballs-2.6.1-py3-none-any.whl", hash = "sha256:f349ba8f4b75cb25c99c5c2d84e997e485204d2902a9597802b0371f09331fb8"}, {file = "aiohappyeyeballs-2.6.1.tar.gz", hash = "sha256:c3f9d0113123803ccadfdf3f0faa505bc78e6a72d1cc4806cbd719826e943558"}, @@ -45,7 +45,7 @@ description = "Async http client/server framework (asyncio)" optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "aiohttp-3.13.5-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:02222e7e233295f40e011c1b00e3b0bd451f22cf853a0304c3595633ee47da4b"}, {file = "aiohttp-3.13.5-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:bace460460ed20614fa6bc8cb09966c0b8517b8c58ad8046828c6078d25333b5"}, @@ -208,7 +208,7 @@ description = "aiosignal: a list of registered asynchronous callbacks" optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "aiosignal-1.4.0-py3-none-any.whl", hash = "sha256:053243f8b92b990551949e63930a839ff0cf0b0ebbe0597b0f3fb19e1a0fe82e"}, {file = "aiosignal-1.4.0.tar.gz", hash = "sha256:f47eecd9468083c2029cc99945502cb7708b082c232f9aca65da147157b251c7"}, @@ -351,6 +351,19 @@ tzdata = {version = "*", markers = "python_version >= \"3.9\""} doc = ["doc8", "sphinx (>=7.0.0)", "sphinx-autobuild", "sphinx-autodoc-typehints", "sphinx_rtd_theme (>=1.3.0)"] test = ["dateparser (==1.*)", "pre-commit", "pytest", "pytest-cov", "pytest-mock", "pytz (==2025.2)", "simplejson (==3.*)"] +[[package]] +name = "asn1crypto" +version = "1.5.1" +description = "Fast ASN.1 parser and serializer with definitions for private keys, public keys, certificates, CRL, OCSP, CMS, PKCS#3, PKCS#7, PKCS#8, PKCS#12, PKCS#5, X.509 and TSP" +optional = true +python-versions = "*" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "asn1crypto-1.5.1-py2.py3-none-any.whl", hash = "sha256:db4e40728b728508912cbb3d44f19ce188f218e9eba635821bb4b68564f8fd67"}, + {file = "asn1crypto-1.5.1.tar.gz", hash = "sha256:13ae38502be632115abf8a24cbe5f4da52e3b5231990aff31123c805306ccb9c"}, +] + [[package]] name = "asttokens" version = "3.0.1" @@ -497,32 +510,11 @@ files = [ {file = "babel-2.18.0-py3-none-any.whl", hash = "sha256:e2b422b277c2b9a9630c1d7903c2a00d0830c409c59ac8cae9081c92f1aeba35"}, {file = "babel-2.18.0.tar.gz", hash = "sha256:b80b99a14bd085fcacfa15c9165f651fbb3406e66cc603abf11c5750937c992d"}, ] -markers = {main = "extra == \"docs\" or extra == \"dbt\" or extra == \"all\""} +markers = {main = "extra == \"dbt\" or extra == \"all\""} [package.extras] dev = ["backports.zoneinfo ; python_version < \"3.9\"", "freezegun (>=1.0,<2.0)", "jinja2 (>=3.0)", "pytest (>=6.0)", "pytest-cov", "pytz", "setuptools", "tzdata ; sys_platform == \"win32\""] -[[package]] -name = "backrefs" -version = "6.2" -description = "A wrapper around re and regex that adds additional back references." -optional = true -python-versions = ">=3.9" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "backrefs-6.2-py310-none-any.whl", hash = "sha256:0fdc7b012420b6b144410342caeb8adc54c6866cf12064abc9bb211302e496f8"}, - {file = "backrefs-6.2-py311-none-any.whl", hash = "sha256:08aa7fae530c6b2361d7bdcbda1a7c454e330cc9dbcd03f5c23205e430e5c3be"}, - {file = "backrefs-6.2-py312-none-any.whl", hash = "sha256:c3f4b9cb2af8cda0d87ab4f57800b57b95428488477be164dd2b47be54db0c90"}, - {file = "backrefs-6.2-py313-none-any.whl", hash = "sha256:12df81596ab511f783b7d87c043ce26bc5b0288cf3bb03610fe76b8189282b2b"}, - {file = "backrefs-6.2-py314-none-any.whl", hash = "sha256:e5f805ae09819caa1aa0623b4a83790e7028604aa2b8c73ba602c4454e665de7"}, - {file = "backrefs-6.2-py39-none-any.whl", hash = "sha256:664e33cd88c6840b7625b826ecf2555f32d491800900f5a541f772c485f7cda7"}, - {file = "backrefs-6.2.tar.gz", hash = "sha256:f44ff4d48808b243b6c0cdc6231e22195c32f77046018141556c66f8bab72a49"}, -] - -[package.extras] -extras = ["regex"] - [[package]] name = "beautifulsoup4" version = "4.14.3" @@ -565,6 +557,48 @@ webencodings = "*" [package.extras] css = ["tinycss2 (>=1.1.0,<1.5)"] +[[package]] +name = "boto3" +version = "1.43.27" +description = "The AWS SDK for Python" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "boto3-1.43.27-py3-none-any.whl", hash = "sha256:b3eea072c2fdbbdd8c6161f912f603be10c8ec477625926dea8b91a0842a3482"}, + {file = "boto3-1.43.27.tar.gz", hash = "sha256:dc0d1b47f391983d8b3047e49402d31f9aaa4d7b398d3b4ea986fe680cbea43a"}, +] + +[package.dependencies] +botocore = ">=1.43.27,<1.44.0" +jmespath = ">=0.7.1,<2.0.0" +s3transfer = ">=0.18.0,<0.19.0" + +[package.extras] +crt = ["botocore[crt] (>=1.21.0,<2.0a0)"] + +[[package]] +name = "botocore" +version = "1.43.27" +description = "Low-level, data-driven core of boto 3." +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "botocore-1.43.27-py3-none-any.whl", hash = "sha256:4976544e652d5a1d8eca135da019f8e1c2d749efa2f9a31a8fb8c76f1895a40b"}, + {file = "botocore-1.43.27.tar.gz", hash = "sha256:2093c316c24214e50e18640b1869513b759bb8cc48b95b004a8306cb9f0d6703"}, +] + +[package.dependencies] +jmespath = ">=0.7.1,<2.0.0" +python-dateutil = ">=2.1,<3.0.0" +urllib3 = ">=1.25.4,<2.2.0 || >2.2.0,<3" + +[package.extras] +crt = ["awscrt (==0.32.2)"] + [[package]] name = "certifi" version = "2026.2.25" @@ -825,7 +859,7 @@ files = [ {file = "charset_normalizer-3.4.7-py3-none-any.whl", hash = "sha256:3dce51d0f5e7951f8bb4900c257dad282f49190fdbebecd4ba99bcc41fef404d"}, {file = "charset_normalizer-3.4.7.tar.gz", hash = "sha256:ae89db9e5f98a11a4bf50407d4363e7b09b31e55bc117b4f7d80aab97ba009e5"}, ] -markers = {main = "extra == \"docs\" or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"embedding-search\""} +markers = {main = "(extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"clickhouse\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\") or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\""} [[package]] name = "ciso8601" @@ -922,10 +956,9 @@ colorama = {version = "*", markers = "platform_system == \"Windows\""} name = "clickhouse-driver" version = "0.2.10" description = "Python driver with native interface for ClickHouse" -optional = true +optional = false python-versions = "<4,>=3.9" -groups = ["main"] -markers = "extra == \"clickhouse\" or extra == \"all\"" +groups = ["main", "dev"] files = [ {file = "clickhouse_driver-0.2.10-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:1dc2b570b3c02baf1ba759f1db3cc5d49b4f968f32dd56710ca4f892700079c1"}, {file = "clickhouse_driver-0.2.10-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:5be143e33e330e856e65d40cdfdd38c0f215ba5140807aee5e923fd0c8cf04c8"}, @@ -1025,6 +1058,7 @@ files = [ {file = "clickhouse_driver-0.2.10-pp39-pypy39_pp73-win_amd64.whl", hash = "sha256:cd3777d74787707d1b273f8491d38deff213dcf1d2669d9d91bf109cd0cd930d"}, {file = "clickhouse_driver-0.2.10.tar.gz", hash = "sha256:925fc6ecda1e5314e3f03bcb493955c068b070cdba221fb8ce27329ee8a7f71b"}, ] +markers = {main = "extra == \"clickhouse\" or extra == \"all\""} [package.dependencies] pytz = "*" @@ -1064,7 +1098,7 @@ files = [ {file = "colorama-0.4.6-py2.py3-none-any.whl", hash = "sha256:4f1d9991f5acc0ca119f9d443620b77f9d6b33703e51011c16baf57afb285fc6"}, {file = "colorama-0.4.6.tar.gz", hash = "sha256:08695f5cb7ed6e0531a20572697297273c47b8cae5a63ffc6d6ed5c201be6e44"}, ] -markers = {main = "platform_system == \"Windows\" or extra == \"docs\" or extra == \"dbt\" or extra == \"all\"", dev = "sys_platform == \"win32\""} +markers = {main = "platform_system == \"Windows\" or extra == \"dbt\" or extra == \"all\"", dev = "sys_platform == \"win32\""} [[package]] name = "comm" @@ -1206,7 +1240,7 @@ version = "46.0.7" description = "cryptography is a package which provides cryptographic recipes and primitives to Python developers." optional = false python-versions = "!=3.9.0,!=3.9.1,>=3.8" -groups = ["main"] +groups = ["main", "dev"] files = [ {file = "cryptography-46.0.7-cp311-abi3-macosx_10_9_universal2.whl", hash = "sha256:ea42cbe97209df307fdc3b155f1b6fa2577c0defa8f1f7d3be7d31d189108ad4"}, {file = "cryptography-46.0.7-cp311-abi3-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:b36a4695e29fe69215d75960b22577197aca3f7a25b9cf9d165dcfe9d80bc325"}, @@ -1552,12 +1586,35 @@ description = "Distro - an OS platform information API" optional = true python-versions = ">=3.6" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "distro-1.9.0-py3-none-any.whl", hash = "sha256:7bffd925d65168f85027d8da9af6bddab658135b840670a223589bc0c8ef02b2"}, {file = "distro-1.9.0.tar.gz", hash = "sha256:2fa77c6fd8940f116ee1d6b94a2f90b13b5ea8d019b98bc8bafdcabcdd9bdbed"}, ] +[[package]] +name = "docker" +version = "7.1.0" +description = "A Python library for the Docker Engine API." +optional = false +python-versions = ">=3.8" +groups = ["dev"] +files = [ + {file = "docker-7.1.0-py3-none-any.whl", hash = "sha256:c96b93b7f0a746f9e77d325bcfb87422a3d8bd4f03136ae8a85b37f1898d5fc0"}, + {file = "docker-7.1.0.tar.gz", hash = "sha256:ad8c70e6e3f8926cb8a92619b832b4ea5299e2831c14284663184e200546fa6c"}, +] + +[package.dependencies] +pywin32 = {version = ">=304", markers = "sys_platform == \"win32\""} +requests = ">=2.26.0" +urllib3 = ">=1.26.0" + +[package.extras] +dev = ["coverage (==7.2.7)", "pytest (==7.4.2)", "pytest-cov (==4.1.0)", "pytest-timeout (==2.1.0)", "ruff (==0.1.8)"] +docs = ["myst-parser (==0.18.0)", "sphinx (==5.1.1)"] +ssh = ["paramiko (>=2.4.3)"] +websockets = ["websocket-client (>=1.3.0)"] + [[package]] name = "duckdb" version = "1.5.2" @@ -1702,7 +1759,7 @@ description = "Python bindings to Rust's UUID library." optional = true python-versions = ">=3.8" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "fastuuid-0.14.0-cp310-cp310-macosx_10_12_x86_64.macosx_11_0_arm64.macosx_10_12_universal2.whl", hash = "sha256:6e6243d40f6c793c3e2ee14c13769e341b90be5ef0c23c82fa6515a96145181a"}, {file = "fastuuid-0.14.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:13ec4f2c3b04271f62be2e1ce7e95ad2dd1cf97e94503a3760db739afbd48f00"}, @@ -1795,7 +1852,7 @@ files = [ {file = "filelock-3.28.0-py3-none-any.whl", hash = "sha256:de9af6712788e7171df1b28b15eba2446c69721433fa427a9bee07b17820a9db"}, {file = "filelock-3.28.0.tar.gz", hash = "sha256:4ed1010aae813c4ee8d9c660e4792475ee60c4a0ba76073ceaf862bd317e3ca6"}, ] -markers = {main = "extra == \"embedding-search\" or extra == \"all\""} +markers = {main = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\" or extra == \"snowflake\""} [[package]] name = "fqdn" @@ -1816,7 +1873,7 @@ description = "A list-like structure which implements collections.abc.MutableSeq optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "frozenlist-1.8.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:b37f6d31b3dcea7deb5e9696e529a6aa4a898adc33db82da12e4c60a7c4d2011"}, {file = "frozenlist-1.8.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:ef2b7b394f208233e471abc541cc6991f907ffd47dc72584acee3147899d6565"}, @@ -1957,7 +2014,7 @@ description = "File-system specification" optional = true python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "fsspec-2026.4.0-py3-none-any.whl", hash = "sha256:11ef7bb35dab8a394fde6e608221d5cf3e8499401c249bebaeaad760a1a8dec2"}, {file = "fsspec-2026.4.0.tar.gz", hash = "sha256:301d8ac70ae90ef3ad05dcf94d6c3754a097f9b5fe4667d2787aa359ec7df7e4"}, @@ -1992,23 +2049,205 @@ test-full = ["adlfs", "aiohttp (!=4.0.0a0,!=4.0.0a1)", "backports-zstd ; python_ tqdm = ["tqdm"] [[package]] -name = "ghp-import" -version = "2.1.0" -description = "Copy your docs directly to the gh-pages branch." +name = "google-api-core" +version = "2.31.0" +description = "Google API client core library" optional = true -python-versions = "*" +python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"docs\"" +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" files = [ - {file = "ghp-import-2.1.0.tar.gz", hash = "sha256:9c535c4c61193c2df8871222567d7fd7e5014d835f97dc7b7439069e2413d343"}, - {file = "ghp_import-2.1.0-py3-none-any.whl", hash = "sha256:8337dd7b50877f163d4c0289bc1f1c7f127550241988d568c1db512c4324a619"}, + {file = "google_api_core-2.31.0-py3-none-any.whl", hash = "sha256:ef79fb3784c71cbac89cbd03301ba0c8fb8ad2aa95d7f9204dd9628f7adf59ab"}, + {file = "google_api_core-2.31.0.tar.gz", hash = "sha256:2be84ee0f584c48e6bde1b36766e23348b361fb7e55e56135fc76ce1c397f9c2"}, ] [package.dependencies] -python-dateutil = ">=2.8.1" +google-auth = ">=2.14.1,<3.0.0" +googleapis-common-protos = ">=1.63.2,<2.0.0" +grpcio = [ + {version = ">=1.49.1,<2.0.0", optional = true, markers = "python_version >= \"3.11\" and extra == \"grpc\" and python_version < \"3.14\""}, + {version = ">=1.75.1,<2.0.0", optional = true, markers = "python_version >= \"3.14\" and extra == \"grpc\""}, +] +grpcio-status = [ + {version = ">=1.49.1,<2.0.0", optional = true, markers = "python_version >= \"3.11\" and extra == \"grpc\""}, + {version = ">=1.75.1,<2.0.0", optional = true, markers = "python_version >= \"3.14\" and extra == \"grpc\""}, +] +proto-plus = [ + {version = ">=1.24.0,<2.0.0"}, + {version = ">=1.25.0,<2.0.0", markers = "python_version >= \"3.13\""}, +] +protobuf = ">=5.29.6,<8.0.0" +requests = ">=2.33.0,<3.0.0" [package.extras] -dev = ["flake8", "markdown", "twine", "wheel"] +async-rest = ["aiohttp (>=3.13.4)", "google-auth[aiohttp] (>=2.14.1,<3.0.0)"] +grpc = ["grpcio (>=1.41.0,<2.0.0)", "grpcio (>=1.49.1,<2.0.0) ; python_version >= \"3.11\"", "grpcio (>=1.75.1,<2.0.0) ; python_version >= \"3.14\"", "grpcio-status (>=1.41.0,<2.0.0)", "grpcio-status (>=1.49.1,<2.0.0) ; python_version >= \"3.11\"", "grpcio-status (>=1.75.1,<2.0.0) ; python_version >= \"3.14\""] + +[[package]] +name = "google-auth" +version = "2.53.0" +description = "Google Authentication Library" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "google_auth-2.53.0-py3-none-any.whl", hash = "sha256:6e7449917c599b35126a99ec268ec6880301f2fea41dce198fe8fd83ff642b68"}, + {file = "google_auth-2.53.0.tar.gz", hash = "sha256:e7e6aa16f6bee7b2b264830fd04f08087a1d5a836df516251a5d15327b246c9c"}, +] + +[package.dependencies] +cryptography = ">=38.0.3" +pyasn1-modules = ">=0.2.1" + +[package.extras] +aiohttp = ["aiohttp (>=3.8.0,<4.0.0)", "requests (>=2.20.0,<3.0.0)"] +cryptography = ["cryptography (>=38.0.3)"] +enterprise-cert = ["pyopenssl"] +pyjwt = ["pyjwt (>=2.0)"] +pyopenssl = ["pyopenssl (>=20.0.0)"] +reauth = ["pyu2f (>=0.1.5)"] +requests = ["requests (>=2.20.0,<3.0.0)"] +rsa = ["rsa (>=3.1.4,<5)"] +testing = ["aiohttp (<3.10.0)", "aiohttp (>=3.8.0,<4.0.0)", "aioresponses", "flask", "freezegun", "grpcio", "packaging", "pyjwt (>=2.0)", "pyopenssl (<24.3.0)", "pyopenssl (>=20.0.0)", "pytest", "pytest-asyncio", "pytest-cov", "pytest-localserver", "pyu2f (>=0.1.5)", "requests (>=2.20.0,<3.0.0)", "responses", "urllib3"] +urllib3 = ["packaging", "urllib3"] + +[[package]] +name = "google-cloud-bigquery" +version = "3.41.0" +description = "Google BigQuery API client library" +optional = true +python-versions = ">=3.8" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "google_cloud_bigquery-3.41.0-py3-none-any.whl", hash = "sha256:2a5b5a737b401cbd824a6e5eac7554100b878668d908e6548836b5d8aaa4dcaa"}, + {file = "google_cloud_bigquery-3.41.0.tar.gz", hash = "sha256:2217e488b47ed576360c9b2cc07d59d883a54b83167c0ef37f915c26b01a06fe"}, +] + +[package.dependencies] +google-api-core = {version = ">=2.11.1,<3.0.0", extras = ["grpc"]} +google-auth = ">=2.14.1,<3.0.0" +google-cloud-core = ">=2.4.1,<3.0.0" +google-resumable-media = ">=2.0.0,<3.0.0" +packaging = ">=24.2.0" +python-dateutil = ">=2.8.2,<3.0.0" +requests = ">=2.21.0,<3.0.0" + +[package.extras] +all = ["google-cloud-bigquery[bigquery-v2,bqstorage,geopandas,ipython,ipywidgets,matplotlib,opentelemetry,pandas,tqdm]"] +bigquery-v2 = ["proto-plus (>=1.22.3,<2.0.0)", "protobuf (>=3.20.2,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<7.0.0)"] +bqstorage = ["google-cloud-bigquery-storage (>=2.18.0,<3.0.0)", "grpcio (>=1.47.0,<2.0.0)", "grpcio (>=1.49.1,<2.0.0) ; python_version >= \"3.11\"", "grpcio (>=1.75.1,<2.0.0) ; python_version >= \"3.14\"", "pyarrow (>=4.0.0)"] +geopandas = ["Shapely (>=1.8.4,<3.0.0)", "geopandas (>=0.9.0,<2.0.0)"] +ipython = ["bigquery-magics (>=0.6.0)", "ipython (>=7.23.1)"] +ipywidgets = ["ipykernel (>=6.2.0)", "ipywidgets (>=7.7.1)"] +matplotlib = ["matplotlib (>=3.10.3) ; python_version >= \"3.10\"", "matplotlib (>=3.7.1,<=3.9.2) ; python_version == \"3.9\""] +opentelemetry = ["opentelemetry-api (>=1.1.0)", "opentelemetry-instrumentation (>=0.20b0)", "opentelemetry-sdk (>=1.1.0)"] +pandas = ["db-dtypes (>=1.0.4,<2.0.0)", "grpcio (>=1.47.0,<2.0.0)", "grpcio (>=1.49.1,<2.0.0) ; python_version >= \"3.11\"", "grpcio (>=1.75.1,<2.0.0) ; python_version >= \"3.14\"", "pandas (>=1.3.0)", "pandas-gbq (>=0.26.1)", "pyarrow (>=3.0.0)"] +tqdm = ["tqdm (>=4.23.4,<5.0.0)"] + +[[package]] +name = "google-cloud-core" +version = "2.6.0" +description = "Google Cloud API client core library" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "google_cloud_core-2.6.0-py3-none-any.whl", hash = "sha256:6d63ac8e5eca6d9e4319d0a1e2265fadcd7f1049904378caecfa01cf52dd869e"}, + {file = "google_cloud_core-2.6.0.tar.gz", hash = "sha256:e76149739f90fac1fc6757c09f47eaccb3145b54adbd7759b0f7c4b235f46c83"}, +] + +[package.dependencies] +google-api-core = ">=2.11.0,<3.0.0" +google-auth = ">=2.14.1,<2.24.0 || >2.24.0,<2.25.0 || >2.25.0,<3.0.0" + +[package.extras] +grpc = ["grpcio (>=1.47.0,<2.0.0) ; python_version < \"3.14\"", "grpcio (>=1.75.1,<2.0.0) ; python_version >= \"3.14\"", "grpcio-status (>=1.47.0,<2.0.0)"] + +[[package]] +name = "google-crc32c" +version = "1.8.0" +description = "A python wrapper of the C library 'Google CRC32C'" +optional = true +python-versions = ">=3.9" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "google_crc32c-1.8.0-cp310-cp310-macosx_12_0_arm64.whl", hash = "sha256:0470b8c3d73b5f4e3300165498e4cf25221c7eb37f1159e221d1825b6df8a7ff"}, + {file = "google_crc32c-1.8.0-cp310-cp310-macosx_12_0_x86_64.whl", hash = "sha256:119fcd90c57c89f30040b47c211acee231b25a45d225e3225294386f5d258288"}, + {file = "google_crc32c-1.8.0-cp310-cp310-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:6f35aaffc8ccd81ba3162443fabb920e65b1f20ab1952a31b13173a67811467d"}, + {file = "google_crc32c-1.8.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:864abafe7d6e2c4c66395c1eb0fe12dc891879769b52a3d56499612ca93b6092"}, + {file = "google_crc32c-1.8.0-cp310-cp310-win_amd64.whl", hash = "sha256:db3fe8eaf0612fc8b20fa21a5f25bd785bc3cd5be69f8f3412b0ac2ffd49e733"}, + {file = "google_crc32c-1.8.0-cp311-cp311-macosx_12_0_arm64.whl", hash = "sha256:014a7e68d623e9a4222d663931febc3033c5c7c9730785727de2a81f87d5bab8"}, + {file = "google_crc32c-1.8.0-cp311-cp311-macosx_12_0_x86_64.whl", hash = "sha256:86cfc00fe45a0ac7359e5214a1704e51a99e757d0272554874f419f79838c5f7"}, + {file = "google_crc32c-1.8.0-cp311-cp311-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:19b40d637a54cb71e0829179f6cb41835f0fbd9e8eb60552152a8b52c36cbe15"}, + {file = "google_crc32c-1.8.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:17446feb05abddc187e5441a45971b8394ea4c1b6efd88ab0af393fd9e0a156a"}, + {file = "google_crc32c-1.8.0-cp311-cp311-win_amd64.whl", hash = "sha256:71734788a88f551fbd6a97be9668a0020698e07b2bf5b3aa26a36c10cdfb27b2"}, + {file = "google_crc32c-1.8.0-cp312-cp312-macosx_12_0_arm64.whl", hash = "sha256:4b8286b659c1335172e39563ab0a768b8015e88e08329fa5321f774275fc3113"}, + {file = "google_crc32c-1.8.0-cp312-cp312-macosx_12_0_x86_64.whl", hash = "sha256:2a3dc3318507de089c5384cc74d54318401410f82aa65b2d9cdde9d297aca7cb"}, + {file = "google_crc32c-1.8.0-cp312-cp312-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:14f87e04d613dfa218d6135e81b78272c3b904e2a7053b841481b38a7d901411"}, + {file = "google_crc32c-1.8.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:cb5c869c2923d56cb0c8e6bcdd73c009c36ae39b652dbe46a05eb4ef0ad01454"}, + {file = "google_crc32c-1.8.0-cp312-cp312-win_amd64.whl", hash = "sha256:3cc0c8912038065eafa603b238abf252e204accab2a704c63b9e14837a854962"}, + {file = "google_crc32c-1.8.0-cp313-cp313-macosx_12_0_arm64.whl", hash = "sha256:3ebb04528e83b2634857f43f9bb8ef5b2bbe7f10f140daeb01b58f972d04736b"}, + {file = "google_crc32c-1.8.0-cp313-cp313-macosx_12_0_x86_64.whl", hash = "sha256:450dc98429d3e33ed2926fc99ee81001928d63460f8538f21a5d6060912a8e27"}, + {file = "google_crc32c-1.8.0-cp313-cp313-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3b9776774b24ba76831609ffbabce8cdf6fa2bd5e9df37b594221c7e333a81fa"}, + {file = "google_crc32c-1.8.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:89c17d53d75562edfff86679244830599ee0a48efc216200691de8b02ab6b2b8"}, + {file = "google_crc32c-1.8.0-cp313-cp313-win_amd64.whl", hash = "sha256:57a50a9035b75643996fbf224d6661e386c7162d1dfdab9bc4ca790947d1007f"}, + {file = "google_crc32c-1.8.0-cp314-cp314-macosx_12_0_arm64.whl", hash = "sha256:e6584b12cb06796d285d09e33f63309a09368b9d806a551d8036a4207ea43697"}, + {file = "google_crc32c-1.8.0-cp314-cp314-macosx_12_0_x86_64.whl", hash = "sha256:f4b51844ef67d6cf2e9425983274da75f18b1597bb2c998e1c0a0e8d46f8f651"}, + {file = "google_crc32c-1.8.0-cp314-cp314-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b0d1a7afc6e8e4635564ba8aa5c0548e3173e41b6384d7711a9123165f582de2"}, + {file = "google_crc32c-1.8.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:8b3f68782f3cbd1bce027e48768293072813469af6a61a86f6bb4977a4380f21"}, + {file = "google_crc32c-1.8.0-cp314-cp314-win_amd64.whl", hash = "sha256:d511b3153e7011a27ab6ee6bb3a5404a55b994dc1a7322c0b87b29606d9790e2"}, + {file = "google_crc32c-1.8.0-cp39-cp39-macosx_12_0_arm64.whl", hash = "sha256:ba6aba18daf4d36ad4412feede6221414692f44d17e5428bdd81ad3fc1eee5dc"}, + {file = "google_crc32c-1.8.0-cp39-cp39-macosx_12_0_x86_64.whl", hash = "sha256:87b0072c4ecc9505cfa16ee734b00cd7721d20a0f595be4d40d3d21b41f65ae2"}, + {file = "google_crc32c-1.8.0-cp39-cp39-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:3d488e98b18809f5e322978d4506373599c0c13e6c5ad13e53bb44758e18d215"}, + {file = "google_crc32c-1.8.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:01f126a5cfddc378290de52095e2c7052be2ba7656a9f0caf4bcd1bfb1833f8a"}, + {file = "google_crc32c-1.8.0-cp39-cp39-win_amd64.whl", hash = "sha256:61f58b28e0b21fcb249a8247ad0db2e64114e201e2e9b4200af020f3b6242c9f"}, + {file = "google_crc32c-1.8.0-pp311-pypy311_pp73-manylinux1_x86_64.manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:87fa445064e7db928226b2e6f0d5304ab4cd0339e664a4e9a25029f384d9bb93"}, + {file = "google_crc32c-1.8.0-pp311-pypy311_pp73-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f639065ea2042d5c034bf258a9f085eaa7af0cd250667c0635a3118e8f92c69c"}, + {file = "google_crc32c-1.8.0.tar.gz", hash = "sha256:a428e25fb7691024de47fecfbff7ff957214da51eddded0da0ae0e0f03a2cf79"}, +] + +[[package]] +name = "google-resumable-media" +version = "2.10.0" +description = "Utilities for Google Media Downloads and Resumable Uploads" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "google_resumable_media-2.10.0-py3-none-any.whl", hash = "sha256:88152884bee37b2bf36a0ab81ad8c7fd12212c9803dd981d77c1b35b02d34e7c"}, + {file = "google_resumable_media-2.10.0.tar.gz", hash = "sha256:e324bc9d0fdae4c52a08ae90456edc4e71ece858399e1217ac0eb3a51d6bc6ee"}, +] + +[package.dependencies] +google-crc32c = ">=1.0.0,<2.0.0" + +[package.extras] +aiohttp = ["aiohttp (>=3.6.2,<4.0.0)", "google-auth (>=1.22.0,<2.0.0)"] +requests = ["requests (>=2.18.0,<3.0.0)"] + +[[package]] +name = "googleapis-common-protos" +version = "1.75.0" +description = "Common protobufs used in Google APIs" +optional = true +python-versions = ">=3.9" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "googleapis_common_protos-1.75.0-py3-none-any.whl", hash = "sha256:961ed60399c457ceb0ee8f285a84c870aabc9c6a832b9d37bb281b5bebde43ed"}, + {file = "googleapis_common_protos-1.75.0.tar.gz", hash = "sha256:53a062ff3c32552fbd62c11fe23768b78e4ddf0494d5e5fd97d3f4689c75fbbd"}, +] + +[package.dependencies] +protobuf = ">=4.25.8,<8.0.0" + +[package.extras] +grpc = ["grpcio (>=1.44.0,<2.0.0)"] [[package]] name = "greenlet" @@ -2084,6 +2323,92 @@ files = [ docs = ["Sphinx", "furo"] test = ["objgraph", "psutil", "setuptools"] +[[package]] +name = "grpcio" +version = "1.81.0" +description = "HTTP/2-based RPC framework" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "grpcio-1.81.0-cp310-cp310-linux_armv7l.whl", hash = "sha256:b4108e5d9d0f651b7eea749116181fe6c315b145661a80ec31f05ec2dbe21af7"}, + {file = "grpcio-1.81.0-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:b76ea9d55cd08fcdbda25d28e0f76679536710acb7fbd5b1f70cb4ac49317265"}, + {file = "grpcio-1.81.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:4e032feb3bfb4e2749b140a2302a6baa8ead1b9781ff5cf7094e4402b5e9372e"}, + {file = "grpcio-1.81.0-cp310-cp310-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:725801c7086d7e4cd160e42bb2f54e0aeb976b9568df3cc6f843b15d29b79fb1"}, + {file = "grpcio-1.81.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:f750a091fff3a3991731abc1f818bdc64874bb3528162732cb4d45f2e07821a6"}, + {file = "grpcio-1.81.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:8226ba097eed660ef14d36c6a69b85038552bb8b6d17b44a5aa6f9abf48b8e08"}, + {file = "grpcio-1.81.0-cp310-cp310-musllinux_1_2_i686.whl", hash = "sha256:40edffb4ec3689373825d367c4457727047a6e554f03245265ecc8cc03215f22"}, + {file = "grpcio-1.81.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:f85570a016d794c29b1e76cf22f67af4486ddbe779e0f30674f138fa4e1769ec"}, + {file = "grpcio-1.81.0-cp310-cp310-win32.whl", hash = "sha256:3755c9669307cad18e7e009860fdea98118978d2300451bd8530a53048e741e7"}, + {file = "grpcio-1.81.0-cp310-cp310-win_amd64.whl", hash = "sha256:909bb3222b53235498d2c5817a0596d82b0aaea490ba93fdf1b060e2938a543c"}, + {file = "grpcio-1.81.0-cp311-cp311-linux_armv7l.whl", hash = "sha256:794e6aa648e8df47d8f908dc8c3b42347d04ec58438f1dcd4e445f09b4f6b0ce"}, + {file = "grpcio-1.81.0-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:cd78145b7f7784661c524624f3526c9c6f891b30a4b54cb93a40806d0d0d61e9"}, + {file = "grpcio-1.81.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:638ccc1b86f7540170a169cb900799b9296a1381e47879ce60b0de9d3db73d33"}, + {file = "grpcio-1.81.0-cp311-cp311-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:21ec30b9ea320c8207ea7cd05873ad64aa69fdd0e81b6758b3347983ba20b50a"}, + {file = "grpcio-1.81.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:dbdb99986548a7e87f8343805ef315fd4eb50ffaabf4fb1206e42f2542bb805d"}, + {file = "grpcio-1.81.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c36f5d5e97944cbda2d4096b4ae262e6e68506246b61582acf1b8591607f3ccc"}, + {file = "grpcio-1.81.0-cp311-cp311-musllinux_1_2_i686.whl", hash = "sha256:9f355384e5543ab77a755a7085225ecc19f32b76032e851cbd8145715d79dec8"}, + {file = "grpcio-1.81.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:77eb4e9fe61486bd1198cc7236ebb0f70e66234e63c0348f40bc2553ed16a88b"}, + {file = "grpcio-1.81.0-cp311-cp311-win32.whl", hash = "sha256:7915a2e63acdc05264a206e1bddfd8e1fb8a29e406c18d72d30f8c124e021374"}, + {file = "grpcio-1.81.0-cp311-cp311-win_amd64.whl", hash = "sha256:5e925a70fe99fe5794f7beca0ea034c75f068afcc356d79047e73f99cdcca34c"}, + {file = "grpcio-1.81.0-cp312-cp312-linux_armv7l.whl", hash = "sha256:57b3b0e73a518fa286959b40c3eddd02703504ca186e8b7b2945954519bd8b2c"}, + {file = "grpcio-1.81.0-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:8bb1789c94322a13336a2b6c58d9c14d68f8628b6e24205a799c69f5bf8516ce"}, + {file = "grpcio-1.81.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:e4d053900a0d24b75d7521139a3872150301b3d6bde3bed5e12318fb25791e4d"}, + {file = "grpcio-1.81.0-cp312-cp312-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:db217c2e52931719f9937bd12082cd4d7b495b35803d5760686975c285924bf8"}, + {file = "grpcio-1.81.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:19f201da7b4e5c0559198abe5a97157e726f3abe6e8f5e832d4a50740f6dcc22"}, + {file = "grpcio-1.81.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:275144b0115353339dbb8a6f28a9cf8997b5bf40e37f8f66ac0b0ea57e95b43f"}, + {file = "grpcio-1.81.0-cp312-cp312-musllinux_1_2_i686.whl", hash = "sha256:5192857589f223e5a98ff0e31f6e551b19040e647d17bfe10116c8a2ce3b8696"}, + {file = "grpcio-1.81.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:c6ff087cb1f563f47b504b4e29e684129fc5ae4863faf3ebca08a327764ee6cb"}, + {file = "grpcio-1.81.0-cp312-cp312-win32.whl", hash = "sha256:98c6240f563178fc5877bd50e6ff274463e53e1472128f4110742450739659fa"}, + {file = "grpcio-1.81.0-cp312-cp312-win_amd64.whl", hash = "sha256:87e33b7afcfb3585121b5f007d2c52b8c534104d18f556e840d35193ca2a9141"}, + {file = "grpcio-1.81.0-cp313-cp313-linux_armv7l.whl", hash = "sha256:62bbe463c9f0f2ff24e31bd25f8dd8b4bae78900e315915a3195a0ef1471a855"}, + {file = "grpcio-1.81.0-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:43c121e135ae44d1559b430db2b2dfad7421cbbe40e1deba506c7dc62b439719"}, + {file = "grpcio-1.81.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:f345de40ef2e65f63645d53d251824e6070e07804827c5b00ec2e44555f9f901"}, + {file = "grpcio-1.81.0-cp313-cp313-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:8c0855a350886f713b9e458e2a10d208009dcaa849f574e39cd6067db1fe1279"}, + {file = "grpcio-1.81.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:a524cd530900bd24511fcb7f2ed144da4ea37711c4b094475d0bceca7a93a170"}, + {file = "grpcio-1.81.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:e7746ba3e6efc9e2b748eff59470a2b8684d5a9ec607c6580bcaa5be175820bc"}, + {file = "grpcio-1.81.0-cp313-cp313-musllinux_1_2_i686.whl", hash = "sha256:aaaa4f7f2057d795952e4eacf3f342be8b5b156992f6ac85023c8b98794ebd47"}, + {file = "grpcio-1.81.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:0fba53cb96004b2b7fb758b46b2288cb49d0b658316a4e73f3ef67230616ee65"}, + {file = "grpcio-1.81.0-cp313-cp313-win32.whl", hash = "sha256:c197e2ef75a442528072b29e9755da299110e8610e8bcbb59a6b4cf55384f005"}, + {file = "grpcio-1.81.0-cp313-cp313-win_amd64.whl", hash = "sha256:194eddfacc84d80f50512e9fd4ee851d5f2499f18f299c95aa8fb4748f0537e0"}, + {file = "grpcio-1.81.0-cp314-cp314-linux_armv7l.whl", hash = "sha256:a9351055f52660b58f3d4890ea66188b5134399f82b11aa0c55bd4b99eff5390"}, + {file = "grpcio-1.81.0-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:300f3337b6425fd16ead9a4f9b2ac25801acb64aa5bc0b99eb69901645b2b1d2"}, + {file = "grpcio-1.81.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:97bbd623f7ded558fd4f7cb5a4f600c4d4de65c5dd364c83a5b14b2a10a2d3b5"}, + {file = "grpcio-1.81.0-cp314-cp314-manylinux2014_i686.manylinux_2_17_i686.whl", hash = "sha256:ff83d889e3ebf6341c8c7864ad8031591ad5ca61599072fc511644d1eb962d2b"}, + {file = "grpcio-1.81.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:c4fe218c5a35e1d87a5a26544237f1fa41dfd9cbd3c856b0810a30061f8b0aaf"}, + {file = "grpcio-1.81.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:b8b025b6af43ee0ad4a70307025d77bcab5adde7c4597786010d802c203e9fc5"}, + {file = "grpcio-1.81.0-cp314-cp314-musllinux_1_2_i686.whl", hash = "sha256:3d4e0ce5a40a998cf608c8ba60ecfe18fdf364a9aa193ae4ac3faeecd0e86757"}, + {file = "grpcio-1.81.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:aa948712c8e5fa40ec250870bda14bc7578e1bb832a8912d9d2a0f720518edbe"}, + {file = "grpcio-1.81.0-cp314-cp314-win32.whl", hash = "sha256:fbbe81314a9d92156abce8b62c09364eb8bafc0ca2a19919a45ec64b5c6cb664"}, + {file = "grpcio-1.81.0-cp314-cp314-win_amd64.whl", hash = "sha256:b93cee313cae4e113fbb3a0ce1ea5633db6f63cfde2b2dc1d817429026b2a50b"}, + {file = "grpcio-1.81.0.tar.gz", hash = "sha256:a5acd7efd3b1fe9b4eb0bcaaa1507eed68a0ad0678b654c3f7b464df9ba9dca5"}, +] + +[package.dependencies] +typing-extensions = ">=4.12,<5.0" + +[package.extras] +protobuf = ["grpcio-tools (>=1.81.0)"] + +[[package]] +name = "grpcio-status" +version = "1.81.0" +description = "Status proto mapping for gRPC" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "grpcio_status-1.81.0-py3-none-any.whl", hash = "sha256:10eb4c2309db902dc26c1873e80a821bf794be772c10dfd83030f7f59f165fab"}, + {file = "grpcio_status-1.81.0.tar.gz", hash = "sha256:b6fe9788cfdd1f0f63c0528a1e0bfdb41e8ff0583e920d2d8e8888598c01bb69"}, +] + +[package.dependencies] +googleapis-common-protos = ">=1.5.5" +grpcio = ">=1.81.0" +protobuf = ">=6.33.5,<8.0.0" + [[package]] name = "h11" version = "0.16.0" @@ -2103,7 +2428,7 @@ description = "Fast transfer of large files with the Hugging Face Hub." optional = true python-versions = ">=3.8" groups = ["main"] -markers = "(extra == \"embedding-search\" or extra == \"all\") and (platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"arm64\" or platform_machine == \"aarch64\")" +markers = "(extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\") and (platform_machine == \"x86_64\" or platform_machine == \"amd64\" or platform_machine == \"AMD64\" or platform_machine == \"arm64\" or platform_machine == \"aarch64\")" files = [ {file = "hf_xet-1.5.0-cp313-cp313t-macosx_10_12_x86_64.whl", hash = "sha256:7d70fe2ce97b9db73b9c9b9c81fe3693640aec83416a966c446afea54acfae3c"}, {file = "hf_xet-1.5.0-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:73a0dae8c71de3b0633a45c73f4a4a5ed09e94b43441d82981a781d4f12baa42"}, @@ -2201,7 +2526,7 @@ description = "Client library to download and publish models, datasets and other optional = true python-versions = ">=3.10.0" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "huggingface_hub-1.14.0-py3-none-any.whl", hash = "sha256:efe075535c62e130b30e836b138e13785f6f043d1f0539e0a39aa411a99e90b8"}, {file = "huggingface_hub-1.14.0.tar.gz", hash = "sha256:d6d2c9cd6be1d02ae9ec6672d5587d10a427f377db688e82528f426a041622c2"}, @@ -2268,7 +2593,7 @@ description = "Read metadata from Python packages" optional = true python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"dbt\" or extra == \"all\" or extra == \"embedding-search\"" +markers = "extra == \"dbt\" or extra == \"all\" or extra == \"advanced-search\" or extra == \"embedding-search\"" files = [ {file = "importlib_metadata-8.9.0-py3-none-any.whl", hash = "sha256:e0f761b6ea91ced3b0844c14c9d955224d538105921f8e6754c00f6ca79fba7f"}, {file = "importlib_metadata-8.9.0.tar.gz", hash = "sha256:58850626cef4bd2df100378b0f2aea9724a7b92f10770d547725b047078f99ee"}, @@ -2339,7 +2664,7 @@ description = "IPython: Productive Interactive Computing" optional = false python-versions = ">=3.11" groups = ["dev"] -markers = "python_version < \"3.14\"" +markers = "python_version < \"3.13\"" files = [ {file = "ipython-9.10.1-py3-none-any.whl", hash = "sha256:82d18ae9fb9164ded080c71ef92a182ee35ee7db2395f67616034bebb020a232"}, {file = "ipython-9.10.1.tar.gz", hash = "sha256:e170e9b2a44312484415bdb750492699bf329233b03f2557a9692cce6466ada4"}, @@ -2373,7 +2698,7 @@ description = "IPython: Productive Interactive Computing" optional = false python-versions = ">=3.12" groups = ["dev"] -markers = "python_version >= \"3.14\"" +markers = "python_version >= \"3.13\"" files = [ {file = "ipython-9.12.0-py3-none-any.whl", hash = "sha256:0f2701e8ee86e117e37f50563205d36feaa259d2e08d4a6bc6b6d74b18ce128d"}, {file = "ipython-9.12.0.tar.gz", hash = "sha256:01daa83f504b693ba523b5a407246cabde4eb4513285a3c6acaff11a66735ee4"}, @@ -2509,7 +2834,7 @@ files = [ {file = "jinja2-3.1.6-py3-none-any.whl", hash = "sha256:85ece4451f492d0c13c5dd7c13a64681a86afae63a5f347908daf103ce6d2f67"}, {file = "jinja2-3.1.6.tar.gz", hash = "sha256:0137fb05990d35f1275a587e9aee6d56da821fc83491a0fb838183be43f66d6d"}, ] -markers = {main = "extra == \"docs\" or extra == \"dbt\" or extra == \"all\" or extra == \"embedding-search\""} +markers = {main = "extra == \"dbt\" or extra == \"all\" or extra == \"advanced-search\" or extra == \"embedding-search\""} [package.dependencies] MarkupSafe = ">=2.0" @@ -2524,7 +2849,7 @@ description = "Fast iterable JSON parser." optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "jiter-0.14.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:02f36a5c700f105ac04a6556fe664a59037a2c200db3b7e88784fac2ddf02531"}, {file = "jiter-0.14.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:41eab6c09ceffb6f0fe25e214b3068146edb1eda3649ca2aee2a061029c7ba2e"}, @@ -2637,6 +2962,19 @@ files = [ {file = "jiter-0.14.0.tar.gz", hash = "sha256:e8a39e66dac7153cf3f964a12aad515afa8d74938ec5cc0018adcdae5367c79e"}, ] +[[package]] +name = "jmespath" +version = "1.1.0" +description = "JSON Matching Expressions" +optional = true +python-versions = ">=3.9" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "jmespath-1.1.0-py3-none-any.whl", hash = "sha256:a5663118de4908c91729bea0acadca56526eb2698e83de10cd116ae0f4e97c64"}, + {file = "jmespath-1.1.0.tar.gz", hash = "sha256:472c87d80f36026ae83c6ddd0f1d05d4e510134ed462851fd5f754c8c3cbb88d"}, +] + [[package]] name = "jpype1" version = "1.7.1" @@ -2645,6 +2983,13 @@ optional = false python-versions = ">=3.8" groups = ["dev"] files = [ + {file = "jpype1-1.7.1-1-cp310-cp310-macosx_11_0_universal2.whl", hash = "sha256:6590cbdb6208e4522fd99ae5f5f4bed5de707122385bc48446a1e7d7b56357ef"}, + {file = "jpype1-1.7.1-1-cp311-cp311-macosx_11_0_universal2.whl", hash = "sha256:4c81ee11aee5ed938d7415877cd9c7a0cc9cbf1dac87f7eab928e641323a385b"}, + {file = "jpype1-1.7.1-1-cp312-cp312-macosx_11_0_universal2.whl", hash = "sha256:b3ddd9f9099202212a34679dfb95dda590bcfbd23289559d104e24abec9120d1"}, + {file = "jpype1-1.7.1-1-cp313-cp313-macosx_11_0_universal2.whl", hash = "sha256:6d491a81281407f8a68552eb3c0e635e576e066c069268dc29a1ea27bb4778ae"}, + {file = "jpype1-1.7.1-1-cp314-cp314-macosx_11_0_universal2.whl", hash = "sha256:ace0ba1a67561358fa5b57b8e93ed8bcf16f0a8d5cba79c875089c56827adf8e"}, + {file = "jpype1-1.7.1-1-cp38-cp38-macosx_11_0_universal2.whl", hash = "sha256:0dc28836cb91218df78db9476e96e6567eb55366120837490edbfc54745048b4"}, + {file = "jpype1-1.7.1-1-cp39-cp39-macosx_11_0_universal2.whl", hash = "sha256:293f558ef43189afff2b501fdb37c7a578111f32d6b9863058d6439115b3d31e"}, {file = "jpype1-1.7.1-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:472b2f53002f5fdf118d2e6b8c6b5441d6e3ca3cf1b1bdb163442be76c8b2859"}, {file = "jpype1-1.7.1-cp310-cp310-manylinux_2_24_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:80c4c8cbab99040b8b56f28ff834e0b089aefccaabe3b472b8b43bb1e4658b86"}, {file = "jpype1-1.7.1-cp310-cp310-manylinux_2_24_i686.manylinux_2_28_i686.whl", hash = "sha256:9c9a08d06016afbe5391daaf843b9e76c79022181685bbb23b64cd3f9aaec30d"}, @@ -2677,8 +3022,10 @@ files = [ {file = "jpype1-1.7.1-cp314-cp314t-manylinux_2_24_i686.manylinux_2_28_i686.whl", hash = "sha256:7bef4ac17e0b0dbb96ee6afbd8878a5fa85353e3eb3eba4fe86e1df3dd62eb1b"}, {file = "jpype1-1.7.1-cp314-cp314t-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:b230c9475525b29114e6396b864c154f02f7cb041f2ac6bde006ed569e579aea"}, {file = "jpype1-1.7.1-cp38-cp38-macosx_14_0_x86_64.whl", hash = "sha256:9f1d0fb81becc32a231bd856bba9ddf4e49389cd6037154bb8c499e4b4eb14fd"}, + {file = "jpype1-1.7.1-cp38-cp38-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7dbbedb99ec99b703fe79b10de2c3430ec5ca181a690ccfa7346d350d171ffb4"}, {file = "jpype1-1.7.1-cp38-cp38-win_amd64.whl", hash = "sha256:89d57d48db2c96047c966a058a96cee53f19969220a792cb240d5e8835578a2e"}, {file = "jpype1-1.7.1-cp39-cp39-macosx_14_0_x86_64.whl", hash = "sha256:d70948f7665e837f9790c0d4aa0add4a555416dc1cd3108d15201a0e40facb64"}, + {file = "jpype1-1.7.1-cp39-cp39-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8fc7f35049f068571053931598c2a40a345053c32e8a839c4cee1ae99b06aaee"}, {file = "jpype1-1.7.1-cp39-cp39-win_amd64.whl", hash = "sha256:36696e850d07fabb920abe63371cc8fda6fa93d9ffeaa52176ddc49c629383dc"}, {file = "jpype1-1.7.1.tar.gz", hash = "sha256:3cd88838dc3d2d546f7eaeadaaff864e590010c15f2b6a44b6f37e60796a14b2"}, ] @@ -2975,6 +3322,111 @@ docs = ["autodoc-traits", "jinja2 (<3.2.0)", "mistune (<4)", "myst-parser", "pyd openapi = ["openapi-core (>=0.18.0,<0.19.0)", "ruamel-yaml"] test = ["hatch", "ipykernel", "openapi-core (>=0.18.0,<0.19.0)", "openapi-spec-validator (>=0.6.0,<0.8.0)", "pytest (>=7.0,<8)", "pytest-console-scripts", "pytest-cov", "pytest-jupyter[server] (>=0.6.2)", "pytest-timeout", "requests-mock", "ruamel-yaml", "sphinxcontrib-spelling", "strict-rfc3339", "werkzeug"] +[[package]] +name = "ladybug" +version = "0.15.3" +description = "Highly scalable, extremely fast, easy-to-use embeddable graph database" +optional = true +python-versions = "*" +groups = ["main"] +markers = "python_version >= \"3.14\" and (extra == \"advanced-search\" or extra == \"all\")" +files = [ + {file = "ladybug-0.15.3-cp310-cp310-macosx_13_0_arm64.whl", hash = "sha256:2db5c91361e0b1610a663c887a3b16c6e39f865c545003b42fe1230bef553511"}, + {file = "ladybug-0.15.3-cp310-cp310-macosx_13_0_x86_64.whl", hash = "sha256:a3d3d6748e851602a302a6a74c1369f7da71a29e1010ccee406a5de3e9eea554"}, + {file = "ladybug-0.15.3-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c9597b49b1087b2f6bf0bfc3d2343fa17e4584d1a223149344760fb3084fe60d"}, + {file = "ladybug-0.15.3-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:71c4bb93b0ff4132047042f9215f719570f81be522cbf553f2a09a433db1e4c4"}, + {file = "ladybug-0.15.3-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f73a236b8bb3f6fcfe14e85853b05e46ad669e12f8b0bad32635ae6fe3494119"}, + {file = "ladybug-0.15.3-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:74b52699e4a0e34da8b8b99c6c509b27721a63bb1f40a55eb904c54aead74614"}, + {file = "ladybug-0.15.3-cp310-cp310-win_amd64.whl", hash = "sha256:0a70a3fa89bce1eb3d814931b30046476f5f5d3fc024259e78bb2bc5ee4e7299"}, + {file = "ladybug-0.15.3-cp311-cp311-macosx_13_0_arm64.whl", hash = "sha256:1af0199af74a4bbe9dea2bd9206323e0a73610ef4cb850857fb7fa2ab5450577"}, + {file = "ladybug-0.15.3-cp311-cp311-macosx_13_0_x86_64.whl", hash = "sha256:18f2f0f51152335b56c87da3be30b303e4009984e949a8ed91fb6cde55913091"}, + {file = "ladybug-0.15.3-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9ba99b50793019de73803bc8a46592830e5faae676581a6f66f4d56444c90aeb"}, + {file = "ladybug-0.15.3-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1f782367e47aca8ad58d88cd1b043466235628100c39192660219dbe0591dc94"}, + {file = "ladybug-0.15.3-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:c5a0efcbfc5beb9645b1a71f03b57bc357f042b37a378674adaedb36cdf68ef6"}, + {file = "ladybug-0.15.3-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:128283f495bcdb1d65821df753237de1e2866d3dd6b43d60ed56a3b8eec43eb7"}, + {file = "ladybug-0.15.3-cp311-cp311-win_amd64.whl", hash = "sha256:82e4891dd38f781f03aa41c9e78c6a3fa067370733c4365ca28f36b6fd51323b"}, + {file = "ladybug-0.15.3-cp312-cp312-macosx_13_0_arm64.whl", hash = "sha256:8bc1d6408cb0b7a827426f0d916470be522a09fcf2ab970e512fa0697a1b942c"}, + {file = "ladybug-0.15.3-cp312-cp312-macosx_13_0_x86_64.whl", hash = "sha256:3f6d4b03fbb13f4a1080241de9947cae922911a912058ff714e3b281cd318154"}, + {file = "ladybug-0.15.3-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:99b04bbb0dd8c603837fb7baab85c43635f836e7ec8fffb478eb36331f2ae8dc"}, + {file = "ladybug-0.15.3-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8242c06143cb176a9ead085a3304aa8edb3979d54cc743ca5e7d7670c4fc03c8"}, + {file = "ladybug-0.15.3-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:baf9e27bd16419bd5913f88e5cfaafa11da8f8772fe4704749255531d9da3edb"}, + {file = "ladybug-0.15.3-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:eaf156bc0c6842f7b4dfc58a850c4f9026603b62e7ad5d1ec1a54f1391368c1c"}, + {file = "ladybug-0.15.3-cp312-cp312-win_amd64.whl", hash = "sha256:0583e1c4c14568da3a2df6f6bf8d72a05802e2be4aec8744976f0326a6410cd5"}, + {file = "ladybug-0.15.3-cp313-cp313-macosx_13_0_arm64.whl", hash = "sha256:01926ef51570c13b535ebb3d9accaa34303eba9f1c4325463d72d703cbbcfd94"}, + {file = "ladybug-0.15.3-cp313-cp313-macosx_13_0_x86_64.whl", hash = "sha256:ba095b21f18b1c47e51a70c9e640e10e42b7b122868a183b44290fac32caa3ae"}, + {file = "ladybug-0.15.3-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:37e4182514b454f63094c41dd55c78d42d693e8f4a2afa92068dd06512ce2e6d"}, + {file = "ladybug-0.15.3-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c52128275e8faec23aa15441c70368e238b831c2d79f5f8351f6fed541157bf5"}, + {file = "ladybug-0.15.3-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:ec3a32153152d235d1ccb51319d1e7cc3934ebd6af46e6649ceaf0e15a206abb"}, + {file = "ladybug-0.15.3-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:17df519815ed2795445119bc94d0d73ef60e1936500024abd8ae5feaa35bf7b5"}, + {file = "ladybug-0.15.3-cp313-cp313-win_amd64.whl", hash = "sha256:a5a874b8bab458b71f2d3634de4015f830de8953a86faf8f91aa00f0780576f1"}, + {file = "ladybug-0.15.3-cp314-cp314-macosx_13_0_arm64.whl", hash = "sha256:be6f5bb67b0335f6c7929301aef664ceb6271ad02c83d93b4a24b09b8120d9d6"}, + {file = "ladybug-0.15.3-cp314-cp314-macosx_13_0_x86_64.whl", hash = "sha256:8d40e8ebd1dbe652390e95116dc7cd6257d2f05fa206d46f30b232a3d0eef56c"}, + {file = "ladybug-0.15.3-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2b1ea519f56fae3a266e1ffa8850d3fb626ef9df7b59e03ab109f678ab750272"}, + {file = "ladybug-0.15.3-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:a38ff8223a8ff8eedeff71664560c68a115cfc6d552d5087a7c074953f8b1b57"}, + {file = "ladybug-0.15.3-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f88a754ea3b1c341fc3963d92441e8809015b6440997a5b6c994906f677ced29"}, + {file = "ladybug-0.15.3-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5983e2714f43d4f13301982f78399be6ba1c663d2117572a182d0b66770d9093"}, + {file = "ladybug-0.15.3-cp314-cp314-win_amd64.whl", hash = "sha256:67243bb6b47f8e699a513a060160e0497b96087f0da936b2973a02defe6c1f12"}, + {file = "ladybug-0.15.3.tar.gz", hash = "sha256:cbb6b672ec7f87714c7b26aa017b0d4ea1595194479a5544b20e6b4c6192e4c7"}, +] + +[package.extras] +dev = ["mypy (==1.16.0)", "networkx (>=3.0,<4.0)", "numpy (>=2.0,<3.0)", "pandas (>=2.2,<3.0)", "polars (>=1.30,<2.0)", "pyarrow (>=20.0,<21.0)", "pybind11 (>=2.13,<3.0)", "pytest", "pytest-asyncio (>=1.0,<2.0)", "ruff (==0.11.12)", "setuptools (>=80.9,<81.0)", "torch (>=2.5.0)", "torch-geometric (>=2.5.0)"] + +[[package]] +name = "ladybug" +version = "0.17.1" +description = "Highly scalable, extremely fast, easy-to-use embeddable graph database" +optional = true +python-versions = "<3.15,>=3.10" +groups = ["main"] +markers = "python_version <= \"3.13\" and (extra == \"advanced-search\" or extra == \"all\")" +files = [ + {file = "ladybug-0.17.1-cp310-cp310-macosx_13_0_arm64.whl", hash = "sha256:faf616d55f8a1269346c66913cabb32a88f9459878adadd7befb5e7adb206ea2"}, + {file = "ladybug-0.17.1-cp310-cp310-macosx_13_0_x86_64.whl", hash = "sha256:6b93d3dc7eb3827bfec82368d1196f78aa529dcddab848a06015a51d05ec622d"}, + {file = "ladybug-0.17.1-cp310-cp310-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:e0624025078a120b4e397d0e84a4de2a931a8a40434fa0594c49da5beee093ce"}, + {file = "ladybug-0.17.1-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3b69c79fafd7fbd2eaa2fa5ee7b891aa5ddf91f913b1d252a983175c1b8c4563"}, + {file = "ladybug-0.17.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:0ed7445e6bf200cc585f7e71e0d3a79b4a78718c1bf3fdcdee99dfadc0ce73c1"}, + {file = "ladybug-0.17.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:99a76128828245e39996c44ba693a754132b785c5db5e70f0e469e73ec24d103"}, + {file = "ladybug-0.17.1-cp310-cp310-win_amd64.whl", hash = "sha256:2748b1bb9e13c6af465f8089eb20e72fb4f1871603b1031eeb3390ab5b9f9baa"}, + {file = "ladybug-0.17.1-cp310-cp310-win_arm64.whl", hash = "sha256:0a7cea07905e7bdfb8a45bd80f3491ff6624ec34d0395d1887be0e4e0e2259d3"}, + {file = "ladybug-0.17.1-cp311-cp311-macosx_13_0_arm64.whl", hash = "sha256:b21e9101855a852c8f810e591d078adc0d28639c45bfaae4f20c20ad07164162"}, + {file = "ladybug-0.17.1-cp311-cp311-macosx_13_0_x86_64.whl", hash = "sha256:d6ed21cc33d636d9e69c9126a0e55142000faa3f402cc8f30a79f03f9c9c997b"}, + {file = "ladybug-0.17.1-cp311-cp311-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:afbf3096d18ab6300a29df15c2a77c936434d11bf9ef2b370076e13450592907"}, + {file = "ladybug-0.17.1-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:3cffc7023414c890f8be7841df8f41a9b3d489715ada9ab7426f943701e64eb6"}, + {file = "ladybug-0.17.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:7eb77583aee8618bcb8d174ec834950b821f44feb21655d502fda415d51e1194"}, + {file = "ladybug-0.17.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:8eb26cb7508541facc38e4d4282583469ef3927fbb99900eb4ae66632a55e215"}, + {file = "ladybug-0.17.1-cp311-cp311-win_amd64.whl", hash = "sha256:b2fd0e4e8948e15e567e8dcdfa38709906f6f1a66552cd44eea873a8ff664df9"}, + {file = "ladybug-0.17.1-cp311-cp311-win_arm64.whl", hash = "sha256:e1b881433e6623daa5c84eb80a9c1d6d5b4a16618c022fa028aa16b9de7a7dd8"}, + {file = "ladybug-0.17.1-cp312-cp312-macosx_13_0_arm64.whl", hash = "sha256:1a5fd5f324df8c5659a8cd6a0c08c58b2f6efbf0319cde6ec7cdcc3480343129"}, + {file = "ladybug-0.17.1-cp312-cp312-macosx_13_0_x86_64.whl", hash = "sha256:17e540cedd30816fad248e4d3cc6444bbc139a4b79e43af5f471acbe790a707a"}, + {file = "ladybug-0.17.1-cp312-cp312-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:067e183bd3fb2094a040cc4e69269fdcc5a770579a89f2d9f5dd8c93da6c08bc"}, + {file = "ladybug-0.17.1-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c11887032a14ca49941cbb5e9ac94adcba804c17fc7f823f598c8cceb5438612"}, + {file = "ladybug-0.17.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:5a3e906981abb4a92f3c2add315ae44419b1c5487bbedfa50eb03a0e2040dc38"}, + {file = "ladybug-0.17.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:bc0d0276b6181c26434f33eb675a304d33ce183fe6f90d04c91e7c001468b407"}, + {file = "ladybug-0.17.1-cp312-cp312-win_amd64.whl", hash = "sha256:4efc3f1117c16d7e9728afe9a0a87b89d4d3fb98c480d16d7f22c64be963a43b"}, + {file = "ladybug-0.17.1-cp312-cp312-win_arm64.whl", hash = "sha256:4c50a9e3e288cb57ede865b16e84bb4fd0471c572a641481e548cd70c4e82973"}, + {file = "ladybug-0.17.1-cp313-cp313-macosx_13_0_arm64.whl", hash = "sha256:785c95d595754be8d16b2e35defc44a906ccbef329c180e97e47ac7df17776bf"}, + {file = "ladybug-0.17.1-cp313-cp313-macosx_13_0_x86_64.whl", hash = "sha256:a168cdeea1f239b698ca222d680faa9b9b01b5ff96a1087b6235be23533d05e0"}, + {file = "ladybug-0.17.1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:fb2d88b4852689d73437c7c31057345fc95540e242c25e71999d541ad104f390"}, + {file = "ladybug-0.17.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:bf0ec0e0c0cb06be60ef83e3d95a3f5c02e3df6fe0270fee576b5688d0e70d28"}, + {file = "ladybug-0.17.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:8fd79f020c5b2db9d2b2b689ac2616f86d43ebc5d36063a001ee1ce158f50bf8"}, + {file = "ladybug-0.17.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:a9706c6a65365955ac3b18688a582c82ac3388362fdccab13a18e66b2c864e6b"}, + {file = "ladybug-0.17.1-cp313-cp313-win_amd64.whl", hash = "sha256:616d5017ee21ac99c4e1251f75c569053b709f73e207bd2d97f7742169680286"}, + {file = "ladybug-0.17.1-cp313-cp313-win_arm64.whl", hash = "sha256:5e84e5329f74e833828eeddd468ac881323fbfdb54a8610ae9f2bbe7bc02f952"}, + {file = "ladybug-0.17.1-cp314-cp314-macosx_13_0_arm64.whl", hash = "sha256:c8a71f90f2fe16396c98380aadd5519ed72346aed4d7872b06e2450e9470ac77"}, + {file = "ladybug-0.17.1-cp314-cp314-macosx_13_0_x86_64.whl", hash = "sha256:08934e417214ca03836934ed7bd1f6ebedf6959db3b4405541aa4389f60800a6"}, + {file = "ladybug-0.17.1-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2dbca5d77c309b6c992f5ab9f5d18b2a8002d4766f09ab7fa6ff9510df304b74"}, + {file = "ladybug-0.17.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:1f206b635840a26dd76e0b2f74e00e25e33fad1eb4e913bcce98cd5e5cf0146d"}, + {file = "ladybug-0.17.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:a4c6701f0abce984e7820f8242754682b7e6ed82787b1316b614c192458bac99"}, + {file = "ladybug-0.17.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:6c53cf98ae5b414af6f6f2025ccbb06dcd368eb1142e9a675ab24ccc4b936ed5"}, + {file = "ladybug-0.17.1-cp314-cp314-win_amd64.whl", hash = "sha256:48efb95673096a7524a1f16943a1885ad016944ea8a5a000746d9939cf2a039b"}, + {file = "ladybug-0.17.1-cp314-cp314-win_arm64.whl", hash = "sha256:2ad869f14b0307c7eb2f91639da7ab887c666c2f93260c46a055c7969e082a09"}, + {file = "ladybug-0.17.1.tar.gz", hash = "sha256:82efea498713c7da7f6f94f80252574599252d26ee10fb11e62ae932b5d8f2ba"}, +] + +[package.extras] +dev = ["mypy (==1.16.0)", "networkx (>=3.0,<4.0)", "numpy (>=2.0,<3.0)", "pandas (>=2.2,<3.0)", "polars (>=1.30,<2.0)", "pyarrow (>=21,<23)", "pybind11 (>=2.13,<3.0)", "pytest", "pytest-asyncio (>=1.0,<2.0)", "ruff (==0.11.12)", "setuptools (>=80.9,<81.0)", "torch (>=2.5.0)", "torch-geometric (>=2.5.0)"] + [[package]] name = "lark" version = "1.3.1" @@ -3029,7 +3481,7 @@ description = "Library to easily interface with LLM API providers" optional = true python-versions = "<4.0,>=3.9" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "litellm-1.83.0-py3-none-any.whl", hash = "sha256:88c536d339248f3987571493015784671ba3f193a328e1ea6780dbebaa2094a8"}, {file = "litellm-1.83.0.tar.gz", hash = "sha256:860bebc76c4bb27b4cf90b4a77acd66dba25aced37e3db98750de8a1766bfb7a"}, @@ -3132,23 +3584,6 @@ docs = ["sphinx (>=1.6.0)", "sphinx_bootstrap_theme"] flake8 = ["flake8"] tests = ["psutil", "pytest (!=3.3.0)", "pytest-cov"] -[[package]] -name = "markdown" -version = "3.10.2" -description = "Python implementation of John Gruber's Markdown." -optional = true -python-versions = ">=3.10" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "markdown-3.10.2-py3-none-any.whl", hash = "sha256:e91464b71ae3ee7afd3017d9f358ef0baf158fd9a298db92f1d4761133824c36"}, - {file = "markdown-3.10.2.tar.gz", hash = "sha256:994d51325d25ad8aa7ce4ebaec003febcce822c3f8c911e3b17c52f7f589f950"}, -] - -[package.extras] -docs = ["mdx_gh_links (>=0.2)", "mkdocs (>=1.6)", "mkdocs-gen-files", "mkdocs-literate-nav", "mkdocs-nature (>=0.6)", "mkdocs-section-index", "mkdocstrings[python] (>=0.28.3)"] -testing = ["coverage", "pyyaml"] - [[package]] name = "markdown-it-py" version = "4.0.0" @@ -3271,7 +3706,7 @@ files = [ {file = "markupsafe-3.0.3-cp39-cp39-win_arm64.whl", hash = "sha256:38664109c14ffc9e7437e86b4dceb442b0096dfe3541d7864d9cbe1da4cf36c8"}, {file = "markupsafe-3.0.3.tar.gz", hash = "sha256:722695808f4b6457b320fdc131280796bdceb04ab50fe1795cd540799ebe1698"}, ] -markers = {main = "extra == \"docs\" or extra == \"dbt\" or extra == \"all\" or extra == \"embedding-search\""} +markers = {main = "extra == \"dbt\" or extra == \"all\" or extra == \"advanced-search\" or extra == \"embedding-search\""} [[package]] name = "mashumaro" @@ -3359,19 +3794,6 @@ files = [ {file = "mdurl-0.1.2.tar.gz", hash = "sha256:bb413d29f5eea38f31dd4754dd7377d4465116fb207585f97bf925588687c1ba"}, ] -[[package]] -name = "mergedeep" -version = "1.3.4" -description = "A deep merge function for 🐍." -optional = true -python-versions = ">=3.6" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "mergedeep-1.3.4-py3-none-any.whl", hash = "sha256:70775750742b25c0d8f36c55aed03d24c3384d17c951b3175d898bd778ef0307"}, - {file = "mergedeep-1.3.4.tar.gz", hash = "sha256:0096d52e9dad9939c3d975a774666af186eda617e6ca84df4c94dec30004f2a8"}, -] - [[package]] name = "mirakuru" version = "3.0.2" @@ -3399,100 +3821,6 @@ files = [ {file = "mistune-3.2.0.tar.gz", hash = "sha256:708487c8a8cdd99c9d90eb3ed4c3ed961246ff78ac82f03418f5183ab70e398a"}, ] -[[package]] -name = "mkdocs" -version = "1.6.1" -description = "Project documentation with Markdown." -optional = true -python-versions = ">=3.8" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "mkdocs-1.6.1-py3-none-any.whl", hash = "sha256:db91759624d1647f3f34aa0c3f327dd2601beae39a366d6e064c03468d35c20e"}, - {file = "mkdocs-1.6.1.tar.gz", hash = "sha256:7b432f01d928c084353ab39c57282f29f92136665bdd6abf7c1ec8d822ef86f2"}, -] - -[package.dependencies] -click = ">=7.0" -colorama = {version = ">=0.4", markers = "platform_system == \"Windows\""} -ghp-import = ">=1.0" -jinja2 = ">=2.11.1" -markdown = ">=3.3.6" -markupsafe = ">=2.0.1" -mergedeep = ">=1.3.4" -mkdocs-get-deps = ">=0.2.0" -packaging = ">=20.5" -pathspec = ">=0.11.1" -pyyaml = ">=5.1" -pyyaml-env-tag = ">=0.1" -watchdog = ">=2.0" - -[package.extras] -i18n = ["babel (>=2.9.0)"] -min-versions = ["babel (==2.9.0)", "click (==7.0)", "colorama (==0.4) ; platform_system == \"Windows\"", "ghp-import (==1.0)", "importlib-metadata (==4.4) ; python_version < \"3.10\"", "jinja2 (==2.11.1)", "markdown (==3.3.6)", "markupsafe (==2.0.1)", "mergedeep (==1.3.4)", "mkdocs-get-deps (==0.2.0)", "packaging (==20.5)", "pathspec (==0.11.1)", "pyyaml (==5.1)", "pyyaml-env-tag (==0.1)", "watchdog (==2.0)"] - -[[package]] -name = "mkdocs-get-deps" -version = "0.2.2" -description = "An extra command for MkDocs that infers required PyPI packages from `plugins` in mkdocs.yml" -optional = true -python-versions = ">=3.9" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "mkdocs_get_deps-0.2.2-py3-none-any.whl", hash = "sha256:e7878cbeac04860b8b5e0ca31d3abad3df9411a75a32cde82f8e44b6c16ff650"}, - {file = "mkdocs_get_deps-0.2.2.tar.gz", hash = "sha256:8ee8d5f316cdbbb2834bc1df6e69c08fe769a83e040060de26d3c19fad3599a1"}, -] - -[package.dependencies] -mergedeep = ">=1.3.4" -platformdirs = ">=2.2.0" -pyyaml = ">=5.1" - -[[package]] -name = "mkdocs-material" -version = "9.7.6" -description = "Documentation that simply works" -optional = true -python-versions = ">=3.8" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "mkdocs_material-9.7.6-py3-none-any.whl", hash = "sha256:71b84353921b8ea1ba84fe11c50912cc512da8fe0881038fcc9a0761c0e635ba"}, - {file = "mkdocs_material-9.7.6.tar.gz", hash = "sha256:00bdde50574f776d328b1862fe65daeaf581ec309bd150f7bff345a098c64a69"}, -] - -[package.dependencies] -babel = ">=2.10" -backrefs = ">=5.7.post1" -colorama = ">=0.4" -jinja2 = ">=3.1" -markdown = ">=3.2" -mkdocs = ">=1.6,<2" -mkdocs-material-extensions = ">=1.3" -paginate = ">=0.5" -pygments = ">=2.16" -pymdown-extensions = ">=10.2" -requests = ">=2.30" - -[package.extras] -git = ["mkdocs-git-committers-plugin-2 (>=1.1)", "mkdocs-git-revision-date-localized-plugin (>=1.2.4)"] -imaging = ["cairosvg (>=2.6)", "pillow (>=10.2)"] -recommended = ["mkdocs-minify-plugin (>=0.7)", "mkdocs-redirects (>=1.2)", "mkdocs-rss-plugin (>=1.6)"] - -[[package]] -name = "mkdocs-material-extensions" -version = "1.3.1" -description = "Extension pack for Python Markdown and MkDocs Material." -optional = true -python-versions = ">=3.8" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "mkdocs_material_extensions-1.3.1-py3-none-any.whl", hash = "sha256:adff8b62700b25cb77b53358dad940f3ef973dd6db797907c49e3c2ef3ab4e31"}, - {file = "mkdocs_material_extensions-1.3.1.tar.gz", hash = "sha256:10c9511cea88f568257f960358a467d12b970e1f7b2c0e5fb2bb48cab1928443"}, -] - [[package]] name = "more-itertools" version = "10.8.0" @@ -3586,7 +3914,7 @@ description = "multidict implementation" optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "multidict-6.7.1-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:c93c3db7ea657dd4637d57e74ab73de31bccefe144d3d4ce370052035bc85fb5"}, {file = "multidict-6.7.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:974e72a2474600827abaeda71af0c53d9ebbc3c2eb7da37b37d7829ae31232d8"}, @@ -3861,7 +4189,7 @@ description = "Python package for creating and manipulating graphs and networks" optional = true python-versions = "!=3.14.1,>=3.11" groups = ["main"] -markers = "python_version < \"3.14\" and (extra == \"dbt\" or extra == \"all\")" +markers = "python_version <= \"3.13\" and (extra == \"dbt\" or extra == \"all\")" files = [ {file = "networkx-3.6.1-py3-none-any.whl", hash = "sha256:d47fbf302e7d9cbbb9e2555a0d267983d2aa476bac30e90dfbe5669bd57f3762"}, {file = "networkx-3.6.1.tar.gz", hash = "sha256:26b7c357accc0c8cde558ad486283728b65b6a95d85ee1cd66bafab4c8168509"}, @@ -4021,7 +4349,7 @@ description = "The official Python library for the openai API" optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "openai-2.36.0-py3-none-any.whl", hash = "sha256:143f6194b548dbc2c921af1f1b03b9f14c85fed8a75b5b516f5bcc11a2a50c63"}, {file = "openai-2.36.0.tar.gz", hash = "sha256:139dea0edd2f1b30c33d46ae1a6929e03906254140318e4608e98fe8c566f2e7"}, @@ -4088,23 +4416,6 @@ files = [ {file = "packaging-26.1.tar.gz", hash = "sha256:f042152b681c4bfac5cae2742a55e103d27ab2ec0f3d88037136b6bfe7c9c5de"}, ] -[[package]] -name = "paginate" -version = "0.5.7" -description = "Divides large result sets into pages for easier browsing" -optional = true -python-versions = "*" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "paginate-0.5.7-py2.py3-none-any.whl", hash = "sha256:b885e2af73abcf01d9559fd5216b57ef722f8c42affbb63942377668e35c7591"}, - {file = "paginate-0.5.7.tar.gz", hash = "sha256:22bd083ab41e1a8b4f3690544afb2c60c25e5c9a63a30fa2f483f6c60c8e5945"}, -] - -[package.extras] -dev = ["pytest", "tox"] -lint = ["black"] - [[package]] name = "pandas" version = "3.0.2" @@ -4246,7 +4557,7 @@ description = "Utility library for gitignore style pattern matching of file path optional = true python-versions = ">=3.8" groups = ["main"] -markers = "extra == \"docs\" or extra == \"dbt\" or extra == \"all\"" +markers = "extra == \"dbt\" or extra == \"all\"" files = [ {file = "pathspec-0.12.1-py3-none-any.whl", hash = "sha256:a0d503e138a4c123b27490a4f7beda6a01c6f288df0e4a8b79c7eb0dc7b4cc08"}, {file = "pathspec-0.12.1.tar.gz", hash = "sha256:a482d51503a1ab33b1c67a6c3813a26953dbdc71c31dacaef9a838c4e29f5712"}, @@ -4279,7 +4590,7 @@ files = [ {file = "platformdirs-4.9.6-py3-none-any.whl", hash = "sha256:e61adb1d5e5cb3441b4b7710bea7e4c12250ca49439228cc1021c00dcfac0917"}, {file = "platformdirs-4.9.6.tar.gz", hash = "sha256:3bfa75b0ad0db84096ae777218481852c0ebc6c727b3168c1b9e0118e458cf0a"}, ] -markers = {main = "extra == \"docs\""} +markers = {main = "extra == \"snowflake\" or extra == \"all\""} [[package]] name = "pluggy" @@ -4367,7 +4678,7 @@ description = "Accelerated property cache" optional = true python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "propcache-0.5.2-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:d5a81be28596d6559f6131ef33e10200de6e17643b3c74ce03f9eb103be6ae8b"}, {file = "propcache-0.5.2-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:29cbaac5ea0212663e6845e04b5e188d5a6ae6dd919810ac835bf1d3b42c3f4c"}, @@ -4492,6 +4803,25 @@ files = [ {file = "propcache-0.5.2.tar.gz", hash = "sha256:01c4fc7480cd0598bb4b57022df55b9ca296da7fc5a8760bd8451a7e63a7d427"}, ] +[[package]] +name = "proto-plus" +version = "1.28.0" +description = "Beautiful, Pythonic protocol buffers" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "proto_plus-1.28.0-py3-none-any.whl", hash = "sha256:a630604310899e73c59ec302e5765c058d412b2f090b9c79c8822589f14955b8"}, + {file = "proto_plus-1.28.0.tar.gz", hash = "sha256:38e5696342835b08fc116f30a25665b29531cda9d5d5643e9b81fc312385abd9"}, +] + +[package.dependencies] +protobuf = ">=4.25.8,<8.0.0" + +[package.extras] +testing = ["google-api-core (>=1.31.5)"] + [[package]] name = "protobuf" version = "6.33.6" @@ -4499,7 +4829,7 @@ description = "" optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"dbt\" or extra == \"all\"" +markers = "(extra == \"dbt\" or extra == \"all\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"dbt\") or extra == \"dbt\" or extra == \"all\"" files = [ {file = "protobuf-6.33.6-cp310-abi3-win32.whl", hash = "sha256:7d29d9b65f8afef196f8334e80d6bc1d5d4adedb449971fefd3723824e6e77d3"}, {file = "protobuf-6.33.6-cp310-abi3-win_amd64.whl", hash = "sha256:0cd27b587afca21b7cfa59a74dcbd48a50f0a6400cfb59391340ad729d91d326"}, @@ -4751,6 +5081,35 @@ files = [ {file = "pyarrow-24.0.0.tar.gz", hash = "sha256:85fe721a14dd823aca09127acbb06c3ca723efbd436c004f16bca601b04dcc83"}, ] +[[package]] +name = "pyasn1" +version = "0.6.3" +description = "Pure-Python implementation of ASN.1 types and DER/BER/CER codecs (X.208)" +optional = true +python-versions = ">=3.8" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "pyasn1-0.6.3-py3-none-any.whl", hash = "sha256:a80184d120f0864a52a073acc6fc642847d0be408e7c7252f31390c0f4eadcde"}, + {file = "pyasn1-0.6.3.tar.gz", hash = "sha256:697a8ecd6d98891189184ca1fa05d1bb00e2f84b5977c481452050549c8a72cf"}, +] + +[[package]] +name = "pyasn1-modules" +version = "0.4.2" +description = "A collection of ASN.1-based protocols modules" +optional = true +python-versions = ">=3.8" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "pyasn1_modules-0.4.2-py3-none-any.whl", hash = "sha256:29253a9207ce32b64c3ac6600edc75368f98473906e8fd1043bd6b5b1de2c14a"}, + {file = "pyasn1_modules-0.4.2.tar.gz", hash = "sha256:677091de870a80aae844b1ca6134f54652fa2c8c5a52aa396440ac3106e941e6"}, +] + +[package.dependencies] +pyasn1 = ">=0.6.1,<0.7.0" + [[package]] name = "pycparser" version = "3.0" @@ -4762,7 +5121,7 @@ files = [ {file = "pycparser-3.0-py3-none-any.whl", hash = "sha256:b727414169a36b7d524c1c3e31839a521725078d7b2ff038656844266160a992"}, {file = "pycparser-3.0.tar.gz", hash = "sha256:600f49d217304a5902ac3c37e1281c9fe94e4d0489de643a9504c5cdfdfc6b29"}, ] -markers = {main = "implementation_name != \"PyPy\" and platform_python_implementation != \"PyPy\"", dev = "implementation_name != \"PyPy\""} +markers = {main = "platform_python_implementation != \"PyPy\" and implementation_name != \"PyPy\"", dev = "implementation_name != \"PyPy\""} [[package]] name = "pydantic" @@ -5062,42 +5421,174 @@ docs = ["sphinx", "sphinx-rtd-theme", "zope.interface"] tests = ["coverage[toml] (==7.10.7)", "pytest (>=8.4.2,<9.0.0)"] [[package]] -name = "pymdown-extensions" -version = "10.21.2" -description = "Extension pack for Python Markdown." -optional = true +name = "pymssql" +version = "2.3.13" +description = "DB-API interface to Microsoft SQL Server for Python. (new Cython-based version)" +optional = false python-versions = ">=3.9" -groups = ["main"] -markers = "extra == \"docs\"" +groups = ["dev"] files = [ - {file = "pymdown_extensions-10.21.2-py3-none-any.whl", hash = "sha256:5c0fd2a2bea14eb39af8ff284f1066d898ab2187d81b889b75d46d4348c01638"}, - {file = "pymdown_extensions-10.21.2.tar.gz", hash = "sha256:c3f55a5b8a1d0edf6699e35dcbea71d978d34ff3fa79f3d807b8a5b3fa90fbdc"}, + {file = "pymssql-2.3.13-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:476f6f06b2ae5dfbfa0b169a6ecdd0d9ddfedb07f2d6dc97d2dd630ff2d6789a"}, + {file = "pymssql-2.3.13-cp310-cp310-macosx_15_0_x86_64.whl", hash = "sha256:17942dc9474693ab2229a8a6013e5b9cb1312a5251207552141bb85fcce8c131"}, + {file = "pymssql-2.3.13-cp310-cp310-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:4d87237500def5f743a52e415cd369d632907212154fcc7b4e13f264b4e30021"}, + {file = "pymssql-2.3.13-cp310-cp310-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:612ac062027d2118879f11a5986e9d9d82d07ca3545bb98c93200b68826ea687"}, + {file = "pymssql-2.3.13-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:f1897c1b767cc143e77d285123ae5fd4fa7379a1bfec5c515d38826caf084eb6"}, + {file = "pymssql-2.3.13-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:48631c7b9fd14a1bd5675c521b6082590bf700b7961c65638d237817b3fde735"}, + {file = "pymssql-2.3.13-cp310-cp310-win_amd64.whl", hash = "sha256:79c759db6e991eeae473b000c2e0a7fb8da799b2da469fe5a10d30916315e0b5"}, + {file = "pymssql-2.3.13-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:152be40c0d7f5e4b1323f7728b0a01f3ee0082190cfbadf84b2c2e930d57e00e"}, + {file = "pymssql-2.3.13-cp311-cp311-macosx_15_0_x86_64.whl", hash = "sha256:d94da3a55545c5b6926cb4d1c6469396f0ae32ad5d6932c513f7a0bf569b4799"}, + {file = "pymssql-2.3.13-cp311-cp311-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:51e42c5defc3667f0803c7ade85db0e6f24b9a1c5a18fcdfa2d09c36bff9b065"}, + {file = "pymssql-2.3.13-cp311-cp311-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:4aa18944a121f996178e26cadc598abdbf73759f03dc3cd74263fdab1b28cd96"}, + {file = "pymssql-2.3.13-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:910404e0ec85c4cc7c633ec3df9b04a35f23bb74a844dd377a387026ae635e3a"}, + {file = "pymssql-2.3.13-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:4b834c34e7600369eee7bc877948b53eb0fe6f3689f0888d005ae47dd53c0a66"}, + {file = "pymssql-2.3.13-cp311-cp311-win_amd64.whl", hash = "sha256:5c2e55b6513f9c5a2f58543233ed40baaa7f91c79e64a5f961ea3fc57a700b80"}, + {file = "pymssql-2.3.13-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:cf4f32b4a05b66f02cb7d55a0f3bcb0574a6f8cf0bee4bea6f7b104038364733"}, + {file = "pymssql-2.3.13-cp312-cp312-macosx_15_0_x86_64.whl", hash = "sha256:2b056eb175955f7fb715b60dc1c0c624969f4d24dbdcf804b41ab1e640a2b131"}, + {file = "pymssql-2.3.13-cp312-cp312-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:319810b89aa64b99d9c5c01518752c813938df230496fa2c4c6dda0603f04c4c"}, + {file = "pymssql-2.3.13-cp312-cp312-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c0ea72641cb0f8bce7ad8565dbdbda4a7437aa58bce045f2a3a788d71af2e4be"}, + {file = "pymssql-2.3.13-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:1493f63d213607f708a5722aa230776ada726ccdb94097fab090a1717a2534e0"}, + {file = "pymssql-2.3.13-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:eb3275985c23479e952d6462ae6c8b2b6993ab6b99a92805a9c17942cf3d5b3d"}, + {file = "pymssql-2.3.13-cp312-cp312-win_amd64.whl", hash = "sha256:a930adda87bdd8351a5637cf73d6491936f34e525a5e513068a6eac742f69cdb"}, + {file = "pymssql-2.3.13-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:30918bb044242865c01838909777ef5e0f1b9ecd7f5882346aefa57f4414b29c"}, + {file = "pymssql-2.3.13-cp313-cp313-macosx_15_0_x86_64.whl", hash = "sha256:1c6d0b2d7961f159a07e4f0d8cc81f70ceab83f5e7fd1e832a2d069e1d67ee4e"}, + {file = "pymssql-2.3.13-cp313-cp313-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:16c5957a3c9e51a03276bfd76a22431e2bc4c565e2e95f2cbb3559312edda230"}, + {file = "pymssql-2.3.13-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:0fddd24efe9d18bbf174fab7c6745b0927773718387f5517cf8082241f721a68"}, + {file = "pymssql-2.3.13-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:123c55ee41bc7a82c76db12e2eb189b50d0d7a11222b4f8789206d1cda3b33b9"}, + {file = "pymssql-2.3.13-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:e053b443e842f9e1698fcb2b23a4bff1ff3d410894d880064e754ad823d541e5"}, + {file = "pymssql-2.3.13-cp313-cp313-win_amd64.whl", hash = "sha256:5c045c0f1977a679cc30d5acd9da3f8aeb2dc6e744895b26444b4a2f20dad9a0"}, + {file = "pymssql-2.3.13-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:fc5482969c813b0a45ce51c41844ae5bfa8044ad5ef8b4820ef6de7d4545b7f2"}, + {file = "pymssql-2.3.13-cp314-cp314-macosx_15_0_x86_64.whl", hash = "sha256:ff5be7ab1d643dbce2ee3424d2ef9ae8e4146cf75bd20946bc7a6108e3ad1e47"}, + {file = "pymssql-2.3.13-cp314-cp314-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:8d66ce0a249d2e3b57369048d71e1f00d08dfb90a758d134da0250ae7bc739c1"}, + {file = "pymssql-2.3.13-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d663c908414a6a032f04d17628138b1782af916afc0df9fefac4751fa394c3ac"}, + {file = "pymssql-2.3.13-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:aa5e07eff7e6e8bd4ba22c30e4cb8dd073e138cd272090603609a15cc5dbc75b"}, + {file = "pymssql-2.3.13-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:db77da1a3fc9b5b5c5400639d79d7658ba7ad620957100c5b025be608b562193"}, + {file = "pymssql-2.3.13-cp314-cp314-win_amd64.whl", hash = "sha256:7d7037d2b5b907acc7906d0479924db2935a70c720450c41339146a4ada2b93d"}, + {file = "pymssql-2.3.13-cp39-cp39-macosx_14_0_arm64.whl", hash = "sha256:b0af51904764811da0bfe4b057b1d72dee11a399ce9ed5770875162772740c8a"}, + {file = "pymssql-2.3.13-cp39-cp39-macosx_15_0_x86_64.whl", hash = "sha256:0a7e6431925572bc75fb47929ae8ca5b0aac26abfe8b98d4c08daf117b5657f1"}, + {file = "pymssql-2.3.13-cp39-cp39-manylinux_2_27_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:7f9b1d5aef2b5f47a7f9d9733caee4d66772681e8f798a0f5e4739a8bdab408c"}, + {file = "pymssql-2.3.13-cp39-cp39-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c690f1869dadbf4201b7f51317fceff6e5d8f5175cec6a4a813e06b0dca2d6ed"}, + {file = "pymssql-2.3.13-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:e7c31f192da9d30f0e03ad99e548120a8740a675302e2f04fa8c929f7cbee771"}, + {file = "pymssql-2.3.13-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:f5d995a80996235ed32102a93067ce6a7143cce3bfd4e5042bf600020fc08456"}, + {file = "pymssql-2.3.13-cp39-cp39-win_amd64.whl", hash = "sha256:6a6c0783d97f57133573a03aad3017917dbdf7831a65e0d84ccf2a85e183ca66"}, + {file = "pymssql-2.3.13.tar.gz", hash = "sha256:2137e904b1a65546be4ccb96730a391fcd5a85aab8a0632721feb5d7e39cfbce"}, ] -[package.dependencies] -markdown = ">=3.6" -pyyaml = "*" - -[package.extras] -extra = ["pygments (>=2.19.1)"] - [[package]] name = "pymysql" version = "1.1.2" description = "Pure Python MySQL Driver" -optional = true +optional = false python-versions = ">=3.8" -groups = ["main"] -markers = "extra == \"mysql\" or extra == \"all\"" +groups = ["main", "dev"] files = [ {file = "pymysql-1.1.2-py3-none-any.whl", hash = "sha256:e6b1d89711dd51f8f74b1631fe08f039e7d76cf67a42a323d3178f0f25762ed9"}, {file = "pymysql-1.1.2.tar.gz", hash = "sha256:4961d3e165614ae65014e361811a724e2044ad3ea3739de9903ae7c21f539f03"}, ] +markers = {main = "extra == \"mysql\" or extra == \"all\""} + +[package.dependencies] +cryptography = {version = "*", optional = true, markers = "extra == \"rsa\""} [package.extras] ed25519 = ["PyNaCl (>=1.4.0)"] rsa = ["cryptography"] +[[package]] +name = "pyodbc" +version = "5.3.0" +description = "DB API module for ODBC" +optional = true +python-versions = ">=3.9" +groups = ["main"] +markers = "extra == \"sqlserver\" or extra == \"all\"" +files = [ + {file = "pyodbc-5.3.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:6682cdec78f1302d0c559422c8e00991668e039ed63dece8bf99ef62173376a5"}, + {file = "pyodbc-5.3.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:9cd3f0a9796b3e1170a9fa168c7e7ca81879142f30e20f46663b882db139b7d2"}, + {file = "pyodbc-5.3.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:46185a1a7f409761716c71de7b95e7bbb004390c650d00b0b170193e3d6224bb"}, + {file = "pyodbc-5.3.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:349a9abae62a968b98f6bbd23d2825151f8d9de50b3a8f5f3271b48958fdb672"}, + {file = "pyodbc-5.3.0-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:ac23feb7ddaa729f6b840639e92f83ff0ccaa7072801d944f1332cd5f5b05f47"}, + {file = "pyodbc-5.3.0-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:8aa396c6d6af52ccd51b8c8a5bffbb46fd44e52ce07ea4272c1d28e5e5b12722"}, + {file = "pyodbc-5.3.0-cp310-cp310-win32.whl", hash = "sha256:46869b9a6555ff003ed1d8ebad6708423adf2a5c88e1a578b9f029fb1435186e"}, + {file = "pyodbc-5.3.0-cp310-cp310-win_amd64.whl", hash = "sha256:705903acf6f43c44fc64e764578d9a88649eb21bf7418d78677a9d2e337f56f2"}, + {file = "pyodbc-5.3.0-cp310-cp310-win_arm64.whl", hash = "sha256:c68d9c225a97aedafb7fff1c0e1bfe293093f77da19eaf200d0e988fa2718d16"}, + {file = "pyodbc-5.3.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ebc3be93f61ea0553db88589e683ace12bf975baa954af4834ab89f5ee7bf8ae"}, + {file = "pyodbc-5.3.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:9b987a25a384f31e373903005554230f5a6d59af78bce62954386736a902a4b3"}, + {file = "pyodbc-5.3.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:676031723aac7dcbbd2813bddda0e8abf171b20ec218ab8dfb21d64a193430ea"}, + {file = "pyodbc-5.3.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:c5c30c5cd40b751f77bbc73edd32c4498630939bcd4e72ee7e6c9a4b982cc5ca"}, + {file = "pyodbc-5.3.0-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:2035c7dfb71677cd5be64d3a3eb0779560279f0a8dc6e33673499498caa88937"}, + {file = "pyodbc-5.3.0-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:5cbe4d753723c8a8f65020b7a259183ef5f14307587165ce37e8c7e251951852"}, + {file = "pyodbc-5.3.0-cp311-cp311-win32.whl", hash = "sha256:d255f6b117d05cfc046a5201fdf39535264045352ea536c35777cf66d321fbb8"}, + {file = "pyodbc-5.3.0-cp311-cp311-win_amd64.whl", hash = "sha256:f1ad0e93612a6201621853fc661209d82ff2a35892b7d590106fe8f97d9f1f2a"}, + {file = "pyodbc-5.3.0-cp311-cp311-win_arm64.whl", hash = "sha256:0df7ff47fab91ea05548095b00e5eb87ed88ddf4648c58c67b4db95ea4913e23"}, + {file = "pyodbc-5.3.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:5ebf6b5d989395efe722b02b010cb9815698a4d681921bf5db1c0e1195ac1bde"}, + {file = "pyodbc-5.3.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:197bb6ddafe356a916b8ee1b8752009057fce58e216e887e2174b24c7ab99269"}, + {file = "pyodbc-5.3.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:c6ccb5315ec9e081f5cbd66f36acbc820ad172b8fa3736cf7f993cdf69bd8a96"}, + {file = "pyodbc-5.3.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5dd3d5e469f89a3112cf8b0658c43108a4712fad65e576071e4dd44d2bd763c7"}, + {file = "pyodbc-5.3.0-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:b180bc5e49b74fd40a24ef5b0fe143d0c234ac1506febe810d7434bf47cb925b"}, + {file = "pyodbc-5.3.0-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:e3c39de3005fff3ae79246f952720d44affc6756b4b85398da4c5ea76bf8f506"}, + {file = "pyodbc-5.3.0-cp312-cp312-win32.whl", hash = "sha256:d32c3259762bef440707098010035bbc83d1c73d81a434018ab8c688158bd3bb"}, + {file = "pyodbc-5.3.0-cp312-cp312-win_amd64.whl", hash = "sha256:fe77eb9dcca5fc1300c9121f81040cc9011d28cff383e2c35416e9ec06d4bc95"}, + {file = "pyodbc-5.3.0-cp312-cp312-win_arm64.whl", hash = "sha256:afe7c4ac555a8d10a36234788fc6cfc22a86ce37fc5ba88a1f75b3e6696665dc"}, + {file = "pyodbc-5.3.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:7e9ab0b91de28a5ab838ac4db0253d7cc8ce2452efe4ad92ee6a57b922bf0c24"}, + {file = "pyodbc-5.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:6132554ffbd7910524d643f13ce17f4a72f3a6824b0adef4e9a7f66efac96350"}, + {file = "pyodbc-5.3.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:1629af4706e9228d79dabb4863c11cceb22a6dab90700db0ef449074f0150c0d"}, + {file = "pyodbc-5.3.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5ceaed87ba2ea848c11223f66f629ef121f6ebe621f605cde9cfdee4fd9f4b68"}, + {file = "pyodbc-5.3.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:3cc472c8ae2feea5b4512e23b56e2b093d64f7cbc4b970af51da488429ff7818"}, + {file = "pyodbc-5.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:c79df54bbc25bce9f2d87094e7b39089c28428df5443d1902b0cc5f43fd2da6f"}, + {file = "pyodbc-5.3.0-cp313-cp313-win32.whl", hash = "sha256:c2eb0b08e24fe5c40c7ebe9240c5d3bd2f18cd5617229acee4b0a0484dc226f2"}, + {file = "pyodbc-5.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:01166162149adf2b8a6dc21a212718f205cabbbdff4047dc0c415af3fd85867e"}, + {file = "pyodbc-5.3.0-cp313-cp313-win_arm64.whl", hash = "sha256:363311bd40320b4a61454bebf7c38b243cd67c762ed0f8a5219de3ec90c96353"}, + {file = "pyodbc-5.3.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:3f1bdb3ce6480a17afaaef4b5242b356d4997a872f39e96f015cabef00613797"}, + {file = "pyodbc-5.3.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:7713c740a10f33df3cb08f49a023b7e1e25de0c7c99650876bbe717bc95ee780"}, + {file = "pyodbc-5.3.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cf18797a12e70474e1b7f5027deeeccea816372497e3ff2d46b15bec2d18a0cc"}, + {file = "pyodbc-5.3.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:08b2439500e212625471d32f8fde418075a5ddec556e095e5a4ba56d61df2dc6"}, + {file = "pyodbc-5.3.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:729c535341bb09c476f219d6f7ab194bcb683c4a0a368010f1cb821a35136f05"}, + {file = "pyodbc-5.3.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:c67e7f2ce649155ea89beb54d3b42d83770488f025cf3b6f39ca82e9c598a02e"}, + {file = "pyodbc-5.3.0-cp314-cp314-win32.whl", hash = "sha256:a48d731432abaee5256ed6a19a3e1528b8881f9cb25cb9cf72d8318146ea991b"}, + {file = "pyodbc-5.3.0-cp314-cp314-win_amd64.whl", hash = "sha256:58635a1cc859d5af3f878c85910e5d7228fe5c406d4571bffcdd281375a54b39"}, + {file = "pyodbc-5.3.0-cp314-cp314-win_arm64.whl", hash = "sha256:754d052030d00c3ac38da09ceb9f3e240e8dd1c11da8906f482d5419c65b9ef5"}, + {file = "pyodbc-5.3.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:f927b440c38ade1668f0da64047ffd20ec34e32d817f9a60d07553301324b364"}, + {file = "pyodbc-5.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:25c4cfb2c08e77bc6e82f666d7acd52f0e52a0401b1876e60f03c73c3b8aedc0"}, + {file = "pyodbc-5.3.0-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bc834567c2990584b9726cba365834d039380c9dbbcef3030ddeb00c6541b943"}, + {file = "pyodbc-5.3.0-cp314-cp314t-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:8339d3094858893c1a68ee1af93efc4dff18b8b65de54d99104b99af6306320d"}, + {file = "pyodbc-5.3.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:74528fe148980d0c735c0ebb4a4dc74643ac4574337c43c1006ac4d09593f92d"}, + {file = "pyodbc-5.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:d89a7f2e24227150c13be8164774b7e1f9678321a4248f1356a465b9cc17d31e"}, + {file = "pyodbc-5.3.0-cp314-cp314t-win32.whl", hash = "sha256:af4d8c9842fc4a6360c31c35508d6594d5a3b39922f61b282c2b4c9d9da99514"}, + {file = "pyodbc-5.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:bfeb3e34795d53b7d37e66dd54891d4f9c13a3889a8f5fe9640e56a82d770955"}, + {file = "pyodbc-5.3.0-cp314-cp314t-win_arm64.whl", hash = "sha256:13656184faa3f2d5c6f19b701b8f247342ed581484f58bf39af7315c054e69db"}, + {file = "pyodbc-5.3.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:0263323fc47082c2bf02562f44149446bbbfe91450d271e44bffec0c3143bfb1"}, + {file = "pyodbc-5.3.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:452e7911a35ee12a56b111ac5b596d6ed865b83fcde8427127913df53132759e"}, + {file = "pyodbc-5.3.0-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:b35b9983ad300e5aea82b8d1661fc9d3afe5868de527ee6bd252dd550e61ecd6"}, + {file = "pyodbc-5.3.0-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:e981db84fee4cebec67f41bd266e1e7926665f1b99c3f8f4ea73cd7f7666e381"}, + {file = "pyodbc-5.3.0-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:25b6766e56748eb1fc1d567d863e06cbb7b7c749a41dfed85db0031e696fa39a"}, + {file = "pyodbc-5.3.0-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:2eb7151ed0a1959cae65b6ac0454f5c8bbcd2d8bafeae66483c09d58b0c7a7fc"}, + {file = "pyodbc-5.3.0-cp39-cp39-win32.whl", hash = "sha256:fc5ac4f2165f7088e74ecec5413b5c304247949f9702c8853b0e43023b4187e8"}, + {file = "pyodbc-5.3.0-cp39-cp39-win_amd64.whl", hash = "sha256:c25dc9c41f61573bdcf61a3408c34b65e4c0f821b8f861ca7531b1353b389804"}, + {file = "pyodbc-5.3.0-cp39-cp39-win_arm64.whl", hash = "sha256:101313a21d2654df856a60e4a13763e4d9f6c5d3fd974bcf3fc6b4e86d1bbe8e"}, + {file = "pyodbc-5.3.0.tar.gz", hash = "sha256:2fe0e063d8fb66efd0ac6dc39236c4de1a45f17c33eaded0d553d21c199f4d05"}, +] + +[[package]] +name = "pyopenssl" +version = "26.2.0" +description = "Python wrapper module around the OpenSSL library" +optional = true +python-versions = ">=3.8" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "pyopenssl-26.2.0-py3-none-any.whl", hash = "sha256:4f9d971bc5298b8bc1fab282803da04bf000c755d4ad9d99b52de2569ca19a70"}, + {file = "pyopenssl-26.2.0.tar.gz", hash = "sha256:8c6fcecd1183a7fc897548dfe388b0cdb7f37e018200d8409cf33959dbe35387"}, +] + +[package.dependencies] +cryptography = ">=46.0.0,<49" +typing-extensions = {version = ">=4.9", markers = "python_version < \"3.13\" and python_version >= \"3.8\""} + +[package.extras] +docs = ["sphinx (!=5.2.0,!=5.2.0.post0,!=7.2.5)", "sphinx_rtd_theme"] +test = ["pretend", "pytest (>=3.0.1)", "pytest-rerunfailures"] + [[package]] name = "pytest" version = "9.0.3" @@ -5256,7 +5747,7 @@ version = "1.2.2" description = "Read key-value pairs from a .env file and set them as environment variables" optional = false python-versions = ">=3.10" -groups = ["main"] +groups = ["main", "dev"] files = [ {file = "python_dotenv-1.2.2-py3-none-any.whl", hash = "sha256:1d8214789a24de455a8b8bd8ae6fe3c6b69a5e3d64aa8a8e5d68e694bbcb285a"}, {file = "python_dotenv-1.2.2.tar.gz", hash = "sha256:2c371a91fbd7ba082c2c1dc1f8bf89ca22564a087c2c287cd9b662adde799cf3"}, @@ -5328,14 +5819,14 @@ files = [ name = "pytz" version = "2025.2" description = "World timezone definitions, modern and historical" -optional = true +optional = false python-versions = "*" -groups = ["main"] -markers = "extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\"" +groups = ["main", "dev"] files = [ {file = "pytz-2025.2-py2.py3-none-any.whl", hash = "sha256:5ddf76296dd8c44c26eb8f4b6f35488f3ccbf6fbbd7adee0b7262d43f0ec2f00"}, {file = "pytz-2025.2.tar.gz", hash = "sha256:360b9e3dbb49a209c21ad61809c7fb453643e048b38924c765813546746e81c3"}, ] +markers = {main = "extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"snowflake\""} [[package]] name = "pywin32" @@ -5343,7 +5834,7 @@ version = "311" description = "Python for Window Extensions" optional = false python-versions = "*" -groups = ["main"] +groups = ["main", "dev"] markers = "sys_platform == \"win32\"" files = [ {file = "pywin32-311-cp310-cp310-win32.whl", hash = "sha256:d03ff496d2a0cd4a5893504789d4a15399133fe82517455e78bad62efbb7f0a3"}, @@ -5478,22 +5969,6 @@ files = [ {file = "pyyaml-6.0.3.tar.gz", hash = "sha256:d76623373421df22fb4cf8817020cbb7ef15c725b9d5e45f17e189bfc384190f"}, ] -[[package]] -name = "pyyaml-env-tag" -version = "1.1" -description = "A custom YAML tag for referencing environment variables in YAML files." -optional = true -python-versions = ">=3.9" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "pyyaml_env_tag-1.1-py3-none-any.whl", hash = "sha256:17109e1a528561e32f026364712fee1264bc2ea6715120891174ed1b980d2e04"}, - {file = "pyyaml_env_tag-1.1.tar.gz", hash = "sha256:2eb38b75a2d21ee0475d6d97ec19c63287a7e140231e4214969d0eac923cd7ff"}, -] - -[package.dependencies] -pyyaml = "*" - [[package]] name = "pyzmq" version = "27.1.0" @@ -5641,7 +6116,7 @@ description = "Alternative regular expression module, to replace re." optional = true python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "regex-2026.5.9-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:a9e1328e17c84c1a5d22ec9f785ecef4a967fab9a42b6a8dc3bcbebd0a0c9e44"}, {file = "regex-2026.5.9-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:bfe1ce50cbfb569d74e1e4337da6468961f31dbea55fd85aa5de59c0947a805a"}, @@ -5770,7 +6245,7 @@ files = [ {file = "requests-2.33.1-py3-none-any.whl", hash = "sha256:4e6d1ef462f3626a1f0a0a9c42dd93c63bad33f9f1c1937509b8c5c8718ab56a"}, {file = "requests-2.33.1.tar.gz", hash = "sha256:18817f8c57c6263968bc123d237e3b8b08ac046f5456bd1e307ee8f4250d3517"}, ] -markers = {main = "extra == \"docs\" or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"embedding-search\""} +markers = {main = "(extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"clickhouse\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\") or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\""} [package.dependencies] certifi = ">=2023.5.7" @@ -5999,6 +6474,25 @@ files = [ {file = "ruff-0.15.10.tar.gz", hash = "sha256:d1f86e67ebfdef88e00faefa1552b5e510e1d35f3be7d423dc7e84e63788c94e"}, ] +[[package]] +name = "s3transfer" +version = "0.18.0" +description = "An Amazon S3 Transfer Manager" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "s3transfer-0.18.0-py3-none-any.whl", hash = "sha256:239c13b09e65ad0346e1be7348b8a202dcad44ac7ea7c6eb858fc881dce739b6"}, + {file = "s3transfer-0.18.0.tar.gz", hash = "sha256:3760b8b7ec1315da54048b2d626276732bee4300d054d492d4e1d43e20d4ecbd"}, +] + +[package.dependencies] +botocore = ">=1.37.4,<2.0a0" + +[package.extras] +crt = ["botocore[crt] (>=1.37.4,<2.0a0)"] + [[package]] name = "send2trash" version = "2.1.0" @@ -6067,12 +6561,95 @@ description = "Sniff out which async library your code is running under" optional = true python-versions = ">=3.7" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "sniffio-1.3.1-py3-none-any.whl", hash = "sha256:2f6da418d1f1e0fddd844478f41680e794e6051915791a034ff65e5f100525a2"}, {file = "sniffio-1.3.1.tar.gz", hash = "sha256:f4324edc670a0f49750a81b895f35c3adb843cca46f0530f79fc1babb23789dc"}, ] +[[package]] +name = "snowflake-connector-python" +version = "4.6.0" +description = "Snowflake Connector for Python" +optional = true +python-versions = ">=3.10" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "snowflake_connector_python-4.6.0-cp310-cp310-macosx_14_0_arm64.whl", hash = "sha256:3ff98c3213674c5ed18ba6bb9288c4e88e790150f350824434d49a23d15c0fc3"}, + {file = "snowflake_connector_python-4.6.0-cp310-cp310-macosx_14_0_x86_64.whl", hash = "sha256:04ea8906ac06bdf98ab265f7870b532f32dd2b0f6b3b06a542b6e25a43e01665"}, + {file = "snowflake_connector_python-4.6.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:324b15278ee84ea6f0af7fef5e916778c23c4569b2c8ba7fdc90d288478772b9"}, + {file = "snowflake_connector_python-4.6.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:fe9005d226b234bf190409e5d7e8db9f7daba271880de9105f5173a6858b8e6b"}, + {file = "snowflake_connector_python-4.6.0-cp310-cp310-win_amd64.whl", hash = "sha256:8edc8bbcbaaa25a08d43f943fe45f00dc465684ef243859b0f3f7498d800f1ce"}, + {file = "snowflake_connector_python-4.6.0-cp311-cp311-macosx_14_0_arm64.whl", hash = "sha256:03b0a232d8d0a1c78eb0d4e9f8a422a1553b2f69ef1387d50a3223bb1829a249"}, + {file = "snowflake_connector_python-4.6.0-cp311-cp311-macosx_14_0_x86_64.whl", hash = "sha256:531dcb07eee8405e5d8a9f4e7f8c1ca7916e3afbb4ffb3dd2c9a12ec5bd0e46a"}, + {file = "snowflake_connector_python-4.6.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:c3124fd4a5dc702173ccd73d821ceba1442134d5f347b4c8d1ecb76489f44671"}, + {file = "snowflake_connector_python-4.6.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:7ab64f46b18d77d1e6c159a29cd86eeff0be9ff01a9904fa873a3c29d20063d1"}, + {file = "snowflake_connector_python-4.6.0-cp311-cp311-win_amd64.whl", hash = "sha256:18cc5402695b8e958503d6d7ab96403db90c481b63c31520305876ef3cb797e9"}, + {file = "snowflake_connector_python-4.6.0-cp312-cp312-macosx_14_0_arm64.whl", hash = "sha256:a7701b702dbeb348769c5d1248231e18544c4ff1fb4118ad73d48e8f801cfb6e"}, + {file = "snowflake_connector_python-4.6.0-cp312-cp312-macosx_14_0_x86_64.whl", hash = "sha256:00abbcfe958f60da18297191f3499b1e61802e64622521a2e8da1c059c14e1c0"}, + {file = "snowflake_connector_python-4.6.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:72aaee21a70e00fbe4dadcc60b9b1012b6411dddc90f94804d5efe5706fb9621"}, + {file = "snowflake_connector_python-4.6.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:6d3f6120edeb0d6edd208831d006cc3e769ec51bc346727f22d7aeaecbf20f77"}, + {file = "snowflake_connector_python-4.6.0-cp312-cp312-win_amd64.whl", hash = "sha256:f15e2493a316ce79ab3d7fb16add10252bb2401723e5cfbc7a2ebc44d89a7b2b"}, + {file = "snowflake_connector_python-4.6.0-cp313-cp313-macosx_14_0_arm64.whl", hash = "sha256:e0ca5a035b1afa690fb36a767ba59c8db85ef6295b88c2bbc2040449e99992ad"}, + {file = "snowflake_connector_python-4.6.0-cp313-cp313-macosx_14_0_x86_64.whl", hash = "sha256:1894504c69a76ac4a205d01fbb3e18c6a6e974e6ad26dad263edd06343bea501"}, + {file = "snowflake_connector_python-4.6.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:ed40d1e9d867253596860b9d5240280489ff4692b7a3fa21e2d45d63b4b61d36"}, + {file = "snowflake_connector_python-4.6.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:1c8476781cfef961fc5f6f75a5238e668d3e0ca5ebf1d055661b2fcf2831c254"}, + {file = "snowflake_connector_python-4.6.0-cp313-cp313-win_amd64.whl", hash = "sha256:e8ccbf8b5e12177a86bd3ab8292cc5a99e9ac97d7645ef4a3ed0f767b4ec6594"}, + {file = "snowflake_connector_python-4.6.0-cp314-cp314-macosx_14_0_arm64.whl", hash = "sha256:1fe93d88278a0b7e0efde6140890bc298a49fbf1e04968a35aa22c801131cced"}, + {file = "snowflake_connector_python-4.6.0-cp314-cp314-macosx_14_0_x86_64.whl", hash = "sha256:0829d57467bf1bb5af411f6e7723058cb2218fb7df07cf15d912e3b1a2c126eb"}, + {file = "snowflake_connector_python-4.6.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.whl", hash = "sha256:676162cd45df744aa966483960d34bf204cdcae87cecad77fba970f1c2fd570d"}, + {file = "snowflake_connector_python-4.6.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.whl", hash = "sha256:eab420406a38ebc059100bb1faa55d7d6306bb224cefadb739ec3cafeff65384"}, + {file = "snowflake_connector_python-4.6.0-cp314-cp314-win_amd64.whl", hash = "sha256:9dd8689123a7e7b873db0846f2d92745a02062b16665d20634fbaf34a9c88e7a"}, + {file = "snowflake_connector_python-4.6.0.tar.gz", hash = "sha256:06e2dba02703da6fd60e07bb0574506f810a85e5831d3461247753ecce4b8335"}, +] + +[package.dependencies] +asn1crypto = ">0.24.0,<2.0.0" +boto3 = ">=1.24" +botocore = ">=1.24" +certifi = ">=2024.7.4" +charset_normalizer = ">=2,<4" +cryptography = ">=46.0.5" +filelock = ">=3.5,<4" +idna = ">=3.7,<4" +packaging = "*" +platformdirs = ">=2.6.0,<5.0.0" +pyjwt = ">=2.10.1,<3.0.0" +pyOpenSSL = ">=24.0.0" +pytz = "*" +requests = ">=2.32.4,<3.0.0" +sortedcontainers = ">=2.4.0" +tomlkit = "*" +typing_extensions = ">=4.3,<5" + +[package.extras] +boto = ["boto3 (>=1.24)", "botocore (>=1.24)"] +development = ["Cython", "coverage", "mitmproxy (>=12.0.0) ; python_version >= \"3.12\"", "more-itertools", "numpy (<=2.4.3)", "pendulum (!=2.1.1)", "pexpect", "pytest (<7.5.0)", "pytest-asyncio", "pytest-cov", "pytest-rerunfailures (<16.0)", "pytest-timeout", "pytest-xdist", "pytzdata", "responses"] +pandas = ["pandas (>=1.0.0,<3.0.0) ; python_version < \"3.13\"", "pandas (>=2.1.2,<3.0.0) ; python_version >= \"3.13\"", "pyarrow (>=14.0.1)"] +secure-local-storage = ["keyring (>=23.1.0,<26.0.0)"] + +[[package]] +name = "snowflake-sqlalchemy" +version = "1.10.0" +description = "Snowflake SQLAlchemy Dialect" +optional = true +python-versions = ">=3.8" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "snowflake_sqlalchemy-1.10.0-py3-none-any.whl", hash = "sha256:e5c1c62d2a203b93beb4a813f6d07f2ea03f2151af90314037b7f4827734cdb0"}, + {file = "snowflake_sqlalchemy-1.10.0.tar.gz", hash = "sha256:70d0caf62ed429e080103211bfa7f221414d7ed7a08caa4c455538159e9b4520"}, +] + +[package.dependencies] +snowflake-connector-python = "<5.0.0" +sqlalchemy = ">=1.4.19" + +[package.extras] +development = ["alembic", "numpy", "pre-commit", "pytest", "pytest-cov", "pytest-rerunfailures", "pytest-timeout", "pytest-xdist", "pytz", "setuptools", "syrupy"] +pandas = ["snowflake-connector-python[pandas]"] + [[package]] name = "snowplow-tracker" version = "1.1.0" @@ -6093,6 +6670,19 @@ typing-extensions = ">=3.7.4" [package.extras] typing = ["mypy (>=0.971)", "types-requests (>=2.25.1,<3.0)"] +[[package]] +name = "sortedcontainers" +version = "2.4.0" +description = "Sorted Containers -- Sorted List, Sorted Dict, Sorted Set" +optional = true +python-versions = "*" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "sortedcontainers-2.4.0-py2.py3-none-any.whl", hash = "sha256:a163dcaede0f1c021485e957a39245190e74249897e2ae4b2aa38595db237ee0"}, + {file = "sortedcontainers-2.4.0.tar.gz", hash = "sha256:25caa5a06cc30b6b83d11423433f65d1f9d76c4c6a0c90e3379eaa43b9bfdb88"}, +] + [[package]] name = "soupsieve" version = "2.8.3" @@ -6207,63 +6797,89 @@ postgresql-psycopgbinary = ["psycopg[binary] (>=3.0.7)"] pymysql = ["pymysql"] sqlcipher = ["sqlcipher3_binary"] +[[package]] +name = "sqlalchemy-bigquery" +version = "1.17.0" +description = "SQLAlchemy dialect for BigQuery" +optional = true +python-versions = "<3.15,>=3.10" +groups = ["main"] +markers = "python_version < \"3.15\" and (extra == \"bigquery\" or extra == \"all\")" +files = [ + {file = "sqlalchemy_bigquery-1.17.0-py3-none-any.whl", hash = "sha256:89c1d4fc9f045ce762c93bf4b73a6c51a203dcf0dbe2d9ade540c7c5e3ed01dd"}, + {file = "sqlalchemy_bigquery-1.17.0.tar.gz", hash = "sha256:472284546a0c79cbf99b1bb0f5f99c5131fa888ea25d2d53208e6863e5094e2f"}, +] + +[package.dependencies] +google-api-core = ">=2.11.1,<3.0.0" +google-auth = ">=2.14.1,<3.0.0" +google-cloud-bigquery = ">=3.20.0,<4.0.0" +packaging = "*" +sqlalchemy = ">=1.4.16,<3.0.0" + +[package.extras] +alembic = ["alembic"] +all = ["GeoAlchemy2", "alembic", "google-cloud-bigquery-storage (>=2.18.0,<3.0.0)", "grpcio (>=1.47.0,<2.0.0)", "grpcio (>=1.49.1,<2.0.0) ; python_version == \"3.11\"", "grpcio (>=1.75.1,<2.0.0) ; python_version >= \"3.14\"", "packaging", "pyarrow (>=6.0.0)", "pytz", "shapely"] +bqstorage = ["google-cloud-bigquery-storage (>=2.18.0,<3.0.0)", "grpcio (>=1.47.0,<2.0.0)", "grpcio (>=1.49.1,<2.0.0) ; python_version == \"3.11\"", "grpcio (>=1.75.1,<2.0.0) ; python_version >= \"3.14\"", "pyarrow (>=6.0.0)"] +geography = ["GeoAlchemy2", "shapely"] +tests = ["packaging", "pytz"] + [[package]] name = "sqlglot" -version = "30.4.3" +version = "30.11.0" description = "An easily customizable SQL parser and transpiler" optional = false python-versions = ">=3.9" groups = ["main"] files = [ - {file = "sqlglot-30.4.3-py3-none-any.whl", hash = "sha256:58ea8e723444569da5cec91e4c8f16e385bce3f0ce0374b8c722c3088e1c1c7a"}, - {file = "sqlglot-30.4.3.tar.gz", hash = "sha256:3a4e9a1e1dd47f8e536ba822d77cb784681704da5e4a3e1a07d2ef86b6067826"}, + {file = "sqlglot-30.11.0-py3-none-any.whl", hash = "sha256:cffdee57d1f2f5472dc9f13087e618cf795841172b7d5ef78b63a051a52d2710"}, + {file = "sqlglot-30.11.0.tar.gz", hash = "sha256:1a23c6e2adb41da61fda46b1848d2fa26341d447fc0f0cd5ca21160362100991"}, ] [package.dependencies] -sqlglotc = {version = "30.4.3", optional = true, markers = "extra == \"c\""} +sqlglotc = {version = "30.11.0", optional = true, markers = "python_version >= \"3.10\" and extra == \"c\""} [package.extras] -c = ["sqlglotc (==30.4.3)"] -dev = ["duckdb (>=0.6)", "pandas", "pandas-stubs", "pdoc", "pre-commit", "pyperf", "python-dateutil", "pytz", "ruff (==0.15.6)", "setuptools_scm", "sqlglot-mypy", "types-python-dateutil", "types-pytz", "typing_extensions"] -rs = ["sqlglotc (==30.4.3)", "sqlglotrs (==0.13.0)"] +c = ["sqlglotc (==30.11.0) ; python_version >= \"3.10\""] +dev = ["duckdb (>=0.6)", "mypy ; python_version < \"3.10\"", "pandas", "pandas-stubs", "pdoc", "pre-commit", "pyperf", "python-dateutil", "pytz", "ruff (==0.15.6)", "setuptools_scm", "sqlglot-mypy (>=2.1.0.post2) ; python_version >= \"3.10\"", "types-python-dateutil", "types-pytz", "typing_extensions"] +rs = ["sqlglotc (==30.11.0) ; python_version >= \"3.10\"", "sqlglotrs (==0.13.0)"] [[package]] name = "sqlglotc" -version = "30.4.3" +version = "30.11.0" description = "mypyc-compiled extensions for sqlglot" optional = false -python-versions = ">=3.9" +python-versions = ">=3.10" groups = ["main"] files = [ - {file = "sqlglotc-30.4.3-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:d4b95ab5da3a1995446092674c13dcc391e5a97cc3a3944451cb060d57420841"}, - {file = "sqlglotc-30.4.3-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:bdd76c39b15a8c6a4c7cad5035be707ddcdfcb9ea100a04a0b1c9c97fc734a44"}, - {file = "sqlglotc-30.4.3-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ead5cb2f12fa3ada72d0da15683e4d2adfb2e89d71140e513dff795e1d7ddcd7"}, - {file = "sqlglotc-30.4.3-cp310-cp310-win_amd64.whl", hash = "sha256:07fe7c9aaaad4f9b0ea3de58cb3ab7ad773d69d31a77ab014f74c21d268f48d2"}, - {file = "sqlglotc-30.4.3-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:1c959a466c553e81a739d43880307098debc84ac2841cdca16768fb3c15af2b0"}, - {file = "sqlglotc-30.4.3-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:5fba18522e6cfc5f528220ac41ac2cf94207ba8d7e3ad2b2804925edf8add015"}, - {file = "sqlglotc-30.4.3-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5fa916534110c6e2f3044c04ddfaa56416fb486cb91527c8bd0bba9cae7a6b2e"}, - {file = "sqlglotc-30.4.3-cp311-cp311-win_amd64.whl", hash = "sha256:c59cdfba3fcaf213e9a728d8ad76373ba3e526cd35e0c7ca85b8af749524eeb3"}, - {file = "sqlglotc-30.4.3-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:431e6a514d50b1aa5b4a1c5d2cca137e472b2d4094afdea9a8bf676379049dc1"}, - {file = "sqlglotc-30.4.3-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0b81f31d3b0aa3e37b72b63397819fb52af10e1aca6dfa6bee9dc31ad65b27e4"}, - {file = "sqlglotc-30.4.3-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9f81d07e24fa4224c65a264e0af5a438a572ba06a3d995016664cfbd0e258650"}, - {file = "sqlglotc-30.4.3-cp312-cp312-win_amd64.whl", hash = "sha256:7e8af179f53ca65e78cfce77135912d1f86ccbf570472849f7531254020a7397"}, - {file = "sqlglotc-30.4.3-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:9aaa294f84ff35ebffd654bac4a8b632eee59187836d77820d0132b534c0c6ee"}, - {file = "sqlglotc-30.4.3-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9f2d949ab45b49e6b8be82d98447ca3e636b70ff1c2f2dea49e3fbf299d7b436"}, - {file = "sqlglotc-30.4.3-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:cc6e1eaee511c149e9c6e834fde24355afed1bbb87189caf46c7a2476283192c"}, - {file = "sqlglotc-30.4.3-cp313-cp313-win_amd64.whl", hash = "sha256:232ac97308d21b68b6e0ab026add69e34117bdd6b9ffa98d431e14325c5c6d63"}, - {file = "sqlglotc-30.4.3-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:d48d78489029f7cac3c6ffeefbec6cb646235f779afd3900e0bf764f4b229763"}, - {file = "sqlglotc-30.4.3-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:dd920c6771b1b622b7e023d4664441407015fb9140bb90ad18842d3b3cd60e4a"}, - {file = "sqlglotc-30.4.3-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:302b9831e1ef174caaac8d3d28b08844b44b3da1d8b5b845bf969e451ca4ccfb"}, - {file = "sqlglotc-30.4.3-cp314-cp314-win_amd64.whl", hash = "sha256:56b3467acedb651a33c53bfbc6edfc33f17fe4473f1917292f14d4d152b925cc"}, - {file = "sqlglotc-30.4.3-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:1ca9024cafa528fe9eb6c8c79dac35333b8edd3ec7f94e1a154cfe710c8796d3"}, - {file = "sqlglotc-30.4.3-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:9a0f6fa6e8b81929933c9d52e92b756135766380316156dcf098d04cda67259f"}, - {file = "sqlglotc-30.4.3-cp39-cp39-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:5b5c25ef3ff05c6d9b6746b321323ce22aaab99702023c012b4b87552e700a8a"}, - {file = "sqlglotc-30.4.3-cp39-cp39-win_amd64.whl", hash = "sha256:5d241f64087695e8ad42ef6f431367347574a45636a4cc629c1e406c7e97939c"}, - {file = "sqlglotc-30.4.3.tar.gz", hash = "sha256:10187877550509a1a546a5f1f24ea2077fa648b55944a9cbee8d9b569d01f0ff"}, -] - -[package.extras] -dev = ["setuptools (>=61.0)", "setuptools_scm", "sqlglot-mypy"] + {file = "sqlglotc-30.11.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:742c108c8089d39280c229eb7a57838fc907f3b01620de948502b2f3211cf301"}, + {file = "sqlglotc-30.11.0-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:76734c1684663fa2899a990446562c635c7057cc431bc1c79920e26423727488"}, + {file = "sqlglotc-30.11.0-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:44eb032a8ebd50bd7d835024f5f50363e1720fbbed1710bca8160f26ebeab59b"}, + {file = "sqlglotc-30.11.0-cp310-cp310-win_amd64.whl", hash = "sha256:3fb6b4aa751485d5415539762ff16668978436eff946de51985d811c23aba1d4"}, + {file = "sqlglotc-30.11.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:349e6b1e79baefebeda9ea457bb06a8f348aaacf0fc39c2f4b6a54ddef19cbb4"}, + {file = "sqlglotc-30.11.0-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:eedab4e58c2baa2bf6b9864505aaf7bdbde92c0370854dfdbce5a3cf72bc4be4"}, + {file = "sqlglotc-30.11.0-cp311-cp311-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:d5759b32f0492f5196f7611f9f5233ff636e91441d9b5cd2309c939cd532f11c"}, + {file = "sqlglotc-30.11.0-cp311-cp311-win_amd64.whl", hash = "sha256:64f0a3d05a92aef5911a2aed058af3195ea08a11a68b3d7befe6dd04d3c2aa66"}, + {file = "sqlglotc-30.11.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:2f6741bff8c92777ecdd05f97d5dcd61614c05b1a01dbf7b10198dfd176060d6"}, + {file = "sqlglotc-30.11.0-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ff9d1917f5238e6d8261c79bc911726787220fa85805beca348a36d4aecc71f8"}, + {file = "sqlglotc-30.11.0-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba3a0f215ba81c276759a4e6331f8af1948d803d38c3f5ed684119c5d6f1d82a"}, + {file = "sqlglotc-30.11.0-cp312-cp312-win_amd64.whl", hash = "sha256:9c1cdfc5544cc4e23d31b966f007cc2be16e697cef86b97cba2ffd246f344256"}, + {file = "sqlglotc-30.11.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:5c36388963d82aacb4287c7859cc40a08c15fa0dfaf0ad055679b53ba602016b"}, + {file = "sqlglotc-30.11.0-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a357a6bf6321796c8506520340feb5052bf76c658a2455aabbbdd7d85c8fd5da"}, + {file = "sqlglotc-30.11.0-cp313-cp313-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:ba16b178d2ad9ab1eac38fb1415434abecaf342d3a487c3128649fe8c0d1e500"}, + {file = "sqlglotc-30.11.0-cp313-cp313-win_amd64.whl", hash = "sha256:20e662593a96c0086edc46b336bc772d213901fd09add99a3326661793ba7018"}, + {file = "sqlglotc-30.11.0-cp314-cp314-macosx_10_15_universal2.whl", hash = "sha256:d49533b4a851f3603ecba9a724c32f227bca90cc0034b8a3ad66acdbde759a90"}, + {file = "sqlglotc-30.11.0-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a2cbeec4ec4591268585e5ad5f55fbee30ccd68227b5d2f77d483cdd63b65963"}, + {file = "sqlglotc-30.11.0-cp314-cp314-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:2f4bcc7b76634171edccc13aabba9c1bea51f97caf2187daacbe433004058dc3"}, + {file = "sqlglotc-30.11.0-cp314-cp314-win_amd64.whl", hash = "sha256:8c6ed84d557772880a082ef29d18b0c2cfee2d177e6528b30df2b7eaf6adb585"}, + {file = "sqlglotc-30.11.0.tar.gz", hash = "sha256:6ce71f4d31459df966f3752848f397e254ff5d4e2772594e539e23cff0f2bbc8"}, +] + +[package.dependencies] +sqlglot = "30.11.0" + +[package.extras] +dev = ["setuptools (>=61.0)", "setuptools_scm", "sqlglot-mypy (>=2.1.0.post2)"] [[package]] name = "sqlparse" @@ -6409,6 +7025,65 @@ docs = ["myst-parser", "pydata-sphinx-theme", "sphinx"] test = ["pre-commit", "pytest (>=7.0)", "pytest-timeout"] typing = ["mypy (>=1.6,<2.0)", "traitlets (>=5.11.1)"] +[[package]] +name = "testcontainers" +version = "4.14.2" +description = "Python library for throwaway instances of anything that can run in a Docker container" +optional = false +python-versions = ">=3.10" +groups = ["dev"] +files = [ + {file = "testcontainers-4.14.2-py3-none-any.whl", hash = "sha256:0d0522c3cd8f8d9627cda41f7a6b51b639fa57bdc492923c045117933c668d68"}, + {file = "testcontainers-4.14.2.tar.gz", hash = "sha256:1340ccf16fe3acd9389a6c9e1d9ab21d9fe99a8afdf8165f89c3e69c1967d239"}, +] + +[package.dependencies] +clickhouse-driver = {version = "*", optional = true, markers = "extra == \"clickhouse\""} +docker = "*" +pymssql = {version = ">=2", optional = true, markers = "extra == \"mssql\""} +pymysql = {version = ">=1", extras = ["rsa"], optional = true, markers = "extra == \"mysql\""} +python-dotenv = "*" +sqlalchemy = {version = ">=2", optional = true, markers = "extra == \"mssql\" or extra == \"mysql\""} +typing-extensions = "*" +urllib3 = "*" +wrapt = "*" + +[package.extras] +arangodb = ["python-arango (>=8)"] +aws = ["boto3 (>=1)", "httpx"] +azurite = ["azure-storage-blob (>=12)"] +chroma = ["chromadb-client (>=1)"] +clickhouse = ["clickhouse-driver"] +cosmosdb = ["azure-cosmos (>=4)"] +db2 = ["ibm-db-sa ; platform_machine != \"aarch64\" and platform_machine != \"arm64\"", "sqlalchemy (>=2)"] +generic = ["httpx", "redis (>=7)"] +google = ["google-cloud-datastore (>=2)", "google-cloud-pubsub (>=2)"] +influxdb = ["influxdb (>=5)", "influxdb-client (>=1)"] +k3s = ["kubernetes", "pyyaml (>=6.0.3)"] +keycloak = ["python-keycloak (>=6) ; python_version < \"4.0\""] +localstack = ["boto3 (>=1)"] +mailpit = ["cryptography"] +minio = ["minio (>=7)"] +mongodb = ["pymongo (>=4)"] +mssql = ["pymssql (>=2)", "sqlalchemy (>=2)"] +mysql = ["pymysql[rsa] (>=1)", "sqlalchemy (>=2)"] +nats = ["nats-py (>=2)"] +neo4j = ["neo4j (>=6)"] +openfga = ["openfga-sdk"] +opensearch = ["opensearch-py (>=3) ; python_version < \"4.0\""] +oracle = ["oracledb (>=3)", "sqlalchemy (>=2)"] +oracle-free = ["oracledb (>=3)", "sqlalchemy (>=2)"] +qdrant = ["qdrant-client (>=1)"] +rabbitmq = ["pika (>=1)"] +redis = ["redis (>=7)"] +registry = ["bcrypt (>=5)"] +scylla = ["cassandra-driver (>=3)"] +selenium = ["selenium (>=4)"] +sftp = ["cryptography"] +test-module-import = ["httpx"] +trino = ["trino"] +weaviate = ["weaviate-client (>=4)"] + [[package]] name = "text-unidecode" version = "1.3" @@ -6429,7 +7104,7 @@ description = "tiktoken is a fast BPE tokeniser for use with OpenAI's models" optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "tiktoken-0.12.0-cp310-cp310-macosx_10_12_x86_64.whl", hash = "sha256:3de02f5a491cfd179aec916eddb70331814bd6bf764075d39e21d5862e533970"}, {file = "tiktoken-0.12.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:b6cfb6d9b7b54d20af21a912bfe63a2727d9cfa8fbda642fd8322c70340aad16"}, @@ -6523,7 +7198,7 @@ description = "" optional = true python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "tokenizers-0.23.1-cp310-abi3-macosx_10_12_x86_64.whl", hash = "sha256:e03d6ffcbe0d56ee9c1ccd070e70a13fa750727c0277e138152acbc0252c2224"}, {file = "tokenizers-0.23.1-cp310-abi3-macosx_11_0_arm64.whl", hash = "sha256:e0948bbb1ac1d7cdfc9fb6d62c596e3b7550036ad60ecd654a66ad273326324e"}, @@ -6552,6 +7227,19 @@ dev = ["tokenizers[testing]"] docs = ["setuptools-rust", "sphinx", "sphinx-rtd-theme"] testing = ["datasets", "numpy", "pytest", "pytest-asyncio", "requests", "ruff", "ty"] +[[package]] +name = "tomlkit" +version = "0.15.0" +description = "Style preserving TOML library" +optional = true +python-versions = ">=3.9" +groups = ["main"] +markers = "extra == \"snowflake\" or extra == \"all\"" +files = [ + {file = "tomlkit-0.15.0-py3-none-any.whl", hash = "sha256:4dbc8f0fc024412b57ced8757ac7461305126a648ff8c2c807fcb8e133a78738"}, + {file = "tomlkit-0.15.0.tar.gz", hash = "sha256:7d1a9ecba3086638211b13814ea79c90dd54dd11993564376f3aa92271f5c7a3"}, +] + [[package]] name = "tornado" version = "6.5.5" @@ -6579,7 +7267,7 @@ description = "Fast, Extensible Progress Meter" optional = true python-versions = ">=3.7" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "tqdm-4.67.3-py3-none-any.whl", hash = "sha256:ee1e4c0e59148062281c49d80b25b67771a127c85fc9676d3be5f243206826bf"}, {file = "tqdm-4.67.3.tar.gz", hash = "sha256:7d825f03f89244ef73f1d4ce193cb1774a8179fd96f31d7e1dcde62092b960bb"}, @@ -6673,14 +7361,14 @@ markers = {main = "platform_system == \"Windows\" or (sys_platform == \"win32\" name = "tzlocal" version = "5.3.1" description = "tzinfo object for the local timezone" -optional = true +optional = false python-versions = ">=3.9" -groups = ["main"] -markers = "extra == \"clickhouse\" or extra == \"all\"" +groups = ["main", "dev"] files = [ {file = "tzlocal-5.3.1-py3-none-any.whl", hash = "sha256:eb1a66c3ef5847adf7a834f1be0800581b683b5608e74f86ecbcef8ab91bb85d"}, {file = "tzlocal-5.3.1.tar.gz", hash = "sha256:cceffc7edecefea1f595541dbd6e990cb1ea3d19bf01b2809f362a03dd7921fd"}, ] +markers = {main = "extra == \"clickhouse\" or extra == \"all\""} [package.dependencies] tzdata = {version = "*", markers = "platform_system == \"Windows\""} @@ -6714,7 +7402,7 @@ files = [ {file = "urllib3-2.6.3-py3-none-any.whl", hash = "sha256:bf272323e553dfb2e87d9bfd225ca7b0f467b919d7bbd355436d3fd37cb0acd4"}, {file = "urllib3-2.6.3.tar.gz", hash = "sha256:1b62b6884944a57dbe321509ab94fd4d3b307075e0c2eae991ac71ee15ad38ed"}, ] -markers = {main = "extra == \"docs\" or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"embedding-search\""} +markers = {main = "(extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\" or extra == \"bigquery\") and python_version < \"3.15\" or python_version == \"3.14\" and (extra == \"bigquery\" or extra == \"all\" or extra == \"clickhouse\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\") or extra == \"clickhouse\" or extra == \"all\" or extra == \"dbt\" or extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"snowflake\""} [package.extras] brotli = ["brotli (>=1.2.0) ; platform_python_implementation == \"CPython\"", "brotlicffi (>=1.2.0.0) ; platform_python_implementation != \"CPython\""] @@ -6759,50 +7447,6 @@ filelock = {version = ">=3.24.2,<4", markers = "python_version >= \"3.10\""} platformdirs = ">=3.9.1,<5" python-discovery = ">=1.2.2" -[[package]] -name = "watchdog" -version = "6.0.0" -description = "Filesystem events monitoring" -optional = true -python-versions = ">=3.9" -groups = ["main"] -markers = "extra == \"docs\"" -files = [ - {file = "watchdog-6.0.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:d1cdb490583ebd691c012b3d6dae011000fe42edb7a82ece80965b42abd61f26"}, - {file = "watchdog-6.0.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:bc64ab3bdb6a04d69d4023b29422170b74681784ffb9463ed4870cf2f3e66112"}, - {file = "watchdog-6.0.0-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:c897ac1b55c5a1461e16dae288d22bb2e412ba9807df8397a635d88f671d36c3"}, - {file = "watchdog-6.0.0-cp311-cp311-macosx_10_9_universal2.whl", hash = "sha256:6eb11feb5a0d452ee41f824e271ca311a09e250441c262ca2fd7ebcf2461a06c"}, - {file = "watchdog-6.0.0-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:ef810fbf7b781a5a593894e4f439773830bdecb885e6880d957d5b9382a960d2"}, - {file = "watchdog-6.0.0-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:afd0fe1b2270917c5e23c2a65ce50c2a4abb63daafb0d419fde368e272a76b7c"}, - {file = "watchdog-6.0.0-cp312-cp312-macosx_10_13_universal2.whl", hash = "sha256:bdd4e6f14b8b18c334febb9c4425a878a2ac20efd1e0b231978e7b150f92a948"}, - {file = "watchdog-6.0.0-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:c7c15dda13c4eb00d6fb6fc508b3c0ed88b9d5d374056b239c4ad1611125c860"}, - {file = "watchdog-6.0.0-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:6f10cb2d5902447c7d0da897e2c6768bca89174d0c6e1e30abec5421af97a5b0"}, - {file = "watchdog-6.0.0-cp313-cp313-macosx_10_13_universal2.whl", hash = "sha256:490ab2ef84f11129844c23fb14ecf30ef3d8a6abafd3754a6f75ca1e6654136c"}, - {file = "watchdog-6.0.0-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:76aae96b00ae814b181bb25b1b98076d5fc84e8a53cd8885a318b42b6d3a5134"}, - {file = "watchdog-6.0.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:a175f755fc2279e0b7312c0035d52e27211a5bc39719dd529625b1930917345b"}, - {file = "watchdog-6.0.0-cp39-cp39-macosx_10_9_universal2.whl", hash = "sha256:e6f0e77c9417e7cd62af82529b10563db3423625c5fce018430b249bf977f9e8"}, - {file = "watchdog-6.0.0-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:90c8e78f3b94014f7aaae121e6b909674df5b46ec24d6bebc45c44c56729af2a"}, - {file = "watchdog-6.0.0-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:e7631a77ffb1f7d2eefa4445ebbee491c720a5661ddf6df3498ebecae5ed375c"}, - {file = "watchdog-6.0.0-pp310-pypy310_pp73-macosx_10_15_x86_64.whl", hash = "sha256:c7ac31a19f4545dd92fc25d200694098f42c9a8e391bc00bdd362c5736dbf881"}, - {file = "watchdog-6.0.0-pp310-pypy310_pp73-macosx_11_0_arm64.whl", hash = "sha256:9513f27a1a582d9808cf21a07dae516f0fab1cf2d7683a742c498b93eedabb11"}, - {file = "watchdog-6.0.0-pp39-pypy39_pp73-macosx_10_15_x86_64.whl", hash = "sha256:7a0e56874cfbc4b9b05c60c8a1926fedf56324bb08cfbc188969777940aef3aa"}, - {file = "watchdog-6.0.0-pp39-pypy39_pp73-macosx_11_0_arm64.whl", hash = "sha256:e6439e374fc012255b4ec786ae3c4bc838cd7309a540e5fe0952d03687d8804e"}, - {file = "watchdog-6.0.0-py3-none-manylinux2014_aarch64.whl", hash = "sha256:7607498efa04a3542ae3e05e64da8202e58159aa1fa4acddf7678d34a35d4f13"}, - {file = "watchdog-6.0.0-py3-none-manylinux2014_armv7l.whl", hash = "sha256:9041567ee8953024c83343288ccc458fd0a2d811d6a0fd68c4c22609e3490379"}, - {file = "watchdog-6.0.0-py3-none-manylinux2014_i686.whl", hash = "sha256:82dc3e3143c7e38ec49d61af98d6558288c415eac98486a5c581726e0737c00e"}, - {file = "watchdog-6.0.0-py3-none-manylinux2014_ppc64.whl", hash = "sha256:212ac9b8bf1161dc91bd09c048048a95ca3a4c4f5e5d4a7d1b1a7d5752a7f96f"}, - {file = "watchdog-6.0.0-py3-none-manylinux2014_ppc64le.whl", hash = "sha256:e3df4cbb9a450c6d49318f6d14f4bbc80d763fa587ba46ec86f99f9e6876bb26"}, - {file = "watchdog-6.0.0-py3-none-manylinux2014_s390x.whl", hash = "sha256:2cce7cfc2008eb51feb6aab51251fd79b85d9894e98ba847408f662b3395ca3c"}, - {file = "watchdog-6.0.0-py3-none-manylinux2014_x86_64.whl", hash = "sha256:20ffe5b202af80ab4266dcd3e91aae72bf2da48c0d33bdb15c66658e685e94e2"}, - {file = "watchdog-6.0.0-py3-none-win32.whl", hash = "sha256:07df1fdd701c5d4c8e55ef6cf55b8f0120fe1aef7ef39a1c6fc6bc2e606d517a"}, - {file = "watchdog-6.0.0-py3-none-win_amd64.whl", hash = "sha256:cbafb470cf848d93b5d013e2ecb245d4aa1c8fd0504e863ccefa32445359d680"}, - {file = "watchdog-6.0.0-py3-none-win_ia64.whl", hash = "sha256:a1914259fa9e1454315171103c6a30961236f508b9b623eae470268bbcc6a22f"}, - {file = "watchdog-6.0.0.tar.gz", hash = "sha256:9ddf7c82fda3ae8e24decda1338ede66e1c99883db93711d8fb941eaa2d8c282"}, -] - -[package.extras] -watchmedo = ["PyYAML (>=3.10)"] - [[package]] name = "wcwidth" version = "0.6.0" @@ -6856,6 +7500,109 @@ docs = ["Sphinx (>=6.0)", "myst-parser (>=2.0.0)", "sphinx_rtd_theme (>=1.1.0)"] optional = ["python-socks", "wsaccel"] test = ["pytest", "websockets"] +[[package]] +name = "wrapt" +version = "2.2.1" +description = "Module for decorators, wrappers and monkey patching." +optional = false +python-versions = ">=3.9" +groups = ["dev"] +files = [ + {file = "wrapt-2.2.1-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:0f68f478004475d97906686e702ddbddeaf717c0b68ad2794384308f2dc713ae"}, + {file = "wrapt-2.2.1-cp310-cp310-macosx_11_0_arm64.whl", hash = "sha256:e422b2d647a65d6b080cad5accd09055d3809bdff00c76fba8dca00ca935572a"}, + {file = "wrapt-2.2.1-cp310-cp310-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:036dfb40128819a751c6f451c6b9c10172c49e4c401aebcdb8ecf2aec1683598"}, + {file = "wrapt-2.2.1-cp310-cp310-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:09ac16c081bebfd15d8e4dfa5bdc805990bbd52249ecff22530da7a129d6120b"}, + {file = "wrapt-2.2.1-cp310-cp310-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:07be671fa8875971222b0ba9059ed8b4dc738631122feba17c93aa36b4213e9a"}, + {file = "wrapt-2.2.1-cp310-cp310-musllinux_1_2_aarch64.whl", hash = "sha256:93fc2bf40cd7f4a0256010dce073d44eeb4a351b9bca94d0477ce2b6e62532b3"}, + {file = "wrapt-2.2.1-cp310-cp310-musllinux_1_2_riscv64.whl", hash = "sha256:ba519b2d765df9871a25879e6f7fa78948ea59a2a31f9c1a257e34b651994afc"}, + {file = "wrapt-2.2.1-cp310-cp310-musllinux_1_2_x86_64.whl", hash = "sha256:9011395be8db1827d106c6449b4bb6dd17e331ff6ec521f227e4588f1c78e46f"}, + {file = "wrapt-2.2.1-cp310-cp310-win32.whl", hash = "sha256:a8f7176b83664af44567e9cc06e0d3827823fcc1a5e52307ebb8ac3aa95860b9"}, + {file = "wrapt-2.2.1-cp310-cp310-win_amd64.whl", hash = "sha256:d7f513d3185e6fec82d0c3518f2e6365d8b4e49f5f45f29640d5162d56a23b54"}, + {file = "wrapt-2.2.1-cp310-cp310-win_arm64.whl", hash = "sha256:44255c84bc57554fed822e83e70036b51afa9edb56fc7ca56c54410ece7898c9"}, + {file = "wrapt-2.2.1-cp311-cp311-macosx_10_9_x86_64.whl", hash = "sha256:dd57607acc85678925940bd5df0385ff8332083a32fa8d7a43f8767f4997263c"}, + {file = "wrapt-2.2.1-cp311-cp311-macosx_11_0_arm64.whl", hash = "sha256:1ae574d65c9fa8e86f64f6a7c2668f9fcd507b183e0e577619f504b883cb0a6c"}, + {file = "wrapt-2.2.1-cp311-cp311-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:9a04c28c10ba7fd12842b109d2edb0678872a2fe65277ca4ff06a0d61edee245"}, + {file = "wrapt-2.2.1-cp311-cp311-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:3e2f02472a1cbbf3884b365714a810b5947134a95ad6952b554cb8cce9d492b0"}, + {file = "wrapt-2.2.1-cp311-cp311-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:ac2745950b2bff80219c15ebf2fa9d8427eba7e249739f97e55c9d169e47e9e1"}, + {file = "wrapt-2.2.1-cp311-cp311-musllinux_1_2_aarch64.whl", hash = "sha256:67a97e5b6c457f0cd3cfc19ebb2d84463e60c3ece754cc831e4281a3ca29bb18"}, + {file = "wrapt-2.2.1-cp311-cp311-musllinux_1_2_riscv64.whl", hash = "sha256:c803a3d331796255af51ba2c79ed0ac8275865b516c09e61f248d1e7aff31ce9"}, + {file = "wrapt-2.2.1-cp311-cp311-musllinux_1_2_x86_64.whl", hash = "sha256:9b984d1eb252145d6302c1dbd5e87fc6d404d45531447c84eadec04bf1fcb027"}, + {file = "wrapt-2.2.1-cp311-cp311-win32.whl", hash = "sha256:8a983a603a18c8708f024f7f6991b2e66159219abbf894634c5056243c55f3cd"}, + {file = "wrapt-2.2.1-cp311-cp311-win_amd64.whl", hash = "sha256:9c210a6994b21aa9b29e81c8d11560e8fdab54c117e9cff37870d0a27bde1343"}, + {file = "wrapt-2.2.1-cp311-cp311-win_arm64.whl", hash = "sha256:401229e9d63ca09f9b8891ecf83798d26c11bbb445d11ed9f1836b6d4585b38a"}, + {file = "wrapt-2.2.1-cp312-cp312-macosx_10_13_x86_64.whl", hash = "sha256:3ffad790d9d11d8ecf9f17c4bb671a5b4089e4d8b575c46c5129597f41f836b0"}, + {file = "wrapt-2.2.1-cp312-cp312-macosx_11_0_arm64.whl", hash = "sha256:628f5220c7a904d5fc78f7075c8d7871433eb6d035c94728a22fdf85f193d2a8"}, + {file = "wrapt-2.2.1-cp312-cp312-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:61acce4257a9883669703c525447c5b4c392edf0f987ae77ec32668440158f0e"}, + {file = "wrapt-2.2.1-cp312-cp312-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:727ab4244622cd6ad2390f322642090c877d2e83a608d2653a7643ae5368d926"}, + {file = "wrapt-2.2.1-cp312-cp312-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:03df9ebed4c73ab93fa8c07e3d41d818dfca1852b15731a3de59457b27814624"}, + {file = "wrapt-2.2.1-cp312-cp312-musllinux_1_2_aarch64.whl", hash = "sha256:0d9ff006f420b2ec8296aa56ade43ea7da3e997e85769f0aafc5e0661aacb710"}, + {file = "wrapt-2.2.1-cp312-cp312-musllinux_1_2_riscv64.whl", hash = "sha256:844c858fc3bb7eacc0ba8efa904935d16aac6a4470948ad1e7e55c9f5a2a665f"}, + {file = "wrapt-2.2.1-cp312-cp312-musllinux_1_2_x86_64.whl", hash = "sha256:87bacdaf225117a342a20d9c03438d701c02112f6e3f351ce9b7f32354f14797"}, + {file = "wrapt-2.2.1-cp312-cp312-win32.whl", hash = "sha256:2f8c90c8afde51969487be4e1343ae049b268854877d415c2510baf833775052"}, + {file = "wrapt-2.2.1-cp312-cp312-win_amd64.whl", hash = "sha256:6ce32763ac31ce94fe9aada947e479b1975012bff166da409b4b9e4e376cf7e5"}, + {file = "wrapt-2.2.1-cp312-cp312-win_arm64.whl", hash = "sha256:8d1b4d0e0c2119587a31f5c029abd547e0c81d93b89d394566fe1588659eb579"}, + {file = "wrapt-2.2.1-cp313-cp313-macosx_10_13_x86_64.whl", hash = "sha256:d2beb1c7cab10603aecdc42f8edd6ff013f9a32e4543474e38e6b77ce9975aeb"}, + {file = "wrapt-2.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:e0cb7e4dd71f4c32e5e84843cd3c4cd65dda034314004bbe1d7f99af2426ab80"}, + {file = "wrapt-2.2.1-cp313-cp313-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:95821352042722cd9f1108874579a47989d0a7e12a37d87d2fc4af20fd99ab8a"}, + {file = "wrapt-2.2.1-cp313-cp313-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:abd621552ede77c4c69be7fac44ba911225b0c812b6ba604e5964cf98085b474"}, + {file = "wrapt-2.2.1-cp313-cp313-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:e3677c7146ce694874941ba82b57092cc4875445aadf29d72807351023105143"}, + {file = "wrapt-2.2.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:9a5934eaea872e17936b5f45501eba5ab0bce9a74122e172b663d7c28c459c4a"}, + {file = "wrapt-2.2.1-cp313-cp313-musllinux_1_2_riscv64.whl", hash = "sha256:f5b9daf6b629fce418e0cc3dd0436eac045188fa35deadb7a7f3941d5b8203f9"}, + {file = "wrapt-2.2.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:f53ac9f3ef573326d009ed809beff4efcac6451931c2b8132586da4b9e53ff31"}, + {file = "wrapt-2.2.1-cp313-cp313-win32.whl", hash = "sha256:1ffa9cfd4bdb581539951b14ae661ff20ed0c3599b3e911a131ee0ec5ac11337"}, + {file = "wrapt-2.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:368eac1e20fd0bb03dd3cc42bf9887154c3861b60989389ccb5fac032617d215"}, + {file = "wrapt-2.2.1-cp313-cp313-win_arm64.whl", hash = "sha256:c754dafdf5aaf0b401b644a90a30046929a0dd1a536e0ff0ec959a59155d9c7f"}, + {file = "wrapt-2.2.1-cp313-cp313t-macosx_10_13_x86_64.whl", hash = "sha256:ed928d0fda15fc0adc8d13305c8b3c0f2fba5b0669950c9e6d019d9162a3b3e8"}, + {file = "wrapt-2.2.1-cp313-cp313t-macosx_11_0_arm64.whl", hash = "sha256:fafb4e739e43544d12cb4abd1605fd4683b6ca6a9ad682b7fd8f4d21973eafa8"}, + {file = "wrapt-2.2.1-cp313-cp313t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:74d6a0c31472fe5d814917266b9f46495d7c61ed890af08b468acea92fb89a8d"}, + {file = "wrapt-2.2.1-cp313-cp313t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:ab5be648d5a0b86b7438864f8df3c705a65cef35a2fd3e5561e3e203167e0f27"}, + {file = "wrapt-2.2.1-cp313-cp313t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:9d8f204c8e3a8bf9ece17e0a83d137fd807440977f8a5e762d59306795011440"}, + {file = "wrapt-2.2.1-cp313-cp313t-musllinux_1_2_aarch64.whl", hash = "sha256:d047f6498c973874ba08ac3f97c69a2c4b2211c8de6f4c205f75cb1c9522596e"}, + {file = "wrapt-2.2.1-cp313-cp313t-musllinux_1_2_riscv64.whl", hash = "sha256:7a4fdb9326aab4a5a477a1640e5ad786a8495901009d7e7b038371edd23a9d2b"}, + {file = "wrapt-2.2.1-cp313-cp313t-musllinux_1_2_x86_64.whl", hash = "sha256:c8cc5094b08abeae52da9c73c8a32003623be691a5193df2f4e3eac3d557c394"}, + {file = "wrapt-2.2.1-cp313-cp313t-win32.whl", hash = "sha256:9907a4402ab6db12b7077a0ea5d7a4d028ecb22c8eee2b53527080d347cd1562"}, + {file = "wrapt-2.2.1-cp313-cp313t-win_amd64.whl", hash = "sha256:5590d63f5243251641cf543009b4c9314a79d0598fdb8a8e4cfc918494536c53"}, + {file = "wrapt-2.2.1-cp313-cp313t-win_arm64.whl", hash = "sha256:c318a64b53d97b841d7b5e637517e50a27be64bc695128422953d4b21710954e"}, + {file = "wrapt-2.2.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:6f56a647e4eaf5f0ca40330fb070f566bdf9f7b0db89a1af20d71c28dcd7a0ab"}, + {file = "wrapt-2.2.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:64b7deeda4b70408e382328d8bbe52a256fe9bc63ae3db86d804608367e5422c"}, + {file = "wrapt-2.2.1-cp314-cp314-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:b9cf53ba90717db2e292401de290776c498d4bbfb0d4a559ca2895db8b9dcb5c"}, + {file = "wrapt-2.2.1-cp314-cp314-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:cf3638274ab9d9b724c9baa0b4c04e132cd6faefb78b4dd3dd1a02a4bdaad41e"}, + {file = "wrapt-2.2.1-cp314-cp314-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:aed9658797d0b45d6c49adcfc6b41f66e6f2d0c6de3ec79e16cf4b1855df240f"}, + {file = "wrapt-2.2.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:1d676ee388bc42a04d56dd7deb5605244dac2e35cc2fadbb43c9fa25bbd93508"}, + {file = "wrapt-2.2.1-cp314-cp314-musllinux_1_2_riscv64.whl", hash = "sha256:e395f7bc31851ef9b612050368cb446e9bc14cd7454b025018980349caf25ae5"}, + {file = "wrapt-2.2.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:5f1845c2a8cc1180ccccfa45785dd06f562730d19ef75be180334254012b6283"}, + {file = "wrapt-2.2.1-cp314-cp314-win32.whl", hash = "sha256:436addbc4bb4fc0a88c702577f51195d7d73683a7f3e0e5b253d8404d7847243"}, + {file = "wrapt-2.2.1-cp314-cp314-win_amd64.whl", hash = "sha256:50972a1d974ea07725a7f6b1cec5f8759008afd030a0024843ebe7d52de47f2b"}, + {file = "wrapt-2.2.1-cp314-cp314-win_arm64.whl", hash = "sha256:1c9934ea5d92957e3cd0adbc0845539dccfd62710ebe16195a8c66c53954db36"}, + {file = "wrapt-2.2.1-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:17de18fc12cea55b8a9587314cb830573e37fb33b247a7515696350863714188"}, + {file = "wrapt-2.2.1-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:a9dec1aca52dddde7df94818310fa2fe79739c8f385b2014c4cb1035f5508199"}, + {file = "wrapt-2.2.1-cp314-cp314t-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:69f2e9244542cb34dd59c7f073445b9e54ad9f3fce8d93606c368a1b499fc413"}, + {file = "wrapt-2.2.1-cp314-cp314t-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:2d83966dc7f4f45e8b97b5933685ac2e6e67fc0e19246ea314bceb9a8970c956"}, + {file = "wrapt-2.2.1-cp314-cp314t-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:78b0aa6bfb7be8deed0ab23e7aa028cc5210c29bc2d32a04d52b50e517a7307e"}, + {file = "wrapt-2.2.1-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:05d5cb74d1b232ec8cfa130a8f900708699ff2491d97b8f85a4cdc5996294b85"}, + {file = "wrapt-2.2.1-cp314-cp314t-musllinux_1_2_riscv64.whl", hash = "sha256:f6518b94edb9150452e9aba08027d4cc293433753ec1fbefb4629a21cbc74181"}, + {file = "wrapt-2.2.1-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:ed55af48b3eb28f43228ca2306788892bcb629eb2b5c4876e2a3659872c2f17a"}, + {file = "wrapt-2.2.1-cp314-cp314t-win32.whl", hash = "sha256:2e08688ab16525897da6589d56d0aebaf417bbe91c2d8e3b96203b1efa596e85"}, + {file = "wrapt-2.2.1-cp314-cp314t-win_amd64.whl", hash = "sha256:fd0135d34387f5fd087d9be368ea77ea89cf2451dc1cd1c622d35021bcb3ab50"}, + {file = "wrapt-2.2.1-cp314-cp314t-win_arm64.whl", hash = "sha256:f70db64e8266d7c45d3b735f2e08eeb434b5e03da9a479ae42b2e2e486a21a00"}, + {file = "wrapt-2.2.1-cp39-cp39-macosx_10_9_x86_64.whl", hash = "sha256:5fa9bf3b9e66336589d03f42abce2da1055ad5c69b0c2b764852a8471c9b9114"}, + {file = "wrapt-2.2.1-cp39-cp39-macosx_11_0_arm64.whl", hash = "sha256:2076d2335085eb09b9547e7688656fa8f5cf0183eab589d33499cd353489d797"}, + {file = "wrapt-2.2.1-cp39-cp39-manylinux1_x86_64.manylinux_2_28_x86_64.manylinux_2_5_x86_64.whl", hash = "sha256:7975bc88ab4b0f72ef2a2d5ae9d77d87efb5ef95e8f8046242fa9afdaaf2030b"}, + {file = "wrapt-2.2.1-cp39-cp39-manylinux2014_aarch64.manylinux_2_17_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:61a0013344674d2b648bc6e6fe9828dd4fc1d3b4eb7523809792f8cb952e2f16"}, + {file = "wrapt-2.2.1-cp39-cp39-manylinux_2_31_riscv64.manylinux_2_39_riscv64.whl", hash = "sha256:b6c0febfe38f22df2eb565c0ce8a092bb80411e56861ca382c443da83105423f"}, + {file = "wrapt-2.2.1-cp39-cp39-musllinux_1_2_aarch64.whl", hash = "sha256:211f595f8e7faae5c5930fcc64708f2ba36849e0ba0fd653a843de9fa8d7db77"}, + {file = "wrapt-2.2.1-cp39-cp39-musllinux_1_2_riscv64.whl", hash = "sha256:f4e1a92032a39cd5e3c647ca57dbf33b6a1938fd975623175793f9dbb63236de"}, + {file = "wrapt-2.2.1-cp39-cp39-musllinux_1_2_x86_64.whl", hash = "sha256:24c52546acf2ab82412f2ab6fc5948a7fe958d3b4f070202e8dcdd865489eaf9"}, + {file = "wrapt-2.2.1-cp39-cp39-win32.whl", hash = "sha256:c3723ff8eb8721f4daac98bc0256f15158e05316d5e52648ce9cebee434fbdd5"}, + {file = "wrapt-2.2.1-cp39-cp39-win_amd64.whl", hash = "sha256:2de9e20769fe9c1f6dcdc893c6a89287c5ccf8537c90b5de78aed8017697aad5"}, + {file = "wrapt-2.2.1-cp39-cp39-win_arm64.whl", hash = "sha256:585916e210db57b23543342c2f298e42331b617fd0c934caf5c64df44de8640e"}, + {file = "wrapt-2.2.1-py3-none-any.whl", hash = "sha256:3aafea2975caef8ca49400640dde02cc7426e798f24870ed01f490bc3cffd32f"}, + {file = "wrapt-2.2.1.tar.gz", hash = "sha256:6744f504375775d7609c82c8d3d94af1c9a6f05586984536905908ba905277b9"}, +] + +[package.extras] +dev = ["pytest", "setuptools"] + [[package]] name = "yarl" version = "1.23.0" @@ -6863,7 +7610,7 @@ description = "Yet another URL library" optional = true python-versions = ">=3.10" groups = ["main"] -markers = "extra == \"embedding-search\" or extra == \"all\"" +markers = "extra == \"advanced-search\" or extra == \"embedding-search\" or extra == \"all\"" files = [ {file = "yarl-1.23.0-cp310-cp310-macosx_10_9_universal2.whl", hash = "sha256:cff6d44cb13d39db2663a22b22305d10855efa0fa8015ddeacc40bc59b9d8107"}, {file = "yarl-1.23.0-cp310-cp310-macosx_10_9_x86_64.whl", hash = "sha256:e4c53f8347cd4200f0d70a48ad059cabaf24f5adc6ba08622a23423bc7efa10d"}, @@ -7007,7 +7754,7 @@ description = "Backport of pathlib-compatible object wrapper for zip files" optional = true python-versions = ">=3.9" groups = ["main"] -markers = "extra == \"dbt\" or extra == \"all\" or extra == \"embedding-search\"" +markers = "extra == \"dbt\" or extra == \"all\" or extra == \"advanced-search\" or extra == \"embedding-search\"" files = [ {file = "zipp-3.23.1-py3-none-any.whl", hash = "sha256:0b3596c50a5c700c9cb40ba8d86d9f2cc4807e9bedb06bcdf7fac85633e444dc"}, {file = "zipp-3.23.1.tar.gz", hash = "sha256:32120e378d32cd9714ad503c1d024619063ec28aad2248dc6672ad13edfa5110"}, @@ -7197,18 +7944,20 @@ files = [ ] [extras] -all = ["aiomysql", "asyncpg", "clickhouse-sqlalchemy", "dbt-core", "httpx", "litellm", "numpy", "pandas", "psycopg2-binary", "pyarrow", "pymysql"] +advanced-search = ["ladybug", "litellm", "numpy"] +all = ["aiomysql", "asyncpg", "clickhouse-sqlalchemy", "dbt-core", "httpx", "ladybug", "litellm", "numpy", "pandas", "psycopg2-binary", "pyarrow", "pymysql", "pyodbc", "snowflake-connector-python", "snowflake-sqlalchemy", "sqlalchemy-bigquery"] +bigquery = ["sqlalchemy-bigquery"] clickhouse = ["clickhouse-sqlalchemy"] client = ["httpx", "pandas"] dbt = ["dbt-core"] -docs = ["mkdocs-material"] embedding-search = ["litellm", "numpy"] flight = ["pyarrow"] mysql = ["aiomysql", "pymysql"] -pg-facade = [] postgres = ["asyncpg", "psycopg2-binary"] +snowflake = ["snowflake-connector-python", "snowflake-sqlalchemy"] +sqlserver = ["pyodbc"] [metadata] lock-version = "2.1" python-versions = "^3.11" -content-hash = "312de7d50d47c0b4be641d5bda4469965a546be00ff454ea1b262a4e9bebde76" +content-hash = "b4fe34cc49941759e67dfadecc27663a9f4467d0322b06a0a797f8ce5d1703cd" diff --git a/pyproject.toml b/pyproject.toml index cdf737ff..844f2cf1 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -1,6 +1,6 @@ [project] name = "motley-slayer" -version = "0.6.10" +version = "0.9.12" description = "A lightweight, agent-first semantic layer for AI agents" requires-python = ">=3.11" license = "MIT" @@ -24,7 +24,7 @@ classifiers = [ [tool.poetry.urls] Repository = "https://github.com/MotleyAI/slayer" -Documentation = "https://motley-slayer.readthedocs.io/" +Documentation = "https://docs.motley.ai/slayer/" Discord = "https://discord.gg/egWxMctHCA" [tool.poetry.dependencies] @@ -39,12 +39,13 @@ uvicorn = ">=0.20" httpx = {version = ">=0.24", optional = true} pandas = {version = ">=2.0", optional = true} mcp = ">=1.0" -mkdocs-material = {version = ">=9.0", optional = true} psycopg2-binary = {version = ">=2.9", optional = true} asyncpg = {version = ">=0.27", optional = true} pymysql = {version = ">=1.0", optional = true} aiomysql = {version = ">=0.2", optional = true} clickhouse-sqlalchemy = {version = ">=0.3", optional = true} +pyodbc = {version = ">=5.0", optional = true} +sqlalchemy-bigquery = {version = ">=1.11", optional = true, python = "<3.15"} # DuckDB + jafgen are core deps so the bundled `slayer datasources create demo` # works after a single `pip install motley-slayer`. duckdb = ">=0.9" @@ -63,20 +64,28 @@ tantivy = "^0.26" # gracefully to tantivy + BM25 when this extra is not installed. litellm = {version = ">=1.50", optional = true} numpy = {version = ">=1.24", optional = true} +# LadybugDB is the active successor to KuzuDB (same codebase, new name after +# the original KuzuDB repo was archived post-acquisition). Import name is +# ``ladybug``. +ladybug = {version = ">=0.1", optional = true} +# Snowflake support (DEV-1551). snowflake-sqlalchemy >= 1.7 is the first +# release with SQLAlchemy 2.x support (a hard repo dep). +snowflake-connector-python = {version = ">=3.13", optional = true} +snowflake-sqlalchemy = {version = ">=1.7", optional = true} [tool.poetry.extras] client = ["httpx", "pandas"] postgres = ["psycopg2-binary", "asyncpg"] mysql = ["pymysql", "aiomysql"] clickhouse = ["clickhouse-sqlalchemy"] +sqlserver = ["pyodbc"] +snowflake = ["snowflake-connector-python", "snowflake-sqlalchemy"] +bigquery = ["sqlalchemy-bigquery"] dbt = ["dbt-core"] -docs = ["mkdocs-material"] flight = ["pyarrow"] -# The Postgres wire-protocol facade is pure-stdlib; the empty extra keeps the -# `pip install motley-slayer[pg_facade]` install path consistent. -pg_facade = [] -embedding_search = ["litellm", "numpy"] -all = ["httpx", "pandas", "psycopg2-binary", "asyncpg", "pymysql", "aiomysql", "clickhouse-sqlalchemy", "dbt-core", "pyarrow", "litellm", "numpy"] +advanced_search = ["litellm", "numpy", "ladybug"] +embedding_search = ["litellm", "numpy"] # legacy alias for advanced_search without graph support +all = ["httpx", "pandas", "psycopg2-binary", "asyncpg", "pymysql", "aiomysql", "clickhouse-sqlalchemy", "pyodbc", "snowflake-connector-python", "snowflake-sqlalchemy", "sqlalchemy-bigquery", "dbt-core", "pyarrow", "litellm", "numpy", "ladybug"] [project.scripts] slayer = "slayer.cli:main" @@ -103,12 +112,18 @@ pyinstrument = "^5.1.2" # wire path a real BI tool would. Java >= 11 must be on PATH. jaydebeapi = "^1.2.3" jpype1 = "^1.5.0" +# DEV-1564: dialect integration suites spin up real DBs via testcontainers. +# The MySQL/ClickHouse/SQL Server suites import `testcontainers.` +# and skip cleanly if it's missing; CI workflows assert the extras are +# importable before invoking pytest so a missing dep surfaces as failure. +testcontainers = {extras = ["mysql", "clickhouse", "mssql"], version = "^4.0"} [tool.pytest.ini_options] testpaths = ["tests"] addopts = "--ignore=tests/perf" markers = [ "integration: marks tests as integration tests (require a real database)", + "metabase_e2e: live-Metabase end-to-end suite (requires Docker)", ] asyncio_mode = "auto" diff --git a/slayer/api/server.py b/slayer/api/server.py index ab544d9e..57769b8e 100644 --- a/slayer/api/server.py +++ b/slayer/api/server.py @@ -2,10 +2,10 @@ import logging from importlib.metadata import PackageNotFoundError, version as _pkg_version -from typing import Any, Dict, List, Optional, Union +from typing import Any from fastapi import FastAPI, HTTPException -from pydantic import BaseModel, ConfigDict +from pydantic import BaseModel, ConfigDict, Field from slayer.mcp.server import create_mcp_server from slayer.core.errors import ( @@ -13,11 +13,14 @@ EntityResolutionError, MemoryNotFoundError, SchemaDriftError, + SlayerError, ) from slayer.core.format import NumberFormat from slayer.core.models import DatasourceConfig, SlayerModel from slayer.core.query import SlayerQuery from slayer.engine.query_engine import SlayerQueryEngine +from slayer.inspect.service import InspectService +from slayer.memories.help_seed import seed_help_memories from slayer.memories.service import MemoryService from slayer.search.service import SearchService from slayer.storage.base import StorageBackend @@ -30,30 +33,34 @@ class QueryRequest(BaseModel): # to pass through to SlayerQuery's pre-validate hook. model_config = ConfigDict(extra="allow") - name: Optional[str] = None # Run-by-name: backing query for a query-backed model + name: str | None = None # Run-by-name: backing query for a query-backed model # ``source_model`` accepts a string (stored model name) or a dict # — the dict form is an inline ``ModelExtension`` (``{"source_name": # "", "columns": [...], "joins": [...]}``) or an inline # ``SlayerModel`` (``{"name": "...", "sql_table": "...", "data_source": # "...", "columns": [...]}``). The full polymorphism is handled by # ``SlayerQuery.model_validate`` downstream. - source_model: Optional[Union[str, Dict[str, Any]]] = None + source_model: str | dict[str, Any] | None = None # ``measures`` and ``dimensions`` accept bare strings as a shorthand, # mirroring the Python API: ``"*:count"`` is lifted to # ``{"formula": "*:count"}``, ``"status"`` to ``{"name": "status"}``. # ``SlayerQuery``'s before-validators (``_coerce_measures`` / # ``_coerce_dimensions``) do the actual lifting downstream. - measures: Optional[List[Union[str, Dict[str, Any]]]] = None - dimensions: Optional[List[Union[str, Dict[str, Any]]]] = None - time_dimensions: Optional[List[Dict[str, Any]]] = None - filters: Optional[List[str]] = None - order: Optional[List[Dict[str, Any]]] = None - limit: Optional[int] = None - offset: Optional[int] = None - whole_periods_only: Optional[bool] = None - dry_run: Optional[bool] = None - explain: Optional[bool] = None - variables: Optional[Dict[str, Any]] = None + measures: list[str | dict[str, Any]] | None = None + dimensions: list[str | dict[str, Any]] | None = None + time_dimensions: list[dict[str, Any]] | None = None + filters: list[str] | None = None + order: list[dict[str, Any]] | None = None + limit: int | None = None + offset: int | None = None + whole_periods_only: bool | None = None + # DEV-1543: opt out of the dim-only auto-dedup GROUP BY. Default + # (``None`` here) keeps the v3 SlayerQuery default (``True``). Set + # ``False`` to emit raw rows. + distinct_dimension_values: bool | None = None + dry_run: bool | None = None + explain: bool | None = None + variables: dict[str, Any] | None = None class QueryListRequest(BaseModel): @@ -68,39 +75,46 @@ class QueryListRequest(BaseModel): model_config = ConfigDict(extra="forbid") - queries: List[Dict[str, Any]] - variables: Optional[Dict[str, Any]] = None - dry_run: Optional[bool] = None - explain: Optional[bool] = None + queries: list[dict[str, Any]] + variables: dict[str, Any] | None = None + dry_run: bool | None = None + explain: bool | None = None class FieldMetadataResponse(BaseModel): - label: Optional[str] = None - format: Optional[NumberFormat] = None + label: str | None = None + format: NumberFormat | None = None class AttributesResponse(BaseModel): - dimensions: Dict[str, FieldMetadataResponse] = {} - measures: Dict[str, FieldMetadataResponse] = {} + dimensions: dict[str, FieldMetadataResponse] = {} + measures: dict[str, FieldMetadataResponse] = {} class QueryResponse(BaseModel): - data: List[Dict[str, Any]] + data: list[dict[str, Any]] row_count: int - columns: List[str] - sql: Optional[str] = None - attributes: Optional[AttributesResponse] = None + columns: list[str] + sql: str | None = None + attributes: AttributesResponse | None = None class IngestRequest(BaseModel): datasource: str - include_tables: Optional[List[str]] = None - exclude_tables: Optional[List[str]] = None - schema_name: Optional[str] = None + include_tables: list[str] | None = None + exclude_tables: list[str] | None = None + schema_name: str | None = None class ValidateModelsRequest(BaseModel): - data_source: Optional[str] = None + data_source: str | None = None + + +class RecommendRootModelRequest(BaseModel): + """Body for ``POST /recommend-root-model``.""" + items: list[str] + data_source: str | None = None + root_hint: str | None = None class DatasourcePriorityRequest(BaseModel): @@ -109,7 +123,7 @@ class DatasourcePriorityRequest(BaseModel): shape and FastAPI rejects mistyped payloads with 422 instead of silently coercing them downstream. """ - priority: List[str] = [] + priority: list[str] = [] class SaveMemoryRequest(BaseModel): @@ -126,30 +140,62 @@ class SaveMemoryRequest(BaseModel): (e.g. for knowledge-base ingestion that wants stable string ids like ``kb.policy.42``). Bad charset → 400. Omit → auto-allocated int-shaped id. + + DEV-1549: optional ``description`` is a short compact preview + surfaced by ``search(compact=True)`` and ``inspect_model``. Hard + cap of 500 chars; over-cap returns HTTP 400. """ learning: str linked_entities: Any - id: Optional[str] = None + id: str | None = None + description: str | None = None class SearchRequest(BaseModel): """Body for ``POST /search`` (DEV-1375). Mirrors the MCP / CLI / SlayerClient surfaces. - All three retrieval inputs are optional. Empty input falls back to - a recency listing of the newest ``max_memories`` learning-only - memories plus the newest ``max_example_queries`` query-bearing - memories. + All retrieval inputs are optional. Empty input falls back to a + recency listing capped at ``max_results`` hits. + + DEV-1549: ``compact`` defaults to ``True`` everywhere. Compact + memory hits surface ``description`` (with a first-paragraph + fallback from ``learning``) and empty ``text``; compact entity + hits surface ``entity.description`` and empty ``text``. Set + ``compact=False`` to restore the verbose pre-0.7.3 shape. """ - entities: Optional[List[str]] = None - query: Optional[Any] = None - question: Optional[str] = None - datasource: Optional[str] = None - max_memories: int = 5 - max_example_queries: int = 2 - max_entities: int = 5 + model_config = ConfigDict(extra="forbid") + + entities: list[str] | None = None + query: Any | None = None + question: str | None = None + datasource: str | None = None + max_results: int = Field(default=10, ge=1) + cypher_filter: str | None = None + compact: bool = True + + +class InspectRequest(BaseModel): + """Body for ``POST /inspect`` (DEV-1588). Mirrors the MCP / CLI / + SlayerClient ``inspect`` surfaces — a point-lookup of one entity, or a + homogeneous-kind batch when ``reference`` is a list (DEV-1612).""" + + model_config = ConfigDict(extra="forbid") + + # DEV-1612: a list is a homogeneous-kind batch (one ``entity_type`` for + # every id). A single str keeps single-id behaviour byte-for-byte. + # DEV-1667: ``None`` / omitted (or ``[]``) renders the whole collection at + # ``entity_type`` (model / datasource only). + reference: str | list[str] | None = None + entity_type: str + compact: bool = True + format: str = "markdown" + num_rows: int = 3 + show_sql: bool = False + sections: list[str] | None = None + descriptions_max_chars: int | None = None def _slayer_version() -> str: @@ -164,10 +210,16 @@ def create_app( # NOSONAR(S3776) — FastAPI route-handler factory; complexity *, ingest_on_startup: bool = False, ) -> FastAPI: + from slayer.async_utils import run_sync + + # DEV-1658: seed conceptual-help memories once here; the embedded MCP + # server below is created with _seed_help=False so the pass never fires + # twice (mirrors the ingest_on_startup single-orchestration rule). + run_sync(seed_help_memories(storage=storage)) + if ingest_on_startup: import sys - from slayer.async_utils import run_sync from slayer.engine.ingestion import ingest_all_datasources_idempotent run_sync( @@ -180,12 +232,12 @@ def create_app( # NOSONAR(S3776) — FastAPI route-handler factory; complexity # does NOT receive `ingest_on_startup` — orchestration happens once, # above, so calling `create_app(ingest_on_startup=True)` doesn't fire # the orchestrator twice. - mcp = create_mcp_server(storage=storage) + mcp = create_mcp_server(storage=storage, _seed_help=False) mcp_app = mcp.sse_app() app.mount("/mcp", mcp_app) @app.get("/health") - async def health() -> Dict[str, str]: + async def health() -> dict[str, str]: return {"status": "ok"} @app.post( @@ -204,7 +256,7 @@ async def health() -> Dict[str, str]: }, ) async def query( - request: Union[QueryRequest, QueryListRequest], + request: QueryRequest | QueryListRequest, ) -> QueryResponse: try: # Multi-stage DAG: body is ``{"queries": [...], "variables": ..., @@ -235,6 +287,7 @@ async def query( request.source_model, request.measures, request.dimensions, request.time_dimensions, request.filters, request.order, request.limit, request.offset, request.whole_periods_only, + request.distinct_dimension_values, ) if f is not None ] if disallowed: @@ -279,7 +332,7 @@ async def query( ) attrs = result.attributes - def _convert_meta(d: dict) -> Dict[str, FieldMetadataResponse]: + def _convert_meta(d: dict) -> dict[str, FieldMetadataResponse]: return {k: FieldMetadataResponse(label=v.label, format=v.format) for k, v in d.items()} attributes = None @@ -314,8 +367,8 @@ def _convert_meta(d: dict) -> Dict[str, FieldMetadataResponse]: @app.get("/models") async def list_models( - data_source: Optional[str] = None, - ) -> List[Dict[str, Any]]: + data_source: str | None = None, + ) -> list[dict[str, Any]]: identities = await storage._list_all_model_identities() result = [] for ds_name, name in identities: @@ -324,7 +377,7 @@ async def list_models( model = await storage.get_model(name, data_source=ds_name) if model is None or model.hidden: continue - entry: Dict[str, Any] = {"name": name, "data_source": ds_name} + entry: dict[str, Any] = {"name": name, "data_source": ds_name} if model.description: entry["description"] = model.description result.append(entry) @@ -344,8 +397,8 @@ async def list_models( ) async def get_model( name: str, - data_source: Optional[str] = None, - ) -> Dict[str, Any]: + data_source: str | None = None, + ) -> dict[str, Any]: try: model = await storage.get_model(name, data_source=data_source) except AmbiguousModelError as exc: @@ -367,7 +420,7 @@ async def get_model( "/models", responses={400: {"description": "Model failed validation (e.g. user-supplied cache fields on a query-backed model, save-time SQL generation failure)."}}, ) - async def create_model(model: SlayerModel) -> Dict[str, str]: + async def create_model(model: SlayerModel) -> dict[str, str]: # Route through engine.save_model so query-backed models get cache # populated (and user-supplied cache fields are rejected). try: @@ -380,7 +433,7 @@ async def create_model(model: SlayerModel) -> Dict[str, str]: "/models/{name}", responses={400: {"description": "Body name does not match path name, or model failed validation."}}, ) - async def update_model(name: str, model: SlayerModel) -> Dict[str, str]: + async def update_model(name: str, model: SlayerModel) -> dict[str, str]: if model.name != name: raise HTTPException( status_code=400, @@ -406,8 +459,8 @@ async def update_model(name: str, model: SlayerModel) -> Dict[str, str]: ) async def delete_model( name: str, - data_source: Optional[str] = None, - ) -> Dict[str, Any]: + data_source: str | None = None, + ) -> dict[str, Any]: try: deleted = await storage.delete_model(name, data_source=data_source) except AmbiguousModelError as exc: @@ -423,7 +476,7 @@ async def delete_model( return {"status": "deleted", "name": name} @app.get("/datasources/priority") - async def get_datasource_priority() -> Dict[str, List[str]]: + async def get_datasource_priority() -> dict[str, list[str]]: return {"priority": await storage.get_datasource_priority()} @app.put( @@ -437,7 +490,7 @@ async def get_datasource_priority() -> Dict[str, List[str]]: } }, ) - async def put_datasource_priority(body: DatasourcePriorityRequest) -> Dict[str, Any]: + async def put_datasource_priority(body: DatasourcePriorityRequest) -> dict[str, Any]: priority = list(body.priority) try: await storage.set_datasource_priority(priority) @@ -446,18 +499,18 @@ async def put_datasource_priority(body: DatasourcePriorityRequest) -> Dict[str, return {"status": "ok", "priority": priority} @app.get("/datasources") - async def list_datasources() -> List[Dict[str, Any]]: + async def list_datasources() -> list[dict[str, Any]]: result = [] for name in await storage.list_datasources(): ds = await storage.get_datasource(name) - entry: Dict[str, Any] = {"name": name} + entry: dict[str, Any] = {"name": name} if ds: entry["type"] = ds.type result.append(entry) return result @app.get("/datasources/{name}") - async def get_datasource(name: str) -> Dict[str, Any]: + async def get_datasource(name: str) -> dict[str, Any]: ds = await storage.get_datasource(name) if ds is None: raise HTTPException( @@ -465,18 +518,24 @@ async def get_datasource(name: str) -> Dict[str, Any]: ) # Mask credentials data = ds.model_dump(exclude_none=True) - for secret_field in ("password", "connection_string"): + for secret_field in ("password", "connection_string", "credentials_json"): if secret_field in data: data[secret_field] = "***" return data - @app.post("/datasources") - async def create_datasource(datasource: DatasourceConfig) -> Dict[str, str]: - await storage.save_datasource(datasource) + @app.post( + "/datasources", + responses={400: {"description": "Name conflicts with an existing datasource (differs only by case)."}}, + ) + async def create_datasource(datasource: DatasourceConfig) -> dict[str, str]: + try: + await storage.save_datasource(datasource) + except ValueError as e: + raise HTTPException(status_code=400, detail=str(e)) return {"status": "created", "name": datasource.name} @app.delete("/datasources/{name}") - async def delete_datasource(name: str) -> Dict[str, Any]: + async def delete_datasource(name: str) -> dict[str, Any]: deleted = await storage.delete_datasource(name) if not deleted: raise HTTPException( @@ -499,7 +558,7 @@ async def delete_datasource(name: str) -> Dict[str, Any]: ) async def validate_models_endpoint( request: ValidateModelsRequest, - ) -> List[Dict[str, Any]]: + ) -> list[dict[str, Any]]: """Diff persisted SlayerModels against live DB schemas. Read-only.""" if request.data_source is not None: ds_check = await storage.get_datasource(request.data_source) @@ -523,6 +582,26 @@ async def validate_models_endpoint( ) return [e.model_dump(mode="json") for e in entries] + @app.post( + "/recommend-root-model", + responses={400: {"description": "Unresolvable / wrong-kind / cross-datasource items."}}, + ) + async def recommend_root_model_endpoint( + request: RecommendRootModelRequest, + ) -> dict[str, Any]: + """Recommend the query ``source_model`` (root) for a set of + ``model.column`` / ``model.metric`` items, plus each item's + join-qualified path from that root. Read-only.""" + engine = SlayerQueryEngine(storage=storage) + try: + rec = await engine.recommend_root_model( + request.items, data_source=request.data_source, + root_hint=request.root_hint, + ) + except (ValueError, SlayerError) as exc: + raise HTTPException(status_code=400, detail=str(exc)) + return rec.model_dump(mode="json") + @app.post( "/ingest", responses={ @@ -538,7 +617,7 @@ async def validate_models_endpoint( }, }, ) - async def ingest(request: IngestRequest) -> Dict[str, Any]: + async def ingest(request: IngestRequest) -> dict[str, Any]: # Strip newlines from the user-controlled datasource name before it # reaches the log or the response detail (S5145 — log-injection # surface). The ds value already round-tripped through Pydantic so @@ -617,12 +696,13 @@ async def ingest(request: IngestRequest) -> Dict[str, Any]: } }, ) - async def save_memory(request: SaveMemoryRequest) -> Dict[str, Any]: + async def save_memory(request: SaveMemoryRequest) -> dict[str, Any]: try: response = await memory_service.save_memory( learning=request.learning, linked_entities=request.linked_entities, id=request.id, + description=request.description, ) except ( EntityResolutionError, @@ -639,7 +719,7 @@ async def save_memory(request: SaveMemoryRequest) -> Dict[str, Any]: 404: {"description": "Memory not found."}, }, ) - async def delete_memory(memory_id: str) -> Dict[str, Any]: + async def delete_memory(memory_id: str) -> dict[str, Any]: try: response = await memory_service.forget_memory(identifier=memory_id) except MemoryNotFoundError as exc: @@ -650,7 +730,9 @@ async def delete_memory(memory_id: str) -> Dict[str, Any]: # ---------- DEV-1375: semantic search ----------------------------- - search_service = SearchService(storage=storage) + # DEV-1516: pass the engine so the search service's post-fusion + # column-hit hook can auto-refresh stale categorical columns. + search_service = SearchService(storage=storage, engine=engine) @app.post( "/search", @@ -663,23 +745,48 @@ async def delete_memory(memory_id: str) -> Dict[str, Any]: } }, ) - async def search(request: SearchRequest) -> Dict[str, Any]: + async def search(request: SearchRequest) -> dict[str, Any]: try: response = await search_service.search( entities=request.entities, query=request.query, question=request.question, datasource=request.datasource, - max_memories=request.max_memories, - max_example_queries=request.max_example_queries, - max_entities=request.max_entities, + max_results=request.max_results, + cypher_filter=request.cypher_filter, + compact=request.compact, ) - except ( - EntityResolutionError, - AmbiguousModelError, - ValueError, - ) as exc: + except (SlayerError, ValueError) as exc: raise HTTPException(status_code=400, detail=str(exc)) return response.model_dump(mode="json") + inspect_service = InspectService(storage=storage, engine=engine) + + @app.post( + "/inspect", + responses={ + 400: { + "description": ( + "Invalid input: bad entity_type / format, negative " + "descriptions_max_chars, or an unresolvable reference." + ) + } + }, + ) + async def inspect(request: InspectRequest) -> dict[str, Any]: + try: + result = await inspect_service.inspect( + reference=request.reference, + entity_type=request.entity_type, + compact=request.compact, + format=request.format, + num_rows=request.num_rows, + show_sql=request.show_sql, + sections=request.sections, + descriptions_max_chars=request.descriptions_max_chars, + ) + except (SlayerError, ValueError) as exc: + raise HTTPException(status_code=400, detail=str(exc)) + return {"result": result} + return app diff --git a/slayer/async_utils.py b/slayer/async_utils.py index 383cbfeb..7daf559c 100644 --- a/slayer/async_utils.py +++ b/slayer/async_utils.py @@ -6,7 +6,8 @@ import asyncio import concurrent.futures -from typing import Any, Coroutine, TypeVar +from typing import Any, TypeVar +from collections.abc import Coroutine T = TypeVar("T") diff --git a/slayer/cli.py b/slayer/cli.py index b2d11041..db116992 100644 --- a/slayer/cli.py +++ b/slayer/cli.py @@ -5,7 +5,7 @@ import json import os import sys -from typing import List, Optional +from typing import Any from pydantic import BaseModel, Field @@ -14,6 +14,7 @@ AmbiguousModelError, EntityResolutionError, MemoryNotFoundError, + SlayerError, ) from slayer.core.models import SlayerModel from slayer.engine.ingestion import ( @@ -25,11 +26,14 @@ refresh_table_backed_model_sampled, ) from slayer.engine.query_engine import SlayerQueryEngine +from slayer.inspect.service import InspectService +from slayer.memories.help_seed import seed_help_memories from slayer.search.service import SearchService from slayer.storage import migrations as _mig from slayer.storage.base import default_storage_path from slayer.storage.type_refinement import ( has_refineable_columns, + has_sqlite_widenable_columns, refine_dict_with_live_schema, ) @@ -66,8 +70,8 @@ class RefreshSamplesResult(BaseModel): that didn't resolve in the requested scope — those are reported as a hard error so typos fail fast.""" - errors: List[str] = Field(default_factory=list) - unresolved_models: List[str] = Field(default_factory=list) + errors: list[str] = Field(default_factory=list) + unresolved_models: list[str] = Field(default_factory=list) def _add_storage_arg(parser): @@ -88,7 +92,7 @@ def _resolve_storage(args): return resolve_storage(path) -def main(): +def main(): # NOSONAR(S3776) — linear top-level CLI command dispatch (one elif per subcommand); splitting the dispatch chain would not improve readability parser = argparse.ArgumentParser( prog="slayer", description="SLayer — a lightweight semantic layer for AI agents", @@ -108,7 +112,12 @@ def main(): slayer serve --storage slayer.db slayer ingest --datasource my_pg --storage slayer.db -docs: https://motley-slayer.readthedocs.io/ +conceptual help: + # SLayer's concepts ship as help memories — read them with inspect: + slayer inspect memory:help.intro --type memory + slayer search --question "how do transforms work" + +docs: https://docs.motley.ai/slayer/ """, formatter_class=argparse.RawDescriptionHelpFormatter, ) @@ -303,6 +312,48 @@ def main(): ) _add_storage_arg(validate_parser) + # ── recommend-root-model ────────────────────────────────────────── + recommend_parser = subparsers.add_parser( + "recommend-root-model", + help="Recommend a query root model + join paths for a set of items", + epilog="""\ +examples: + slayer recommend-root-model orders.revenue customers.name + slayer recommend-root-model customers.name products.category --data-source my_pg + slayer recommend-root-model orders.revenue:sum regions.name --format json + slayer recommend-root-model customers.name products.category --root-hint orders +""", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + recommend_parser.add_argument( + "items", + nargs="+", + help="model.column / model.metric items (aggregation suffixes allowed)", + ) + recommend_parser.add_argument( + "--data-source", + dest="data_source", + default=None, + help="Datasource scope. If omitted, names resolve via the priority list.", + ) + recommend_parser.add_argument( + "--root-hint", + dest="root_hint", + default=None, + help=( + "Intended root model (bare name or '.'). " + "Honored when it reaches every item; otherwise the auto-pick is " + "used and a warning explains why." + ), + ) + recommend_parser.add_argument( + "--format", + choices=["text", "json"], + default="text", + help="Output format (default: text).", + ) + _add_storage_arg(recommend_parser) + # ── import-dbt ──────────────────────────────────────────────────── import_dbt_parser = subparsers.add_parser( "import-dbt", @@ -327,6 +378,26 @@ def main(): ) _add_storage_arg(import_dbt_parser) + # ── import-osi ──────────────────────────────────────────────────── + import_osi_parser = subparsers.add_parser( + "import-osi", + help="Import OSI (Open Semantic Interchange) configs into SLayer models", + epilog="""\ +examples: + slayer import-osi ./osi_configs --datasource my_postgres + slayer import-osi ./model.yaml --datasource my_pg --dialect SNOWFLAKE +""", + formatter_class=argparse.RawDescriptionHelpFormatter, + ) + import_osi_parser.add_argument("osi_path", help="Path to an OSI file or directory (YAML/JSON)") + import_osi_parser.add_argument("--datasource", required=True, help="SLayer datasource name for the imported models") + import_osi_parser.add_argument( + "--dialect", default="ANSI_SQL", + help="OSI expression dialect to read (default: ANSI_SQL). Falls back to " + "another SQL dialect when the requested one is absent.", + ) + _add_storage_arg(import_osi_parser) + # ── models ──────────────────────────────────────────────────────── models_parser = subparsers.add_parser( "models", @@ -501,6 +572,16 @@ def main(): "Duplicate id → upsert." ), ) + memory_save_parser.add_argument( + "--description", + default=None, + help=( + "Optional compact preview (<= 500 chars) surfaced by " + "`slayer search` and `inspect_model` when run in compact " + "mode (the default). Omit to let the renderer compute the " + "preview from the first paragraph of --learning." + ), + ) memory_forget_parser = memory_subparsers.add_parser( "forget", help="Delete a memory by id" @@ -539,6 +620,77 @@ def main(): ) _add_storage_arg(migrate_types_parser) + # ── inspect (DEV-1588) ─────────────────────────────────────────── + inspect_parser = subparsers.add_parser( + "inspect", + help=( + "Inspect one entity by reference and kind, or several of the same " + "kind at once (pass multiple references) (DEV-1588, DEV-1612)" + ), + ) + inspect_parser.add_argument( + "reference", + nargs="*", + help=( + "Entity reference(s): canonical id (mydb.orders.amount), bare " + "name, join path, or memory:. Pass two or more for a " + "homogeneous-kind batch (one --type for all). Omit entirely to " + "list the whole collection at --type (model / datasource only)." + ), + ) + inspect_parser.add_argument( + "--type", + required=True, + dest="entity_type", + choices=[ + "datasource", "model", "column", "measure", "aggregation", + "memory", + ], + help="Required. The entity kind to inspect.", + ) + inspect_parser.add_argument( + "--no-compact", + action="store_false", + dest="compact", + default=True, + help="Return the full render instead of the compact description.", + ) + inspect_parser.add_argument( + "--format", + choices=["markdown", "json"], + default="markdown", + help="Output format (default: markdown).", + ) + inspect_parser.add_argument( + "--num-rows", + type=int, + default=3, + dest="num_rows", + help="Sample-data rows (model entity_type only; default 3).", + ) + inspect_parser.add_argument( + "--show-sql", + action="store_true", + default=False, + dest="show_sql", + help="Include generated SQL (model entity_type only).", + ) + inspect_parser.add_argument( + "--section", + action="append", + default=None, + dest="sections", + help="Section subset (model entity_type only; repeatable).", + ) + inspect_parser.add_argument( + "--descriptions-max-chars", + type=int, + default=None, + dest="descriptions_max_chars", + help="Truncate description fields to this many characters.", + ) + _add_storage_arg(inspect_parser) + # ── search (DEV-1375) ──────────────────────────────────────────── search_parser = subparsers.add_parser( "search", @@ -584,25 +736,24 @@ def main(): ), ) search_parser.add_argument( - "--max-memories", + "--max-results", type=int, - default=5, - dest="max_memories", - help="Cap on returned learning-only memory hits (default 5).", - ) - search_parser.add_argument( - "--max-example-queries", - type=int, - default=2, - dest="max_example_queries", - help="Cap on returned query-bearing memory hits (default 2 — bulky).", + default=10, + dest="max_results", + help="Maximum total number of hits to return (default 10).", ) search_parser.add_argument( - "--max-entities", - type=int, - default=5, - dest="max_entities", - help="Cap on returned entity hits (default 5).", + "--cypher-filter", + default=None, + dest="cypher_filter", + help=( + "openCypher MATCH query returning '… AS id' to pre-filter all " + "channels to matching canonical IDs. When advanced_search is not " + "installed, only simple MATCH (n:Label1:Label2) RETURN n.id AS id " + "patterns are supported as a kind filter (multi-label uses union " + "semantics; allowed labels: Memory, Datasource, Model, Column, " + "Measure, Aggregation)." + ), ) search_parser.add_argument( "--format", @@ -610,6 +761,19 @@ def main(): default="text", help="Output format (default: text).", ) + search_parser.add_argument( + "--verbose", + action="store_true", + default=False, + help=( + "Opt out of compact rendering (DEV-1549). Default is " + "compact: memory hits surface ``description`` (or a " + "first-paragraph fallback from ``learning``) and an " + "empty ``text``; entity hits surface ``entity.description`` " + "and an empty ``text``. With ``--verbose`` the full hit " + "text is restored." + ), + ) refresh_parser = search_subparsers.add_parser( "refresh-samples", help="Re-profile and persist Column.sampled for table-backed models.", @@ -629,27 +793,9 @@ def main(): help="Model name(s) to refresh (repeatable; default: all in scope).", ) - # ── help ────────────────────────────────────────────────────────── - from slayer.help import TOPIC_SUMMARY_LINE - - help_parser = subparsers.add_parser( - "help", - help="Show conceptual help on SLayer (concepts, query composition, transforms, joins, workflow)", - epilog=( - f"{TOPIC_SUMMARY_LINE}\n\n" - "examples:\n" - " slayer help # intro\n" - " slayer help queries # deep dive on a topic\n" - " slayer help transforms\n" - ), - formatter_class=argparse.RawDescriptionHelpFormatter, - ) - help_parser.add_argument( - "topic", - nargs="?", - default=None, - help="Topic name (optional). If omitted, prints the intro.", - ) + # DEV-1658: the standalone `slayer help` subcommand is removed. SLayer's + # concepts ship as help memories — read them with + # `slayer inspect memory:help.intro --type memory` (see the epilog above). args = parser.parse_args() @@ -669,25 +815,64 @@ def main(): _run_ingest(args) elif args.command == "validate-models": _run_validate_models(args) + elif args.command == "recommend-root-model": + _run_recommend_root_model(args) elif args.command == "import-dbt": _run_import_dbt(args) + elif args.command == "import-osi": + _run_import_osi(args) elif args.command == "models": _run_models(args) elif args.command == "datasources": _run_datasources(args) elif args.command == "memory": _run_memory(args) + elif args.command == "inspect": + _run_inspect(args=args, storage=_resolve_storage(args)) elif args.command == "search": _run_search(args) elif args.command == "storage": _run_storage(args) - elif args.command == "help": - _run_help(args) else: parser.print_help() sys.exit(1) +def _run_inspect(*, args, storage) -> None: + """Run ``slayer inspect`` — a single-entity point-lookup (DEV-1588).""" + # DEV-1658: ensure the help.* memories exist so `inspect memory:help.intro` + # works on a fresh store (idempotent / warm no-op). + run_sync(seed_help_memories(storage=storage)) + service = InspectService( + storage=storage, engine=SlayerQueryEngine(storage=storage), + ) + # argparse ``nargs="*"`` always yields a list; map zero positionals to + # ``None`` (the collection sentinel, DEV-1667) and a single positional back + # to a bare str so single-id output stays byte-for-byte (DEV-1612). A direct + # str (older callers / tests) is passed through unchanged. + reference = args.reference + if isinstance(reference, list): + if len(reference) == 0: + reference = None + elif len(reference) == 1: + reference = reference[0] + try: + out = run_sync(service.inspect( + reference=reference, + entity_type=args.entity_type, + compact=args.compact, + format=args.format, + num_rows=args.num_rows, + show_sql=args.show_sql, + sections=args.sections, + descriptions_max_chars=args.descriptions_max_chars, + )) + except (SlayerError, ValueError) as exc: + _exit_with_error(exc) + return # for type checkers; _exit_with_error never returns + print(out) + + def _run_search(args) -> None: """Dispatch ``slayer search [...]`` and ``slayer search refresh-samples``.""" storage = _resolve_storage(args) @@ -703,8 +888,8 @@ async def _refresh_samples_async(*, args, storage) -> "RefreshSamplesResult": and (optional) model filters, accumulate per-column errors and the names of any user-specified models that didn't resolve.""" engine = SlayerQueryEngine(storage=storage) - errors: List[str] = [] - unresolved_models: List[str] = [] + errors: list[str] = [] + unresolved_models: list[str] = [] data_source = args.data_source models = args.models if data_source is None: @@ -765,7 +950,15 @@ def _run_search_refresh_samples(*, args, storage) -> None: def _print_search_response_text(response) -> None: - """Pretty-print a ``SearchResponse`` for the default text format.""" + """Pretty-print a ``SearchResponse`` for the default text format. + + DEV-1549: under compact mode ``hit.text`` is empty and the preview + lives in ``hit.description``; under ``--verbose`` (compact=False) + ``hit.text`` carries the full body and is what the caller wants to + see. Prefer ``text`` when non-empty so ``--verbose`` actually shows + the restored body; fall back to ``description`` for compact + responses. + """ for w in response.warnings: print(f"[warning] {w}") if response.resolved_input_entities: @@ -773,22 +966,24 @@ def _print_search_response_text(response) -> None: "\nResolved input entities: " + ", ".join(response.resolved_input_entities) ) - print(f"\nMemories ({len(response.memories)}):") - for hit in response.memories: - print(f" M{hit.id} (score={hit.score:.4f})") - print(f" {hit.text.splitlines()[0] if hit.text else ''}") - print(f"\nExample queries ({len(response.example_queries)}):") - for hit in response.example_queries: - print(f" M{hit.id} (score={hit.score:.4f})") - print(f" {hit.text.splitlines()[0] if hit.text else ''}") - print(f"\nEntities ({len(response.entities)}):") - for hit in response.entities: - print(f" [{hit.kind}] {hit.id} (score={hit.score:.4f})") + print(f"\nResults ({len(response.results)}):") + for hit in response.results: + if hit.kind == "memory": + prefix = "Q" if hit.query is not None else "M" + else: + prefix = f"[{hit.kind}]" + print(f" {prefix} {hit.id} (score={hit.score:.4f})") + preview = hit.text if hit.text else (hit.description or "") + preview_line = preview.splitlines()[0] if preview else "" + print(f" {preview_line}") def _run_search_query(args, storage) -> None: """``slayer search [...]`` — call the SearchService and emit JSON or pretty text.""" + # DEV-1658: seed help.* memories so concept searches surface them on a + # fresh store. Only on the query path — NOT `search refresh-samples`. + run_sync(seed_help_memories(storage=storage)) service = SearchService(storage=storage) query_input = _load_query_arg(args.query) if args.query else None try: @@ -797,11 +992,11 @@ def _run_search_query(args, storage) -> None: query=query_input, question=args.question, datasource=args.datasource, - max_memories=args.max_memories, - max_example_queries=args.max_example_queries, - max_entities=args.max_entities, + max_results=args.max_results, + cypher_filter=args.cypher_filter, + compact=not getattr(args, "verbose", False), )) - except (EntityResolutionError, AmbiguousModelError, ValueError) as exc: + except (SlayerError, ValueError) as exc: _exit_with_error(exc) return if args.format == "json": @@ -858,6 +1053,49 @@ def _run_storage_migrate_types(args) -> None: print(f"\nDone: refined {refined_total} model(s).") +def _resolve_datasource_for_cli_refinement( + *, inner, ds_name: str, model_name: str, needs_double: bool, +) -> Any | None: + """Resolve the datasource for ``slayer storage migrate-types``. + + Returns the ``DatasourceConfig`` when present, ``None`` when missing + and the model is SQLite-INT-only (best-effort skip — prints a stderr + skip notice). Raises ``ValueError`` when the model has DOUBLE base + columns and the DS is missing (DEV-1361 hard-fail contract). + """ + ds = run_sync(inner.get_datasource(ds_name)) + if ds is not None: + return ds + if needs_double: + raise ValueError( + f"Cannot refine model {ds_name!r}.{model_name!r}: datasource " + f"{ds_name!r} is unavailable for type refinement. Restore the " + f"datasource entry or remove the stale model file." + ) + print( + f"skipped {ds_name}.{model_name}: datasource {ds_name!r} unavailable " + f"for SQLite affinity probe (INT columns untouched). Restore the " + f"datasource and re-run.", + file=sys.stderr, + ) + return None + + +def _print_refinement_diff( + *, ds_name: str, model_name: str, upgraded: dict, types_before: dict[str, str], +) -> None: + """Print before→after diffs for columns whose type changed during + refinement. Migration-only aliases are excluded because ``types_before`` + is captured AFTER migration.""" + print(f"refined {ds_name}.{model_name}:") + for col in upgraded.get("columns", []) or []: + name = col.get("name") or "?" + before = types_before.get(name, None) + after = col.get("type", "?") + if before != after: + print(f" - {name}: {before or '?'} → {after}") + + def _refine_one_model_for_cli( *, inner, ds_name: str, model_name: str, dry_run: bool, ) -> bool: @@ -871,38 +1109,39 @@ def _refine_one_model_for_cli( dict has refineable DOUBLE base columns AND the datasource entry is missing, raises ``ValueError`` rather than silently reporting "nothing to refine" for a model the CLI never had enough information to inspect. - Models with no refineable columns (text-only, query-backed, sql-mode, + DEV-1538 SQLite-INT widening is best-effort: a missing datasource for + an INT-only model logs a skip notice and returns False. Models with no + refineable or widenable columns (text-only, query-backed, sql-mode, already-narrowed) skip silently and don't require a live datasource. """ raw = run_sync(_load_raw_model_dict(inner, ds_name, model_name)) if raw is None: return False - # Snapshot the original column types before migration mutates the - # shared inner dicts, so we can show before/after diffs. - original_types = { + upgraded = _mig.migrate("SlayerModel", copy.deepcopy(raw)) + # Snapshot column types AFTER migration but BEFORE refinement so the + # before/after diff reports only actual refinement changes — migration- + # only aliases like ``number → DOUBLE`` are not refinement events. + types_before_refinement = { (c.get("name") or "?"): c.get("type", "?") - for c in raw.get("columns", []) or [] + for c in upgraded.get("columns", []) or [] if isinstance(c, dict) } - upgraded = _mig.migrate("SlayerModel", copy.deepcopy(raw)) - if not has_refineable_columns(upgraded): + needs_double = has_refineable_columns(upgraded) + needs_sqlite_int = has_sqlite_widenable_columns(upgraded) + if not (needs_double or needs_sqlite_int): return False - ds = run_sync(inner.get_datasource(ds_name)) + ds = _resolve_datasource_for_cli_refinement( + inner=inner, ds_name=ds_name, model_name=model_name, + needs_double=needs_double, + ) if ds is None: - raise ValueError( - f"Cannot refine model {ds_name!r}.{model_name!r}: datasource " - f"{ds_name!r} is unavailable for type refinement. Restore the " - f"datasource entry or remove the stale model file." - ) + return False if not refine_dict_with_live_schema(upgraded, ds): return False - print(f"refined {ds_name}.{model_name}:") - for col in upgraded.get("columns", []) or []: - if col.get("type") != "INT": - continue - before = original_types.get(col.get("name") or "", None) - if before != "INT": - print(f" - {col['name']}: {before or '?'} → INT") + _print_refinement_diff( + ds_name=ds_name, model_name=model_name, + upgraded=upgraded, types_before=types_before_refinement, + ) if not dry_run: model = SlayerModel.model_validate(upgraded) # Save through inner so we don't re-trigger the load-time @@ -911,7 +1150,7 @@ def _refine_one_model_for_cli( return True -async def _load_raw_model_dict(storage, data_source: str, name: str) -> Optional[dict]: +async def _load_raw_model_dict(storage, data_source: str, name: str) -> dict | None: """Read a model's raw on-disk dict bypassing Pydantic's validator chain.""" import json as _json import os as _os @@ -932,12 +1171,6 @@ async def _load_raw_model_dict(storage, data_source: str, name: str) -> Optional return None -def _run_help(args): - from slayer.help import render_help - - print(render_help(topic=args.topic)) - - def _parse_cli_variables(args) -> dict: """Combine ``--variables KEY=VALUE`` (repeatable) and ``--variables-json`` into a single dict. Errors out if both forms are mixed. @@ -1133,7 +1366,7 @@ def _run_ingest(args): ) -def _format_edit_entry_lines(entry) -> List[str]: +def _format_edit_entry_lines(entry) -> list[str]: lines = [f"EDIT MODEL: {entry.model_name} (datasource: {entry.data_source})"] for attr, label in _REMOVE_SECTIONS: values = getattr(entry.remove, attr) @@ -1148,7 +1381,7 @@ def _format_validate_models_output(entries) -> str: """Render a List[ToDeleteEntry] as human-readable text for CLI output.""" if not entries: return "No drift detected." - lines: List[str] = [] + lines: list[str] = [] for entry in entries: if entry.tool == "delete_model": lines.append( @@ -1211,8 +1444,29 @@ def _run_validate_models(args): print("\n✓ no remaining drift") +def _run_recommend_root_model(args): + import json as _json + + from slayer.core.recommend import render_recommendation_markdown + from slayer.engine.query_engine import SlayerQueryEngine + + storage = _resolve_storage(args) + engine = SlayerQueryEngine(storage=storage) + try: + rec = engine.recommend_root_model_sync( + args.items, data_source=args.data_source, + root_hint=getattr(args, "root_hint", None), + ) + except Exception as exc: # noqa: BLE001 — surface resolution/validation errors cleanly + print(f"recommend-root-model failed: {exc}") + sys.exit(1) + if args.format == "json": + print(_json.dumps(rec.model_dump(mode="json"), indent=2)) + else: + print(render_recommendation_markdown(rec)) + + def _run_import_dbt(args): - import sqlalchemy as sa from slayer.dbt.converter import DbtToSlayerConverter from slayer.dbt.parser import parse_dbt_project @@ -1229,8 +1483,8 @@ def _run_import_dbt(args): sys.exit(1) sa_engine = None + ds = run_sync(storage.get_datasource(args.datasource)) if include_hidden: - ds = run_sync(storage.get_datasource(args.datasource)) if ds is None: storage_path = args.storage or args.models_dir or _STORAGE_DEFAULT print( @@ -1238,19 +1492,22 @@ def _run_import_dbt(args): "required for --include-hidden-models." ) sys.exit(1) - sa_engine = sa.create_engine(ds.resolve_env_vars().get_connection_string()) + from slayer.sql import engine_factory + sa_engine = engine_factory.get_engine(ds.resolve_env_vars()) - try: - converter = DbtToSlayerConverter( - project=project, - data_source=args.datasource, - sa_engine=sa_engine, - include_hidden_models=include_hidden, - ) - result = converter.convert() - finally: - if sa_engine is not None: - sa_engine.dispose() + # DEV-1595: pass the datasource dialect (best-effort) so the converter can + # emit percentile/median caveats for dialects that lack them (MySQL/T-SQL). + target_dialect = ds.type if ds is not None else None + + converter = DbtToSlayerConverter( + project=project, + data_source=args.datasource, + sa_engine=sa_engine, + include_hidden_models=include_hidden, + target_dialect=target_dialect, + ) + result = converter.convert() + # Cached engine — engine_factory owns its lifecycle; don't dispose. hidden_count = 0 for model in result.models: @@ -1263,19 +1520,72 @@ def _run_import_dbt(args): f"({len(model.columns)} columns, {len(model.measures)} measures)" ) - for u in result.unconverted_metrics: - context = u.model_name or u.metric_name or "general" - print(f" UNCONVERTED [{context}]: {u.message}") - - for w in result.warnings: - context = w.model_name or w.metric_name or "general" - print(f" WARNING [{context}]: {w.message}") + # DEV-1595: grouped, category-keyed conversion report + a severity tally. + if result.unconverted_metrics or result.warnings: + print("\nConversion report:") + print(result.render_report()) visible_count = len(result.models) - hidden_count + unconverted, dropped = result.tally() print( f"\nDone: {visible_count} models, {hidden_count} hidden, " - f"{len(result.unconverted_metrics)} unconverted metrics, " - f"{len(result.warnings)} warnings" + f"{unconverted} unconverted, {dropped} dropped" + ) + + +def _run_import_osi(args): + from slayer.osi.converter import OsiConversionError, OsiToSlayerConverter + from slayer.osi.parser import parse_osi_path + from slayer.sql import engine_factory + + storage = _resolve_storage(args) + try: + documents = parse_osi_path(args.osi_path) + except FileNotFoundError as exc: + print(str(exc)) + sys.exit(1) + if not documents: + print(f"No OSI documents found in {args.osi_path}") + sys.exit(1) + + ds = run_sync(storage.get_datasource(args.datasource)) + if ds is None: + storage_path = args.storage or args.models_dir or _STORAGE_DEFAULT + print( + f"Datasource '{args.datasource}' not found in {storage_path}; " + "a reachable datasource is required (types come from live introspection)." + ) + sys.exit(1) + + sa_engine = engine_factory.get_engine(ds.resolve_env_vars()) + converter = OsiToSlayerConverter( + documents=documents, + data_source=args.datasource, + sa_engine=sa_engine, + dialect=args.dialect, + target_dialect=ds.type, + ) + try: + result = converter.convert() + except OsiConversionError as exc: + print(str(exc)) + sys.exit(1) + + for model in result.models: + run_sync(storage.save_model(model)) + print( + f"Imported model: {model.name} " + f"({len(model.columns)} columns, {len(model.measures)} measures)" + ) + + if result.unconverted_metrics or result.warnings: + print("\nConversion report:") + print(result.render_report()) + + unconverted, dropped = result.tally() + print( + f"\nDone: {len(result.models)} models, " + f"{unconverted} unconverted, {dropped} dropped" ) @@ -1356,10 +1666,9 @@ def _run_datasources(args): print(f"Datasource '{args.name}' not found.") sys.exit(1) data = ds.model_dump(mode="json", exclude_none=True) - if "password" in data: - data["password"] = "********" - if "connection_string" in data: - data["connection_string"] = "********" + for secret_field in ("password", "connection_string", "credentials_json"): + if secret_field in data: + data[secret_field] = "********" print(yaml.dump(data, sort_keys=False, default_flow_style=False).rstrip()) elif args.datasources_command == "create": @@ -1379,12 +1688,13 @@ def _run_datasources(args): print(f"Datasource '{args.name}' not found.") sys.exit(1) import sqlalchemy as sa + from slayer.sql import engine_factory try: - engine = sa.create_engine(ds.resolve_env_vars().get_connection_string()) + engine = engine_factory.get_engine(ds.resolve_env_vars()) with engine.connect() as conn: conn.execute(sa.text("SELECT 1")) - engine.dispose() + # Cached engine — engine_factory owns lifecycle; don't dispose. print(f"OK — connected to '{args.name}' ({ds.type}).") except Exception as e: print(f"FAILED — {e}") @@ -1429,6 +1739,22 @@ def _parse_connection_string(url: str) -> tuple[str, str]: ) return ds_type, stem + # DEV-1551: Snowflake connection_name sentinel URL has no path segment — + # all routing lives in the query string + the TOML profile. Use the + # connection_name itself as the derived datasource name fallback so + # ``slayer datasources create "snowflake://?connection_name=default"`` + # works without --name. + if ds_type == "snowflake": + connection_name = "" + if parsed.query: + from urllib.parse import parse_qs # noqa: PLC0415 + params = parse_qs(parsed.query) + connection_name = (params.get("connection_name") or [""])[0] + if connection_name: + return ds_type, connection_name + # Inline form: snowflake://user:pw@account/db/schema?warehouse=... — + # take the first path segment (database) like other networked dialects. + # Networked: take the first non-empty path segment (Postgres/MySQL/ClickHouse all put db there). segments = [s for s in parsed.path.split("/") if s] if not segments: @@ -1479,7 +1805,11 @@ def _persist_ingested_models(models, storage, *, assume_yes: bool, pre_save=None for model in models: if pre_save is not None: pre_save(model) - run_sync(storage.save_model(model)) + try: + run_sync(storage.save_model(model)) + except ValueError as e: + print(f"Skipped {model.name}: {e}") + continue print(f"Ingested: {model.name} ({len(model.columns)} columns, {len(model.measures)} measures)") @@ -1513,7 +1843,11 @@ def _run_datasources_create(args, storage): print("Aborted.") sys.exit(1) - run_sync(storage.save_datasource(ds)) + try: + run_sync(storage.save_datasource(ds)) + except ValueError as e: + print(f"Error: {e}") + sys.exit(1) print(f"Created datasource '{ds.name}' ({ds.type}).") if not args.ingest: @@ -1581,7 +1915,11 @@ def _run_datasources_create_demo(args, storage): # NOSONAR S3776 — linear dem print("Aborted.") sys.exit(1) - run_sync(storage.save_datasource(ds)) + try: + run_sync(storage.save_datasource(ds)) + except ValueError as e: + print(f"Error: {e}") + sys.exit(1) print(f"Created datasource '{ds.name}' (duckdb).") if not args.ingest: @@ -1651,6 +1989,7 @@ def _run_memory_save(args, service): learning=args.learning, linked_entities=linked, id=getattr(args, "id", None), + description=getattr(args, "description", None), ) ) except (EntityResolutionError, AmbiguousModelError, ValueError) as exc: diff --git a/slayer/client/slayer_client.py b/slayer/client/slayer_client.py index fa9ea227..cc63de22 100644 --- a/slayer/client/slayer_client.py +++ b/slayer/client/slayer_client.py @@ -1,16 +1,15 @@ """Python client for SLayer API.""" import logging -from collections.abc import Mapping as ABCMapping, Sequence as ABCSequence +from collections.abc import ( + Mapping, + Mapping as ABCMapping, + Sequence, + Sequence as ABCSequence, +) from typing import ( TYPE_CHECKING, Any, - Dict, - List, - Mapping, - Optional, - Sequence, - Union, ) from urllib.parse import quote @@ -22,6 +21,8 @@ ) if TYPE_CHECKING: + from slayer.core.policy import SessionPolicy + from slayer.core.recommend import RootModelRecommendation from slayer.search.service import SearchResponse logger = logging.getLogger(__name__) @@ -33,12 +34,12 @@ # ``{"queries": [...]}`` (REST) also accept; ``str`` runs a query-backed model # by name. ``Mapping``/``Sequence`` (not ``Dict``/``List``) so callers passing # ``list[dict[str, str]]`` aren't rejected by pyright's invariance check. -QueryInput = Union[ - SlayerQuery, - Mapping[str, Any], - Sequence[Union[SlayerQuery, Mapping[str, Any]]], - str, -] +QueryInput = ( + SlayerQuery + | Mapping[str, Any] + | Sequence[SlayerQuery | Mapping[str, Any]] + | str +) class SlayerClient: @@ -62,7 +63,9 @@ class SlayerClient: def __init__( self, url: str = "http://localhost:5143", - storage: Optional[Any] = None, + storage: Any | None = None, + *, + policy: "SessionPolicy | None" = None, ): self.url = url.rstrip("/") self._storage = storage @@ -70,14 +73,23 @@ def __init__( if storage is not None: from slayer.engine.query_engine import SlayerQueryEngine - self._engine = SlayerQueryEngine(storage=storage) + self._engine = SlayerQueryEngine(storage=storage, policy=policy) + elif policy is not None: + # DEV-1578: forced-filter policy is enforced in the local engine + # only. Silently ignoring it in HTTP mode would disable a security + # control, so fail fast instead. + raise ValueError( + "policy= is only supported in local-engine mode (pass " + "storage=...); server-side policy over HTTP is not yet " + "available." + ) async def _request( self, method: str, path: str, - json: Optional[Dict] = None, - params: Optional[Dict] = None, + json: dict | None = None, + params: dict | None = None, ) -> Any: try: import httpx @@ -94,8 +106,8 @@ def _request_sync( self, method: str, path: str, - json: Optional[Dict] = None, - params: Optional[Dict] = None, + json: dict | None = None, + params: dict | None = None, ) -> Any: try: import httpx @@ -109,7 +121,7 @@ def _request_sync( return resp.json() @staticmethod - def _validated_dump(payload: Mapping[str, Any]) -> Dict[str, Any]: + def _validated_dump(payload: Mapping[str, Any]) -> dict[str, Any]: """Round-trip a single-query payload through ``SlayerQuery`` so the server sees the normalised JSON-mode shape. Necessary because the server's ``QueryRequest`` declares ``measures`` / @@ -128,7 +140,7 @@ def _build_query_body( *, dry_run: bool = False, explain: bool = False, - ) -> Dict[str, Any]: + ) -> dict[str, Any]: """Convert any accepted input shape into the JSON body for ``POST /query``. Single source of truth shared by sync + async transports. Never mutates caller-owned dicts or lists. @@ -149,7 +161,7 @@ def _build_query_body( both accept them). """ if isinstance(query, str): - body: Dict[str, Any] = {"name": query} + body: dict[str, Any] = {"name": query} elif isinstance(query, SlayerQuery): body = query.model_dump(mode="json", exclude_none=True) elif isinstance(query, ABCSequence) and not isinstance( @@ -157,7 +169,7 @@ def _build_query_body( ): # ``str`` is also a Sequence but already handled above; guard the # binary string types here too so they raise via the else-branch. - serialised: List[Dict[str, Any]] = [] + serialised: list[dict[str, Any]] = [] for i, item in enumerate(query): if isinstance(item, SlayerQuery): serialised.append( @@ -214,7 +226,7 @@ def _parse_response(result: dict) -> SlayerResponse: """Parse an API JSON response into a SlayerResponse.""" from slayer.core.format import NumberFormat - def _parse_meta_dict(d: dict) -> Dict[str, FieldMetadata]: + def _parse_meta_dict(d: dict) -> dict[str, FieldMetadata]: out = {} for k, v in (d or {}).items(): fmt = None @@ -274,7 +286,7 @@ async def explain(self, query: QueryInput) -> SlayerResponse: """ return await self.query(query=query, explain=True) - async def list_models(self, data_source: Optional[str] = None) -> List[str]: + async def list_models(self, data_source: str | None = None) -> list[str]: if self._storage is not None: names = await self._storage.list_models(data_source=data_source) return list(names) @@ -284,29 +296,29 @@ async def list_models(self, data_source: Optional[str] = None) -> List[str]: async def get_model( self, name: str, - data_source: Optional[str] = None, - ) -> Optional[Any]: + data_source: str | None = None, + ) -> Any | None: if self._storage is not None: return await self._storage.get_model(name, data_source=data_source) params = {"data_source": data_source} if data_source else None return await self._request(method="GET", path=f"/models/{name}", params=params) # NOSONAR(S1192) — REST path is the API contract; defining a constant adds indirection without value - async def create_model(self, model: Dict[str, Any]) -> Dict[str, str]: + async def create_model(self, model: dict[str, Any]) -> dict[str, str]: return await self._request(method="POST", path="/models", json=model) # NOSONAR(S1192) — REST path is the API contract; defining a constant adds indirection without value - async def list_datasources(self) -> List[str]: + async def list_datasources(self) -> list[str]: return await self._request(method="GET", path="/datasources") - async def create_datasource(self, datasource: Dict[str, Any]) -> Dict[str, str]: + async def create_datasource(self, datasource: dict[str, Any]) -> dict[str, str]: return await self._request(method="POST", path="/datasources", json=datasource) - async def get_datasource_priority(self) -> List[str]: + async def get_datasource_priority(self) -> list[str]: if self._storage is not None: return await self._storage.get_datasource_priority() body = await self._request(method="GET", path="/datasources/priority") return list(body.get("priority", [])) - async def set_datasource_priority(self, priority: List[str]) -> None: + async def set_datasource_priority(self, priority: list[str]) -> None: if self._storage is not None: await self._storage.set_datasource_priority(list(priority)) return @@ -342,30 +354,38 @@ async def save_memory( self, *, learning: str, - linked_entities: Union[List[str], SlayerQuery, Dict[str, Any]], - id: Optional[str] = None, # noqa: A002 — public kwarg matching MCP / REST + linked_entities: list[str] | SlayerQuery | dict[str, Any], + id: str | None = None, # noqa: A002 — public kwarg matching MCP / REST + description: str | None = None, ) -> SaveMemoryResponse: """Save a memory: a learning text + linked entities (or an inline SlayerQuery to extract entities from). DEV-1428: - optional ``id`` lets callers pin the canonical memory id.""" + optional ``id`` lets callers pin the canonical memory id. + + DEV-1549: optional ``description`` is a ≤ 500-char preview + surfaced by ``search(compact=True)`` and ``inspect_model``. + """ if self._storage is not None: response = await self._memory_service().save_memory( learning=learning, linked_entities=self._coerce_linked_entities(linked_entities), id=id, + description=description, ) return response - body: Dict[str, Any] = { + body: dict[str, Any] = { "learning": learning, "linked_entities": self._coerce_linked_entities(linked_entities), } if id is not None: body["id"] = id + if description is not None: + body["description"] = description result = await self._request(method="POST", path="/memories", json=body) return SaveMemoryResponse.model_validate(result) async def forget_memory( - self, identifier: Union[int, str] + self, identifier: int | str ) -> ForgetMemoryResponse: if self._storage is not None: return await self._memory_service().forget_memory( @@ -385,23 +405,22 @@ async def forget_memory( async def search( self, *, - entities: Optional[List[str]] = None, - query: Optional[Union[SlayerQuery, Dict[str, Any]]] = None, - question: Optional[str] = None, - datasource: Optional[str] = None, - max_memories: int = 5, - max_example_queries: int = 2, - max_entities: int = 5, + entities: list[str] | None = None, + query: SlayerQuery | dict[str, Any] | None = None, + question: str | None = None, + datasource: str | None = None, + max_results: int = 10, + cypher_filter: str | None = None, + compact: bool = True, ) -> "SearchResponse": """Up to three-channel semantic search over memories + canonical entities. Channels: (1) entity-overlap BM25 over memories; (2) tantivy full-text over memories ∪ entities; (3) optional dense embedding - similarity (gated by the ``embedding_search`` extra and a - configured provider API key). Memory rankings from all active - channels and entity rankings from channels 2 and 3 are fused via - Reciprocal Rank Fusion (``k=60``). + similarity (gated by the ``advanced_search`` extra and a + configured provider API key). All hits are fused via Reciprocal + Rank Fusion (``k=60``) into a single ranked ``results`` list. ``datasource`` (DEV-1409, optional): when set, scope memories and entities to that one datasource. Entity hits are limited to docs @@ -435,14 +454,13 @@ async def search( query=coerced_query, question=question, datasource=datasource, - max_memories=max_memories, - max_example_queries=max_example_queries, - max_entities=max_entities, + max_results=max_results, + cypher_filter=cypher_filter, + compact=compact, ) - body: Dict[str, Any] = { - "max_memories": max_memories, - "max_example_queries": max_example_queries, - "max_entities": max_entities, + body: dict[str, Any] = { + "max_results": max_results, + "compact": compact, } if entities is not None: body["entities"] = entities @@ -452,9 +470,95 @@ async def search( body["question"] = question if datasource is not None: body["datasource"] = datasource + if cypher_filter is not None: + body["cypher_filter"] = cypher_filter result = await self._request(method="POST", path="/search", json=body) return SearchResponse.model_validate(result) + async def inspect( + self, + *, + reference: str | list[str] | None = None, + entity_type: str, + compact: bool = True, + format: str = "markdown", + num_rows: int = 3, + show_sql: bool = False, + sections: list[str] | None = None, + descriptions_max_chars: int | None = None, + ) -> str: + """Inspect EXACTLY one entity by reference and kind (DEV-1588), a + homogeneous-kind BATCH when ``reference`` is a list (DEV-1612), or the + whole COLLECTION at a kind when ``reference`` is ``None`` / ``[]`` + (DEV-1667). + + A point-lookup (no fusion / ranking / bundled memories). + ``entity_type`` is required, one of + ``datasource``/``model``/``column``/``measure``/``aggregation``/ + ``memory``, and applies to every id in a list. A single ``str`` keeps + its byte-for-byte single output; a list returns, in input order, one + ``## `` block per id under ``format="markdown"`` or a JSON + array under ``format="json"``, with per-id error isolation. ``None`` / + ``[]`` lists the whole collection (``model`` / ``datasource`` only). + """ + if self._storage is not None: + # Local import: slayer.inspect.service transitively imports the + # search render stack (tantivy), which is not part of the client + # extras. Remote-only installs that never call .inspect() must + # not blow up at module-import time. + from slayer.inspect.service import InspectService + + return await InspectService( + storage=self._storage, engine=self._engine, + ).inspect( + reference=reference, + entity_type=entity_type, + compact=compact, + format=format, + num_rows=num_rows, + show_sql=show_sql, + sections=sections, + descriptions_max_chars=descriptions_max_chars, + ) + body = self._build_inspect_body( + reference=reference, + entity_type=entity_type, + compact=compact, + format=format, + num_rows=num_rows, + show_sql=show_sql, + sections=sections, + descriptions_max_chars=descriptions_max_chars, + ) + resp = await self._request(method="POST", path="/inspect", json=body) + return resp["result"] + + @staticmethod + def _build_inspect_body( + *, + reference: str | list[str] | None, + entity_type: str, + compact: bool, + format: str, + num_rows: int, + show_sql: bool, + sections: list[str] | None, + descriptions_max_chars: int | None, + ) -> dict[str, Any]: + body: dict[str, Any] = { + "reference": reference, + "entity_type": entity_type, + "compact": compact, + "format": format, + "num_rows": num_rows, + "show_sql": show_sql, + } + if sections is not None: + body["sections"] = sections + if descriptions_max_chars is not None: + body["descriptions_max_chars"] = descriptions_max_chars + return body + # ----- Sync API (for notebooks, scripts, CLI) ----- def query_sync( @@ -494,6 +598,83 @@ def explain_sync(self, query: QueryInput) -> SlayerResponse: """ return self.query_sync(query=query, explain=True) + def inspect_sync( + self, + *, + reference: str | list[str] | None = None, + entity_type: str, + compact: bool = True, + format: str = "markdown", + num_rows: int = 3, + show_sql: bool = False, + sections: list[str] | None = None, + descriptions_max_chars: int | None = None, + ) -> str: + """Synchronous variant of :meth:`inspect` (DEV-1588; batch DEV-1612; + collection DEV-1667).""" + if self._storage is not None: + from slayer.async_utils import run_sync + + return run_sync(self.inspect( + reference=reference, + entity_type=entity_type, + compact=compact, + format=format, + num_rows=num_rows, + show_sql=show_sql, + sections=sections, + descriptions_max_chars=descriptions_max_chars, + )) + body = self._build_inspect_body( + reference=reference, + entity_type=entity_type, + compact=compact, + format=format, + num_rows=num_rows, + show_sql=show_sql, + sections=sections, + descriptions_max_chars=descriptions_max_chars, + ) + resp = self._request_sync(method="POST", path="/inspect", json=body) + return resp["result"] + + async def recommend_root_model( + self, items: list[str], *, data_source: str | None = None, + root_hint: str | None = None, + ) -> "RootModelRecommendation": + """Recommend the query ``source_model`` (root) for a set of + ``model.column`` / ``model.metric`` items, plus each item's + join-qualified path from that root (DEV-1626). + + ``root_hint`` forces the intended root when it reaches every item + (else the auto-pick is used with an explanatory warning).""" + from slayer.core.recommend import RootModelRecommendation + + if self._engine is not None: + return await self._engine.recommend_root_model( + items, data_source=data_source, root_hint=root_hint + ) + body: dict[str, Any] = {"items": items} + if data_source is not None: + body["data_source"] = data_source + if root_hint is not None: + body["root_hint"] = root_hint + result = await self._request( + method="POST", path="/recommend-root-model", json=body + ) + return RootModelRecommendation.model_validate(result) + + def recommend_root_model_sync( + self, items: list[str], *, data_source: str | None = None, + root_hint: str | None = None, + ) -> "RootModelRecommendation": + """Synchronous variant of :meth:`recommend_root_model`.""" + from slayer.async_utils import run_sync + + return run_sync(self.recommend_root_model( + items, data_source=data_source, root_hint=root_hint + )) + def query_df(self, query: QueryInput): """Execute a query and return a pandas DataFrame (sync). @@ -506,8 +687,8 @@ def query_df(self, query: QueryInput): result = self.query_sync(query=query) return pd.DataFrame(result.data) - def list_models_sync(self) -> List[str]: + def list_models_sync(self) -> list[str]: return self._request_sync(method="GET", path="/models") - def get_model_sync(self, name: str) -> Dict[str, Any]: + def get_model_sync(self, name: str) -> dict[str, Any]: return self._request_sync("GET", f"/models/{name}") diff --git a/slayer/core/enums.py b/slayer/core/enums.py index 51a67ba9..9da753fa 100644 --- a/slayer/core/enums.py +++ b/slayer/core/enums.py @@ -1,8 +1,9 @@ """Core enums for SLayer.""" import datetime # noqa: F401 (kept for downstream imports of TimeGranularity) +import difflib from enum import Enum -from typing import Any, Optional +from typing import Any class StrEnum(str, Enum): @@ -13,7 +14,24 @@ def __str__(self) -> str: class DataType(StrEnum): """SLayer data types — values match sqlglot's ``exp.DataType.Type`` byte-for-byte so SQL generation can ``CAST`` to the declared type without - a translation map. (DEV-1361.)""" + a translation map. (DEV-1361.) + + ``UNKNOWN`` is the explicit *opaque* type: the column's database type was + detected but SLayer cannot operate on it — it has no default btree/hash + operator class, which is exactly what ``GROUP BY`` / ``DISTINCT`` require + (``json``, ``xml``, the geometric / PostGIS types, range types, ...). + Comparable types keep working as ``TEXT`` even when unmapped — ``jsonb``, + ``uuid``, ``bytea``, arrays and ``tsvector`` are all groupable and are + deliberately *not* opaque. ``slayer.engine.ingestion._OPAQUE_SA_TYPE_NAMES`` + is the source of truth for that classification. Such a column is + **stored and displayed** — its raw DB type string is kept on + ``Column.db_type`` — but it is never used in ``GROUP BY``, ``DISTINCT``, + aggregation, or ``CAST``, because those operations fail at the database + (e.g. "could not identify an equality operator for type point"). Use the + :attr:`is_opaque` property rather than comparing against the member + directly. ``UNKNOWN`` is also a real ``sqlglot`` type name, so the + byte-equality invariant above still holds. + """ TEXT = "TEXT" INT = "INT" @@ -21,13 +39,19 @@ class DataType(StrEnum): BOOLEAN = "BOOLEAN" DATE = "DATE" TIMESTAMP = "TIMESTAMP" + UNKNOWN = "UNKNOWN" + + @property + def is_opaque(self) -> bool: + """True when SLayer can store/display the column but not operate on it.""" + return self is DataType.UNKNOWN # DEV-1361: lenient before-validator absorbs legacy lowercase type spellings # from older agent input (MCP/REST/CLI), pseudo-types (count/sum/...) drop to # None so the field falls through to its default. Used by both Column and # ModelMeasure validators in slayer/core/models.py. -_LEGACY_DATATYPE_ALIASES: dict[str, Optional[str]] = { +_LEGACY_DATATYPE_ALIASES: dict[str, str | None] = { # Pre-rename canonical values. "string": "TEXT", "number": "DOUBLE", @@ -35,6 +59,7 @@ class DataType(StrEnum): "time": "TIMESTAMP", "date": "DATE", "boolean": "BOOLEAN", + "unknown": "UNKNOWN", # Aggregation pseudo-types — dropped in v5 because they were unused. "count": None, "count_distinct": None, @@ -68,6 +93,10 @@ class TimeGranularity(StrEnum): HOUR = "hour" DAY = "day" WEEK = "week" + # DEV-1572: Sunday-anchored week (weeks start on Sunday, end on Saturday). + # WEEK is Monday-anchored (ISO-8601); WEEK_SUNDAY exists so Metabase week + # breakouts — which use Sunday weeks — bucket the way Metabase asked for. + WEEK_SUNDAY = "week_sunday" MONTH = "month" QUARTER = "quarter" YEAR = "year" @@ -79,6 +108,10 @@ def period_start(self, date: datetime.date) -> datetime.date: return date elif self == TimeGranularity.WEEK: return date - datetime.timedelta(days=date.weekday()) + elif self == TimeGranularity.WEEK_SUNDAY: + # Round back to the Sunday at or before ``date``. weekday(): Mon=0.. + # Sun=6, so (weekday + 1) % 7 is the number of days since Sunday. + return date - datetime.timedelta(days=(date.weekday() + 1) % 7) elif self == TimeGranularity.MONTH: return date.replace(day=1) elif self == TimeGranularity.QUARTER: @@ -95,6 +128,10 @@ def period_end(self, date: datetime.date) -> datetime.date: return date elif self == TimeGranularity.WEEK: return date + datetime.timedelta(days=6 - date.weekday()) + elif self == TimeGranularity.WEEK_SUNDAY: + # Advance to the Saturday at or after ``date`` (last day of the + # Sunday-anchored week). weekday(): Mon=0..Sun=6, Saturday=5. + return date + datetime.timedelta(days=(5 - date.weekday()) % 7) elif self == TimeGranularity.MONTH: if date.month == 12: return date.replace(year=date.year + 1, month=1, day=1) - datetime.timedelta(days=1) @@ -128,7 +165,7 @@ class JoinType(StrEnum): # Built-in aggregation names (always available without model-level definition). BUILTIN_AGGREGATIONS: frozenset[str] = frozenset({ "sum", "avg", "min", "max", - "count", "count_distinct", + "count", "count_distinct", "count_distinct_approx", "first", "last", "weighted_avg", "median", "percentile", @@ -137,6 +174,59 @@ class JoinType(StrEnum): "corr", "covar_samp", "covar_pop", }) +# DEV-1576: unambiguous aggregation-name aliases that LLM agents routinely +# emit. ``normalize_aggregation_name`` lowercases the incoming token and maps +# it through this table; the result is only adopted when it lands in +# ``BUILTIN_AGGREGATIONS``. ``stddev``/``var``/``variance`` map to the *sample* +# variants, matching Postgres' bare ``stddev``/``variance`` defaults. +AGGREGATION_ALIASES: dict[str, str] = { + "countd": "count_distinct", + "countdistinct": "count_distinct", # also matches "countDistinct" once lowercased + # DEV-1595: approximate-distinct spellings agents / dbt-to-cube emit. + "approx_count_distinct": "count_distinct_approx", + "countdistinctapprox": "count_distinct_approx", # matches "countDistinctApprox" lowercased + "stddev": "stddev_samp", + "var": "var_samp", + "variance": "var_samp", +} + + +def normalize_aggregation_name(name: str) -> str: + """Coerce an aggregation token to its canonical SLayer spelling. + + Lowercases the token and applies :data:`AGGREGATION_ALIASES`. The + normalized form is only adopted when it is a real built-in aggregation; + otherwise the **original** string is returned unchanged so genuinely + unknown names still raise downstream (and custom aggregation names keep + their exact casing). Examples:: + + "countd" -> "count_distinct" + "countDistinct" -> "count_distinct" + "stddev" -> "stddev_samp" + "SUM" -> "sum" + "myCustomAgg" -> "myCustomAgg" (unchanged — not a builtin) + "bogus" -> "bogus" (unchanged — still raises later) + """ + lowered = name.lower() + candidate = AGGREGATION_ALIASES.get(lowered, lowered) + return candidate if candidate in BUILTIN_AGGREGATIONS else name + + +def format_unknown_aggregation(name: str, known: "set[str] | frozenset[str]") -> str: + """DEV-1576: the shared 'Unknown aggregation' error message. + + Used by both the ``enrich_query`` gate (``slayer/engine/enrichment.py``) + and the typed binding gate (``slayer/engine/binding.py``) so the wording + stays byte-identical: an unknown aggregation name is distinguished from a + known-but-disallowed one, with a close-match suggestion and the model-wide + known list. ``known`` = ``BUILTIN_AGGREGATIONS`` unioned with the owning + model's custom aggregation names. + """ + suggestion = difflib.get_close_matches(word=name, possibilities=sorted(known), n=1) + hint = f" Did you mean '{suggestion[0]}'?" if suggestion else "" + return f"Unknown aggregation '{name}'.{hint} Known: {sorted(known)}." + + # Built-in aggregation SQL formulas (for aggregations that use a template). # {value} = measure's SQL expression; {param_name} = parameter values. # Note: percentile is dialect-dependent (no single template works on @@ -171,7 +261,7 @@ class JoinType(StrEnum): # to gate ``column:agg`` expressions (e.g., ``revenue:sum`` requires ``sum`` to # be eligible for the ``revenue`` column's data type). _NUMERIC_AGGREGATIONS: frozenset[str] = frozenset({ - "sum", "avg", "min", "max", "count", "count_distinct", + "sum", "avg", "min", "max", "count", "count_distinct", "count_distinct_approx", "median", "weighted_avg", "percentile", "first", "last", "stddev_samp", "stddev_pop", "var_samp", "var_pop", "corr", "covar_samp", "covar_pop", @@ -183,21 +273,21 @@ class JoinType(StrEnum): DataType.INT: _NUMERIC_AGGREGATIONS, DataType.DOUBLE: _NUMERIC_AGGREGATIONS, DataType.TEXT: frozenset({ - "count", "count_distinct", "first", "last", "min", "max", + "count", "count_distinct", "count_distinct_approx", "first", "last", "min", "max", }), DataType.BOOLEAN: frozenset({ - "count", "count_distinct", "sum", "min", "max", "first", "last", + "count", "count_distinct", "count_distinct_approx", "sum", "min", "max", "first", "last", }), DataType.DATE: frozenset({ - "count", "count_distinct", "first", "last", "min", "max", + "count", "count_distinct", "count_distinct_approx", "first", "last", "min", "max", }), DataType.TIMESTAMP: frozenset({ - "count", "count_distinct", "first", "last", "min", "max", + "count", "count_distinct", "count_distinct_approx", "first", "last", "min", "max", }), } # Primary-key columns are always restricted to row-counting aggregations, # regardless of data type. (You can ``count`` customer_ids, but not ``sum`` them.) PRIMARY_KEY_AGGREGATIONS: frozenset[str] = frozenset({ - "count", "count_distinct", + "count", "count_distinct", "count_distinct_approx", }) diff --git a/slayer/core/errors.py b/slayer/core/errors.py index 4a7cfb92..9882bd24 100644 --- a/slayer/core/errors.py +++ b/slayer/core/errors.py @@ -35,7 +35,7 @@ class AmbiguousModelError(SlayerError): priority`` CLI subcommand, etc.). """ - def __init__(self, name: str, candidates: List[str]) -> None: + def __init__(self, name: str, candidates: list[str]) -> None: self.name = name self.candidates = list(candidates) super().__init__( @@ -85,8 +85,8 @@ class SchemaDriftError(SlayerError): def __init__( self, - models: List[str], - to_delete: List[Any], + models: list[str], + to_delete: list[Any], original: BaseException, ) -> None: self.models = list(models) @@ -110,8 +110,8 @@ class ColumnCycleError(SlayerError, ValueError): compile-time cycle raise) continue to work unchanged. """ - def __init__(self, cycle: List[Tuple[str, str]]) -> None: - self.cycle: List[Tuple[str, str]] = list(cycle) + def __init__(self, cycle: list[tuple[str, str]]) -> None: + self.cycle: list[tuple[str, str]] = list(cycle) chain = " → ".join(f"{m}.{c}" for m, c in self.cycle) super().__init__(f"Circular column reference detected: {chain}") @@ -418,3 +418,102 @@ def __init__(self, filter_text: str, reason: str) -> None: f"Filter {filter_text!r} dropped from cross-model CTE " f"(unreachable from CTE root): {reason}" ) + + +class IdCollisionError(SlayerError, ValueError): + """Raised by filename-backed (YAML) storage when saving an entity + whose id differs from an existing id only by letter case — such ids + collide as filenames on case-insensitive filesystems. ``kind`` is + ``"model"`` / ``"datasource"`` / ``"memory"``. Multi-inherits + ``ValueError`` so existing ``except ValueError`` call sites continue + to work unchanged. + """ + + _LABELS = { + "model": "Model name", + "datasource": "Datasource name", + "memory": "Memory id", + } + + def __init__( + self, + *, + kind: str, + new_id: str, + existing_id: str, + data_source: str | None = None, + ) -> None: + self.kind = kind + self.new_id = new_id + self.existing_id = existing_id + self.data_source = data_source + label = self._LABELS.get(kind, "Id") + scope = f" in datasource '{data_source}'" if data_source else "" + super().__init__( + f"{label} '{new_id}' conflicts with existing '{existing_id}'" + f"{scope} (differs only by case). Rename or delete one." + ) + + +class ForcedFilterError(SlayerError): + """Raised when the session policy's ruleset cannot be safely applied to a query. + + Carries the offending ``table`` and ``column`` for diagnostics; either may be + ``None`` (``column`` is, for the unlisted-table and statement-root guards). + """ + + def __init__( + self, + message: str, + *, + table: str | None = None, + column: str | None = None, + ) -> None: + self.table = table + self.column = column + super().__init__(message) + + +class DistinctDimensionValuesError(SlayerError, ValueError): + """Raised when ``distinct_dimension_values=False`` conflicts with the + query shape (DEV-1543). + + ``distinct_dimension_values=False`` asks for raw rows — no top-level + ``GROUP BY``. It is incompatible with any aggregation: a non-empty + ``measures`` list, a filter / order item referencing a measure + (colon-form ``col:agg`` / ``*:count``, a transform call like + ``rank(...)``, or a bare saved ``ModelMeasure`` name), or a query + with no projected columns at all (both ``dimensions`` and + ``time_dimensions`` empty). + + Multi-inherits ``ValueError`` so existing ``except ValueError`` + call sites continue to work unchanged. + """ + + +class UnresolvableOrderColumnError(SlayerError, ValueError): + """Raised when an ``order`` item references a column that cannot be bound + to the query's FROM scope (DEV-1645). + + Fires when the sort key is neither a projected output alias, a base-model + column, nor a column on a join that the query already pulled into scope + (via a dimension, measure, or filter). The common case is ordering by an + *unprojected joined column* — e.g. ``order=[{"column": + "customers.regions.name"}]`` — whose join was never resolved, so there is + no in-scope table to qualify against. Emitting a reference anyway would + produce SQL that fails at the database with UndefinedTable/UndefinedColumn; + rejecting at compile time surfaces an actionable error instead. + + Multi-inherits ``ValueError`` so existing ``except ValueError`` call sites + continue to work unchanged. + """ + + def __init__(self, *, column: str, qualifier: str) -> None: + self.column = column + self.qualifier = qualifier + super().__init__( + f"ORDER BY column '{qualifier}.{column}' cannot be resolved: it is not a " + f"projected field, a base column, or a column on a join that is in scope. " + f"Project it (add to dimensions/measures), reference it in a filter, or " + f"order by a projected field instead." + ) diff --git a/slayer/core/format.py b/slayer/core/format.py index 520884fd..0938a650 100644 --- a/slayer/core/format.py +++ b/slayer/core/format.py @@ -4,7 +4,6 @@ import math import numbers from enum import Enum -from typing import Optional from pydantic import BaseModel, Field, model_validator @@ -28,12 +27,12 @@ class NumberFormat(BaseModel): default=NumberFormatType.FLOAT, description="The format type for number display", ) - precision: Optional[int] = Field( + precision: int | None = Field( default=None, ge=0, description="Number of decimal places to show", ) - symbol: Optional[str] = Field( + symbol: str | None = Field( default=None, description="Currency symbol (defaults to $ for CURRENCY type, must be None otherwise)", ) @@ -51,8 +50,8 @@ def validate_symbol(self) -> "NumberFormat": def _format_with_notation( value: float, default_precision: int, - explicit_precision: Optional[int] = None, - max_precision: Optional[int] = None, + explicit_precision: int | None = None, + max_precision: int | None = None, ) -> tuple[float, str, int]: """Core formatting logic with K/M notation and dynamic precision calculation. diff --git a/slayer/core/formula.py b/slayer/core/formula.py index d54c94fe..d41dd5a2 100644 --- a/slayer/core/formula.py +++ b/slayer/core/formula.py @@ -17,11 +17,12 @@ import re import tokenize import warnings -from typing import Any, Dict, List, Literal, Mapping, Optional, Union +from typing import Any, Literal +from collections.abc import Mapping from pydantic import BaseModel, Field -from slayer.core.enums import BUILTIN_AGGREGATIONS +from slayer.core.enums import BUILTIN_AGGREGATIONS, normalize_aggregation_name from slayer.core.refs import ( AGG_REF_RE as _AGG_REF_RE, IDENT_OR_PATH_RE as _IDENT_OR_PATH_RE, @@ -52,45 +53,90 @@ ALL_TRANSFORMS = TIME_TRANSFORMS | TIMELESS_TRANSFORMS -# DEV-1378: string-hygiene scalar functions accepted inline in Mode B -# (DSL) filters. These are pass-through to the emitted SQL; sqlglot -# handles per-dialect spelling at SQL-generation time. Names are -# lowercase only (matching SLayer's existing transform convention — -# ``cumsum``, ``rank``, ``time_shift``). The SQL ``||`` concat operator -# is rewritten to ``concat(...)`` by ``_preprocess_concat`` before AST -# parsing. -STRING_HYGIENE_OPS = frozenset({ - "lower", - "upper", - "trim", - "replace", - "substr", - "instr", - "length", - "concat", +# Canonical Mode B scalar-function allowlist. Consulted uniformly by every +# Mode B surface — top-level ``ModelMeasure.formula`` calls, inside-arithmetic +# calls, and ``SlayerQuery.filters`` — case-insensitively. Pass-through to +# emitted SQL; sqlglot handles per-dialect spelling. Extending: add to this +# set only — never create parallel allowlists. +SCALAR_PASSTHROUGH = frozenset({ + # NULL handling + "coalesce", "nullif", "ifnull", + # Math + "round", "abs", "ceil", "ceiling", "floor", + "power", "pow", "sqrt", "exp", + "ln", "log", "log10", "log2", + "mod", "sign", "trunc", + # Min/max scalar (NOT the agg forms — those are min:/max:) + "greatest", "least", + # String + "lower", "upper", "trim", "ltrim", "rtrim", + "replace", "substr", "substring", + "instr", "length", "concat", }) -CallCategory = Literal["transform", "hygiene", "like_internal", "unknown"] +CallCategory = Literal["transform", "scalar", "like_internal", "unknown"] _LIKE_INTERNAL_NAMES = frozenset({"__like__", "__notlike__"}) def _classify_call_name(name: str) -> CallCategory: - """Categorize a function-call identifier for the formula and filter walkers. - - Single source of truth shared by ``_parse_node`` (formula → FieldSpec) and - ``_call_to_sql`` (filter → SQL string). Each walker still decides what - to do with the category; this helper just classifies the name. - """ + """Categorize a Mode B function-call identifier — shared by the formula + and filter walkers.""" if name in ALL_TRANSFORMS: return "transform" - if name in STRING_HYGIENE_OPS: - return "hygiene" + if name.lower() in SCALAR_PASSTHROUGH: + return "scalar" if name in _LIKE_INTERNAL_NAMES: return "like_internal" return "unknown" +def _validate_scalar_call(node: ast.Call, original: str) -> None: + """Arity check for top-level scalar calls. ``round`` / ``abs`` have specific + shapes worth catching early; other scalars in ``SCALAR_PASSTHROUGH`` have + variable arity (``coalesce``, ``greatest``, …) and are validated by sqlglot + at SQL-emission time.""" + name = node.func.id.lower() + if node.keywords: + raise ValueError( + f"'{name}' does not accept keyword arguments. Formula: {original!r}" + ) + if name == "abs": + if len(node.args) != 1: + raise ValueError( + f"'abs' requires exactly one argument. Formula: {original!r}" + ) + elif name == "round": + if len(node.args) not in (1, 2): + raise ValueError( + f"'round' accepts 1 or 2 arguments (expression[, ndigits]). " + f"Formula: {original!r}" + ) + if len(node.args) == 2 and not _is_integer_literal(node.args[1]): + raise ValueError( + f"'round' ndigits (2nd argument) must be an integer literal. " + f"Formula: {original!r}" + ) + + +def _is_integer_literal(node: ast.AST) -> bool: + """True for an integer constant or a negated integer constant (``-2``). + + Booleans are rejected — ``True``/``False`` are ``int`` subclasses but are + not meaningful ndigits values. + """ + if isinstance(node, ast.Constant): + return isinstance(node.value, int) and not isinstance(node.value, bool) + if ( + isinstance(node, ast.UnaryOp) + and isinstance(node.op, ast.USub) + and isinstance(node.operand, ast.Constant) + ): + v = node.operand.value + return isinstance(v, int) and not isinstance(v, bool) + return False + + class AggregatedMeasureRef(BaseModel): """A measure reference with explicit aggregation (new colon syntax). @@ -105,15 +151,15 @@ class AggregatedMeasureRef(BaseModel): """ measure_name: str = Field(description="Measure name, e.g. 'revenue', 'customers.revenue', '*'") aggregation_name: str = Field(description="Aggregation name, e.g. 'sum', 'weighted_avg'") - agg_args: List[str] = Field(default_factory=list, description="Positional aggregation args") - agg_kwargs: Dict[str, str] = Field(default_factory=dict, description="Keyword aggregation args") + agg_args: list[str] = Field(default_factory=list, description="Positional aggregation args") + agg_kwargs: dict[str, str] = Field(default_factory=dict, description="Keyword aggregation args") class ArithmeticField(BaseModel): """An arithmetic expression over measures only (no transform calls inside).""" sql: str = Field(description="Preprocessed formula with placeholders for aggregated refs") - measure_names: List[str] = Field(description="Placeholder IDs or bare measure names") - agg_refs: Dict[str, AggregatedMeasureRef] = Field(default_factory=dict) + measure_names: list[str] = Field(description="Placeholder IDs or bare measure names") + agg_refs: dict[str, AggregatedMeasureRef] = Field(default_factory=dict) is_predicate: bool = Field( default=False, description="True when the top-level AST node is a comparison or boolean op " @@ -126,8 +172,8 @@ class TransformField(BaseModel): """A transform function call, possibly wrapping another transform or arithmetic.""" transform: str = Field(description="Transform name: cumsum, lag, lead, change, change_pct, rank, percent_rank, dense_rank, ntile, time_shift, first, last, consecutive_periods") inner: "FieldSpec" = Field(description="The measure or expression being transformed") - args: List[Any] = Field(default_factory=list, description="Extra transform args (offset, granularity, etc.)") - kwargs: Dict[str, Any] = Field( + args: list[Any] = Field(default_factory=list, description="Extra transform args (offset, granularity, etc.)") + kwargs: dict[str, Any] = Field( default_factory=dict, description=( "Keyword args from the call site, e.g. partition_by=[...] for the " @@ -143,9 +189,9 @@ class MixedArithmeticField(BaseModel): as a CTE step, then the arithmetic references its result. """ sql: str = Field(description="Preprocessed formula with placeholders") - measure_names: List[str] = Field(description="Placeholder IDs or bare measure names") - sub_transforms: List[tuple] = Field(description="List of (placeholder_name, TransformField)") - agg_refs: Dict[str, AggregatedMeasureRef] = Field(default_factory=dict) + measure_names: list[str] = Field(description="Placeholder IDs or bare measure names") + sub_transforms: list[tuple] = Field(description="List of (placeholder_name, TransformField)") + agg_refs: dict[str, AggregatedMeasureRef] = Field(default_factory=dict) is_predicate: bool = Field( default=False, description="True when the top-level AST node is a comparison or boolean op " @@ -155,7 +201,7 @@ class MixedArithmeticField(BaseModel): # The parsed result of a single field -FieldSpec = Union[AggregatedMeasureRef, ArithmeticField, TransformField, MixedArithmeticField] +FieldSpec = AggregatedMeasureRef | ArithmeticField | TransformField | MixedArithmeticField # Rebuild TransformField which uses a forward reference to FieldSpec TransformField.model_rebuild() @@ -198,7 +244,7 @@ def _find_balanced_close(s: str, start: int) -> int: def _rewrite_funcstyle_aggregations( formula: str, - extra_agg_names: Optional[frozenset[str]] = None, + extra_agg_names: frozenset[str] | None = None, ) -> str: """Rewrite function-style aggregation calls to colon syntax. @@ -344,31 +390,66 @@ def _split_args(s: str) -> list[str]: # --------------------------------------------------------------------------- -def _preprocess_agg_refs(formula: str) -> tuple[str, Dict[str, AggregatedMeasureRef]]: +def _split_agg_arglist(args_str: str | None) -> tuple[list[str], dict[str, str]]: + """Parse a colon-aggregation ``(...)`` arglist into (positional, keyword). + + ``args_str`` is the raw ``(...)`` capture (with parens) or ``None``. + ``price:weighted_avg(weight=quantity)`` → ``([], {"weight": "quantity"})``; + ``revenue:last(ordered_at)`` → ``(["ordered_at"], {})``. + """ + agg_args: list[str] = [] + agg_kwargs: dict[str, str] = {} + if not args_str: + return agg_args, agg_kwargs + inner = args_str[1:-1].strip() + if not inner: + return agg_args, agg_kwargs + for part in inner.split(","): + part = part.strip() + if "=" in part: + key, val = part.split("=", 1) + agg_kwargs[key.strip()] = val.strip() + else: + agg_args.append(part) + return agg_args, agg_kwargs + + +def _preprocess_agg_refs( + formula: str, + custom_agg_names: frozenset[str] = frozenset(), +) -> tuple[str, dict[str, AggregatedMeasureRef]]: """Replace colon-syntax aggregated measure refs with placeholder identifiers. Returns (preprocessed_formula, {placeholder: AggregatedMeasureRef}). + + ``custom_agg_names`` are the reachable custom-aggregation names (the source + model plus its join graph); a colon token matching one exactly is left + un-normalized so a custom aggregation takes precedence over alias/casing + healing (DEV-1576). This is parse-time and model-set-scoped, not per-ref + target-scoped: in a mixed-model graph a custom name on one model suppresses + healing of that same token on a different model. The failure mode is benign + — the un-healed token simply resolves (custom agg) or raises an explicit + "Unknown aggregation" at enrichment; it never silently picks the wrong + aggregation. Per-ref scoping would require deferring the heal past parse, + which would desync the canonical filter/ORDER-BY aliases built here. """ - refs: Dict[str, AggregatedMeasureRef] = {} + refs: dict[str, AggregatedMeasureRef] = {} counter = [0] def _replace(match: re.Match) -> str: measure_name = match.group(1) - agg_name = match.group(2) - args_str = match.group(3) - - agg_args: list[str] = [] - agg_kwargs: dict[str, str] = {} - if args_str: - inner = args_str[1:-1].strip() - if inner: - for part in inner.split(","): - part = part.strip() - if "=" in part: - key, val = part.split("=", 1) - agg_kwargs[key.strip()] = val.strip() - else: - agg_args.append(part) + # DEV-1576: heal aggregation-name aliases / casing at the single colon- + # syntax chokepoint (shared by parse_formula and parse_filter). Unknown + # names pass through unchanged so the §3 enrichment error still fires. + # A model-level custom aggregation named like an alias key / builtin + # casing wins — skip healing for an exact custom-name match so it still + # resolves at enrichment. + raw_agg = match.group(2) + agg_name = ( + raw_agg if raw_agg in custom_agg_names + else normalize_aggregation_name(raw_agg) + ) + agg_args, agg_kwargs = _split_agg_arglist(match.group(3)) placeholder = f"__agg{counter[0]}__" counter[0] += 1 @@ -426,7 +507,7 @@ def _expand_named_measures( except (tokenize.TokenError, IndentationError, SyntaxError): return formula - def _significant(idx: int, step: int) -> Optional[tokenize.TokenInfo]: + def _significant(idx: int, step: int) -> tokenize.TokenInfo | None: skip_types = {tokenize.NEWLINE, tokenize.NL, tokenize.ENCODING, tokenize.ENDMARKER, tokenize.COMMENT, tokenize.INDENT, tokenize.DEDENT} @@ -439,7 +520,7 @@ def _significant(idx: int, step: int) -> Optional[tokenize.TokenInfo]: return t return None - replacements: List[tuple] = [] # (start_pos, end_pos, replacement_text) + replacements: list[tuple] = [] # (start_pos, end_pos, replacement_text) for i, tok in enumerate(tokens): if tok.type != tokenize.NAME or tok.string not in named_measures: continue @@ -481,8 +562,8 @@ def _significant(idx: int, step: int) -> Optional[tokenize.TokenInfo]: def parse_formula( formula: str, - extra_agg_names: Optional[frozenset[str]] = None, - named_measures: Optional[Mapping[str, str]] = None, + extra_agg_names: frozenset[str] | None = None, + named_measures: Mapping[str, str] | None = None, ) -> FieldSpec: """Parse a formula string into a FieldSpec. @@ -510,7 +591,9 @@ def parse_formula( # Rewrite function-style aggregations (e.g., sum(revenue) → revenue:sum) formula = _rewrite_funcstyle_aggregations(formula, extra_agg_names) # Preprocess colon syntax into ast-parseable placeholders - processed, agg_refs = _preprocess_agg_refs(formula) + processed, agg_refs = _preprocess_agg_refs( + formula=formula, custom_agg_names=extra_agg_names or frozenset() + ) try: tree = ast.parse(processed, mode="eval") @@ -523,7 +606,7 @@ def parse_formula( def _parse_node( node: ast.AST, original: str, - agg_refs: Optional[Dict[str, AggregatedMeasureRef]] = None, + agg_refs: dict[str, AggregatedMeasureRef] | None = None, ) -> FieldSpec: """Recursively parse an AST node into a FieldSpec.""" if agg_refs is None: @@ -554,10 +637,19 @@ def _parse_node( raise ValueError(f"Unsupported function call in formula: {original!r}") func_name = node.func.id - if _classify_call_name(func_name) != "transform": + category = _classify_call_name(func_name) + if category == "scalar": + # Pass-through SCALAR_PASSTHROUGH call. Route through + # _parse_mixed_arithmetic so inner aggregated refs and nested + # transforms are registered/extracted. + _validate_scalar_call(node, original) + return _parse_mixed_arithmetic(node, original, agg_refs) + if category != "transform": raise ValueError( - f"Unknown transform function '{func_name}'. " - f"Supported: {', '.join(sorted(ALL_TRANSFORMS))}" + f"Unknown function '{func_name}' in formula {original!r}. " + f"Supported scalar functions: " + f"{', '.join(sorted(SCALAR_PASSTHROUGH))}. " + f"Transforms: {', '.join(sorted(ALL_TRANSFORMS))}." ) if not node.args: @@ -634,7 +726,7 @@ def _replace_calls_in_arith( sub_transforms: list[tuple], measure_names: list[str], counter: list[int], - agg_refs: Dict[str, AggregatedMeasureRef], + agg_refs: dict[str, AggregatedMeasureRef], original: str, ) -> ast.AST: """Walk the AST, replacing transform Call nodes with Name placeholders.""" @@ -677,9 +769,20 @@ def _replace_calls_in_arith( return node if isinstance(node, ast.Call): - # Non-transform call (e.g. nullif, coalesce) wrapping aggregated refs. - # Recurse into args/keywords so any __aggN__ placeholders inside get - # registered in measure_names; otherwise they leak to emitted SQL. + # Non-transform scalar call wrapping aggregated refs (e.g. nullif, + # coalesce, abs). Validate the name against the canonical Mode B + # allowlist — previously any name passed through, which was a + # consistency bug (top-level had a strict allowlist while + # inside-arithmetic accepted anything). + if isinstance(node.func, ast.Name): + name = node.func.id + if name not in ALL_TRANSFORMS and name.lower() not in SCALAR_PASSTHROUGH: + raise ValueError( + f"Unknown function {name!r} in formula {kwargs['original']!r}. " + f"Supported scalar functions: " + f"{', '.join(sorted(SCALAR_PASSTHROUGH))}. " + f"Transforms: {', '.join(sorted(ALL_TRANSFORMS))}." + ) node.args = [_replace_calls_in_arith(a, **kwargs) for a in node.args] for kw in node.keywords: kw.value = _replace_calls_in_arith(kw.value, **kwargs) @@ -691,7 +794,7 @@ def _replace_calls_in_arith( def _parse_mixed_arithmetic( node: ast.AST, original: str, - agg_refs: Optional[Dict[str, AggregatedMeasureRef]] = None, + agg_refs: dict[str, AggregatedMeasureRef] | None = None, ) -> MixedArithmeticField: """Parse arithmetic that contains transform calls. @@ -737,7 +840,7 @@ def _parse_literal(node: ast.AST, original: str) -> Any: # Per-transform kwarg whitelist. Empty set means the transform takes no kwargs. -_ALLOWED_TRANSFORM_KWARGS: Dict[str, frozenset] = { +_ALLOWED_TRANSFORM_KWARGS: dict[str, frozenset] = { "rank": frozenset({"partition_by"}), "percent_rank": frozenset({"partition_by"}), "dense_rank": frozenset({"partition_by"}), @@ -761,8 +864,8 @@ def _parse_dotted_name(node: ast.AST, original: str) -> str: def _parse_transform_kwargs( # NOSONAR S3776 — straight-line whitelist + per-kwarg validation; splitting into helpers would force threading transform/original through every call just to preserve the error-message context - transform: str, keywords: List[ast.keyword], original: str -) -> Dict[str, Any]: + transform: str, keywords: list[ast.keyword], original: str +) -> dict[str, Any]: """Parse and validate a transform's keyword arguments. Each transform has a fixed kwarg whitelist (see ``_ALLOWED_TRANSFORM_KWARGS``). @@ -771,7 +874,7 @@ def _parse_transform_kwargs( # NOSONAR S3776 — straight-line whitelist + per- path, or list of those. """ allowed = _ALLOWED_TRANSFORM_KWARGS.get(transform, frozenset()) - parsed: Dict[str, Any] = {} + parsed: dict[str, Any] = {} for kw in keywords: if kw.arg is None: @@ -839,10 +942,10 @@ class ParsedFilter(BaseModel): as-is (they get qualified with the model name during SQL generation). """ sql: str = Field(description="SQL WHERE condition, e.g. \"status = 'completed'\"") - columns: List[str] = Field(description="Column names referenced in the filter") + columns: list[str] = Field(description="Column names referenced in the filter") is_having: bool = Field(default=False, description="True if this is a HAVING filter (aggregate condition)") is_post_filter: bool = Field(default=False, description="True if this references a computed column (transform/expression)") - synthesized_aliases: List[str] = Field( + synthesized_aliases: list[str] = Field( default_factory=list, description=( "Canonical aggregation aliases this filter introduced from " @@ -852,7 +955,7 @@ class ParsedFilter(BaseModel): "regex that would let typos like ``made_up_sum`` through." ), ) - agg_refs: List["AggregatedMeasureRef"] = Field( + agg_refs: list["AggregatedMeasureRef"] = Field( default_factory=list, description=( "Aggregated measure references extracted from colon syntax in " @@ -866,11 +969,10 @@ class ParsedFilter(BaseModel): # LHS of a LIKE / NOT LIKE may be a bare/dotted identifier OR a single -# hygiene-call (e.g. ``lower(name)``, ``trim(customers.email)``). The -# call alternative is matched before the dotted-identifier alternative -# so that ``lower(name) like 'a%'`` resolves to the call form. -# ``[^()]*`` keeps this to one level of parens — nested hygiene calls -# inside LIKE (e.g. ``concat(lower(a), b) like '%'``) still fall through. +# scalar call (e.g. ``lower(name)``, ``trim(customers.email)``). The call +# alternative is matched first so ``lower(name) like 'a%'`` resolves to +# the call form. ``[^()]*`` keeps this to one level of parens — nested +# scalar calls (``concat(lower(a), b) like '%'``) still fall through. _LIKE_RE = re.compile( r"\b(\w+\([^()]*\)|(?:\w+\.)*\w+)\s+(not\s+)?like\s+('[^']*')", flags=re.IGNORECASE, @@ -930,7 +1032,7 @@ def _preprocess_concat(formula: str) -> str: parts = _STRING_LITERAL_RE.split(formula) literals = _STRING_LITERAL_RE.findall(formula) rewritten_parts = [p.replace("||", "<<") for p in parts] - out: List[str] = [] + out: list[str] = [] for i, p in enumerate(rewritten_parts): out.append(p) if i < len(literals): @@ -967,8 +1069,8 @@ def _preprocess_sql_operators(formula: str) -> str: def parse_filter( formula: str, - extra_agg_names: Optional[frozenset[str]] = None, - named_measures: Optional[Mapping[str, str]] = None, + extra_agg_names: frozenset[str] | None = None, + named_measures: Mapping[str, str] | None = None, ) -> ParsedFilter: """Parse a Mode B (DSL) filter formula into a ParsedFilter. @@ -982,8 +1084,8 @@ def parse_filter( - ``LIKE`` / ``NOT LIKE``, ``IS NULL`` / ``IS NOT NULL`` Rejects unknown function calls (the Python AST walker raises on any - call other than the internal ``__like__`` / ``__notlike__`` helpers - and a small allowlist of string-hygiene scalar operators). + call outside ``SCALAR_PASSTHROUGH`` and the internal ``__like__`` / + ``__notlike__`` helpers). Pre-rejects raw ``OVER (...)`` window-function syntax via :func:`has_window_function` (the rank-family transforms cover the @@ -1035,7 +1137,9 @@ def parse_filter( # Include agg args/kwargs in the canonical name so e.g. # ``revenue:sum(window='90d') > 100`` matches the windowed measure's alias # ``orders.revenue_sum_window_90d`` and not the bare ``orders.revenue_sum``. - processed, agg_refs = _preprocess_agg_refs(processed) + processed, agg_refs = _preprocess_agg_refs( + formula=processed, custom_agg_names=extra_agg_names or frozenset() + ) agg_canonical = { ph: canonical_agg_name( measure_name=ref.measure_name, @@ -1064,12 +1168,30 @@ def parse_filter( ) -_BINOP_OP_MAP: Dict[type, str] = { +_BINOP_OP_MAP: dict[type, str] = { ast.Add: "+", ast.Sub: "-", ast.Mult: "*", ast.Div: "/", ast.Mod: "%", ast.Pow: "**", } +# DEV-1539: SQL-precedence tier for each supported ``ast.BinOp`` op. +# Higher = tighter binding. Used by ``_binop_to_sql`` to decide whether +# a child BinOp's operands need parenthesising so the tree-encoded +# precedence survives serialisation. Only left-associative arithmetic +# ops live here: ``ast.Pow`` is intentionally absent because it is +# right-associative — its equal-precedence rule is the mirror of the +# others (wrap on LEFT, not right) and the simplest way to stay correct +# is to fall through to the ``parent_prec is None`` fallback in +# ``_emit_binop_operand`` (wrap every BinOp child unconditionally). +# That adds at most one harmless paren on mixed-precedence Pow +# expressions and guarantees ``(a ** b) ** c`` doesn't silently +# re-associate to ``a ** (b ** c)``. +_BINOP_PRECEDENCE: dict[type, int] = { + ast.Mult: 2, ast.Div: 2, ast.Mod: 2, + ast.Add: 1, ast.Sub: 1, +} + + def _resolve_dotted_attribute(node: ast.expr) -> str: """Render an ``ast.Attribute`` (or ``ast.Name`` leaf) as a dotted string.""" if isinstance(node, ast.Name): @@ -1080,16 +1202,57 @@ def _resolve_dotted_attribute(node: ast.expr) -> str: def _compare_to_sql(node: ast.Compare, recur) -> str: - parts = [recur(node.left)] + # DEV-1539: chained comparisons (``a < b < c``) have different + # semantics in Python (``(a < b) AND (b < c)``) and SQL (left-to-right + # ``(a < b) < c``, a boolean re-compared with ``c``). Reject up front + # with a pointer at the ``AND`` rewrite rather than emit silently + # wrong SQL. + if len(node.ops) > 1: + raise ValueError( + "Chained comparisons (e.g. `a < b < c`) are not supported in " + "DSL filters because their Python semantics differ from SQL. " + "Rewrite using AND: `a < b AND b < c`." + ) + # DEV-1539: wrap a Compare LHS/RHS that is ``ast.BinOp`` in outer + # parens so the comparator's precedence is explicit in the emitted + # SQL. ``ast.BoolOp`` is intentionally NOT included — ``_boolop_to_sql`` + # already self-wraps multi-operand outputs in ``(...)`` and a second + # layer would add noise. ``ast.LShift`` is also excluded — it is a + # marker for the SQL ``||`` concat operator (pre-processed by + # ``_preprocess_concat``) and ``_binop_to_sql`` rewrites the chain + # into a single ``concat(...)`` function call, which doesn't need + # an outer paren. + def _needs_wrap(n: ast.AST) -> bool: + return isinstance(n, ast.BinOp) and not isinstance(n.op, ast.LShift) + + left_sql = recur(node.left) + if _needs_wrap(node.left): + left_sql = f"({left_sql})" + parts = [left_sql] for op, comparator in zip(node.ops, node.comparators): + # ``is None`` / ``is not None`` map to ``IS NULL`` / ``IS NOT NULL`` + # — ``_compare_op_to_sql`` already returns the complete operator + # string and there is no RHS to render. Every other ``is`` / + # ``is not`` (e.g. ``flag is True``) falls through to the + # standard ``IS `` / ``IS NOT `` emission; without + # this fall-through ``flag is True`` previously serialised as + # the broken ``flag IS`` (no RHS). + is_null_check = ( + isinstance(op, (ast.Is, ast.IsNot)) + and isinstance(comparator, ast.Constant) + and comparator.value is None + ) sql_op = _compare_op_to_sql(op, comparator) - right = recur(comparator) - if isinstance(op, (ast.Is, ast.IsNot)): - parts.append(sql_op) # "IS NULL" / "IS NOT NULL" already complete - else: - # Both regular comparisons and IN / NOT IN take " "; - # for IN/NotIn the right is already "(val1, val2, ...)". - parts.append(f"{sql_op} {right}") + if is_null_check: + parts.append(sql_op) + continue + right_sql = recur(comparator) + if _needs_wrap(comparator): + right_sql = f"({right_sql})" + # Regular comparisons, IN / NOT IN, and IS / IS NOT with + # non-None RHS all take " "; for IN/NotIn the right + # is already "(val1, val2, ...)". + parts.append(f"{sql_op} {right_sql}") return " ".join(parts) @@ -1133,7 +1296,7 @@ def _seq_to_sql(node, recur) -> str: return f"({', '.join(elts)})" -def _flatten_lshift_chain(node: ast.AST, recur) -> List[str]: +def _flatten_lshift_chain(node: ast.AST, recur) -> list[str]: """Flatten a chain of ``ast.BinOp(LShift)`` nodes into a flat list of SQL strings. Used by ``_binop_to_sql`` to fold a ``a || b || c`` chain (which arrives as left-associative LShift after `_preprocess_concat`) @@ -1153,7 +1316,57 @@ def _binop_to_sql(node: ast.BinOp, original: str, recur) -> str: op_str = _BINOP_OP_MAP.get(type(node.op)) if op_str is None: raise ValueError(f"Unsupported arithmetic operator in filter: {original!r}") - return f"{recur(node.left)} {op_str} {recur(node.right)}" + # DEV-1539: precedence-aware operand wrapping. Without this, + # ``(a + b) * c`` and ``a + b * c`` both serialise to + # ``a + b * c`` — the Python AST's parens are tracked only by tree + # shape, not by an explicit node, so re-emission via plain + # ``left op right`` silently drops grouping. Wrap a child BinOp + # whose operator has *strictly lower* precedence than this op + # (preserves grouping like ``(a + b) * c``); for the right operand + # also wrap on *equal* precedence so left-to-right associativity is + # preserved (``a - (b - c)`` vs ``a - b - c``). LShift / concat + # children carry their own grouping via ``concat(...)`` and are + # already self-contained. + parent_prec = _BINOP_PRECEDENCE.get(type(node.op)) + return f"{_emit_binop_operand(node.left, parent_prec, is_right=False, recur=recur)} {op_str} {_emit_binop_operand(node.right, parent_prec, is_right=True, recur=recur)}" + + +def _emit_binop_operand( + child: ast.AST, + parent_prec: int | None, + *, + is_right: bool, + recur, +) -> str: + """Render a ``BinOp`` operand, wrapping in ``(...)`` when the child's + precedence-tier rule says it would otherwise be misread on re-parse. + + The wrap rule (left-associative ops): + + - Left operand: wrap iff child has *strictly lower* precedence than + parent (``(a + b) * c`` — parent ``*``, left child ``+``). + - Right operand: wrap iff child has *lower or equal* precedence + (``a / (b * c)`` — parent ``/``, right child ``*``, equal — must + wrap to keep left-associative grouping intact). + + Conservative fallbacks (return wrap=True): the child is a BinOp + using a non-arithmetic op we haven't registered in + ``_BINOP_PRECEDENCE``, or ``parent_prec`` is missing. + """ + sql = recur(child) + if not isinstance(child, ast.BinOp): + return sql + if isinstance(child.op, ast.LShift): + # ``concat(...)`` is self-grouped — no extra wrap needed. + return sql + child_prec = _BINOP_PRECEDENCE.get(type(child.op)) + if parent_prec is None or child_prec is None: + return f"({sql})" + if child_prec < parent_prec: + return f"({sql})" + if is_right and child_prec == parent_prec: + return f"({sql})" + return sql def _call_to_sql(node: ast.Call, original: str, recur) -> str: @@ -1164,15 +1377,12 @@ def _call_to_sql(node: ast.Call, original: str, recur) -> str: if category == "like_internal" and len(node.args) >= 2: sql_op = "LIKE" if func_name == "__like__" else "NOT LIKE" return f"{recur(node.args[0])} {sql_op} '{_get_string_arg(node.args[1], original)}'" - if category == "hygiene": - # DEV-1378: lowercase string-hygiene scalars (lower / upper / trim / - # replace / substr / instr / length / concat). Args recurse through - # the standard handler, so nested calls and literal/integer - # constants render correctly. sqlglot translates per-dialect at - # SQL-generation time. + if category == "scalar": + # SCALAR_PASSTHROUGH: pass through with the user-written casing + # preserved; sqlglot re-spells per dialect at SQL-generation time. if node.keywords: raise ValueError( - f"String-hygiene function {func_name!r} does not accept " + f"Filter scalar function {func_name!r} does not accept " f"keyword arguments: {original!r}" ) arg_sqls = [recur(a) for a in node.args] @@ -1279,7 +1489,7 @@ def _get_string_arg(node: ast.AST, original: str) -> str: raise ValueError(f"Expected a string argument in filter: {original!r}") -def _collect_names(node: ast.AST) -> List[str]: +def _collect_names(node: ast.AST) -> list[str]: """Collect all Name and dotted Attribute references from an AST subtree.""" names = [] # Collect dotted names (model.measure) first to avoid also collecting the bare Name part diff --git a/slayer/core/keys.py b/slayer/core/keys.py index df57c4ee..b07296cf 100644 --- a/slayer/core/keys.py +++ b/slayer/core/keys.py @@ -442,6 +442,99 @@ def __eq__(self, other: object) -> bool: ) +def _reroot_path_ref(ref, *, target_path: Tuple[str, ...]): + """Re-anchor a single embedded reference from the host coordinate system + into the target's local scope (DEV-1707). + + Prefix-strip with residual: if ``ref`` carries a join ``path`` that starts + with ``target_path``, drop that prefix and keep the residual hops + (``("customers", "regions")`` under target ``("customers",)`` → + ``("regions",)``; an exact match → ``()``). A ``path`` that does NOT start + with ``target_path`` — or a value with no ``path`` at all (a scalar + ``Decimal`` / ``str`` / ``bool`` / ``None``) — is returned unchanged. The + strip applies uniformly to ``ColumnKey``, ``ColumnSqlKey``, and + ``StarKey``; the non-``path`` fields (``leaf`` / ``model`` / + ``column_name``) ride along untouched via ``model_copy``. + """ + path = getattr(ref, "path", None) + if path is None: + return ref + path = tuple(path) + if path[: len(target_path)] != tuple(target_path): + return ref + residual = path[len(target_path):] + if residual == path: + return ref + return ref.model_copy(update={"path": residual}) + + +def reroot_aggregate_key( + key: "AggregateKey", *, target_path: Tuple[str, ...], +) -> "AggregateKey": + """Re-anchor EVERY embedded reference of a cross-model ``AggregateKey`` + from the host's coordinate system into its target's local scope + (DEV-1707 / DEV-1703 Stage 3). + + When a cross-model aggregate (``customers.revenue:sum``, + ``customers.amount:last(customers.signup_at)``) is rendered inside its + target-rooted CTE, its ``source``, positional ``args``, keyword ``kwargs`` + values, and — invariantly — its ``column_filter_key`` must all be + expressed relative to the target rather than the query root. This is the + single, symmetric replacement for the per-field strip logic that used to + live in ``slayer/engine/cross_model_planner.py`` (``_local_agg_formula`` / + ``_reroot_col_kwarg``) and ``slayer/sql/generator.py`` (the inline + ``_reroot_kwarg`` / ``local_args`` / ``_reroot_having`` blocks) with two + divergent semantics. + + ``target_path`` is the join path of the aggregate's source (the hops from + the query root to the target model). Semantics — prefix-strip with + residual, applied uniformly (see ``_reroot_path_ref``): + + * ``source``, each positional arg, and each kwarg VALUE whose ``path`` + starts with ``target_path`` drops that prefix (exact match → local); + * a ref whose ``path`` does not start with ``target_path`` is left + unchanged — the function is TOTAL and never raises; a genuinely + mis-pathed ref surfaces at the downstream binder / kwarg-path validator + exactly as before; + * scalar args / kwargs pass through untouched; kwarg NAMES are preserved + (and the ``AggregateKey`` validator keeps them canonically sorted); + * ``target_path == ()`` is the identity (the filtered-local case, where + the source is already host-local — the empty prefix strips zero hops). + + ``column_filter_key`` is copied UNCHANGED. Its ``canonical_sql`` and + ``referenced_join_paths`` are anchored at the OWNING MODEL of the source + column (stamped by ``slayer.engine.binding._resolve_column_filter_key`` + via ``compute_column_filter_join_paths`` with ``anchor_model`` = that + owning model). Rerooting only changes how that owner is REACHED from the + query root; it never moves the owner, so the filter's owner-relative SQL + and paths are invariant under reroot. After rerooting a filtered + cross-model aggregate the source reads local (``path == ()``) while + ``referenced_join_paths`` stays non-empty — precisely the DEV-1503 + filtered-local isolation trigger shape. + """ + target_path = tuple(target_path) + if not target_path: + return key + # Rebuild from the existing key so fields reroot does NOT own (``agg``, + # ``column_filter_key``, and any field added to ``AggregateKey`` later) + # ride through automatically rather than being silently dropped. Only + # ``source`` / ``args`` / ``kwargs`` carry rerootable paths. ``model_copy`` + # skips the ``_canonicalize_kwargs`` validator, which is a no-op here: + # reroot preserves kwarg names and order, so the input's already-canonical + # sort is unchanged (pinned by + # ``test_kwargs_canonical_sort_preserved_after_reroot``). + return key.model_copy(update={ + "source": _reroot_path_ref(key.source, target_path=target_path), + "args": tuple( + _reroot_path_ref(a, target_path=target_path) for a in key.args + ), + "kwargs": tuple( + (k, _reroot_path_ref(v, target_path=target_path)) + for k, v in key.kwargs + ), + }) + + class TransformKey(_FrozenKey): """Identity for a transform slot (window / temporal operator over a value). diff --git a/slayer/core/models.py b/slayer/core/models.py index 6382ab9f..ff565f7c 100644 --- a/slayer/core/models.py +++ b/slayer/core/models.py @@ -3,9 +3,10 @@ import logging import os import re -from typing import Annotated, Any, Dict, List, Optional +from typing import Annotated, Any, Optional from pydantic import BaseModel, BeforeValidator, Field, field_validator, model_validator +from sqlalchemy.engine import URL as _SA_URL from slayer.core.enums import ( BUILTIN_AGGREGATIONS, @@ -23,6 +24,19 @@ logger = logging.getLogger(__name__) +# Host-field normalization for the generic connection URL. ``URL.create`` +# wants a raw host (IPv6 without brackets) plus a separate port, but the +# pre-fix string branch tolerated the port being embedded in the host +# field. These two patterns split it back out. See +# ``DatasourceConfig.get_connection_string``. +# +# ``[]`` or ``[]:`` — bracketed IPv6, optional port. +_BRACKETED_HOST_RE = re.compile(r"^\[(.+)\](?::(\d+))?$") +# ``:`` — the leading class excludes ``:`` and ``[`` so +# bare IPv6 (``::1``) and bracketed IPv6 never match here; only a +# single-colon, numeric-tail host does. +_HOST_EMBEDDED_PORT_RE = re.compile(r"^([^:\[]+):(\d+)$") + class _SubstringRule: """Single source of truth for a forbidden substring inside a name. @@ -122,19 +136,28 @@ class Column(BaseModel): Replaces v1 ``Dimension`` and ``Measure`` (which were merged in v2). """ name: str - sql: Optional[str] = None + sql: str | None = None type: DataType = DataType.TEXT + db_type: str | None = Field( + default=None, + description=( + "Raw database type string (e.g. 'point', 'jsonb'), retained when " + "the declared DataType loses information. Populated by ingestion " + "for UNKNOWN (opaque) columns; None for mapped types, where the " + "declared DataType already carries everything we need." + ), + ) primary_key: bool = False - description: Optional[str] = None - label: Optional[str] = None + description: str | None = None + label: str | None = None hidden: bool = False - format: Optional[NumberFormat] = None - allowed_aggregations: Optional[List[str]] = None - filter: Optional[str] = None # Applied inside CASE WHEN at aggregation time only - meta: Optional[Dict[str, Any]] = None - sampled: Optional[str] = None # DEV-1375: cached sample-value snapshot - sampled_values: Optional[List[str]] = None # DEV-1480: structured top-N - distinct_count: Optional[int] = None # DEV-1480: true cardinality at profile time + format: NumberFormat | None = None + allowed_aggregations: list[str] | None = None + filter: str | None = None # Applied inside CASE WHEN at aggregation time only + meta: dict[str, Any] | None = None + sampled: str | None = None # DEV-1375: cached sample-value snapshot + sampled_values: list[str] | None = None # DEV-1480: structured top-N + distinct_count: int | None = None # DEV-1480: true cardinality at profile time @model_validator(mode="before") @classmethod @@ -181,11 +204,11 @@ class ModelMeasure(BaseModel): contexts; the difference is scope. """ formula: str - name: Optional[str] = None - label: Optional[str] = None - description: Optional[str] = None - type: Optional[DataType] = None - meta: Optional[Dict[str, Any]] = None + name: str | None = None + label: str | None = None + description: str | None = None + type: DataType | None = None + meta: dict[str, Any] | None = None @model_validator(mode="before") @classmethod @@ -203,7 +226,7 @@ def _coerce_legacy_type(cls, data: Any) -> Any: @field_validator("name") @classmethod - def _validate_name(cls, v: Optional[str]) -> Optional[str]: + def _validate_name(cls, v: str | None) -> str | None: if v is not None and not _NAME_PATTERN.match(v): raise ValueError( f"Invalid name '{v}': must contain only letters, digits, " @@ -213,7 +236,7 @@ def _validate_name(cls, v: Optional[str]) -> Optional[str]: @field_validator("name") @classmethod - def _reject_transform_shadowing(cls, v: Optional[str]) -> Optional[str]: + def _reject_transform_shadowing(cls, v: str | None) -> str | None: """A saved measure named after a built-in transform (``cumsum`` etc.) would shadow the transform when written as ``cumsum(...)`` in another formula. Reject these names at construction time. @@ -261,10 +284,24 @@ class Aggregation(BaseModel): For fully custom aggregations, ``formula`` is required. """ name: str - formula: Optional[str] = None # SQL template; None = use built-in formula - params: List[AggregationParam] = Field(default_factory=list) - description: Optional[str] = None - meta: Optional[Dict[str, Any]] = None + formula: str | None = None # SQL template; None = use built-in formula + params: list[AggregationParam] = Field(default_factory=list) + description: str | None = None + meta: dict[str, Any] | None = None + + @field_validator("name") + @classmethod + def _validate_name(cls, v: str) -> str: + # DEV-1567: same identifier rule as Column.name / ModelMeasure.name. + # The pg-facade catalog flattens cross-model entries with a dotted + # prefix; the local/cross-model split would misclassify a dotted + # Aggregation.name as cross-model. + if not _NAME_PATTERN.match(v): + raise ValueError( + f"Invalid name '{v}': must contain only letters, digits, " + f"and underscores, and start with a letter or underscore" + ) + return v @model_validator(mode="after") def _require_formula_for_custom(self) -> "Aggregation": @@ -347,7 +384,7 @@ class SourceModelOrigin(BaseModel): ``exclude=True`` so accidental save paths drop it cleanly. """ name: str - data_source: Optional[str] = None + data_source: str | None = None parent: Optional["SourceModelOrigin"] = None agg_column_names: frozenset[str] = Field(default_factory=frozenset) @@ -355,12 +392,17 @@ class SourceModelOrigin(BaseModel): class ModelJoin(BaseModel): """A join relationship to another model.""" target_model: str # Name of the joined model - join_pairs: List[List[str]] = Field(...) # [["source_dim", "target_dim"], ...] + join_pairs: list[list[str]] = Field(...) # [["source_dim", "target_dim"], ...] join_type: JoinType = JoinType.LEFT # LEFT (default) or INNER + # DEV-1643: optional human/agent metadata (e.g. carrying OSI relationship + # ai_context on import). Purely additive/optional — old data omits them and + # validates unchanged, so no SlayerModel schema-version bump is needed. + description: str | None = None + meta: dict[str, Any] | None = None @field_validator("join_pairs") @classmethod - def _validate_join_pairs(cls, v: List[List[str]]) -> List[List[str]]: + def _validate_join_pairs(cls, v: list[list[str]]) -> list[list[str]]: if not v: raise ValueError("join_pairs must be non-empty") for i, pair in enumerate(v): @@ -374,17 +416,17 @@ def _validate_join_pairs(cls, v: List[List[str]]) -> List[List[str]]: class SlayerModel(BaseModel): version: int = 7 name: str - sql_table: Optional[str] = None - sql: Optional[str] = None + sql_table: str | None = None + sql: str | None = None source_queries: Annotated[ - Optional[List], BeforeValidator(_coerce_source_queries) + list | None, BeforeValidator(_coerce_source_queries) ] = None # List of SlayerQuery — query-backed source mode - query_variables: Dict[str, Any] = Field(default_factory=dict) - backing_query_sql: Optional[str] = None + query_variables: dict[str, Any] = Field(default_factory=dict) + backing_query_sql: str | None = None data_source: str = "" - columns: List[Column] = Field(default_factory=list) - measures: List[ModelMeasure] = Field(default_factory=list) - aggregations: List[Aggregation] = Field(default_factory=list) + columns: list[Column] = Field(default_factory=list) + measures: list[ModelMeasure] = Field(default_factory=list) + aggregations: list[Aggregation] = Field(default_factory=list) @model_validator(mode="before") @classmethod @@ -436,20 +478,20 @@ def _require_data_source_unless_query_backed(self) -> "SlayerModel": f"model belongs to." ) return self - joins: List[ModelJoin] = Field(default_factory=list) - filters: List[str] = Field(default_factory=list) # Model-level filters (always applied) - default_time_dimension: Optional[str] = None - description: Optional[str] = None + joins: list[ModelJoin] = Field(default_factory=list) + filters: list[str] = Field(default_factory=list) # Model-level filters (always applied) + default_time_dimension: str | None = None + description: str | None = None hidden: bool = False - meta: Optional[Dict[str, Any]] = None + meta: dict[str, Any] | None = None # DEV-1449: in-memory breadcrumb for virtual stage models produced by # ``_query_as_model``. ``exclude=True`` keeps it out of YAML/SQLite # roundtrips; virtual stage models are not persisted in the first place. - source_model_origin: Optional[SourceModelOrigin] = Field(default=None, exclude=True) + source_model_origin: SourceModelOrigin | None = Field(default=None, exclude=True) @field_validator("filters") @classmethod - def _validate_filter_predicates(cls, v: List[str]) -> List[str]: + def _validate_filter_predicates(cls, v: list[str]) -> list[str]: """Validate each model filter as a SQL-mode predicate (DEV-1369). Model filters are SQL snippets: joined column references use the @@ -508,6 +550,9 @@ def _validate_allowed_aggregations(self) -> "SlayerModel": A whitelist entry is accepted iff: + 0. **Opaque rule** — the column's type is not opaque. An opaque + (``UNKNOWN``) column cannot be aggregated at all, so declaring a + whitelist on one is always a mistake. 1. It is a known aggregation name (built-in or custom on this model). 2. **PK rule** — if the column is a primary key, the entry must be in ``PRIMARY_KEY_AGGREGATIONS`` (``count`` / ``count_distinct`` only), @@ -531,6 +576,14 @@ def _validate_allowed_aggregations(self) -> "SlayerModel": for c in self.columns: if c.allowed_aggregations is None: continue + if c.type.is_opaque: + db_type_note = f" (db_type={c.db_type!r})" if c.db_type else "" + raise ValueError( + f"Column '{c.name}'{db_type_note}: allowed_aggregations " + f"cannot be declared on a column of type {c.type} — " + f"aggregations are not supported for that type. Remove " + f"allowed_aggregations, or give the column an operable type." + ) for agg_name in c.allowed_aggregations: if agg_name not in valid_names: raise ValueError( @@ -619,7 +672,7 @@ def _validate_source_query_stages(self) -> "SlayerModel": f"in source_queries must have a 'name'." ) seen: set = set() - dupes: List[str] = [] + dupes: list[str] = [] for stage in stages: n = getattr(stage, "name", None) if not n: @@ -634,19 +687,19 @@ def _validate_source_query_stages(self) -> "SlayerModel": ) return self - def get_column(self, name: str) -> Optional[Column]: + def get_column(self, name: str) -> Column | None: for c in self.columns: if c.name == name: return c return None - def get_measure(self, name: str) -> Optional[ModelMeasure]: + def get_measure(self, name: str) -> ModelMeasure | None: for m in self.measures: if m.name == name: return m return None - def get_aggregation(self, name: str) -> Optional[Aggregation]: + def get_aggregation(self, name: str) -> Aggregation | None: for a in self.aggregations: if a.name == name: return a @@ -654,17 +707,40 @@ def get_aggregation(self, name: str) -> Optional[Aggregation]: class DatasourceConfig(BaseModel): - version: int = 1 + version: int = 2 name: str - type: Optional[str] = None - host: Optional[str] = None - port: Optional[int] = None - database: Optional[str] = None - username: Optional[str] = None - password: Optional[str] = None - connection_string: Optional[str] = None - schema_name: Optional[str] = None - description: Optional[str] = None + type: str | None = None + host: str | None = None + port: int | None = None + database: str | None = None + username: str | None = None + password: str | None = None + connection_string: str | None = None + schema_name: str | None = None + # ``schema_name`` (above) is the UPSTREAM physical schema this datasource + # reads from (e.g. a Snowflake/Postgres source schema). ``postgres_schema`` + # is unrelated: it's the schema name the Postgres *facade* advertises this + # datasource's models under. Default ``None`` => "public", so by default + # every datasource's models share one schema; set it to separate them. + postgres_schema: str | None = None + description: str | None = None + # Snowflake-specific (v2, DEV-1551). Other dialects ignore them. + # ``connection_name`` is the primary auth path — when set, all credentials + # come from ``~/.snowflake/connections.toml`` via + # ``snowflake.connector.connect(connection_name=...)``. Inline form + # (host as account, username, password, database, schema_name) is the + # secondary path; ``warehouse`` and ``role`` populate the URL's query + # string for that path. See docs/configuration/datasources.md#snowflake. + connection_name: str | None = None + warehouse: str | None = None + role: str | None = None + # BigQuery-specific. Other dialects ignore it. The contents of a Google + # service-account key file as a JSON string. When set, the BigQuery dialect + # constructs the SQLAlchemy engine with ``credentials_info=json.loads(...)`` + # so the connection authenticates against that service account directly. + # When unset, BigQuery falls back to Application Default Credentials + # (``GOOGLE_APPLICATION_CREDENTIALS`` env var or attached compute identity). + credentials_json: str | None = Field(default=None, repr=False) @model_validator(mode="before") @classmethod @@ -696,11 +772,54 @@ def _validate_name(cls, v: str) -> str: _NO_COLON.check(name=v, context=label) return v + @field_validator("postgres_schema") + @classmethod + def _validate_postgres_schema(cls, v: str | None) -> str | None: + # The facade advertises models under this schema, so it must be a + # plain unquoted Postgres identifier. Postgres folds unquoted + # identifiers to lowercase, so we require lowercase to avoid the + # quoting ambiguity that would otherwise surprise BI tools. + if v is None: + return v + _require_non_empty_trimmed(v=v, context="Datasource 'postgres_schema'") + if not re.fullmatch(r"[a-z_][a-z0-9_]*", v): + raise ValueError( + f"Datasource 'postgres_schema' must be a lowercase Postgres " + f"identifier matching [a-z_][a-z0-9_]*, got {v!r}" + ) + return v + + def _get_tsql_connection_string(self) -> str: + return _SA_URL.create( + "mssql+pyodbc", + username=self.username or None, + password=self.password or None, + host=self.host or "localhost", + port=self.port, + database=self.database or "", + query={ + "driver": "ODBC Driver 18 for SQL Server", + "TrustServerCertificate": "yes", + }, + ).render_as_string(hide_password=False) + def get_connection_string(self) -> str: if self.connection_string: return self.connection_string + # Dialect-specific hook (DEV-1551): SnowflakeDialect builds the + # sentinel ``snowflake://?connection_name=`` URL or the + # full snowflake-sqlalchemy URL from inline fields. Tier-1 + # dialects without a custom hook return None and fall through + # to the standard branches below. + from slayer.sql.dialects import dialect_for_ds_type # noqa: PLC0415 + dialect = dialect_for_ds_type(self.type) + url_from_dialect = dialect.build_connection_url(self) + if url_from_dialect is not None: + return str(url_from_dialect) if self.type in ("sqlite", "duckdb"): return f"{self.type}:///{self.database}" + if self.type in ("mssql", "sqlserver", "tsql"): + return self._get_tsql_connection_string() driver_map = { "postgres": "postgresql", "postgresql": "postgresql", @@ -709,17 +828,49 @@ def get_connection_string(self) -> str: "clickhouse": "clickhouse+http", } driver = driver_map.get(self.type, self.type) - auth = "" - if self.username: - auth = self.username - if self.password: - auth += f":{self.password}" - auth += "@" - host_port = self.host or "localhost" - if self.port: - host_port += f":{self.port}" - db = self.database or "" - return f"{driver}://{auth}{host_port}/{db}" + # Build the URL with SQLAlchemy's structured builder (issue #240) + # rather than manual string concatenation, so credentials containing + # reserved URL characters (``@``, ``/``, ``:``, ...) are + # percent-encoded in the userinfo section instead of being misparsed + # as URL delimiters. ``username``/``password`` are treated as raw + # credentials; ``host or "localhost"`` preserves the pre-fix default. + # (SQLAlchemy renders the database path unencoded, matching the + # pre-fix behavior — a ``?`` in a database name is not our concern.) + # Mirrors ``_get_tsql_connection_string``. + host, port = self.host or "localhost", self.port + # Backward-compat with the pre-fix string branch, which tolerated the + # port (and IPv6 brackets) living in the host field. ``URL.create`` + # wants a raw host + separate port, so normalize: + # [::1] / [::1]:5432 -> strip brackets, lift embedded port + # db.example:5432 -> split single-colon numeric port + # A bare IPv6 host (``::1``) is left as-is — ``URL.create`` brackets + # it correctly. If the host embeds a port AND the ``port`` field is + # also set, that is contradictory config — raise rather than guess. + embedded_port: str | None = None + bracketed = _BRACKETED_HOST_RE.match(host) + if bracketed: + host = bracketed.group(1) + embedded_port = bracketed.group(2) + else: + embedded = _HOST_EMBEDDED_PORT_RE.match(host) + if embedded: + host, embedded_port = embedded.group(1), embedded.group(2) + if embedded_port is not None: + if port is not None: + raise ValueError( + f"Datasource '{self.name}': port is set both in the host " + f"field ({self.host!r}) and in the 'port' field ({port}); " + f"specify it in only one place." + ) + port = int(embedded_port) + return _SA_URL.create( + drivername=driver, + username=self.username or None, + password=self.password or None, + host=host, + port=port, + database=self.database or "", + ).render_as_string(hide_password=False) def resolve_env_vars(self) -> "DatasourceConfig": data = self.model_dump() diff --git a/slayer/core/policy.py b/slayer/core/policy.py new file mode 100644 index 00000000..e3959768 --- /dev/null +++ b/slayer/core/policy.py @@ -0,0 +1,377 @@ +"""Session-policy data model for forced-filter RLS. + +A ``SessionPolicy`` is immutable, agent-invisible engine state carrying exactly +one required ``ruleset``; the no-filtering case is ``policy=None`` at the +engine/client, so a bare ``SessionPolicy()`` raises. The rewrite it drives lives +in ``slayer/sql/session_policy.py``. +""" + +from __future__ import annotations + +from typing import Annotated, Literal, Tuple, Union + +from pydantic import ( + BaseModel, + ConfigDict, + Field, + ValidationInfo, + field_validator, + model_validator, +) + +# A scalar value implies ``=``; a non-empty list/tuple implies ``IN (...)``. +PolicyScalar = Union[str, int, float, bool] +OnUnapplicable = Literal["block", "pass"] + + +def _coerce_policy_value(v): + """Freeze a list/tuple ``value`` into a tuple; a degenerate empty ``IN`` is rejected.""" + if isinstance(v, (list, tuple)): + if len(v) == 0: + raise ValueError("policy rule value list/tuple must be non-empty") + return tuple(v) + return v + + +def _require_non_blank(v, info: ValidationInfo): + if not isinstance(v, str) or not v.strip(): + raise ValueError(f"{info.field_name} must be a non-empty string") + return v + + +def _table_parts(table: str) -> Tuple[str, ...]: + """Split a possibly qualified table name into whitespace-trimmed dotted parts.""" + return tuple(p.strip() for p in table.split(".")) + + +def _table_names_match(a: str, b: str) -> bool: + """Symmetric case-insensitive table match over the qualifier parts both names state, + so a bare name matches any schema and two qualified names must agree on every part.""" + ap, bp = _table_parts(a), _table_parts(b) + for pa, pb in zip(reversed(ap), reversed(bp)): + if pa.casefold() != pb.casefold(): + return False + return True + + +def _reaches_anchor(endpoint: str, anchor: str) -> bool: + """Whether a join-path endpoint reaches ``anchor``, qualified at least as fully as it. + + The endpoint is emitted verbatim, so a bare endpoint reaching a qualified anchor + would silently scope against the default-schema table instead. + """ + return _table_names_match(endpoint, anchor) and len( + _table_parts(endpoint) + ) >= len(_table_parts(anchor)) + + +class ColumnFilterRuleset(BaseModel): + """Filter every physical table that has ``column``; a scalar emits ``=``, a tuple ``IN``. + + ``on_unapplicable`` governs a table that confirms it lacks the column. A table whose + column presence cannot be confirmed always fails closed regardless. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + kind: Literal["column"] = "column" + column: str + value: Union[PolicyScalar, Tuple[PolicyScalar, ...]] + on_unapplicable: OnUnapplicable = "block" + + @field_validator("column") + @classmethod + def _non_blank_column(cls, v: str, info: ValidationInfo) -> str: + return _require_non_blank(v, info) + + @field_validator("value", mode="before") + @classmethod + def _coerce_value(cls, v): + return _coerce_policy_value(v) + + +class JoinHop(BaseModel): + """One parsed join hop: ``from_table.from_column`` -> ``to_table.to_column``. + + Internal only — callers author hops as strings (see :func:`_parse_hop`). + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + from_table: str + from_column: str + to_table: str + to_column: str + + @field_validator("from_table", "from_column", "to_table", "to_column") + @classmethod + def _non_blank(cls, v: str, info: ValidationInfo) -> str: + return _require_non_blank(v, info) + + +def _parse_hop(spec: str) -> JoinHop: + """Parse ``"from_table.from_column = to_table.to_column"`` into a :class:`JoinHop`. + + Each side splits on its last dot, so a column containing a dot is not expressible. + """ + if not isinstance(spec, str): + raise ValueError( + f"join_path hop must be a string, got {type(spec).__name__}" + ) + sides = spec.split("=") + if len(sides) != 2: + raise ValueError( + f"join_path hop {spec!r} must be " + "'from_table.from_column = to_table.to_column' (exactly one '=')" + ) + + def _split(side: str) -> tuple[str, str]: + table, dot, column = side.strip().rpartition(".") + if not dot: + raise ValueError( + f"join_path hop side {side.strip()!r} must be 'table.column' " + f"(in hop {spec!r})" + ) + return table.strip(), column.strip() + + from_table, from_column = _split(sides[0]) + to_table, to_column = _split(sides[1]) + return JoinHop( + from_table=from_table, + from_column=from_column, + to_table=to_table, + to_column=to_column, + ) + + +def _validate_hop_chain(*, hops: Tuple["JoinHop", ...]) -> None: + """Assert ``hops`` is non-empty and each hop starts where the previous one ended. + + Runs on every ``parsed_hops`` access, so a ``model_copy`` bypassing validation + still fails closed rather than reaching SQL generation. + """ + if not hops: + raise ValueError("JoinFilterRule.join_path must be non-empty") + for prev, cur in zip(hops, hops[1:]): + if cur.from_table.casefold() != prev.to_table.casefold(): + raise ValueError( + "JoinFilterRule.join_path hops must chain: hop from_table " + f"{cur.from_table!r} must equal the previous hop's to_table " + f"{prev.to_table!r}" + ) + + +def _reverse_hop(hop: JoinHop) -> JoinHop: + """Swap a hop's endpoints (used to normalize a master-first path).""" + return JoinHop( + from_table=hop.to_table, + from_column=hop.to_column, + to_table=hop.from_table, + to_column=hop.from_column, + ) + + +class JoinFilterRule(BaseModel): + """Scope ``target_table`` via an explicit join path to the ruleset's anchor. + + ``join_path`` holds hop strings (``"from_table.from_column = to_table.to_column"``, + physical names) whose two endpoints are ``target_table`` and the anchor, in either + written order. The tenant ``column``/``value`` live on the owning ruleset. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + target_table: str + join_path: Tuple[str, ...] + + @property + def parsed_hops(self) -> Tuple[JoinHop, ...]: + """``join_path`` parsed and chain-validated, derived fresh so it can never go stale.""" + hops = tuple(_parse_hop(spec) for spec in self.join_path) + _validate_hop_chain(hops=hops) + return hops + + @property + def _endpoints(self) -> Tuple[str, str]: + hops = self.parsed_hops + return (hops[0].from_table, hops[-1].to_table) + + def oriented_hops(self) -> Tuple[JoinHop, ...]: + """``parsed_hops`` normalized target-first, reversing a path authored anchor-first. + + Lets the EXISTS builder always find the wrapped source at the start and the + tenant column on the terminal hop. + """ + hops = self.parsed_hops + start, end = hops[0].from_table, hops[-1].to_table + if _table_names_match(start, self.target_table): + return hops + if _table_names_match(end, self.target_table): + return tuple(_reverse_hop(h) for h in reversed(hops)) + raise ValueError( + f"JoinFilterRule.target_table ({self.target_table!r}) is not an " + f"endpoint of join_path (endpoints {start!r}, {end!r})" + ) + + @field_validator("target_table") + @classmethod + def _non_blank(cls, v: str, info: ValidationInfo) -> str: + return _require_non_blank(v, info) + + @field_validator("join_path", mode="before") + @classmethod + def _coerce_path(cls, v): + if isinstance(v, str): + # Would otherwise be iterated into a tuple of single characters. + raise ValueError( + "JoinFilterRule.join_path must be a list of hop strings, not a " + "single string" + ) + if isinstance(v, list): + return tuple(v) + return v + + @model_validator(mode="after") + def _validate_endpoints(self): + # The matching "non-target endpoint == anchor" check lives on the ruleset, + # which is what knows the anchor. + hops = self.parsed_hops + start, end = hops[0].from_table, hops[-1].to_table + if not ( + _table_names_match(start, self.target_table) + or _table_names_match(end, self.target_table) + ): + raise ValueError( + "JoinFilterRule.target_table " + f"({self.target_table!r}) must be one of the join_path " + f"endpoints ({start!r}, {end!r})" + ) + return self + + +def _validate_join_rule_anchor( + rule: JoinFilterRule, anchor: str +) -> Tuple[JoinHop, ...]: + """Validate one join rule against the ruleset ``anchor``, returning target-first hops. + + Single source of truth for the per-rule anchor invariants, shared by ruleset + construction and the SQL boundary so both enforce the same set. Cross-rule checks + (duplicate targets, whitelist overlaps) stay on the ruleset validator. + """ + oriented = rule.oriented_hops() + terminal = oriented[-1].to_table + if not _reaches_anchor(terminal, anchor): + raise ValueError( + f"JoinFilterRule for target '{rule.target_table}': the join_path " + f"must reach the anchor table '{anchor}' at its non-target endpoint, " + f"qualified at least as fully as the anchor (got '{terminal}')." + ) + # The anchor may not also appear as an intermediate hop. + node_sequence = [oriented[0].from_table] + [h.to_table for h in oriented] + if sum(1 for n in node_sequence if _table_names_match(n, anchor)) != 1: + raise ValueError( + f"JoinFilterRule for target '{rule.target_table}': the anchor table " + f"'{anchor}' must appear exactly once in the join path, only as the " + "terminal endpoint (not as an intermediate hop)." + ) + if _table_names_match(rule.target_table, anchor): + raise ValueError( + f"JoinFilterRule may not target the anchor table '{anchor}' " + "(the anchor is filtered directly)." + ) + return oriented + + +class JoinFilterRuleset(BaseModel): + """Single-anchor join model: the tenant identifier lives on one ``table`` + ``column``. + + The anchor is filtered directly, each :class:`JoinFilterRule` target is scoped via a + correlated ``EXISTS`` to it, ``whitelist`` tables are emitted unfiltered, and any + other physical table fails closed. Fully structural — no column introspection. + """ + + model_config = ConfigDict(extra="forbid", frozen=True) + + kind: Literal["join"] = "join" + table: str + column: str + value: Union[PolicyScalar, Tuple[PolicyScalar, ...]] + joins: Tuple[JoinFilterRule, ...] = () + whitelist: Tuple[str, ...] = () + + @field_validator("table", "column") + @classmethod + def _non_blank(cls, v: str, info: ValidationInfo) -> str: + return _require_non_blank(v, info) + + @field_validator("value", mode="before") + @classmethod + def _coerce_value(cls, v): + return _coerce_policy_value(v) + + @field_validator("joins", "whitelist", mode="before") + @classmethod + def _coerce_tuple(cls, v): + return tuple(v) if isinstance(v, list) else v + + @field_validator("whitelist") + @classmethod + def _non_blank_whitelist(cls, v): + for entry in v: + if not isinstance(entry, str) or not entry.strip(): + raise ValueError("whitelist entries must be non-empty strings") + return v + + @model_validator(mode="after") + def _validate_anchor_reachability(self): + master = self.table + seen_targets: list[str] = [] + for rule in self.joins: + _validate_join_rule_anchor(rule, master) + if any(_table_names_match(rule.target_table, t) for t in seen_targets): + raise ValueError( + f"Duplicate JoinFilterRule target '{rule.target_table}' " + "(one path per target)." + ) + seen_targets.append(rule.target_table) + for entry in self.whitelist: + if _table_names_match(entry, master): + raise ValueError( + f"whitelist entry '{entry}' is the anchor table; the anchor " + "is filtered, not passed through." + ) + if any(_table_names_match(entry, t) for t in seen_targets): + raise ValueError( + f"whitelist entry '{entry}' is also a join target; a table " + "cannot be both whitelisted and join-scoped." + ) + return self + + +# Keyed on the explicit ``kind`` field — no inference, so a kind-less dict fails +# to discriminate. +FilterRuleset = Annotated[ + Union[ColumnFilterRuleset, JoinFilterRuleset], Field(discriminator="kind") +] + + +class SessionPolicy(BaseModel): + """Immutable, engine-global forced-filter configuration.""" + + model_config = ConfigDict(extra="forbid", frozen=True) + + # Pinned so an unknown schema version fails closed rather than being silently + # interpreted by the v1 rewrite path. + version: Literal[1] = 1 + ruleset: FilterRuleset + + +__all__ = [ + "PolicyScalar", + "OnUnapplicable", + "ColumnFilterRuleset", + "JoinFilterRule", + "JoinFilterRuleset", + "FilterRuleset", + "SessionPolicy", +] diff --git a/slayer/core/query.py b/slayer/core/query.py index 9b988100..fd117bc3 100644 --- a/slayer/core/query.py +++ b/slayer/core/query.py @@ -9,7 +9,7 @@ import datetime import logging import re -from typing import Annotated, Any, Dict, List, Optional +from typing import Annotated, Any from pydantic import BaseModel, BeforeValidator, ConfigDict, field_validator, model_validator @@ -38,7 +38,7 @@ def _validate_query_filter_string(formula: str) -> None: raise ValueError(f"Filter '{formula}' {WINDOW_IN_FILTER_ERROR}") -def substitute_variables(filter_str: str, variables: Dict[str, Any]) -> str: +def substitute_variables(filter_str: str, variables: dict[str, Any]) -> str: """Substitute {variable} placeholders in a filter string. - {var_name} is replaced with the variable's value (str or number, inserted as-is). @@ -110,8 +110,8 @@ class ColumnRef(BaseModel): on the query's model. """ name: str - model: Optional[str] = None - label: Optional[str] = None + model: str | None = None + label: str | None = None @model_validator(mode="after") def _parse_dotted_name(self) -> "ColumnRef": @@ -163,6 +163,47 @@ def _coerce_column_ref(v: Any) -> Any: _FUNCSTYLE_CALL_PATTERN = re.compile(r"^\w+\([^()]*\)$") +# DEV-1733: sentinel ``ColumnRef.name`` values meaning "this ORDER BY item is +# an EXPRESSION — resolve it from ``raw_formula``, not as a column reference". +# ``_funcstyle_pending`` marks an unrewritten function-style call (a custom +# aggregation or a transform); ``_expr_pending`` marks any other formula shape +# that is not expressible as a ``ColumnRef`` (composite arithmetic, a scalar +# call over an aggregation, arithmetic over a transform). Consumers MUST also +# require a non-empty ``raw_formula`` before treating a name as a sentinel, so +# a model that genuinely has a column of that name still resolves normally. +_FUNCSTYLE_PENDING = "_funcstyle_pending" +_EXPR_PENDING = "_expr_pending" +ORDER_PLACEHOLDER_NAMES = frozenset({_FUNCSTYLE_PENDING, _EXPR_PENDING}) + + +def _order_formula_candidate(v: str) -> str | None: + """The func-style-rewritten form of ``v`` when it carries a measure + expression (a colon aggregation, or a function-style call), else ``None``. + + Single source of truth for "this ORDER BY string is a formula, not a column + reference". Shared by :meth:`OrderItem._capture_raw_formula` (which + preserves the original text) and :func:`_coerce_order_column` (which emits + the placeholder ``ColumnRef``) so the two cannot drift — if only one of + them recognised a shape, the item would either lose its formula or bind a + meaningless placeholder name. + """ + from slayer.core.formula import _rewrite_funcstyle_aggregations + + rewritten = _rewrite_funcstyle_aggregations(v) + if ":" in rewritten or _FUNCSTYLE_CALL_PATTERN.match(rewritten): + return rewritten + return None + + +def _is_valid_column_ref_name(name: str) -> bool: + """Whether ``name`` parses as a ``ColumnRef`` (bare leaf or dotted path).""" + try: + ColumnRef.model_validate({"name": name}) + except Exception: + return False + return True + + def _coerce_order_column(v: Any) -> Any: """Coerce ORDER BY column, normalizing aggregation syntax. @@ -177,15 +218,27 @@ def _coerce_order_column(v: Any) -> Any: - "revenue:last(ordered_at)" → "revenue_last" - "rolling_avg(revenue)" → placeholder, raw_formula carries the call so enrichment can resolve it via ``extra_agg_names``. + - "revenue:sum / cnt:sum" → placeholder (DEV-1733), raw_formula carries the + composite so the planner binds it as an expression. + + DEV-1733: a composite that is NOT a formula candidate — ``"rev / cnt"``, + arithmetic over declared measure ALIASES — falls through to normal + ``ColumnRef`` validation and keeps its original error. Alias references + inside expressions are unsupported everywhere in SLayer, so failing fast at + construction is better than a deep binder error. """ if isinstance(v, str): from slayer.core.formula import _rewrite_funcstyle_aggregations - rewritten = _rewrite_funcstyle_aggregations(v) + candidate = _order_formula_candidate(v) + rewritten = ( + candidate if candidate is not None + else _rewrite_funcstyle_aggregations(v) + ) if _FUNCSTYLE_CALL_PATTERN.match(rewritten): # Unrewritten function-style call (custom aggregation). Enrichment # parses raw_formula with custom_agg_names and overwrites # column.name with the canonical alias, so a placeholder is fine. - return {"name": "_funcstyle_pending"} + return {"name": _FUNCSTYLE_PENDING} if ":" in rewritten: base, agg = rewritten.rsplit(":", 1) agg_name = agg.split("(", 1)[0] # strip arglist @@ -193,35 +246,84 @@ def _coerce_order_column(v: Any) -> Any: rewritten = f"_{agg_name}" else: rewritten = f"{base}_{agg_name}" + if candidate is not None and not _is_valid_column_ref_name(rewritten): + # A formula that does not canonicalise to a column reference — + # composite arithmetic, a scalar call over an aggregation, or + # arithmetic over a transform. ``raw_formula`` carries the original. + return {"name": _EXPR_PENDING} return {"name": rewritten} return v +# DEV-1575: the accepted ORDER BY direction vocabulary (case-insensitive), +# mapping every synonym to the canonical lowercase form the SQL generator +# compares against (``direction == "asc"``). Single source of truth shared by +# the shorthand-healing detector (``_process_order_item``) and the +# ``OrderItem.direction`` normalizing validator. +_DIRECTION_NORMALIZE = { + "asc": "asc", + "ascending": "asc", + "desc": "desc", + "descending": "desc", +} + + +def _is_direction(value: Any) -> bool: + """True if ``value`` is a recognized direction word (case/whitespace-insensitive).""" + return isinstance(value, str) and value.strip().lower() in _DIRECTION_NORMALIZE + + class TimeDimension(BaseModel): dimension: Annotated[ColumnRef, BeforeValidator(_coerce_column_ref)] granularity: TimeGranularity - date_range: Optional[List[str]] = None - label: Optional[str] = None + date_range: list[str] | None = None + label: str | None = None class OrderItem(BaseModel): + # DEV-1575: reject stray keys so a mixed canonical+shorthand item + # (e.g. ``{"column": "x", "b": "asc"}``) raises loudly instead of silently + # dropping the extra key. + model_config = ConfigDict(extra="forbid") + column: Annotated[ColumnRef, BeforeValidator(_coerce_order_column)] direction: str = "asc" - raw_formula: Optional[str] = None + raw_formula: str | None = None @model_validator(mode="before") @classmethod def _capture_raw_formula(cls, data: Any) -> Any: - """Capture the raw column formula before coercion normalizes it.""" + """Capture the raw column formula before coercion normalizes it. + + Shares :func:`_order_formula_candidate` with ``_coerce_order_column`` + so a shape can never be recognised by one and not the other. + """ if isinstance(data, dict): col = data.get("column") if isinstance(col, str): - from slayer.core.formula import _rewrite_funcstyle_aggregations - rewritten = _rewrite_funcstyle_aggregations(col) - if ":" in rewritten or _FUNCSTYLE_CALL_PATTERN.match(rewritten): - data = {**data, "raw_formula": rewritten} + candidate = _order_formula_candidate(col) + if candidate is not None: + data = {**data, "raw_formula": candidate} return data + @field_validator("direction") + @classmethod + def _normalize_direction(cls, v: str) -> str: + """DEV-1575: normalize direction to canonical ``asc``/``desc`` (case- and + whitespace-insensitive, accepting ``ascending``/``descending`` synonyms), + rejecting anything else. + + The SQL generator compares ``direction == "asc"`` strictly, so a + non-normalized value (``"ASC"``, ``"ascending"``) would silently emit + DESC. Normalizing here fixes that for both healed and canonical items. + """ + if _is_direction(v): + return _DIRECTION_NORMALIZE[v.strip().lower()] + raise ValueError( + "order direction must be one of asc/desc/ascending/descending " + f"(case-insensitive), got {v!r}" + ) + def _coerce_measures(v: Any) -> Any: """Allow plain strings in the measures list: "count" → {"formula": "count"}.""" @@ -241,6 +343,56 @@ def _coerce_dimensions(v: Any) -> Any: return [{"name": item} if isinstance(item, str) else item for item in v] +def _process_order_item(item: Any) -> list: + """DEV-1575: heal a single ``order`` entry, returning the list of canonical + items it expands to. + + LLM agents frequently write a shorthand dict mapping column → direction + instead of the canonical ``{"column", "direction"}`` shape. A dict with no + ``column``/``direction`` key whose values are *all* direction words is + treated as shorthand and expanded — one canonical item per key, preserving + insertion order (so a single-key dict yields one item and a multi-key dict + yields several). Everything else passes through unchanged: canonical dicts + (their stray keys are policed by ``OrderItem``'s ``extra="forbid"``) and + malformed input (rejected by ``OrderItem`` validation). + """ + if not isinstance(item, dict): + return [item] + # A dict carrying a reserved key is canonical-intended; never reinterpret it + # as shorthand. extra="forbid" on OrderItem rejects any stray keys. + if "column" in item or "direction" in item: + return [item] + if ( + item + and all(isinstance(k, str) for k in item) + and all(_is_direction(val) for val in item.values()) + ): + return [{"column": k, "direction": val} for k, val in item.items()] + return [item] + + +def _coerce_order(v: Any) -> Any: + """DEV-1575: heal shorthand ``order`` items before ``OrderItem`` validation. + + Accepts the canonical ``list``/``tuple`` of items, healing each shorthand + dict (see ``_process_order_item``). A bare single item passed without the + enclosing list (a ``dict`` or an ``OrderItem``) is wrapped into a + one-element list; any other non-list input raises (mirrors the + ``_coerce_measures``/``_coerce_dimensions`` convention). + """ + if v is None: + return v + if not isinstance(v, (list, tuple)): + if isinstance(v, (dict, OrderItem)): + v = [v] + else: + raise TypeError(f"'order' must be a list, got {type(v).__name__}") + result: list = [] + for item in v: + result.extend(_process_order_item(item)) + return result + + class ModelExtension(BaseModel): """Extend an existing model with extra columns, measures, or joins. @@ -249,12 +401,12 @@ class ModelExtension(BaseModel): stored model. """ source_name: str # Model/query to extend - columns: Optional[List] = None # Extra Column objects - measures: Optional[List[ModelMeasure]] = None # Extra ModelMeasure formulas - joins: Optional[List] = None # Extra ModelJoin objects + columns: list | None = None # Extra Column objects + measures: list[ModelMeasure] | None = None # Extra ModelMeasure formulas + joins: list | None = None # Extra ModelJoin objects -def _get_source_model_name(source_model: object) -> Optional[str]: +def _get_source_model_name(source_model: object) -> str | None: """Extract the model name from any source_model type. Works before model resolution — handles str, dict, ModelExtension, @@ -308,9 +460,9 @@ class SlayerQuery(BaseModel): model_config = ConfigDict(extra="forbid") version: int = 3 - name: Optional[str] = None # For referencing this query from other queries in a list + name: str | None = None # For referencing this query from other queries in a list source_model: object # str (model name), SlayerModel (inline), or ModelExtension - measures: Annotated[Optional[List[ModelMeasure]], BeforeValidator(_coerce_measures)] = None + measures: Annotated[list[ModelMeasure] | None, BeforeValidator(_coerce_measures)] = None @model_validator(mode="before") @classmethod @@ -319,7 +471,7 @@ def _apply_schema_migrations(cls, data: Any) -> Any: @field_validator("name") @classmethod - def _validate_query_name(cls, v: Optional[str]) -> Optional[str]: + def _validate_query_name(cls, v: str | None) -> str | None: # Share the same rejection rules as SlayerModel.name — # SlayerQuery names occupy the same naming space when persisted # as query-backed models. Rejects ``__`` (join-path alias @@ -329,15 +481,24 @@ def _validate_query_name(cls, v: Optional[str]) -> Optional[str]: return v from slayer.core.models import _validate_model_name return _validate_model_name(v, "Query") - dimensions: Annotated[Optional[List[ColumnRef]], BeforeValidator(_coerce_dimensions)] = None - time_dimensions: Optional[List[TimeDimension]] = None - main_time_dimension: Optional[str] = None # Explicit time dimension for transforms (overrides auto-detection) - filters: Optional[List[str]] = None - variables: Optional[Dict[str, Any]] = None # Variable values for filter substitution - order: Optional[List[OrderItem]] = None - limit: Optional[int] = None - offset: Optional[int] = None + dimensions: Annotated[list[ColumnRef] | None, BeforeValidator(_coerce_dimensions)] = None + time_dimensions: list[TimeDimension] | None = None + main_time_dimension: str | None = None # Explicit time dimension for transforms (overrides auto-detection) + filters: list[str] | None = None + variables: dict[str, Any] | None = None # Variable values for filter substitution + order: Annotated[list[OrderItem] | None, BeforeValidator(_coerce_order)] = None + limit: int | None = None + offset: int | None = None whole_periods_only: bool = False + # DEV-1543: opt out of the auto "distinct dimension tuples" GROUP BY + # for dim-only queries. Default ``True`` preserves the Cube.js-style + # dedup that fires when ``measures`` is empty. Setting ``False`` emits + # a flat ``SELECT FROM ... WHERE ... ORDER BY ... + # LIMIT ...`` projection. Any measure reference (in ``measures``, in + # ``filters``, or in ``order``) is rejected with + # ``DistinctDimensionValuesError``; both ``dimensions`` and + # ``time_dimensions`` empty is also rejected (nothing to project). + distinct_dimension_values: bool = True @model_validator(mode="after") def _validate_dsl_user_input(self) -> "SlayerQuery": @@ -358,8 +519,46 @@ def _validate_dsl_user_input(self) -> "SlayerQuery": if self.filters: for f in self.filters: _validate_query_filter_string(f) + self._validate_distinct_dimension_values() return self + def _validate_distinct_dimension_values(self) -> None: + """DEV-1543: structural rejection rules for ``distinct_dimension_values=False``. + + Only the cheap, model-free checks fire here: + + * ``measures`` non-empty — flag asks for raw rows, but the query + asks for aggregations. + * Both ``dimensions`` and ``time_dimensions`` empty — there are + no projected columns to ``SELECT``. + + Deep filter / order measure-reference checks happen at enrichment, + where named measures, custom aggregations, and post-substitution + text are all available. Detecting them here would either reject + valid ``{var}`` filters before substitution or miss model-defined + custom aggregations. + """ + if self.distinct_dimension_values: + return + from slayer.core.errors import DistinctDimensionValuesError + + if self.measures: + n = len(self.measures) + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False requires an empty `measures` " + f"field, but {n} measure(s) were supplied. Either remove the " + f"measures (and any other measure references) or set " + f"distinct_dimension_values=True (the default) to keep the " + f"auto-aggregating behaviour." + ) + if not self.dimensions and not self.time_dimensions: + raise DistinctDimensionValuesError( + "distinct_dimension_values=False requires at least one of " + "`dimensions` or `time_dimensions` to be non-empty — there " + "are no columns to SELECT. Add the columns you want to " + "project." + ) + def snap_to_whole_periods(self) -> "SlayerQuery": """Adjust date filters to align with period boundaries when whole_periods_only=True. @@ -396,7 +595,7 @@ def strip_source_model_prefix(self) -> "SlayerQuery": if model_name is None: return self - updates: Dict[str, Any] = {} + updates: dict[str, Any] = {} pattern = re.compile(r"\b" + re.escape(model_name) + r"\.") # Dimensions diff --git a/slayer/core/recommend.py b/slayer/core/recommend.py new file mode 100644 index 00000000..b5c3eb99 --- /dev/null +++ b/slayer/core/recommend.py @@ -0,0 +1,85 @@ +"""Result models + rendering for ``recommend_root_model`` (DEV-1626). + +Given a set of ``model.column`` / ``model.metric`` items an agent wants in +one query, the engine recommends a root model (query ``source_model``) and +the join-qualified reference path for each item from that root. These +models are the caller-facing shape returned by the engine and echoed +verbatim through MCP / REST / CLI / SlayerClient. +""" + +from __future__ import annotations + +from pydantic import BaseModel + + +class ItemPath(BaseModel): + """One input item paired with its join-qualified reference path from + the recommended root (relative to the root, root name excluded — the + same form used inside a ``SlayerQuery`` whose ``source_model`` is the + root). Any aggregation suffix is preserved verbatim.""" + + input_item: str + path: str + + +class CandidateCoverage(BaseModel): + """A partial-root candidate for the no-common-root diagnostic: which + input items it can reach and which it cannot.""" + + model_name: str + reachable_items: list[str] + unreachable_items: list[str] + + +class RootModelRecommendation(BaseModel): + """Outcome of :meth:`SlayerQueryEngine.recommend_root_model`. + + When ``reachable`` is ``True``, ``root_model`` names the recommended + ``source_model`` and ``item_paths`` gives every item's path from it. + When ``False``, no single model reaches all items: ``root_model`` is + ``None``, ``item_paths`` is empty, and ``coverage`` lists the Pareto + frontier of partial roots so the caller can split the work into a + multi-stage query. + """ + + data_source: str + root_model: str | None + reachable: bool + item_paths: list[ItemPath] = [] + coverage: list[CandidateCoverage] = [] + message: str = "" + warnings: list[str] = [] + + +def render_recommendation_markdown(rec: RootModelRecommendation) -> str: + """Render a recommendation as human-readable markdown (MCP / CLI).""" + lines: list[str] = [] + lines.append(f"Datasource: `{rec.data_source}`") + if rec.reachable and rec.root_model is not None: + lines.append(f"Recommended root model: `{rec.root_model}`") + lines.append("") + lines.append("| Input item | Path from root |") + lines.append("|------------|----------------|") + for ip in rec.item_paths: + lines.append(f"| `{ip.input_item}` | `{ip.path}` |") + else: + lines.append("Recommended root model: _none_ — no single model " + "reaches every item.") + if rec.message: + lines.append("") + lines.append(rec.message) + if rec.coverage: + lines.append("") + lines.append("Best partial roots (Pareto frontier):") + lines.append("") + lines.append("| Candidate root | Reaches | Cannot reach |") + lines.append("|----------------|---------|--------------|") + for c in rec.coverage: + reaches = ", ".join(f"`{i}`" for i in c.reachable_items) or "_(none)_" + misses = ", ".join(f"`{i}`" for i in c.unreachable_items) or "_(none)_" + lines.append(f"| `{c.model_name}` | {reaches} | {misses} |") + if rec.warnings: + lines.append("") + lines.append("Warnings:") + lines.extend(f"- {w}" for w in rec.warnings) + return "\n".join(lines) diff --git a/slayer/core/refs.py b/slayer/core/refs.py index dadf648d..0d0f5eb3 100644 --- a/slayer/core/refs.py +++ b/slayer/core/refs.py @@ -17,7 +17,7 @@ import re from decimal import Decimal -from typing import Any, List, Optional, Tuple +from typing import Any from slayer.core.keys import ColumnKey, ColumnSqlKey @@ -60,8 +60,8 @@ def agg_signature_suffix( - agg_args: Optional[List[str]], - agg_kwargs: Optional[dict], + agg_args: list[str] | None, + agg_kwargs: dict | None, ) -> str: """Build a deterministic identifier suffix from aggregation args/kwargs. @@ -76,7 +76,7 @@ def agg_signature_suffix( kwargs = agg_kwargs or {} if not args and not kwargs: return "" - parts: List[str] = [] + parts: list[str] = [] for a in args: sanitized = _NON_IDENT_RE.sub("_", str(a)).strip("_") if sanitized: @@ -192,8 +192,8 @@ def agg_kwarg_canonical_str(value: Any) -> str: def canonical_agg_name( measure_name: str, aggregation_name: str, - agg_args: Optional[List[str]] = None, - agg_kwargs: Optional[dict] = None, + agg_args: list[str] | None = None, + agg_kwargs: dict | None = None, ) -> str: """Canonical hidden-column name for an aggregated measure ref. @@ -207,7 +207,7 @@ def canonical_agg_name( return f"{measure_name}_{aggregation_name}{suffix}" -def strip_agg_suffix(raw: str) -> Tuple[str, Optional[str]]: +def strip_agg_suffix(raw: str) -> tuple[str, str | None]: """Return ``(prefix, agg_name)`` after stripping a trailing ``:agg`` or ``:agg(...)``. @@ -233,6 +233,28 @@ def strip_agg_suffix(raw: str) -> Tuple[str, Optional[str]]: return raw, None +def split_agg_suffix(raw: str) -> tuple[str, str | None]: + """Return ``(prefix, suffix)`` splitting off a trailing ``:agg`` / + ``:agg(...)``, keeping the *full* suffix text (args included). + + Unlike :func:`strip_agg_suffix` (which discards the arglist and returns + only the aggregation name), this preserves the entire suffix so callers + that re-root a reference can re-attach it verbatim — + ``"orders.revenue:weighted_avg(weight=qty)"`` → + ``("orders.revenue", "weighted_avg(weight=qty)")``. Returns + ``(raw, None)`` when there is no top-level ``:`` aggregation. + """ + depth = 0 + for i, ch in enumerate(raw): + if ch == "(": + depth += 1 + elif ch == ")": + depth -= 1 + elif ch == ":" and depth == 0: + return raw[:i], raw[i + 1:] + return raw, None + + # --------------------------------------------------------------------------- # User-input validation # --------------------------------------------------------------------------- diff --git a/slayer/core/time_bounds.py b/slayer/core/time_bounds.py new file mode 100644 index 00000000..6f12ec43 --- /dev/null +++ b/slayer/core/time_bounds.py @@ -0,0 +1,150 @@ +"""Frame-bound predicate analysis for trailing-window / shifted CTEs (DEV-1732). + +Some CTEs must read rows from OUTSIDE the query's visible time frame: + +* a duration-windowed measure's ``_src`` subquery (``revenue:sum(window='90d')``) + — the trailing window reaches back before the earliest visible bucket, or that + bucket under-counts; +* a ``time_shift`` shifted CTE — the shifted value for the earliest visible + bucket comes from a bucket outside the frame. + +``TimeDimension.date_range`` has always been excluded from those CTEs for that +reason. This module generalises the exclusion from that one carrier to the +*semantic class* it belongs to, so the two spellings of one intent agree: + + A ROW-phase filter conjunct that is a relational bound, with a temporal + literal, on the raw column of one of the query's time dimensions is a FRAME + bound, not a population filter. Frame bounds constrain the visible buckets + only. Everything else is a population filter and is applied unchanged. + +Dependency-free by design (imports only :mod:`slayer.core.keys`) so both the +engine planner and the SQL generator can call it without either importing the +other — the same placement rationale as :mod:`slayer.core.window_duration`. +""" + +from __future__ import annotations + +from typing import AbstractSet, Optional + +from slayer.core.keys import ArithmeticKey, BetweenKey, LiteralKey, ValueKey + +__all__ = [ + "RELATIONAL_OPS", + "is_temporal_literal", + "is_frame_bound", + "strip_frame_bounds", +] + +#: Operators that can express a frame bound. ``==`` / ``!=`` / ``in`` / ``is`` +#: are deliberately absent: an equality on a raw timestamp means "this instant" +#: or "this set", never a range, and stripping one would sum the whole window +#: where a single instant was asked for. +RELATIONAL_OPS = frozenset({"<", "<=", ">", ">="}) + +_AND = "and" + + +def is_temporal_literal(key: object) -> bool: + """Is ``key`` a literal usable as a frame-bound endpoint? + + A **bare** ``LiteralKey`` holding a non-``None`` ``str`` — nothing else. + "Bare" means the operand IS the literal, not an expression tree that merely + contains one: ``ArithmeticKey('+', (LiteralKey(1), LiteralKey(2)))`` does not + qualify. (``isinstance`` is deliberate — ``LiteralKey`` has no subclasses, + and an exact ``type(...) is`` check would be unidiomatic here.) + + Deliberately a whitelist of one shape rather than "contains no column + reference", which would also admit dynamic expressions (a zero-argument + scalar call, say) and quietly treat them as frame bounds. + + Mirrors ``BetweenKey``, whose ``low``/``high`` are + ``LiteralKey(value=normalize_scalar(...))`` and are strings for dates — so + the explicit spelling is recognised on exactly the same terms as the + ``date_range`` one. + + Two cases the strictness matters for: + + * ``created_at < None`` binds to ``LiteralKey(value=None)``. ``col < NULL`` + matches nothing; stripping it would turn an empty result into the full + population. + * ``created_at >= 5`` binds to ``LiteralKey(value=Decimal(5))`` — a + type-invalid comparison, not a frame bound. + + ``bool`` is excluded for free: ``isinstance(True, str)`` is ``False``. + """ + return isinstance(key, LiteralKey) and isinstance(key.value, str) + + +def is_frame_bound(*, key: object, time_columns: AbstractSet[ValueKey]) -> bool: + """Is ``key`` a single frame bound on one of ``time_columns``? + + ``time_columns`` holds the RAW column keys (``ColumnKey`` / ``ColumnSqlKey``) + of the query's non-hidden time dimensions. Matching is by ValueKey identity, + so a derived (``Column.sql``) temporal column is covered without any special + casing, and a same-named column on a joined model cannot collide. + + Both operand orders count — ``'2024-06-01' <= created_at`` says the same + thing as ``created_at >= '2024-06-01'``. + """ + if isinstance(key, BetweenKey): + return ( + key.column in time_columns + and is_temporal_literal(key.low) + and is_temporal_literal(key.high) + ) + if not isinstance(key, ArithmeticKey) or key.op not in RELATIONAL_OPS: + return False + if len(key.operands) != 2: + return False + lhs, rhs = key.operands + if lhs in time_columns: + return is_temporal_literal(rhs) + if rhs in time_columns: + return is_temporal_literal(lhs) + return False + + +def strip_frame_bounds( + *, key: ValueKey, time_columns: AbstractSet[ValueKey], +) -> Optional[ValueKey]: + """Return ``key`` with its top-level frame bounds removed. + + * ``None`` — the whole predicate was a frame bound (or a conjunction of + them); the caller omits the filter from the CTE entirely. + * the **same object** — nothing was stripped; the caller can skip building a + rewrite entry, and the CTE renders the host's predicate verbatim. + * a new key — the residual population predicate. + + A top-level ``and`` is split: each operand is tested independently, frame + bounds are dropped, survivors are rebuilt in their original order (a lone + survivor replaces the conjunction). Nested ``and`` is recursed into. + + ``or`` and ``not`` are never descended into — no sound split exists under a + disjunction or a negation, and keeping the predicate whole preserves the + pre-DEV-1732 result, which is the safe direction to err in. + """ + if not time_columns: + return key + if is_frame_bound(key=key, time_columns=time_columns): + return None + if not (isinstance(key, ArithmeticKey) and key.op == _AND): + return key + + kept: list[ValueKey] = [] + changed = False + for operand in key.operands: + residual = strip_frame_bounds(key=operand, time_columns=time_columns) + if residual is None: + changed = True + continue + if residual is not operand: + changed = True + kept.append(residual) + + if not changed: + return key + if not kept: + return None + if len(kept) == 1: + return kept[0] + return ArithmeticKey(op=_AND, operands=tuple(kept)) diff --git a/slayer/core/window_duration.py b/slayer/core/window_duration.py new file mode 100644 index 00000000..2e857b55 --- /dev/null +++ b/slayer/core/window_duration.py @@ -0,0 +1,49 @@ +"""Compact duration parsing for windowed measures (``window='90d'``). + +Shared by the plan-time windowed guard (engine layer, DEV-1714) and the SQL +generator's per-unit interval emission (sql layer). Lives in ``slayer.core`` and +is dependency-free so the engine planner can validate a window duration at plan +time WITHOUT importing the SQL layer. + +Compact syntax only — an integer immediately followed by a unit, repeated with +no separators: ``1y2m3w5d6h7min8s``. Units: ``y`` year, ``m`` month, ``w`` week, +``d`` day, ``h`` hour, ``min`` minute, ``s`` second. +""" + +from __future__ import annotations + +import re + +# ``min`` must precede the single-char alternation so ``7min`` parses the whole +# ``min`` unit rather than a bare ``m`` followed by a stray ``in``. +_WINDOW_DURATION_RE = re.compile(r"(?P\d+)(?Pmin|[ymwdhs])") + + +def parse_window_duration(value: str) -> list[tuple[int, str]]: + """Parse a compact duration like ``1y2m3w5d6h7min8s`` into ``(amount, unit)`` + parts, in written order. + + Raises ``ValueError`` on an empty string, a non-positive amount, or any + malformed / gapped input (e.g. ``'90x'``, ``'d90'``). The error messages are + a stable contract — the plan-time guard and its tests match on them. + """ + if not value: + raise ValueError("Window duration cannot be empty") + pos = 0 + parts: list[tuple[int, str]] = [] + for match in _WINDOW_DURATION_RE.finditer(value): + if match.start() != pos: + raise ValueError( + f"Invalid window duration '{value}'. Use syntax like '1y2m3w5d6h7min8s'." + ) + amount = int(match.group("num")) + unit = match.group("unit") + if amount <= 0: + raise ValueError(f"Window duration parts must be positive in '{value}'") + parts.append((amount, unit)) + pos = match.end() + if pos != len(value) or not parts: + raise ValueError( + f"Invalid window duration '{value}'. Use syntax like '1y2m3w5d6h7min8s'." + ) + return parts diff --git a/slayer/dbt/converter.py b/slayer/dbt/converter.py index 0f3bb545..ca3ad7ae 100644 --- a/slayer/dbt/converter.py +++ b/slayer/dbt/converter.py @@ -8,29 +8,35 @@ The converter never emits ``SlayerQuery`` definitions: every dbt artefact that produces a query-shaped result is expressed as a named formula on a -model. Metrics that cannot be expressed that way (e.g. transform-name -collisions, conversion metrics) are returned in -``ConversionResult.unconverted_metrics``. +model. Constructs that cannot be expressed exactly (conversion metrics, +windowed / grain-to-date cumulatives, semi-additive measures, …) are *failed +cleanly*: routed to the structured ``ConversionResult`` report with a precise +reason + workaround, and the raw construct is stashed into the owning entity's +``meta`` so the dropped semantics are retained, never silently lost. (DEV-1595.) """ import logging import re from collections import defaultdict -from typing import Dict, List, Optional, Tuple +from typing import Any, Literal import sqlalchemy as sa -from pydantic import BaseModel, Field from slayer.core.enums import DataType, JoinType from slayer.core.format import NumberFormat, NumberFormatType +from slayer.core.formula import parse_formula from slayer.core.models import Column, ModelJoin, ModelMeasure, SlayerModel from slayer.core.refs import IDENTIFIER_RE as _IDENTIFIER_RE from slayer.dbt.entities import EntityRegistry -from slayer.dbt.filters import convert_dbt_filter +from slayer.dbt.filters import _DIMENSION_RE, convert_dbt_filter from slayer.dbt.models import ( + DbtConfig, + DbtConversionTypeParams, DbtDimension, DbtMeasure, DbtMetric, + DbtMetricInput, + DbtMetricTimeWindow, DbtMetricTypeParams, DbtProject, DbtRegularModel, @@ -38,16 +44,25 @@ ) from slayer.dbt.sql_resolver import resolve_refs from slayer.engine.ingestion import introspect_table_to_model +# DEV-1643: the conversion-report types are shared with the OSI importer. They +# live in the neutral ``slayer.ingest_report`` module and are re-exported here so +# existing ``from slayer.dbt.converter import ConversionResult`` imports keep +# working against the same class objects. +from slayer.ingest_report import ConversionResult, ConversionWarning logger = logging.getLogger(__name__) # Map dbt aggregation names to SLayer aggregation names -_AGG_MAP: Dict[str, str] = { +_AGG_MAP: dict[str, str] = { "sum": "sum", "average": "avg", "avg": "avg", "count": "count", "count_distinct": "count_distinct", + # DEV-1595: SLayer-added dialect-aware approximate-distinct. Not in + # MetricFlow's AggregationType enum — mapped defensively for non-canonical + # / legacy inputs (e.g. dbt-to-cube's countDistinctApprox). + "count_distinct_approx": "count_distinct_approx", "min": "min", "max": "max", "median": "median", @@ -57,6 +72,21 @@ _FLOAT_FORMAT = NumberFormat(type=NumberFormatType.FLOAT) +# Standard SLayer time granularities accepted in offset_window shifts. Custom / +# non-standard granularities (e.g. "fortnight") are routed to a clean-fail. +_STANDARD_GRAINS: frozenset = frozenset({ + "second", "minute", "hour", "day", "week", "month", "quarter", "year", +}) + +# Dialects with no GROUP-BY percentile / median aggregate. A percentile/median +# measure imports fine but fails at query time on these, so the converter emits +# an info-level caveat when ``target_dialect`` is one of them. +_NO_PERCENTILE_DIALECTS: frozenset = frozenset({"mysql", "tsql", "mssql", "sqlserver"}) + +# Shared workaround text for an unreachable cross-model filter (used by the +# simple / ratio / derived push-down paths). +_JOIN_REACHABILITY_SUGGESTION = "Add the required join, or filter on a local dimension." + class DbtConversionError(Exception): """Raised when a dbt project cannot be converted to SLayer shape. @@ -66,20 +96,6 @@ class DbtConversionError(Exception): """ -class ConversionWarning(BaseModel): - """A warning or info message from the conversion process.""" - model_name: Optional[str] = None - metric_name: Optional[str] = None - message: str - - -class ConversionResult(BaseModel): - """Result of converting a DbtProject to SLayer representations.""" - models: List[SlayerModel] = Field(default_factory=list) - unconverted_metrics: List[ConversionWarning] = Field(default_factory=list) - warnings: List[ConversionWarning] = Field(default_factory=list) - - def _map_agg(dbt_agg: str) -> str: """Map a dbt aggregation name to a SLayer aggregation name.""" mapped = _AGG_MAP.get(dbt_agg.lower()) @@ -94,6 +110,13 @@ def _is_simple_identifier(s: str) -> bool: return bool(_IDENTIFIER_RE.match(s)) +def _meta_of(config: DbtConfig | None) -> dict[str, Any] | None: + """Extract ``config.meta`` (or ``None``).""" + if config is not None and config.meta: + return dict(config.meta) + return None + + def _convert_dimension(dim: DbtDimension) -> Column: """Convert a dbt dimension to a SLayer column.""" if dim.type == "time": @@ -109,90 +132,10 @@ def _convert_dimension(dim: DbtDimension) -> Column: type=data_type, description=dim.description, label=dim.label, + meta=_meta_of(dim.config), ) -def _convert_measures( - dbt_measures: List[DbtMeasure], - *, - sm_name: str, - existing_column_names: set, - unconverted: List[ConversionWarning], -) -> Tuple[List[Column], List[ModelMeasure]]: - """Convert dbt measures into a (Columns, ModelMeasures) pair. - - Each unique measure expression yields a single ``Column`` whose name is - either the bare expression (when it is a SQL identifier) or - ``_col`` (when the expression is a SQL fragment - like ``amount * quantity``). Each dbt measure yields one ``ModelMeasure`` - whose formula is ``:``. label and description live on the - ``ModelMeasure`` only — they belong to the named formula, not the raw - column. - - Column-name collisions with already-emitted dimensions/entities or with - any of the about-to-be-emitted ``ModelMeasure`` names are resolved by - suffixing the Column with ``_col`` (Q-2 / Q-I in the S4 spec). - - A ``ModelMeasure`` whose name shadows a built-in transform (e.g. - ``cumsum``) is rejected by the Pydantic validator; the dbt measure is - routed to ``unconverted`` and skipped. - """ - groups: Dict[str, List[DbtMeasure]] = defaultdict(list) - for m in dbt_measures: - key = m.expr or m.name - groups[key].append(m) - - # All dbt-measure names, taken to be the eventual ``ModelMeasure`` names. - measure_names = {m.name for m in dbt_measures} - - columns: List[Column] = [] - measures: List[ModelMeasure] = [] - used_column_names = set(existing_column_names) - - for expr_key, group in groups.items(): - if _is_simple_identifier(expr_key): - base_name = expr_key - else: - base_name = f"{group[0].name}_col" - - col_name = base_name - # Q-2: avoid collision with any ModelMeasure name in this model. - # Q-I: also avoid collision with the dimensions/entities already on - # the model, by suffixing ``_col`` until unique. - while col_name in measure_names or col_name in used_column_names: - col_name = f"{col_name}_col" - used_column_names.add(col_name) - - sql = expr_key if expr_key != col_name else None - columns.append(Column( - name=col_name, - sql=sql, - type=DataType.DOUBLE, - format=_FLOAT_FORMAT, - )) - - for m in group: - mapped_agg = _map_agg(m.agg) - try: - measures.append(ModelMeasure( - name=m.name, - formula=f"{col_name}:{mapped_agg}", - label=m.label, - description=m.description, - )) - except ValueError as exc: - unconverted.append(ConversionWarning( - model_name=sm_name, - metric_name=m.name, - message=( - f"dbt measure '{m.name}' could not be converted to a " - f"ModelMeasure: {exc}" - ), - )) - - return columns, measures - - class DbtToSlayerConverter: """Convert a DbtProject into SLayer models.""" @@ -200,23 +143,29 @@ def __init__( self, project: DbtProject, data_source: str, - sa_engine: Optional[sa.Engine] = None, + sa_engine: sa.Engine | None = None, include_hidden_models: bool = False, + target_dialect: str | None = None, ) -> None: self.project = project self.data_source = data_source self.sa_engine = sa_engine self.include_hidden_models = include_hidden_models + # DEV-1595: when set to a dialect that lacks percentile/median + # (mysql / tsql), the converter emits info caveats for those measures. + self.target_dialect = target_dialect self.entity_registry = EntityRegistry() - self._warnings: List[ConversionWarning] = [] - self._unconverted: List[ConversionWarning] = [] + self._warnings: list[ConversionWarning] = [] + self._unconverted: list[ConversionWarning] = [] # {model_name: SlayerModel} for metric resolution - self._models_by_name: Dict[str, SlayerModel] = {} + self._models_by_name: dict[str, SlayerModel] = {} # {model_name: DbtSemanticModel} for looking up entities - self._dbt_models_by_name: Dict[str, DbtSemanticModel] = {} + self._dbt_models_by_name: dict[str, DbtSemanticModel] = {} + # Filtered-column dedup: {(model, column_expr, normalized_filter): col_name} + self._filtered_columns: dict[tuple[str, str, str | None], str] = {} # {regular_model_name: raw_code} — used to inline SQL into semantic # models whose underlying dbt model is a query rather than a table. - self._regular_models_sql: Dict[str, str] = { + self._regular_models_sql: dict[str, str] = { rm.name: rm.raw_code for rm in project.regular_models if rm.raw_code @@ -229,7 +178,7 @@ def convert(self) -> ConversionResult: for sm in self.project.semantic_models: self._dbt_models_by_name[sm.name] = sm - models: List[SlayerModel] = [] + models: list[SlayerModel] = [] for sm in self.project.semantic_models: model = self._convert_semantic_model(sm) models.append(model) @@ -238,6 +187,7 @@ def convert(self) -> ConversionResult: for metric in self.project.metrics: self._convert_metric(metric) + self._prune_dangling_measures() self._mirror_inner_joins() if self.include_hidden_models and self.project.regular_models: @@ -270,10 +220,56 @@ def _mirror_inner_joins(self) -> None: join_type=JoinType.INNER, )) - def _convert_regular_models(self, existing_names: set) -> List[SlayerModel]: + def _prune_dangling_measures(self) -> None: + """Drop+report any ``ModelMeasure`` whose formula references a name that + does not resolve on its model (DEV-1595 robust validation pass). + + A derived / ratio metric whose input metric was itself clean-failed + (measure-less, time-spine gap-fill, unreachable filter, filtered-leaf + clean-fail, or a transitively-dropped dependency) leaves a bare formula + reference to a measure that was never materialized. Rather than predict + every such case inline at conversion time, this final pass validates the + emitted formulas against the actual model and removes the ones that + can't resolve — running to a fixpoint so a measure depending on a + just-dropped one is dropped too. + """ + for model in self._models_by_name.values(): + self._prune_model_measures(model) + + def _prune_model_measures(self, model: SlayerModel) -> None: + agg_names = frozenset(a.name for a in model.aggregations) + changed = True + while changed: + changed = False + named = {m.name: m.formula for m in model.measures if m.name} + survivors: list[ModelMeasure] = [] + for m in model.measures: + others = {k: v for k, v in named.items() if k != m.name} + try: + parse_formula(m.formula, extra_agg_names=agg_names, named_measures=others) + survivors.append(m) + except (ValueError, RecursionError) as exc: + self._unconverted.append(ConversionWarning( + model_name=model.name, + metric_name=m.name, + category="dangling_reference", + severity="dropped", + message=( + f"Measure '{m.name}' references a name that does not resolve " + f"on model '{model.name}' and was dropped: {exc}" + ), + suggestion="Ensure every referenced metric/measure converts successfully.", + )) + changed = True + if changed: + model.measures = survivors + + def _convert_regular_models(self, existing_names: set) -> list[SlayerModel]: """Convert orphan dbt models (not wrapped by semantic_models) to hidden SLayer models.""" if self.sa_engine is None: self._warnings.append(ConversionWarning( + category="hidden_models", + severity="info", message=( "include_hidden_models=True but no SQLAlchemy engine was provided; " "skipping regular-model import." @@ -283,7 +279,7 @@ def _convert_regular_models(self, existing_names: set) -> List[SlayerModel]: engine = self.sa_engine inspector = sa.inspect(engine) - results: List[SlayerModel] = [] + results: list[SlayerModel] = [] for rm in self.project.regular_models: if rm.name in existing_names: continue @@ -298,7 +294,7 @@ def _convert_regular_model( rm: DbtRegularModel, sa_engine: sa.Engine, inspector: sa.engine.Inspector, - ) -> Optional[SlayerModel]: + ) -> SlayerModel | None: """Introspect a regular dbt model and wrap it as a hidden SlayerModel.""" table_name = rm.alias or rm.name try: @@ -313,6 +309,8 @@ def _convert_regular_model( except Exception as exc: self._warnings.append(ConversionWarning( model_name=rm.name, + category="hidden_models", + severity="info", message=( f"Skipped hidden import of dbt model '{rm.name}' " f"(table '{table_name}'): {type(exc).__name__}: {exc}" @@ -353,8 +351,8 @@ def _convert_semantic_model(self, sm: DbtSemanticModel) -> SlayerModel: ref_name = sm.model or sm.name - sql_source: Optional[str] = None - sql_table: Optional[str] = None + sql_source: str | None = None + sql_table: str | None = None if ref_name in self._regular_models_sql: resolved, warnings = resolve_refs( self._regular_models_sql[ref_name], @@ -364,6 +362,8 @@ def _convert_semantic_model(self, sm: DbtSemanticModel) -> SlayerModel: for message in warnings: self._warnings.append(ConversionWarning( model_name=sm.name, + category="sql_inline", + severity="info", message=message, )) else: @@ -373,7 +373,16 @@ def _convert_semantic_model(self, sm: DbtSemanticModel) -> SlayerModel: if sm.defaults and sm.defaults.agg_time_dimension: default_time_dim = sm.defaults.agg_time_dimension - cols: List[Column] = [_convert_dimension(d) for d in sm.dimensions] + # DEV-1595: accumulate model-level meta (config.meta + label + any + # clean-fail raw stashes added during measure conversion). + model_meta: dict[str, Any] = {} + cfg_meta = _meta_of(sm.config) + if cfg_meta: + model_meta.update(cfg_meta) + if sm.label: + model_meta.setdefault("label", sm.label) + + cols: list[Column] = [_convert_dimension(d) for d in sm.dimensions] # Add primary key column for primary/unique entities. entity_col_names = {c.name for c in cols} @@ -386,12 +395,25 @@ def _convert_semantic_model(self, sm: DbtSemanticModel) -> SlayerModel: type=DataType.DOUBLE, primary_key=True, description=entity.description, + label=entity.label, + meta=self._entity_meta(entity), )) entity_col_names.add(col_name) else: + entity_meta = self._entity_meta(entity) for c in cols: if c.name == col_name: c.primary_key = True + # Carry the entity's metadata onto the reused column + # without clobbering anything the column already has + # (parity with the synthetic-column branch above). + if c.description is None: + c.description = entity.description + if c.label is None: + c.label = entity.label + if entity_meta: + c.meta = {**entity_meta, **(c.meta or {})} + break if sm.primary_entity: pe_name = sm.primary_entity @@ -408,11 +430,11 @@ def _convert_semantic_model(self, sm: DbtSemanticModel) -> SlayerModel: )) entity_col_names.add(pe_expr) - measure_cols, measures = _convert_measures( + measure_cols, measures = self._convert_measures( dbt_measures=sm.measures, sm_name=sm.name, existing_column_names={c.name for c in cols}, - unconverted=self._unconverted, + model_meta=model_meta, ) cols.extend(measure_cols) @@ -428,16 +450,246 @@ def _convert_semantic_model(self, sm: DbtSemanticModel) -> SlayerModel: columns=cols, measures=measures, joins=joins, + meta=model_meta or None, ) + @staticmethod + def _entity_meta(entity) -> dict[str, Any] | None: + """Build a PK-column meta blob from an entity's config.meta + role.""" + meta = _meta_of(getattr(entity, "config", None)) or {} + role = getattr(entity, "role", None) + if role: + meta.setdefault("role", role) + return meta or None + + # ── Measure conversion ──────────────────────────────────────────── + + def _convert_measures( + self, + dbt_measures: list[DbtMeasure], + *, + sm_name: str, + existing_column_names: set, + model_meta: dict[str, Any], + ) -> tuple[list[Column], list[ModelMeasure]]: + """Convert dbt measures into a (Columns, ModelMeasures) pair. + + Each unique measure expression yields a single ``Column``; each dbt + measure yields one ``ModelMeasure`` whose formula is ``:``. + Special handling (DEV-1595): + + * ``sum_boolean`` → a dedicated ``CASE WHEN () THEN 1 ELSE 0 END`` + ``INT`` column aggregated with ``:sum`` (cross-DB safe; null bool → 0). + * ``percentile`` → ``:percentile(p=)``; clean-fails when the + value is absent or discrete/approximate flags are set. + * ``non_additive_dimension`` (semi-additive) → clean-fail. + """ + measure_names = {m.name for m in dbt_measures} + columns: list[Column] = [] + measures: list[ModelMeasure] = [] + used_column_names = set(existing_column_names) + + def _alloc(base: str) -> str: + col_name = base + while col_name in measure_names or col_name in used_column_names: + col_name = f"{col_name}_col" + used_column_names.add(col_name) + return col_name + + groups: dict[str, list[DbtMeasure]] = defaultdict(list) + for m in dbt_measures: + if m.non_additive_dimension is not None: + self._fail_measure( + m, sm_name, model_meta, + category="non_additive_dimension", severity="dropped", + message=( + f"Measure '{m.name}' uses a non_additive_dimension " + f"(semi-additive aggregation), which is not exactly expressible." + ), + suggestion=( + "Express as balance:last() / first(...) " + "or a multi-stage query." + ), + raw={"non_additive_dimension": m.non_additive_dimension.model_dump()}, + ) + continue + if m.agg.lower() == "sum_boolean": + expr = m.expr or m.name + col_name = _alloc(f"{m.name}_col") + columns.append(Column( + name=col_name, + sql=f"CASE WHEN ({expr}) THEN 1 ELSE 0 END", + type=DataType.INT, + meta=_meta_of(m.config), + )) + self._emit_model_measure(measures, m, f"{col_name}:sum", sm_name) + continue + groups[m.expr or m.name].append(m) + + for expr_key, group in groups.items(): + if _is_simple_identifier(expr_key): + base_name = expr_key + else: + base_name = f"{group[0].name}_col" + col_name = _alloc(base_name) + sql = expr_key if expr_key != col_name else None + columns.append(Column( + name=col_name, + sql=sql, + type=DataType.DOUBLE, + format=_FLOAT_FORMAT, + )) + for m in group: + formula = self._measure_formula(m, col_name, sm_name, model_meta) + if formula is None: + continue + self._emit_model_measure(measures, m, formula, sm_name) + + return columns, measures + + def _measure_formula( + self, m: DbtMeasure, col_name: str, sm_name: str, model_meta: dict[str, Any] + ) -> str | None: + """Build the ``:`` formula for a dbt measure, or ``None`` if + it clean-fails (percentile without a value / discrete-approx flags).""" + mapped = _map_agg(m.agg) + if mapped == "percentile": + ap = m.agg_params + if ap is None or ap.percentile is None: + self._fail_measure( + m, sm_name, model_meta, + category="percentile", severity="dropped", + message=f"Measure '{m.name}' is a percentile aggregation with no percentile value.", + suggestion="Set agg_params.percentile (continuous, in [0, 1]).", + raw={"agg": m.agg, "agg_params": ap.model_dump() if ap else None}, + ) + return None + if ap.use_discrete_percentile or ap.use_approximate_percentile: + self._fail_measure( + m, sm_name, model_meta, + category="percentile", severity="dropped", + message=( + f"Measure '{m.name}' uses discrete/approximate percentile; " + f"only continuous-exact PERCENTILE_CONT is supported." + ), + suggestion="Remove use_discrete_percentile / use_approximate_percentile.", + raw={"agg_params": ap.model_dump()}, + ) + return None + self._maybe_dialect_caveat(m.name, "percentile") + return f"{col_name}:percentile(p={ap.percentile})" + if mapped == "median": + self._maybe_dialect_caveat(m.name, "median") + return f"{col_name}:{mapped}" + + def _emit_model_measure( + self, measures: list[ModelMeasure], m: DbtMeasure, formula: str, sm_name: str + ) -> None: + """Append a ``ModelMeasure`` for a dbt measure, routing transform-name + collisions to the report instead of raising.""" + try: + measures.append(ModelMeasure( + name=m.name, + formula=formula, + label=m.label, + description=m.description, + meta=_meta_of(m.config), + )) + except ValueError as exc: + self._unconverted.append(ConversionWarning( + model_name=sm_name, + metric_name=m.name, + category="measure", + severity="unconverted", + message=( + f"dbt measure '{m.name}' could not be converted to a " + f"ModelMeasure: {exc}" + ), + )) + + def _maybe_dialect_caveat(self, name: str, agg: str) -> None: + """Emit an info caveat when the target dialect lacks ``agg``.""" + if self.target_dialect and self.target_dialect.lower() in _NO_PERCENTILE_DIALECTS: + self._warnings.append(ConversionWarning( + metric_name=name, + category="percentile", + severity="info", + message=( + f"'{name}' uses {agg}, which is not supported on dialect " + f"'{self.target_dialect}'; the measure imports but will fail " + f"at query time." + ), + suggestion="Query on a dialect with native percentile/median support.", + )) + + # ── Clean-fail helpers ───────────────────────────────────────────── + + def _fail_measure( + self, + m: DbtMeasure, + sm_name: str, + model_meta: dict[str, Any], + *, + category: str, + severity: Literal["unconverted", "dropped", "info"], + message: str, + suggestion: str | None = None, + raw: dict[str, Any] | None = None, + ) -> None: + """Route a measure-level clean-fail to the report + stash raw into meta.""" + self._unconverted.append(ConversionWarning( + model_name=sm_name, + metric_name=m.name, + category=category, + severity=severity, + message=message, + suggestion=suggestion, + )) + if raw is not None: + self._stash_meta(model_meta, m.name, category, raw) + + def _fail_metric( + self, + metric: DbtMetric, + *, + category: str, + severity: Literal["unconverted", "dropped", "info"], + message: str, + suggestion: str | None = None, + raw: dict[str, Any] | None = None, + model_name: str | None = None, + ) -> None: + """Route a metric-level clean-fail to the report + best-effort meta stash.""" + self._unconverted.append(ConversionWarning( + model_name=model_name, + metric_name=metric.name, + category=category, + severity=severity, + message=message, + suggestion=suggestion, + )) + if raw is not None: + src = self._find_metric_source_model(metric) + slayer_model = self._models_by_name.get(src) if src else None + if slayer_model is not None: + if slayer_model.meta is None: + slayer_model.meta = {} + self._stash_meta(slayer_model.meta, metric.name, category, raw) + + @staticmethod + def _stash_meta(meta_dict: dict[str, Any], name: str, category: str, raw: Any) -> None: + """Append a raw dropped construct to ``meta['dbt_unconverted']``.""" + bucket = meta_dict.setdefault("dbt_unconverted", []) + bucket.append({"name": name, "category": category, "raw": raw}) + # ── Metric conversion ───────────────────────────────────────────── def _convert_metric(self, metric: DbtMetric) -> None: """Route a dbt metric to the appropriate handler. All handlers fold their output into a ``ModelMeasure`` on the source - semantic model (or report ``unconverted_metrics`` on failure). No - ``SlayerQuery`` is produced. + semantic model (or route to the report on failure). No ``SlayerQuery`` + is produced. """ metric_type = metric.type.lower() @@ -450,15 +702,32 @@ def _convert_metric(self, metric: DbtMetric) -> None: elif metric_type == "cumulative": self._convert_cumulative_metric(metric) elif metric_type == "conversion": - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message="Conversion metrics are not supported in SLayer. Skipped.", - )) + self._fail_metric( + metric, + category="conversion_metric", + severity="dropped", + message=f"Conversion metric '{metric.name}' (funnel) is not supported in SLayer.", + suggestion="Express the funnel as a multi-stage query.", + raw={ + "type": metric.type, + # Stash the parsed funnel details (base/conversion measure, + # entity, calculation, window) so nothing is silently lost, + # consistent with the other clean-fail branches. + "conversion_type_params": ( + metric.type_params.conversion_type_params.model_dump() + if metric.type_params + and metric.type_params.conversion_type_params + else None + ), + }, + ) else: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message=f"Unknown metric type '{metric.type}'. Skipped.", - )) + self._fail_metric( + metric, + category="unknown_metric_type", + severity="unconverted", + message=f"Unknown metric type '{metric.type}' for metric '{metric.name}'.", + ) def _add_model_measure( self, @@ -469,21 +738,11 @@ def _add_model_measure( ) -> None: """Append a ``ModelMeasure`` to ``slayer_model``. - Routes transform-name collisions (Q-F) to ``unconverted_metrics`` - instead of raising. Skips silently with a warning if the name - collides with an existing column or measure on the model. + Routes transform-name collisions to ``unconverted_metrics`` instead of + raising. Skips with a warning if the name collides with an existing + column or measure on the model. """ - existing_names = {c.name for c in slayer_model.columns} - existing_names.update(m.name for m in slayer_model.measures if m.name is not None) - if metric.name in existing_names: - self._warnings.append(ConversionWarning( - model_name=slayer_model.name, - metric_name=metric.name, - message=( - f"Metric '{metric.name}' collides with an existing column or " - f"measure on model '{slayer_model.name}'. Skipped." - ), - )) + if self._metric_name_collides(metric.name, slayer_model): return try: slayer_model.measures.append(ModelMeasure( @@ -491,259 +750,789 @@ def _add_model_measure( formula=formula, label=metric.label, description=metric.description, + meta=_meta_of(metric.config), )) except ValueError as exc: self._unconverted.append(ConversionWarning( model_name=slayer_model.name, metric_name=metric.name, + category="metric", + severity="unconverted", message=( f"Metric '{metric.name}' could not be converted to a " f"ModelMeasure: {exc}" ), )) + def _metric_name_collides(self, name: str, slayer_model: SlayerModel) -> bool: + """Whether ``name`` collides with a column/measure on ``slayer_model``. + Emits a collision warning when it does.""" + existing_names = {c.name for c in slayer_model.columns} + existing_names.update(m.name for m in slayer_model.measures if m.name is not None) + if name in existing_names: + self._warnings.append(ConversionWarning( + model_name=slayer_model.name, + metric_name=name, + category="collision", + severity="info", + message=( + f"Metric '{name}' collides with an existing column or " + f"measure on model '{slayer_model.name}'. Skipped." + ), + )) + return True + return False + + def _simple_metric_unsupported( + self, metric: DbtMetric, tp: DbtMetricTypeParams | None + ) -> bool: + """Route the unsupported simple-metric shapes (measure-less aggregation + via ``metric_aggregation_params``, time-spine gap filling) to the + report; return ``True`` when one fired.""" + if tp is None: + return False + if tp.metric_aggregation_params is not None: + self._fail_metric( + metric, + category="measure_less_metric", + severity="dropped", + message=( + f"Simple metric '{metric.name}' uses metric_aggregation_params " + f"(a measure-less aggregation), an unsupported shape." + ), + suggestion="Define an explicit measure on the semantic model.", + raw={"metric_aggregation_params": tp.metric_aggregation_params.model_dump()}, + ) + return True + mref = tp.measure + if mref and (mref.join_to_timespine or mref.fill_nulls_with is not None): + self._fail_metric( + metric, + category="timespine_gap_fill", + severity="dropped", + message=( + f"Metric '{metric.name}' uses join_to_timespine / fill_nulls_with; " + f"SLayer has no time-spine gap filling." + ), + suggestion="Remove join_to_timespine / fill_nulls_with.", + raw={"join_to_timespine": mref.join_to_timespine, + "fill_nulls_with": mref.fill_nulls_with}, + ) + return True + return False + def _convert_simple_metric(self, metric: DbtMetric) -> None: """A simple metric is a (filtered) re-aggregation of a single measure. Without a filter: nothing to do — the underlying measure is already - addressable as a ModelMeasure. With a filter: emit a Column carrying - the CASE-WHEN ``filter`` and a ModelMeasure pointing at it. + addressable as a ModelMeasure. With a filter: push it down into a leaf + Column carrying the CASE-WHEN ``filter`` and reference it from a + ModelMeasure. """ - if not metric.type_params or not metric.type_params.measure: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message="Simple metric has no measure reference. Skipped.", - )) - return + tp = metric.type_params - measure_name = metric.type_params.measure + if self._simple_metric_unsupported(metric, tp): + return - if not metric.filter: + measure_name = tp.measure_name if tp else None + if not measure_name: + self._fail_metric( + metric, + category="simple_metric", + severity="unconverted", + message=f"Simple metric '{metric.name}' has no measure reference.", + ) return + mref = tp.measure if tp else None + raw_filter = self._combine_filters(metric.filter, mref.filter if mref else None) + if not raw_filter: + return # unfiltered simple metric — the measure is already addressable. + source_sm = self._find_measure_model(measure_name) if source_sm is None: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message=f"Cannot find measure '{measure_name}' in any semantic model. Skipped.", - )) + self._fail_metric( + metric, + category="simple_metric", + severity="unconverted", + message=f"Cannot find measure '{measure_name}' in any semantic model.", + ) + return + + ok, reason = self._filter_reachable(raw_filter, source_sm) + if not ok: + self._fail_metric( + metric, + category="cross_model_filter", + severity="dropped", + message=f"Metric '{metric.name}': {reason}.", + suggestion=_JOIN_REACHABILITY_SUGGESTION, + ) return dbt_measure = next((m for m in source_sm.measures if m.name == measure_name), None) if dbt_measure is None: return - model_entities = {e.name: e.type for e in source_sm.entities} - sm_by_name = {sm.name: sm for sm in self.project.semantic_models} - slayer_filter = convert_dbt_filter( - filter_str=metric.filter, - source_model_name=source_sm.name, - entity_registry=self.entity_registry, - model_entity_names=model_entities, - all_semantic_models=sm_by_name, - ) - - mapped_agg = _map_agg(dbt_measure.agg) slayer_model = self._models_by_name.get(source_sm.name) if slayer_model is None: return - # Q-3: filtered simple metrics get a Column carrying the filter, with - # NO allowed_aggregations. The metric becomes a ModelMeasure that - # references that Column with the dbt-defined aggregation. - existing_names = {c.name for c in slayer_model.columns} - existing_names.update(m.name for m in slayer_model.measures if m.name is not None) - if metric.name in existing_names: - self._warnings.append(ConversionWarning( - model_name=slayer_model.name, - metric_name=metric.name, - message=( - f"Filtered metric '{metric.name}' collides with an existing column " - f"or measure on model '{slayer_model.name}'. Skipped." - ), - )) + if self._metric_name_collides(metric.name, slayer_model): return - col_name = f"{metric.name}_col" - while col_name in existing_names: - col_name = f"{col_name}_col" - - underlying_sql = ( - dbt_measure.expr - if dbt_measure.expr and dbt_measure.expr != dbt_measure.name - else dbt_measure.name + leaf_ref = self._filtered_leaf_ref( + metric=metric, + slayer_model=slayer_model, + source_sm=source_sm, + dbt_measure=dbt_measure, + raw_filter=raw_filter, ) - slayer_model.columns.append(Column( - name=col_name, - sql=underlying_sql, - type=DataType.DOUBLE, - format=_FLOAT_FORMAT, - filter=slayer_filter, - )) - try: - slayer_model.measures.append(ModelMeasure( - name=metric.name, - formula=f"{col_name}:{mapped_agg}", - label=metric.label, - description=metric.description or f"Filtered metric: {metric.name}", - )) - except ValueError as exc: - # Roll back the column we just appended so the model stays consistent. - slayer_model.columns.pop() - self._unconverted.append(ConversionWarning( - model_name=slayer_model.name, - metric_name=metric.name, - message=( - f"Filtered metric '{metric.name}' could not be converted: {exc}" - ), - )) + if leaf_ref is None: + return # clean-failed (e.g. filtered percentile without a value) + self._add_model_measure(slayer_model=slayer_model, metric=metric, formula=leaf_ref) def _convert_derived_metric(self, metric: DbtMetric) -> None: """A derived metric expresses a formula over other metrics/measures. - The ``ModelMeasure.formula`` references inputs by **bare name** — - either another ``ModelMeasure`` on the same model (which the formula - parser resolves) or a column-with-aggregation when bare names cannot - be located locally. + Input references are substituted in the ``expr``; an ``offset_window`` + on a single-aggregate input is lowered to a ``time_shift`` call + (DEV-1595). Inexpressible shapes (offset_to_grain, offset on a + multi-aggregate input, custom granularity, metric-level filter on a + derived expr) clean-fail. """ - if not metric.type_params: + tp = metric.type_params + if not tp: return - expr = metric.type_params.expr + expr = tp.expr if not expr: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message="Derived metric has no expr. Skipped.", - )) + self._fail_metric( + metric, category="derived_metric", severity="unconverted", + message=f"Derived metric '{metric.name}' has no expr.", + ) return - formula = expr - if metric.type_params.metrics: - for m_input in metric.type_params.metrics: - ref_name = m_input.alias or m_input.name - resolved = self._resolve_metric_to_name(m_input.name) - if resolved and resolved != ref_name: - formula = re.sub( - rf"\b{re.escape(ref_name)}\b", - resolved.replace("\\", r"\\"), - formula, - ) + if metric.filter: + self._fail_metric( + metric, category="filter_pushdown", severity="dropped", + message=( + f"Derived metric '{metric.name}' has a metric-level filter; " + f"leaf push-down across a derived expression is not supported." + ), + suggestion="Push the filter into the input metrics instead.", + ) + return + + formula, clean_failed = self._substitute_derived_inputs(metric, tp, expr) + if clean_failed: + return source_model_name = self._find_metric_source_model(metric) if source_model_name is None: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message=f"Could not determine source model for derived metric '{metric.name}'. Skipped.", - )) + self._fail_metric( + metric, category="derived_metric", severity="unconverted", + message=f"Could not determine source model for derived metric '{metric.name}'.", + ) return slayer_model = self._models_by_name.get(source_model_name) if slayer_model is None: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, + self._fail_metric( + metric, category="derived_metric", severity="unconverted", message=( f"Source model '{source_model_name}' for derived metric " - f"'{metric.name}' was not converted. Skipped." + f"'{metric.name}' was not converted." ), - )) + ) return - self._add_model_measure( - slayer_model=slayer_model, + self._add_model_measure(slayer_model=slayer_model, metric=metric, formula=formula) + + def _substitute_derived_inputs( + self, metric: DbtMetric, tp: DbtMetricTypeParams, expr: str + ) -> tuple[str, bool]: + """Substitute each derived input ref in ``expr`` with its resolved form. + + Returns ``(formula, clean_failed)``; ``clean_failed=True`` means an + input routed a report entry and the metric should be abandoned. + """ + formula = expr + for m_input in tp.metrics or []: + ref_name = m_input.alias or m_input.name + replacement, clean_failed = self._derived_input_replacement(metric, m_input) + if clean_failed: + return formula, True + if replacement and replacement != ref_name: + formula = re.sub( + rf"\b{re.escape(ref_name)}\b", + replacement.replace("\\", r"\\"), + formula, + ) + return formula, False + + def _offset_window_ref(self, metric: DbtMetric, m_input: DbtMetricInput) -> str | None: + """Lower an ``offset_window`` input to ``time_shift(, -N, '')``. + + Only when the input resolves to a single aggregate (measure / simple + metric) and the granularity is standard; otherwise clean-fail. + """ + window = DbtMetricTimeWindow.parse(m_input.offset_window) + if window is None: + self._fail_metric( + metric, category="offset_window", severity="dropped", + message=( + f"Derived metric '{metric.name}': could not parse offset_window " + f"'{m_input.offset_window}'." + ), + ) + return None + + gran = window.granularity.lower() + if gran not in _STANDARD_GRAINS: + self._fail_metric( + metric, category="custom_granularity", severity="dropped", + message=( + f"Derived metric '{metric.name}': offset_window uses non-standard " + f"granularity '{window.granularity}'." + ), + suggestion="Use a standard granularity (day/week/month/quarter/year).", + ) + return None + + leaf = self._resolve_input_to_leaf(m_input.name) + if leaf is None: + self._fail_metric( + metric, category="offset_window", severity="dropped", + message=( + f"Derived metric '{metric.name}': offset_window on input " + f"'{m_input.name}', which is not a single aggregate " + f"(measure / simple metric); not exactly expressible." + ), + suggestion="Offset only single-aggregate inputs.", + ) + return None + + resolved_input = self._resolve_metric_to_name(m_input.name) + if not resolved_input: + _, dbt_measure = leaf + resolved_input = dbt_measure.name + return f"time_shift({resolved_input}, -{window.count}, '{gran}')" + + def _derived_input_replacement( + self, metric: DbtMetric, m_input: DbtMetricInput + ) -> tuple[str | None, bool]: + """Resolve one derived-metric input to its formula replacement. + + Returns ``(replacement, clean_failed)``. ``clean_failed=True`` means a + report entry was emitted and the whole metric should be abandoned; + ``replacement is None`` with ``clean_failed=False`` means no + substitution is needed (the input ref already matches its name). + Handles ``offset_to_grain`` (clean-fail), ``offset_window`` → + ``time_shift``, per-input ``filter`` → leaf push-down, and the + unsupported offset+filter combination (clean-fail). + """ + ref_name = m_input.alias or m_input.name + if m_input.offset_to_grain is not None: + self._fail_metric( + metric, category="offset_to_grain", severity="dropped", + message=( + f"Derived metric '{metric.name}': input '{ref_name}' uses " + f"offset_to_grain; SLayer has no truncate-to-grain shift." + ), + suggestion="Use cumsum(...) and put the grain dimension in the query.", + ) + return None, True + + has_offset = m_input.offset_window is not None + has_filter = bool(m_input.filter) + if has_offset and has_filter: + self._fail_metric( + metric, category="filter_pushdown", severity="dropped", + message=( + f"Derived metric '{metric.name}': input '{ref_name}' combines " + f"offset_window with a per-input filter; not exactly expressible." + ), + suggestion="Split the offset and the filter into separate inputs.", + ) + return None, True + if has_offset: + rep = self._offset_window_ref(metric, m_input) + return rep, rep is None + if has_filter: + rep = self._derived_filtered_input_ref(metric, m_input) + return rep, rep is None + # A plain reference to an input metric that was itself clean-failed (and + # so never materialized) leaves a dangling bare name in the formula; the + # post-conversion _prune_dangling_measures pass catches and reports it. + return self._resolve_metric_to_name(m_input.name), False + + def _derived_filtered_input_ref( + self, metric: DbtMetric, m_input: DbtMetricInput + ) -> str | None: + """Push a derived input's per-input filter into its single-aggregate + leaf, returning the filtered colon-form ref (or ``None`` on clean-fail). + """ + leaf = self._resolve_input_to_leaf_filtered(m_input.name) + if leaf is None: + self._fail_metric( + metric, category="filter_pushdown", severity="dropped", + message=( + f"Derived metric '{metric.name}': input '{m_input.name}' carries a " + f"filter but is not a single aggregate; not exactly expressible." + ), + suggestion="Filter a simple-aggregate input, or use a multi-stage model.", + ) + return None + source_sm, dbt_measure, chain_filter = leaf + # Intersect the input's filter with any filter the referenced simple + # metric already carries, so the referenced metric's filter isn't lost. + raw_filter = self._combine_filters(chain_filter, m_input.filter) + ok, reason = self._filter_reachable(raw_filter, source_sm) + if not ok: + self._fail_metric( + metric, category="cross_model_filter", severity="dropped", + message=f"Derived metric '{metric.name}': {reason}.", + suggestion=_JOIN_REACHABILITY_SUGGESTION, + ) + return None + slayer_model = self._models_by_name.get(source_sm.name) + if slayer_model is None: + return None + return self._filtered_leaf_ref( metric=metric, - formula=formula, + slayer_model=slayer_model, + source_sm=source_sm, + dbt_measure=dbt_measure, + raw_filter=raw_filter, ) def _convert_ratio_metric(self, metric: DbtMetric) -> None: - """A ratio metric is numerator / denominator over two measures/metrics.""" - if not metric.type_params: + """A ratio metric is numerator / denominator over two measures/metrics. + + The denominator is NULL-guarded (``nullif(den, 0)``). Metric-level and + per-input filters push down independently into each leaf (DEV-1595). + """ + tp = metric.type_params + if not tp: return - num = metric.type_params.numerator - den = metric.type_params.denominator + num = tp.numerator + den = tp.denominator if not num or not den: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message="Ratio metric missing numerator or denominator. Skipped.", - )) + self._fail_metric( + metric, category="ratio_metric", severity="unconverted", + message=f"Ratio metric '{metric.name}' missing numerator or denominator.", + ) return - num_formula = self._resolve_metric_to_name(num.name) or num.name - den_formula = self._resolve_metric_to_name(den.name) or den.name - source_model_name = self._find_metric_source_model(metric) if source_model_name is None: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message=f"Could not determine source model for ratio metric '{metric.name}'. Skipped.", - )) + self._fail_metric( + metric, category="ratio_metric", severity="unconverted", + message=f"Could not determine source model for ratio metric '{metric.name}'.", + ) return slayer_model = self._models_by_name.get(source_model_name) if slayer_model is None: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, + self._fail_metric( + metric, category="ratio_metric", severity="unconverted", message=( f"Source model '{source_model_name}' for ratio metric " - f"'{metric.name}' was not converted. Skipped." + f"'{metric.name}' was not converted." ), - )) + ) + return + + num_ref = self._ratio_side_ref(metric, num, slayer_model) + if num_ref is None: + return + den_ref = self._ratio_side_ref(metric, den, slayer_model) + if den_ref is None: return self._add_model_measure( slayer_model=slayer_model, metric=metric, - formula=f"{num_formula} / {den_formula}", + formula=f"{num_ref} / nullif({den_ref}, 0)", + ) + + def _ratio_side_ref( + self, metric: DbtMetric, side: DbtMetricInput, slayer_model: SlayerModel + ) -> str | None: + """Resolve one ratio side to a formula reference, pushing down the + combined (metric-level + per-input) filter into a leaf when present.""" + raw_filter = self._combine_filters(metric.filter, side.filter) + if not raw_filter: + # A ratio side referencing a clean-failed (non-materialized) metric + # leaves a dangling formula name that _prune_dangling_measures drops. + return self._resolve_metric_to_name(side.name) or side.name + + leaf = self._resolve_input_to_leaf_filtered(side.name) + if leaf is None: + self._fail_metric( + metric, category="filter_pushdown", severity="dropped", + message=( + f"Ratio metric '{metric.name}': input '{side.name}' is a " + f"ratio/derived (multi-aggregate) metric carrying a filter; " + f"filtering after metric calculation is not exactly expressible." + ), + suggestion="Restructure as a multi-stage source_queries model.", + ) + return None + + source_sm, dbt_measure, chain_filter = leaf + # Intersect with any filter the referenced simple metric already carries + # so it isn't silently dropped. + raw_filter = self._combine_filters(chain_filter, raw_filter) + ok, reason = self._filter_reachable(raw_filter, source_sm) + if not ok: + self._fail_metric( + metric, category="cross_model_filter", severity="dropped", + message=f"Ratio metric '{metric.name}': {reason}.", + suggestion=_JOIN_REACHABILITY_SUGGESTION, + ) + return None + + return self._filtered_leaf_ref( + metric=metric, + slayer_model=slayer_model, + source_sm=source_sm, + dbt_measure=dbt_measure, + raw_filter=raw_filter, ) def _convert_cumulative_metric(self, metric: DbtMetric) -> None: - """A cumulative metric is a running total of one underlying measure.""" - if not metric.type_params or not metric.type_params.measure: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message="Cumulative metric has no measure reference. Skipped.", - )) + """A cumulative metric is a running total of one underlying measure. + + Only the *unbounded* form (``period_agg=first``, no window, no + grain_to_date) maps to ``cumsum(measure)``. Windowed / grain-to-date / + non-default-period-agg variants are query-grain-dependent and clean-fail. + """ + tp = metric.type_params + if not tp: + self._fail_metric( + metric, category="cumulative_metric", severity="unconverted", + message=f"Cumulative metric '{metric.name}' has no type_params.", + ) return - measure_ref = self._resolve_measure_to_name(metric.type_params.measure) + ctp = tp.cumulative_type_params + window = tp.window or (ctp.window if ctp else None) + grain_to_date = tp.grain_to_date or (ctp.grain_to_date if ctp else None) + period_agg = ctp.period_agg if ctp else None + measure_name = tp.measure_name or (ctp.measure if ctp else None) + + if self._cumulative_clean_fail( + metric, window=window, grain_to_date=grain_to_date, period_agg=period_agg + ): + return + + if not measure_name: + self._fail_metric( + metric, category="cumulative_metric", severity="unconverted", + message=f"Cumulative metric '{metric.name}' has no measure reference.", + ) + return + + measure_ref = self._resolve_measure_to_name(measure_name) if not measure_ref: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, + self._fail_metric( + metric, category="cumulative_metric", severity="unconverted", message=( - f"Cumulative metric '{metric.name}' references unknown " - f"measure '{metric.type_params.measure}'. Skipped." + f"Cumulative metric '{metric.name}' references unknown measure " + f"'{measure_name}'." ), - )) + ) return - source_model_name = self._find_metric_source_model(metric) - if source_model_name is None: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, - message=f"Could not determine source model for cumulative metric '{metric.name}'. Skipped.", - )) + slayer_model = self._cumulative_source_model(metric, measure_name) + if slayer_model is None: return + self._add_model_measure( + slayer_model=slayer_model, + metric=metric, + formula=f"cumsum({measure_ref})", + ) + + def _cumulative_source_model( + self, metric: DbtMetric, measure_name: str + ) -> SlayerModel | None: + """Resolve the SlayerModel a cumulative metric folds into, routing the + not-found / not-converted cases to the report (returns ``None``).""" + source_model_name = self._find_metric_source_model(metric) + if source_model_name is None: + # measure lives on exactly one model — fall back to that. + sm = self._find_measure_model(measure_name) + source_model_name = sm.name if sm else None + if source_model_name is None: + self._fail_metric( + metric, category="cumulative_metric", severity="unconverted", + message=f"Could not determine source model for cumulative metric '{metric.name}'.", + ) + return None slayer_model = self._models_by_name.get(source_model_name) if slayer_model is None: - self._unconverted.append(ConversionWarning( - metric_name=metric.name, + self._fail_metric( + metric, category="cumulative_metric", severity="unconverted", message=( f"Source model '{source_model_name}' for cumulative metric " - f"'{metric.name}' was not converted. Skipped." + f"'{metric.name}' was not converted." + ), + ) + return None + return slayer_model + + def _cumulative_clean_fail( + self, + metric: DbtMetric, + *, + window: DbtMetricTimeWindow | None, + grain_to_date: str | None, + period_agg: str | None, + ) -> bool: + """Route the query-grain-dependent cumulative variants to the report. + + Returns ``True`` (and emits a report entry) for a rolling window, + grain-to-date reset, or non-default ``period_agg``; ``False`` when the + cumulative is the unbounded form SLayer maps to ``cumsum(measure)``. + """ + if window is not None: + self._fail_metric( + metric, category="windowed_cumulative", severity="dropped", + message=( + f"Cumulative metric '{metric.name}' has a rolling window; a windowed " + f"running total with period re-aggregation is not exactly expressible." + ), + suggestion="Use cumsum(measure) for an unbounded running total.", + raw={"window": window.model_dump()}, + ) + return True + if grain_to_date is not None: + self._fail_metric( + metric, category="grain_to_date_cumulative", severity="dropped", + message=( + f"Cumulative metric '{metric.name}' uses grain_to_date; reset-at-grain " + f"can't bake into a saved measure (it is query-grain-dependent)." + ), + suggestion="Use cumsum(measure) and put the grain dimension in the query.", + raw={"grain_to_date": grain_to_date}, + ) + return True + if period_agg is not None and period_agg.lower() != "first": + self._fail_metric( + metric, category="period_agg", severity="dropped", + message=( + f"Cumulative metric '{metric.name}' uses period_agg='{period_agg}'; " + f"only the default (first) running total is exactly expressible." ), + suggestion="Use the default period_agg (first) for cumsum(measure).", + raw={"period_agg": period_agg}, + ) + return True + return False + + # ── Filter push-down helpers ─────────────────────────────────────── + + @staticmethod + def _combine_filters(a: str | None, b: str | None) -> str | None: + """AND-join two raw dbt-Jinja filter strings (each parenthesised).""" + parts = [p for p in (a, b) if p] + if not parts: + return None + if len(parts) == 1: + return parts[0] + return " AND ".join(f"({p})" for p in parts) + + def _convert_filter(self, raw: str, source_sm: DbtSemanticModel) -> str: + """Convert a raw dbt-Jinja filter to a SLayer filter string.""" + model_entities = {e.name: e.type for e in source_sm.entities} + sm_by_name = {sm.name: sm for sm in self.project.semantic_models} + return convert_dbt_filter( + filter_str=raw, + source_model_name=source_sm.name, + entity_registry=self.entity_registry, + model_entity_names=model_entities, + all_semantic_models=sm_by_name, + ) + + def _filter_reachable( + self, raw: str, source_sm: DbtSemanticModel + ) -> tuple[bool, str | None]: + """Whether every ``Dimension('entity__dim')`` in ``raw`` resolves to a + filter SLayer can actually emit from ``source_sm``. + + ``convert_dbt_filter`` lowers ``Dimension('entity__dim')`` to a + **one-hop** ``.`` reference, which only + resolves when that model is **directly** joined to the source model — + i.e. the entity is declared on the source model itself. A multi-hop + filter (e.g. ``orders → customers → regions``) would need the full + ``customers__regions.dim`` join path, which the dbt filter converter + cannot produce, so it is clean-failed rather than emitted as a broken + one-hop path. (Full multi-hop cross-model filter support is tracked + separately — DEV-1445.) + + A foreign entity declared on the source but with **no joinable owner + model** is also clean-failed: ``convert_dbt_filter`` would fall back to + a bare ``dim`` name that doesn't exist on the source table. (Whether a + reachable model actually has the column isn't verified here — undeclared + dimensions are legitimately bare table columns, and column existence is + a query-time schema-drift concern, consistent with the rest of the + converter.) + """ + entity_types = {e.name: e.type for e in source_sm.entities} + for mm in _DIMENSION_RE.finditer(raw): + ok, reason = self._entity_filter_reachable( + mm.group(1), mm.group(2), source_sm, entity_types + ) + if not ok: + return False, reason + return True, None + + def _entity_filter_reachable( + self, + entity_name: str, + dim_name: str, + source_sm: DbtSemanticModel, + entity_types: dict[str, str], + ) -> tuple[bool, str | None]: + """Reachability decision for a single ``entity__dim`` filter token.""" + if entity_name == source_sm.name: + return True, None # the source table's own column + etype = entity_types.get(entity_name) + if etype in ("primary", "unique") or entity_name == source_sm.primary_entity: + return True, None # local primary/unique → bare dim on the source table + if etype == "foreign": + owners = sorted({ + m for m, _expr in self.entity_registry._primaries.get(entity_name, []) + if m != source_sm.name + }) + if not owners: + return False, ( + f"filter dimension '{entity_name}__{dim_name}' references entity " + f"'{entity_name}', which has no joinable owner model" + ) + if len(owners) > 1: + # ``convert_dbt_filter`` qualifies the filter to a single owner + # (the lexicographically first), so a multi-owner entity would + # be lowered to a possibly-wrong model. Clean-fail rather than + # emit an ambiguously-qualified filter. + return False, ( + f"filter dimension '{entity_name}__{dim_name}' references entity " + f"'{entity_name}', which is owned by multiple models {owners}; " + f"the filter cannot be unambiguously qualified to one join" + ) + return True, None # one-hop join → . + return False, ( + f"filter dimension '{entity_name}__{dim_name}' is not reachable from " + f"model '{source_sm.name}' via a direct join (multi-hop cross-model " + f"filters are not exactly expressible)" + ) + + def _filtered_leaf_ref( + self, + *, + metric: DbtMetric, + slayer_model: SlayerModel, + source_sm: DbtSemanticModel, + dbt_measure: DbtMeasure, + raw_filter: str, + ) -> str | None: + """Get-or-create the filtered leaf Column for ``(model, expr, filter)`` + and return the colon-form formula referencing it (or ``None`` on a + clean-fail that has already been routed to the report). + + Dedup key is ``(model, column_expr, normalized_filter)`` — the + aggregation lives on the formula, so multiple aggregations over the + same filtered column share one Column. Special measure forms are + preserved exactly: ``sum_boolean`` builds the CASE-WHEN INT column and + aggregates with ``:sum``; ``percentile`` keeps its ``p=`` argument + (and clean-fails on a missing value / discrete / approximate flags). + """ + leaf = self._filtered_leaf_spec(metric, dbt_measure) + if leaf is None: + return None # clean-failed inside _filtered_leaf_spec + column_expr, col_type, col_format, agg_call = leaf + + slayer_filter = self._convert_filter(raw_filter, source_sm) + key = (slayer_model.name, column_expr, slayer_filter) + col_name = self._filtered_columns.get(key) + if col_name is None: + col_name = self._alloc_column_name(slayer_model, f"{dbt_measure.name}_filtered") + slayer_model.columns.append(Column( + name=col_name, + sql=column_expr, + type=col_type, + format=col_format, + filter=slayer_filter, )) - return + self._filtered_columns[key] = col_name + return f"{col_name}:{agg_call}" + + def _filtered_leaf_spec( + self, metric: DbtMetric, dbt_measure: DbtMeasure + ) -> tuple[str, DataType, NumberFormat | None, str] | None: + """Compute ``(column_expr, type, format, agg_call)`` for a filtered + leaf, preserving special measure semantics; ``None`` on clean-fail.""" + if dbt_measure.non_additive_dimension is not None: + # A semi-additive measure can't be lowered to a plain filtered + # aggregate — that would drop the non-additive semantics. This is the + # choke point for both the simple-metric-filter path and the + # derived/ratio push-down path. + self._fail_metric( + metric, category="non_additive_dimension", severity="dropped", + message=( + f"Filtered metric '{metric.name}' wraps a non-additive " + f"(semi-additive) measure '{dbt_measure.name}', which is not " + f"exactly expressible as a filtered aggregate." + ), + suggestion=( + "Express as balance:last() / first(...) or a " + "multi-stage query." + ), + raw={"non_additive_dimension": dbt_measure.non_additive_dimension.model_dump()}, + ) + return None + agg = dbt_measure.agg.lower() + if agg == "sum_boolean": + expr = dbt_measure.expr or dbt_measure.name + return f"CASE WHEN ({expr}) THEN 1 ELSE 0 END", DataType.INT, None, "sum" - self._add_model_measure( - slayer_model=slayer_model, - metric=metric, - formula=f"cumsum({measure_ref})", + column_expr = ( + dbt_measure.expr + if (dbt_measure.expr and dbt_measure.expr != dbt_measure.name) + else dbt_measure.name ) + mapped = _map_agg(dbt_measure.agg) + if mapped == "percentile": + ap = dbt_measure.agg_params + if ap is None or ap.percentile is None or ap.use_discrete_percentile or ap.use_approximate_percentile: + self._fail_metric( + metric, category="percentile", severity="dropped", + message=( + f"Filtered metric '{metric.name}' wraps a percentile measure " + f"with no usable continuous percentile value." + ), + suggestion="Set agg_params.percentile (continuous, in [0, 1]).", + ) + return None + return column_expr, DataType.DOUBLE, _FLOAT_FORMAT, f"percentile(p={ap.percentile})" + return column_expr, DataType.DOUBLE, _FLOAT_FORMAT, mapped + + @staticmethod + def _alloc_column_name(slayer_model: SlayerModel, base: str) -> str: + used = {c.name for c in slayer_model.columns} + used |= {m.name for m in slayer_model.measures if m.name} + name = base + while name in used: + name = f"{name}_col" + return name # ── Resolution helpers ──────────────────────────────────────────── - def _find_measure_model(self, measure_name: str) -> Optional[DbtSemanticModel]: + def _find_measure_model(self, measure_name: str) -> DbtSemanticModel | None: """Find which dbt semantic model contains a given measure.""" for sm in self.project.semantic_models: for m in sm.measures: @@ -751,28 +1540,74 @@ def _find_measure_model(self, measure_name: str) -> Optional[DbtSemanticModel]: return sm return None - def _find_metric_source_model(self, metric: DbtMetric) -> Optional[str]: + def _resolve_input_to_leaf( + self, name: str + ) -> tuple[DbtSemanticModel, DbtMeasure] | None: + """Resolve a ratio/derived input to its single-aggregate leaf measure, + or ``None`` when it's a multi-aggregate (ratio/derived/cumulative) metric.""" + res = self._resolve_input_to_leaf_filtered(name) + return (res[0], res[1]) if res else None + + def _resolve_input_to_leaf_filtered( + self, name: str + ) -> tuple[DbtSemanticModel, DbtMeasure, str | None] | None: + """Like :meth:`_resolve_input_to_leaf`, but also accumulates the raw + filter(s) encountered along the resolution chain. + + When an input names a *filtered* simple metric (its own ``filter`` or a + ``measure.filter``), that filter must be intersected with any additional + per-input / metric-level filter during push-down — otherwise the + referenced metric's filter is silently dropped, widening results. + Returns ``(source_model, leaf_measure, accumulated_raw_filter)`` or + ``None`` for a multi-aggregate input. + """ + sm = self._find_measure_model(name) + if sm is not None: + dbt_measure = next((m for m in sm.measures if m.name == name), None) + if dbt_measure is not None: + return sm, dbt_measure, None + metric = next((m for m in self.project.metrics if m.name == name), None) + if metric is None: + return None + tp = metric.type_params + if (metric.type or "").lower() == "simple" and tp and tp.measure_name: + return self._resolve_simple_metric_leaf(metric) + return None # ratio / derived / cumulative / conversion → multi-aggregate + + def _resolve_simple_metric_leaf( + self, mtc: DbtMetric + ) -> tuple[DbtSemanticModel, DbtMeasure, str | None] | None: + """Resolve a *simple* metric input to its filtered leaf, accumulating + the metric's own filter. Unsupported shapes (measure-less, time-spine) + return ``None`` so the push-down clean-fails rather than resurrecting + them as plain aggregates.""" + tp = mtc.type_params + mref = tp.measure + if tp.metric_aggregation_params is not None: + return None + if mref and (mref.join_to_timespine or mref.fill_nulls_with is not None): + return None + inner = self._resolve_input_to_leaf_filtered(tp.measure_name) + if inner is None: + return None + inner_sm, inner_measure, inner_filter = inner + own_filter = self._combine_filters(mtc.filter, mref.filter if mref else None) + return inner_sm, inner_measure, self._combine_filters(own_filter, inner_filter) + + def _find_metric_source_model(self, metric: DbtMetric) -> str | None: """Determine the source model for a metric. - Walks ``measure``, ``metrics``, and ``numerator``/``denominator`` and - returns the unique source semantic-model name. When the metric's - inputs span multiple semantic models, returns ``None`` so the caller - routes the metric to ``unconverted_metrics`` rather than silently - anchoring it to whichever model is discovered first. + Walks ``measure``, ``metrics``, ``numerator``/``denominator``, and the + nested cumulative measure, and returns the unique source model name — + or ``None`` when inputs span multiple models. """ if metric.type_params is None: return None sources = self._collect_metric_sources_from_params(metric.type_params) return next(iter(sources)) if len(sources) == 1 else None - def _collect_metric_sources(self, metric_name: str, _seen: Optional[set] = None) -> set: - """Collect every distinct semantic-model name a metric ultimately resolves to. - - Recurses through derived (``metrics``) and ratio - (``numerator``/``denominator``) inputs. Falls back to looking - ``metric_name`` up as a dbt measure when no metric of that name - exists. ``_seen`` guards against pathological metric cycles. - """ + def _collect_metric_sources(self, metric_name: str, _seen: set | None = None) -> set: + """Collect every distinct semantic-model name a metric ultimately resolves to.""" seen = _seen if _seen is not None else set() if metric_name in seen: return set() @@ -789,58 +1624,82 @@ def _collect_metric_sources(self, metric_name: str, _seen: Optional[set] = None) return {sm.name} if sm else set() def _collect_metric_sources_from_params( - self, type_params: DbtMetricTypeParams, *, seen: Optional[set] = None + self, type_params: DbtMetricTypeParams, *, seen: set | None = None ) -> set: """Shared shape-walker used by both entry points above.""" sources: set = set() - if type_params.measure: - sm = self._find_measure_model(type_params.measure) - if sm: - sources.add(sm.name) - if type_params.metrics: - for m_input in type_params.metrics: - sources |= self._collect_metric_sources(m_input.name, _seen=seen) + self._add_measure_source(sources, type_params.measure_name) + ctp = type_params.cumulative_type_params + if ctp: + self._add_measure_source(sources, ctp.measure) + self._add_conversion_sources(sources, type_params.conversion_type_params, seen) + for m_input in type_params.metrics or (): + sources |= self._collect_metric_sources(m_input.name, _seen=seen) for side in (type_params.numerator, type_params.denominator): - if side is None: - continue - sources |= self._collect_metric_sources(side.name, _seen=seen) + if side is not None: + sources |= self._collect_metric_sources(side.name, _seen=seen) return sources - def _resolve_metric_to_name(self, metric_name: str) -> Optional[str]: - """Resolve a metric name to a formula reference. + def _add_measure_source(self, sources: set, measure_name: str | None) -> None: + """Resolve a measure name to its owning model and add it to ``sources``.""" + if not measure_name: + return + sm = self._find_measure_model(measure_name) + if sm: + sources.add(sm.name) - Returns the bare ``ModelMeasure`` name when the metric was lowered - into a ``ModelMeasure`` (filtered simple, derived, ratio, cumulative). - For an *unfiltered* simple metric — which ``_convert_simple_metric`` - deliberately does not materialize — resolves to the backing dbt - measure name instead, since that is what's actually addressable on - the model. Falls back to ``_resolve_measure_to_name`` when - ``metric_name`` is a dbt measure rather than a metric. + def _add_conversion_sources( + self, sources: set, conv: DbtConversionTypeParams | None, seen: set | None + ) -> None: + """Add the source models of a conversion (funnel) metric's base/conversion + measures and metric refs.""" + if conv is None: + return + for meas in (conv.base_measure, conv.conversion_measure): + if meas: + self._add_measure_source(sources, meas.name) + for metric_ref in (conv.base_metric, conv.conversion_metric): + if metric_ref: + sources |= self._collect_metric_sources(metric_ref, _seen=seen) + + def _resolve_metric_to_name(self, metric_name: str) -> str | None: + """Resolve a metric name to a formula reference (bare ModelMeasure name). + + For a *plain unfiltered* simple metric — which the converter does not + materialize — resolves to the backing dbt measure name instead. Falls + back to ``_resolve_measure_to_name`` when ``metric_name`` is a measure. """ for m in self.project.metrics: if m.name != metric_name: continue - if ( - m.type - and m.type.lower() == "simple" - and not m.filter - and m.type_params is not None - and m.type_params.measure - ): - # Unfiltered simple metric was not materialized — point at - # the backing measure's ModelMeasure on its own model. - return self._resolve_measure_to_name(m.type_params.measure) + if m.type and m.type.lower() == "simple" and self._simple_metric_is_plain(m): + return self._resolve_measure_to_name(m.type_params.measure_name) return metric_name return self._resolve_measure_to_name(metric_name) - def _resolve_measure_to_name(self, measure_name: str) -> Optional[str]: - """Resolve a dbt measure name to a formula reference. - - After ``_convert_measures`` has run, the dbt measure name is the - ``ModelMeasure`` name on its semantic model. Reference it by bare - name (Q-5) so the formula parser can resolve it relative to the - current model. + @staticmethod + def _simple_metric_is_plain(m: DbtMetric) -> bool: + """Whether a simple metric is a *plain* re-aggregation that the + converter does NOT materialize as its own ``ModelMeasure`` (so a + reference collapses to the backing measure). + + It is plain only when it carries no filter at all — neither + ``metric.filter`` NOR ``type_params.measure.filter`` — and no time-spine + gap fill. A filter on either side means it was materialized as a + filtered ModelMeasure under the metric's own name, and a time-spine + metric is clean-failed; in both cases the reference must stay the + metric name, not the unfiltered base measure. """ + tp = m.type_params + if tp is None or not tp.measure_name or m.filter: + return False + mref = tp.measure + if mref and (mref.filter or mref.join_to_timespine or mref.fill_nulls_with is not None): + return False + return True + + def _resolve_measure_to_name(self, measure_name: str) -> str | None: + """Resolve a dbt measure name to a formula reference (bare name).""" sm = self._find_measure_model(measure_name) if sm is None: return None @@ -850,6 +1709,4 @@ def _resolve_measure_to_name(self, measure_name: str) -> Optional[str]: for m in slayer_model.measures: if m.name == measure_name: return measure_name - # Fallback: shouldn't happen for converted measures, but if the - # measure was routed to unconverted_metrics we have nothing to point at. return None diff --git a/slayer/dbt/entities.py b/slayer/dbt/entities.py index b3329133..e794c355 100644 --- a/slayer/dbt/entities.py +++ b/slayer/dbt/entities.py @@ -6,7 +6,6 @@ """ import logging -from typing import Dict, List, Optional, Tuple from slayer.core.enums import JoinType from slayer.core.models import ModelJoin @@ -20,9 +19,9 @@ class EntityRegistry: def __init__(self) -> None: # {entity_name: [(model_name, expr), ...]} - self._primaries: Dict[str, List[Tuple[str, str]]] = {} + self._primaries: dict[str, list[tuple[str, str]]] = {} - def build(self, models: List[DbtSemanticModel]) -> None: + def build(self, models: list[DbtSemanticModel]) -> None: """First pass: register all primary and unique entities.""" for model in models: # Check primary_entity shorthand @@ -61,7 +60,7 @@ def _register(self, entity_name: str, model_name: str, expr: str) -> None: ) self._primaries[entity_name].append((model_name, expr)) - def get_primary_model(self, entity_name: str) -> Optional[Tuple[str, str]]: + def get_primary_model(self, entity_name: str) -> tuple[str, str] | None: """Look up which model owns this entity as primary. Returns (model_name, expr) or None. When multiple models share the @@ -73,13 +72,13 @@ def get_primary_model(self, entity_name: str) -> Optional[Tuple[str, str]]: return None return min(entries, key=lambda e: e[0]) - def resolve_joins_for_model(self, model: DbtSemanticModel) -> List[ModelJoin]: + def resolve_joins_for_model(self, model: DbtSemanticModel) -> list[ModelJoin]: """For each foreign entity in the model, generate a ModelJoin to the primary model. Returns a list of ModelJoin objects. Skips entities whose primary model is the same as the current model (self-joins are not useful). """ - joins: List[ModelJoin] = [] + joins: list[ModelJoin] = [] # Dedupe by full join signature (target + FK columns) so distinct FKs # to the same target — e.g. buyer_id -> users.id AND seller_id -> users.id — # each get their own ModelJoin instead of silently collapsing. @@ -135,14 +134,14 @@ def resolve_joins_for_model(self, model: DbtSemanticModel) -> List[ModelJoin]: return joins - def resolve_entity_to_model(self, entity_name: str) -> Optional[str]: + def resolve_entity_to_model(self, entity_name: str) -> str | None: """Given an entity name, return the first model that owns it as primary.""" entry = self.get_primary_model(entity_name) if entry is None: return None return entry[0] - def get_entity_expr(self, entity_name: str) -> Optional[str]: + def get_entity_expr(self, entity_name: str) -> str | None: """Get the SQL expression for an entity's primary key column.""" entry = self.get_primary_model(entity_name) if entry is None: diff --git a/slayer/dbt/filters.py b/slayer/dbt/filters.py index 10e0e7b1..e69f4a52 100644 --- a/slayer/dbt/filters.py +++ b/slayer/dbt/filters.py @@ -12,7 +12,6 @@ import logging import re -from typing import Dict, Optional from slayer.dbt.entities import EntityRegistry from slayer.dbt.models import DbtSemanticModel @@ -36,8 +35,8 @@ def convert_dbt_filter( filter_str: str, source_model_name: str, entity_registry: EntityRegistry, - model_entity_names: Optional[Dict[str, str]] = None, - all_semantic_models: Optional[Dict[str, DbtSemanticModel]] = None, + model_entity_names: dict[str, str] | None = None, + all_semantic_models: dict[str, DbtSemanticModel] | None = None, ) -> str: """Convert a dbt Jinja filter string to a SLayer filter string. diff --git a/slayer/dbt/manifest.py b/slayer/dbt/manifest.py index d7331ee6..73ee93ef 100644 --- a/slayer/dbt/manifest.py +++ b/slayer/dbt/manifest.py @@ -14,7 +14,7 @@ import importlib.util import logging import os -from typing import Any, Dict, List, Optional, Set +from typing import Any from slayer.dbt.models import DbtColumnMeta, DbtRegularModel @@ -34,7 +34,7 @@ def _manifest_path(project_path: str) -> str: return os.path.join(project_path, "target", "manifest.json") -def _load_manifest_file(path: str) -> Optional[dict]: +def _load_manifest_file(path: str) -> dict | None: try: with open(path, encoding="utf-8") as f: return json.load(f) @@ -67,7 +67,7 @@ def _run_dbt_parse(project_path: str) -> bool: return bool(success) -def load_or_generate_manifest(project_path: str) -> Optional[dict]: +def load_or_generate_manifest(project_path: str) -> dict | None: """Return the dbt manifest dict for a project, or None. Resolution order: @@ -95,9 +95,9 @@ def load_or_generate_manifest(project_path: str) -> Optional[dict]: return _load_manifest_file(path) -def _semantic_model_referenced_nodes(manifest: dict) -> Set[str]: +def _semantic_model_referenced_nodes(manifest: dict) -> set[str]: """Collect every dbt node key referenced by any semantic_model.""" - referenced: Set[str] = set() + referenced: set[str] = set() semantic_models = manifest.get("semantic_models") or {} for sm in semantic_models.values(): # Prefer node_relation when present, fall back to depends_on.nodes @@ -108,11 +108,11 @@ def _semantic_model_referenced_nodes(manifest: dict) -> Set[str]: return referenced -def find_orphan_model_nodes(manifest: dict) -> List[dict]: +def find_orphan_model_nodes(manifest: dict) -> list[dict]: """Return manifest nodes for regular models not wrapped by any semantic_model.""" referenced = _semantic_model_referenced_nodes(manifest) nodes = manifest.get("nodes") or {} - orphans: List[dict] = [] + orphans: list[dict] = [] for node_key, node in nodes.items(): if node.get("resource_type") != "model": continue @@ -122,7 +122,7 @@ def find_orphan_model_nodes(manifest: dict) -> List[dict]: return orphans -def _column_from_manifest(raw: Dict[str, Any]) -> DbtColumnMeta: +def _column_from_manifest(raw: dict[str, Any]) -> DbtColumnMeta: return DbtColumnMeta( name=raw.get("name", ""), description=raw.get("description") or None, @@ -131,7 +131,7 @@ def _column_from_manifest(raw: Dict[str, Any]) -> DbtColumnMeta: ) -def _regular_model_from_node(node: Dict[str, Any]) -> DbtRegularModel: +def _regular_model_from_node(node: dict[str, Any]) -> DbtRegularModel: columns_raw = node.get("columns") or {} columns = [_column_from_manifest(c) for c in columns_raw.values() if c.get("name")] return DbtRegularModel( @@ -145,6 +145,6 @@ def _regular_model_from_node(node: Dict[str, Any]) -> DbtRegularModel: ) -def regular_models_from_manifest(manifest: dict) -> List[DbtRegularModel]: +def regular_models_from_manifest(manifest: dict) -> list[DbtRegularModel]: """Turn the orphan nodes of a dbt manifest into ``DbtRegularModel`` instances.""" return [_regular_model_from_node(node) for node in find_orphan_model_nodes(manifest)] diff --git a/slayer/dbt/models.py b/slayer/dbt/models.py index d0e30b12..db001a24 100644 --- a/slayer/dbt/models.py +++ b/slayer/dbt/models.py @@ -3,42 +3,113 @@ Lightweight representations of dbt's semantic_models and metrics YAML. We don't use metricflow-semantic-interfaces because it requires a Pydantic v1 compatibility shim and has heavy transitive dependencies we don't need. + +DEV-1595: parser completeness — every semantically-relevant +dbt-semantic-interfaces (DSI) field is parsed (so the converter can either +represent it or route it to a clean-fail report), never silently dropped. +Pydantic ``extra="ignore"`` is kept everywhere (DSI is forward-compatible and +adds fields over time; ``forbid`` would break on a newer manifest). """ -from typing import List, Optional +import re +from typing import Any, Optional from pydantic import BaseModel, Field, field_validator +# ───────────────────────── shared helpers ───────────────────────── + + +# Plural → singular granularity normalization for offset/window strings +# (DSI accepts ``"2 weeks"``; SLayer's transforms want ``week``). +_PLURAL_GRANULARITY_RE = re.compile(r"s$", re.IGNORECASE) + + +def _clause_to_str(clause: Any) -> str | None: + """Extract one where-clause as a string from a bare string or a DSI + ``{"where_sql_template": "..."}`` dict; ``None`` when empty.""" + if isinstance(clause, dict): + tmpl = clause.get("where_sql_template") + return str(tmpl) if tmpl else None + return str(clause) if clause else None + + +def _normalize_filter(value: Any) -> str | None: + """Normalize a DSI ``WhereFilterIntersection`` to a single filter string. + + DSI filters are a *string*, a *list of strings*, or the structured + ``{"where_filters": [{"where_sql_template": "..."}]}`` dict. SLayer carries + a single ``Optional[str]`` per filter, so multiple where-clauses are + AND-joined (each parenthesised to preserve precedence). The raw Jinja is + preserved verbatim inside each clause — ``convert_dbt_filter`` resolves it + downstream. + """ + if value is None: + return None + if isinstance(value, str): + return value or None + # Reduce the intersection (dict / list / tuple) to its clause iterable. + if isinstance(value, dict): + clauses: Any = value.get("where_filters") or [] + elif isinstance(value, (list, tuple)): + clauses = value + else: + return str(value) + + parts = [c for c in (_clause_to_str(x) for x in clauses) if c] + if not parts: + return None + if len(parts) == 1: + return parts[0] + return " AND ".join(f"({p})" for p in parts) + + +class DbtConfig(BaseModel): + """DSI ``config`` block. Only ``meta`` is semantically carried by SLayer.""" + meta: dict[str, Any] | None = None + + class DbtTimeTypeParams(BaseModel): - time_granularity: Optional[str] = None - is_partition: Optional[bool] = None + time_granularity: str | None = None + is_partition: bool | None = None class DbtNonAdditiveDimension(BaseModel): name: str window_choice: str = "min" - window_groupings: List[str] = Field(default_factory=list) + window_groupings: list[str] = Field(default_factory=list) + + +class DbtValidityParams(BaseModel): + """SCD validity-window params on a dimension (recognized, not represented).""" + is_start: bool | None = None + is_end: bool | None = None class DbtEntity(BaseModel): name: str type: str # "primary", "foreign", "unique", "natural" - expr: Optional[str] = None # defaults to name if omitted - description: Optional[str] = None + expr: str | None = None # defaults to name if omitted + description: str | None = None + label: str | None = None + role: str | None = None # recognized for report/meta; not represented + config: DbtConfig | None = None class DbtDimension(BaseModel): name: str type: str = "categorical" # "categorical" or "time" - expr: Optional[str] = None - description: Optional[str] = None - label: Optional[str] = None - type_params: Optional[DbtTimeTypeParams] = None + expr: str | None = None + description: str | None = None + label: str | None = None + type_params: DbtTimeTypeParams | None = None + is_partition: bool | None = None # recognized for report/meta + validity_params: DbtValidityParams | None = None # recognized for report/meta + config: DbtConfig | None = None class DbtMeasureAggParams(BaseModel): - percentile: Optional[float] = None + percentile: float | None = None use_discrete_percentile: bool = False use_approximate_percentile: bool = False @@ -46,17 +117,18 @@ class DbtMeasureAggParams(BaseModel): class DbtMeasure(BaseModel): name: str agg: str # "sum", "count", "average", "count_distinct", "min", "max", etc. - expr: Optional[str] = None - description: Optional[str] = None - label: Optional[str] = None - create_metric: Optional[bool] = None - agg_time_dimension: Optional[str] = None - agg_params: Optional[DbtMeasureAggParams] = None - non_additive_dimension: Optional[DbtNonAdditiveDimension] = None + expr: str | None = None + description: str | None = None + label: str | None = None + create_metric: bool | None = None + agg_time_dimension: str | None = None + agg_params: DbtMeasureAggParams | None = None + non_additive_dimension: DbtNonAdditiveDimension | None = None + config: DbtConfig | None = None @field_validator("expr", mode="before") @classmethod - def _coerce_expr_to_str(cls, v: object) -> Optional[str]: + def _coerce_expr_to_str(cls, v: object) -> str | None: """Coerce numeric expr values to strings (e.g. dbt `expr: 1`).""" if v is None: return None @@ -64,60 +136,196 @@ def _coerce_expr_to_str(cls, v: object) -> Optional[str]: class DbtDefaults(BaseModel): - agg_time_dimension: Optional[str] = None + agg_time_dimension: str | None = None class DbtSemanticModel(BaseModel): name: str - model: Optional[str] = None # raw string, e.g. "ref('claim')" - description: Optional[str] = None - defaults: Optional[DbtDefaults] = None - primary_entity: Optional[str] = None - entities: List[DbtEntity] = Field(default_factory=list) - dimensions: List[DbtDimension] = Field(default_factory=list) - measures: List[DbtMeasure] = Field(default_factory=list) - label: Optional[str] = None + model: str | None = None # raw string, e.g. "ref('claim')" + description: str | None = None + defaults: DbtDefaults | None = None + primary_entity: str | None = None + entities: list[DbtEntity] = Field(default_factory=list) + dimensions: list[DbtDimension] = Field(default_factory=list) + measures: list[DbtMeasure] = Field(default_factory=list) + label: str | None = None + config: DbtConfig | None = None + + +class DbtMetricTimeWindow(BaseModel): + """A DSI metric time window: ``{count, granularity}`` or ``"7 days"``.""" + count: int + granularity: str + + @field_validator("granularity", mode="before") + @classmethod + def _normalize_granularity(cls, v: Any) -> Any: + """Singularize plural granularities (``weeks`` → ``week``).""" + if isinstance(v, str): + return _PLURAL_GRANULARITY_RE.sub("", v.strip()).lower() or v + return v + + @classmethod + def parse(cls, value: Any) -> Optional["DbtMetricTimeWindow"]: + """Coerce a ``" "`` string / dict into the model.""" + if value is None: + return None + if isinstance(value, DbtMetricTimeWindow): + return value + if isinstance(value, dict): + return cls.model_validate(value) + if isinstance(value, str): + parts = value.strip().split() + if len(parts) == 2 and parts[0].lstrip("-").isdigit(): + return cls(count=int(parts[0]), granularity=parts[1]) + # Single token (e.g. "month") → count 1. + if len(parts) == 1 and parts[0]: + return cls(count=1, granularity=parts[0]) + return None class DbtMetricInputMeasure(BaseModel): """A measure reference within a metric's type_params.""" name: str - filter: Optional[str] = None - alias: Optional[str] = None + filter: str | None = None + alias: str | None = None + join_to_timespine: bool = False + fill_nulls_with: int | None = None + + @field_validator("filter", mode="before") + @classmethod + def _normalize_filter(cls, v: Any) -> str | None: + return _normalize_filter(v) class DbtMetricInput(BaseModel): """A metric reference within a derived metric's type_params.""" name: str - alias: Optional[str] = None - offset_window: Optional[str] = None - offset_to_grain: Optional[str] = None - filter: Optional[str] = None + alias: str | None = None + offset_window: str | None = None + offset_to_grain: str | None = None + filter: str | None = None + + @field_validator("filter", mode="before") + @classmethod + def _normalize_filter(cls, v: Any) -> str | None: + return _normalize_filter(v) + + @field_validator("offset_window", mode="before") + @classmethod + def _coerce_offset_window(cls, v: Any) -> Any: + """Accept the DSI object form ``{count, granularity}`` as well as the + string form (``"1 month"``); store canonically as a string so + ``DbtMetricTimeWindow.parse`` (plural normalization, custom-grain + clean-fail) handles it downstream in the converter.""" + if isinstance(v, dict): + count = v.get("count") + gran = v.get("granularity") + if count is not None and gran: + return f"{count} {gran}" + return v + + +class DbtCumulativeTypeParams(BaseModel): + """DSI ``cumulative_type_params`` (window / grain_to_date / period_agg).""" + measure: str | None = None + metric: str | None = None + window: DbtMetricTimeWindow | None = None + grain_to_date: str | None = None + period_agg: str | None = None # default "first" in DSI + + @field_validator("window", mode="before") + @classmethod + def _coerce_window(cls, v: Any) -> Any: + return DbtMetricTimeWindow.parse(v) + + +class DbtConversionTypeParams(BaseModel): + """DSI ``conversion_type_params`` — parsed so conversion metrics fail + cleanly (funnel SQL is unsupported), never crash.""" + base_measure: DbtMetricInputMeasure | None = None + conversion_measure: DbtMetricInputMeasure | None = None + base_metric: str | None = None + conversion_metric: str | None = None + entity: str | None = None + calculation: str | None = None + window: DbtMetricTimeWindow | None = None + constant_properties: list[dict[str, Any]] | None = None + + @field_validator("window", mode="before") + @classmethod + def _coerce_window(cls, v: Any) -> Any: + return DbtMetricTimeWindow.parse(v) + + +class DbtMetricAggregationParams(BaseModel): + """DSI ``metric_aggregation_params`` — a measure-less simple metric that + aggregates a semantic-model expression directly. Unsupported shape in + SLayer (parsed for clean-fail routing).""" + semantic_model: str | None = None + agg: str | None = None + expr: str | None = None + agg_params: DbtMeasureAggParams | None = None + agg_time_dimension: str | None = None class DbtMetricTypeParams(BaseModel): - measure: Optional[str] = None # simple metrics: measure name (string shorthand) - expr: Optional[str] = None # derived metrics: formula expression - metrics: Optional[List[DbtMetricInput]] = None # derived: input metric refs - numerator: Optional[DbtMetricInput] = None # ratio - denominator: Optional[DbtMetricInput] = None # ratio + measure: DbtMetricInputMeasure | None = None # simple: measure ref (str shorthand or obj) + expr: str | None = None # derived metrics: formula expression + metrics: list[DbtMetricInput] | None = None # derived: input metric refs + numerator: DbtMetricInput | None = None # ratio + denominator: DbtMetricInput | None = None # ratio + # Cumulative — both the flat legacy fields and the nested struct. + window: DbtMetricTimeWindow | None = None + grain_to_date: str | None = None + cumulative_type_params: DbtCumulativeTypeParams | None = None + # Conversion / measure-less / metadata. + conversion_type_params: DbtConversionTypeParams | None = None + metric_aggregation_params: DbtMetricAggregationParams | None = None + time_granularity: str | None = None + is_private: bool | None = None + + @field_validator("measure", mode="before") + @classmethod + def _coerce_measure(cls, v: Any) -> Any: + """Accept the string shorthand (``measure: revenue``) or the full + ``MetricInputMeasure`` object.""" + if isinstance(v, str): + return {"name": v} + return v + + @field_validator("window", mode="before") + @classmethod + def _coerce_window(cls, v: Any) -> Any: + return DbtMetricTimeWindow.parse(v) + + @property + def measure_name(self) -> str | None: + return self.measure.name if self.measure else None class DbtMetric(BaseModel): name: str type: str # "simple", "derived", "cumulative", "ratio", "conversion" - description: Optional[str] = None - label: Optional[str] = None - type_params: Optional[DbtMetricTypeParams] = None - filter: Optional[str] = None + description: str | None = None + label: str | None = None + type_params: DbtMetricTypeParams | None = None + filter: str | None = None + time_granularity: str | None = None + config: DbtConfig | None = None + + @field_validator("filter", mode="before") + @classmethod + def _normalize_filter(cls, v: Any) -> str | None: + return _normalize_filter(v) class DbtColumnMeta(BaseModel): """Column-level metadata from dbt's manifest for a regular (non-semantic) model.""" name: str - description: Optional[str] = None - data_type: Optional[str] = None - tags: List[str] = Field(default_factory=list) + description: str | None = None + data_type: str | None = None + tags: list[str] = Field(default_factory=list) class DbtRegularModel(BaseModel): @@ -137,17 +345,22 @@ class DbtRegularModel(BaseModel): regular model's SQL into a semantic-model-derived ``SlayerModel``. """ name: str - database: Optional[str] = None - schema_name: Optional[str] = None # avoids shadowing pydantic's `schema` method - alias: Optional[str] = None # materialized table name; falls back to `name` - description: Optional[str] = None - tags: List[str] = Field(default_factory=list) - columns: List[DbtColumnMeta] = Field(default_factory=list) - raw_code: Optional[str] = None # SQL body from the .sql file on disk, Jinja unresolved + database: str | None = None + schema_name: str | None = None # avoids shadowing pydantic's `schema` method + alias: str | None = None # materialized table name; falls back to `name` + description: str | None = None + tags: list[str] = Field(default_factory=list) + columns: list[DbtColumnMeta] = Field(default_factory=list) + raw_code: str | None = None # SQL body from the .sql file on disk, Jinja unresolved class DbtProject(BaseModel): - """Aggregated result of parsing all YAML files in a dbt project.""" - semantic_models: List[DbtSemanticModel] = Field(default_factory=list) - metrics: List[DbtMetric] = Field(default_factory=list) - regular_models: List[DbtRegularModel] = Field(default_factory=list) + """Aggregated result of parsing all YAML files in a dbt project. + + ``saved_queries`` / ``exports`` (and any other top-level DSI artefacts) are + out of scope for the importer but accepted via ``extra="ignore"`` so a full + manifest doesn't crash the parser. + """ + semantic_models: list[DbtSemanticModel] = Field(default_factory=list) + metrics: list[DbtMetric] = Field(default_factory=list) + regular_models: list[DbtRegularModel] = Field(default_factory=list) diff --git a/slayer/dbt/parser.py b/slayer/dbt/parser.py index fe324ad0..4a5c3903 100644 --- a/slayer/dbt/parser.py +++ b/slayer/dbt/parser.py @@ -13,7 +13,6 @@ import os import re from pathlib import Path -from typing import Dict, List import yaml @@ -56,7 +55,7 @@ def _extract_ref_name(raw: str) -> str: return raw -def _collect_yaml_paths(directory: str) -> List[str]: +def _collect_yaml_paths(directory: str) -> list[str]: """Recursively collect .yaml and .yml file paths, skipping hidden dirs/files.""" paths = [] for root, dirs, files in os.walk(directory): @@ -69,7 +68,7 @@ def _collect_yaml_paths(directory: str) -> List[str]: return paths -def _collect_sql_files(directory: str) -> Dict[str, str]: +def _collect_sql_files(directory: str) -> dict[str, str]: """Recursively collect .sql file bodies keyed by filename stem. dbt models are named after their `.sql` filename (without the extension), @@ -77,7 +76,7 @@ def _collect_sql_files(directory: str) -> Dict[str, str]: hidden dirs/files and any target/build directories dbt may have left behind. """ - result: Dict[str, str] = {} + result: dict[str, str] = {} for root, dirs, files in os.walk(directory): dirs[:] = [d for d in dirs if not d.startswith(".") and d != "target"] for filename in sorted(files): @@ -121,8 +120,8 @@ def parse_dbt_project( models whose underlying dbt model is a query rather than a physical table. """ - all_semantic_models: List[DbtSemanticModel] = [] - all_metrics: List[DbtMetric] = [] + all_semantic_models: list[DbtSemanticModel] = [] + all_metrics: list[DbtMetric] = [] yaml_paths = _collect_yaml_paths(project_path) if not yaml_paths: @@ -206,7 +205,7 @@ def parse_dbt_project( ) -def _parse_regular_models(project_path: str) -> List[DbtRegularModel]: +def _parse_regular_models(project_path: str) -> list[DbtRegularModel]: """Discover regular (non-semantic) dbt models via the dbt manifest. Returns an empty list when the manifest is absent and dbt-core is not diff --git a/slayer/dbt/sql_resolver.py b/slayer/dbt/sql_resolver.py index 3285833e..9603b5b7 100644 --- a/slayer/dbt/sql_resolver.py +++ b/slayer/dbt/sql_resolver.py @@ -24,7 +24,6 @@ import logging import re -from typing import Dict, List, Optional, Set, Tuple logger = logging.getLogger(__name__) @@ -68,11 +67,11 @@ def resolve_refs( sql: str, - regular_models_sql: Dict[str, str], + regular_models_sql: dict[str, str], *, max_depth: int = 16, - _visited: Optional[Set[str]] = None, -) -> Tuple[str, List[str]]: + _visited: set[str] | None = None, +) -> tuple[str, list[str]]: """Resolve dbt Jinja refs/sources in a SQL body. Refs pointing at other regular models are recursively inlined as @@ -99,7 +98,7 @@ def resolve_refs( if _visited is None: _visited = set() - warnings: List[str] = [] + warnings: list[str] = [] if max_depth <= 0: warnings.append( diff --git a/slayer/demo/jaffle_shop.py b/slayer/demo/jaffle_shop.py index 32aaa02f..1ac751ce 100644 --- a/slayer/demo/jaffle_shop.py +++ b/slayer/demo/jaffle_shop.py @@ -2,7 +2,8 @@ Generates ~2 years of synthetic coffee-shop data via ``jafgen``, loads it into a DuckDB file under the storage directory, registers a ``jaffle_shop`` -datasource, and (optionally) auto-ingests SLayer models. The default is kept +datasource, and (optionally) auto-ingests SLayer models enriched with curated +labels, descriptions, formats, and example measures (``DEMO_ENRICHMENT``). The default is kept small so ``slayer serve --demo`` / ``slayer mcp --demo`` finish quickly enough to fit inside MCP-client startup timeouts; bump ``--years`` for a richer dataset (only the first four jafgen stores open within the first 2 years). @@ -14,13 +15,27 @@ import datetime as dt import io import os +import shutil import subprocess import sys import tempfile -from typing import IO, TYPE_CHECKING, List, Optional, Tuple +from collections import deque +from importlib.util import find_spec +from typing import IO, TYPE_CHECKING + +from pydantic import BaseModel, Field from slayer.async_utils import run_sync -from slayer.core.models import DatasourceConfig, SlayerModel +from slayer.core.enums import DataType +from slayer.core.format import NumberFormat, NumberFormatType +from slayer.core.models import ( + Aggregation, + AggregationParam, + Column, + DatasourceConfig, + ModelMeasure, + SlayerModel, +) from slayer.storage.base import StorageBackend, storage_base_dir if TYPE_CHECKING: @@ -116,6 +131,379 @@ """ +# --- curated semantic enrichment -------------------------------------------- +# +# Hand-curated labels, descriptions, formats, and measures layered on top of +# the bare auto-ingested models. Monetary columns are in dollars (the loader +# converts jafgen's cents on insert — see CENTS_COLUMNS). + + +def _currency() -> NumberFormat: + return NumberFormat(type=NumberFormatType.CURRENCY, symbol="$", precision=2) + + +def _percent() -> NumberFormat: + return NumberFormat(type=NumberFormatType.PERCENT, precision=1) + + +class _ColumnEnrichment(BaseModel): + label: str + description: str | None = None + format: NumberFormat | None = None + + +class _TableEnrichment(BaseModel): + description: str | None = None + columns: dict[str, _ColumnEnrichment] = Field(default_factory=dict) + measures: list[ModelMeasure] = Field(default_factory=list) + aggregations: list[Aggregation] = Field(default_factory=list) + + +def _build_demo_enrichment() -> dict[str, _TableEnrichment]: + return { + "orders": _TableEnrichment( + description=( + "Customer orders — one row per order. Monetary amounts are in " + "dollars. The fact table at the center of the demo: joins to " + "customers and stores, and is referenced by items." + ), + # Default the weighted_avg weight to subtotal, so a bare + # ``order_total:weighted_avg`` is a sales-weighted average. + aggregations=[ + Aggregation( + name="weighted_avg", + params=[AggregationParam(name="weight", sql="subtotal")], + description="Weighted average defaulting the weight to order subtotal.", + ), + ], + columns={ + "id": _ColumnEnrichment(label="Order ID"), + "ordered_at": _ColumnEnrichment( + label="Order Date", description="When the order was placed." + ), + "store_id": _ColumnEnrichment( + label="Store ID", description="Store where the order was placed." + ), + "customer_id": _ColumnEnrichment( + label="Customer ID", description="Customer who placed the order." + ), + "subtotal": _ColumnEnrichment( + label="Net Sales", + description="Pre-tax order amount, in dollars.", + format=_currency(), + ), + "tax_paid": _ColumnEnrichment( + label="Tax Paid", + description="Tax charged on the order, in dollars.", + format=_currency(), + ), + "order_total": _ColumnEnrichment( + label="Order Total", + description="Final order amount including tax, in dollars.", + format=_currency(), + ), + }, + measures=[ + ModelMeasure( + name="total_revenue", + formula="order_total:sum", + label="Total Revenue", + description="Gross sales including tax, in dollars.", + type=DataType.DOUBLE, + ), + ModelMeasure( + name="net_sales", + formula="subtotal:sum", + label="Net Sales (pre-tax)", + description="Sales before tax, in dollars.", + type=DataType.DOUBLE, + ), + ModelMeasure( + name="tax_collected", + formula="tax_paid:sum", + label="Tax Collected", + description="Total tax collected, in dollars.", + type=DataType.DOUBLE, + ), + ModelMeasure( + name="order_count", + formula="id:count", + label="Orders", + description="Number of orders.", + type=DataType.INT, + ), + ModelMeasure( + name="unique_customers", + formula="customer_id:count_distinct", + label="Unique Customers", + description="Distinct customers who ordered.", + type=DataType.INT, + ), + ModelMeasure( + name="avg_order_value", + formula="order_total:sum / nullif(id:count, 0)", + label="Average Order Value", + description="Revenue per order, in dollars.", + type=DataType.DOUBLE, + ), + ModelMeasure( + name="effective_tax_rate", + formula="tax_paid:sum / nullif(subtotal:sum, 0)", + label="Effective Tax Rate", + description="Tax collected as a share of net sales.", + type=DataType.DOUBLE, + ), + ModelMeasure( + name="sales_weighted_aov", + formula="order_total:weighted_avg", + label="Sales-Weighted Avg Order", + description=( + "Average order total weighted by subtotal (larger orders " + "weigh more), in dollars." + ), + type=DataType.DOUBLE, + ), + ], + ), + "customers": _TableEnrichment( + description="Customers of the Jaffle Shop — one row per person.", + columns={ + "id": _ColumnEnrichment(label="Customer ID"), + "name": _ColumnEnrichment(label="Customer", description="Customer name."), + }, + measures=[ + ModelMeasure( + name="customer_count", + formula="id:count_distinct", + label="Customers", + description="Number of distinct customers.", + type=DataType.INT, + ), + ], + ), + "stores": _TableEnrichment( + description="Physical Jaffle Shop locations.", + columns={ + "id": _ColumnEnrichment(label="Store ID"), + "name": _ColumnEnrichment(label="Store", description="Store name."), + "opened_at": _ColumnEnrichment( + label="Opened", description="When the store opened." + ), + "tax_rate": _ColumnEnrichment( + label="Tax Rate", + description="Local sales-tax rate.", + format=_percent(), + ), + }, + measures=[ + ModelMeasure( + name="store_count", + formula="id:count_distinct", + label="Stores", + description="Number of stores.", + type=DataType.INT, + ), + ], + ), + "products": _TableEnrichment( + description="Menu items sold at the Jaffle Shop — jaffles and beverages.", + columns={ + "sku": _ColumnEnrichment(label="SKU", description="Product identifier."), + "name": _ColumnEnrichment(label="Product", description="Product name."), + "type": _ColumnEnrichment( + label="Category", description="Product category (jaffle or beverage)." + ), + "price": _ColumnEnrichment( + label="Price", + description="List price, in dollars.", + format=_currency(), + ), + "description": _ColumnEnrichment( + label="Description", description="Product description." + ), + }, + measures=[ + ModelMeasure( + name="product_count", + formula="sku:count_distinct", + label="Products", + description="Number of distinct products.", + type=DataType.INT, + ), + ModelMeasure( + name="avg_price", + formula="price:avg", + label="Average Price", + description="Average list price, in dollars.", + type=DataType.DOUBLE, + ), + ], + ), + "items": _TableEnrichment( + description="Order line items — one row per unit sold on an order.", + columns={ + "id": _ColumnEnrichment(label="Item ID"), + "order_id": _ColumnEnrichment( + label="Order", description="Order this line belongs to." + ), + "sku": _ColumnEnrichment( + label="Product", description="Product sold on this line." + ), + }, + measures=[ + ModelMeasure( + name="units_sold", + formula="id:count", + label="Units Sold", + description="Number of item units sold.", + type=DataType.INT, + ), + ], + ), + "supplies": _TableEnrichment( + description=( + "Supplies (ingredients and packaging) used per product, with " + "unit costs in dollars." + ), + columns={ + "id": _ColumnEnrichment(label="Supply ID"), + "name": _ColumnEnrichment(label="Supply", description="Supply name."), + "cost": _ColumnEnrichment( + label="Unit Cost", + description="Cost per unit, in dollars.", + format=_currency(), + ), + "perishable": _ColumnEnrichment( + label="Perishable", description="Whether the supply is perishable." + ), + "sku": _ColumnEnrichment( + label="Product", description="Product this supply is used for." + ), + }, + measures=[ + ModelMeasure( + name="total_supply_cost", + formula="cost:sum", + label="Total Supply Cost", + description="Total supply cost, in dollars.", + type=DataType.DOUBLE, + ), + ModelMeasure( + name="avg_unit_cost", + formula="cost:avg", + label="Avg Unit Cost", + description="Average supply unit cost, in dollars.", + type=DataType.DOUBLE, + ), + ], + ), + "tweets": _TableEnrichment( + description="Synthetic customer tweets mentioning the Jaffle Shop.", + columns={ + "id": _ColumnEnrichment(label="Tweet ID"), + "user_id": _ColumnEnrichment( + label="Customer ID", description="Customer who tweeted." + ), + "tweeted_at": _ColumnEnrichment( + label="Tweet Date", description="When the tweet was posted." + ), + "content": _ColumnEnrichment(label="Tweet", description="Tweet text."), + }, + measures=[ + ModelMeasure( + name="tweet_count", + formula="id:count", + label="Tweets", + description="Number of tweets.", + type=DataType.INT, + ), + ], + ), + } + + +DEMO_ENRICHMENT = _build_demo_enrichment() + + +def _is_auto_default_format(fmt: NumberFormat | None) -> bool: + """True when ``fmt`` is unset or the bare INTEGER/FLOAT default that + auto-ingestion stamps on numeric columns (safe to override).""" + if fmt is None: + return True + if fmt.type not in (NumberFormatType.INTEGER, NumberFormatType.FLOAT): + return False + return fmt.precision is None and fmt.symbol is None + + +def _enrich_column(column: Column, col_spec: _ColumnEnrichment) -> bool: + """Fill unset label / description / auto-default format on one column.""" + changed = False + if column.label is None: + column.label = col_spec.label + changed = True + if column.description is None and col_spec.description is not None: + column.description = col_spec.description + changed = True + if ( + col_spec.format is not None + and column.format != col_spec.format + and _is_auto_default_format(column.format) + ): + column.format = col_spec.format.model_copy(deep=True) + changed = True + return changed + + +def _new_named_entries(existing: list, additions: list) -> list: + """Deep-copied ``additions`` whose ``name`` isn't already in ``existing``. + Copies keep the shared spec isolated from later model mutations.""" + taken = {item.name for item in existing if item.name} + return [item.model_copy(deep=True) for item in additions if item.name not in taken] + + +def apply_demo_enrichment(model: SlayerModel) -> bool: + """Additively apply the curated enrichment to an ingested demo model. + + Fills only unset fields and merges measures/aggregations by name, so + user edits survive and re-runs are no-ops. Skips models not backed by + ``sql_table``. Returns ``True`` if the model was modified. + """ + spec = DEMO_ENRICHMENT.get(model.name) + if spec is None or model.sql_table is None: + return False + + changed = False + + if model.description is None and spec.description is not None: + model.description = spec.description + changed = True + + if ( + model.default_time_dimension is None + and model.name in DEFAULT_TIME_DIMENSIONS + ): + model.default_time_dimension = DEFAULT_TIME_DIMENSIONS[model.name] + changed = True + + by_name: dict[str, Column] = {c.name: c for c in model.columns} + for col_name, col_spec in spec.columns.items(): + column = by_name.get(col_name) + if column is not None and _enrich_column(column, col_spec): + changed = True + + new_measures = _new_named_entries(model.measures, spec.measures) + if new_measures: + model.measures = list(model.measures) + new_measures + changed = True + + new_aggs = _new_named_entries(model.aggregations, spec.aggregations) + if new_aggs: + model.aggregations = list(model.aggregations) + new_aggs + changed = True + + return changed + + def resolve_demo_db_path(storage_path: str) -> str: """Return the path to the Jaffle Shop DuckDB file for a given storage path. @@ -126,7 +514,7 @@ def resolve_demo_db_path(storage_path: str) -> str: return os.path.join(demo_dir, "jaffle_shop.duckdb") -def _stream_fileno(stream) -> Optional[int]: +def _stream_fileno(stream) -> int | None: """Return ``stream.fileno()`` if it points at a real file descriptor, else None. ipykernel / nbclient replace ``sys.stdout`` / ``sys.stderr`` with shim @@ -141,11 +529,34 @@ def _stream_fileno(stream) -> Optional[int]: return None +def _jafgen_cmd(years: int) -> list[str]: + """Build the jafgen invocation without relying on PATH exposure. + + ``uv tool install motley-slayer`` (and pipx) link only slayer's own entry + points onto PATH; the ``jafgen`` console script of the dependency stays + inside the tool venv's ``bin/``, so a bare ``jafgen`` subprocess fails + with FileNotFoundError. Since jafgen is a core dependency, prefer running + its CLI through the current interpreter; fall back to a PATH lookup for + environments where the package somehow isn't importable. + """ + years_arg = str(max(1, years)) + if find_spec("jafgen") is not None: + return [sys.executable, "-c", "from jafgen.cli import app; app()", years_arg] + exe = shutil.which("jafgen") + if exe is not None: + return [exe, years_arg] + raise RuntimeError( + "jafgen is required to generate the demo data but was not found. " + "It ships with motley-slayer — reinstall the package, or run " + "`pip install jafgen` in the environment running slayer." + ) + + def generate_data( output_dir: str, years: int = 1, *, - stream: Optional[IO[str]] = None, + stream: IO[str] | None = None, ) -> str: """Run ``jafgen`` into ``output_dir``; return the path to the generated CSVs. @@ -157,25 +568,40 @@ def generate_data( shim without ``fileno()`` (Jupyter ``OutStream``, ``io.StringIO``, …), we pump the child's output line by line into ``stream`` instead. """ - cmd = ["jafgen", str(max(1, years))] + cmd = _jafgen_cmd(years) out = stream if stream is not None else sys.stderr + # Force the child into Python UTF-8 mode (PEP 540). jafgen's Rich progress + # bars emit non-Latin-1 glyphs (e.g. the 🥪 emoji); on Windows the child's + # default stdio encoding is the ANSI code page (cp1252), which can't encode + # them, so jafgen would die with UnicodeEncodeError and exit 1. PYTHONUTF8=1 + # switches its stdio to UTF-8 regardless of the host code page. + child_env = {**os.environ, "PYTHONUTF8": "1", "PYTHONIOENCODING": "utf-8"} if _stream_fileno(out) is not None: try: - subprocess.run(args=cmd, cwd=output_dir, check=True, stdout=out, stderr=out) + subprocess.run( + args=cmd, cwd=output_dir, check=True, stdout=out, stderr=out, env=child_env + ) except subprocess.CalledProcessError as e: raise RuntimeError(f"jafgen failed with exit code {e.returncode}") from e return os.path.join(output_dir, "jaffle-data") + # Decode the pipe as UTF-8 (matching the child's forced encoding); errors are + # replaced so the pump loop never crashes on a stray byte. + tail: deque = deque(maxlen=25) with subprocess.Popen( args=cmd, cwd=output_dir, stdout=subprocess.PIPE, stderr=subprocess.STDOUT, text=True, + encoding="utf-8", + errors="replace", bufsize=1, + env=child_env, ) as proc: assert proc.stdout is not None for line in proc.stdout: + tail.append(line) out.write(line) try: out.flush() @@ -183,13 +609,15 @@ def generate_data( pass rc = proc.wait() if rc != 0: - raise RuntimeError(f"jafgen failed with exit code {rc}") + detail = "".join(tail).rstrip() + suffix = f":\n{detail}" if detail else "" + raise RuntimeError(f"jafgen failed with exit code {rc}{suffix}") return os.path.join(output_dir, "jaffle-data") def create_schema( conn: "duckdb.DuckDBPyConnection", - schema_path: Optional[str] = None, + schema_path: str | None = None, ) -> None: """Create the Jaffle Shop tables on ``conn``. @@ -329,7 +757,7 @@ def build_jaffle_shop( *, years: int = 2, force: bool = False, - stream: Optional[IO[str]] = None, + stream: IO[str] | None = None, ) -> bool: """Generate the Jaffle Shop DuckDB at ``db_path`` if it does not already exist. @@ -372,14 +800,15 @@ def ensure_demo_datasource( years: int = 2, ingest_models: bool = True, assume_yes: bool = True, - stream: Optional[IO[str]] = None, -) -> Tuple[DatasourceConfig, List[SlayerModel], bool]: + stream: IO[str] | None = None, +) -> tuple[DatasourceConfig, list[SlayerModel], bool]: """Ensure the Jaffle Shop demo is fully set up in ``storage``. - Builds the DuckDB at ``/demo/jaffle_shop.duckdb`` if missing. - Registers a datasource record (``name``, default ``jaffle_shop``). - - Optionally auto-ingests models and sets ``default_time_dimension`` on - ``orders``/``tweets``. + - Optionally auto-ingests models, sets ``default_time_dimension`` on + ``orders``/``tweets``, and applies the curated semantic enrichment + (labels, descriptions, formats, measures — see ``DEMO_ENRICHMENT``). Returns ``(datasource, jaffle_models, db_built)``. ``jaffle_models`` is the full set of Jaffle Shop ``SlayerModel`` objects currently in storage @@ -407,22 +836,32 @@ def ensure_demo_datasource( return ds, [], db_built # Fast path: DB was reused and models are already stored — return what's - # on disk so callers can report the real count. - existing_model_names = set(run_sync(storage.list_models())) + # on disk. Lookups are scoped to the demo datasource (bare-name calls + # raise on multi-datasource storages). Enrichment still runs so demos + # set up by older versions gain labels/measures on next startup. + existing_model_names = set(run_sync(storage.list_models(data_source=name))) if not db_built and all(t in existing_model_names for t in TABLE_NAMES): - jaffle_models = [ - run_sync(storage.get_model(t)) for t in TABLE_NAMES if t in existing_model_names - ] - return ds, [m for m in jaffle_models if m is not None], db_built + jaffle_models = [] + for t in TABLE_NAMES: + if t not in existing_model_names: + continue + model = run_sync(storage.get_model(name=t, data_source=name)) + if model is None: + continue + if assume_yes and apply_demo_enrichment(model): + run_sync(storage.save_model(model)) + jaffle_models.append(model) + return ds, jaffle_models, db_built from slayer.engine.ingestion import ingest_datasource models = ingest_datasource(datasource=ds) - written: List[SlayerModel] = [] + written: list[SlayerModel] = [] for model in models: - if model.name in DEFAULT_TIME_DIMENSIONS: - model.default_time_dimension = DEFAULT_TIME_DIMENSIONS[model.name] - existing_model: Optional[SlayerModel] = run_sync(storage.get_model(model.name)) + apply_demo_enrichment(model) + existing_model: SlayerModel | None = run_sync( + storage.get_model(name=model.name, data_source=name) + ) if existing_model is not None and not assume_yes: written.append(existing_model) continue diff --git a/slayer/embeddings/__init__.py b/slayer/embeddings/__init__.py index 6c01fb36..2e04afe7 100644 --- a/slayer/embeddings/__init__.py +++ b/slayer/embeddings/__init__.py @@ -1,10 +1,10 @@ -"""Embedding-based semantic search channel (DEV-1386). +"""Embedding storage + litellm client wrapper (DEV-1386). -Exposes the persisted ``Embedding`` row. The ``EmbeddingService`` orchestrator -and the litellm client wrapper are intentionally not re-exported here — they -import from ``slayer.storage.base``, which imports from this package, so eager -re-export would create a cycle. Import them directly from -``slayer.embeddings.service`` / ``slayer.embeddings.client`` when needed. +The orchestrator that owns the refresh pipeline + cosine ranking lives +in :mod:`slayer.search.retrievers.embeddings` as of DEV-1514. This +package exposes only the persisted ``Embedding`` row; the litellm +client wrapper is in :mod:`slayer.embeddings.client` and the cosine +helpers are in :mod:`slayer.embeddings.ranker`. """ from slayer.embeddings.models import Embedding diff --git a/slayer/embeddings/client.py b/slayer/embeddings/client.py index 49bb6c18..518e108e 100644 --- a/slayer/embeddings/client.py +++ b/slayer/embeddings/client.py @@ -1,7 +1,7 @@ """Litellm wrapper for embedding generation (DEV-1386). This module is the only place that imports ``litellm`` (and only lazily). -When the ``embedding_search`` extra is not installed, ``is_available()`` +When the ``advanced_search`` extra is not installed, ``is_available()`` returns ``False`` and every call returns the no-op shape — the caller is expected to short-circuit and skip the embedding channel entirely. @@ -9,15 +9,23 @@ ``openai/text-embedding-3-small``. Provider credentials (``OPENAI_API_KEY``, ``AZURE_API_KEY``, etc.) are read by litellm itself per its standard env-var conventions. + +DEV-1557: ``embed_batch`` no longer treats one over-cap input as a +batch-killer. Each text is token-truncated to the model's reported cap +(minus a 256-token margin) via ``truncate_text_for_model`` before the +call; if the batch still raises ``BadRequestError``, we fall back to +embedding each text individually so good inputs survive. """ from __future__ import annotations +import hashlib +import inspect import logging import os import warnings from functools import lru_cache -from typing import List, Optional +from typing import Any DEFAULT_EMBEDDING_MODEL = "openai/text-embedding-3-small" @@ -54,7 +62,7 @@ def is_available() -> bool: Two conditions, both required: - 1. The ``embedding_search`` extra is installed (``litellm`` imports). + 1. The ``advanced_search`` extra is installed (``litellm`` imports). 2. The configured embedding model has a usable API key in the environment, per ``litellm.validate_environment``. @@ -88,33 +96,175 @@ def is_available() -> bool: return bool(validation.get("keys_in_environment", False)) -async def embed_batch( - texts: List[str], *, model: Optional[str] = None, -) -> List[Optional[List[float]]]: - """Embed a batch of texts via ``litellm.aembedding``. +# DEV-1557: 256-token margin between the model's reported cap and our +# usable budget. text-embedding-3-small reports cap=8191 from litellm +# and accepts up to 8192 tokens server-side; 256 leaves room for any +# provider-side BOS/role-token overhead and minor cap-introspection +# drift across litellm versions, while still keeping the budget useful +# for very large caps (Voyage's 32K → 31744-token budget). +_TRUNCATE_MARGIN_TOKENS = 256 +_CAP_FALLBACK = 8192 +# Hash bytes logged on truncation events. SHA-256 prefix is enough to +# correlate two log lines without leaking the embedded user content. +_HASH_PREFIX_CHARS = 16 + + +_CAP_CACHE: dict[str, int] = {} + + +def _try_get_max_tokens(model: str) -> int | None: + """Best-effort ``litellm.utils.get_max_tokens`` lookup. Returns the + cap if it's a positive int, else ``None``. Never raises.""" + try: + from litellm import utils as litellm_utils + except Exception: # noqa: BLE001 — litellm absent or import-broken + return None + try: + cap = litellm_utils.get_max_tokens(model) + except Exception: # noqa: BLE001 — litellm model-map drift / network blip + return None + if isinstance(cap, int) and cap > 0: + return cap + return None + + +def _resolve_model_cap(resolved_model: str) -> int | None: + """Look up the model's max-token cap, trying provider-prefixed name + first and the bare name as a fallback. Caches only positive-int + successes so a transient lookup failure isn't sticky.""" + cached = _CAP_CACHE.get(resolved_model) + if cached is not None: + return cached + cap = _try_get_max_tokens(resolved_model) + if cap is None: + bare = resolved_model.rsplit("/", 1)[-1] + if bare and bare != resolved_model: + cap = _try_get_max_tokens(bare) + if cap is not None: + _CAP_CACHE[resolved_model] = cap + return cap + + +def _clear_cap_cache() -> None: + """Test hook (matches the ``cache_clear`` shape used by lru_cache).""" + _CAP_CACHE.clear() + + +# Surface ``cache_clear`` on the function itself so test fixtures can +# clear it uniformly with the lru-cached encoder helper below. +_resolve_model_cap.cache_clear = _clear_cap_cache # type: ignore[attr-defined] - Returns one vector per input text in input order. On any exception - (rate limit, bad key, network), logs a warning and returns - ``[None] * len(texts)`` — callers persist only the non-None entries. - Empty ``texts`` short-circuits to ``[]`` without an API call. +@lru_cache(maxsize=8) +def _resolve_encoder(bare_model_name: str) -> Any: + """Return a tiktoken encoder for the given bare model name (with + provider prefix already stripped). Falls back to ``cl100k_base`` on + KeyError. Raises ``ImportError`` if tiktoken is unavailable — + callers must catch and degrade to identity truncation.""" + import tiktoken # noqa: PLC0415 — lazy import gated by advanced_search + try: + return tiktoken.encoding_for_model(bare_model_name) + except KeyError: + return tiktoken.get_encoding("cl100k_base") + + +def _strip_provider_prefix(model: str) -> str: + return model.rsplit("/", 1)[-1] if "/" in model else model + + +def truncate_text_for_model( + text: str, *, model: str | None = None, +) -> str: + """Truncate ``text`` to the resolved model's token cap minus a + fixed 256-token margin (DEV-1557). + + Returns ``text`` unchanged when already under budget (no decode + round-trip). Head-keep slicing — the prefix of the input is + preserved because SLayer's rendered entity / memory text leads + with the most signal-rich fields. + + Defensive: if tiktoken is unavailable (the lazy import in + ``_resolve_encoder`` raises) or any other encoder-resolution + failure surfaces, returns ``text`` unchanged. The per-input retry + in :func:`embed_batch` is what saves the batch when truncation + degrades to identity. """ - if not texts: - return [] - if not is_available(): - return [None] * len(texts) resolved_model = model or current_model() try: - import litellm - response = await litellm.aembedding(model=resolved_model, input=texts) - except Exception as exc: - _log.warning( - "embed_batch failed for model=%s (n=%d): %s", - resolved_model, len(texts), exc, - ) - return [None] * len(texts) + encoder = _resolve_encoder(_strip_provider_prefix(resolved_model)) + except Exception: # noqa: BLE001 — tiktoken missing / unknown failure + return text + + # ``disallowed_special=()`` keeps tiktoken from raising on literal + # ``<|endoftext|>`` and similar markers that a user-controlled + # memory / entity description might happen to contain — without + # this, a single such input would propagate the ValueError out + # past embed_batch's per-input retry and regress to the all-None + # batch-killer this PR is supposed to fix. + try: + tokens = encoder.encode(text, disallowed_special=()) + except Exception: # noqa: BLE001 — tokenisation failure → degrade gracefully + return text + cap = _resolve_model_cap(resolved_model) or _CAP_FALLBACK + budget = max(0, cap - _TRUNCATE_MARGIN_TOKENS) + + if len(tokens) <= budget: + return text + + truncated_tokens = tokens[:budget] + truncated = encoder.decode(truncated_tokens) if truncated_tokens else "" + # Log a content hash, not a preview. The preview was originally + # there for operator correlation, but it leaks embedded user + # content into application logs — the hash gives correlation + # (two log lines for the same input share a digest) without + # leaking the content. + text_digest = hashlib.sha256(text.encode("utf-8")).hexdigest()[ + :_HASH_PREFIX_CHARS + ] + _log.warning( + "truncated text for model=%s: original_tokens=%d post_tokens=%d " + "original_chars=%d post_chars=%d sha256_prefix=%s", + resolved_model, len(tokens), len(truncated_tokens), + len(text), len(truncated), text_digest, + ) + return truncated + + +def _get_bad_request_exception_classes() -> tuple[type[BaseException], ...]: + """Return the exception class(es) representing litellm's + ``BadRequestError`` — empty tuple if neither ``litellm`` nor + ``litellm.exceptions`` exposes one. Tuple form lets us use the + result directly in an ``except`` clause; an empty tuple safely + catches nothing so the generic-exception fallback path takes over. + """ + classes: list[type[BaseException]] = [] + try: + import litellm # noqa: PLC0415 — lazy + except Exception: # noqa: BLE001 + return () + cls = getattr(litellm, "BadRequestError", None) + if inspect.isclass(cls) and issubclass(cls, BaseException): + classes.append(cls) + try: + from litellm import exceptions as _exc_mod # noqa: PLC0415 + except Exception: # noqa: BLE001 + _exc_mod = None + if _exc_mod is not None: + cls2 = getattr(_exc_mod, "BadRequestError", None) + if ( + inspect.isclass(cls2) + and issubclass(cls2, BaseException) + and cls2 not in classes + ): + classes.append(cls2) + return tuple(classes) + + +def _parse_vectors(response: Any, n: int) -> list[list[float] | None]: + """Pack a litellm aembedding response into ``n`` per-slot vectors, + padding short responses with ``None``.""" data = getattr(response, "data", None) or [] - out: List[Optional[List[float]]] = [] + out: list[list[float] | None] = [] for entry in data: if isinstance(entry, dict): vec = entry.get("embedding") @@ -124,17 +274,109 @@ async def embed_batch( out.append([float(v) for v in vec]) else: out.append(None) - # If litellm returned fewer rows than requested, pad with None. - while len(out) < len(texts): + while len(out) < n: out.append(None) - return out[: len(texts)] + return out[:n] + + +async def _per_input_retry( + litellm: Any, + resolved_model: str, + truncated_texts: list[str], + bad_request_classes: tuple[type[BaseException], ...], +) -> list[list[float] | None]: + """Embed each text in ``truncated_texts`` individually. On a + per-text ``BadRequestError`` the slot is ``None`` and the loop + continues; on any other exception we treat it as a global failure + shape (rate limit / auth / network), mark the current slot and + every remaining slot ``None``, and return early.""" + results: list[list[float] | None] = [] + for idx, text in enumerate(truncated_texts): + try: + response = await litellm.aembedding( + model=resolved_model, input=[text], + ) + except bad_request_classes as exc: + _log.warning( + "embed_batch per-input retry: BadRequestError at idx=%d " + "for model=%s: %s — slot marked None", + idx, resolved_model, exc, + ) + results.append(None) + continue + except Exception as exc: # noqa: BLE001 — see docstring + _log.warning( + "embed_batch per-input retry: %s at idx=%d for model=%s: " + "%s — global failure shape, aborting retries " + "(remaining %d slot(s) None)", + type(exc).__name__, idx, resolved_model, exc, + len(truncated_texts) - len(results) - 1, + ) + results.append(None) + results.extend([None] * (len(truncated_texts) - len(results))) + return results + results.extend(_parse_vectors(response, 1)) + return results + + +async def embed_batch( + texts: list[str], *, model: str | None = None, +) -> list[list[float] | None]: + """Embed a batch of texts via ``litellm.aembedding`` (DEV-1557). + + Each text is preemptively truncated to fit the resolved model's + token cap (see :func:`truncate_text_for_model`). If the batch call + still raises ``litellm.BadRequestError``, we fall back to embedding + each text individually so a single over-cap input no longer + poisons the whole batch. + + Returns one vector (or ``None``) per input text in input order: + + * Empty ``texts`` → ``[]`` (no API call). + * ``is_available()`` False → ``[None] * len(texts)`` (no truncation, + no API call). + * Batch ``BadRequestError`` → per-input retry; each text gets its + own success/failure verdict. + * Any other batch exception (rate limit / auth / network) → log + warning, return ``[None] * len(texts)``. + """ + if not texts: + return [] + if not is_available(): + return [None] * len(texts) + resolved_model = model or current_model() + truncated = [ + truncate_text_for_model(t, model=resolved_model) for t in texts + ] + import litellm # noqa: PLC0415 — lazy + bad_request_classes = _get_bad_request_exception_classes() + try: + response = await litellm.aembedding( + model=resolved_model, input=truncated, + ) + except bad_request_classes as exc: + _log.warning( + "embed_batch BadRequestError for model=%s (n=%d): %s — " + "falling back to per-input retry", + resolved_model, len(truncated), exc, + ) + return await _per_input_retry( + litellm, resolved_model, truncated, bad_request_classes, + ) + except Exception as exc: + _log.warning( + "embed_batch failed for model=%s (n=%d): %s", + resolved_model, len(truncated), exc, + ) + return [None] * len(truncated) + return _parse_vectors(response, len(truncated)) -_QUERY_CACHE: "dict[tuple[str, str], List[float]]" = {} +_QUERY_CACHE: "dict[tuple[str, str], list[float]]" = {} _QUERY_CACHE_MAX = 64 -async def embed_query(text: str, *, model: Optional[str] = None) -> Optional[List[float]]: +async def embed_query(text: str, *, model: str | None = None) -> list[float] | None: """Embed a single query string with a small process-wide LRU cache. Returns ``None`` when the extra is not installed or the embedding call diff --git a/slayer/embeddings/models.py b/slayer/embeddings/models.py index 4110fc15..bf534914 100644 --- a/slayer/embeddings/models.py +++ b/slayer/embeddings/models.py @@ -18,7 +18,7 @@ """ from datetime import datetime, timezone -from typing import Any, List, Literal +from typing import Any, Literal from pydantic import BaseModel, Field, model_validator @@ -42,7 +42,7 @@ class Embedding(BaseModel): embedding_model_name: str entity_kind: EntityKind content_hash: str - embedding: List[float] + embedding: list[float] created_at: datetime = Field(default_factory=_utcnow) @model_validator(mode="before") diff --git a/slayer/embeddings/ranker.py b/slayer/embeddings/ranker.py index 37f78dda..49055933 100644 --- a/slayer/embeddings/ranker.py +++ b/slayer/embeddings/ranker.py @@ -7,18 +7,17 @@ scale and avoid the operational burden of a persistent ANN index. Imports numpy at module top — this module is only imported behind the -``embedding_search`` extra's gate, so a missing numpy is a programming +``advanced_search`` extra's gate, so a missing numpy is a programming error here, not a runtime fallback. """ from __future__ import annotations -from typing import List, Tuple import numpy as np -def normalise(vector: List[float]) -> np.ndarray: +def normalise(vector: list[float]) -> np.ndarray: """Return a unit-L2 numpy view of ``vector``. Zero vectors come back unchanged so we never divide by zero.""" arr = np.asarray(vector, dtype=np.float32) @@ -42,7 +41,7 @@ def top_k_cosine( query: np.ndarray, matrix: np.ndarray, k: int, -) -> List[Tuple[int, float]]: +) -> list[tuple[int, float]]: """Return the top-``k`` ``(row_index, cosine_similarity)`` pairs. Assumes ``query`` is already unit-normalised (1D shape ``(dim,)``) and diff --git a/slayer/embeddings/service.py b/slayer/embeddings/service.py deleted file mode 100644 index 671406f6..00000000 --- a/slayer/embeddings/service.py +++ /dev/null @@ -1,292 +0,0 @@ -"""EmbeddingService — orchestrates embedding refresh + corpus fetch. - -Refresh routines are called from the same write-side edges that maintain -``Column.sampled``: ``save_memory``, ``edit_model``, and ``slayer ingest``. -Each refresh hashes the rendered text of the affected entity, compares to -the stored ``content_hash`` for ``(canonical_id, embedding_model_name)``, -and only calls litellm when the text has actually changed. - -Per-entity embed failures are non-fatal: the corresponding row is simply -not written. Search degrades gracefully via the remaining tantivy + BM25 -channels. When the ``embedding_search`` extra is not installed, -``is_available()`` returns ``False`` and all refresh methods short-circuit -to "no-op + warning". -""" - -from __future__ import annotations - -import hashlib -import logging -from typing import List, Optional, Tuple - -from slayer.core.models import SlayerModel -from slayer.embeddings import client as embedding_client -from slayer.embeddings.client import current_model, embed_batch -from slayer.embeddings.models import Embedding, EntityKind -from slayer.memories.models import MEMORY_CANONICAL_PREFIX as _MEMORY_PREFIX -from slayer.memories.models import Memory -from slayer.search.render import ( - render_aggregation_text, - render_column_text, - render_datasource_text, - render_measure_text, - render_memory_text_for_embedding, - render_model_text, -) -from slayer.storage.base import StorageBackend - - -_log = logging.getLogger(__name__) - - -def _sha256(text: str) -> str: - return hashlib.sha256(text.encode("utf-8")).hexdigest() - - -def _memory_canonical_id(memory_id: str) -> str: - return f"{_MEMORY_PREFIX}{memory_id}" - - -def _model_canonical_id(model: SlayerModel) -> str: - return f"{model.data_source}.{model.name}" - - -def _column_canonical_id(model: SlayerModel, column_name: str) -> str: - return f"{model.data_source}.{model.name}.{column_name}" - - -def _measure_canonical_id(model: SlayerModel, measure_name: str) -> str: - return f"{model.data_source}.{model.name}.{measure_name}" - - -def _aggregation_canonical_id(model: SlayerModel, aggregation_name: str) -> str: - return f"{model.data_source}.{model.name}.{aggregation_name}" - - -class _PendingRefresh: - """One unit of work — rendered text needing an embedding.""" - - __slots__ = ("canonical_id", "entity_kind", "text", "content_hash") - - canonical_id: str - entity_kind: EntityKind - text: str - content_hash: str - - def __init__( - self, - *, - canonical_id: str, - entity_kind: EntityKind, - text: str, - ) -> None: - self.canonical_id = canonical_id - self.entity_kind = entity_kind - self.text = text - self.content_hash = _sha256(text) - - -class EmbeddingService: - """Orchestrates refresh + corpus retrieval for embedding-based search.""" - - def __init__( - self, - *, - storage: StorageBackend, - model_name: Optional[str] = None, - ) -> None: - self._storage = storage - self._model_name = model_name or current_model() - - @property - def model_name(self) -> str: - return self._model_name - - # ------------------------------------------------------------------ - # Refresh — write-side hooks - # ------------------------------------------------------------------ - - async def refresh_memory(self, memory: Memory) -> List[str]: - """Refresh the embedding for a single memory. Returns warning - strings (empty on success or hash-skip).""" - if not embedding_client.is_available(): - # Channel disabled (no extra installed, or no API key - # configured for the active embedding model). Stay silent on - # the write path — this is "feature not configured", not a - # runtime failure. The search-side surface emits one - # user-visible warning into ``SearchResponse.warnings`` on - # the next query. - return [] - pending = _PendingRefresh( - canonical_id=_memory_canonical_id(memory.id), - entity_kind="memory", - text=render_memory_text_for_embedding(memory=memory), - ) - return await self._apply_pending([pending]) - - async def refresh_datasource( - self, *, name: str, models: List[SlayerModel], - ) -> List[str]: - """Refresh the embedding for one datasource doc.""" - if not embedding_client.is_available(): - # Channel disabled (no extra installed, or no API key - # configured for the active embedding model). Stay silent on - # the write path — this is "feature not configured", not a - # runtime failure. The search-side surface emits one - # user-visible warning into ``SearchResponse.warnings`` on - # the next query. - return [] - pending = _PendingRefresh( - canonical_id=name, - entity_kind="datasource", - text=render_datasource_text(name=name, models=models), - ) - return await self._apply_pending([pending]) - - async def refresh_model_subtree(self, model: SlayerModel) -> List[str]: - """Refresh the model doc + every visible column + named measures + - custom aggregations in a single batch call. - - Hidden models / hidden columns are skipped entirely (matches the - tantivy indexing rules). - """ - if not embedding_client.is_available(): - # Channel disabled (no extra installed, or no API key - # configured for the active embedding model). Stay silent on - # the write path — this is "feature not configured", not a - # runtime failure. The search-side surface emits one - # user-visible warning into ``SearchResponse.warnings`` on - # the next query. - return [] - if model.hidden: - return [] - pending: List[_PendingRefresh] = [] - pending.append(_PendingRefresh( - canonical_id=_model_canonical_id(model), - entity_kind="model", - text=render_model_text(model=model), - )) - for column in model.columns: - if column.hidden: - continue - pending.append(_PendingRefresh( - canonical_id=_column_canonical_id(model, column.name), - entity_kind="column", - text=render_column_text(model=model, column=column), - )) - for measure in model.measures: - if measure.name is None: - continue - pending.append(_PendingRefresh( - canonical_id=_measure_canonical_id(model, measure.name), - entity_kind="measure", - text=render_measure_text(model=model, measure=measure), - )) - for aggregation in model.aggregations: - pending.append(_PendingRefresh( - canonical_id=_aggregation_canonical_id(model, aggregation.name), - entity_kind="aggregation", - text=render_aggregation_text(model=model, aggregation=aggregation), - )) - return await self._apply_pending(pending) - - # ------------------------------------------------------------------ - # Read — search-side - # ------------------------------------------------------------------ - - async def fetch_corpus(self) -> List[Embedding]: - """Return every embedding row under the active model name.""" - return await self._storage.list_embeddings( - embedding_model_name=self._model_name, - ) - - async def embed_question(self, question: str) -> Optional[List[float]]: - """Embed a search query string. ``None`` when unavailable / failed. - - Calls through the module attribute (``embedding_client.embed_query``) - rather than an import-time binding so tests can monkeypatch the - client module without having to also reach into this module. - """ - return await embedding_client.embed_query( - question, model=self._model_name, - ) - - # ------------------------------------------------------------------ - # Internals - # ------------------------------------------------------------------ - - async def _apply_pending( - self, pending: List[_PendingRefresh], - ) -> List[str]: - """Hash-skip, batch-embed, and persist. Returns warning strings. - - DEV-1405: hot-path uses two batched storage round-trips per call — - one ``get_embeddings_for_canonical_ids`` for the hash-skip filter, - one ``save_embeddings`` for the persist step. The previous code - did M point ``get_embedding`` + M point ``save_embedding`` calls. - """ - if not pending: - return [] - stale, fresh_count = await self._filter_stale(pending) - if not stale: - return [] - texts = [p.text for p in stale] - vectors = await embed_batch(texts, model=self._model_name) - warnings: List[str] = [] - rows: List[Embedding] = [] - for p, vec in zip(stale, vectors): - if vec is None: - warnings.append( - f"embedding refresh failed for {p.canonical_id}; " - f"skipped (search will still find this entity via " - f"tantivy + BM25)." - ) - continue - rows.append(Embedding( - canonical_id=p.canonical_id, - embedding_model_name=self._model_name, - entity_kind=p.entity_kind, - content_hash=p.content_hash, - embedding=vec, - )) - if rows: - try: - await self._storage.save_embeddings(rows) - except Exception as exc: # NOSONAR(S112) — best-effort persistence - # Include canonical ids so a caller doing failure - # attribution by entity (e.g. ``ingest_datasource_idempotent`` - # tagging memory failures as ``model_name="memory:"``) - # can see which rows did not land. - canonical_ids = ", ".join(r.canonical_id for r in rows) - warnings.append( - f"embedding batch persist failed for " - f"{len(rows)} row(s) [{canonical_ids}]: {exc}" - ) - _log.debug( - "EmbeddingService: refreshed=%d stale=%d total=%d warnings=%d", - fresh_count, len(stale), len(pending), len(warnings), - ) - return warnings - - async def _filter_stale( - self, pending: List[_PendingRefresh], - ) -> Tuple[List[_PendingRefresh], int]: - """Drop pending entries whose stored content_hash already matches. - - Returns ``(stale_entries, fresh_skipped_count)``. DEV-1405: one - batched ``get_embeddings_for_canonical_ids`` call replaces the - previous M-iteration point-read loop. - """ - existing = await self._storage.get_embeddings_for_canonical_ids( - canonical_ids=[p.canonical_id for p in pending], - embedding_model_name=self._model_name, - ) - stale: List[_PendingRefresh] = [] - fresh = 0 - for p in pending: - match = existing.get(p.canonical_id) - if match is not None and match.content_hash == p.content_hash: - fresh += 1 - continue - stale.append(p) - return stale, fresh diff --git a/slayer/engine/aggregate_input_paths.py b/slayer/engine/aggregate_input_paths.py new file mode 100644 index 00000000..fc669216 --- /dev/null +++ b/slayer/engine/aggregate_input_paths.py @@ -0,0 +1,184 @@ +"""DEV-1709 (Stage 5) — plan-time crossing-input discovery for aggregates. + +The widened Law-3 trigger isolates a LOCAL aggregate into a host-rooted CTE +when ANY of its explicit inputs crosses a join. This module answers "which +join paths do the aggregate's inputs cross?" for every input kind: + +* **source** — a structural ``source.path`` contributes as-is; a derived + ``ColumnSqlKey`` with ``path == ()`` has its ``Column.sql`` expanded and + scanned with the shared Law-1 scanner (via + ``compute_column_filter_join_paths``, the same parse → expand → walk + pipeline the ``Column.filter`` trigger half uses). +* **positional args** (covers the explicit first/last time arg) — same + structural + derived-sql treatment. +* **kwargs** — column-valued kwargs same as args; template-fragment STRING + kwargs (user-supplied values for custom-aggregation params) are parsed + with the dialect-fallback chain and scanned. Model-default + ``AggregationParam.sql`` fragments of the custom aggregation named by + ``key.agg`` are scanned too — but only for params NOT overridden by a + user kwarg (an overridden default never renders). +* **``column_filter_key`` is deliberately NOT re-scanned** — the trigger + reads its bind-time ``SqlExprKey.referenced_join_paths`` directly + (DEV-1503, unchanged). + +Defensive fallbacks mirror ``column_filter_paths.py``: an unparseable +fragment contributes nothing (parity with the ``Column.filter`` scan — +pre-Stage-5 behavior is preserved for fragments the dialect fallback chain +cannot parse; a documented D1 carve-out, not an endorsement), and scalar / +duration / literal kwarg values contribute nothing. +""" + +from __future__ import annotations + +from typing import List, Optional, Tuple, Union + +from slayer.core.keys import AggregateKey, ColumnKey, ColumnSqlKey, StarKey +from slayer.core.models import SlayerModel +from slayer.engine.column_filter_paths import compute_column_filter_join_paths +from slayer.engine.source_bundle import ResolvedSourceBundle + +_PathList = List[Tuple[str, ...]] +_StructuralRef = Union[ColumnKey, ColumnSqlKey, StarKey] + + +def _add_path_prefixes(path: Tuple[str, ...], out: _PathList) -> None: + """Emit every prefix of ``path`` once (``("a", "b")`` → ``("a",)`` AND + ``("a", "b")``) — same prefix semantics as the Law-1 scanner.""" + for i in range(1, len(path) + 1): + prefix = tuple(path[:i]) + if prefix not in out: + out.append(prefix) + + +def _scan_sql_fragment( + sql: str, + *, + anchor_model: SlayerModel, + anchor_relation: str, + bundle: ResolvedSourceBundle, + out: _PathList, +) -> None: + """Scan a free-SQL fragment (derived ``Column.sql`` or a template + fragment) for crossed join paths, reusing the filter-side pipeline + (dialect-fallback parse → anchor-derived expansion → root-scope walk). + Unparseable fragments contribute nothing.""" + for path in compute_column_filter_join_paths( + canonical_sql=sql, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + ): + if path not in out: + out.append(path) + + +def _collect_ref_paths( + ref: object, + *, + anchor_model: SlayerModel, + anchor_relation: str, + bundle: ResolvedSourceBundle, + out: _PathList, +) -> None: + """Crossed paths of one embedded reference (source / arg / kwarg value). + + Scalars (Decimal / int / float / None) contribute nothing; strings are + template fragments and get the free-SQL scan. + """ + if isinstance(ref, (ColumnKey, StarKey)): + _add_path_prefixes(tuple(getattr(ref, "path", ()) or ()), out) + return + if isinstance(ref, ColumnSqlKey): + if ref.path: + # Structural crossing; any FURTHER crossing inside the target's + # own Column.sql is the target-rooted CTE's concern (Stage 4). + _add_path_prefixes(tuple(ref.path), out) + return + col = next( + (c for c in anchor_model.columns if c.name == ref.column_name), + None, + ) + if col is not None and col.sql: + _scan_sql_fragment( + col.sql, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + return + if isinstance(ref, str): + _scan_sql_fragment( + ref, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + + +def _collect_default_fragment_paths( + key: AggregateKey, + *, + anchor_model: SlayerModel, + anchor_relation: str, + bundle: ResolvedSourceBundle, + out: _PathList, +) -> None: + """Scan the model-default ``AggregationParam.sql`` fragments of the + custom aggregation named by ``key.agg`` — skipping params a user kwarg + overrides (the default never renders for those).""" + agg_def = next( + (a for a in (anchor_model.aggregations or []) if a.name == key.agg), + None, + ) + if agg_def is None: + return + overridden = {name for name, _ in key.kwargs} + for param in agg_def.params or []: + param_sql: Optional[str] = getattr(param, "sql", None) + if param.name in overridden or not param_sql: + continue + _scan_sql_fragment( + param_sql, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + + +def compute_aggregate_input_join_paths( + *, + key: AggregateKey, + anchor_model: Optional[SlayerModel], + anchor_relation: str, + bundle: ResolvedSourceBundle, +) -> Tuple[Tuple[str, ...], ...]: + """Ordered, de-duplicated tuple of join-path prefixes crossed by the + aggregate's explicit inputs (source, positional args, kwargs, and + non-overridden custom-aggregation default fragments). + + ``()`` for a purely-local aggregate. ``column_filter_key`` crossing is + intentionally excluded — read ``referenced_join_paths`` on the key. + """ + if anchor_model is None: + return () + out: _PathList = [] + refs: List[object] = [key.source, *key.args, *(v for _, v in key.kwargs)] + for ref in refs: + _collect_ref_paths( + ref, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + _collect_default_fragment_paths( + key, + anchor_model=anchor_model, + anchor_relation=anchor_relation, + bundle=bundle, + out=out, + ) + return tuple(out) diff --git a/slayer/engine/binding.py b/slayer/engine/binding.py index a75b0d76..0a8eb811 100644 --- a/slayer/engine/binding.py +++ b/slayer/engine/binding.py @@ -45,9 +45,12 @@ UnknownReferenceError, ) from slayer.core.enums import ( + BUILTIN_AGGREGATIONS, DEFAULT_AGGREGATIONS_BY_TYPE, PRIMARY_KEY_AGGREGATIONS, DataType, + format_unknown_aggregation, + normalize_aggregation_name, ) from slayer.core.keys import ( SCALAR_FUNCTIONS, @@ -831,12 +834,15 @@ def _bind_agg( # silently in the typed pipeline. The check is best-effort against # the bundle — sources whose target model can't be resolved (e.g. # an unreferenced join target) skip the check. - _validate_agg_eligibility( + # DEV-1576 / DEV-1717: heal alias + gate, then store the EFFECTIVE + # (healed) name on the key so the generator resolves the canonical + # aggregation rather than the raw parser token. + effective_agg = _validate_agg_eligibility( source=source, agg=parsed.agg, bundle=bundle, ) return AggregateKey( source=source, - agg=parsed.agg, + agg=effective_agg, args=args, kwargs=kwargs, column_filter_key=column_filter_key, @@ -891,20 +897,59 @@ def _resolve_column_filter_key( return SqlExprKey(canonical_sql=col.filter, referenced_join_paths=paths) -def _validate_agg_eligibility( - *, source, agg: str, bundle: ResolvedSourceBundle, -) -> None: - """Enforce per-column aggregation eligibility gates. +def _resolve_gate_owner( + source, bundle: ResolvedSourceBundle, +) -> "Optional[tuple[SlayerModel, str]]": + """Resolve the ``(owning_model, leaf)`` an aggregation gate applies to. - Mirrors the legacy ``enrichment.py:401-417`` v2 contract: + Returns ``None`` when the target can't be confirmed — a ``StarKey`` + (``*:count`` has no column), a source with no leaf, no host model, or an + unresolved join hop — so the caller best-effort skips the gate (the + compile-time path validator catches truly broken refs). + """ + if isinstance(source, StarKey): + return None + leaf = getattr(source, "leaf", None) or getattr(source, "column_name", None) + if leaf is None: + return None + host = bundle.source_model + if host is None: + return None + current: SlayerModel = host + for hop in tuple(getattr(source, "path", ())): + nxt = bundle.get_referenced_model(hop) + if nxt is None: + return None + current = nxt + return current, leaf - 1. Primary-key columns are always restricted to ``count`` / - ``count_distinct`` regardless of type or explicit whitelist. + +def _validate_agg_eligibility( + *, source, agg: str, bundle: ResolvedSourceBundle, +) -> str: + """Heal the aggregation name and enforce per-column eligibility gates. + + Returns the **effective** (alias-healed) aggregation name, which the + caller stores on ``AggregateKey.agg`` (DEV-1576 / DEV-1717) — the typed + colon parser (``syntax.py``) does not normalise, so healing must land + here, after the owning model is resolved, or the generator later fails on + the raw token at ``_resolve_aggregation_def``. + + Healing (:func:`normalize_aggregation_name`) is **skipped** when the raw + token exactly matches a custom aggregation registered on the owning model, + so a custom ``countd`` wins over the ``countd -> count_distinct`` alias. + + Gate order (mirrors the legacy ``enrichment.py`` v2 contract): + + 0. Unknown-name-first: a name that is neither a built-in nor a model + custom aggregation raises ``"Unknown aggregation ..."`` **before** the + PK / whitelist / type gates, so a misspelled agg on an otherwise + aggregatable column is not mislabelled as a type restriction. + 1. Primary-key columns are restricted to ``count`` / ``count_distinct``. 2. An explicit ``Column.allowed_aggregations`` whitelist overrides type defaults. 3. Otherwise, built-in aggregations are gated by - ``DEFAULT_AGGREGATIONS_BY_TYPE``; model-custom aggregations - (registered in ``SlayerModel.aggregations``) are exempt. + ``DEFAULT_AGGREGATIONS_BY_TYPE``; model-custom aggregations are exempt. ``StarKey`` sources (``*:count`` / ``customers.*:count``) have no column to attach a whitelist to and pass through. Cross-model and @@ -913,60 +958,57 @@ def _validate_agg_eligibility( target) the gate is skipped — the compile-time path validator would have raised earlier on a truly broken ref. """ - if isinstance(source, StarKey): - return - path = tuple(getattr(source, "path", ())) - leaf = getattr(source, "leaf", None) or getattr(source, "column_name", None) - if leaf is None: - return - host = bundle.source_model - if host is None: - return - current: SlayerModel = host - for hop in path: - nxt = bundle.get_referenced_model(hop) - if nxt is None: - return - current = nxt + owner = _resolve_gate_owner(source, bundle) + if owner is None: + return normalize_aggregation_name(agg) + current, leaf = owner + # DEV-1576 alias healing — custom aggregation named like an alias wins. + custom_names = {a.name for a in (current.aggregations or [])} + effective = agg if agg in custom_names else normalize_aggregation_name(agg) + # Gate 0: unknown-name-first (precedence over PK / whitelist / type). + known = BUILTIN_AGGREGATIONS | custom_names + if effective not in known: + raise ValueError(format_unknown_aggregation(effective, known)) col = next((c for c in current.columns if c.name == leaf), None) if col is None: - return + return effective if col.primary_key: - if agg not in PRIMARY_KEY_AGGREGATIONS: + if effective not in PRIMARY_KEY_AGGREGATIONS: raise AggregationNotAllowedError( column=leaf, - agg=agg, + agg=effective, reason=( f"primary-key column {leaf!r} restricted to " - f"{sorted(PRIMARY_KEY_AGGREGATIONS)}; got {agg!r}." + f"{sorted(PRIMARY_KEY_AGGREGATIONS)}; got {effective!r}." ), ) - return + return effective if col.allowed_aggregations is not None: - if agg not in col.allowed_aggregations: + if effective not in col.allowed_aggregations: raise AggregationNotAllowedError( column=leaf, - agg=agg, + agg=effective, reason=( f"column {leaf!r} restricts allowed_aggregations to " - f"{sorted(col.allowed_aggregations)}; got {agg!r}." + f"{sorted(col.allowed_aggregations)}; got {effective!r}." ), ) - return + return effective # Model-custom aggregations are exempt from the type-default gate. - if any(a.name == agg for a in (current.aggregations or [])): - return + if effective in custom_names: + return effective allowed = DEFAULT_AGGREGATIONS_BY_TYPE.get(col.type, frozenset()) - if agg not in allowed: + if effective not in allowed: raise AggregationNotAllowedError( column=leaf, - agg=agg, + agg=effective, reason=( - f"aggregation {agg!r} is not applicable to " + f"aggregation {effective!r} is not applicable to " f"{col.type} column {leaf!r}; default aggregations are " f"{sorted(allowed)}." ), ) + return effective def _bind_agg_arg( diff --git a/slayer/engine/cache.py b/slayer/engine/cache.py new file mode 100644 index 00000000..8ca2631f --- /dev/null +++ b/slayer/engine/cache.py @@ -0,0 +1,336 @@ +"""Per-engine, in-memory query result cache (DEV-1587). + +A query-level result cache local to a single :class:`SlayerQueryEngine` +instance, modelled on Cube's in-memory cache. Caching is opt-in per call +via ``execute(query, cache=True)``. Staleness is governed by an optional +time-to-live (``ttl_seconds``, checked lazily on read) and an optional set +of Cube-style ``(physical_table, select_expression)`` refresh keys scanned +by an explicit ``engine.refresh()``. + +This module holds the DB-free half of the feature: the cache dict, the +cache key, TTL bookkeeping, sqlglot-based table detection (CTE-alias +excluding), refresh-key applicability, refresh-key scan-SQL building, and +value comparison. All of it is unit-testable without a database. The +engine (``slayer/engine/query_engine.py``) owns the DB awaits (data query, +refresh-key scans, re-execution) and never holds the cache lock across +one. +""" + +import asyncio +import hashlib +import time +from collections.abc import Callable +from typing import Any + +from pydantic import BaseModel, ConfigDict, Field, field_validator +from sqlglot import exp, parse_one +from sqlglot.optimizer.normalize_identifiers import normalize_identifiers +from sqlglot.optimizer.scope import Scope, traverse_scope + +# Marker alias prefix for refresh-key scan projections. Also lets tests +# distinguish a refresh-key scan query from a data query. +_RK_ALIAS_PREFIX = "slayer_rk_" + +# Normalized physical-table identity: (catalog, db, name). Parts absent +# from the source expression are ``None``. +NormalizedTable = tuple[str | None, str | None, str] + + +class CacheConfig(BaseModel): + """Per-engine cache configuration. + + ``ttl_seconds`` bounds wall-clock entry age (``None`` => no time-based + expiry). ``refresh_keys`` is a sequence of ``(physical_table, + select_expression)`` pairs; the same table may repeat with different + expressions. Each ``select_expression`` is a scalar SQL expression + evaluated verbatim as ``SELECT FROM `` — the + user supplies it in full (SLayer does NOT wrap it in ``MAX(...)``). + + The model is **frozen** and ``refresh_keys`` is a tuple, so the config is + genuinely immutable (mirrors ``SessionPolicy``). To change an engine's + policy, reassign ``engine.cache_config`` — the setter clears the cache so + stale entries can't survive under a new TTL / refresh-key set. In-place + mutation (``engine.cache_config.refresh_keys += ...``) is rejected rather + than silently leaving already-cached entries on the old policy. + """ + + model_config = ConfigDict(frozen=True) + + ttl_seconds: float | None = None + refresh_keys: tuple[tuple[str, str], ...] = () + + @field_validator("refresh_keys", mode="before") + @classmethod + def _coerce_refresh_keys(cls, v: Any) -> Any: + """Accept a list of lists/tuples and freeze it into a tuple of tuples.""" + if v is None: + return () + return tuple(tuple(pair) for pair in v) + + +class RefreshKeyValue(BaseModel): + """A captured refresh-key baseline: the value of ``expression`` scanned + from ``table`` at cache-write (or last-refresh) time.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + table: str + expression: str + value: Any = None + + +class RefreshError(BaseModel): + """A per-entry / per-table failure recorded during ``refresh()``. + + ``phase`` is ``"refresh_key_scan"`` (a table's batched scan raised) or + ``"re_execute"`` (re-running a stale entry raised). ``key`` is the cache + key for a re-execute failure, or the physical table for a scan failure. + """ + + key: str + phase: str + message: str + + +class RefreshResult(BaseModel): + """Outcome of ``engine.refresh()``. + + ``refreshed`` — keys re-run because an applicable refresh-key value + moved. ``expired_refreshed`` — keys re-run because their TTL lapsed. + ``unchanged`` — keys left as-is. ``errors`` — continue-on-failure + diagnostics. + """ + + refreshed: list[str] = Field(default_factory=list) + expired_refreshed: list[str] = Field(default_factory=list) + unchanged: list[str] = Field(default_factory=list) + errors: list[RefreshError] = Field(default_factory=list) + + +class _CacheEntry(BaseModel): + """One cached result plus everything needed to re-scan its refresh keys + and re-prepare + re-execute it from the original user input. + + ``response`` holds a :class:`SlayerResponse` (typed ``Any`` to avoid a + circular import with ``query_engine``). ``original_input`` is the raw + user input shape (``SlayerQuery`` / ``dict`` / ``list`` / ``str``) so + ``refresh()`` can replay through the full ``execute`` normalization. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + response: Any + sql: str + ds_fingerprint: str + dialect: str + ds_key: tuple[str, str] + resolved_data_source: str | None = None + original_input: Any = None + variables: dict[str, Any] | None = None + data_source: str | None = None + created_at: float = 0.0 + applicable: list[tuple[str, str]] = Field(default_factory=list) + refresh_key_values: list[RefreshKeyValue] = Field(default_factory=list) + + +class QueryCache: + """In-memory ``dict[str, _CacheEntry]`` with TTL-aware reads. + + The :class:`asyncio.Lock` guards only in-memory dict operations + (get / put / delete / snapshot / commit) — the engine performs every DB + await outside the lock. The ``clock`` is injectable for deterministic + TTL testing (``time.monotonic`` by default). + """ + + def __init__( + self, + config: CacheConfig, + clock: Callable[[], float] = time.monotonic, + ) -> None: + self.config = config + self._clock = clock + self._entries: dict[str, _CacheEntry] = {} + self._lock = asyncio.Lock() + + # ---- key / clock / size ------------------------------------------------ + + @staticmethod + def make_key(sql: str, ds_fingerprint: str) -> str: + """``sha256(final_sql + "|" + ds_fingerprint)``. + + ``ds_fingerprint`` is the engine's SQL-client cache fingerprint + (``connection_string|runtime_fingerprint``), so a config edit under + the same datasource name never serves the wrong rows. + """ + return hashlib.sha256(f"{sql}|{ds_fingerprint}".encode()).hexdigest() + + def now(self) -> float: + """Read the injectable clock (used for entry ``created_at``).""" + return self._clock() + + def size(self) -> int: + return len(self._entries) + + def clear(self) -> None: + self._entries.clear() + + # ---- lock-guarded dict ops -------------------------------------------- + + async def get(self, key: str) -> _CacheEntry | None: + """Return the live entry, or ``None``. TTL-expired entries are + deleted and reported as a miss (re-execution re-populates them).""" + async with self._lock: + entry = self._entries.get(key) + if entry is None: + return None + if self.config.ttl_seconds is not None: + if (self._clock() - entry.created_at) > self.config.ttl_seconds: + del self._entries[key] + return None + return entry + + async def put(self, key: str, entry: _CacheEntry) -> None: + async with self._lock: + self._entries[key] = entry + + async def delete(self, key: str) -> bool: + async with self._lock: + if key in self._entries: + del self._entries[key] + return True + return False + + async def snapshot(self) -> dict[str, _CacheEntry]: + """A shallow copy of the ``{key: entry}`` map so ``refresh()`` can + iterate without racing concurrent ``execute()`` writes.""" + async with self._lock: + return dict(self._entries) + + async def commit_replace( + self, + *, + old_key: str, + expected: _CacheEntry, + new_key: str, + new_entry: _CacheEntry, + ) -> bool: + """Identity-guarded write used by ``refresh()``. + + Only mutate if the live entry at ``old_key`` is still the SAME + object ``refresh()`` snapshotted. If it was evicted / cleared / + replaced by a newer ``execute()`` during ``refresh()``'s DB awaits, + skip the write (return ``False``) — never resurrect a gone entry or + clobber a newer result. On a re-key (``new_key != old_key``) the old + key is dropped; a ``new_key`` collision is last-writer-wins (both + results are interchangeable — identical SQL + ds fingerprint). + """ + async with self._lock: + if self._entries.get(old_key) is not expected: + return False + if new_key != old_key: + self._entries.pop(old_key, None) + self._entries[new_key] = new_entry + return True + + # ---- table detection --------------------------------------------------- + + def parse_referenced_tables( + self, sql: str, dialect: str + ) -> list[NormalizedTable]: + """Normalized physical tables referenced by ``sql``. + + Parses with sqlglot and uses **scope analysis** (``traverse_scope``, + the same mechanism the forced-filter policy uses) to keep only + genuinely physical tables: a table reference whose name resolves to a + CTE / derived table in its scope is skipped. This is name-collision + safe — a physical table that happens to share a name with a CTE alias + (SLayer wraps queries in CTEs) is still detected, because scope + resolution distinguishes them structurally rather than by bare name. + Each physical :class:`exp.Table` is normalized to ``(catalog, db, + name)`` with the dialect's identifier folding (quoted identifiers + preserved exactly, unquoted folded per dialect). + """ + tree = parse_one(sql, dialect=dialect) + out: list[NormalizedTable] = [] + for scope in traverse_scope(tree): + for table in scope.tables: + # A qualified reference (has a db/catalog part) can never be a + # CTE — CTE names are always unqualified — so it is always + # physical. Only an unqualified name can shadow a CTE / derived + # source in its scope; skip those (they aren't physical tables). + if not table.db and not table.catalog: + if isinstance(scope.sources.get(table.alias_or_name), Scope): + continue + out.append(self._normalize_table_expr(table, dialect)) + return out + + @staticmethod + def _normalize_table_expr(table: exp.Table, dialect: str) -> NormalizedTable: + norm = normalize_identifiers(table.copy(), dialect=dialect) + return (norm.catalog or None, norm.db or None, norm.name) + + @classmethod + def _normalize_config_table(cls, table: str, dialect: str) -> NormalizedTable: + return cls._normalize_table_expr(exp.to_table(table, dialect=dialect), dialect) + + @staticmethod + def _table_matches(config: NormalizedTable, sql_table: NormalizedTable) -> bool: + """A config table matches a SQL table iff their normalized parts are + equal, treating parts unspecified (``None``) in the config as + wildcards. So ``orders`` matches ``orders`` / ``public.orders`` / + ``db.public.orders``; ``public.orders`` matches only ``db=public``. + """ + c_cat, c_db, c_name = config + s_cat, s_db, s_name = sql_table + if c_name != s_name: + return False + if c_db is not None and c_db != s_db: + return False + if c_cat is not None and c_cat != s_cat: + return False + return True + + def applicable_keys(self, sql: str, dialect: str) -> list[tuple[str, str]]: + """The configured ``(table, expression)`` refresh keys whose table is + referenced by ``sql``. The original config table string is preserved + (duplicate expressions per table are kept).""" + sql_tables = self.parse_referenced_tables(sql, dialect) + out: list[tuple[str, str]] = [] + for table, expr in self.config.refresh_keys: + config_norm = self._normalize_config_table(table, dialect) + if any(self._table_matches(config_norm, s) for s in sql_tables): + out.append((table, expr)) + return out + + def build_refresh_key_sql( + self, table: str, expressions: list[str], dialect: str + ) -> str: + """``SELECT () AS "slayer_rk_0", ... FROM
``. + + Each user expression is parsed with the entry's dialect and + re-emitted so quoting/dialect are consistent; the table identifier is + quoted dialect-safely. One scan covers all of a table's applicable + refresh keys. + """ + projections = [ + exp.alias_( + exp.paren(parse_one(e, dialect=dialect)), + f"{_RK_ALIAS_PREFIX}{i}", + quoted=True, + ) + for i, e in enumerate(expressions) + ] + select = exp.select(*projections).from_(exp.to_table(table, dialect=dialect)) + return select.sql(dialect=dialect) + + @staticmethod + def rk_alias(index: int) -> str: + """The scan projection alias for the ``index``-th expression.""" + return f"{_RK_ALIAS_PREFIX}{index}" + + @staticmethod + def values_differ(a: Any, b: Any) -> bool: + """Equality comparison across DB scalar types (timestamps / ints / + strings / concatenations). Any inequality — including a decrease — + signals staleness.""" + return a != b diff --git a/slayer/engine/column_dependency.py b/slayer/engine/column_dependency.py index c3cfaf99..3211c2f5 100644 --- a/slayer/engine/column_dependency.py +++ b/slayer/engine/column_dependency.py @@ -18,7 +18,7 @@ from __future__ import annotations from collections import deque -from typing import TYPE_CHECKING, Deque, Dict, List, Optional, Set, Tuple +from typing import TYPE_CHECKING import sqlglot from sqlglot import exp @@ -26,6 +26,7 @@ from slayer.core.errors import ColumnCycleError from slayer.core.models import Column, SlayerModel from slayer.engine.column_expansion import _is_trivial_base, _root_scope_column_ids +from slayer.sql.reserved_keywords import prequote_reserved_identifiers if TYPE_CHECKING: from slayer.storage.base import StorageBackend @@ -35,15 +36,15 @@ # ``exp.Column`` identifier shape — dialect choice does not change which # columns appear in the AST. Using sqlglot's default keeps the validator # independent of the model's runtime datasource dialect. -_DEPENDENCY_DIALECT: Optional[str] = None +_DEPENDENCY_DIALECT: str | None = None def _resolve_target_for_ref( *, - table_alias: Optional[str], + table_alias: str | None, host: SlayerModel, - reachable: Dict[str, SlayerModel], -) -> Optional[SlayerModel]: + reachable: dict[str, SlayerModel], +) -> SlayerModel | None: """Return the model that a column reference resolves to, or ``None``. Mirrors the runtime alias resolution in @@ -85,9 +86,9 @@ def _resolve_single_column( *, node: exp.Column, host: SlayerModel, - reachable: Dict[str, SlayerModel], - root_ids: Set[int], -) -> Optional[Tuple[str, str]]: + reachable: dict[str, SlayerModel], + root_ids: set[int], +) -> tuple[str, str] | None: """Resolve one ``exp.Column`` node to a (model_name, column_name) dep, or ``None`` if the node is out of scope. @@ -116,8 +117,8 @@ def _column_dependencies( *, column: Column, host: SlayerModel, - reachable: Dict[str, SlayerModel], -) -> List[Tuple[str, str]]: + reachable: dict[str, SlayerModel], +) -> list[tuple[str, str]]: """Extract the root-scope derived-column dependencies of ``column``. Returns a list of ``(model_name, column_name)`` tuples — only refs @@ -128,14 +129,20 @@ def _column_dependencies( if column.sql is None or _is_trivial_base(column=column): return [] try: - parsed = sqlglot.parse_one(column.sql, dialect=_DEPENDENCY_DIALECT) + # DEV-1686: prequote reserved qualifiers/leaves so a derived column + # referencing a reserved joined model (e.g. ``grant.amount``) parses + # cleanly here instead of falling back to a noisy ``Command`` parse. + parsed = sqlglot.parse_one( + prequote_reserved_identifiers(sql=column.sql, dialect=_DEPENDENCY_DIALECT), + dialect=_DEPENDENCY_DIALECT, + ) except Exception: # Parse failure on a save attempt — let the actual save proceed so # the surface-level error (storage / pydantic) is what the user # sees, not a noisy validator complaint about unparseable SQL. return [] root_ids = _root_scope_column_ids(parsed=parsed) - deps: List[Tuple[str, str]] = [] + deps: list[tuple[str, str]] = [] for node in parsed.find_all(exp.Column): resolved = _resolve_single_column( node=node, host=host, reachable=reachable, root_ids=root_ids, @@ -147,9 +154,9 @@ def _column_dependencies( def _node_dependencies( *, - node: Tuple[str, str], - reachable: Dict[str, SlayerModel], -) -> List[Tuple[str, str]]: + node: tuple[str, str], + reachable: dict[str, SlayerModel], +) -> list[tuple[str, str]]: """Return the dependency edges leaving ``node = (model_name, col_name)``. Empty list when the model or column is missing — those are dead-ends, not errors. @@ -166,12 +173,12 @@ def _node_dependencies( def _dfs_visit( *, - node: Tuple[str, str], - reachable: Dict[str, SlayerModel], - on_stack: List[Tuple[str, str]], - on_stack_set: Set[Tuple[str, str]], - visited: Set[Tuple[str, str]], -) -> Optional[List[Tuple[str, str]]]: + node: tuple[str, str], + reachable: dict[str, SlayerModel], + on_stack: list[tuple[str, str]], + on_stack_set: set[tuple[str, str]], + visited: set[tuple[str, str]], +) -> list[tuple[str, str]] | None: """Recursive DFS visit. Returns the first cycle reachable from ``node``, or ``None``. Mutates ``on_stack`` / ``on_stack_set`` / ``visited`` in place — the caller initialises them empty and @@ -199,9 +206,9 @@ def _dfs_visit( def _detect_cycle_dfs( *, - start: Tuple[str, str], - reachable: Dict[str, SlayerModel], -) -> Optional[List[Tuple[str, str]]]: + start: tuple[str, str], + reachable: dict[str, SlayerModel], +) -> list[tuple[str, str]] | None: """DFS from ``start = (model_name, column_name)``. Returns the first cycle found as an ordered list (start may appear at both ends if the cycle closes through it), or ``None`` if the subgraph is acyclic. @@ -216,14 +223,14 @@ async def _prefetch_reachable_models( *, model: SlayerModel, storage: "StorageBackend", -) -> Dict[str, SlayerModel]: +) -> dict[str, SlayerModel]: """BFS over ``model.joins`` (transitively), pulling each target model in the same ``data_source``. Returns ``{model_name: model}`` including ``model`` itself. Unresolvable target names (model not persisted yet) are silently omitted — save-time is best-effort. """ - out: Dict[str, SlayerModel] = {model.name: model} - queue: Deque[SlayerModel] = deque([model]) + out: dict[str, SlayerModel] = {model.name: model} + queue: deque[SlayerModel] = deque([model]) while queue: current = queue.popleft() for join in current.joins: @@ -259,7 +266,7 @@ async def validate_no_column_cycles( reachable = await _prefetch_reachable_models(model=model, storage=storage) # Iterate roots in a deterministic order so the reported cycle is # stable across runs. - roots: List[Tuple[str, str]] = [] + roots: list[tuple[str, str]] = [] for entity_name in sorted(reachable.keys()): entity = reachable[entity_name] for col in entity.columns: diff --git a/slayer/engine/column_expansion.py b/slayer/engine/column_expansion.py index e928d9b9..0a29e6bb 100644 --- a/slayer/engine/column_expansion.py +++ b/slayer/engine/column_expansion.py @@ -17,7 +17,8 @@ """ from __future__ import annotations -from typing import Any, Awaitable, Callable, Dict, List, Optional, Protocol, Set, Tuple +from collections.abc import Awaitable, Callable +from typing import Any, List, Optional, Protocol, Set, Tuple import sqlglot from sqlglot import exp @@ -25,8 +26,9 @@ from slayer.core.errors import ColumnCycleError from slayer.core.models import Column, SlayerModel +from slayer.sql.reserved_keywords import prequote_reserved_identifiers -ResolveModel = Callable[..., Awaitable[Optional[SlayerModel]]] +ResolveModel = Callable[..., Awaitable[SlayerModel | None]] def _is_trivial_base(*, column: Column) -> bool: @@ -35,10 +37,18 @@ def _is_trivial_base(*, column: Column) -> bool: """ if column.sql is None: return True - return column.sql.strip() == column.name - - -def _root_scope_column_ids(*, parsed: exp.Expression) -> Set[int]: + sql = column.sql.strip() + # A double-quoted self-identity (``"legalEntityType"`` for a column named + # ``legalEntityType``) is still a bare base reference — required to point + # at a mixed-case physical column on case-folding dialects. Strip the + # surrounding identifier quotes before comparing so it is not mistaken + # for a derived expression (which would self-recurse into a false cycle). + if len(sql) >= 2 and sql[0] == '"' and sql[-1] == '"': + sql = sql[1:-1].replace('""', '"') + return sql == column.name + + +def _root_scope_column_ids(*, parsed: exp.Expression) -> set[int]: """Return the ``id()`` set of ``exp.Column`` nodes that lexically belong to the root scope of ``parsed`` (DEV-1410). @@ -59,7 +69,7 @@ def _root_scope_column_ids(*, parsed: exp.Expression) -> Set[int]: if not isinstance(parsed, exp.Expression): return set() wrapper = exp.Select(expressions=[exp.Alias(this=parsed.copy(), alias="_")]) - scope_node_ids: Dict[int, ScopeType] = {} + scope_node_ids: dict[int, ScopeType] = {} for scope in traverse_scope(wrapper): scope_node_ids[id(scope.expression)] = scope.scope_type if not scope_node_ids: @@ -81,10 +91,10 @@ def _root_scope_column_ids(*, parsed: exp.Expression) -> Set[int]: # wrapper just wraps a deep copy and ``find_all`` walks in # document order. return set() - root_ids: Set[int] = set() + root_ids: set[int] = set() for w_col, p_col in zip(wrapper_cols, parsed_cols): - node: Optional[exp.Expression] = w_col.parent - scope_type: Optional[ScopeType] = None + node: exp.Expression | None = w_col.parent + scope_type: ScopeType | None = None while node is not None: if id(node) in scope_node_ids: scope_type = scope_node_ids[id(node)] @@ -186,9 +196,9 @@ async def _walk_path_to_target( source_alias: str, table_alias: str, resolve_model: ResolveModel, - named_queries: Dict[str, Any], + named_queries: dict[str, Any], is_root: bool, -) -> Tuple[Optional[SlayerModel], Optional[str]]: +) -> tuple[SlayerModel | None, str | None]: """Resolve a ``table_alias`` (e.g. ``B`` or ``B__C``) seen inside a Column.sql to the terminal joined model and the canonical alias to use in emitted SQL. @@ -234,11 +244,11 @@ async def _process_column_node( model: SlayerModel, alias_path: str, resolve_model: ResolveModel, - named_queries: Dict[str, Any], + named_queries: dict[str, Any], dialect: str, - visited: Tuple[Tuple[str, str], ...], + visited: tuple[tuple[str, str], ...], is_root: bool, - root_scope_ids: Set[int], + root_scope_ids: set[int], ) -> None: """Resolve one ``exp.Column`` node in the parsed AST, mutating it in place. Encapsulates the multi-branch decision that drives expansion: @@ -319,21 +329,26 @@ async def _process_column_node( return # Splice in, parenthesized so the surrounding expression's precedence # is preserved. - expanded_ast = sqlglot.parse_one(expanded_sql, dialect=dialect) + # DEV-1686: quote bare reserved-word qualifiers/leaves (e.g. a derived + # column referencing a reserved joined model like ``grant.amount``) so the + # generated SQL parses. + expanded_ast = sqlglot.parse_one( + prequote_reserved_identifiers(sql=expanded_sql, dialect=dialect), dialect=dialect + ) col.replace(exp.Paren(this=expanded_ast)) async def expand_derived_refs( *, - sql: Optional[str], + sql: str | None, model: SlayerModel, alias_path: str, resolve_model: ResolveModel, - named_queries: Optional[Dict[str, Any]] = None, + named_queries: dict[str, Any] | None = None, dialect: str, - visited: Optional[Tuple[Tuple[str, str], ...]] = None, + visited: tuple[tuple[str, str], ...] | None = None, is_root: bool = True, -) -> Optional[str]: +) -> str | None: """Recursively expand cross-model and local derived-column references inside ``sql``. @@ -363,7 +378,11 @@ async def expand_derived_refs( visited = visited or () named_queries = named_queries or {} - parsed = sqlglot.parse_one(sql, dialect=dialect) + # DEV-1686: prequote reserved-word qualifiers/leaves before parsing user + # ``Column.sql`` (may reference a reserved joined model, e.g. ``grant.x``). + parsed = sqlglot.parse_one( + prequote_reserved_identifiers(sql=sql, dialect=dialect), dialect=dialect + ) # Materialize the columns first — we may mutate them in place via .replace(). column_nodes = list(parsed.find_all(exp.Column)) # DEV-1410: compute root-scope membership once. Derived-column inlining diff --git a/slayer/engine/cross_model_planner.py b/slayer/engine/cross_model_planner.py index b8277db9..15ff9d8e 100644 --- a/slayer/engine/cross_model_planner.py +++ b/slayer/engine/cross_model_planner.py @@ -63,11 +63,15 @@ TimeTruncKey, ValueKey, column_path, + reroot_aggregate_key, ) from slayer.core.models import ModelMeasure, SlayerModel from slayer.core.query import ColumnRef, SlayerQuery, TimeDimension from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name from slayer.core.scope import ModelScope, StageColumn, StageSchema +from slayer.engine.aggregate_input_paths import ( + compute_aggregate_input_join_paths, +) from slayer.engine.binding import ( bind_expr, bind_filter, @@ -528,6 +532,84 @@ def _classify_subplan_filters( return sub_filter_texts or None +def _route_host_filters( + *, + host_filters: List[HostFilterRouting], + host_slots: List[ValueSlot], + target_path: Tuple[str, ...], + host_model: SlayerModel, + terminal_model: SlayerModel, +) -> Tuple[ + List[BoundFilterId], List[BoundFilterId], List[BoundFilterId], + List[UnreachableFilterDroppedWarning], +]: + """Classify each host filter via the ``inherited_filter_policy`` decision + table (``classify_host_filter``) into ``(applied, where_ids, having_ids, + dropped)`` — extracted from ``IsolatedCteCrossModelPlanner.plan`` (DEV-1708) + to keep that method focused. ``DROP_HOST_LOCAL`` / ``STAY_AT_HOST_POST`` are + neither propagated nor warned.""" + applied: List[BoundFilterId] = [] + where_ids: List[BoundFilterId] = [] + having_ids: List[BoundFilterId] = [] + dropped: List[UnreachableFilterDroppedWarning] = [] + for hf in host_filters: + route = classify_host_filter( + host_filter=hf, + host_slots=host_slots, + target_path=target_path, + host_model_name=host_model.name, + ) + if route is FilterRoute.PROPAGATE_WHERE: + where_ids.append(hf.filter_id) + applied.append(hf.filter_id) + elif route is FilterRoute.PROPAGATE_HAVING: + having_ids.append(hf.filter_id) + applied.append(hf.filter_id) + elif route is FilterRoute.DROP_UNREACHABLE: + dropped.append(UnreachableFilterDroppedWarning( + filter_text=hf.text or hf.filter_id, + reason=( + f"filter {hf.filter_id!r} references slot(s) outside " + f"the join path to {terminal_model.name!r}; " + f"unreachable filters are dropped." + ), + )) + return applied, where_ids, having_ids, dropped + + +def _compute_shared_grain_slots( + *, host_slots: List[ValueSlot], target_path: Tuple[str, ...], +) -> List[SlotId]: + """Host ROW slots (dimensions / time-dimensions) whose path lies on the + target's join chain flow through as the cross-model CTE's shared grain + (extracted from ``IsolatedCteCrossModelPlanner.plan`` — DEV-1708). Cross- + branch and aggregate/transform slots do not. + + A path-bearing **plain derived** (``ColumnSqlKey``) dimension on the target + path flows through identically to a base ``ColumnKey`` dim (DEV-1728): the + generator expands its ``Column.sql`` inside the ``_cm_*`` CTE, groups by it, + and joins back on the DOTTED host alias (the DEV-1708 raise is gone now that + DEV-1713 fixed the naming half). ``path == ()`` (a host-local derived dim) + still broadcasts by design — the generator's grain loop skips empty-path + slots — and a hidden filter-only derived ref is excluded there via the + ``base_projection_ids`` intersection, so no ``not s.hidden`` guard is needed. + """ + shared_grain: List[SlotId] = [] + for s in host_slots: + # Base and derived dims carry their path directly; a time dimension + # carries it on the wrapped column. One prefix test then serves all + # three kinds — the DEV-1728 merge of what were two identical branches. + if isinstance(s.key, (ColumnKey, ColumnSqlKey)): + p = s.key.path + elif isinstance(s.key, TimeTruncKey): + p = column_path(s.key.column) + else: + continue + if not p or p == target_path[: len(p)]: + shared_grain.append(s.id) + return shared_grain + + class IsolatedCteCrossModelPlanner: """Default impl — one CTE per (target_model, shared_grain) tuple. @@ -595,54 +677,22 @@ def plan( )) target_path = path - applied: List[BoundFilterId] = [] - where_ids: List[BoundFilterId] = [] - having_ids: List[BoundFilterId] = [] - dropped: List[UnreachableFilterDroppedWarning] = [] - for hf in host_filters: - route = classify_host_filter( - host_filter=hf, - host_slots=host_slots, - target_path=target_path, - host_model_name=host_model.name, - ) - if route is FilterRoute.PROPAGATE_WHERE: - where_ids.append(hf.filter_id) - applied.append(hf.filter_id) - elif route is FilterRoute.PROPAGATE_HAVING: - having_ids.append(hf.filter_id) - applied.append(hf.filter_id) - elif route is FilterRoute.DROP_UNREACHABLE: - dropped.append(UnreachableFilterDroppedWarning( - filter_text=hf.text or hf.filter_id, - reason=( - f"filter {hf.filter_id!r} references slot(s) outside " - f"the join path to {terminal_model.name!r}; " - f"unreachable filters are dropped." - ), - )) - # DROP_HOST_LOCAL and STAY_AT_HOST_POST: not propagated, not warned. + applied, where_ids, having_ids, dropped = _route_host_filters( + host_filters=host_filters, + host_slots=host_slots, + target_path=target_path, + host_model=host_model, + terminal_model=terminal_model, + ) target_model_filters = list(terminal_model.filters or []) - # Shared grain: local ROW slots on host (dimensions) flow through. - # Cross-branch ROW slots and aggregate / transform slots do not. - # DEV-1450 stage 7b.12: ``TimeTruncKey`` slots count as grain - # candidates too — a joined TD (``customers.created_at`` MONTH) - # whose column path lies on the target's join chain is shared - # between the host base and the cross-model CTE, so legacy - # ``LEFT JOIN`` on the truncated alias replaces the global - # ``CROSS JOIN``. - shared_grain: List[SlotId] = [] - for s in host_slots: - if isinstance(s.key, ColumnKey): - p = s.key.path - if not p or p == target_path[: len(p)]: - shared_grain.append(s.id) - elif isinstance(s.key, TimeTruncKey): - td_path = column_path(s.key.column) - if not td_path or td_path == target_path[: len(td_path)]: - shared_grain.append(s.id) + # Shared grain: host ROW dimensions / time-dimensions on the target's + # join chain flow through (a plain derived one on the target path + # raises — DEV-1708). Extracted to keep this method focused. + shared_grain = _compute_shared_grain_slots( + host_slots=host_slots, target_path=target_path, + ) first_hop = join_chain[0] first_hop_target = ( @@ -710,19 +760,32 @@ def _dispatch_filtered_local( Callable[[SlayerQuery, ResolvedSourceBundle], PlannedQuery] ], ) -> CrossModelAggregatePlan: - """Validate the filtered-local trigger preconditions and dispatch + """Validate the host-rooted trigger preconditions and dispatch into ``_plan_filtered_local`` — the aggregate is on a HOST column - but its ``Column.filter`` crosses a join, so a host-rooted nested - sub-plan owns the aggregation and the host base LEFT JOINs back. + but at least one of its inputs crosses a join (``Column.filter`` + per DEV-1503; source ``Column.sql`` / positional args / kwargs per + DEV-1709), so a host-rooted nested sub-plan owns the aggregation + and the host base LEFT JOINs back. """ agg_source = aggregate_key.source cfk = aggregate_key.column_filter_key - if cfk is None or not cfk.referenced_join_paths: + has_crossing_filter = cfk is not None and bool( + cfk.referenced_join_paths, + ) + has_crossing_input = has_crossing_filter or bool( + compute_aggregate_input_join_paths( + key=aggregate_key, + anchor_model=host_model, + anchor_relation=host_model.name, + bundle=bundle, + ), + ) + if not has_crossing_input: raise ValueError( - f"AggregateKey on {agg_source!r} has empty source.path " - f"AND no cross-model column_filter_key — this is a plain " - f"local aggregate. The cross-model planner should not " - f"have been invoked." + f"AggregateKey on {agg_source!r} has empty source.path, " + f"no cross-model column_filter_key, AND no other crossing " + f"input — this is a plain local aggregate. The cross-model " + f"planner should not have been invoked." ) if subplan_builder is None or host_query is None: # The DEV-1503 strategy requires a sub-plan builder + the host @@ -917,53 +980,47 @@ def _filter_ref_paths(value_key: ValueKey) -> List[Tuple[str, ...]]: return paths +def _render_ref_formula(ref) -> str: + """Render one already-rerooted embedded reference back into formula text. + + Column-like refs dot-join their (residual) path with the leaf; scalars + fall through to ``_scalar_formula_literal``. Contains NO path-stripping + decisions — the reroot has already happened (DEV-1707). + """ + if isinstance(ref, ColumnSqlKey): + return ".".join((*ref.path, ref.column_name)) + if isinstance(ref, ColumnKey): + return ".".join((*ref.path, ref.leaf)) + return _scalar_formula_literal(ref) + + def _local_agg_formula(key: AggregateKey) -> str: """Reconstruct the LOCAL colon-formula for a cross-model aggregate (``customers.revenue:sum`` -> ``revenue:sum``) so it can be re-planned against the target model as a plain local measure. - Column-valued kwargs (``corr(other=customers.region_id)``) are - re-rooted too: their join path is bound from the HOST, so the leading - agg-source (target) prefix is stripped to express the ref in the - target's local scope (``other=region_id``; a deeper hop keeps its - residual path, ``other=regions.code``). Dropping the path outright - would mis-bind or fail to bind the nested sub-query (CR review).""" - src = key.source - target_path = tuple(getattr(src, "path", ())) + Every embedded reference — source, positional args, and column-valued + kwargs — is re-anchored symmetrically via the unified + ``reroot_aggregate_key`` (DEV-1707), then rendered by the path-free + ``_render_ref_formula``. A kwarg / arg one hop past the target keeps its + residual path (``other=regions.code``); an exact match becomes local + (``other=region_id``). The public string contract is unchanged — the + strip logic simply no longer lives here. + """ + local = reroot_aggregate_key( + key, target_path=tuple(getattr(key.source, "path", ())), + ) + src = local.source if isinstance(src, StarKey): base = "*" elif isinstance(src, ColumnSqlKey): - base = src.column_name + base = ".".join((*src.path, src.column_name)) else: # ColumnKey - base = src.leaf - - def _reroot_col_kwarg(v) -> str: - leaf = v.leaf if isinstance(v, ColumnKey) else v.column_name - vpath = tuple(getattr(v, "path", ())) - # Strip the agg-source (target) prefix so the ref is target-local. - residual = ( - vpath[len(target_path):] - if vpath[: len(target_path)] == target_path - else vpath - ) - return ".".join((*residual, leaf)) - - formula = f"{base}:{key.agg}" - parts: List[str] = [] - # Positional args may carry ColumnKey / ColumnSqlKey just like kwargs do - # (rerooting needs path-aware handling on both — CR review). Falling - # through to ``_scalar_formula_literal`` would emit Pydantic-repr noise - # for a column-valued positional arg, mis-binding the nested sub-query. - for a in key.args: - if isinstance(a, (ColumnKey, ColumnSqlKey)): - parts.append(_reroot_col_kwarg(a)) - else: - parts.append(_scalar_formula_literal(a)) - for k, v in key.kwargs: - if isinstance(v, (ColumnKey, ColumnSqlKey)): - parts.append(f"{k}={_reroot_col_kwarg(v)}") - else: - parts.append(f"{k}={_scalar_formula_literal(v)}") + base = ".".join((*src.path, src.leaf)) + + formula = f"{base}:{local.agg}" + parts: List[str] = [_render_ref_formula(a) for a in local.args] + parts += [f"{k}={_render_ref_formula(v)}" for k, v in local.kwargs] if parts: formula += "(" + ", ".join(parts) + ")" return formula diff --git a/slayer/engine/enriched.py b/slayer/engine/enriched.py index b28e5ce3..a6ef17c1 100644 --- a/slayer/engine/enriched.py +++ b/slayer/engine/enriched.py @@ -17,7 +17,7 @@ - The query engine controls resolution logic (placeholder expansion, join resolution) """ -from typing import Dict, List, Optional +from typing import Optional from pydantic import BaseModel, Field @@ -32,19 +32,19 @@ class EnrichedDimension(BaseModel): """A dimension with its SQL expression fully resolved.""" name: str - sql: Optional[str] + sql: str | None type: DataType alias: str = Field(description="Result column name, e.g. 'orders.status'") model_name: str - label: Optional[str] = Field(default=None, description="Human-readable label") - format: Optional[NumberFormat] = Field(default=None, description="Number format from the source dimension") + label: str | None = Field(default=None, description="Human-readable label") + format: NumberFormat | None = Field(default=None, description="Number format from the source dimension") class EnrichedMeasure(BaseModel): """A measure with its SQL expression and aggregation fully resolved.""" name: str - sql: Optional[str] = Field(description="SQL expression; None for *:count (COUNT(*))") + sql: str | None = Field(description="SQL expression; None for *:count (COUNT(*))") aggregation: str = Field(description="Aggregation name: sum, avg, count, weighted_avg, etc.") alias: str = Field(description="Result column name, e.g. 'orders.revenue_sum'") user_declared: bool = Field( @@ -57,19 +57,19 @@ class EnrichedMeasure(BaseModel): ), ) model_name: str - aggregation_def: Optional[Aggregation] = Field(default=None, description="Full aggregation definition (formula, params)") - agg_kwargs: Dict[str, str] = Field(default_factory=dict, description="Query-time aggregation param overrides") - window: Optional[str] = Field(default=None, description="Trailing time window for windowed sum/avg aggregations") - window_time_alias: Optional[str] = Field(default=None, description="Time dimension alias used for windowed aggregations") - label: Optional[str] = Field(default=None, description="Human-readable label") - time_column: Optional[str] = Field(default=None, description="Explicit time col for first/last (overrides query default)") - source_measure_name: Optional[str] = Field(default=None, description="Original measure name before canonicalization") - filter_sql: Optional[str] = Field(default=None, description="Resolved SQL condition for filtered measures (CASE WHEN)") - filter_columns: List[str] = Field( + aggregation_def: Aggregation | None = Field(default=None, description="Full aggregation definition (formula, params)") + agg_kwargs: dict[str, str] = Field(default_factory=dict, description="Query-time aggregation param overrides") + window: str | None = Field(default=None, description="Trailing time window for windowed sum/avg aggregations") + window_time_alias: str | None = Field(default=None, description="Time dimension alias used for windowed aggregations") + label: str | None = Field(default=None, description="Human-readable label") + time_column: str | None = Field(default=None, description="Explicit time col for first/last (overrides query default)") + source_measure_name: str | None = Field(default=None, description="Original measure name before canonicalization") + filter_sql: str | None = Field(default=None, description="Resolved SQL condition for filtered measures (CASE WHEN)") + filter_columns: list[str] = Field( default_factory=list, description="Resolved (qualified) column names referenced by the filter, for join planning", ) - type: Optional[DataType] = Field( + type: DataType | None = Field( default=None, description=( "DEV-1361: declared result type of the aggregation. When set, " @@ -77,7 +77,7 @@ class EnrichedMeasure(BaseModel): "Inherits from ModelMeasure.type at enrichment time." ), ) - column_type: Optional[DataType] = Field( + column_type: DataType | None = Field( default=None, description=( "DEV-1361: source column's declared type — wraps the inner " @@ -104,12 +104,12 @@ class EnrichedTimeDimension(BaseModel): """A time dimension with resolved SQL and granularity.""" name: str - sql: Optional[str] + sql: str | None granularity: TimeGranularity - date_range: Optional[List[str]] + date_range: list[str] | None alias: str model_name: str - label: Optional[str] = None + label: str | None = None class EnrichedExpression(BaseModel): @@ -130,8 +130,8 @@ class EnrichedExpression(BaseModel): "expressions (e.g. desugared change/change_pct arithmetic)." ), ) - label: Optional[str] = None - type: Optional[DataType] = Field( + label: str | None = None + type: DataType | None = Field( default=None, description=( "DEV-1361: declared result type — when set, the outer SELECT " @@ -163,10 +163,10 @@ class EnrichedTransform(BaseModel): measure_alias: str = Field(description="Alias of the measure in the base CTE to transform") alias: str = Field(description="Result column name") offset: int = Field(description="For time_shift: number of rows or calendar units") - granularity: Optional[str] = Field(default=None, description="For time_shift: year, month, quarter, etc.") - time_alias: Optional[str] = Field(default=None, description="Alias of the time dimension column for ORDER BY") - partition_aliases: List[str] = Field(default_factory=list, description="Dimension aliases to PARTITION BY") - n: Optional[int] = Field(default=None, description="Bucket count for ntile(measure, n=...)") + granularity: str | None = Field(default=None, description="For time_shift: year, month, quarter, etc.") + time_alias: str | None = Field(default=None, description="Alias of the time dimension column for ORDER BY") + partition_aliases: list[str] = Field(default_factory=list, description="Dimension aliases to PARTITION BY") + n: int | None = Field(default=None, description="Bucket count for ntile(measure, n=...)") predicate_is_boolean: bool = Field( default=False, description="True when the transform's measure_alias points at a boolean expression " @@ -174,8 +174,8 @@ class EnrichedTransform(BaseModel): "Postgres rejects 'boolean <> integer' so the numeric `IS NOT NULL AND <> 0` " "predicate cannot be used.", ) - label: Optional[str] = None - type: Optional[DataType] = Field( + label: str | None = None + type: DataType | None = Field( default=None, description=( "DEV-1361: declared result type — when set, the window-layer " @@ -192,30 +192,30 @@ class EnrichedQuery(BaseModel): """ model_name: str - sql_table: Optional[str] = None - sql: Optional[str] = None + sql_table: str | None = None + sql: str | None = None - resolved_joins: List[tuple] = Field(default_factory=list, description="[(target_table_sql, target_alias, join_condition, join_type), ...]") + resolved_joins: list[tuple] = Field(default_factory=list, description="[(target_table_sql, target_alias, join_condition, join_type), ...]") - dimensions: List[EnrichedDimension] = Field(default_factory=list) - measures: List[EnrichedMeasure] = Field(default_factory=list) - time_dimensions: List[EnrichedTimeDimension] = Field(default_factory=list) + dimensions: list[EnrichedDimension] = Field(default_factory=list) + measures: list[EnrichedMeasure] = Field(default_factory=list) + time_dimensions: list[EnrichedTimeDimension] = Field(default_factory=list) - expressions: List[EnrichedExpression] = Field(default_factory=list) - transforms: List[EnrichedTransform] = Field(default_factory=list) + expressions: list[EnrichedExpression] = Field(default_factory=list) + transforms: list[EnrichedTransform] = Field(default_factory=list) - cross_model_measures: List["CrossModelMeasure"] = Field(default_factory=list) + cross_model_measures: list["CrossModelMeasure"] = Field(default_factory=list) - last_agg_time_column: Optional[str] = Field(default=None, description="Time column for first/last aggregation (ORDER BY for ROW_NUMBER)") + last_agg_time_column: str | None = Field(default=None, description="Time column for first/last aggregation (ORDER BY for ROW_NUMBER)") - filters: List[ParsedFilter] = Field(default_factory=list) - order: Optional[List[OrderItem]] = None - limit: Optional[int] = None - offset: Optional[int] = None + filters: list[ParsedFilter] = Field(default_factory=list) + order: list[OrderItem] | None = None + limit: int | None = None + offset: int | None = None - field_name_aliases: Dict[str, str] = Field(default_factory=dict, description="Custom field name → enriched alias mapping (for ORDER BY resolution)") + field_name_aliases: dict[str, str] = Field(default_factory=dict, description="Custom field name → enriched alias mapping (for ORDER BY resolution)") - user_projection: List[str] = Field( + user_projection: list[str] = Field( default_factory=list, description=( "DEV-1444: ordered list of result-column aliases that the user " @@ -225,6 +225,15 @@ class EnrichedQuery(BaseModel): ), ) + distinct_dimension_values: bool = Field( + default=True, + description=( + "DEV-1543: pass-through of ``SlayerQuery.distinct_dimension_values``. " + "When ``False``, the SQL generator skips the dim-only-dedup " + "``GROUP BY`` clause and emits raw rows." + ), + ) + class CrossModelMeasure(BaseModel): """A measure from a joined model, computed as a separate sub-query. @@ -245,18 +254,18 @@ class CrossModelMeasure(BaseModel): ) alias: str = Field(description="Result column name, e.g. 'orders.customers__avg_score'") target_model_name: str = Field(description="The joined model name") - target_model_sql_table: Optional[str] - target_model_sql: Optional[str] + target_model_sql_table: str | None + target_model_sql: str | None measure: EnrichedMeasure = Field(description="The measure to aggregate") - join_pairs: List[List[str]] = Field(description="[[source_dim, target_dim], ...] from ModelJoin") - shared_dimensions: List[EnrichedDimension] = Field(description="Dimensions shared between main and target") - shared_time_dimensions: List[EnrichedTimeDimension] = Field(description="Time dims shared between main and target") + join_pairs: list[list[str]] = Field(description="[[source_dim, target_dim], ...] from ModelJoin") + shared_dimensions: list[EnrichedDimension] = Field(description="Dimensions shared between main and target") + shared_time_dimensions: list[EnrichedTimeDimension] = Field(description="Time dims shared between main and target") source_model_name: str = Field(description="The main query's model name") - source_sql_table: Optional[str] = Field(description="Main model's table") - source_sql: Optional[str] = Field(description="Main model's SQL") + source_sql_table: str | None = Field(description="Main model's table") + source_sql: str | None = Field(description="Main model's SQL") join_type: str = Field(default="left", description="'left' or 'inner'") - label: Optional[str] = None - format: Optional[NumberFormat] = Field(default=None, description="Inferred format for this cross-model measure") + label: str | None = None + format: NumberFormat | None = Field(default=None, description="Inferred format for this cross-model measure") rerooted_enriched: Optional["EnrichedQuery"] = Field(default=None, description="Re-rooted subquery with target as source") @@ -265,7 +274,7 @@ class CrossModelMeasure(BaseModel): CrossModelMeasure.model_rebuild() -def public_projection_aliases(enriched: EnrichedQuery) -> List[str]: +def public_projection_aliases(enriched: EnrichedQuery) -> list[str]: """Return the ordered list of public-projection aliases for ``enriched``. DEV-1444: the outer rendered SELECT projects exactly the @@ -292,7 +301,7 @@ def public_projection_aliases(enriched: EnrichedQuery) -> List[str]: # desugaring). Matches the pre-DEV-1444 ``expected_columns`` rule # in ``query_engine.py``. internal_prefixes = ("_inner_", "_ft", "_ts") - out: List[str] = [d.alias for d in enriched.dimensions] + out: list[str] = [d.alias for d in enriched.dimensions] out.extend(td.alias for td in enriched.time_dimensions) out.extend( m.alias for m in enriched.measures diff --git a/slayer/engine/enrichment.py b/slayer/engine/enrichment.py index fdaa9132..b02498e1 100644 --- a/slayer/engine/enrichment.py +++ b/slayer/engine/enrichment.py @@ -9,7 +9,8 @@ """ import re -from typing import Any, Dict, List, Mapping, Optional, Set, Tuple +from typing import Any +from collections.abc import Mapping import sqlglot from sqlglot import exp @@ -19,6 +20,7 @@ DEFAULT_AGGREGATIONS_BY_TYPE, DataType, PRIMARY_KEY_AGGREGATIONS, + format_unknown_aggregation, ) from slayer.core.formula import ( canonical_agg_name, @@ -36,7 +38,7 @@ parse_formula, ) from slayer.core.models import Column, SlayerModel -from slayer.core.query import SlayerQuery +from slayer.core.query import OrderItem, SlayerQuery from slayer.core.refs import DOTTED_IDENT_REF_RE as _DOTTED_IDENT_REF_RE from slayer.engine.column_expansion import _is_trivial_base, expand_derived_refs from slayer.engine.enriched import ( @@ -48,11 +50,21 @@ EnrichedTimeDimension, EnrichedTransform, ) +from slayer.sql.naming import flat_name +from slayer.sql.reserved_keywords import prequote_reserved_identifiers from slayer.sql.sql_predicate import parse_sql_predicate from slayer.sql.window_detect import WINDOW_IN_FILTER_ERROR, has_window_function _SELF_JOIN_TRANSFORMS = {"time_shift"} -_TABLE_COL_RE = re.compile(r"\b([a-zA-Z_]\w*)\.([a-zA-Z_]\w*)\b") +# DEV-1686: quote-tolerant so join-path discovery matches a reserved qualifier +# that RESERVED_KEYWORDS emits quoted in expanded derived-column SQL. Tolerates +# every dialect's identifier quote char — ANSI ``"grant"``, MySQL/BigQuery +# `` `grant` ``, T-SQL ``[grant]`` — as well as bare refs and ``__``-path +# aliases (strict superset of the old bare ``word.word`` form). group(1) is +# still the unquoted qualifier name. +_TABLE_COL_RE = re.compile( + r'(? str: """Strip one layer of single/double quotes from a query parameter value.""" if len(value) >= 2 and value[0] == value[-1] and value[0] in ("'", '"'): @@ -63,7 +75,7 @@ def _strip_string_literal(value: str) -> str: _canonical_agg_name = canonical_agg_name # Module-internal alias for the shared helper -def canonical_expression_key(node: Any) -> Tuple[Any, ...]: # NOSONAR(S8495) — variable-length tuple shape IS the discriminator; type signature already declares Tuple[Any, ...] +def canonical_expression_key(node: Any) -> tuple[Any, ...]: # NOSONAR(S8495) — variable-length tuple shape IS the discriminator; type signature already declares Tuple[Any, ...] """DEV-1444: build an alias-independent, structural hash key for a parsed formula AST node. @@ -110,8 +122,8 @@ def canonical_expression_key(node: Any) -> Tuple[Any, ...]: # NOSONAR(S8495) async def _collect_reachable_agg_names( model: SlayerModel, resolve_join_target, - named_queries: Dict, -) -> Optional[frozenset[str]]: + named_queries: dict, +) -> frozenset[str] | None: """Collect custom aggregation names from the source model and all reachable joined models. Walks the full reachable join graph via BFS, bounded only by the ``visited`` @@ -146,10 +158,22 @@ async def _collect_reachable_agg_names( return frozenset(names) if names else None +def _public_field_name(qfield: Any) -> str: + """The public name a query measure surfaces under. + + An explicit ``name`` wins; otherwise the formula is mangled into an + identifier. Shared by the main measure loop and the reserved-name set + that hidden-transform allocation checks against, so the two can't drift. + """ + return qfield.name or qfield.formula.replace(" ", "_").replace("/", "_div_").replace(":", "_").replace( + "*", "" + ) + + async def enrich_query( query: SlayerQuery, model: SlayerModel, - named_queries: Optional[Dict[str, SlayerQuery]] = None, + named_queries: dict[str, SlayerQuery] | None = None, *, resolve_dimension_via_joins, resolve_cross_model_measure, @@ -193,7 +217,7 @@ async def enrich_query( # Saved-formula library for bare-name resolution. Only the source model's # named measures are in scope here; cross-model references (`other.aov`) # remain handled by the cross-model resolver. - named_measures: Dict[str, str] = {} + named_measures: dict[str, str] = {} for m in model.measures: if not m.name: continue @@ -204,6 +228,20 @@ async def enrich_query( ) named_measures[m.name] = m.formula + # A bare named-measure reference like ``measures=["companies_count"]`` is + # an implicit rename: the user already chose ``companies_count`` as the + # surfacing name when they declared the ModelMeasure. Promote it to an + # explicit ``qf.name`` so DEV-1443's surface / collision / canonical + # machinery (which keys on ``qf.name``) treats it identically to an + # explicit ``{"formula": "...", "name": "..."}`` rename. Without this, + # the SELECT alias canonicalises off the expanded formula + # (``company_name_count_distinct``) while ORDER BY / result-key lookups + # still use the declared measure name (``companies_count``) — they + # diverge and the emitted SQL references a column that isn't projected. + for qf in (query.measures or []): + if qf.name is None and qf.formula.strip() in named_measures: + qf.name = qf.formula.strip() + # --- Dimensions --- dimensions = await _resolve_dimensions( query=query, @@ -216,7 +254,7 @@ async def enrich_query( ) # --- Measures (populated from fields below) --- - measures: List[EnrichedMeasure] = [] + measures: list[EnrichedMeasure] = [] # --- Time dimensions --- time_dimensions = await _resolve_time_dimensions( @@ -258,22 +296,22 @@ async def enrich_query( ) # --- Process fields --- - enriched_expressions: List[EnrichedExpression] = [] - enriched_transforms: List[EnrichedTransform] = [] - cross_model_measures: List[CrossModelMeasure] = [] - known_aliases: Dict[str, str] = {} - field_name_aliases: Dict[str, str] = {} + enriched_expressions: list[EnrichedExpression] = [] + enriched_transforms: list[EnrichedTransform] = [] + cross_model_measures: list[CrossModelMeasure] = [] + known_aliases: dict[str, str] = {} + field_name_aliases: dict[str, str] = {} # DEV-1443: canonical-agg alias → user-supplied measure name. Populated # when a query measure renames the canonical (``{"formula": "col:agg", # "name": "alias"}``). Consumed by the filter pre-pass (so a filter # written as ``col:agg N`` resolves to the user alias and HAVINGs # correctly) and by the ORDER BY enrichment (same shape). - canonical_to_user_name: Dict[str, str] = {} + canonical_to_user_name: dict[str, str] = {} # Cached source-column name set for the remap eligibility guard # (Codex Finding 1 — skip remap when the canonical alias also literally # names a source column on the model, since the regex sub would then # clobber the literal source-column reference). - _source_column_names: Set[str] = {c.name for c in model.columns} + _source_column_names: set[str] = {c.name for c in model.columns} # DEV-1444 provenance-merge index: canonical_expression_key → # surfaced alias. Populated when an EnrichedMeasure is created; @@ -282,7 +320,7 @@ async def enrich_query( # (e.g. order-by ``revenue:sum`` matching a user-declared # ``{"formula":"revenue:sum","name":"total"}``) reuses the existing # alias instead of materialising a phantom ``orders.revenue_sum``. - measure_canonical_key_to_alias: Dict[Tuple[Any, ...], str] = {} + measure_canonical_key_to_alias: dict[tuple[Any, ...], str] = {} def _mark_user_declared(alias: str) -> bool: """DEV-1444: flip ``user_declared=True`` on the enriched entry that @@ -312,8 +350,8 @@ async def _ensure_aggregated_measure( alias_key: str, measure_name: str, aggregation_name: str, - agg_args: Optional[list] = None, - agg_kwargs: Optional[dict] = None, + agg_args: list | None = None, + agg_kwargs: dict | None = None, ): """Create an EnrichedMeasure for an aggregated measure ref. @@ -395,6 +433,21 @@ async def _ensure_aggregated_measure( raise ValueError( f"Column '{measure_name}' not found in model '{model.name}'" ) + # DEV-1576 §3: distinguish "unknown aggregation name" from "known + # but not allowed for this column type". The name check runs BEFORE + # the PK / whitelist / type gates so an unknown name never gets + # mislabelled as a type restriction (a perfectly aggregatable DOUBLE + # column with a misspelled agg should say "Unknown aggregation", + # not "not applicable to DOUBLE column"). + known_aggregations = BUILTIN_AGGREGATIONS | { + a.name for a in model.aggregations + } + if aggregation_name not in known_aggregations: + # DEV-1576 / DEV-1717: shared formatter, so this local gate and + # the typed binding gate (binding.py) stay byte-identical. + raise ValueError( + format_unknown_aggregation(aggregation_name, known_aggregations) + ) # Apply aggregation eligibility gates per the v2 contract: # 1. Primary-key columns are always restricted to count/count_distinct # (regardless of type or any explicit whitelist). @@ -463,7 +516,7 @@ async def _ensure_aggregated_measure( # syntax, transform calls, ``OVER``) were rejected at construction # by ``parse_sql_predicate``. filter_sql = None - filter_columns: List[str] = [] + filter_columns: list[str] = [] if measure_def and measure_def.filter: parsed = parse_sql_predicate(measure_def.filter) resolved = await resolve_filter_columns( @@ -520,7 +573,7 @@ def _resolve_sql(sql: str) -> str: resolved = re.sub(rf'(? List[str]: + def _resolve_rank_partition(transform: str, partition_by: list[str]) -> list[str]: """Resolve partition_by= column references to base-CTE aliases. partition_by entries must reference query dimensions or time dimensions — @@ -536,7 +589,7 @@ def _resolve_rank_partition(transform: str, partition_by: List[str]) -> List[str by_name.setdefault(td.name, td.alias) by_alias.setdefault(td.alias, td.alias) - resolved: List[str] = [] + resolved: list[str] = [] for col in partition_by: if col in by_alias: resolved.append(by_alias[col]) @@ -551,6 +604,36 @@ def _resolve_rank_partition(transform: str, partition_by: List[str]) -> List[str ) return resolved + # DEV-1692: names for transforms hoisted out of an arithmetic formula are + # derived from the owning measure's field_name, which keeps them distinct + # from one another — but nothing stops a user from *also* selecting a + # measure or dimension literally named ``_t0_growth``. Both would then + # claim the alias ``._t0_growth``: the self-join CTE projects two + # columns under that name and the hoisted reference silently resolves to + # the user's column (no error, wrong numbers). Allocate against every + # projected name instead of trusting field_name uniqueness on its own. + # Dimensions and time dimensions alias as ``.`` just like + # hoisted transforms do, so they are reserved by bare name too. + _reserved_public_names: set[str] = ( + {_public_field_name(qf) for qf in (query.measures or [])} + | {d.name for d in dimensions} + | {td.name for td in time_dimensions} + ) + _hidden_names: set[str] = set() + + def _allocate_hidden_name(preferred: str) -> str: + candidate = preferred + suffix = 2 + while ( + candidate in _reserved_public_names + or candidate in _hidden_names + or candidate in known_aliases + ): + candidate = f"{preferred}_{suffix}" + suffix += 1 + _hidden_names.add(candidate) + return candidate + def _add_transform( name: str, transform: str, @@ -558,7 +641,7 @@ def _add_transform( offset: int = 1, granularity: str = None, predicate_is_boolean: bool = False, - kwargs: Optional[Dict[str, Any]] = None, + kwargs: dict[str, Any] | None = None, ): needs_time = transform in TIME_TRANSFORMS if needs_time and resolved_time_alias is None: @@ -609,7 +692,7 @@ def _add_transform( # falls through to the cross-model CTE path for them. _DISTRIBUTIVE_AGGS = frozenset({"sum", "min", "max"}) - def _intercept_candidate_for_cross_model(ref) -> "Optional[Tuple[str, str]]": + def _intercept_candidate_for_cross_model(ref) -> "tuple[str, str] | None": """DEV-1449: return ``(flat_with_agg, outer_agg)`` if the intercept would resolve a virtual-stage cross-model agg ref to a local re-aggregation on a flat column, or ``None`` if the @@ -672,7 +755,7 @@ def _intercept_candidate_for_cross_model(ref) -> "Optional[Tuple[str, str]]": async def _try_intercept_cross_model_as_local( ref, field_name: str, - ) -> Optional[str]: + ) -> str | None: """Apply the intercept (computes candidate + builds the EnrichedMeasure). Returns the full enriched alias the caller can use, or ``None`` if no candidate.""" @@ -714,7 +797,7 @@ async def _try_intercept_cross_model_as_local( field_name_aliases[ref_canonical] = local_alias return local_alias - async def _ensure_measure_from_spec(mname: str, agg_refs: Optional[dict] = None): + async def _ensure_measure_from_spec(mname: str, agg_refs: dict | None = None): """Ensure a measure is resolved — handles agg refs only.""" agg_refs = agg_refs or {} if mname in agg_refs: @@ -837,13 +920,42 @@ async def _flatten_spec(spec, field_name: str) -> str: elif isinstance(spec, MixedArithmeticField): for mname in spec.measure_names: await _ensure_measure_from_spec(mname, spec.agg_refs) + # DEV-1692: the ``_t{n}`` placeholder counter restarts on every + # formula parse, so two measures that each wrap a transform in + # arithmetic would both flatten under the name ``_t0`` — colliding + # on the self-join CTE names (``shifted__t0``) and, worse, on the + # expression alias, silently making the second measure read the + # first one's value. Qualify with the owning measure's field_name + # and run it through _allocate_hidden_name so the result can't + # collide with a user measure that happens to share the shape. + placeholder_aliases: list[tuple[str, str]] = [] for placeholder, sub_transform in spec.sub_transforms: - await _flatten_spec(sub_transform, placeholder) + hidden_name = _allocate_hidden_name(f"{placeholder}_{field_name}") + sub_alias = await _flatten_spec( + spec=sub_transform, field_name=hidden_name + ) + placeholder_aliases.append((placeholder, sub_alias)) + # Bind the placeholders just long enough for _resolve_sql to rewrite + # this formula's references, then restore. A user measure may itself + # be named `_t0`, so a shadowed binding is put back rather than + # dropped. + shadowed: list[tuple[str, str | None]] = [ + (placeholder, known_aliases.get(placeholder)) + for placeholder, _ in placeholder_aliases + ] + for placeholder, sub_alias in placeholder_aliases: + known_aliases[placeholder] = sub_alias + resolved_sql = _resolve_sql(spec.sql) + for placeholder, prior in shadowed: + if prior is None: + del known_aliases[placeholder] + else: + known_aliases[placeholder] = prior alias = f"{model_name_str}.{field_name}" enriched_expressions.append( EnrichedExpression( name=field_name, - sql=_resolve_sql(spec.sql), + sql=resolved_sql, alias=alias, ) ) @@ -951,7 +1063,7 @@ async def _flatten_spec(spec, field_name: str) -> str: # DEV-1444: track aliases in declared order so EnrichedQuery.user_projection # can be populated at the end. Dims and time-dims come first. - user_projection: List[str] = [d.alias for d in dimensions] + user_projection: list[str] = [d.alias for d in dimensions] user_projection.extend(td.alias for td in time_dimensions) # DEV-1444 (Codex review on PR #134): the provenance-merge index in @@ -964,7 +1076,7 @@ async def _flatten_spec(spec, field_name: str) -> str: # outer trim would then project a column the inner SELECT doesn't # expose. Track the set of canonical keys already owned by a # user-declared qfield and refuse the duplicate. - user_declared_canon_keys: Dict[Tuple[Any, ...], str] = {} + user_declared_canon_keys: dict[tuple[Any, ...], str] = {} # DEV-1443 (CodeRabbit thread + Codex round 4 on PR #133): the # duplicate-explicit-name check must run for every query measure @@ -973,7 +1085,7 @@ async def _flatten_spec(spec, field_name: str) -> str: # arithmetic/transform measures fall through to ``_flatten_spec``; in # both cases a duplicate ``name`` would silently collapse two # measures onto a single alias. Run the pairwise check once up front. - _seen_explicit_names: Dict[str, str] = {} + _seen_explicit_names: dict[str, str] = {} for qf in (query.measures or []): if not qf.name: continue @@ -1036,8 +1148,9 @@ def _surfaces_for(qf, sp): # _query_as_model derives the downstream short from # _alias_to_short(cm.alias) for unrenamed cross-model: # the source-model prefix is stripped, then dots are - # converted to ``__``. Mirror that here. - short = f"{hop}.{cm_leaf}".replace(".", "__") + # converted to ``__``. Mirror that here via the naming + # module's single flatten owner (DEV-1713). + short = flat_name(f"{hop}.{cm_leaf}") else: if renamed: public = f"{model_name_str}.{qf.name}" @@ -1065,11 +1178,12 @@ def _surfaces_for(qf, sp): # prefix (``model_name_str.`` portion) and converts remaining dots # to ``__``. def _alias_to_short_local(alias: str) -> str: - stripped = alias.split(".", 1)[-1] if "." in alias else alias - return stripped.replace(".", "__") + # DEV-1713: delegates to the naming module's single flatten owner. + strip = alias.split(".", 1)[0] if "." in alias else None + return flat_name(alias, strip_relation=strip) - _occupied_aliases: Dict[str, str] = {} - _occupied_shorts: Dict[str, str] = {} + _occupied_aliases: dict[str, str] = {} + _occupied_shorts: dict[str, str] = {} for _d in dimensions: _occupied_aliases[_d.alias] = f"dimension '{_d.name}'" _occupied_shorts[_alias_to_short_local(_d.alias)] = ( @@ -1223,9 +1337,7 @@ def _mangled_formula(formula: str) -> str: f"ORDER BY would otherwise bind to the source column " f"instead of the renamed aggregate." ) - field_name = qfield.name or qfield.formula.replace(" ", "_").replace("/", "_div_").replace(":", "_").replace( - "*", "" - ) + field_name = _public_field_name(qfield) if isinstance(spec, AggregatedMeasureRef): # New colon syntax: "revenue:sum", "*:count", etc. @@ -1235,7 +1347,10 @@ def _mangled_formula(formula: str) -> str: agg_args=spec.agg_args, agg_kwargs=spec.agg_kwargs, ) - if field_name == qfield.formula.replace(" ", "_").replace("/", "_div_").replace(":", "_").replace("*", ""): + if ( + field_name == qfield.formula.replace(" ", "_").replace("/", "_div_").replace(":", "_").replace("*", "") + and qfield.formula.strip() not in named_measures + ): field_name = canonical_name if "." in spec.measure_name and spec.measure_name != "*": @@ -1368,7 +1483,8 @@ def _mangled_formula(formula: str) -> str: if qfield.name and qfield.name != canonical_name: em.name = qfield.name else: - em.name = canonical_name.replace(".", "__") + # DEV-1713: single flatten owner. + em.name = flat_name(canonical_name) em.alias = target_alias break known_aliases[target_name] = target_alias @@ -1652,7 +1768,7 @@ def _mangled_formula(formula: str) -> str: # operators) flow through unchanged. The construction-time validator # at ``slayer/core/models.py:412`` already rejected DSL constructs. measure_names_set = {m.name for m in measures} - parsed_model_filters: List[ParsedFilter] = [] + parsed_model_filters: list[ParsedFilter] = [] for mf in model.filters: parsed_mf = parse_sql_predicate(mf) for col in parsed_mf.columns: @@ -1676,11 +1792,26 @@ def _mangled_formula(formula: str) -> str: substitute_variables(filter_str=f, variables=query.variables) for f in query_filters ] + # DEV-1543: distinct_dimension_values=False rejects any measure + # reference in filters / order. This pass runs AFTER variable + # substitution (so a ``{var}`` revealing an aggregation is caught) + # and BEFORE ``extract_filter_transforms`` lifts transforms into + # hidden fields (so the original measure-reference shape is still + # visible). Pre-empts the construction-time check which is structural + # only. + if not query.distinct_dimension_values: + _reject_measure_references_for_raw_rows( + query=query, + query_filters=query_filters, + custom_agg_names=custom_agg_names, + named_measures=named_measures, + ) + # Transform extraction runs only on Mode B (DSL) query filters. Model # filters are SQL mode — they don't carry SLayer transforms (rejected # at construction by ``parse_sql_predicate``) and don't go through # ``_preprocess_like`` / ``_preprocess_agg_refs``. - processed_query_filters: List[str] = [] + processed_query_filters: list[str] = [] ft_counter = [0] for f_str in query_filters: rewritten, extra_fields = extract_filter_transforms( @@ -1700,7 +1831,7 @@ def _mangled_formula(formula: str) -> str: # query filters. Used by the windowed-column scan, ``_resolve_joins`` / # ``_collect_needed_paths``, and the ordering of the final # ``EnrichedQuery.filters`` list. - processed_filters_with_mode: List[Tuple[str, str]] = ( + processed_filters_with_mode: list[tuple[str, str]] = ( [(mf, "sql") for mf in model.filters] + [(qf, "dsl") for qf in processed_query_filters] ) @@ -1713,7 +1844,7 @@ def _mangled_formula(formula: str) -> str: # (`rank` / `percent_rank` / `dense_rank` / `ntile`) cover top-N # filtering in pure DSL. Applied to both modes — neither standard SQL # nor SLayer DSL allows window functions in WHERE. - _windowed_columns: Dict[str, str] = { + _windowed_columns: dict[str, str] = { c.name: c.sql for c in model.columns if c.sql and has_window_function(c.sql) } if _windowed_columns: @@ -1747,7 +1878,7 @@ def _mangled_formula(formula: str) -> str: # Names that resolve at the query level (named measures, transforms, # expressions) — pass through as legitimate filter targets even though # they are not Columns / ModelMeasures on the source model. - _query_aliases: Set[str] = set() + _query_aliases: set[str] = set() _query_aliases.update(m.name for m in measures if m.name) _query_aliases.update(t.name for t in enriched_transforms if t.name) _query_aliases.update(e.name for e in enriched_expressions if e.name) @@ -1841,6 +1972,7 @@ def _mangled_formula(formula: str) -> str: offset=query.offset, field_name_aliases=field_name_aliases, user_projection=user_projection, + distinct_dimension_values=query.distinct_dimension_values, ) @@ -1863,15 +1995,15 @@ def _unpack_dim_resolution(result): async def _maybe_expand( *, - sql: Optional[str], - terminal_model: Optional[SlayerModel], + sql: str | None, + terminal_model: SlayerModel | None, fallback_model: SlayerModel, alias_path: str, resolve_model, named_queries: dict, dialect: str, is_root: bool = True, -) -> Optional[str]: +) -> str | None: """Run the column-SQL expander when we have what we need; otherwise return ``sql`` unchanged. Lets tests that don't supply ``resolve_model`` keep getting the legacy unexpanded behavior — production always supplies @@ -1895,8 +2027,8 @@ async def _maybe_expand( def resolve_via_stage_origin( - *, model: SlayerModel, parts: List[str], -) -> Optional[Column]: + *, model: SlayerModel, parts: list[str], +) -> Column | None: """DEV-1449: Resolve a dotted reference against a virtual stage model produced by ``_query_as_model``. @@ -1954,7 +2086,7 @@ async def _resolve_dotted_dim_with_stage_fallback( model_name_str: str, named_queries: dict, resolve_dimension_via_joins, -) -> "tuple[Optional[Column], Optional[SlayerModel], str]": +) -> "tuple[Column | None, SlayerModel | None, str]": """Resolve a dotted dim / time-dim reference for one query field. Shared by ``_resolve_dimensions`` and ``_resolve_time_dimensions`` @@ -1990,10 +2122,10 @@ async def _resolve_dimensions( resolve_dimension_via_joins, resolve_model=None, dialect: str = "postgres", -) -> List[EnrichedDimension]: +) -> list[EnrichedDimension]: dimensions = [] for dim_ref in query.dimensions or []: - terminal_model: Optional[SlayerModel] = None + terminal_model: SlayerModel | None = None is_local = dim_ref.model is None if is_local: dim_def = model.get_column(dim_ref.name) @@ -2010,6 +2142,19 @@ async def _resolve_dimensions( resolve_dimension_via_joins=resolve_dimension_via_joins, ) ) + # Grouping by an opaque column emits SQL the database rejects (no + # equality operator), so fail here with an actionable message instead + # of surfacing a raw driver error. Projecting such a column is fine — + # only its use as a GROUP BY / DISTINCT key is refused. + if dim_def is not None and dim_def.type.is_opaque: + db_type = getattr(dim_def, "db_type", None) + described = f" (database type {db_type!r})" if db_type else "" + raise ValueError( + f"Column '{dim_ref.full_name}'{described} cannot be used as a " + f"dimension: its type does not support the grouping this query " + f"requires. Define a derived column that extracts a comparable " + f"value instead, e.g. sql=\"payload->>'status'\" with type TEXT." + ) expanded_sql = await _maybe_expand( sql=dim_def.sql if dim_def else None, terminal_model=terminal_model, @@ -2042,10 +2187,10 @@ async def _resolve_time_dimensions( resolve_dimension_via_joins, resolve_model=None, dialect: str = "postgres", -) -> List[EnrichedTimeDimension]: +) -> list[EnrichedTimeDimension]: time_dimensions = [] for td in query.time_dimensions or []: - terminal_model: Optional[SlayerModel] = None + terminal_model: SlayerModel | None = None is_local = td.dimension.model is None if is_local: dim_def = model.get_column(td.dimension.name) @@ -2087,10 +2232,10 @@ async def _resolve_time_dimensions( def _resolve_time_alias( - time_dimensions: List[EnrichedTimeDimension], + time_dimensions: list[EnrichedTimeDimension], query: SlayerQuery, model: SlayerModel, -) -> Optional[str]: +) -> str | None: if len(time_dimensions) == 1: return time_dimensions[0].alias elif len(time_dimensions) > 1: @@ -2108,16 +2253,16 @@ def _resolve_time_alias( def _resolve_last_agg_time( query: SlayerQuery, model: SlayerModel, - dimensions: List[EnrichedDimension], - time_dimensions: List[EnrichedTimeDimension], -) -> Optional[str]: + dimensions: list[EnrichedDimension], + time_dimensions: list[EnrichedTimeDimension], +) -> str | None: if query.main_time_dimension: mtd = query.main_time_dimension if "." not in mtd: mtd = f"{model.name}.{mtd}" return mtd - def _qualified(model_name: str, sql: Optional[str], name: str) -> str: + def _qualified(model_name: str, sql: str | None, name: str) -> str: # Once derived-ref expansion has run, `sql` may already be qualified # (e.g. ``orders.created_at`` instead of bare ``created_at``); don't # double-prefix in that case. @@ -2148,14 +2293,14 @@ def _qualified(model_name: str, sql: Optional[str], name: str) -> str: # --------------------------------------------------------------------------- -def _add_with_prefixes(segments: List[str], paths: Set[Tuple[str, ...]]) -> None: +def _add_with_prefixes(segments: list[str], paths: set[tuple[str, ...]]) -> None: """Add ``segments[:1], segments[:2], …, segments`` to ``paths``.""" for i in range(1, len(segments) + 1): paths.add(tuple(segments[:i])) def _raise_column_cycle( - visited: Tuple[Tuple[str, str], ...], key: Tuple[str, str], + visited: tuple[tuple[str, str], ...], key: tuple[str, str], ) -> None: """Raise a deterministic ``Circular column reference`` error matching the chain format used by ``expand_derived_refs``. @@ -2166,7 +2311,7 @@ def _raise_column_cycle( raise ValueError(f"Circular column reference detected: {chain}") -def _scan_sql_table_refs(*, sql: str, model_name: str, paths: Set[Tuple[str, ...]]) -> None: +def _scan_sql_table_refs(*, sql: str, model_name: str, paths: set[tuple[str, ...]]) -> None: """Regex-fallback scan: pick out ``
.`` shapes and add the table prefix paths (skipping references to ``model_name`` itself). """ @@ -2180,9 +2325,9 @@ def _process_node_for_paths( *, node: exp.Column, model: SlayerModel, - paths: Set[Tuple[str, ...]], - visited: Tuple[Tuple[str, str], ...], - dialect: Optional[str] = None, + paths: set[tuple[str, ...]], + visited: tuple[tuple[str, str], ...], + dialect: str | None = None, ) -> None: """Resolve one ``exp.Column`` node into either a recursion into a local derived column or a join-path-prefix add. @@ -2218,9 +2363,9 @@ def _collect_paths_from_local_column_chain( *, model: SlayerModel, col_name: str, - paths: Set[Tuple[str, ...]], - visited: Tuple[Tuple[str, str], ...] = (), - dialect: Optional[str] = None, + paths: set[tuple[str, ...]], + visited: tuple[tuple[str, str], ...] = (), + dialect: str | None = None, ) -> None: """Walk the SQL of a *local* derived column on ``model`` to discover the join paths its expression implies — recursing through references @@ -2250,7 +2395,13 @@ def _collect_paths_from_local_column_chain( next_visited = (*visited, key) try: - parsed = sqlglot.parse_one(sql, dialect=dialect) + # DEV-1686: quote bare reserved-word qualifiers/leaves (a derived + # column referencing a reserved joined model, e.g. ``grant.amount``) + # so join-path discovery finds the ref instead of silently falling + # back to a ref-less ``Command`` parse (which would drop the JOIN). + parsed = sqlglot.parse_one( + prequote_reserved_identifiers(sql=sql, dialect=dialect), dialect=dialect + ) except Exception: _scan_sql_table_refs(sql=sql, model_name=model.name, paths=paths) return @@ -2264,14 +2415,14 @@ def _collect_paths_from_local_column_chain( def _collect_needed_paths( model: SlayerModel, - dimensions: List[EnrichedDimension], - time_dimensions: List[EnrichedTimeDimension], - measures: List[EnrichedMeasure], + dimensions: list[EnrichedDimension], + time_dimensions: list[EnrichedTimeDimension], + measures: list[EnrichedMeasure], cross_model_measures: list, - processed_filters: List[Tuple[str, str]], - extra_agg_names: Optional[frozenset] = None, - dialect: Optional[str] = None, -) -> Set[Tuple[str, ...]]: + processed_filters: list[tuple[str, str]], + extra_agg_names: frozenset | None = None, + dialect: str | None = None, +) -> set[tuple[str, ...]]: """Extract ordered join-path tuples the query needs (including all prefixes). ``processed_filters`` is a list of ``(filter_text, mode)`` tuples @@ -2280,7 +2431,7 @@ def _collect_needed_paths( matching parser so model filters with arbitrary SQL functions don't trip the DSL allowlist (DEV-1378). """ - paths: Set[Tuple[str, ...]] = set() + paths: set[tuple[str, ...]] = set() for d in dimensions: if d.model_name != model.name: @@ -2326,8 +2477,8 @@ def _scan_filter_column_ref( *, model: SlayerModel, col: str, - paths: Set[Tuple[str, ...]], - dialect: Optional[str] = None, + paths: set[tuple[str, ...]], + dialect: str | None = None, ) -> None: """Route one entry from a parsed filter's column list to the right path-discovery branch. @@ -2350,7 +2501,7 @@ def _scan_filter_column_ref( return if _looks_like_dotted_identifier_ref(col): parts = col.split(".") - expanded: List[str] = [] + expanded: list[str] = [] for part in parts[:-1]: # Model filters convert dots to __; expand both forms. expanded.extend(part.split("__")) @@ -2376,16 +2527,16 @@ def _looks_like_dotted_identifier_ref(value: str) -> bool: async def _resolve_joins( model: SlayerModel, model_name_str: str, - dimensions: List[EnrichedDimension], - time_dimensions: List[EnrichedTimeDimension], - measures: List[EnrichedMeasure], + dimensions: list[EnrichedDimension], + time_dimensions: list[EnrichedTimeDimension], + measures: list[EnrichedMeasure], cross_model_measures: list, - processed_filters: List[Tuple[str, str]], + processed_filters: list[tuple[str, str]], named_queries: dict, resolve_join_target, - extra_agg_names: Optional[frozenset] = None, - dialect: Optional[str] = None, -) -> List[tuple]: + extra_agg_names: frozenset | None = None, + dialect: str | None = None, +) -> list[tuple]: """Resolve only the JOINs the query actually needs by walking the join graph. Instead of relying on baked-in multi-hop joins, this walks each intermediate @@ -2410,8 +2561,8 @@ async def _resolve_joins( # Sort shorter paths first so prefixes are resolved before extensions sorted_paths = sorted(needed_paths, key=len) - resolved_joins: Dict[str, tuple] = {} # alias -> (table_sql, alias, condition) - resolved_models: Dict[str, SlayerModel] = {} # model_name -> SlayerModel + resolved_joins: dict[str, tuple] = {} # alias -> (table_sql, alias, condition) + resolved_models: dict[str, SlayerModel] = {} # model_name -> SlayerModel for path in sorted_paths: alias = "__".join(path) @@ -2479,7 +2630,7 @@ async def _resolve_joins( def _remap_renamed_aliases_in_filter( *, pf: ParsedFilter, - canonical_to_user_name: Dict[str, str], + canonical_to_user_name: dict[str, str], ) -> None: """DEV-1443: rewrite canonical-agg aliases in a parsed query filter to the user-supplied alias when the same node renamed the measure. @@ -2538,11 +2689,172 @@ def _remap_renamed_aliases_in_filter( pf.columns = [eligible.get(c, c) for c in pf.columns] +# DEV-1543: rejection-helper module-level state. Pre-compiled string-literal +# regex used to mask quoted ``'a:b'`` content before colon-aggregation +# detection so the literal's colon isn't mis-classified. +_RAW_ROW_FIX_HINT = ( + "Either remove the measure reference, or set " + "distinct_dimension_values=True (the default) to keep the " + "auto-aggregating behaviour." +) +_RAW_ROW_STR_LIT_RE = re.compile(r"'(?:[^'\\]|\\.)*'") + + +def _reject_measure_ref_in_filter( + *, + raw_filter: str, + custom_agg_names: frozenset, + named_measures: dict[str, str], +) -> None: + """DEV-1543: walk a single (substituted) query filter for measure + references and raise ``DistinctDimensionValuesError`` on any match. + Co-defined with :func:`_reject_measure_ref_in_order_item` below; + both are dispatched by :func:`_reject_measure_references_for_raw_rows`. + """ + from slayer.core.errors import DistinctDimensionValuesError + from slayer.core.formula import parse_filter + + masked = _RAW_ROW_STR_LIT_RE.sub("''", raw_filter) + + # Transform-call detection FIRST. ``extract_filter_transforms`` is + # robust against filter shapes ``parse_filter`` rejects (e.g. + # ``rank(amount:sum) <= 5``). + try: + _, lifted = extract_filter_transforms( + masked, + counter=[0], + extra_agg_names=custom_agg_names, + named_measures=named_measures, + ) + except Exception: + lifted = [] + if lifted: + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False rejects measure references. " + f"Filter {raw_filter!r} contains a transform call " + f"({lifted[0][1]}). {_RAW_ROW_FIX_HINT}" + ) + + # Colon-aggregation + saved-measure detection via ``parse_filter``. + try: + parsed = parse_filter(masked, extra_agg_names=custom_agg_names) + except Exception: + # The real parser raises downstream with a useful message tied + # to the original filter text; don't pre-empt. + return + if parsed.agg_refs: + ref = parsed.agg_refs[0] + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False rejects measure references. " + f"Filter {raw_filter!r} contains an aggregation " + f"({ref.measure_name}:{ref.aggregation_name}). " + f"{_RAW_ROW_FIX_HINT}" + ) + for col in parsed.columns: + if col in named_measures: + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False rejects measure references. " + f"Filter {raw_filter!r} references saved ModelMeasure " + f"{col!r}. {_RAW_ROW_FIX_HINT}" + ) + + +def _reject_measure_ref_in_order_item( + *, + item: OrderItem, + custom_agg_names: frozenset, + named_measures: dict[str, str], +) -> None: + """DEV-1543: walk a single ``OrderItem`` for measure references. + + Covers ``raw_formula`` shapes (``AggregatedMeasureRef``, + ``TransformField``, ``MixedArithmeticField``, and ``ArithmeticField`` + when its ``agg_refs`` is non-empty — i.e. arithmetic OVER + aggregations like ``revenue:sum / *:count``; a scalar arithmetic + formula like ``amount + 1`` is fine in raw-row mode) plus the + bare-name case where ``item.column.name`` resolves to a saved + ``ModelMeasure``. + """ + from slayer.core.errors import DistinctDimensionValuesError + from slayer.core.formula import ( + AggregatedMeasureRef as _AggRef, + ArithmeticField as _ArithField, + MixedArithmeticField as _MixedField, + TransformField as _TransformField, + parse_formula, + ) + + if item.raw_formula is not None: + try: + spec = parse_formula( + item.raw_formula, + extra_agg_names=custom_agg_names, + named_measures=named_measures, + ) + except Exception: + spec = None + # Direct aggregation / transform / mixed forms are always rejected. + is_agg_form = isinstance(spec, (_AggRef, _TransformField, _MixedField)) + # ArithmeticField is only an aggregation form when it actually + # carries aggregate refs (``revenue:sum / *:count``); a pure + # scalar arithmetic like ``amount + 1`` is fine in raw-row mode. + is_agg_arith = isinstance(spec, _ArithField) and bool(spec.agg_refs) + if is_agg_form or is_agg_arith: + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False rejects measure references. " + f"Order item raw_formula={item.raw_formula!r} contains a " + f"measure / transform reference. {_RAW_ROW_FIX_HINT}" + ) + + # Bare-name resolution: OrderItem.column.name matches a saved measure. + col_name = item.column.name if item.column else None + if col_name and col_name in named_measures: + raise DistinctDimensionValuesError( + f"distinct_dimension_values=False rejects measure references. " + f"Order item column={col_name!r} resolves to a saved " + f"ModelMeasure on the source model. {_RAW_ROW_FIX_HINT}" + ) + + +def _reject_measure_references_for_raw_rows( + *, + query: SlayerQuery, + query_filters: list[str], + custom_agg_names: frozenset, + named_measures: dict[str, str], +) -> None: + """DEV-1543: when ``query.distinct_dimension_values is False``, reject + any measure reference in ``query.filters`` or ``query.order``. + + Hooks AFTER variable substitution and BEFORE + ``extract_filter_transforms`` / order-formula hoisting, so the + original measure-reference shape is still visible. The construction- + time check in ``SlayerQuery._validate_distinct_dimension_values`` is + structural only (``measures`` non-empty, dims+tds both empty); this + pass is the authoritative measure-reference catch. + + Dispatches per-filter and per-order-item to the focused helpers + above so each unit stays cognitively simple. + """ + for raw_filter in query_filters: + _reject_measure_ref_in_filter( + raw_filter=raw_filter, + custom_agg_names=custom_agg_names, + named_measures=named_measures, + ) + for item in query.order or []: + _reject_measure_ref_in_order_item( + item=item, + custom_agg_names=custom_agg_names, + named_measures=named_measures, + ) + + def extract_filter_transforms( filter_str: str, - counter: Optional[List[int]] = None, - extra_agg_names: Optional[frozenset[str]] = None, - named_measures: Optional[Mapping[str, str]] = None, + counter: list[int] | None = None, + extra_agg_names: frozenset[str] | None = None, + named_measures: Mapping[str, str] | None = None, ) -> tuple: """Extract transform function calls from a filter string. @@ -2578,7 +2890,9 @@ def extract_filter_transforms( preprocessed = _preprocess_concat(preprocessed) preprocessed = _preprocess_like(preprocessed) # Preprocess colon syntax (e.g., "order_total:sum") into ast-safe placeholders - preprocessed, agg_refs = _preprocess_agg_refs(preprocessed) + preprocessed, agg_refs = _preprocess_agg_refs( + formula=preprocessed, custom_agg_names=extra_agg_names or frozenset() + ) # Build reverse map: placeholder → original colon form _agg_reverse = { ph: ( @@ -2594,7 +2908,7 @@ def extract_filter_transforms( except SyntaxError: return filter_str, [] - transforms: List[tuple] = [] + transforms: list[tuple] = [] def _unmangle(s: str) -> str: """Restore colon syntax from placeholders in unparsed formulas.""" @@ -2627,6 +2941,65 @@ def _replace(node): return _unmangle(_ast.unparse(modified)), transforms +# DEV-1539: compound AST shapes that ALWAYS need an outer ``(...)`` +# wrap when their SQL is substituted into a filter context with a +# surrounding comparator. Checked **before** the atomic list below +# because in sqlglot 30.4.3 ``exp.And`` and ``exp.Or`` inherit from +# ``exp.Func`` — a single inverse-atomic check would mis-classify +# ``a AND b`` as atomic and skip the wrap (Codex finding). +_COMPOUND_FILTER_INLINE_TYPES: tuple = ( + exp.Binary, # arith / comparison + exp.Connector, # AND / OR + exp.Unary, # NOT, -x + exp.Predicate, # BETWEEN, IN, LIKE, IS, … +) + +# DEV-1539: AST shapes whose precedence is already unambiguous when +# substituted into a filter context. A ``Column.sql`` body whose root +# is one of these does NOT need an outer paren wrap. Used as the +# fallback after the compound-types check. +_ATOMIC_FILTER_INLINE_TYPES: tuple = ( + exp.Column, + exp.Literal, + exp.Func, # covers function calls, CAST, CASE, Anonymous, … + exp.Paren, # already self-wrapped + exp.Boolean, # TRUE / FALSE + exp.Null, +) + + +def _filter_inline_needs_paren_wrap(*, sql: str, dialect: str) -> bool: + """DEV-1539: decide whether an inlined ``Column.sql`` body needs an + outer ``(...)`` wrap when substituted into a filter's text. + + The wrap matters when the body is a multi-term / predicate + expression whose precedence is ambiguous against the surrounding + comparator. Atomic shapes — bare columns, literals, single + function calls, single CASE / CAST — are already unambiguous; + wrapping them adds noise without changing meaning. **Anything + else** (BinOp, BoolOp, ``NOT``, ``BETWEEN``, ``IN``, ``LIKE``, + ``IS``, …) needs wrapping. + + Parses ``sql`` once via sqlglot to determine the root AST shape. + The compound-type check fires first because in sqlglot 30.4.3 + ``exp.And`` / ``exp.Or`` inherit from ``exp.Func``; a single + inverse-atomic check would mis-classify ``a AND b`` as atomic. + Conservative on parse failure: returns ``True`` so the caller wraps + (errs on the side of correctness over noise). + """ + try: + # DEV-1686: prequote reserved qualifiers so a filter over a reserved + # joined model classifies correctly instead of conservatively wrapping. + tree = sqlglot.parse_one( + prequote_reserved_identifiers(sql=sql, dialect=dialect), dialect=dialect + ) + except Exception: # noqa: BLE001 — sqlglot raises a variety of error types + return True + if isinstance(tree, _COMPOUND_FILTER_INLINE_TYPES): + return True + return not isinstance(tree, _ATOMIC_FILTER_INLINE_TYPES) + + async def resolve_filter_columns( parsed_filters: list, model: SlayerModel, @@ -2638,7 +3011,7 @@ async def resolve_filter_columns( *, strict: bool = False, drop_if_unresolved: bool = False, - query_aliases: Optional[Set[str]] = None, + query_aliases: set[str] | None = None, ) -> list: """Resolve filter column references through model dimensions/measures. @@ -2708,9 +3081,24 @@ async def _expanded_sql_expr(*, sql_expr: str, owning_model: SlayerModel, alias_path=model_name, is_root=True, ) + # DEV-1539: wrap the inlined non-bare Column.sql + # in outer parens so the precedence of any + # surrounding comparator is explicit. Mirrors the + # ``exp.Paren`` wrap that ``expand_derived_refs`` + # already applies to spliced derived bodies. Skip + # the wrap when the inlined body is already a + # single atomic expression (literal / column / + # function call) — its precedence is unambiguous + # and the wrap would only add noise. + if _filter_inline_needs_paren_wrap(sql=qualified, dialect=dialect): + qualified = f"({qualified})" + # DEV-1539: lambda replacement so backslashes inside + # ``qualified`` aren't interpreted as ``re`` escape + # sequences (which would silently halve ``\\`` or + # raise on a ``\1`` backref). resolved_sql = _re.sub( rf"(? list: """Classify filters as WHERE, HAVING, or post-filter. diff --git a/slayer/engine/ingestion.py b/slayer/engine/ingestion.py index 61ade171..a4202009 100644 --- a/slayer/engine/ingestion.py +++ b/slayer/engine/ingestion.py @@ -11,16 +11,22 @@ import logging import sys from collections import defaultdict, deque -from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set, TextIO, Tuple +from typing import TYPE_CHECKING, Any, TextIO import sqlalchemy as sa +import sqlalchemy.dialects.mssql as _sqla_mssql from pydantic import BaseModel, Field from slayer.core.enums import DataType from slayer.core.format import NumberFormat, NumberFormatType from slayer.core.models import Column, DatasourceConfig, ModelJoin, SlayerModel -from slayer.engine.profiling import refresh_all_table_backed_sampled -from slayer.engine.query_engine import SlayerQueryEngine +from slayer.engine.introspect_utils import ( # noqa: F401 (re-exported for back-compat) + _FLOAT_LIKE_INFO_SCHEMA_TYPES, + _INFO_SCHEMA_TYPE_MAP, + _get_columns_fallback, + _parse_info_schema_is_float, + _safe_get_columns, +) from slayer.core.errors import AmbiguousModelError, EntityResolutionError from slayer.memories.models import MEMORY_CANONICAL_PREFIX as _MEMORY_PREFIX from slayer.memories.resolver import ( @@ -31,16 +37,49 @@ if TYPE_CHECKING: # The runtime import lives inside ``_refresh_datasource_embeddings`` - # so the embeddings module stays off the cold-start import graph - # when the optional extra isn't installed. - from slayer.embeddings.service import EmbeddingService + # so the search module stays off the cold-start import graph + # when the optional embedding extra isn't installed. + from slayer.search.service import SearchService logger = logging.getLogger(__name__) # Module-level dedup set for unrecognized SA type warnings (see # _sa_type_to_data_type). Keyed by upper-cased class name. -_logged_unmapped_sa_types: Set[str] = set() +_logged_unmapped_sa_types: set[str] = set() + +# Database types with no usable equality operator — grouping, DISTINCT or +# aggregating them fails at the database ("could not identify an equality +# operator for type point"). These map to ``DataType.UNKNOWN``: stored and +# displayed, never operated on. To query inside one, define a derived Column +# whose ``sql`` is a dialect-specific expression — e.g. +# ``Column(name="status", sql="payload->>'status'", type=TEXT)`` — which is +# emitted into the generated SQL and groups/filters like any other column. +# +# Deliberately a small allow-list of known-bad types rather than "everything +# unrecognized": comparable-but-unmapped types (uuid, jsonb, bytea, arrays, +# inet, citext, ...) are common and must keep working as TEXT. Marking one of +# those opaque would tell an agent that a perfectly groupable column is +# unusable — a worse failure than the query-time error opacity exists to +# prevent, which the Data Profile fallback already degrades gracefully. +# Membership verified against the Postgres catalog: a type is groupable iff it +# has a *default* btree/hash operator class (that is exactly what GROUP BY and +# DISTINCT require) — +# SELECT EXISTS (SELECT 1 FROM pg_opclass oc JOIN pg_am am ON am.oid = oc.opcmethod +# WHERE oc.opcintype = t.oid AND am.amname IN ('btree','hash') +# AND oc.opcdefault) +# Note ``tsvector`` / ``tsquery`` ARE groupable and must not be listed here, +# and ``jsonb`` is groupable while ``json`` is not. +_OPAQUE_SA_TYPE_NAMES = frozenset({ + "JSON", # ``jsonb`` is groupable and deliberately absent + "XML", + "TXID_SNAPSHOT", + # Geometric / spatial — none have a default btree/hash opclass + "POINT", "LINE", "LSEG", "BOX", "PATH", "POLYGON", "CIRCLE", + "GEOMETRY", "GEOGRAPHY", "RASTER", + # Range types + "INT4RANGE", "INT8RANGE", "NUMRANGE", "TSRANGE", "TSTZRANGE", "DATERANGE", +}) # Map SQLAlchemy types to SLayer DataTypes. # DEV-1361: integer family → INT, floating family → DOUBLE, NUMERIC/DECIMAL @@ -70,11 +109,16 @@ # Boolean "BOOLEAN": DataType.BOOLEAN, "BOOL": DataType.BOOLEAN, + "BIT": DataType.BOOLEAN, # T-SQL (SQL Server) boolean type # Temporal "TIMESTAMP": DataType.TIMESTAMP, "DATETIME": DataType.TIMESTAMP, "TIMESTAMP WITHOUT TIME ZONE": DataType.TIMESTAMP, "TIMESTAMP WITH TIME ZONE": DataType.TIMESTAMP, + # Snowflake (DEV-1551) — three timestamp variants by timezone semantics. + "TIMESTAMP_NTZ": DataType.TIMESTAMP, + "TIMESTAMP_LTZ": DataType.TIMESTAMP, + "TIMESTAMP_TZ": DataType.TIMESTAMP, "DATE": DataType.DATE, "TIME": DataType.TIMESTAMP, # ClickHouse adapter integer types → INT @@ -95,6 +139,18 @@ "FLOAT64": DataType.DOUBLE, "DATETIME64": DataType.TIMESTAMP, "DATE32": DataType.DATE, + # T-SQL (SQL Server) types; TINYINT also covers MySQL/MariaDB + "TINYINT": DataType.INT, + "DATETIME2": DataType.TIMESTAMP, + "SMALLDATETIME": DataType.TIMESTAMP, + "DATETIMEOFFSET": DataType.TIMESTAMP, + "NVARCHAR": DataType.TEXT, + "NCHAR": DataType.TEXT, + "NTEXT": DataType.TEXT, + "MONEY": DataType.DOUBLE, + "SMALLMONEY": DataType.DOUBLE, + # SQL Server rowversion — 8-byte binary counter, not temporal + "ROWVERSION": DataType.TEXT, } _NUMERIC_TYPES = {DataType.INT, DataType.DOUBLE} @@ -111,6 +167,9 @@ # ClickHouse adapter (clickhouse-sqlalchemy) "FLOAT32", "FLOAT64", + # T-SQL monetary types (fixed-precision decimal, no integer rounding) + "MONEY", + "SMALLMONEY", } ) @@ -122,39 +181,9 @@ # NUMERIC/DECIMAL type names — float-like only when scale > 0 _NUMERIC_DECIMAL_TYPES = frozenset({"NUMERIC", "DECIMAL"}) -# Float-like INFORMATION_SCHEMA type names -_FLOAT_LIKE_INFO_SCHEMA_TYPES = frozenset( - { - "FLOAT", - "DOUBLE", - "REAL", - } -) - -# Map INFORMATION_SCHEMA type names to SLayer DataTypes (for DuckDB fallback). -# DEV-1361: integer family → INT, floating family → DOUBLE. -_INFO_SCHEMA_TYPE_MAP = { - # Integer family - "INTEGER": DataType.INT, - "BIGINT": DataType.INT, - "SMALLINT": DataType.INT, - "TINYINT": DataType.INT, - "HUGEINT": DataType.INT, - # Floating family - "FLOAT": DataType.DOUBLE, - "DOUBLE": DataType.DOUBLE, - "REAL": DataType.DOUBLE, - # Strings / boolean / temporal - "VARCHAR": DataType.TEXT, - "CHAR": DataType.TEXT, - "TEXT": DataType.TEXT, - "BOOLEAN": DataType.BOOLEAN, - "TIMESTAMP": DataType.TIMESTAMP, - "TIMESTAMP WITH TIME ZONE": DataType.TIMESTAMP, - "DATETIME": DataType.TIMESTAMP, - "DATE": DataType.DATE, - "TIME": DataType.TIMESTAMP, -} +# INFORMATION_SCHEMA type maps + ``_safe_get_columns`` / ``_get_columns_fallback`` +# now live in the dependency-free ``introspect_utils`` leaf module (DEV-1578); +# imported + re-exported at the top of this file for back-compat. def _is_id_column(name: str) -> bool: @@ -184,8 +213,17 @@ def _unwrap_clickhouse_wrappers(sa_type: sa.types.TypeEngine) -> sa.types.TypeEn def _sa_type_to_data_type(sa_type: sa.types.TypeEngine) -> DataType: sa_type = _unwrap_clickhouse_wrappers(sa_type) + # mssql.TIMESTAMP is SQL Server's rowversion (8-byte binary counter), not + # a temporal type. Its class name collides with sa.TIMESTAMP, so we must + # check isinstance before the generic name-based _SA_TYPE_MAP lookup. + if isinstance(sa_type, _sqla_mssql.TIMESTAMP): + return DataType.TEXT type_name = type(sa_type).__name__.upper() type_str = str(sa_type).split("(")[0].upper().strip() + # Types with no equality operator are opaque: querying them fails at the + # database, so declare that explicitly instead of pretending they're TEXT. + if type_name in _OPAQUE_SA_TYPE_NAMES or type_str in _OPAQUE_SA_TYPE_NAMES: + return DataType.UNKNOWN # DEV-1361: NUMERIC/DECIMAL with scale=0 are integer-shaped → INT. # Anything float-like (scale>0 or unknown) → DOUBLE. if type_name in _NUMERIC_DECIMAL_TYPES or type_str in _NUMERIC_DECIMAL_TYPES: @@ -198,13 +236,34 @@ def _sa_type_to_data_type(sa_type: sa.types.TypeEngine) -> DataType: _logged_unmapped_sa_types.add(type_name) logger.warning( "Unrecognized SQLAlchemy type %r (str=%r); falling back to " - "DataType.TEXT. Consider adding to _SA_TYPE_MAP.", + "DataType.TEXT. Most unmapped types (uuid, jsonb, bytea, arrays) " + "are comparable and work as TEXT; add genuinely non-comparable " + "ones to _OPAQUE_SA_TYPE_NAMES and the rest to _SA_TYPE_MAP.", type_name, str(sa_type), ) return DataType.TEXT +def _raw_db_type_str(sa_type: sa.types.TypeEngine) -> str | None: + """Best-effort raw database type string for ``Column.db_type``. + + ``str(sa_type)`` renders the dialect-level spelling (``"point"``, + ``"jsonb"``, ``"geometry(Point,4326)"``). Some third-party types raise + when compiled without a dialect, so fall back to the SA class name and + finally to ``None`` — ``db_type`` is metadata, never worth aborting an + ingest over. + """ + try: + text = str(sa_type).strip() + except Exception: + text = "" + if text: + return text + name = type(sa_type).__name__ + return name or None + + def _sa_type_is_float(sa_type: sa.types.TypeEngine) -> bool: """Return True if the SQLAlchemy type is float-like. @@ -242,9 +301,9 @@ class RollupGraphError(Exception): def _get_fk_relationships( inspector: sa.engine.Inspector, table_name: str, - schema: Optional[str], - table_set: Set[str], -) -> List[tuple]: + schema: str | None, + table_set: set[str], +) -> list[tuple]: """Get FK relationships for a table, filtered to tables in table_set. Returns list of (source_column, target_table, target_column). @@ -264,12 +323,12 @@ def _get_fk_relationships( def _build_fk_graph( inspector: sa.engine.Inspector, - table_names: List[str], - schema: Optional[str], -) -> Dict[str, Set[str]]: + table_names: list[str], + schema: str | None, +) -> dict[str, set[str]]: """Build directed graph: graph[table] = set of tables it references via FK.""" table_set = set(table_names) - graph: Dict[str, Set[str]] = defaultdict(set) + graph: dict[str, set[str]] = defaultdict(set) for table_name in table_names: for _, ref_table, _ in _get_fk_relationships( inspector=inspector, @@ -281,12 +340,12 @@ def _build_fk_graph( return dict(graph) -def _check_acyclic(graph: Dict[str, Set[str]]) -> None: +def _check_acyclic(graph: dict[str, set[str]]) -> None: """Check that FK graph is a DAG. Raises RollupGraphError if cycles found.""" - visited: Set[str] = set() - rec_stack: Set[str] = set() + visited: set[str] = set() + rec_stack: set[str] = set() - def dfs(node: str, path: List[str]) -> None: + def dfs(node: str, path: list[str]) -> None: visited.add(node) rec_stack.add(node) path.append(node) @@ -300,7 +359,7 @@ def dfs(node: str, path: List[str]) -> None: path.pop() rec_stack.remove(node) - all_nodes: Set[str] = set(graph.keys()) + all_nodes: set[str] = set(graph.keys()) for neighbors in graph.values(): all_nodes.update(neighbors) for node in all_nodes: @@ -308,9 +367,9 @@ def dfs(node: str, path: List[str]) -> None: dfs(node, []) -def _compute_transitive_closure(graph: Dict[str, Set[str]], source: str) -> Set[str]: +def _compute_transitive_closure(graph: dict[str, set[str]], source: str) -> set[str]: """BFS to find all tables transitively reachable from source (excluding source).""" - reachable: Set[str] = set() + reachable: set[str] = set() queue = deque([source]) visited = {source} while queue: @@ -331,10 +390,10 @@ def _compute_transitive_closure(graph: Dict[str, Set[str]], source: str) -> Set[ def _generate_joins( inspector: sa.engine.Inspector, source_table: str, - referenced_tables: Set[str], - schema: Optional[str], - table_set: Set[str], -) -> List[ModelJoin]: + referenced_tables: set[str], + schema: str | None, + table_set: set[str], +) -> list[ModelJoin]: """Generate direct ModelJoin objects from the source table's own FK relationships. Only emits joins for FKs defined on ``source_table`` itself — multi-hop @@ -349,7 +408,7 @@ def _generate_joins( ) joins = [] - seen_signatures: Set[Tuple[str, str, str]] = set() + seen_signatures: set[tuple[str, str, str]] = set() for src_col, ref_table, tgt_col in fk_rels: if ref_table not in referenced_tables: continue @@ -373,75 +432,11 @@ def _generate_joins( # --------------------------------------------------------------------------- -def _parse_info_schema_is_float(data_type_str: str) -> bool: - """Determine if a NUMERIC/DECIMAL info-schema type string is float-like. - - Parses scale from strings like "DECIMAL(10,2)" or "NUMERIC(10,0)". - Scale > 0 means float-like; scale == 0 means integer-like; no scale - info defaults to float-like. - """ - if "(" in data_type_str and "," in data_type_str: - try: - scale_str = data_type_str.split(",")[-1].rstrip(")").strip() - return int(scale_str) > 0 - except (ValueError, IndexError): - return True # Can't parse scale, default to float - return True # No precision/scale info, default to float - - -def _get_columns_fallback( - sa_engine: sa.Engine, - table_name: str, - schema: Optional[str], -) -> List[Dict]: - """Get columns via INFORMATION_SCHEMA when Inspector.get_columns() fails.""" - if schema: - sql = ( - "SELECT column_name, data_type " - "FROM information_schema.columns " - "WHERE table_name = :table_name " - "AND table_schema = :schema " - "ORDER BY ordinal_position" - ) - params = {"table_name": table_name, "schema": schema} - else: - sql = ( - "SELECT column_name, data_type " - "FROM information_schema.columns " - "WHERE table_name = :table_name " - "ORDER BY ordinal_position" - ) - params = {"table_name": table_name} - with sa_engine.connect() as conn: - rows = conn.execute(sa.text(sql), params).fetchall() - result = [] - for col_name, data_type_str in rows: - # Strip precision info (e.g. "DECIMAL(10,2)" → "DECIMAL") - base_type = data_type_str.split("(")[0].upper().strip() - sa_type = _INFO_SCHEMA_TYPE_MAP.get(base_type) - is_float = base_type in _FLOAT_LIKE_INFO_SCHEMA_TYPES - # NUMERIC/DECIMAL: check scale to decide float vs integer - if base_type in ("NUMERIC", "DECIMAL") or ( - sa_type is None and ("DECIMAL" in base_type or "NUMERIC" in base_type) - ): - sa_type = sa_type or DataType.DOUBLE - is_float = _parse_info_schema_is_float(data_type_str) - elif sa_type is None and "INT" in base_type: - # DEV-1361: integer-shaped types should narrow to INT, not the - # coarse DOUBLE fallback (e.g. MEDIUMINT, TINYINT variants not - # otherwise mapped). - sa_type = DataType.INT - elif sa_type is None and ("CHAR" in base_type or "TEXT" in base_type): - sa_type = DataType.TEXT - result.append({"name": col_name, "type": sa_type or DataType.TEXT, "is_float": is_float}) - return result - - def _get_pk_constraint_fallback( sa_engine: sa.Engine, table_name: str, - schema: Optional[str], -) -> Dict: + schema: str | None, +) -> dict: """Get PK constraint via INFORMATION_SCHEMA when Inspector.get_pk_constraint() fails.""" if schema: sql = ( @@ -471,25 +466,12 @@ def _get_pk_constraint_fallback( return {"constrained_columns": [row[0] for row in rows]} -def _safe_get_columns( - inspector: sa.engine.Inspector, - sa_engine: sa.Engine, - table_name: str, - schema: Optional[str], -) -> List[Dict]: - """Get columns, falling back to INFORMATION_SCHEMA on failure.""" - try: - return inspector.get_columns(table_name, schema=schema) - except Exception: - return _get_columns_fallback(sa_engine, table_name, schema) - - def _safe_get_pk_constraint( inspector: sa.engine.Inspector, sa_engine: sa.Engine, table_name: str, - schema: Optional[str], -) -> Dict: + schema: str | None, +) -> dict: """Get PK constraint, falling back to INFORMATION_SCHEMA on failure. SQLite has no information_schema views; its stock inspector reads @@ -515,15 +497,20 @@ def _introspect_query_columns_via_inspector( sa_engine: sa.Engine, inspector: sa.engine.Inspector, table_name: str, - schema: Optional[str], - rollup_sql: Optional[str], - referenced_tables: Set[str], - fk_columns_by_table: Dict[str, Set[str]], - joins: Optional[List[ModelJoin]] = None, -) -> List[tuple]: + schema: str | None, + rollup_sql: str | None, + referenced_tables: set[str], + fk_columns_by_table: dict[str, set[str]], + joins: list[ModelJoin] | None = None, +) -> list[tuple]: """Introspect columns from a rollup query or plain table. - Returns list of (column_name, DataType, is_primary_key, is_float) tuples. + Returns list of ``(column_name, DataType, is_primary_key, is_float, + db_type)`` tuples. ``db_type`` is the raw database type string and is only + populated when ``DataType`` came out opaque (``UNKNOWN``) — for mapped + types the declared ``DataType`` already carries everything, so leaving it + ``None`` keeps stored models and golden tests clean. + For rollup queries, uses per-table inspector data since LIMIT 0 type inference can be unreliable across databases. """ @@ -537,18 +524,21 @@ def _introspect_query_columns_via_inspector( for col in columns: col_name = col["name"] col_type = col["type"] + db_type: str | None = None if isinstance(col_type, DataType): data_type = col_type is_float = col.get("is_float", False) else: data_type = _sa_type_to_data_type(col_type) is_float = _sa_type_is_float(col_type) + if data_type.is_opaque: + db_type = _raw_db_type_str(col_type) is_pk = col_name in pk_columns - results.append((col_name, data_type, is_pk, is_float)) + results.append((col_name, data_type, is_pk, is_float, db_type)) # Build list of (ref_table, dotted_path) from joins — supports diamond joins # where the same table appears via multiple paths - table_path_pairs: List[tuple] = [] + table_path_pairs: list[tuple] = [] if joins: for mj in joins: if mj.join_pairs and "." in mj.join_pairs[0][0]: @@ -574,14 +564,17 @@ def _introspect_query_columns_via_inspector( continue alias = f"{path}.{col['name']}" col_type = col["type"] + ref_db_type: str | None = None if isinstance(col_type, DataType): data_type = col_type is_float = col.get("is_float", False) else: data_type = _sa_type_to_data_type(col_type) is_float = _sa_type_is_float(col_type) + if data_type.is_opaque: + ref_db_type = _raw_db_type_str(col_type) is_pk = col["name"] in ref_pk_cols - results.append((alias, data_type, is_pk, is_float)) + results.append((alias, data_type, is_pk, is_float, ref_db_type)) return results @@ -593,23 +586,25 @@ def _introspect_query_columns_via_inspector( def _columns_to_model( name: str, - columns: List[tuple], + columns: list[tuple], data_source: str, - sql_table: Optional[str] = None, - joins: Optional[List[ModelJoin]] = None, + sql_table: str | None = None, + joins: list[ModelJoin] | None = None, ) -> SlayerModel: - """Generate a SlayerModel from introspected (column_name, DataType, is_pk, is_float) tuples. + """Generate a SlayerModel from introspected ``(column_name, DataType, + is_pk, is_float, db_type)`` tuples. In v2 every Column is potentially both a dimension and a measure — what it's used as is decided per query. This function emits one Column per non-joined - column, with format inferred from the column's data type. + column, with format inferred from the column's data type. ``db_type`` is + carried through verbatim (set only for opaque ``UNKNOWN`` columns). """ - cols: List[Column] = [] + cols: list[Column] = [] _INT_FORMAT = NumberFormat(type=NumberFormatType.INTEGER) _FLOAT_FORMAT = NumberFormat(type=NumberFormatType.FLOAT) - for col_name, data_type, is_pk, is_float in columns: + for col_name, data_type, is_pk, is_float, db_type in columns: # Skip joined columns — they live on the target model and are # resolved via the join graph at query time. if "." in col_name: @@ -631,6 +626,7 @@ def _columns_to_model( name=column_name, sql=col_name, type=data_type, + db_type=db_type, primary_key=is_pk, format=fmt, ) @@ -645,14 +641,92 @@ def _columns_to_model( ) +def _sqlite_probe_integer_columns( + *, + sa_engine: sa.Engine, + sql_table: str, + columns: list[tuple], +) -> list[tuple]: + """DEV-1538: per-column SQLite affinity probe. + + Walks the tuples ``(col_name, DataType, is_pk, is_float, db_type)`` produced by + :func:`_introspect_query_columns_via_inspector` and, for every base + column (alias without ``.``) that the SA inspector reported as + :class:`DataType.INT`, runs + :func:`slayer.sql.sqlite_introspect.probe_sqlite_integer_column` against + the actual storage classes. Mutates the tuple to the widened + :class:`DataType` whenever the probe disagrees with the declared + affinity. + + No-op on non-SQLite engines. + + Failure modes: + * Non-SQLite engine → input returned verbatim. + * Probe returns ``None`` (failure or saturation) → keep the SA-derived + INT type, leave the warning already logged by the probe in place. + * Joined-column alias (``"."`` in the name) → skipped; joined references + inherit their type from the target model's own probe pass. + """ + if sa_engine.dialect.name != "sqlite": + return columns + + from slayer.sql.sqlite_introspect import probe_sqlite_integer_column + + schema, table = _parse_qualified_sql_table(sql_table) + out: list[tuple] = [] + with sa_engine.connect() as conn: + for col_name, data_type, is_pk, is_float, db_type in columns: + if data_type is not DataType.INT or "." in col_name: + out.append((col_name, data_type, is_pk, is_float, db_type)) + continue + try: + verdict = probe_sqlite_integer_column( + conn=conn, + table=table, + column=col_name, + schema=schema, + ) + except Exception as exc: + # Defence-in-depth: the helper catches its own errors but a + # caller-level guard keeps ingest from aborting on unexpected + # exceptions outside the helper's scope (e.g. import-time + # failures on environments missing sqlite_introspect). + logger.warning( + "probe call raised for %s.%s; keeping declared INT: %s", + sql_table, + col_name, + exc, + ) + verdict = None + if verdict is None or verdict is DataType.INT: + out.append((col_name, data_type, is_pk, is_float, db_type)) + continue + new_is_float = verdict is DataType.DOUBLE + out.append((col_name, verdict, is_pk, new_is_float, db_type)) + return out + + +def _parse_qualified_sql_table(sql_table: str) -> tuple[str | None, str]: + """Split ``"schema.table"`` into ``(schema, table)`` or ``(None, table)``. + + Only splits on a single dot — table/schema names containing dots are + out of scope for the auto-ingest path (the dotted form would never have + survived ``Inspector.get_table_names`` either). + """ + if "." in sql_table: + schema, _, table = sql_table.partition(".") + return schema or None, table + return None, sql_table + + def introspect_table_to_model( *, sa_engine: sa.Engine, inspector: sa.engine.Inspector, table_name: str, - schema: Optional[str], + schema: str | None, data_source: str, - model_name: Optional[str] = None, + model_name: str | None = None, ) -> SlayerModel: """Introspect a single table (no FK rollup) and return a SlayerModel. @@ -669,6 +743,11 @@ def introspect_table_to_model( fk_columns_by_table={}, ) sql_table = f"{schema}.{table_name}" if schema else table_name + columns = _sqlite_probe_integer_columns( + sa_engine=sa_engine, + sql_table=sql_table, + columns=columns, + ) return _columns_to_model( name=model_name or table_name, columns=columns, @@ -684,11 +763,12 @@ def introspect_table_to_model( def ingest_datasource( datasource: DatasourceConfig, - include_tables: Optional[List[str]] = None, - exclude_tables: Optional[List[str]] = None, - schema: Optional[str] = None, -) -> List[SlayerModel]: - sa_engine = sa.create_engine(datasource.resolve_env_vars().get_connection_string()) + include_tables: list[str] | None = None, + exclude_tables: list[str] | None = None, + schema: str | None = None, +) -> list[SlayerModel]: + from slayer.sql import engine_factory + sa_engine = engine_factory.get_engine(datasource.resolve_env_vars()) inspector = sa.inspect(sa_engine) table_names = inspector.get_table_names(schema=schema) @@ -709,7 +789,7 @@ def ingest_datasource( has_cycles = True # Collect FK columns per table (for excluding from rollup) - fk_columns_by_table: Dict[str, Set[str]] = defaultdict(set) + fk_columns_by_table: dict[str, set[str]] = defaultdict(set) for table_name in table_names: fks = inspector.get_foreign_keys(table_name, schema=schema) for fk in fks: @@ -740,6 +820,11 @@ def ingest_datasource( fk_columns_by_table=fk_columns_by_table, joins=model_joins, ) + columns = _sqlite_probe_integer_columns( + sa_engine=sa_engine, + sql_table=sql_table, + columns=columns, + ) model = _columns_to_model( name=table_name, columns=columns, @@ -758,6 +843,11 @@ def ingest_datasource( referenced_tables=set(), fk_columns_by_table=fk_columns_by_table, ) + columns = _sqlite_probe_integer_columns( + sa_engine=sa_engine, + sql_table=sql_table, + columns=columns, + ) model = _columns_to_model( name=table_name, columns=columns, @@ -767,6 +857,12 @@ def ingest_datasource( models.append(model) + # ingest_datasource is a one-shot admin operation, not a hot query + # path. Disposing here releases the underlying connection so other + # consumers (notably ``duckdb.connect(file)`` in notebooks) can open + # the same file. The engine_factory cache will rebuild on the next + # call; the cost is one extra ``sa.create_engine`` per ingest, which + # is negligible compared to the actual schema-introspection work. sa_engine.dispose() return models @@ -776,56 +872,89 @@ def ingest_datasource( # --------------------------------------------------------------------------- -def _existing_join_signatures(model: SlayerModel) -> Set[Tuple[str, Tuple[Tuple[str, str], ...]]]: +def _existing_join_signatures(model: SlayerModel) -> set[tuple[str, tuple[tuple[str, str], ...]]]: """Return the set of (target_model, sorted join_pair tuples) signatures for joins already on ``model``. Used to detect new joins. """ - out: Set[Tuple[str, Tuple[Tuple[str, str], ...]]] = set() + out: set[tuple[str, tuple[tuple[str, str], ...]]] = set() for j in model.joins: sig_pairs = tuple(sorted((p[0], p[1]) for p in j.join_pairs)) out.add((j.target_model, sig_pairs)) return out -def _additive_merge_existing( - *, - persisted: SlayerModel, - fresh: SlayerModel, -) -> Tuple[SlayerModel, List[str], List[str]]: - """Merge a freshly-ingested ``fresh`` model into ``persisted`` additively. +def _is_auto_default_integer_format(fmt: NumberFormat | None) -> bool: + """Return True when ``fmt`` looks like the auto-ingested ``NumberFormat + (type=INTEGER)`` default (no custom precision / symbol set). Used by + DEV-1538's widening path to decide whether to flip the format alongside + the type; user-set custom formats are preserved verbatim. + """ + if fmt is None: + return False + if fmt.type != NumberFormatType.INTEGER: + return False + return fmt.precision is None and fmt.symbol is None - Returns ``(merged, new_column_names, new_join_target_names)``. - * Existing columns are preserved verbatim (description / label / format / - meta / allowed_aggregations / filter never overwritten). - * Live columns whose names are absent from ``persisted.columns`` are - appended from ``fresh.columns``. - * Joins with new ``(target_model, join_pairs)`` signatures are appended. +def _format_for_widened_type(verdict: DataType) -> NumberFormat | None: + """Return the auto-default format for a probed widening verdict.""" + if verdict is DataType.DOUBLE: + return NumberFormat(type=NumberFormatType.FLOAT) + return None # TEXT clears format + + +def _merge_persisted_column_with_probe( + *, + persisted_col: Column, + fresh_col: Column | None, + model_name: str, + sqlite_widen_enabled: bool, +) -> tuple[Column, bool]: + """DEV-1538: decide whether a persisted column should be widened based + on a freshly-probed type, and return ``(merged_column, did_widen)``. + + The widen branch only fires when ``sqlite_widen_enabled`` is True + (SQLite-only auto-heal), the fresh column exists, the persisted column + is ``DataType.INT``, and the fresh type is ``DataType.DOUBLE`` or + ``DataType.TEXT``. All other cases return ``persisted_col`` unchanged. """ - existing_col_names = {c.name for c in persisted.columns} - new_columns: List[Column] = list(persisted.columns) - new_column_names: List[str] = [] - for c in fresh.columns: - if c.name in existing_col_names: - continue - new_columns.append(c) - new_column_names.append(c.name) + if not ( + sqlite_widen_enabled + and fresh_col is not None + and persisted_col.type is DataType.INT + and fresh_col.type in (DataType.DOUBLE, DataType.TEXT) + ): + return persisted_col, False + + updates: dict[str, Any] = {"type": fresh_col.type} + if _is_auto_default_integer_format(persisted_col.format): + updates["format"] = _format_for_widened_type(fresh_col.type) + else: + logger.info( + "Custom format on %s.%s preserved on SQLite probe widening " + "(persisted INT -> %s). Review whether the format still applies.", + model_name, + persisted_col.name, + fresh_col.type.value, + ) + return persisted_col.model_copy(update=updates), True + +def _merge_joins_strict( + persisted: SlayerModel, fresh: SlayerModel, +) -> tuple[list[ModelJoin], list[str]]: + """Append joins whose signature isn't already present. Raises on the + duplicate-target / different-pairs conflict so callers don't end up + with two joins pointing at the same target_model.""" existing_join_sigs = _existing_join_signatures(persisted) existing_join_targets = {j.target_model for j in persisted.joins} - new_joins: List[ModelJoin] = list(persisted.joins) - new_join_targets: List[str] = [] + new_joins: list[ModelJoin] = list(persisted.joins) + new_join_targets: list[str] = [] for j in fresh.joins: sig = (j.target_model, tuple(sorted((p[0], p[1]) for p in j.join_pairs))) if sig in existing_join_sigs: continue if j.target_model in existing_join_targets: - # Same target_model already present with a different - # join_pairs signature. Downstream consumers key joins by - # target_model only — appending a second one would let the - # stale join shadow the live one and ``remove.joins=[name]`` - # would wipe both. Surface the conflict so the user can - # decide instead of silently breaking. raise ValueError( f"Model {persisted.name!r} already has a join targeting " f"{j.target_model!r} with different join_pairs; the " @@ -836,14 +965,67 @@ def _additive_merge_existing( ) new_joins.append(j) new_join_targets.append(j.target_model) + return new_joins, new_join_targets + + +def _additive_merge_existing( + *, + persisted: SlayerModel, + fresh: SlayerModel, + sqlite_widen_enabled: bool = False, +) -> tuple[SlayerModel, list[str], list[str], list[str]]: + """Merge a freshly-ingested ``fresh`` model into ``persisted`` additively. - if not new_column_names and not new_join_targets: - return persisted, [], [] + Returns ``(merged, new_column_names, new_join_target_names, + widened_column_names)``. + + * Existing columns are preserved verbatim (description / label / format / + meta / allowed_aggregations / filter never overwritten). + * DEV-1538 carve-out (SQLite only — ``sqlite_widen_enabled=True``): a + fresh column whose type widened from the persisted ``DataType.INT`` + (i.e. fresh type is ``DOUBLE`` or ``TEXT``) replaces ONLY the persisted + type — and the persisted ``format`` IF the persisted format is the + auto-ingested ``NumberFormat(INTEGER)`` default. Custom formats are + preserved verbatim and an INFO log line is emitted naming the column. + Widening never narrows DOUBLE → INT. On non-SQLite datasources the + additive contract stays strict — schema drift surfaces via + ``slayer validate-models``, not via silent re-ingest overwrites. + * Live columns whose names are absent from ``persisted.columns`` are + appended from ``fresh.columns``. + * Joins with new ``(target_model, join_pairs)`` signatures are appended. + """ + existing_by_name: dict[str, Column] = {c.name: c for c in persisted.columns} + fresh_by_name: dict[str, Column] = {c.name: c for c in fresh.columns} + + widened_column_names: list[str] = [] + merged_columns: list[Column] = [] + for persisted_col in persisted.columns: + merged_col, did_widen = _merge_persisted_column_with_probe( + persisted_col=persisted_col, + fresh_col=fresh_by_name.get(persisted_col.name), + model_name=persisted.name, + sqlite_widen_enabled=sqlite_widen_enabled, + ) + merged_columns.append(merged_col) + if did_widen: + widened_column_names.append(persisted_col.name) + + new_column_names: list[str] = [] + for fresh_col in fresh.columns: + if fresh_col.name in existing_by_name: + continue + merged_columns.append(fresh_col) + new_column_names.append(fresh_col.name) + + new_joins, new_join_targets = _merge_joins_strict(persisted, fresh) + + if not new_column_names and not new_join_targets and not widened_column_names: + return persisted, [], [], [] merged = persisted.model_copy( - update={"columns": new_columns, "joins": new_joins} + update={"columns": merged_columns, "joins": new_joins} ) - return merged, new_column_names, new_join_targets + return merged, new_column_names, new_join_targets, widened_column_names async def _process_one_table( @@ -873,10 +1055,12 @@ async def _process_one_table( # User-authored sql / query-backed model with the matching name — # leave it alone. return None - merged, new_cols, new_joins = _additive_merge_existing( - persisted=persisted, fresh=fresh + merged, new_cols, new_joins, widened_cols = _additive_merge_existing( + persisted=persisted, + fresh=fresh, + sqlite_widen_enabled=(datasource.type or "").lower() == "sqlite", ) - if new_cols or new_joins: + if new_cols or new_joins or widened_cols: await storage.save_model(merged) return ModelAddition( model_name=table_name, @@ -884,6 +1068,7 @@ async def _process_one_table( created=False, new_columns=new_cols, new_joins=new_joins, + widened_columns=widened_cols, ) @@ -896,8 +1081,8 @@ async def _scoped_models_for_validation( *, storage: StorageBackend, datasource: DatasourceConfig, - in_scope_table_names: Set[str], -) -> List[SlayerModel]: + in_scope_table_names: set[str], +) -> list[SlayerModel]: """Build the list of persisted models to feed to ``validate_datasource``. sql_table-mode models are included only when their live table is in @@ -907,7 +1092,7 @@ async def _scoped_models_for_validation( """ identities = await storage._list_all_model_identities() ds_model_names = [n for d, n in identities if d == datasource.name] - scoped: List[SlayerModel] = [] + scoped: list[SlayerModel] = [] for name in ds_model_names: m = await storage.get_model(name, data_source=datasource.name) if m is None: @@ -924,9 +1109,9 @@ async def ingest_datasource_idempotent( *, datasource: DatasourceConfig, storage: StorageBackend, - include_tables: Optional[List[str]] = None, - exclude_tables: Optional[List[str]] = None, - schema: Optional[str] = None, + include_tables: list[str] | None = None, + exclude_tables: list[str] | None = None, + schema: str | None = None, ): """Idempotent re-ingestion (DEV-1356). @@ -950,8 +1135,8 @@ async def ingest_datasource_idempotent( validate_datasource, ) - additions: List[ModelAddition] = [] - errors: List[IngestionError] = [] + additions: list[ModelAddition] = [] + errors: list[IngestionError] = [] # ``ingest_datasource`` is sync (it drives SQLAlchemy ``Inspector``). # Offload to a thread so a slow / large datasource doesn't block the @@ -964,7 +1149,7 @@ async def ingest_datasource_idempotent( schema=schema, ) fresh_by_name = {m.name: m for m in fresh_models} - in_scope_table_names: Set[str] = set(fresh_by_name.keys()) + in_scope_table_names: set[str] = set(fresh_by_name.keys()) for table_name, fresh in fresh_by_name.items(): try: @@ -994,31 +1179,20 @@ async def ingest_datasource_idempotent( datasource=datasource, models=scoped_models ) - # DEV-1375: refresh persisted Column.sampled values for every - # table-backed model in this datasource. Best-effort: per-column - # failures are accumulated as IngestionError entries; an unexpected - # raise is also caught so ingestion's idempotent contract holds. - refresh_engine = SlayerQueryEngine(storage=storage) - try: - refresh_errors = await refresh_all_table_backed_sampled( - engine=refresh_engine, - storage=storage, - data_source=datasource.name, - ) - except Exception as exc: - refresh_errors = [f"{datasource.name}: {exc}"] - for err in refresh_errors: - errors.append(IngestionError( - model_name=err.split(".", 1)[0] if "." in err else "", - data_source=datasource.name, - error=f"sample-value refresh: {err}", - )) + # Column sample-value profiling is NOT run at ingest time — it fires a + # per-column full-table scan and, on a wide datasource (dozens of tables + # × ~10 columns each), would run hundreds of full scans and dominate + # ingest wall-clock. Samples are instead refreshed on demand on a cache + # miss by the async ``ensure_column_sample_fresh`` helper, invoked from + # the read paths that surface samples — ``inspect_model``, the ``inspect`` + # point-lookup, and ``search()``. Use ``slayer search refresh-samples`` + # to warm the cache explicitly. # DEV-1386: refresh persisted embeddings for the datasource doc plus # every visible model + its visible children. Best-effort: per-entity # failures are surfaced as IngestionError entries, never aborts - # ingestion. When the `embedding_search` extra is not installed, - # EmbeddingService returns a single warning and does no work. + # ingestion. When the `advanced_search` extra is not installed, + # EmbeddingRetriever returns a single warning and does no work. embedding_errors = await _refresh_datasource_embeddings( datasource_name=datasource.name, storage=storage, ) @@ -1081,7 +1255,7 @@ def _friendly_db_error(exc: Exception) -> str: def _print_ingest_addition( - addition, *, file: Optional[TextIO] = None + addition, *, file: TextIO | None = None ) -> None: out = file if file is not None else sys.stdout if addition.created: @@ -1090,18 +1264,21 @@ def _print_ingest_addition( file=out, ) return - if not (addition.new_columns or addition.new_joins): + widened = getattr(addition, "widened_columns", []) or [] + if not (addition.new_columns or addition.new_joins or widened): return details = [] if addition.new_columns: details.append(f"+columns: {', '.join(addition.new_columns)}") if addition.new_joins: details.append(f"+joins: {', '.join(addition.new_joins)}") + if widened: + details.append(f"widened: {', '.join(widened)}") print(f"Updated: {addition.model_name} ({'; '.join(details)})", file=out) def _print_ingest_drift_and_errors( - result, *, file: Optional[TextIO] = None + result, *, file: TextIO | None = None ) -> None: out = file if file is not None else sys.stdout if result.to_delete: @@ -1136,15 +1313,15 @@ class StartupIngestSummary(BaseModel): ``EditModelDelete | WholeModelDelete``. """ - succeeded: List[str] = Field(default_factory=list) - failures: List[StartupIngestFailure] = Field(default_factory=list) - drift_pending: List[Any] = Field(default_factory=list) + succeeded: list[str] = Field(default_factory=list) + failures: list[StartupIngestFailure] = Field(default_factory=list) + drift_pending: list[Any] = Field(default_factory=list) async def ingest_all_datasources_idempotent( *, storage: StorageBackend, - stream: Optional[TextIO] = None, + stream: TextIO | None = None, ) -> StartupIngestSummary: """Run idempotent auto-ingestion across every configured datasource. @@ -1217,8 +1394,8 @@ async def _refresh_models_for_datasource( *, datasource_name: str, storage: StorageBackend, - service: "EmbeddingService", -) -> Tuple[List[Tuple[str, str]], List[SlayerModel]]: + search: "SearchService", +) -> tuple[list[tuple[str, str]], list[SlayerModel]]: """Refresh embeddings for every visible model in the datasource. Returns ``(warnings, models_in_ds)``. Each warning is tagged with @@ -1226,8 +1403,8 @@ async def _refresh_models_for_datasource( right ``IngestionError.model_name``. ``models_in_ds`` is forwarded to the datasource-doc refresh that follows. """ - warnings: List[Tuple[str, str]] = [] - models_in_ds: List[SlayerModel] = [] + warnings: list[tuple[str, str]] = [] + models_in_ds: list[SlayerModel] = [] try: identities = await storage._list_all_model_identities() except Exception as exc: # noqa: BLE001 — defensive @@ -1245,7 +1422,7 @@ async def _refresh_models_for_datasource( continue models_in_ds.append(m) try: - subtree_warnings = await service.refresh_model_subtree(m) + subtree_warnings = await search.refresh_model_subtree(m) except Exception as exc: # noqa: BLE001 — defensive per-model subtree_warnings = [str(exc)] for w in subtree_warnings: @@ -1256,14 +1433,21 @@ async def _refresh_models_for_datasource( async def _refresh_datasource_doc( *, datasource_name: str, - models: List[SlayerModel], - service: "EmbeddingService", -) -> List[Tuple[str, str]]: + models: list[SlayerModel], + search: "SearchService", + storage: StorageBackend, +) -> list[tuple[str, str]]: """Refresh the datasource doc embedding. Warnings are tagged with - an empty ``model_name`` since the doc has no specific entity name.""" + an empty ``model_name`` since the doc has no specific entity name. + + DEV-1549: ``DatasourceConfig.description`` is threaded through so + description text contributes to lexical + embedding recall. + """ + cfg = await storage.get_datasource(datasource_name) + description = cfg.description if cfg is not None else None try: - doc_warnings = await service.refresh_datasource( - name=datasource_name, models=models, + doc_warnings = await search.refresh_datasource( + name=datasource_name, models=models, description=description, ) except Exception as exc: # noqa: BLE001 — defensive return [("", f"{datasource_name} (datasource doc): {exc}")] @@ -1272,7 +1456,7 @@ async def _refresh_datasource_doc( async def _entity_ref_exists( *, entity: str, storage: StorageBackend, -) -> Optional[bool]: +) -> bool | None: """DEV-1428 defense-in-depth cleanup probe. Returns: * ``True`` when the canonical ref still resolves. @@ -1322,8 +1506,8 @@ async def _refresh_memories_for_datasource( # NOSONAR(S3776) — straight-line *, datasource_name: str, storage: StorageBackend, - service: "EmbeddingService", -) -> List[Tuple[str, str]]: + search: "SearchService", +) -> list[tuple[str, str]]: """Refresh embeddings for every memory whose canonical entities are rooted at this datasource. Each warning is tagged with ``memory:`` so a startup log inspection can distinguish memory @@ -1343,7 +1527,7 @@ async def _refresh_memories_for_datasource( # NOSONAR(S3776) — straight-line memories = await storage.list_memories() except Exception as exc: # noqa: BLE001 — defensive return [("", f"{datasource_name} (memories): {exc}")] - warnings: List[Tuple[str, str]] = [] + warnings: list[tuple[str, str]] = [] for memory in memories: rooted_at_ds = any( canonical_id_rooted_at(e, datasource_name) @@ -1363,14 +1547,14 @@ async def _refresh_memories_for_datasource( # NOSONAR(S3776) — straight-line tag = f"{_MEMORY_PREFIX}{memory.id}" if rooted_at_ds: try: - memory_warnings = await service.refresh_memory(memory) + memory_warnings = await search.upsert_memory(memory) except Exception as exc: # noqa: BLE001 — defensive per-memory memory_warnings = [str(exc)] for w in memory_warnings: warnings.append((tag, w)) # DEV-1428 cleanup pass: drop refs that resolve to False # (definitive not-found); keep refs that raise (transient). - cleaned: List[str] = [] + cleaned: list[str] = [] changed = False for entity in memory.entities: exists = await _entity_ref_exists( @@ -1401,7 +1585,7 @@ async def _refresh_memories_for_datasource( # NOSONAR(S3776) — straight-line async def _refresh_datasource_embeddings( *, datasource_name: str, storage: StorageBackend, -) -> List[Tuple[str, str]]: +) -> list[tuple[str, str]]: """Refresh persisted embeddings for everything reachable from this datasource: every visible model + its visible children, the datasource doc itself, and every memory whose canonical entities @@ -1413,18 +1597,21 @@ async def _refresh_datasource_embeddings( doc) used by ``ingest_datasource_idempotent`` to route per-entity failures to the matching ``IngestionError``. """ - # Local import to avoid pulling embeddings into ingestion's import - # graph on a cold start without the optional extra installed. - from slayer.embeddings.service import EmbeddingService + # Local import: keep the search module off the cold-start path + # when the optional embedding extra isn't installed. + from slayer.search.service import SearchService - service = EmbeddingService(storage=storage) + search = SearchService(storage=storage) model_warnings, models_in_ds = await _refresh_models_for_datasource( - datasource_name=datasource_name, storage=storage, service=service, + datasource_name=datasource_name, storage=storage, search=search, ) doc_warnings = await _refresh_datasource_doc( - datasource_name=datasource_name, models=models_in_ds, service=service, + datasource_name=datasource_name, + models=models_in_ds, + search=search, + storage=storage, ) memory_warnings = await _refresh_memories_for_datasource( - datasource_name=datasource_name, storage=storage, service=service, + datasource_name=datasource_name, storage=storage, search=search, ) return model_warnings + doc_warnings + memory_warnings diff --git a/slayer/engine/introspect_utils.py b/slayer/engine/introspect_utils.py new file mode 100644 index 00000000..17ea153d --- /dev/null +++ b/slayer/engine/introspect_utils.py @@ -0,0 +1,131 @@ +"""Dependency-free column-introspection helpers. + +Extracted from ``slayer/engine/ingestion.py`` (DEV-1578) so the +forced-filter column-presence probe in ``slayer/engine/query_engine.py`` +can reuse ``_safe_get_columns`` without importing ``ingestion`` (which +imports ``query_engine`` — a cycle). ``ingestion`` and ``schema_drift`` +import these from here; ``ingestion`` also re-exports them for back-compat. + +``_safe_get_columns`` tries SQLAlchemy's ``Inspector.get_columns`` first and +falls back to a parameterized ``INFORMATION_SCHEMA.columns`` query when +reflection raises — see ``docs`` / the ingestion module for rationale. +""" + +from __future__ import annotations + +from typing import Dict, List, Optional + +import sqlalchemy as sa + +from slayer.core.enums import DataType + +# Float-like INFORMATION_SCHEMA type names +_FLOAT_LIKE_INFO_SCHEMA_TYPES = frozenset( + { + "FLOAT", + "DOUBLE", + "REAL", + } +) + +# Map INFORMATION_SCHEMA type names to SLayer DataTypes (for DuckDB fallback). +# DEV-1361: integer family → INT, floating family → DOUBLE. +_INFO_SCHEMA_TYPE_MAP = { + # Integer family + "INTEGER": DataType.INT, + "BIGINT": DataType.INT, + "SMALLINT": DataType.INT, + "TINYINT": DataType.INT, + "HUGEINT": DataType.INT, + # Floating family + "FLOAT": DataType.DOUBLE, + "DOUBLE": DataType.DOUBLE, + "REAL": DataType.DOUBLE, + # Strings / boolean / temporal + "VARCHAR": DataType.TEXT, + "CHAR": DataType.TEXT, + "TEXT": DataType.TEXT, + "BOOLEAN": DataType.BOOLEAN, + "TIMESTAMP": DataType.TIMESTAMP, + "TIMESTAMP WITH TIME ZONE": DataType.TIMESTAMP, + "DATETIME": DataType.TIMESTAMP, + "DATE": DataType.DATE, + "TIME": DataType.TIMESTAMP, +} + + +def _parse_info_schema_is_float(data_type_str: str) -> bool: + """Determine if a NUMERIC/DECIMAL info-schema type string is float-like. + + Parses scale from strings like "DECIMAL(10,2)" or "NUMERIC(10,0)". + Scale > 0 means float-like; scale == 0 means integer-like; no scale + info defaults to float-like. + """ + if "(" in data_type_str and "," in data_type_str: + try: + scale_str = data_type_str.split(",")[-1].rstrip(")").strip() + return int(scale_str) > 0 + except (ValueError, IndexError): + return True # Can't parse scale, default to float + return True # No precision/scale info, default to float + + +def _get_columns_fallback( + sa_engine: sa.Engine, + table_name: str, + schema: Optional[str], +) -> List[Dict]: + """Get columns via INFORMATION_SCHEMA when Inspector.get_columns() fails.""" + if schema: + sql = ( + "SELECT column_name, data_type " + "FROM information_schema.columns " + "WHERE table_name = :table_name " + "AND table_schema = :schema " + "ORDER BY ordinal_position" + ) + params = {"table_name": table_name, "schema": schema} + else: + sql = ( + "SELECT column_name, data_type " + "FROM information_schema.columns " + "WHERE table_name = :table_name " + "ORDER BY ordinal_position" + ) + params = {"table_name": table_name} + with sa_engine.connect() as conn: + rows = conn.execute(sa.text(sql), params).fetchall() + result = [] + for col_name, data_type_str in rows: + # Strip precision info (e.g. "DECIMAL(10,2)" → "DECIMAL") + base_type = data_type_str.split("(")[0].upper().strip() + sa_type = _INFO_SCHEMA_TYPE_MAP.get(base_type) + is_float = base_type in _FLOAT_LIKE_INFO_SCHEMA_TYPES + # NUMERIC/DECIMAL: check scale to decide float vs integer + if base_type in ("NUMERIC", "DECIMAL") or ( + sa_type is None and ("DECIMAL" in base_type or "NUMERIC" in base_type) + ): + sa_type = sa_type or DataType.DOUBLE + is_float = _parse_info_schema_is_float(data_type_str) + elif sa_type is None and "INT" in base_type: + # DEV-1361: integer-shaped types should narrow to INT, not the + # coarse DOUBLE fallback (e.g. MEDIUMINT, TINYINT variants not + # otherwise mapped). + sa_type = DataType.INT + elif sa_type is None and ("CHAR" in base_type or "TEXT" in base_type): + sa_type = DataType.TEXT + result.append({"name": col_name, "type": sa_type or DataType.TEXT, "is_float": is_float}) + return result + + +def _safe_get_columns( + inspector: sa.engine.Inspector, + sa_engine: sa.Engine, + table_name: str, + schema: Optional[str], +) -> List[Dict]: + """Get columns, falling back to INFORMATION_SCHEMA on failure.""" + try: + return inspector.get_columns(table_name, schema=schema) + except Exception: + return _get_columns_fallback(sa_engine, table_name, schema) diff --git a/slayer/engine/join_graph.py b/slayer/engine/join_graph.py new file mode 100644 index 00000000..8f628962 --- /dev/null +++ b/slayer/engine/join_graph.py @@ -0,0 +1,129 @@ +"""Pure in-memory join-graph routing primitive (DEV-1626). + +``JoinGraph`` builds a directed adjacency from a set of models' *stored +outgoing* joins and answers reachability / shortest-path questions. It is +join-type-agnostic: it simply reads each model's outgoing ``joins``. + +INNER joins are kept symmetric by the storage layer +(``slayer/storage/join_sync.py`` materialises a reverse ``B→A`` edge for +every ``A→B`` INNER join) — the same invariant the query engine's own +``_walk_join_chain`` relies on. So a symmetric INNER pair appears here as +two directed edges and is therefore traversable in both directions, and +every path this primitive emits is walkable by the engine at query time. + +The module is dependency-light (only ``slayer.core.models`` for typing) +and free of storage / async, so it is trivially unit-testable and reusable +— ``SlayerQueryEngine._expand_join_graph`` delegates its directed +reachability here. +""" + +from __future__ import annotations + +from collections import deque + +from slayer.core.models import SlayerModel + + +class JoinGraph: + """Directed graph over model names built from stored outgoing joins.""" + + def __init__(self, adjacency: dict[str, set[str]]) -> None: + # Every referenced node (source or target present in the model set) + # is a key; targets not in the node set are dropped at build time. + self._adj: dict[str, set[str]] = adjacency + + @classmethod + def build_from_models(cls, models: list[SlayerModel]) -> "JoinGraph": + """Build from a single datasource's models. Node keys are model + names (unique within a datasource); edges point ``source → + join.target_model`` for every stored join whose target is also in + the given model set (edges to unknown targets are skipped). + """ + names = {m.name for m in models} + adj: dict[str, set[str]] = {name: set() for name in names} + for m in models: + for j in m.joins: + if j.target_model in names: + adj[m.name].add(j.target_model) + return cls(adj) + + def reachable_from(self, root: str) -> set[str]: + """Set of nodes reachable from ``root`` (including ``root``) by + following directed edges. Visited-guarded for cyclic graphs.""" + seen: set[str] = {root} + frontier: deque[str] = deque([root]) + while frontier: + node = frontier.popleft() + for nbr in self._adj.get(node, ()): # noqa: SIM118 — .get default + if nbr not in seen: + seen.add(nbr) + frontier.append(nbr) + return seen + + def shortest_path(self, root: str, target: str) -> list[str] | None: + """Return the hop-name sequence from ``root`` to ``target`` + (excluding ``root``), or ``None`` if unreachable. + + ``[]`` when ``root == target``. Among all minimal-distance paths, + the lexicographically-smallest hop-name sequence is returned so + diamond graphs resolve deterministically. Distances are computed + first (BFS); the lexicographically-smallest path at each node's + minimal distance is then propagated layer by layer. + """ + if root == target: + return [] + if target not in self._adj and root not in self._adj: + return None + + # BFS layer assignment: dist[node] = min hop count from root. + dist: dict[str, int] = {root: 0} + frontier: deque[str] = deque([root]) + while frontier: + node = frontier.popleft() + for nbr in sorted(self._adj.get(node, ())): + if nbr not in dist: + dist[nbr] = dist[node] + 1 + frontier.append(nbr) + if target not in dist: + return None + + # Propagate the lexicographically-smallest hop sequence per node, + # in order of increasing distance. best[v] = min over predecessors + # u at dist-1 of (best[u] + [v]); since [v] is fixed for v, this + # reduces to min(best[u]) + [v]. + best: dict[str, list[str]] = {root: []} + nodes_by_dist: dict[int, list[str]] = {} + for node, d in dist.items(): + nodes_by_dist.setdefault(d, []).append(node) + for d in range(1, dist[target] + 1): + for v in nodes_by_dist.get(d, []): + preds = [ + u for u in dist + if dist[u] == d - 1 and v in self._adj.get(u, ()) + ] + best[v] = min(best[u] for u in preds) + [v] + return best[target] + + +def min_hops_root( + graph: "JoinGraph", candidates: list[str], mentioned: set[str] +) -> str | None: + """Pick the root that reaches every ``mentioned`` model over ``graph``. + + Shared selection core (DEV-1626 / DEV-1643): among ``candidates`` that reach + all mentioned models, minimize total hops summed over the mentioned set, + prefer a mentioned candidate on ties, then the lexicographically smallest + name. Returns ``None`` when no candidate reaches every mentioned model. An + empty ``mentioned`` set makes every candidate trivially valid (0 hops), so + the lexicographically smallest candidate is returned. + """ + def total_hops(root: str) -> int: + return sum(len(graph.shortest_path(root, m) or []) for m in mentioned) + + def reaches_all(root: str) -> bool: + return all(graph.shortest_path(root, m) is not None for m in mentioned) + + valid = [c for c in candidates if reaches_all(c)] + if not valid: + return None + return min(valid, key=lambda n: (total_hops(n), 0 if n in mentioned else 1, n)) diff --git a/slayer/engine/planned.py b/slayer/engine/planned.py index 5f329396..a55b48a6 100644 --- a/slayer/engine/planned.py +++ b/slayer/engine/planned.py @@ -68,6 +68,7 @@ "SlotId", "TransformLayer", "ValueSlot", + "WindowedAggregatePlan", ] @@ -242,6 +243,81 @@ class CrossModelAggregatePlan(BaseModel): cte_root_model: Optional[str] = None +# --------------------------------------------------------------------------- +# WindowedAggregatePlan +# --------------------------------------------------------------------------- + + +class WindowedAggregatePlan(BaseModel): + """Plan for one duration-windowed aggregate slot (DEV-1714 Stage 10). + + A windowed measure (``revenue:sum(window='90d')``) is a trailing rolling + aggregate: for each output bucket, SLayer sums source rows in the trailing + ``window`` interval ending at that bucket's end. It renders as a HOST-ROOTED + ``_wm___`` CTE — an inner ``_src`` self-join subquery joined + to ``_base`` on the query grain with an ``INTERVAL`` range predicate — then + LEFT-JOINed back to ``_base`` on the shared grain (same join-back machinery + as the cross-model ``_cm_*`` CTEs, so adding a windowed measure never + changes host cardinality). + + The renderer looks up the aggregate ``ValueSlot`` (source column, ``agg``, + result ``type``, ``column_filter_key``) and the grain ``ValueSlot``s by id + from the owning ``PlannedQuery``; this plan carries the window duration, the + resolved window time-dimension slot + its granularity, the per-role grain + slot partition, and the WHERE-phase filter ids inherited into ``_src``. + + Frame bounds are excluded from ``where_filter_ids`` so the trailing window + reaches rows before the visible frame starts (DEV-1732). A filter that is + only PARTLY a frame bound (``created_at >= X and status = 'paid'``) stays in + ``where_filter_ids`` and gets a ``src_filter_rewrites`` entry carrying the + residual — the population half — which the renderer substitutes for the + host's predicate. + + Scope for Stage 10 is ``sum``/``avg`` local measures only; cross-model, + transform-combined, composite, hidden, and mixed-filter windowed shapes are + guarded loudly at plan time (DEV-1504 lifts those guards). + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + aggregate_slot_id: SlotId + agg: str + window_raw: str + window_parts: List[Tuple[int, str]] + window_time_dimension_slot_id: SlotId + window_granularity: str + dimension_slot_ids: List[SlotId] = Field(default_factory=list) + other_time_dimension_slot_ids: List[SlotId] = Field(default_factory=list) + grain_slot_ids: List[SlotId] = Field(default_factory=list) + where_filter_ids: List[BoundFilterId] = Field(default_factory=list) + src_filter_rewrites: List["SrcFilterRewrite"] = Field(default_factory=list) + public_alias: Optional[str] = None + hidden: bool = False + + +# --------------------------------------------------------------------------- +# SrcFilterRewrite — DEV-1732 +# --------------------------------------------------------------------------- + + +class SrcFilterRewrite(BaseModel): + """A ROW filter whose CTE-local form differs from the host's (DEV-1732). + + Emitted when a filter is only PARTLY a frame bound, so it must still apply + inside the CTE but with its frame-bound conjuncts removed: + ``created_at >= '2024-06-01' and status = 'paid'`` becomes ``status = + 'paid'``. + + A filter that is ENTIRELY a frame bound needs no rewrite — the planner just + leaves its id out of ``where_filter_ids``. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + filter_id: BoundFilterId + expression: BoundExpr + + # --------------------------------------------------------------------------- # TransformLayer # --------------------------------------------------------------------------- @@ -339,6 +415,7 @@ class PlannedQuery(BaseModel): row_slots: List[ValueSlot] = Field(default_factory=list) aggregate_slots: List[ValueSlot] = Field(default_factory=list) cross_model_aggregate_plans: List[CrossModelAggregatePlan] = Field(default_factory=list) + windowed_aggregate_plans: List["WindowedAggregatePlan"] = Field(default_factory=list) combined_expression_slots: List[ValueSlot] = Field(default_factory=list) transform_layers: List[TransformLayer] = Field(default_factory=list) filters_by_phase: List[FilterPhase] = Field(default_factory=list) @@ -363,9 +440,26 @@ class PlannedQuery(BaseModel): # generator builds a synthetic model from the upstream schema) and for a # plain single-model query (the generator uses ``bundle.source_model``). render_source_model: Optional[SlayerModel] = None + # DEV-1543 — pass-through of ``SlayerQuery.distinct_dimension_values``. When + # ``False`` the generator skips the dim-only dedup GROUP BY and emits raw + # rows for a measure-less dimension query. + distinct_dimension_values: bool = True + # DEV-1732 — raw column keys of this stage's NON-HIDDEN time dimensions: the + # set of columns on which an explicit relational bound counts as a FRAME + # bound rather than a population filter. Computed once here so the windowed + # ``_src`` path (planner) and the ``time_shift`` shifted-CTE path + # (generator) cannot drift apart. + # + # Hidden ``TimeTruncKey`` slots are excluded deliberately: they are not + # equality-joined into ``_src`` (``_build_windowed_plans`` skips them), so + # stripping a bound on one would leave that axis unconstrained. + frame_bound_columns: List[ValueKey] = Field(default_factory=list) # ``CrossModelAggregatePlan.rerooted_plan`` is a forward reference to # ``PlannedQuery`` (defined above only after the CMA plan). Resolve it now # that both classes exist (DEV-1450 stage 7b.15e, C1). CrossModelAggregatePlan.model_rebuild() +# ``WindowedAggregatePlan.src_filter_rewrites`` forward-references +# ``SrcFilterRewrite``, declared just after it (DEV-1732). +WindowedAggregatePlan.model_rebuild() diff --git a/slayer/engine/planning.py b/slayer/engine/planning.py index b909c7af..604996d9 100644 --- a/slayer/engine/planning.py +++ b/slayer/engine/planning.py @@ -25,7 +25,7 @@ from __future__ import annotations -from typing import Dict, FrozenSet, List, Optional +from typing import Callable, Dict, FrozenSet, List, Optional from pydantic import BaseModel, ConfigDict, Field @@ -54,6 +54,7 @@ column_path, normalize_scalar, ) +from slayer.core.formula import RANK_FAMILY_TRANSFORMS from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name from slayer.engine.binding import BoundExpr, BoundFilter from slayer.engine.planned import SlotId, ValueSlot @@ -127,38 +128,76 @@ def __init__( self._by_key: Dict[ValueKey, SlotId] = {} self._declared_names: Dict[str, SlotId] = {} self._counter = 0 + # DEV-1733: every alias name already spoken for — user-declared public + # names (reserved up front, see ``reserve_public_names``) plus the + # resolved ``declared_name`` of each interned slot. HIDDEN slots are + # uniquified against this set at intern time so no renderer can emit + # two different expressions under one alias. + self._taken_names: set = set() def _next_id(self) -> SlotId: self._counter += 1 return f"s{self._counter}" - def intern( + def reserve_public_names(self, names) -> None: + """Claim user-declared names BEFORE any hidden slot is interned. + + DEV-1733: hidden-name uniquification must not depend on intern order. + Measures are interned public-slot-then-hidden-deps, so measure *i*'s + hidden dependency would otherwise be able to claim a name that measure + *j > i* later declares publicly — and public names are never renamed, + so the collision would come straight back. Reserving every declared + name first makes the outcome order-independent. + """ + for name in names: + if name: + self._taken_names.add(name) + + def _unique_hidden_name(self, declared_name: str) -> str: + """``declared_name``, suffixed ``_2`` / ``_3`` / … if already taken. + + Hidden canonical names are structural, not user-facing + (``_cumsum_inner``, ``_arith_/``, ``_scalar_abs``), so two distinct + keys of the same shape collide by construction: + ``cumsum(a:sum) + cumsum(b:sum)`` interned two hidden slots both named + ``_cumsum_inner``, the step CTE projected two columns under that one + alias, and the composite silently evaluated ``cumsum(a) + cumsum(a)``. + DEV-1692 fixed this inside the ``time_shift`` / ``consecutive_periods`` + emitters only; owning it here covers every renderer at once. + """ + if declared_name not in self._taken_names: + return declared_name + n = 2 + while f"{declared_name}_{n}" in self._taken_names: + n += 1 + return f"{declared_name}_{n}" + + def _validate_alias_collisions( self, *, key: ValueKey, declared_name: str, - phase: Phase, - public_name: Optional[str] = None, - canonical_alias: Optional[str] = None, - hidden: bool = False, - label: Optional[str] = None, - type: Optional[DataType] = None, - expression: Optional["BoundExpr"] = None, - format: Optional[NumberFormat] = None, - description: Optional[str] = None, - ) -> SlotId: - # Alias-collision validations (P4 / DEV-1443). - # Exemption: a dimension whose public name IS its own column - # name (``ColumnKey(path=(), leaf=X)`` declared as ``X``) is the - # column, not a rename of it — collision check skipped. Same - # exemption for a local ``TimeTruncKey`` over that same column - # since a time dimension on ``created_at`` projects the - # (truncated) ``created_at`` column rather than introducing a - # new alias. DEV-1450 stage 7b.13: also exempt - # ``ColumnSqlKey(model=..., column_name=X)`` declared as ``X`` - # -- a derived column (``Column.sql`` set) selected as a - # dimension projects the column unchanged, identical to the - # plain-column case. + public_name: Optional[str], + canonical_alias: Optional[str], + ) -> None: + """Alias-collision validations (P4 / DEV-1443). + + Split out of :meth:`intern` so the interning path reads as + validate → merge-or-create. Raises + ``MeasureNameCollidesWithColumnError`` when a public name shadows a + source column, and ``CanonicalAliasShadowsColumnError`` when a + renamed measure's canonical alias does. + + Exemption: a dimension whose public name IS its own column name + (``ColumnKey(path=(), leaf=X)`` declared as ``X``) is the column, not a + rename of it — collision check skipped. Same exemption for a local + ``TimeTruncKey`` over that same column, since a time dimension on + ``created_at`` projects the (truncated) ``created_at`` column rather + than introducing a new alias. DEV-1450 stage 7b.13: also exempt + ``ColumnSqlKey(model=..., column_name=X)`` declared as ``X`` — a + derived column (``Column.sql`` set) selected as a dimension projects + the column unchanged, identical to the plain-column case. + """ is_self_named_dimension = ( isinstance(key, ColumnKey) and key.path == () @@ -203,6 +242,28 @@ def intern( model=self._host_model_name, ) + def intern( + self, + *, + key: ValueKey, + declared_name: str, + phase: Phase, + public_name: Optional[str] = None, + canonical_alias: Optional[str] = None, + hidden: bool = False, + label: Optional[str] = None, + type: Optional[DataType] = None, + expression: Optional["BoundExpr"] = None, + format: Optional[NumberFormat] = None, + description: Optional[str] = None, + ) -> SlotId: + self._validate_alias_collisions( + key=key, + declared_name=declared_name, + public_name=public_name, + canonical_alias=canonical_alias, + ) + existing_sid = self._by_key.get(key) if existing_sid is not None: return self._merge_into_existing( @@ -229,6 +290,14 @@ def intern( ) sid = self._next_id() + # DEV-1733: uniquify HIDDEN slot names only. A public name is the + # user's result-key contract and is never rewritten — a genuine + # duplicate there already raised ``DuplicateMeasureNameError`` above. + if hidden: + declared_name = self._unique_hidden_name(declared_name) + self._taken_names.add(declared_name) + if public_name is not None: + self._taken_names.add(public_name) public_aliases = [public_name] if public_name is not None else [] slot = ValueSlot( id=sid, @@ -401,6 +470,63 @@ def lower_sugar_transforms(key: ValueKey) -> ValueKey: return key +def rewrite_rank_partition_keys( # NOSONAR(S3776) — sequential isinstance dispatch over the closed ValueKey union; each branch is the per-type identity-preserving rebuild contract, mirroring lower_sugar_transforms. Extracting per-type helpers would scatter the contract across the module. + key: ValueKey, *, rewrite_fn: Callable[[TransformKey], FrozenSet], +) -> ValueKey: + """Walk ``key``; for every rank-family ``TransformKey`` carrying an + explicit ``partition_by`` (non-empty ``partition_keys``), replace those + keys with ``rewrite_fn(transform_key)`` (DEV-1497). + + ``rewrite_fn`` receives the whole ``TransformKey`` and returns a new + ``frozenset`` of partition keys — validating that each resolves to a query + dimension / time-dimension and rewriting a time-dimension source column to + its ``TimeTruncKey`` bucket. It may raise ``ValueError`` for a partition + column that is not a query dimension. + + Identity-preserving (mirrors :func:`lower_sugar_transforms`): parents are + rebuilt only where a child changed, so this runs BEFORE interning without + churning unrelated slots. Reaches rank transforms nested in composite + measures (``ArithmeticKey`` / ``ScalarCallKey``) and in filter predicates + (comparisons are ``ArithmeticKey``). + """ + def _rec(k: ValueKey) -> ValueKey: + return rewrite_rank_partition_keys(key=k, rewrite_fn=rewrite_fn) + + if isinstance(key, TransformKey): + new_input = _rec(key.input) + new_pk = key.partition_keys + if key.op in RANK_FAMILY_TRANSFORMS and key.partition_keys: + new_pk = rewrite_fn(key) + if new_input is key.input and new_pk == key.partition_keys: + return key + return key.model_copy(update={"input": new_input, "partition_keys": new_pk}) + if isinstance(key, ArithmeticKey): + new_ops = tuple(_rec(op) for op in key.operands) + unchanged = all(a is b for a, b in zip(new_ops, key.operands)) + return key if unchanged else ArithmeticKey(op=key.op, operands=new_ops) + if isinstance(key, ScalarCallKey): + rewritable = _SLOTTABLE_KIND + (ArithmeticKey, ScalarCallKey, BetweenKey) + new_args = tuple( + _rec(a) if isinstance(a, rewritable) else a for a in key.args + ) + unchanged = all(a is b for a, b in zip(new_args, key.args)) + return key if unchanged else ScalarCallKey(name=key.name, args=new_args) + if isinstance(key, BetweenKey): + new_col, new_low, new_high = _rec(key.column), _rec(key.low), _rec(key.high) + unchanged = ( + new_col is key.column and new_low is key.low and new_high is key.high + ) + return key if unchanged else BetweenKey( + column=new_col, low=new_low, high=new_high, + ) + if isinstance(key, InKey): + new_col = _rec(key.column) + return key if new_col is key.column else InKey( + column=new_col, values=key.values, negated=key.negated, + ) + return key + + def desugar_change_pct(key: TransformKey) -> ArithmeticKey: """``change_pct(x)`` → ``(x - time_shift(x, periods=-1)) / NULLIF(time_shift(x, periods=-1), 0)``. @@ -554,6 +680,23 @@ class ProjectionPlanner: """Allocate slots for declared measures + hidden slots for refs only used in order/filter.""" + @staticmethod + def _intern_hidden(registry: "ValueRegistry", key: ValueKey) -> None: + """Intern ``key`` as a hidden slot unless it already has one. + + The single rule every hidden-slot site shares — measure aux deps, + filter operands, order operands, and (DEV-1733) an order target that + is itself a composite. Keeping it in one place is what stops the four + call sites drifting on ``declared_name`` / ``phase``. + """ + if registry.find_by_key(key) is None: + registry.intern( + key=key, + declared_name=_canonical_name(key), + hidden=True, + phase=key.phase, + ) + def plan( self, *, @@ -567,6 +710,15 @@ def plan( source_column_names=source_column_names, host_model_name=host_model_name, ) + # DEV-1733: claim every user-declared name before the first intern, so + # hidden-name uniquification is independent of intern order (a hidden + # dependency of measure 1 must not be able to take a name measure 2 + # declares publicly — public names are never renamed). + registry.reserve_public_names( + name + for m in measures + for name in (m.declared_name, m.public_name, m.canonical_alias) + ) public_projection: List[SlotId] = [] for m in measures: sid = registry.intern( @@ -587,38 +739,33 @@ def plan( # rendered by the generator into the inner SELECT but not # surfaced in the public projection. for dep in _iter_slot_deps(m.bound.value_key): - if dep == m.bound.value_key: - continue - if registry.find_by_key(dep) is None: - registry.intern( - key=dep, - declared_name=_canonical_name(dep), - hidden=True, - phase=dep.phase, - ) + if dep != m.bound.value_key: + self._intern_hidden(registry, dep) # Filter and order share the same dependency-selection rule: walk # the bound expression, intern each slot-worthy key as a hidden # slot if not already present. for f in filters: for dep in _iter_slot_deps(f.value_key): - if registry.find_by_key(dep) is None: - registry.intern( - key=dep, - declared_name=_canonical_name(dep), - hidden=True, - phase=dep.phase, - ) + self._intern_hidden(registry, dep) for o in order: for dep in _iter_slot_deps(o.bound.value_key): - if registry.find_by_key(dep) is None: - registry.intern( - key=dep, - declared_name=_canonical_name(dep), - hidden=True, - phase=dep.phase, - ) + self._intern_hidden(registry, dep) + # DEV-1733: ``_iter_slot_deps`` yields a composite's OPERANDS but + # never the composite itself (the generator normally inlines those + # nodes). An ORDER BY target that IS a composite therefore had no + # slot of its own, so ``plan_query``'s ``find_by_key`` lookup + # returned None and the order entry was SILENTLY DROPPED — the + # ``change`` / ``change_pct`` / scalar-call ORDER BY bug. Intern the + # top-level key so it gets a hidden slot the generator can + # materialise and the outer wrap can order on. + # + # ORDER ONLY: filters keep the operands-only walk. A filter's + # top-level composite is rendered inline into WHERE / HAVING, and + # giving it a slot would change that emission. + if isinstance(o.bound.value_key, (ArithmeticKey, ScalarCallKey)): + self._intern_hidden(registry, o.bound.value_key) return ProjectionPlan( registry=registry, diff --git a/slayer/engine/profiling.py b/slayer/engine/profiling.py index 752407d5..2e5f908c 100644 --- a/slayer/engine/profiling.py +++ b/slayer/engine/profiling.py @@ -27,24 +27,46 @@ - Categorical query orders by per-value count desc (alphabetical tie-break in SQL) so the persisted top-N is "most common values first". - New ``Column.sampled_values: Optional[List[str]]`` carries the top-50 - list verbatim (no ambiguous text split). Stays ``None`` for overflow >50 - and for numeric/temporal columns. -- New ``Column.distinct_count: Optional[int]`` carries the true total - cardinality; the overflow branch fires a second ``count_distinct`` query - via a transient ``ModelExtension`` (bypassing ``Column.allowed_aggregations`` - and ``Column.filter``). + list verbatim (no ambiguous text split). For categorical columns it is + populated on ≤50 distinct AND on overflow (the top-50 is kept). It stays + ``None`` only for numeric/temporal columns. +- New ``Column.distinct_count: Optional[int]`` carries the exact distinct + count when ≤50; on overflow it is ``None`` (see the single-scan note). - Text ``sampled`` format unchanged for ≤ 50 distinct (top-20 joined). For - overflow it becomes ``", ".join(top_20) + " ... (N distinct)"`` carrying - the true total — replacing the legacy ``"> 50 distinct"`` marker. + overflow it becomes ``", ".join(top_20) + " ... (50+ distinct)"`` — a + marker, not the exact total (see the single-scan note). - The internal ``_DimProfileEntry`` shape stays the same — overflow keeps ``values=None, distinct_count=None`` to signal "data omitted from the legacy entry". The richer DEV-1480 data only lives on ``ColumnSample`` produced by ``profile_column``. + +Single-scan overflow (team decision, 2026-07): profiling never fires a +secondary ``count_distinct`` query for the exact total on overflow — one +full-table scan per categorical column is enough. The top-50 is still +populated so ``_is_sample_cached`` marks the column cached (no re-scan on +every read); ``distinct_count`` stays ``None`` on overflow. Sample +profiling is also NOT run at ingest time — it is lazy, populated on the +first ``inspect`` of a column (or explicitly via ``refresh-samples``). + +DEV-1516 additions: +- :func:`ensure_column_sample_fresh` — shared cache-aware refresh helper + used by ``inspect_model``'s categorical loop, the search service's + post-fusion column-hit hook, and (DEV-1615) the single-entity ``inspect`` + point-lookup. Returns the input column on cache hit / failure, and an + in-memory refreshed copy on success (after persisting via storage). + +DEV-1615 change: +- :func:`ensure_column_sample_fresh` back-fills BOTH categorical (top-50 + + distinct_count) AND numeric/temporal (min/max range) uncached columns — + the prior categorical-only early-return was removed. Cached columns still + short-circuit at :func:`_is_sample_cached` (zero added cost), so the + common already-profiled case pays nothing. """ from __future__ import annotations -from typing import Any, Dict, List, NamedTuple, Optional, Set, Tuple +import logging +from typing import Any, NamedTuple from slayer.core.enums import DataType from slayer.core.models import Column, SlayerModel @@ -53,6 +75,9 @@ from slayer.storage.base import StorageBackend +logger = logging.getLogger(__name__) + + # --------------------------------------------------------------------------- # DEV-1480: categorical cap and public-ish result type # --------------------------------------------------------------------------- @@ -74,9 +99,9 @@ class ColumnSample(NamedTuple): ``None`` for numeric/temporal columns. """ - sampled: Optional[str] - sampled_values: Optional[List[str]] - distinct_count: Optional[int] + sampled: str | None + sampled_values: list[str] | None + distinct_count: int | None # --------------------------------------------------------------------------- @@ -95,10 +120,10 @@ class _DimProfileEntry(NamedTuple): name: str type_str: str - distinct_count: Optional[int] - values: Optional[List[Any]] - min_value: Optional[Any] - max_value: Optional[Any] + distinct_count: int | None + values: list[Any] | None + min_value: Any | None + max_value: Any | None def _format_dim_profile_value(entry: _DimProfileEntry) -> str: @@ -135,7 +160,7 @@ async def _profile_categorical_column( column: Column, engine: SlayerQueryEngine, max_values: int, -) -> Optional[_DimProfileEntry]: +) -> _DimProfileEntry | None: """Profile one string/boolean column. DEV-1480: orders by per-value count desc with alphabetical tie-break in @@ -145,8 +170,9 @@ async def _profile_categorical_column( Returns ``None`` when the column query fails — caller skips the column. The returned entry uses the legacy shape (``values=None, distinct_count=None`` - signals overflow); DEV-1480's true-total ``distinct_count`` is filled - by ``profile_column``. + signals overflow). The structured top-50 + ``distinct_count`` live on the + ``ColumnSample`` produced by ``profile_column`` (which routes categorical + columns through ``_profile_categorical_with_total``, not this entry path). """ try: q = SlayerQuery.model_validate({ @@ -165,7 +191,7 @@ async def _profile_categorical_column( value_key = f"{model.name}.{column.name}" # Filter NULL values out — they map to ``col IS NULL`` predicates, not # to literal-equality use cases the validator cares about. - raw_pairs: List[Tuple[Any, Any]] = [] + raw_pairs: list[tuple[Any, Any]] = [] count_key = f"{model.name}._count" for row in r.data: v = row.get(value_key) @@ -177,7 +203,7 @@ async def _profile_categorical_column( # equally-ranked rows in arbitrary order. NB: this only re-orders what # we received — the LIMIT cutoff is the SQL's responsibility. raw_pairs.sort(key=lambda p: (-(p[1] or 0), str(p[0]))) - values: List[str] = [str(v) for v, _ in raw_pairs] + values: list[str] = [str(v) for v, _ in raw_pairs] overflow = len(values) > max_values return _DimProfileEntry( name=column.name, @@ -192,9 +218,9 @@ async def _profile_categorical_column( async def _profile_numeric_temporal_columns( *, model: SlayerModel, - columns: List[Column], + columns: list[Column], engine: SlayerQueryEngine, -) -> Dict[str, _DimProfileEntry]: +) -> dict[str, _DimProfileEntry]: """Profile every numeric/temporal column in a single batched min/max query.""" if not columns: return {} @@ -208,11 +234,11 @@ async def _profile_numeric_temporal_columns( {"name": f"_slayer_range_{c.name}", "sql": c.sql if c.sql else c.name} for c in columns ] - measures_payload: List[Dict[str, str]] = [] + measures_payload: list[dict[str, str]] = [] for c in columns: measures_payload.append({"formula": f"_slayer_range_{c.name}:min"}) measures_payload.append({"formula": f"_slayer_range_{c.name}:max"}) - row: Dict[str, Any] = {} + row: dict[str, Any] = {} try: q = SlayerQuery.model_validate({ "source_model": {"source_name": model.name, "columns": ext_columns}, @@ -223,7 +249,7 @@ async def _profile_numeric_temporal_columns( row = r.data[0] except Exception: row = {} - out: Dict[str, _DimProfileEntry] = {} + out: dict[str, _DimProfileEntry] = {} for c in columns: mn = row.get(f"{model.name}._slayer_range_{c.name}_min") mx = row.get(f"{model.name}._slayer_range_{c.name}_max") @@ -246,8 +272,8 @@ async def _collect_dim_profile( engine: SlayerQueryEngine, max_values: int = _MAX_CATEGORICAL_VALUES, max_dims: int = 10, - only_columns: Optional[Set[str]] = None, -) -> List[_DimProfileEntry]: + only_columns: set[str] | None = None, +) -> list[_DimProfileEntry]: """Produce one profile entry per eligible column (non-hidden, non-pk). - string/boolean columns: distinct values (or overflow marker) via one @@ -262,9 +288,10 @@ async def _collect_dim_profile( with the set, so callers can profile a single column cheaply. DEV-1480: ``max_values`` defaults to 50 (was 20). Callers that need - the structured top-50 + true total should use :func:`profile_column` - per column, which fires the secondary ``count_distinct`` query on - overflow and returns a :class:`ColumnSample`. + the structured top-50 list should use :func:`profile_column` per + column, which returns a :class:`ColumnSample`. On overflow that path + keeps the top-50 and reports ``distinct_count=None`` — one scan only, + no secondary ``count_distinct`` query for the exact total. """ eligible = [ c for c in model.columns @@ -277,7 +304,7 @@ async def _collect_dim_profile( if c.type in (DataType.INT, DataType.DOUBLE, DataType.DATE, DataType.TIMESTAMP) ] - entries: Dict[str, _DimProfileEntry] = {} + entries: dict[str, _DimProfileEntry] = {} for c in categorical: entry = await _profile_categorical_column( model=model, column=c, engine=engine, max_values=max_values, @@ -318,6 +345,11 @@ def _is_sample_cached(column: Column) -> bool: """ if column.hidden or column.primary_key: return True + if column.type.is_opaque: + # Opaque columns are never profiled (DISTINCT / min / max fail on the + # underlying DB type), so report them as "cached" — same convention as + # hidden / PK columns — and keep callers from re-querying every read. + return True if column.type in _CATEGORICAL_TYPES: return column.sampled_values is not None return column.sampled is not None @@ -337,64 +369,18 @@ def _is_table_backed(model: SlayerModel) -> bool: return bool(model.sql_table) and not model.sql and not model.source_queries -async def _count_distinct_via_model_extension( - *, - model: SlayerModel, - column: Column, - engine: SlayerQueryEngine, -) -> Optional[int]: - """Fire a secondary ``count_distinct`` query via a transient - ``ModelExtension`` column. - - Bypasses both ``Column.allowed_aggregations`` (which might omit - ``count_distinct``) and ``Column.filter`` (which would otherwise apply - a CASE-WHEN at aggregation time and under-count). Mirrors the existing - ``_profile_numeric_temporal_columns`` pattern. - - Returns ``None`` when the query fails. - """ - try: - ext_q = SlayerQuery.model_validate({ - "source_model": { - "source_name": model.name, - "columns": [{ - "name": "_slayer_distinct_probe", - "sql": column.sql if column.sql else column.name, - "type": str(column.type), - }], - }, - "measures": [{ - "formula": "_slayer_distinct_probe:count_distinct", - }], - }) - r = await engine.execute(query=ext_q, data_source=model.data_source or None) - except Exception: # NOSONAR(S112) — best-effort: see module docstring - return None - if not r.data: - return None - raw = r.data[0].get( - f"{model.name}._slayer_distinct_probe_count_distinct", - ) - if raw is None: - return None - try: - return int(raw) - except (TypeError, ValueError): - return None - - async def _profile_categorical_with_total( *, model: SlayerModel, column: Column, engine: SlayerQueryEngine, -) -> Optional[ColumnSample]: - """DEV-1480 categorical profile: top-50 by frequency + true total on - overflow. +) -> ColumnSample | None: + """DEV-1480 categorical profile: top-50 values by frequency in a SINGLE + full-table scan. - Re-runs the query without the post-overflow path of - ``_profile_categorical_column`` so we can keep the top-50 list when - overflow is detected (the legacy entry shape would have discarded it). + On overflow (> 50 distinct) we keep the top-50 and report the total as + unknown rather than firing a second ``count_distinct`` scan — one scan + is enough; the exact distinct count isn't worth a second full scan. """ # Run the top-values query directly (instead of going through # ``_profile_categorical_column``) so we retain the values list even @@ -415,14 +401,14 @@ async def _profile_categorical_with_total( return None value_key = f"{model.name}.{column.name}" count_key = f"{model.name}._count" - raw_pairs: List[Tuple[Any, Any]] = [] + raw_pairs: list[tuple[Any, Any]] = [] for row in r.data: v = row.get(value_key) if v is None: continue raw_pairs.append((v, row.get(count_key))) raw_pairs.sort(key=lambda p: (-(p[1] or 0), str(p[0]))) - values: List[str] = [str(v) for v, _ in raw_pairs] + values: list[str] = [str(v) for v, _ in raw_pairs] overflow = len(values) > _MAX_CATEGORICAL_VALUES if not overflow: text = ", ".join(values[:_TEXT_SAMPLE_CAP]) @@ -431,29 +417,18 @@ async def _profile_categorical_with_total( sampled_values=values, distinct_count=len(values), ) - # Overflow: fire the secondary count_distinct query for the true total. - total = await _count_distinct_via_model_extension( - model=model, column=column, engine=engine, - ) + # Overflow (> _MAX_CATEGORICAL_VALUES distinct). We deliberately do NOT + # fire a secondary count_distinct query for the exact total — one full + # scan is enough. Keep the top-50 and report the total as unknown + # (``distinct_count=None``, sampled text carries a "50+" marker). The + # top-50 is still populated so ``_is_sample_cached`` marks the column + # cached and we don't re-scan on every read. top_50 = values[:_MAX_CATEGORICAL_VALUES] top_20_text = ", ".join(top_50[:_TEXT_SAMPLE_CAP]) - if total is None: - # Defensive: count_distinct query failed (transient backend error, - # missing permission, etc.). Persist ``sampled_values=None`` rather - # than the top-50 list so ``_is_sample_cached`` classifies the - # column as a cache miss and the next ``inspect_model`` / - # ``refresh-samples`` call retries the secondary query. Persisting - # the top-50 here would mark the column "cached" forever despite - # ``distinct_count`` being permanently None. - return ColumnSample( - sampled=f"> {_MAX_CATEGORICAL_VALUES} distinct", - sampled_values=None, - distinct_count=None, - ) return ColumnSample( - sampled=f"{top_20_text} ... ({total} distinct)", + sampled=f"{top_20_text} ... ({_MAX_CATEGORICAL_VALUES}+ distinct)", sampled_values=top_50, - distinct_count=total, + distinct_count=None, ) @@ -462,12 +437,12 @@ async def profile_column( model: SlayerModel, column: Column, engine: SlayerQueryEngine, -) -> Optional[ColumnSample]: +) -> ColumnSample | None: """Return the :class:`ColumnSample` for ``column`` on ``model``. - Returns ``None`` for primary-key / hidden columns and when the - profile query fails or yields no data. Caller decides whether to - persist the ``None`` (clearing any stale value) or skip it. + Returns ``None`` for primary-key / hidden / opaque (``UNKNOWN``) columns + and when the profile query fails or yields no data. Caller decides whether + to persist the ``None`` (clearing any stale value) or skip it. DEV-1480: signature widened from ``Optional[str]`` to ``Optional[ColumnSample]`` so the structured ``sampled_values`` and @@ -475,6 +450,11 @@ async def profile_column( """ if column.hidden or column.primary_key: return None + if column.type.is_opaque: + # No equality operator / no orderable comparison on the underlying DB + # type — both the categorical top-values scan and the batched min/max + # query would raise. Skip sample-value profiling entirely. + return None if column.type in _CATEGORICAL_TYPES: return await _profile_categorical_with_total( model=model, column=column, engine=engine, @@ -502,14 +482,14 @@ async def _refresh_one_column( column: Column, engine: SlayerQueryEngine, storage: StorageBackend, -) -> List[str]: +) -> list[str]: """Profile + persist a single column. Best-effort — returns the list of error strings produced (empty on full success). Extracted from ``refresh_table_backed_model_sampled`` to keep that function's cognitive complexity low. """ - errors: List[str] = [] - sample: Optional[ColumnSample] = None + errors: list[str] = [] + sample: ColumnSample | None = None try: sample = await profile_column(model=model, column=column, engine=engine) except Exception as exc: # NOSONAR(S112) — best-effort: see module docstring @@ -539,8 +519,8 @@ async def refresh_table_backed_model_sampled( model: SlayerModel, engine: SlayerQueryEngine, storage: StorageBackend, - only_columns: Optional[Set[str]] = None, -) -> List[str]: + only_columns: set[str] | None = None, +) -> list[str]: """Refresh ``Column.sampled``, ``Column.sampled_values``, and ``Column.distinct_count`` for each eligible column on ``model``. @@ -551,7 +531,7 @@ async def refresh_table_backed_model_sampled( """ if not _is_table_backed(model): return [] - errors: List[str] = [] + errors: list[str] = [] for column in model.columns: if column.hidden or column.primary_key: continue @@ -568,10 +548,10 @@ async def refresh_all_table_backed_sampled( engine: SlayerQueryEngine, storage: StorageBackend, data_source: str, -) -> List[str]: +) -> list[str]: """Refresh ``Column.sampled`` for every table-backed model in ``data_source``. Best-effort across all models.""" - errors: List[str] = [] + errors: list[str] = [] identities = await storage._list_all_model_identities() for ds, name in identities: if ds != data_source: @@ -593,9 +573,9 @@ async def handle_edit_refresh( storage: StorageBackend, data_source: str, model_name: str, - changed_columns: Set[str], + changed_columns: set[str], model_level_change: bool, -) -> List[str]: +) -> list[str]: """Refresh entry point for ``edit_model``. * ``model_level_change=True`` → refresh every non-hidden column on @@ -622,18 +602,123 @@ async def handle_edit_refresh( # match the new content_hash. reloaded = await storage.get_model(model_name, data_source=data_source) if reloaded is not None: - # Local import: keep embeddings off the cold-start path when the - # extra is not installed. - from slayer.embeddings.service import EmbeddingService - - try: - warnings.extend( - await EmbeddingService(storage=storage).refresh_model_subtree( - reloaded, - ) - ) - except Exception as exc: # noqa: BLE001 — best-effort - warnings.append( - f"{model_name}: embedding refresh failed: {exc}" + # DEV-1514: fan out through SearchService so every registered + # retriever sees the refresh. SearchService isolates per-retriever + # exceptions as prefixed warnings. + # Local import: keep the search module off the cold-start path. + from slayer.search.service import SearchService + + warnings.extend( + await SearchService(storage=storage).refresh_model_subtree( + reloaded, ) + ) return warnings + + +# --------------------------------------------------------------------------- +# DEV-1516: shared cache-aware refresh helper +# --------------------------------------------------------------------------- + + +async def ensure_column_sample_fresh( + *, + model: SlayerModel, + column: Column, + engine: SlayerQueryEngine, + storage: StorageBackend, +) -> Column: + """Best-effort refresh of a stale column's persisted sample. + + Used by ``inspect_model`` (categorical cache-miss path), + :class:`slayer.search.service.SearchService` (post-fusion column-hit + hook), and — DEV-1615 — the single-entity ``inspect`` point-lookup + (`slayer.inspect.service.InspectService`), so the "stale columns + auto-refresh on the spot" contract has a single source of truth. + + DEV-1615: back-fills BOTH categorical (top-50 + distinct_count) AND + numeric/temporal (min/max range) columns — :func:`profile_column` + already handles both kinds. The prior numeric/temporal early-return was + removed (see the inline note below). + + Returns the **input column unchanged** when: + + - ``_is_sample_cached(column)`` is True (cache hit; includes hidden / + primary-key / opaque ``UNKNOWN`` columns by convention), + - :func:`profile_column` returns ``None`` (e.g. transient query failure + or no rows), + - :func:`profile_column` raises (logged + swallowed), + - ``storage.update_column_sampled`` raises (logged + swallowed; the + in-memory refresh is still returned so the caller can render fresh + data this call). + + Returns a Pydantic ``model_copy``'d column with refreshed + ``sampled`` / ``sampled_values`` / ``distinct_count`` fields on + success (after persisting via storage). + + Logs ``WARNING`` on profile + persist failures with + ``(data_source, model_name, column_name)`` context so observability + matches the pre-DEV-1516 inline implementation in ``inspect_model``. + """ + if _is_sample_cached(column): + return column + # DEV-1615: no categorical-only gate here. ``profile_column`` handles + # BOTH categorical (top-50 + distinct_count) AND numeric/temporal + # (min/max range) columns, so an uncached column of either kind is + # back-filled. The previous early-return for numeric/temporal existed + # only so the search post-fusion hook would skip ranges; that skip was + # an assumption (numeric is reliably profiled at ingest), not a + # correctness requirement. ``inspect`` and ``search`` both now fill + # ranges on read. Already-profiled columns short-circuit above via + # ``_is_sample_cached`` so the common case stays free. + try: + sample = await profile_column( + model=model, column=column, engine=engine, + ) + except Exception as exc: # NOSONAR(S112) — best-effort: see module docstring + logger.warning( + "ensure_column_sample_fresh: failed to profile %s.%s.%s: %s", + model.data_source, model.name, column.name, exc, + ) + return column + if sample is None: + # No data to persist (e.g. PK / hidden / no rows). Helper short- + # circuits without writing — keeps cache predicate from flipping. + return column + if ( + sample.sampled_values is None + and sample.distinct_count is None + and column.sampled + ): + # Overflow-retry path failed to recover structured data: the + # ``ColumnSample`` carries only the generic ``"> 50 distinct"`` + # marker. The column already has a richer ``sampled`` text + # (e.g. v6 legacy ``"a, b, c ... (1234 distinct)"`` or a + # previous successful-overflow run). Skip the persist + return + # the input so the rich text survives — cache predicate still + # flags the column stale so the next call retries. + return column + try: + await storage.update_column_sampled( + data_source=model.data_source, + model_name=model.name, + column_name=column.name, + sampled=sample.sampled, + sampled_values=sample.sampled_values, + distinct_count=sample.distinct_count, + ) + except Exception as exc: # NOSONAR(S112) — best-effort: see module docstring + logger.warning( + "ensure_column_sample_fresh: failed to persist sample for " + "%s.%s.%s via update_column_sampled: %s", + model.data_source, model.name, column.name, exc, + ) + # Fall through: surface the in-memory refresh so the caller can + # still render fresh data this call. Next call will retry — the + # cache predicate still flags the column stale because the persist + # never landed. + return column.model_copy(update={ + "sampled": sample.sampled, + "sampled_values": sample.sampled_values, + "distinct_count": sample.distinct_count, + }) diff --git a/slayer/engine/query_engine.py b/slayer/engine/query_engine.py index f0f6725b..08f80766 100644 --- a/slayer/engine/query_engine.py +++ b/slayer/engine/query_engine.py @@ -3,15 +3,25 @@ Flow: SlayerQuery → _enrich() → EnrichedQuery → SQLGenerator → SQL → execute """ +import copy import decimal import logging +import re +from collections.abc import Callable from contextvars import ContextVar from typing import Any, Dict, List, Optional -from pydantic import BaseModel, Field as PydanticField, model_validator +import sqlalchemy as sa +from pydantic import ( + BaseModel, + ConfigDict as PydanticConfigDict, + Field as PydanticField, + model_validator, +) from slayer.core.enums import DEFAULT_AGGREGATIONS_BY_TYPE, DataType -from slayer.core.errors import AmbiguousModelError +from slayer.core.errors import AmbiguousModelError, ForcedFilterError +from slayer.core.policy import JoinFilterRuleset, SessionPolicy from slayer.core.format import NumberFormat, NumberFormatType, format_number from slayer.core.models import ( Column, @@ -22,12 +32,26 @@ SourceModelOrigin, ) from slayer.core.warnings import NormalizationWarning +from slayer.core.recommend import ( + CandidateCoverage, + ItemPath, + RootModelRecommendation, +) +from slayer.core.refs import split_agg_suffix from slayer.core.query import ( ColumnRef, SlayerQuery, TimeDimension, extract_placeholder_names, ) +from slayer.engine.cache import ( + CacheConfig, + QueryCache, + RefreshError, + RefreshKeyValue, + RefreshResult, + _CacheEntry, +) from slayer.engine.enriched import ( CrossModelMeasure, EnrichedMeasure, @@ -52,14 +76,143 @@ from slayer.engine.stage_ordering import topologically_order_stages from slayer.engine.stage_planner import plan_stages from slayer.engine.variables import apply_variables_to_query +from slayer.engine.introspect_utils import _safe_get_columns +from slayer.engine.join_graph import JoinGraph, min_hops_root +from slayer.memories.resolver import _all_models_in_datasource, resolve_entity +from slayer.sql import engine_factory from slayer.sql.client import SlayerSQLClient +from slayer.sql.engine_factory import _runtime_fingerprint +from slayer.sql.dialects import dialect_for_ds_type, get_dialect from slayer.sql.generator import SQLGenerator, generate_planned_stages +from slayer.sql.naming import flat_name +from slayer.sql.session_policy import ScopedTable, apply_session_policy from slayer.sql.stage_wrapper import build_flat_rename_wrapper from slayer.storage.base import StorageBackend logger = logging.getLogger(__name__) +# ------------------------------------------------------------------ +# recommend_root_model (DEV-1626) — ported from origin/main (DEV-1717) +# ------------------------------------------------------------------ +class _ResolvedItem(BaseModel): + """A recommend_root_model input item after resolution/validation.""" + + input_item: str + data_source: str + model: str + leaf: str + suffix: str | None = None + + +def _emit_recommend_path(graph: JoinGraph, root: str, item: "_ResolvedItem") -> str: + """Join-qualified path to ``item`` from ``root`` (root name excluded), + with the original aggregation suffix re-attached verbatim.""" + hops = graph.shortest_path(root, item.model) or [] + core = item.leaf if not hops else ".".join(hops) + "." + item.leaf + return core if item.suffix is None else f"{core}:{item.suffix}" + + +def _resolve_root_hint( + raw_hint: str | None, *, data_source: str, all_names: list[str] +) -> tuple[str, str] | None: + """Resolve a caller-supplied ``root_hint`` to a bare model name within + ``data_source`` (follow-up to DEV-1626). + + Returns ``(model, display)`` where ``model`` is the validated bare model + name used for graph logic and ``display`` is the caller's original string + (whitespace-trimmed) reused verbatim in diagnostics — so a + ``mydb.customers`` hint surfaces as ``mydb.customers`` in warnings, not the + resolved ``customers``. Returns ``None`` when the hint is empty / + whitespace-only (treated as "no hint" — a no-op). + + Raises ``ValueError`` for a non-existent / wrong-kind / cross-datasource / + otherwise malformed hint (the caller surfaces it loudly). + """ + if raw_hint is None: + return None + display = raw_hint.strip() + if not display: + return None + if "." in display: + segs = display.split(".") + if len(segs) == 2 and segs[0] == data_source: + model = segs[1] + else: + raise ValueError( + f"root_hint '{display}' must be a bare model name or " + f"'{data_source}.' within datasource '{data_source}'." + ) + else: + model = display + if model not in all_names: + raise ValueError( + f"root_hint '{display}' is not a model in datasource '{data_source}'." + ) + return model, display + + +def _build_recommend_coverage( + graph: JoinGraph, + all_names: list[str], + mentioned: set[str], + resolved: list["_ResolvedItem"], + *, + force_include: set[str] | None = None, +) -> list[CandidateCoverage]: + """Pareto frontier of partial-root candidates for the no-common-root + diagnostic. Item reachability ≡ owning-model reachability, so dominance + is computed on reached owning-model sets + per-model hop counts. + + ``force_include`` names models whose row must appear even when they reach + zero mentioned models or are Pareto-dominated (used to surface a caller's + ``root_hint``). Forced rows still sort by the same key, so a hint that is + genuinely on the frontier is not duplicated. + """ + forced = force_include or set() + items_in_order = [r.input_item for r in resolved] + model_of_item = {r.input_item: r.model for r in resolved} + + candidates: list[tuple[str, set[str], dict[str, int]]] = [] + for name in all_names: + reach = {m for m in mentioned if graph.shortest_path(name, m) is not None} + if not reach and name not in forced: + continue + hops = {m: len(graph.shortest_path(name, m) or []) for m in reach} + candidates.append((name, reach, hops)) + + def dominates(a: tuple, b: tuple) -> bool: + _an, ar, ah = a + _bn, br, bh = b + if ar > br: # strict superset → covers strictly more + return True + if ar == br: # same coverage, no path longer, at least one shorter + return all(ah[m] <= bh[m] for m in ar) and any(ah[m] < bh[m] for m in ar) + return False + + frontier = [ + c for c in candidates + if c[0] in forced + or not any(dominates(o, c) for o in candidates if o[0] != c[0]) + ] + + entries: list[tuple[CandidateCoverage, int]] = [] + for name, reach, hops in frontier: + reachable_items = [it for it in items_in_order if model_of_item[it] in reach] + unreachable_items = [it for it in items_in_order if model_of_item[it] not in reach] + entries.append(( + CandidateCoverage( + model_name=name, + reachable_items=reachable_items, + unreachable_items=unreachable_items, + ), + sum(hops.values()), + )) + entries.sort(key=lambda e: (-len(e[0].reachable_items), e[1], e[0].model_name)) + return [e[0] for e in entries] + + + # Per-task in-flight join-target names. Used by _resolve_join_target to break # loops when a query-backed target's own join graph references it back. Lives # in a ContextVar (not on the engine) so concurrent requests through the same @@ -82,30 +235,18 @@ ) -_EXPLAIN_PREFIX = { - "postgres": "EXPLAIN ANALYZE", - "redshift": "EXPLAIN", - "mysql": "EXPLAIN FORMAT=JSON", - "sqlite": "EXPLAIN QUERY PLAN", - "duckdb": "EXPLAIN ANALYZE", - "clickhouse": "EXPLAIN", - "snowflake": "EXPLAIN USING JSON", - "bigquery": None, # BigQuery doesn't support EXPLAIN via SQL - "trino": "EXPLAIN ANALYZE", - "presto": "EXPLAIN ANALYZE", - "databricks": "EXPLAIN EXTENDED", - "spark": "EXPLAIN EXTENDED", - "tsql": "SET SHOWPLAN_ALL ON;", # SQL Server: batch prefix, needs suffix too - "oracle": "EXPLAIN PLAN FOR", -} - +_PLACEHOLDER_FILL_VALUE = "0" -_EXPLAIN_POSTFIX = { - "tsql": "; SET SHOWPLAN_ALL OFF", -} +def _sql_client_cache_key(datasource: DatasourceConfig) -> tuple[str, str]: + """Cache key for ``SlayerQueryEngine._sql_clients``. -_PLACEHOLDER_FILL_VALUE = "0" + Mirrors ``engine_factory``'s cache key so two datasources differing + in (e.g.) Snowflake ``warehouse`` get distinct ``SlayerSQLClient`` + instances (and therefore distinct factory-cached engines with the + correct per-connection ``USE`` listener) — DEV-1551. + """ + return (datasource.get_connection_string(), _runtime_fingerprint(datasource)) def _merge_query_variables( @@ -139,14 +280,13 @@ def _apply_placeholder_fill( def _build_explain_sql(dialect: str, sql: str) -> str: - """Build a dialect-appropriate EXPLAIN statement.""" - prefix = _EXPLAIN_PREFIX.get(dialect) - if prefix is None: - raise ValueError( - f"EXPLAIN is not supported for dialect '{dialect}'. Use dry_run=True to inspect the generated SQL instead." - ) - suffix = _EXPLAIN_POSTFIX.get(dialect, "") - return f"{prefix} {sql}{suffix}" + """Build a dialect-appropriate EXPLAIN statement. + + DEV-1716: delegates to the dialect strategy's ``build_explain_sql`` hook + (raises ``ValueError`` for dialects without SQL-level EXPLAIN, e.g. + BigQuery) instead of an inline prefix/postfix map. + """ + return get_dialect(dialect).build_explain_sql(sql) class SlayerResponse(BaseModel): @@ -216,6 +356,38 @@ def _normalize_source_query_stages( return model.model_copy(update={"source_queries": new_stages}) +class _Prepared(BaseModel): + """DB-free product of ``_prepare_pipeline`` (DEV-1715). + + Everything the execute / cache-hook / evict / refresh paths need after + resolve→enrich→plan→SQL-gen→policy but BEFORE any SQL-client construction. + ``sql`` is the FINAL, policy-rewritten SQL that is actually executed, so a + cache key computed from it (``make_key(sql, ds_fingerprint)``) always + matches the executed statement. ``resolved_data_source`` is + ``datasource.name`` — the authoritative datasource used to run the query + (not ``model.data_source``, which is less authoritative for inline / + expanded query-backed models). + + The ds-key / ds-fingerprint are intentionally NOT eager fields: computing + them calls ``datasource.get_connection_string()``, which some dialects + (e.g. Snowflake) reject without credentials. A ``dry_run`` must still + render SQL for a credential-less datasource, so the fingerprint is computed + lazily via ``_ds_fingerprint`` only on the cache / execute paths. + """ + + model_config = PydanticConfigDict(arbitrary_types_allowed=True) + + sql: str + dialect: str + datasource: DatasourceConfig + resolved_data_source: Optional[str] = None + attributes: Any + expected_columns: List[str] + touched: set + model: SlayerModel + slack_warnings: List[Any] = PydanticField(default_factory=list) + + class SlayerQueryEngine: """Central orchestrator: resolves queries via storage, generates SQL, executes. @@ -224,9 +396,239 @@ class SlayerQueryEngine: SQLGenerator for SQL generation. """ - def __init__(self, storage: StorageBackend): + def __init__( + self, + storage: StorageBackend, + *, + policy: Optional[SessionPolicy] = None, + cache_config: Optional[CacheConfig] = None, + ): self.storage = storage - self._sql_clients: Dict[str, SlayerSQLClient] = {} # connection string → cached client + # DEV-1587 / DEV-1715: per-engine, in-memory, opt-in query result cache. + # ``cache_config`` defaults to an empty ``CacheConfig()`` (caches + # indefinitely with no auto-staleness). The cache lives on the engine + # instance, so two engines with different connection settings / policy + # keep separate caches. + self._cache = QueryCache(config=cache_config or CacheConfig()) + # Cache key: (connection_string, runtime_fingerprint) — matches + # ``engine_factory``'s cache so Snowflake datasources sharing a + # connection_name but differing in warehouse/role/database/schema get + # distinct clients (DEV-1551). + self._sql_clients: dict[tuple[str, str], SlayerSQLClient] = {} + # DEV-1578: immutable, engine-global forced-filter policy. When set, + # every generated SQL is rewritten to scope each physical table to the + # configured tenant before execution / dry-run / explain. + self.policy = policy + # Cache of confirmed column-presence facts keyed by + # (ds_key, catalog, schema, table, column). Only ``True``/``False`` are + # cached; an unconfirmable ``None`` is re-probed so a transient + # introspection failure self-heals once the datasource recovers. + self._column_presence_cache: dict[tuple, bool] = {} + # DEV-1627: cached ClickHouse ``(major, minor)`` server version per + # datasource, for the correlated-subquery join-rule gate. ``None`` (or a + # missing entry) fails closed. Populated by + # ``_preflight_clickhouse_correlated`` before the policy rewrite. + self._ch_version_cache: dict[tuple[str, str], tuple[int, int] | None] = {} + + # ---- query cache management (DEV-1587 / DEV-1715) ---------------------- + + @property + def cache_config(self) -> CacheConfig: + """The active :class:`CacheConfig` (read-only view of ``_cache``).""" + return self._cache.config + + @cache_config.setter + def cache_config(self, config: CacheConfig) -> None: + """Reassign the cache policy. This **clears the cache** — stale entries + must not survive under a new TTL / refresh-key set. The existing clock + is preserved so an injected test clock survives the reassignment.""" + self._cache = QueryCache(config=config, clock=self._cache._clock) + + @property + def cache_size(self) -> int: + """Number of live cache entries.""" + return self._cache.size() + + def clear_cache(self) -> None: + """Drop every cached entry.""" + self._cache.clear() + + def _apply_policy( + self, *, sql: str, dialect: str, datasource: DatasourceConfig + ) -> str: + """Rewrite ``sql`` to enforce the forced-filter policy, or return it + unchanged when no policy is configured (zero overhead).""" + if not self.policy: + return sql + return apply_session_policy( + sql, + dialect=dialect, + policy=self.policy, + has_column=lambda scoped, column: self._column_present( + datasource=datasource, scoped_table=scoped, column=column + ), + on_correlated_emitted=self._clickhouse_correlated_guard( + dialect=dialect, datasource=datasource + ), + ) + + def _policy_has_join_rules(self) -> bool: + return bool( + self.policy + and isinstance(self.policy.ruleset, JoinFilterRuleset) + and self.policy.ruleset.joins + ) + + @staticmethod + def _parse_clickhouse_version(raw: Any) -> tuple[int, int] | None: + """Parse a ClickHouse ``version()`` string to ``(major, minor)``. + + Tolerates a leading ``v``, extra patch/build segments, and prerelease + suffixes (``25.4.1-lts``). Returns ``None`` when the input is not a + string or has no leading ``.`` — so an unparseable version + fails closed at the guard. + """ + if not isinstance(raw, str): + return None + match = re.match(r"\s*v?(\d+)\.(\d+)", raw) + if not match: + return None + return (int(match.group(1)), int(match.group(2))) + + def _clickhouse_correlated_guard( + self, *, dialect: str, datasource: DatasourceConfig + ) -> Callable[[], None] | None: + """Return a sync guard invoked when the rewrite emits a correlated + ``EXISTS``, or ``None`` for non-ClickHouse dialects. + + The guard reads the cached server version and raises + :class:`ForcedFilterError` when it is unknown (missing/None) or + ``< (25, 4)`` (correlated subqueries unsupported); on a supported + version it logs a warning and returns. + """ + if dialect != "clickhouse": + return None + ds_key = _sql_client_cache_key(datasource) + + def guard() -> None: + version = self._ch_version_cache.get(ds_key) + if version is None: + raise ForcedFilterError( + "ClickHouse join-based forced filter needs a correlated " + "subquery (server >= 25.4), but the server version could " + "not be determined; failing closed." + ) + if version < (25, 4): + raise ForcedFilterError( + "ClickHouse join-based forced filter needs a correlated " + "subquery, which requires server >= 25.4; detected " + f"{version[0]}.{version[1]}; failing closed." + ) + logger.warning( + "Applying a join-based forced filter on ClickHouse via an " + "experimental correlated subquery " + "(allow_experimental_correlated_subqueries=1); requires " + "server >= 25.4 (detected %d.%d).", + version[0], + version[1], + ) + + return guard + + async def _preflight_clickhouse_correlated( + self, *, dialect: str, datasource: DatasourceConfig + ) -> None: + """Probe and cache the ClickHouse server version once per datasource + when the policy has join rules. No-op for non-ClickHouse dialects and + for policies without join rules. A probe failure caches ``None`` (fail + closed at the guard). Cheap: cached, and only fires when a join rule + could actually emit a correlated subquery.""" + if dialect != "clickhouse" or not self._policy_has_join_rules(): + return + ds_key = _sql_client_cache_key(datasource) + if ds_key in self._ch_version_cache: + return # already probed (value may be None) + try: + if ds_key not in self._sql_clients: + self._sql_clients[ds_key] = SlayerSQLClient(datasource=datasource) + client = self._sql_clients[ds_key] + rows = await client.execute("SELECT version()") + raw = None + if rows and isinstance(rows[0], dict): + raw = next(iter(rows[0].values()), None) + self._ch_version_cache[ds_key] = self._parse_clickhouse_version(raw) + except Exception as exc: + logger.warning( + "ClickHouse version preflight failed for datasource '%s'; " + "join-based forced filters will fail closed: %s", + datasource.name, + exc, + ) + self._ch_version_cache[ds_key] = None + + def _column_present( + self, + *, + datasource: DatasourceConfig, + scoped_table: ScopedTable, + column: str, + ) -> bool | None: + """Return whether ``column`` exists on ``scoped_table`` in + ``datasource``: ``True`` / ``False`` / ``None`` (cannot confirm). + + Introspects via the shared ``_safe_get_columns`` helper (Inspector + with INFORMATION_SCHEMA fallback). The schema is the table's parsed + qualifier, else the datasource default. Only confirmed ``True`` / + ``False`` results are cached; a ``None`` (any introspection error or + empty result) is returned uncached so a transient failure is + re-probed on the next query. + """ + schema = scoped_table.schema_name or datasource.schema_name + # Cross-catalog refs can't be confirmed: SQLAlchemy's column + # introspection takes no catalog argument, so a three-part + # ``catalog.schema.table`` naming a catalog other than the + # connection's own would probe the wrong relation. Fail closed + # (consistent with the unconfirmable-presence rule) rather than risk + # an under-filter under ``on_unapplicable="pass"``. Single-catalog + # refs (catalog matches the connection, or no catalog) probe normally. + if scoped_table.catalog and ( + not datasource.database + or scoped_table.catalog.casefold() != datasource.database.casefold() + ): + return None + # Include catalog in the key so two tables differing only by catalog + # (e.g. BigQuery project) never share a cached presence fact. + key = ( + _sql_client_cache_key(datasource), + scoped_table.catalog, + schema, + scoped_table.name, + column, + ) + if key in self._column_presence_cache: + return self._column_presence_cache[key] + try: + sa_engine = engine_factory.get_engine(datasource.resolve_env_vars()) + inspector = sa.inspect(sa_engine) + cols = _safe_get_columns( + inspector, sa_engine, scoped_table.name, schema + ) + except Exception as exc: # introspection failed -> cannot confirm + logger.warning( + "Forced filter: column-presence probe failed for %s.%s " + "(column %r): %s", + schema or "", + scoped_table.name, + column, + exc, + ) + return None + if not cols: + return None # no columns resolved -> cannot confirm + names = {str(c.get("name", "")).lower() for c in cols} + present = column.lower() in names + self._column_presence_cache[key] = present + return present def _get_join_target_resolving(self) -> set: """Return the per-task in-flight join-target name set, allocating one @@ -280,7 +682,20 @@ def _topologically_order_queries( """ return topologically_order_stages(queries) - async def execute( # NOSONAR S3776 — public dispatch over str/dict/list/SlayerQuery; splitting hides the input-shape contract + async def aclose(self) -> None: + """Dispose every cached client's async engine; keep the clients themselves. + + Per-instance async engines bind their asyncpg/aiomysql pool to the loop + that first opened a connection; closing that loop without disposing + leaks the server-side connections (asyncpg.Connection.close needs a + live loop). Clients are kept so ``_sync_engine`` survives — important + for ``:memory:`` SQLite, whose StaticPool pins the connection holding + all data. + """ + for client in self._sql_clients.values(): + await client.aclose() + + async def execute( self, query: "SlayerQuery | dict | list[SlayerQuery | dict] | str", variables: Optional[Dict[str, Any]] = None, @@ -288,21 +703,49 @@ async def execute( # NOSONAR S3776 — public dispatch over str/dict/list/Slaye dry_run: bool = False, explain: bool = False, data_source: Optional[str] = None, + cache: bool = False, ) -> SlayerResponse: runtime_kwarg = variables or {} + main_query, named_queries, prefer_data_source = await self._normalize_input( + query, runtime_kwarg=runtime_kwarg, prefer_data_source=data_source + ) + return await self._execute_pipeline( + query=main_query, + named_queries=named_queries, + runtime_kwarg=runtime_kwarg, + dry_run=dry_run, + explain=explain, + prefer_data_source=prefer_data_source, + cache=cache, + original_input=query, + original_data_source=data_source, + ) - # Run-by-name dispatch: ``execute("model_name", variables=...)`` runs - # the backing query of a query-backed model. + async def _normalize_input( # NOSONAR S3776 — public dispatch over str/dict/list/SlayerQuery; splitting hides the input-shape contract + self, + query: "SlayerQuery | dict | list[SlayerQuery | dict] | str", + *, + runtime_kwarg: Dict[str, Any], + prefer_data_source: Optional[str], + ) -> "tuple[SlayerQuery, Dict[str, SlayerQuery], Optional[str]]": + """Resolve the user input union into ``(main_query, named_queries, + prefer_data_source)`` shared by ``execute()``, ``evict()``, and + ``refresh()`` re-exec (DEV-1715). + + Async because the ``str`` (run-by-name) branch reads storage — which is + exactly what lets ``refresh()`` re-prep pick up ``source_queries`` + edits and pin the model lookup to the entry's originally-resolved + datasource (``prefer_data_source``). + """ + # Run-by-name dispatch: ``execute("model_name", ...)`` runs the backing + # query of a query-backed model. if isinstance(query, str): - return await self._execute_by_name( + return await self._normalize_by_name( name=query, runtime_kwarg=runtime_kwarg, - dry_run=dry_run, - explain=explain, - data_source=data_source, + prefer_data_source=prefer_data_source, ) - # Accept dicts and validate them into SlayerQuery objects if isinstance(query, list): if not query: @@ -315,40 +758,39 @@ async def execute( # NOSONAR S3776 — public dispatch over str/dict/list/Slaye # last entry stays last as the entry point. Validates names, # duplicates, self-refs, root-as-sink, and cycles up front. queries = self._topologically_order_queries(queries) - query = queries[-1] + main_query = queries[-1] named_queries = {q.name: q for q in queries[:-1] if q.name} else: if isinstance(query, dict): query = SlayerQuery.model_validate(query) + main_query = query named_queries = {} # Merge ``variables=`` kwarg into query.variables so filter # substitution and downstream resolution see the merged set. # ``runtime_kwarg`` always wins (per spec precedence). if runtime_kwarg: - merged_top = {**(query.variables or {}), **runtime_kwarg} - if merged_top != (query.variables or {}): - query = query.model_copy(update={"variables": merged_top}) + merged_top = {**(main_query.variables or {}), **runtime_kwarg} + if merged_top != (main_query.variables or {}): + main_query = main_query.model_copy(update={"variables": merged_top}) - return await self._execute_pipeline( - query=query, - named_queries=named_queries, - runtime_kwarg=runtime_kwarg, - dry_run=dry_run, - explain=explain, - prefer_data_source=data_source, - ) + return main_query, named_queries, prefer_data_source - async def _execute_by_name( + async def _normalize_by_name( self, + *, name: str, runtime_kwarg: Dict[str, Any], - dry_run: bool = False, - explain: bool = False, - data_source: Optional[str] = None, - ) -> SlayerResponse: - """Run the backing query of a query-backed model by name.""" - model = await self.storage.get_model(name, data_source=data_source) + prefer_data_source: Optional[str], + ) -> "tuple[SlayerQuery, Dict[str, SlayerQuery], Optional[str]]": + """Normalize a run-by-name input into the shared prepare tuple. + + ``prefer_data_source`` pins the ``storage.get_model`` lookup: on a + ``refresh()`` re-exec it is the entry's originally-resolved datasource, + so a datasource-priority flip after caching cannot re-read a different + query-backed model (Codex #3). + """ + model = await self.storage.get_model(name, data_source=prefer_data_source) if model is None: raise ValueError(f"Model '{name}' not found") if not model.source_queries: @@ -387,29 +829,29 @@ async def _execute_by_name( if merged != (main_query.variables or {}): main_query = main_query.model_copy(update={"variables": merged}) - return await self._execute_pipeline( - query=main_query, - named_queries=named_queries, - runtime_kwarg=runtime_kwarg, - dry_run=dry_run, - explain=explain, - prefer_data_source=model.data_source or data_source, - ) + return main_query, named_queries, model.data_source or prefer_data_source - async def _execute_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enrich→generate→execute); breaking it up obscures the order of operations + async def _prepare_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enrich→generate→policy); breaking it up obscures the order of operations self, query: SlayerQuery, named_queries: Dict[str, SlayerQuery], runtime_kwarg: Dict[str, Any], *, - dry_run: bool = False, - explain: bool = False, prefer_data_source: Optional[str] = None, - ) -> SlayerResponse: - """Shared pipeline used by both ``execute()`` and ``_execute_by_name()``. + override_datasource: Optional[DatasourceConfig] = None, + ) -> _Prepared: + """DB-free-ish prepare portion shared by execute / evict / refresh + (DEV-1715): resolve→enrich→normalize→plan→SQL-gen→ClickHouse-preflight→ + policy-rewrite→response-metadata. Produces the FINAL executed SQL and + the datasource fingerprint but constructs **no** SQL client on the + common (no-policy) path — so ``evict()`` recomputes a cache key without + connecting. When a policy IS configured, producing the correct SQL may + introspect column presence / preflight ClickHouse; that is inherent to + the rewrite. Assumes ``query.variables`` already reflects the resolved variable - context for the top of the chain (kwarg merged in by the caller). + context for the top of the chain (kwarg merged in by ``_normalize_ + input``). """ # Pre-processing: strip redundant source model name prefixes from all references query = query.strip_source_model_prefix() @@ -503,11 +945,25 @@ async def _execute_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enr planned_list = plan_stages(queries=stages, bundle=bundle) root_planned = planned_list[-1] - datasource = await self._resolve_datasource(model=model) + # ``override_datasource`` pins the connection identity (used by + # refresh() re-exec): the query is re-run against the EXACT datasource + # the entry was cached under (its ds_key), not whatever the model's + # ``data_source`` name resolves to now — so a same-name repoint can't + # migrate the entry or store rows from a different database. + datasource = override_datasource or await self._resolve_datasource(model=model) dialect = self._dialect_for_type(datasource.type) sql = generate_planned_stages( planned_list, bundle=bundle, dialect=dialect, ) + # DEV-1578: forced-filter (RLS) rewrite — scope each physical table to + # the configured tenant. Applied to the rendered SQL before dry-run / + # explain / execute so all three surfaces (and the cache key) see the + # policy-rewritten SQL. Zero overhead (returns unchanged) when no + # policy is configured. + await self._preflight_clickhouse_correlated( + dialect=dialect, datasource=datasource + ) + sql = self._apply_policy(sql=sql, dialect=dialect, datasource=datasource) logger.debug("Generated SQL:\n%s", sql) # Response metadata (attributes + expected_columns) from the typed @@ -526,46 +982,478 @@ async def _execute_pipeline( # NOSONAR S3776 — linear pipeline (resolve→enr original_source_model=original_source_model, ) - # dry_run: return SQL without executing - if dry_run: - return SlayerResponse( - data=[], columns=expected_columns, sql=sql, - attributes=attributes, warnings=slack_warnings, - ) + return _Prepared( + sql=sql, + dialect=dialect, + datasource=datasource, + resolved_data_source=datasource.name, + attributes=attributes, + expected_columns=list(expected_columns), + touched=touched, + model=model, + slack_warnings=slack_warnings, + ) - # Execute — reuse SQL client (and its connection pool) per datasource - ds_key = datasource.get_connection_string() + @staticmethod + def _ds_fingerprint(datasource: DatasourceConfig) -> str: + """The datasource identity fingerprint used in the cache key — + ``connection_string|runtime_fingerprint``. Computed lazily (only on the + cache / execute paths) because ``get_connection_string`` rejects some + credential-less dialects that a ``dry_run`` must still render.""" + return "|".join(_sql_client_cache_key(datasource)) + + def _client_for(self, datasource: DatasourceConfig) -> SlayerSQLClient: + """Reuse (or lazily construct) the cached SQL client for a datasource. + + Keyed by ``_sql_client_cache_key`` so two datasources differing only in + (e.g.) Snowflake warehouse get distinct clients (DEV-1551). + """ + ds_key = _sql_client_cache_key(datasource) if ds_key not in self._sql_clients: self._sql_clients[ds_key] = SlayerSQLClient(datasource=datasource) - client = self._sql_clients[ds_key] + return self._sql_clients[ds_key] + + async def _execute_pipeline( + self, + query: SlayerQuery, + named_queries: Dict[str, SlayerQuery], + runtime_kwarg: Dict[str, Any], + *, + dry_run: bool = False, + explain: bool = False, + prefer_data_source: Optional[str] = None, + cache: bool = False, + original_input: Any = None, + original_data_source: Optional[str] = None, + ) -> SlayerResponse: + """Prepare (DB-free-ish) then dry-run / explain / cache-hook / execute. - # explain: run dialect-appropriate EXPLAIN on the query + The cache hook sits at the one seam between ``_prepare_pipeline`` (which + produces the final policy-rewritten SQL + ds fingerprint) and SQL-client + construction: a hit skips only the DB execute + decode; a miss scans + refresh-key baselines BEFORE the data query and stores a deep copy. + """ + prepared = await self._prepare_pipeline( + query=query, + named_queries=named_queries, + runtime_kwarg=runtime_kwarg, + prefer_data_source=prefer_data_source, + ) + + # dry_run: return SQL without executing. NEVER cached. + if dry_run: + return SlayerResponse( + data=[], columns=prepared.expected_columns, sql=prepared.sql, + attributes=prepared.attributes, warnings=prepared.slack_warnings, + ) + + use_cache = cache and not dry_run and not explain + # Bind the cache instance ONCE for the whole cached path. A concurrent + # ``cache_config`` reassignment (the setter swaps in a fresh QueryCache) + # during the DB awaits below must not split the read (get) and write + # (put) across two caches — that would land an entry whose applicable / + # baselines were computed under the old refresh-key set into the new + # cache, defeating the "reassigning cache_config clears stale entries" + # contract on CacheConfig. + cache_obj = self._cache + key: Optional[str] = None + if use_cache: + key = QueryCache.make_key(prepared.sql, self._ds_fingerprint(prepared.datasource)) + entry = await cache_obj.get(key) + if entry is not None: + # Hit: return an independent deep copy so caller mutation can't + # poison the cached response. + return entry.response.model_copy(deep=True) + + # Miss (or cache=False) → a SQL client is required. + client = self._client_for(prepared.datasource) + + # explain: run dialect-appropriate EXPLAIN on the query. NEVER cached. if explain: - explain_sql = _build_explain_sql(dialect=dialect, sql=sql) + explain_sql = _build_explain_sql(dialect=prepared.dialect, sql=prepared.sql) try: rows = await client.execute(sql=explain_sql) except Exception as exc: await self._maybe_raise_schema_drift( - err=exc, model=model, touched_models=touched + err=exc, model=prepared.model, touched_models=prepared.touched ) raise return SlayerResponse( - data=rows, sql=sql, attributes=attributes, - warnings=slack_warnings, + data=rows, sql=prepared.sql, attributes=prepared.attributes, + warnings=prepared.slack_warnings, + ) + + # On a cache miss, capture refresh-key baselines BEFORE the data query + # (so the cached data reflects a state >= the baseline). A write-time + # baseline-scan failure PROPAGATES — nothing is stored. + applicable: list[tuple[str, str]] = [] + refresh_key_values: list[RefreshKeyValue] = [] + if use_cache: + applicable, refresh_key_values = await self._scan_refresh_key_baselines( + prepared=prepared, client=client, cache=cache_obj ) + rows = await self._run_data_query(prepared=prepared, client=client) + columns = prepared.expected_columns if not rows else [] # [] triggers auto-derive + response = SlayerResponse( + data=rows, columns=columns, sql=prepared.sql, + attributes=prepared.attributes, warnings=prepared.slack_warnings, + ) + + if use_cache: + entry = self._build_cache_entry( + prepared=prepared, + response=response, + original_input=original_input, + variables=runtime_kwarg, + data_source=original_data_source, + created_at=cache_obj.now(), + applicable=applicable, + refresh_key_values=refresh_key_values, + ) + await cache_obj.put(key, entry) + # Return the original response; the stored copy is a defensive deep copy. + return response + + async def _run_data_query( + self, *, prepared: _Prepared, client: SlayerSQLClient + ) -> "list[dict]": + """Run the prepared data query with schema-drift attribution on error. + + DEV-1716: applies the dialect's read-side ``decode_result_keys`` hook so + BigQuery / T-SQL alias-mangled result keys are reversed back to SLayer's + universal dotted shape (identity for every other dialect / on empty + rows). Shared by the execute miss path and the refresh() re-exec. + """ try: - rows = await client.execute(sql=sql) + rows = await client.execute(sql=prepared.sql) except Exception as exc: await self._maybe_raise_schema_drift( - err=exc, model=model, touched_models=touched + err=exc, model=prepared.model, touched_models=prepared.touched ) raise - columns = expected_columns if not rows else [] # fallback for empty results; [] triggers auto-derive - return SlayerResponse( - data=rows, columns=columns, sql=sql, attributes=attributes, - warnings=slack_warnings, + return get_dialect(prepared.dialect).decode_result_keys(rows) + + async def _scan_one_table_values( + self, + *, + table: str, + exprs: "list[str]", + dialect: str, + datasource: DatasourceConfig, + client: SlayerSQLClient, + ) -> "dict[str, Any]": + """Run one batched refresh-key scan for a table and return + ``{expression: value}`` (read from the ``slayer_rk_`` aliases). + + The scan SQL is policy-rewritten identically to the data query so a + tenant-scoped query's baseline can't be masked by a global MAX/COUNT. + """ + scan_sql = self._cache.build_refresh_key_sql(table, exprs, dialect) + scan_sql = self._apply_policy(sql=scan_sql, dialect=dialect, datasource=datasource) + rows = await client.execute(sql=scan_sql) + row0 = rows[0] if rows else {} + return {e: row0.get(self._cache.rk_alias(i)) for i, e in enumerate(exprs)} + + async def _scan_refresh_key_baselines( + self, *, prepared: _Prepared, client: SlayerSQLClient, cache: QueryCache + ) -> "tuple[list[tuple[str, str]], list[RefreshKeyValue]]": + """Capture the write-time refresh-key baselines for a cache entry. + + Returns ``(applicable, refresh_key_values)`` where ``applicable`` is the + entry's applicable ``(table, expression)`` refresh keys (order + dups + preserved) and ``refresh_key_values`` are the scanned baselines in the + same order. Scan failures propagate (write-time contract). + """ + applicable = cache.applicable_keys(prepared.sql, prepared.dialect) + if not applicable: + return [], [] + by_table = self._group_expressions_by_table(applicable) + scanned: dict[str, dict[str, Any]] = {} + for table, exprs in by_table.items(): + scanned[table] = await self._scan_one_table_values( + table=table, exprs=exprs, dialect=prepared.dialect, + datasource=prepared.datasource, client=client, + ) + values = [ + RefreshKeyValue(table=t, expression=e, value=scanned[t][e]) + for (t, e) in applicable + ] + return applicable, values + + @staticmethod + def _group_expressions_by_table( + applicable: "list[tuple[str, str]]", + ) -> "dict[str, list[str]]": + """Collate ``(table, expression)`` pairs into ``{table: [exprs]}``, + preserving first-occurrence order and de-duplicating identical + expressions (so the ``slayer_rk_`` alias order is stable).""" + by_table: dict[str, list[str]] = {} + for table, expr in applicable: + exprs = by_table.setdefault(table, []) + if expr not in exprs: + exprs.append(expr) + return by_table + + def _build_cache_entry( + self, + *, + prepared: _Prepared, + response: SlayerResponse, + original_input: Any, + variables: Optional[Dict[str, Any]], + data_source: Optional[str], + created_at: float, + applicable: "list[tuple[str, str]]", + refresh_key_values: "list[RefreshKeyValue]", + ) -> _CacheEntry: + """Build a ``_CacheEntry`` holding a defensive deep copy of the response + and the original user input (so later caller mutation can't change a + cached hit or a refresh replay).""" + ds_key = _sql_client_cache_key(prepared.datasource) + return _CacheEntry( + response=response.model_copy(deep=True), + sql=prepared.sql, + ds_fingerprint="|".join(ds_key), + dialect=prepared.dialect, + ds_key=ds_key, + resolved_data_source=prepared.resolved_data_source, + original_input=copy.deepcopy(original_input), + # Deep copy (not a shallow ``dict(...)``) so nested list/dict values + # can't be mutated by the caller after execute and leak into a + # refresh() replay — matching the ``original_input`` snapshot above. + variables=copy.deepcopy(dict(variables)) if variables else None, + data_source=data_source, + created_at=created_at, + applicable=list(applicable), + refresh_key_values=list(refresh_key_values), + ) + + async def evict( + self, + query: "SlayerQuery | dict | list[SlayerQuery | dict] | str", + variables: Optional[Dict[str, Any]] = None, + *, + data_source: Optional[str] = None, + ) -> bool: + """Remove one cached entry, recomputing its key DB-free (resolve→enrich + →SQL-gen→policy). Returns ``True`` if an entry was present. Never + constructs a SQL client on the no-policy path.""" + runtime_kwarg = variables or {} + main_query, named_queries, prefer_ds = await self._normalize_input( + query, runtime_kwarg=runtime_kwarg, prefer_data_source=data_source + ) + prepared = await self._prepare_pipeline( + query=main_query, + named_queries=named_queries, + runtime_kwarg=runtime_kwarg, + prefer_data_source=prefer_ds, + ) + key = QueryCache.make_key(prepared.sql, self._ds_fingerprint(prepared.datasource)) + return await self._cache.delete(key) + + def evict_sync( + self, + query: "SlayerQuery | dict | list[SlayerQuery | dict] | str", + variables: Optional[Dict[str, Any]] = None, + *, + data_source: Optional[str] = None, + ) -> bool: + """Synchronous wrapper for :meth:`evict`.""" + from slayer.async_utils import run_sync + + async def _run() -> bool: + try: + return await self.evict(query, variables=variables, data_source=data_source) + finally: + await self.aclose() + + return run_sync(_run()) + + async def _reexecute_entry(self, entry: _CacheEntry, now: float) -> _CacheEntry: + """Re-prepare a stale entry from its ORIGINAL input and re-execute it, + returning a fresh entry (``created_at=now`` → TTL reset). + + Replays through the full ``_normalize_input`` / ``_prepare_pipeline`` + pipeline — so model / ``source_queries`` edits and ``whole_periods_only`` + re-snapping are picked up — but the connection identity is PINNED to the + entry's ``ds_key`` via ``override_datasource`` (the datasource carried by + the client cached at write time). So neither a datasource-priority flip + NOR a same-name config edit can migrate the entry or re-execute against a + different database; a new identity is a new cache entry via ``execute``, + never a refresh migration. If the write-time client is gone the re-exec + can't be pinned faithfully → raise, and ``refresh()`` records the + ``re_execute`` error and keeps the stale entry. Any other error (e.g. the + re-exec baseline scan hitting a dropped table) propagates the same way. + """ + client = self._sql_clients.get(entry.ds_key) + if client is None: + raise RuntimeError( + f"no cached SQL client for datasource fingerprint {entry.ds_key!r}; " + "cannot pin re-execution to the entry's connection identity" + ) + main_query, named_queries, prefer_ds = await self._normalize_input( + entry.original_input, + runtime_kwarg=entry.variables or {}, + prefer_data_source=entry.resolved_data_source, + ) + prepared = await self._prepare_pipeline( + query=main_query, + named_queries=named_queries, + runtime_kwarg=entry.variables or {}, + prefer_data_source=prefer_ds, + override_datasource=client.datasource, + ) + applicable, refresh_key_values = await self._scan_refresh_key_baselines( + prepared=prepared, client=client, cache=self._cache ) + rows = await self._run_data_query(prepared=prepared, client=client) + columns = prepared.expected_columns if not rows else [] + response = SlayerResponse( + data=rows, columns=columns, sql=prepared.sql, + attributes=prepared.attributes, warnings=prepared.slack_warnings, + ) + return self._build_cache_entry( + prepared=prepared, + response=response, + original_input=entry.original_input, + variables=entry.variables, + data_source=entry.data_source, + created_at=now, + applicable=applicable, + refresh_key_values=refresh_key_values, + ) + + async def refresh(self) -> RefreshResult: # NOSONAR S3776 — Cube-style refresh: snapshot → collate scans → per-entry TTL/refresh-key decision + """Cube-style explicit refresh over all cached entries. + + Snapshots the cache, runs one batched refresh-key scan per + ``(datasource-fingerprint, table)`` — through the SQL client cached at + write time, so each entry is scanned against the exact connection + identity (``ds_key``) it was cached under. Keying by fingerprint (not + the bare datasource name) mirrors the cache key: a same-name config + edit or a datasource-priority flip cannot migrate the scan to a + different database. Then per entry: TTL-expired ⇒ re-exec + (``expired_refreshed``); an applicable table whose scan failed ⇒ keep + ``unchanged``; any applicable refresh-key value moved ⇒ re-exec + (``refreshed``); else ``unchanged``. Continue-on-failure: scan / + re-exec errors become :class:`RefreshError` and keep the stale entry. + """ + result = RefreshResult() + snapshot = await self._cache.snapshot() + if not snapshot: + return result + + # Collate {ds_key: {table: ordered exprs}} across entries — keyed by the + # SQL-client fingerprint (connection_string|runtime_fingerprint), NOT + # the bare datasource name. An entry cached under one connection + # identity is thus scanned against THAT identity even if the datasource + # was later edited under the same name (the cache key is fingerprint- + # scoped too, so such entries coexist). + collate: dict[tuple[str, str], dict[str, list[str]]] = {} + for entry in snapshot.values(): + if not entry.applicable: + continue + tables = collate.setdefault(entry.ds_key, {}) + for table, expr in entry.applicable: + exprs = tables.setdefault(table, []) + if expr not in exprs: + exprs.append(expr) + + # One batched scan per (ds_key, table), continue-on-error per table. + # The scan runs through the client cached at write time (which carries + # its exact DatasourceConfig) — no name re-resolution, so a priority + # flip or same-name config edit can't migrate the scan. + scanned: dict[tuple[tuple[str, str], str], dict[str, Any]] = {} + failed: set[tuple[tuple[str, str], str]] = set() + for ds_key, tables in collate.items(): + client = self._sql_clients.get(ds_key) + for table, exprs in tables.items(): + if client is None: + # The write-time client is gone (should not happen — clients + # live for the engine's lifetime). Fail-soft: keep the entry + # rather than re-resolve the name to a possibly-different + # fingerprint and scan the wrong database. + failed.add((ds_key, table)) + result.errors.append(RefreshError( + key=table, phase="refresh_key_scan", + message=f"no cached SQL client for datasource fingerprint {ds_key!r}", + )) + continue + try: + datasource = client.datasource + dialect = self._dialect_for_type(datasource.type) + # Codex #5: warm the ClickHouse correlated-subquery version + # cache before policy-applying the standalone scan SQL, so + # a join-policy refresh scan matches normal execution. + await self._preflight_clickhouse_correlated( + dialect=dialect, datasource=datasource + ) + scanned[(ds_key, table)] = await self._scan_one_table_values( + table=table, exprs=exprs, dialect=dialect, + datasource=datasource, client=client, + ) + except Exception as exc: + failed.add((ds_key, table)) + result.errors.append(RefreshError( + key=table, phase="refresh_key_scan", message=str(exc), + )) + + for key, entry in snapshot.items(): + now = self._cache.now() + ttl = self._cache.config.ttl_seconds + ttl_expired = ttl is not None and (now - entry.created_at) > ttl + if ttl_expired: + bucket = result.expired_refreshed + else: + dk = entry.ds_key + if any((dk, t) in failed for (t, _e) in entry.applicable): + result.unchanged.append(key) + continue + moved = any( + QueryCache.values_differ( + scanned.get((dk, rkv.table), {}).get(rkv.expression), + rkv.value, + ) + for rkv in entry.refresh_key_values + ) + if not moved: + result.unchanged.append(key) + continue + bucket = result.refreshed + + # Re-exec. Bucket only after a SUCCESSFUL re-exec AND a landed + # commit: a re-exec failure records a re_execute error and keeps the + # stale entry; a commit skipped by the identity guard (the entry was + # concurrently evicted / cleared / superseded) must NOT be reported + # as refreshed, since the cache was not actually updated. + try: + new_entry = await self._reexecute_entry(entry, now) + except Exception as exc: + result.errors.append(RefreshError( + key=key, phase="re_execute", message=str(exc), + )) + continue + new_key = QueryCache.make_key(new_entry.sql, new_entry.ds_fingerprint) + replaced = await self._cache.commit_replace( + old_key=key, expected=entry, new_key=new_key, new_entry=new_entry, + ) + if replaced: + bucket.append(key) + + return result + + def refresh_sync(self) -> RefreshResult: + """Synchronous wrapper for :meth:`refresh`.""" + from slayer.async_utils import run_sync + + async def _run() -> RefreshResult: + try: + return await self.refresh() + finally: + await self.aclose() + + return run_sync(_run()) def _normalize_stage( self, @@ -866,7 +1754,7 @@ async def get_column_types( # NOSONAR(S3776) — linear probe pipeline: query-b except ValueError: return {} - ds_key = datasource.get_connection_string() + ds_key = _sql_client_cache_key(datasource) if ds_key not in self._sql_clients: self._sql_clients[ds_key] = SlayerSQLClient(datasource=datasource) client = self._sql_clients[ds_key] @@ -900,6 +1788,16 @@ async def get_column_types( # NOSONAR(S3776) — linear probe pipeline: query-b root = planned[-1] dialect = self._dialect_for_type(datasource.type) sql = generate_planned_stages(planned, bundle=bundle, dialect=dialect) + # DEV-1578: type probing is a user-visible execution path, so it + # honours the forced-filter policy too — a policy failure + # (block / fail-closed) degrades to {} via this try/except rather + # than leaking an unscoped probe. DEV-1627: preflight the ClickHouse + # version before the rewrite so the correlated-subquery guard can + # gate (no-op otherwise, and no-op entirely when no policy is set). + await self._preflight_clickhouse_correlated( + dialect=dialect, datasource=datasource + ) + sql = self._apply_policy(sql=sql, dialect=dialect, datasource=datasource) except Exception: logger.warning( "get_column_types plan/generate failed for model '%s'", @@ -915,6 +1813,12 @@ async def get_column_types( # NOSONAR(S3776) — linear probe pipeline: query-b ) return {} + # DEV-1716: on BigQuery / T-SQL the probe SQL is alias-mangled (it has to + # be, to execute), so the cursor returns mangled keys like + # ``orders___revenue_max``. Decode them back to the canonical dotted form + # the ``full`` lookups below use. Identity for every non-mangling dialect. + raw_types = get_dialect(dialect).decode_result_keys([raw_types])[0] + # Map qualified aliases (e.g., "orders.revenue_max") back to bare # measure names. Probe sources can be ColumnKey (.leaf) or # ColumnSqlKey (.column_name) per DEV-1369 derived columns. @@ -943,13 +1847,282 @@ def execute_sync( *, dry_run: bool = False, explain: bool = False, + data_source: Optional[str] = None, + cache: bool = False, ) -> SlayerResponse: - """Synchronous wrapper for execute(). For CLI, notebooks, and scripts.""" + """Synchronous wrapper for execute(). For CLI, notebooks, and scripts. + + Forwards ``cache`` and ``data_source`` (the sync surface previously + lacked ``data_source``; DEV-1715 closes that gap). Disposes per-call + async engines in ``finally`` so they don't outlive their owning loop — + see ``aclose`` (DEV-1656). + """ from slayer.async_utils import run_sync - return run_sync( - self.execute(query, variables=variables, dry_run=dry_run, explain=explain) + async def _run_and_cleanup() -> SlayerResponse: + try: + return await self.execute( + query, variables=variables, dry_run=dry_run, + explain=explain, data_source=data_source, cache=cache, + ) + finally: + await self.aclose() + + return run_sync(_run_and_cleanup()) + + # ------------------------------------------------------------------ + async def _scope_bare_name_to_datasource( + self, *, raw: str, name: str, data_source: str + ) -> "tuple[str, str, str]": + """Resolve a bare (dotless) item within a single datasource, so the + requested ``data_source`` genuinely scopes it (rather than deferring + to the global datasource-priority list). Returns ``(ds, model, leaf)``. + """ + models = await _all_models_in_datasource(self.storage, data_source) + if any(m.name == name for m in models): + raise ValueError( + f"'{raw}' resolves to model '{name}' in '{data_source}', which " + f"is not a column or metric. recommend_root_model needs " + f"'model.column' / 'model.metric' items." + ) + # Ownership is column/metric only — a model whose sole match is a + # custom aggregation named the same must NOT make a valid column + # ambiguous; it only drives the aggregation-specific error below. + owners = [ + m for m in models + if m.get_column(name) is not None or m.get_measure(name) is not None + ] + if len(owners) == 1: + return data_source, owners[0].name, name + if len(owners) > 1: + names = sorted(m.name for m in owners) + raise ValueError( + f"'{raw}' is ambiguous in datasource '{data_source}' — matches " + f"{names}. Qualify it as '.{name}'." + ) + if any(m.get_aggregation(name) is not None for m in models): + raise ValueError( + f"'{raw}' names a custom aggregation in datasource " + f"'{data_source}'. Aggregations are operators applied to " + f"columns, not join-path targets — pass the column." + ) + raise ValueError( + f"'{raw}' does not name a column or metric in datasource " + f"'{data_source}'." + ) + + async def _resolve_recommend_item( + self, *, raw: str, data_source: str | None + ) -> "tuple[_ResolvedItem, list[str]]": + """Resolve + validate a single recommend_root_model item. + + Reuses ``resolve_entity`` for normalization; requires the resolved + entity to be a column or a named measure (metric). Raises on a + malformed / wrong-kind item or a datasource-scope mismatch. + """ + entity_ref, suffix = split_agg_suffix(raw) + warnings: list[str] = [] + if data_source and "." not in entity_ref: + # Bare name + explicit scope → resolve within the datasource. + ds, model_name, leaf = await self._scope_bare_name_to_datasource( + raw=raw, name=entity_ref, data_source=data_source, + ) + else: + # With an explicit data_source, force every dotted item into that + # datasource unless it already leads with it — so a model name + # colliding with a *datasource* name (``shared.status`` under + # ``data_source='mydb'``) still resolves to the model, and a ref + # naming a different datasource fails as a cross-datasource item. + resolution_input = entity_ref + if data_source and entity_ref.split(".")[0] != data_source: + resolution_input = f"{data_source}.{entity_ref}" + res = await resolve_entity(resolution_input, storage=self.storage) + warnings = res.warnings + segs = res.canonical_forms[0].split(".") + if len(segs) < 3: + raise ValueError( + f"'{raw}' resolves to '{res.canonical_forms[0]}', which is " + f"not a column or metric. recommend_root_model needs " + f"'model.column' / 'model.metric' items." + ) + ds, model_name, leaf = segs[0], segs[1], segs[2] + owning = await self.storage.get_model(model_name, data_source=ds) + if owning is None: + raise ValueError( + f"'{raw}' resolves to model '{model_name}' in '{ds}', which " + f"is not a saved model." + ) + if owning.get_column(leaf) is None and owning.get_measure(leaf) is None: + if owning.get_aggregation(leaf) is not None: + raise ValueError( + f"'{raw}' names the custom aggregation '{leaf}' on " + f"'{model_name}'. Aggregations are operators applied to " + f"columns, not join-path targets — pass the column." + ) + raise ValueError( + f"'{raw}' does not name a column or metric on '{model_name}'." + ) + item = _ResolvedItem( + input_item=raw, data_source=ds, model=model_name, + leaf=leaf, suffix=suffix, ) + return item, warnings + + async def _recommend_resolve_items( + self, *, items: list[str], data_source: str | None + ) -> "tuple[list[_ResolvedItem], list[str]]": + """Resolve every input item, then enforce single-datasource + dedup. + + Raises on malformed / wrong-kind items and on a cross-datasource mix. + """ + resolved: list[_ResolvedItem] = [] + warnings: list[str] = [] + seen_inputs: set[str] = set() + for original in items: + raw = original.strip() + if not raw or raw in seen_inputs: + continue + seen_inputs.add(raw) + item, item_warnings = await self._resolve_recommend_item( + raw=raw, data_source=data_source, + ) + resolved.append(item) + warnings.extend(item_warnings) + + if not resolved: + raise ValueError("recommend_root_model requires at least one item.") + datasources = {r.data_source for r in resolved} + if len(datasources) > 1: + raise ValueError( + f"items span multiple datasources {sorted(datasources)}; " + f"cross-datasource queries are not supported. Pass items from " + f"a single datasource (optionally set data_source=...)." + ) + deduped_warnings: list[str] = [] + seen_w: set[str] = set() + for w in warnings: + if w not in seen_w: + seen_w.add(w) + deduped_warnings.append(w) + return resolved, deduped_warnings + + async def recommend_root_model( + self, + items: list[str], + *, + data_source: str | None = None, + root_hint: str | None = None, + ) -> RootModelRecommendation: + """Recommend the query ``source_model`` (root) for a set of + ``model.column`` / ``model.metric`` items, plus each item's + join-qualified path from that root (DEV-1626). + + A valid root reaches every mentioned owning model over the join + graph (LEFT = directed, INNER = symmetric via storage). Selection + minimizes total hops summed over distinct owning models, prefers a + mentioned model on ties, then the lexicographically smallest name. + When no single model reaches everything, returns a structured + ``reachable=False`` result with the Pareto frontier of partial + roots in ``coverage`` (never raises for a valid-but-unroutable set). + + ``root_hint`` (optional) names the intended root — a bare model name + or ``.`` within the resolved datasource. When the + hint is a feasible root (reaches every item), it is honored outright, + overriding the min-hops selection so a caller can force a *bridge* + model that owns none of the items but matches the intended grain. + When it is infeasible, the auto-pick is used and a warning explains + why; whether the hint was honored is conveyed via ``message`` / + ``warnings`` (no structured field). A non-existent / wrong-kind / + cross-datasource hint raises ``ValueError``. Note: ``root_hint`` is + resolved *after* the datasource is determined from ``items`` / + ``data_source``, so it cannot influence datasource selection. + """ + resolved, base_warnings = await self._recommend_resolve_items( + items=items, data_source=data_source + ) + ds = resolved[0].data_source + models = await _all_models_in_datasource(self.storage, ds) + graph = JoinGraph.build_from_models(models) + all_names = sorted(m.name for m in models) + mentioned = {r.model for r in resolved} + warnings = list(base_warnings) + + hint = _resolve_root_hint(root_hint, data_source=ds, all_names=all_names) + hint_model = hint[0] if hint is not None else None + hint_display = hint[1] if hint is not None else None + + def reaches_all(root: str) -> bool: + return all(graph.shortest_path(root, m) is not None for m in mentioned) + + def missing_models(root: str) -> list[str]: + return sorted(m for m in mentioned if graph.shortest_path(root, m) is None) + + # Shared selection core (also used by the OSI importer's anchor pick). + auto = min_hops_root(graph, all_names, mentioned) + if auto is not None: + if hint_model is not None and reaches_all(hint_model): + root = hint_model + message = ( + f"All items are reachable from '{root}' (requested via root_hint)." + ) + else: + root = auto + if hint_model is not None: + missing = ", ".join(missing_models(hint_model)) + warnings.append( + f"root_hint '{hint_display}' cannot reach {{{missing}}}; " + f"fell back to '{auto}'." + ) + message = f"All items are reachable from '{root}'." + item_paths = [ + ItemPath(input_item=r.input_item, path=_emit_recommend_path(graph, root, r)) + for r in resolved + ] + return RootModelRecommendation( + data_source=ds, root_model=root, reachable=True, + item_paths=item_paths, warnings=warnings, message=message, + ) + + force_include = {hint_model} if hint_model is not None else None + coverage = _build_recommend_coverage( + graph, all_names, mentioned, resolved, force_include=force_include + ) + if hint_model is not None: + missing = ", ".join(missing_models(hint_model)) + warnings.append( + f"root_hint '{hint_display}' cannot reach {{{missing}}}; " + f"no single model reaches every item — see coverage." + ) + return RootModelRecommendation( + data_source=ds, root_model=None, reachable=False, item_paths=[], + coverage=coverage, warnings=warnings, + message=( + "No single model reaches every requested item. See 'coverage' " + "for the best partial roots — split the request into a " + "multi-stage query rooted at those models." + ), + ) + + def recommend_root_model_sync( + self, + items: list[str], + *, + data_source: str | None = None, + root_hint: str | None = None, + ) -> RootModelRecommendation: + """Synchronous wrapper for :meth:`recommend_root_model`.""" + from slayer.async_utils import run_sync + + async def _run() -> RootModelRecommendation: + try: + return await self.recommend_root_model( + items, data_source=data_source, root_hint=root_hint + ) + finally: + await self.aclose() + + return run_sync(_run()) + async def edit_model_remove( self, @@ -2085,10 +3258,11 @@ def _alias_to_short(alias: str) -> str: 'orders.customers.regions.name' → 'customers__regions__name' 'orders.count' → 'count' """ - # Strip source model prefix - stripped = alias.split(".", 1)[-1] if "." in alias else alias - # Replace remaining dots with __ to encode the original join path - return stripped.replace(".", "__") + # DEV-1713: the strip-source-prefix + ``__`` flatten is owned by + # the naming module (single owner). The first path segment is the + # source-model prefix to strip. + strip = alias.split(".", 1)[0] if "." in alias else None + return flat_name(alias, strip_relation=strip) # (inner_alias, short_name, data_type, label, description, format) column_map = [] @@ -2142,8 +3316,19 @@ def _alias_to_short(alias: str) -> str: short = _alias_to_short(cm.alias) column_map.append((cm.alias, short, DataType.DOUBLE, cm.label, None, cm.format)) - # Wrap inner SQL: SELECT "orders.id" AS id, "orders.count" AS count, ... FROM (inner) AS _inner - rename_parts = [f'"{alias}" AS {short}' for alias, short, _, _, _, _ in column_map] + # Wrap inner SQL: SELECT AS id, AS count, ... FROM (inner) AS _inner + # DEV-1716: ``generate(render_mode="wrapped")`` dialect-quotes AND + # (BigQuery / T-SQL) alias-mangles the inner query's projection, so the + # outer reference must match. Dialect-quote each alias and apply the same + # ``rewrite_emitted_sql`` — identity for Postgres/SQLite/DuckDB and for + # MySQL's dot-preserving backticks; mangles the dotted alias on + # BigQuery/T-SQL to the ``___`` form the inner actually exposes. A raw + # ANSI ``"{alias}"`` would reference a column the mangled inner no longer + # has (and be a string literal, not an identifier, on MySQL/BigQuery/T-SQL). + def _inner_ref(alias: str) -> str: + return generator._dialect.rewrite_emitted_sql(generator._quote_ident(alias)) + + rename_parts = [f'{_inner_ref(alias)} AS {short}' for alias, short, _, _, _, _ in column_map] wrapped_sql = f"SELECT {', '.join(rename_parts)} FROM ({inner_sql}) AS _inner" # One Column per result column — each is potentially both a dimension @@ -2502,8 +3687,6 @@ async def _build_rerooted_enriched( Dimensions and filters referencing models not reachable from the target are dropped. """ - import re - from slayer.core.formula import parse_filter target_model = await self._resolve_model( @@ -2662,25 +3845,10 @@ async def _resolve_datasource(self, model: SlayerModel) -> DatasourceConfig: @staticmethod def _dialect_for_type(ds_type: Optional[str]) -> str: - _DIALECT_MAP = { - "postgres": "postgres", - "postgresql": "postgres", - "mysql": "mysql", - "mariadb": "mysql", - "clickhouse": "clickhouse", - "bigquery": "bigquery", - "snowflake": "snowflake", - "sqlite": "sqlite", - "duckdb": "duckdb", - "redshift": "redshift", - "trino": "trino", - "presto": "presto", - "athena": "presto", - "databricks": "databricks", - "spark": "spark", - "mssql": "tsql", - "sqlserver": "tsql", - "tsql": "tsql", - "oracle": "oracle", - } - return _DIALECT_MAP.get(ds_type or "", "postgres") + """Map a datasource ``type`` to its sqlglot dialect name. + + DEV-1716: delegates to the DEV-1542 registry (``dialect_for_ds_type``) + — single source of truth for the ds-type → dialect mapping — instead + of an inline duplicate map. Lenient (unknown / None → ``postgres``). + """ + return dialect_for_ds_type(ds_type).sqlglot_name diff --git a/slayer/engine/response_meta.py b/slayer/engine/response_meta.py index 9e34a14b..7a5ed70a 100644 --- a/slayer/engine/response_meta.py +++ b/slayer/engine/response_meta.py @@ -42,6 +42,8 @@ from slayer.core.models import Column, SlayerModel from slayer.engine.planned import PlannedQuery, ValueSlot from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.sql.dialects import get_dialect +from slayer.sql.naming import result_key, result_key_from_alias # --------------------------------------------------------------------------- @@ -75,7 +77,7 @@ def _infer_aggregated_format( """Infer NumberFormat for an aggregated measure based on aggregation type and source measure format. Rules: - - count, count_distinct: always INTEGER + - count, count_distinct, count_distinct_approx: always INTEGER - avg, weighted_avg, median: always FLOAT - sum, min, max, first, last: inherit from source measure - *:count (measure_name="*"): INTEGER @@ -83,7 +85,7 @@ def _infer_aggregated_format( if measure_name == "*": return NumberFormat(type=NumberFormatType.INTEGER) - if aggregation in ("count", "count_distinct"): + if aggregation in ("count", "count_distinct", "count_distinct_approx"): return NumberFormat(type=NumberFormatType.INTEGER) if aggregation in ("avg", "weighted_avg", "median"): @@ -129,23 +131,37 @@ def _model_for_path( def _slot_result_keys(*, slot: ValueSlot, source_relation: str) -> List[str]: """The public result-key alias(es) for ``slot``. - Mirrors ``SQLGenerator._full_alias_for_slot``: joined ROW slots emit the - full dotted path (``orders.customers.region``); everything else uses the - slot's public alias(es) — multiple for a C13 multi-name interned slot — - prefixed by the stage's source relation. + Mirrors ``SQLGenerator._full_alias_for_slot`` via the SAME naming builders + (``slayer.sql.naming.result_key`` / ``result_key_from_alias``) so the SQL + alias and the response result key cannot drift. Joined ROW slots — base + ``ColumnKey``, derived ``ColumnSqlKey`` (DEV-1713 D3 / DEV-1495 bug 1), and + ``TimeTruncKey`` over either — emit the full dotted path + (``orders.customers.region``); everything else uses the slot's public + alias(es) — multiple for a C13 multi-name interned slot. """ key = slot.key if slot.phase == Phase.ROW: if isinstance(key, ColumnKey) and key.path: - return [f"{source_relation}." + ".".join(key.path) + f".{key.leaf}"] + return [result_key( + source_relation=source_relation, path=key.path, leaf=key.leaf, + )] + if isinstance(key, ColumnSqlKey) and key.path: + return [result_key( + source_relation=source_relation, + path=key.path, + leaf=key.column_name, + )] if isinstance(key, TimeTruncKey) and column_path(key.column): - return [ - f"{source_relation}." - + ".".join(column_path(key.column)) - + f".{column_leaf(key.column)}" - ] + return [result_key( + source_relation=source_relation, + path=column_path(key.column), + leaf=column_leaf(key.column), + )] aliases = slot.public_aliases or [slot.declared_name] - return [f"{source_relation}.{a}" for a in aliases] + return [ + result_key_from_alias(source_relation=source_relation, alias=a) + for a in aliases + ] def _column_for_row_slot( @@ -237,7 +253,7 @@ def _measure_label( return None -def build_response_metadata( +def build_response_metadata( # NOSONAR(S3776) — flat per-slot metadata classification (dimension vs measure, TimeTruncKey, label/format lookup) over one candidate-slot loop; complexity is inherent to the projection-to-metadata mapping and pre-dates this change. Splitting the loop body out would scatter the shared public_keys / source_relation state without improving readability. *, root_planned: PlannedQuery, bundle: ResolvedSourceBundle, @@ -253,6 +269,16 @@ def build_response_metadata( a guard against any divergence between this derivation and the generator. """ expected_columns = expected_columns_from_sql(sql=sql, dialect=dialect) + # DEV-1716: on BigQuery / T-SQL the rendered SQL carries alias-mangled + # projection names (``orders___status``); decode them back to the canonical + # dotted form so ``expected_columns`` and the attribute-matching below + # operate in the same space as the plan's slot result keys. Reuses the + # dialect read-side hook (identity for every non-mangling dialect) via a + # synthetic-row wrap — no ``decode_columns`` method needed. + if expected_columns: + expected_columns = list( + get_dialect(dialect).decode_result_keys([dict.fromkeys(expected_columns)])[0] + ) public_keys = set(expected_columns) source_relation = root_planned.source_relation diff --git a/slayer/engine/schema_drift.py b/slayer/engine/schema_drift.py index dffbd2c6..0dff9a04 100644 --- a/slayer/engine/schema_drift.py +++ b/slayer/engine/schema_drift.py @@ -17,12 +17,10 @@ import logging from typing import ( Annotated, - Dict, List, Literal, Optional, Set, - Tuple, Union, ) @@ -40,8 +38,8 @@ ) from slayer.core.query import SlayerQuery from slayer.sql.sql_predicate import parse_sql_predicate +from slayer.engine.introspect_utils import _safe_get_columns from slayer.engine.ingestion import ( - _safe_get_columns, _safe_get_pk_constraint, _sa_type_is_float, _sa_type_to_data_type, @@ -75,10 +73,10 @@ class DeleteReason(BaseModel): class RemoveSpec(BaseModel): """Per-entity removal spec, mirroring the MCP ``edit_model`` ``remove=`` shape.""" - columns: List[str] = Field(default_factory=list) - measures: List[str] = Field(default_factory=list) - aggregations: List[str] = Field(default_factory=list) - joins: List[str] = Field(default_factory=list) + columns: list[str] = Field(default_factory=list) + measures: list[str] = Field(default_factory=list) + aggregations: list[str] = Field(default_factory=list) + joins: list[str] = Field(default_factory=list) class EditModelDelete(BaseModel): @@ -88,8 +86,8 @@ class EditModelDelete(BaseModel): model_name: str data_source: str remove: RemoveSpec = Field(default_factory=RemoveSpec) - remove_filters: List[str] = Field(default_factory=list) - reasons: List[DeleteReason] = Field(default_factory=list) + remove_filters: list[str] = Field(default_factory=list) + reasons: list[DeleteReason] = Field(default_factory=list) class WholeModelDelete(BaseModel): @@ -98,11 +96,11 @@ class WholeModelDelete(BaseModel): tool: Literal["delete_model"] = "delete_model" model_name: str data_source: str - reasons: List[DeleteReason] = Field(default_factory=list) + reasons: list[DeleteReason] = Field(default_factory=list) ToDeleteEntry = Annotated[ - Union[EditModelDelete, WholeModelDelete], Field(discriminator="tool") + EditModelDelete | WholeModelDelete, Field(discriminator="tool") ] @@ -112,8 +110,11 @@ class ModelAddition(BaseModel): model_name: str data_source: str created: bool = False # True if the model was new - new_columns: List[str] = Field(default_factory=list) - new_joins: List[str] = Field(default_factory=list) + new_columns: list[str] = Field(default_factory=list) + new_joins: list[str] = Field(default_factory=list) + # DEV-1538: persisted INT columns whose type widened (to DOUBLE or TEXT) + # because the SQLite affinity probe disagreed with the declared type. + widened_columns: list[str] = Field(default_factory=list) class IngestionError(BaseModel): @@ -127,9 +128,9 @@ class IngestionError(BaseModel): class IdempotentIngestResult(BaseModel): """Combined return shape of the idempotent ``ingest_datasource`` pass.""" - additions: List[ModelAddition] = Field(default_factory=list) - to_delete: List[ToDeleteEntry] = Field(default_factory=list) - errors: List[IngestionError] = Field(default_factory=list) + additions: list[ModelAddition] = Field(default_factory=list) + to_delete: list[ToDeleteEntry] = Field(default_factory=list) + errors: list[IngestionError] = Field(default_factory=list) class AppliedEntry(BaseModel): @@ -152,9 +153,9 @@ class ApplyError(BaseModel): class ApplyDriftResult(BaseModel): """Combined return shape of ``apply_drift_deletes``.""" - applied: List[AppliedEntry] = Field(default_factory=list) - errors: List[ApplyError] = Field(default_factory=list) - residual: List[ToDeleteEntry] = Field(default_factory=list) + applied: list[AppliedEntry] = Field(default_factory=list) + errors: list[ApplyError] = Field(default_factory=list) + residual: list[ToDeleteEntry] = Field(default_factory=list) # =========================================================================== @@ -171,10 +172,10 @@ class LiveTable(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) - columns: Dict[str, DataType] = Field(default_factory=dict) - pk_columns: Set[str] = Field(default_factory=set) + columns: dict[str, DataType] = Field(default_factory=dict) + pk_columns: set[str] = Field(default_factory=set) # Each entry: (local_column, ref_table, ref_column) - fk_relationships: List[Tuple[str, str, str]] = Field(default_factory=list) + fk_relationships: list[tuple[str, str, str]] = Field(default_factory=list) # =========================================================================== @@ -203,7 +204,35 @@ def data_type_bucket(dt: DataType) -> str: return str(dt) -def _is_bare_identifier(s: Optional[str]) -> bool: +def _type_buckets_conflict(*, persisted: DataType, live: DataType) -> bool: + """True when persisted vs live types are genuinely incompatible. + + Opacity is deliberately **one-way**: + + - *live* opaque + operable persisted type → **conflict**. The physical + column has no equality operator, so the persisted model is promising a + ``GROUP BY`` / ``DISTINCT`` / aggregation the database will refuse. That + is a real runtime hazard and must surface rather than be hidden. + - *persisted* opaque + known live type → no conflict. ``UNKNOWN`` makes no + type claim to contradict; it only says "we could not classify this", and + the live type is strictly more information. + - both opaque → no conflict (they agree). + - both known → ordinary bucket comparison. + + Note this means models ingested before opaque classification existed (an + exotic column coarsed to TEXT back then, read as UNKNOWN now) will be + reported. That is intentional: those columns really are unusable as + declared. The remedy is to re-ingest / retype the column to ``UNKNOWN``, + after which the conflict disappears. + """ + if live.is_opaque: + return not persisted.is_opaque + if persisted.is_opaque: + return False + return data_type_bucket(persisted) != data_type_bucket(live) + + +def _is_bare_identifier(s: str | None) -> bool: """``s`` is a bare SQL identifier (alphanumeric + underscore, no leading digit).""" if not s: return False @@ -213,7 +242,7 @@ def _is_bare_identifier(s: Optional[str]) -> bool: return all(c.isalnum() or c == "_" for c in s) -def _column_is_base(col_sql: Optional[str]) -> bool: +def _column_is_base(col_sql: str | None) -> bool: """A Column whose ``sql`` is None or a bare identifier is a "base" column — it claims a live column. Derived expressions (``amount * 2``, ``customers.region``, etc.) do not. @@ -230,10 +259,10 @@ def _column_is_base(col_sql: Optional[str]) -> bool: def _diff_sql_table_columns( *, model: SlayerModel, live_table: LiveTable -) -> Tuple[List[str], List[DeleteReason]]: +) -> tuple[list[str], list[DeleteReason]]: """Per-column diff of a sql_table-mode model against live columns.""" - dropped: List[str] = [] - reasons: List[DeleteReason] = [] + dropped: list[str] = [] + reasons: list[DeleteReason] = [] for col in model.columns: # Only compare base columns directly. Derived columns are handled # by cascade. @@ -250,7 +279,7 @@ def _diff_sql_table_columns( ) continue live_dt = live_table.columns[bare_name] - if data_type_bucket(col.type) != data_type_bucket(live_dt): + if _type_buckets_conflict(persisted=col.type, live=live_dt): dropped.append(col.name) reasons.append( DeleteReason( @@ -268,12 +297,12 @@ def _diff_sql_table_joins( *, model: SlayerModel, live_table: LiveTable, - available_models_in_ds: Set[str], -) -> Tuple[List[str], List[DeleteReason]]: + available_models_in_ds: set[str], +) -> tuple[list[str], list[DeleteReason]]: """Per-join diff of a sql_table-mode model against live FK columns and in-datasource model availability.""" - dropped: List[str] = [] - reasons: List[DeleteReason] = [] + dropped: list[str] = [] + reasons: list[DeleteReason] = [] # ``join.join_pairs[*][0]`` is the semantic column name (``Column.name``). # Resolve to the physical column name via ``Column.sql`` before checking # against the live table — for a base column like @@ -320,9 +349,9 @@ def _diff_sql_table_joins( def diff_sql_table_model( *, model: SlayerModel, - live_table: Optional[LiveTable], - available_models_in_ds: Set[str], -) -> Tuple[Optional[ToDeleteEntry], Set[str]]: + live_table: LiveTable | None, + available_models_in_ds: set[str], +) -> tuple[ToDeleteEntry | None, set[str]]: """Diff a sql_table-mode model against live introspection. Returns ``(entry_or_None, dropped_column_names)``. @@ -382,8 +411,8 @@ def diff_sql_table_model( def diff_sql_model( *, model: SlayerModel, - live_columns: Optional[Dict[str, DataType]], -) -> Tuple[Optional[ToDeleteEntry], Set[str]]: + live_columns: dict[str, DataType] | None, +) -> tuple[ToDeleteEntry | None, set[str]]: """Diff a sql-mode model against trial-execute cursor metadata. ``live_columns is None`` ⇒ trial-execute failed ⇒ ``WholeModelDelete``. @@ -407,8 +436,8 @@ def diff_sql_model( {c.name for c in model.columns}, ) - dropped_cols: List[str] = [] - reasons: List[DeleteReason] = [] + dropped_cols: list[str] = [] + reasons: list[DeleteReason] = [] for col in model.columns: # Cursor exposes ALIAS names — match by col.name first, fall back to # col.sql for legacy cases where a Column's name differs from its @@ -431,7 +460,7 @@ def diff_sql_model( ) ) continue - if data_type_bucket(col.type) != data_type_bucket(live_dt): + if _type_buckets_conflict(persisted=col.type, live=live_dt): dropped_cols.append(col.name) reasons.append( DeleteReason( @@ -461,7 +490,7 @@ def diff_sql_model( # =========================================================================== -def _extract_column_refs_from_sql(sql: str) -> List[Tuple[Optional[str], str]]: +def _extract_column_refs_from_sql(sql: str) -> list[tuple[str | None, str]]: """Return all ``(table_alias, column_name)`` refs in a SQL expression. ``table_alias`` is ``None`` for bare identifiers, the raw alias string @@ -473,7 +502,7 @@ def _extract_column_refs_from_sql(sql: str) -> List[Tuple[Optional[str], str]]: parsed = sqlglot.parse_one(sql) except Exception: return [] - refs: List[Tuple[Optional[str], str]] = [] + refs: list[tuple[str | None, str]] = [] for col in parsed.find_all(exp.Column): if col.args.get("db") or col.args.get("catalog"): continue @@ -542,7 +571,7 @@ def _measure_formula_refs( return out -def _filter_refs(filter_str: str) -> List[str]: +def _filter_refs(filter_str: str) -> list[str]: """Best-effort: return list of column references in a SQL-mode filter. Used to scan ``Column.filter`` / ``SlayerModel.filters`` strings (Mode A @@ -555,7 +584,7 @@ def _filter_refs(filter_str: str) -> List[str]: return list(pf.columns) -def _filter_refs_dsl(filter_str: str) -> List[str]: +def _filter_refs_dsl(filter_str: str) -> list[str]: """Best-effort: return list of column / measure references in a DSL filter. Used to scan ``SlayerQuery.filters`` strings (Mode B DSL — DEV-1369), @@ -583,8 +612,8 @@ def _walk_alias_to_target_model( *, source_model: SlayerModel, table_alias: str, - models_by_name: Dict[str, SlayerModel], -) -> Optional[SlayerModel]: + models_by_name: dict[str, SlayerModel], +) -> SlayerModel | None: """Resolve a ``__``-delimited path alias starting from ``source_model`` to the terminal joined model. Returns None if any hop fails. """ @@ -607,8 +636,8 @@ def _resolve_dotted_ref_to_model( *, source_model: SlayerModel, dotted_ref: str, - models_by_name: Dict[str, SlayerModel], -) -> Tuple[Optional[SlayerModel], Optional[str]]: + models_by_name: dict[str, SlayerModel], +) -> tuple[SlayerModel | None, str | None]: """Resolve a dotted measure/column ref like ``customers.region`` or ``customers.regions.name`` to ``(target_model, leaf_name)``. @@ -630,13 +659,13 @@ def _resolve_dotted_ref_to_model( # =========================================================================== -def _pk_columns(model: SlayerModel) -> Set[str]: +def _pk_columns(model: SlayerModel) -> set[str]: return {c.name for c in model.columns if c.primary_key} def _ensure_edit_entry( *, - edit_entries: Dict[str, EditModelDelete], + edit_entries: dict[str, EditModelDelete], model: SlayerModel, ) -> EditModelDelete: if model.name not in edit_entries: @@ -649,8 +678,8 @@ def _ensure_edit_entry( def _add_dropped_column( *, - edit_entries: Dict[str, EditModelDelete], - dropped_cols: Dict[str, Set[str]], + edit_entries: dict[str, EditModelDelete], + dropped_cols: dict[str, set[str]], model: SlayerModel, column_name: str, reason: str, @@ -670,8 +699,8 @@ def _add_dropped_column( def _add_dropped_measure( *, - edit_entries: Dict[str, EditModelDelete], - dropped_measures: Dict[str, Set[str]], + edit_entries: dict[str, EditModelDelete], + dropped_measures: dict[str, set[str]], model: SlayerModel, measure_name: str, reason: str, @@ -690,8 +719,8 @@ def _add_dropped_measure( def _add_dropped_join( *, - edit_entries: Dict[str, EditModelDelete], - dropped_joins: Dict[str, Set[str]], + edit_entries: dict[str, EditModelDelete], + dropped_joins: dict[str, set[str]], model: SlayerModel, target_name: str, reason: str, @@ -710,7 +739,7 @@ def _add_dropped_join( def _add_remove_filter( *, - edit_entries: Dict[str, EditModelDelete], + edit_entries: dict[str, EditModelDelete], model: SlayerModel, filter_text: str, reason: str, @@ -733,8 +762,8 @@ def _add_remove_filter( def _resolve_stage_source_to_base( *, source_model: object, - prior_stages_by_name: Dict[str, SlayerQuery], -) -> Optional[str]: + prior_stages_by_name: dict[str, SlayerQuery], +) -> str | None: """Walk a ``source_model`` reference (str / SlayerModel / ModelExtension / prior-stage-name) back to a real persisted base model name. @@ -742,7 +771,7 @@ def _resolve_stage_source_to_base( underlying model — we follow it transparently so query-backed drift attribution doesn't silently skip extension-wrapped stages. """ - seen: Set[str] = set() + seen: set[str] = set() current = source_model while True: if isinstance(current, str): @@ -775,29 +804,29 @@ class _StageGraph(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) - stage_source_name: Optional[str] = None - extension_targets: Set[str] = Field(default_factory=set) - reachable: Set[str] = Field(default_factory=set) - models_by_name: Dict[str, SlayerModel] = Field(default_factory=dict) + stage_source_name: str | None = None + extension_targets: set[str] = Field(default_factory=set) + reachable: set[str] = Field(default_factory=set) + models_by_name: dict[str, SlayerModel] = Field(default_factory=dict) def _build_stage_graph( *, stage: SlayerQuery, - stage_source_name: Optional[str], - models_by_name: Dict[str, SlayerModel], + stage_source_name: str | None, + models_by_name: dict[str, SlayerModel], ) -> _StageGraph: """Build a ``_StageGraph`` for a single stage. ``stage_source_name`` is the resolved base model name (str), or ``None`` for inline / unresolved sources. """ extension_targets = _stage_join_targets(stage) - reachable: Set[str] = set() + reachable: set[str] = set() if stage_source_name: reachable.add(stage_source_name) reachable |= extension_targets frontier = list(reachable) - visited: Set[str] = set() + visited: set[str] = set() while frontier: name = frontier.pop() if name in visited: @@ -823,7 +852,7 @@ def _attribute_ref_to_base( ref: str, base_name: str, graph: _StageGraph, -) -> Optional[str]: +) -> str | None: """Walk ``ref`` through the stage's join graph and return the leaf column name when it resolves to ``base_name``, else ``None``. @@ -888,8 +917,8 @@ def _measure_refs_on_base( def _dimension_refs_on_base( stage: SlayerQuery, base_name: str, graph: _StageGraph -) -> Set[str]: - out: Set[str] = set() +) -> set[str]: + out: set[str] = set() for d in stage.dimensions or []: full = getattr(d, "full_name", None) or str(d) attributed = _attribute_ref_to_base( @@ -902,8 +931,8 @@ def _dimension_refs_on_base( def _time_dimension_refs_on_base( stage: SlayerQuery, base_name: str, graph: _StageGraph -) -> Set[str]: - out: Set[str] = set() +) -> set[str]: + out: set[str] = set() for td in stage.time_dimensions or []: attributed = _attribute_ref_to_base( ref=td.dimension.full_name, base_name=base_name, graph=graph @@ -915,8 +944,8 @@ def _time_dimension_refs_on_base( def _filter_refs_on_base( stage: SlayerQuery, base_name: str, graph: _StageGraph -) -> Set[str]: - out: Set[str] = set() +) -> set[str]: + out: set[str] = set() # ``SlayerQuery.filters`` are Mode B (DSL) — go through the DSL parser # so colon-syntax aggregations and transforms surface their underlying # measure names. ``_filter_refs`` (SQL-mode) would drop them silently. @@ -934,8 +963,8 @@ def _stage_referenced_columns_for_base( *, stage: SlayerQuery, base_name: str, - graph: Optional[_StageGraph] = None, -) -> Set[str]: + graph: _StageGraph | None = None, +) -> set[str]: """Return the set of column names referenced *on* ``base_name`` by a single source_queries stage. Walks the stage's join graph (passed via ``graph``) so multi-hop dotted refs and ModelExtension-added joins are @@ -960,7 +989,7 @@ def _stage_referenced_columns_for_base( ) -def _stage_join_targets(stage: SlayerQuery) -> Set[str]: +def _stage_join_targets(stage: SlayerQuery) -> set[str]: """Return the set of join target_model names referenced by a stage. ``SlayerQuery`` itself has no ``joins`` field; joins on a stage live @@ -971,7 +1000,7 @@ def _stage_join_targets(stage: SlayerQuery) -> Set[str]: """ source = getattr(stage, "source_model", None) joins = getattr(source, "joins", None) or [] - out: Set[str] = set() + out: set[str] = set() for j in joins: target = getattr(j, "target_model", None) if isinstance(target, str): @@ -984,9 +1013,9 @@ def _check_stage_against_base( stage: SlayerQuery, base_name: str, graph: _StageGraph, - dropped_cols: Dict[str, Set[str]], - pk_per_model: Dict[str, Set[str]], -) -> Set[str]: + dropped_cols: dict[str, set[str]], + pk_per_model: dict[str, set[str]], +) -> set[str]: """Return the set of dropped column names on ``base_name`` that this stage references (resolved through the stage's join graph). @@ -1007,8 +1036,8 @@ def _stage_uses_dropped_join( stage: SlayerQuery, base_name: str, graph: _StageGraph, - dropped_joins: Dict[str, Set[str]], -) -> Optional[str]: + dropped_joins: dict[str, set[str]], +) -> str | None: """If ``stage`` references any column under a join target that's been dropped on ``base_name``, return the conflicting target; else None. @@ -1039,12 +1068,12 @@ def _check_stage_for_whole_drop( base_name: str, qb_name: str, graph: _StageGraph, - whole_dropped_models: Set[str], - dropped_cols: Dict[str, Set[str]], - dropped_joins: Dict[str, Set[str]], - pk_per_model: Dict[str, Set[str]], - candidate_base_names: Set[str], -) -> Optional[DeleteReason]: + whole_dropped_models: set[str], + dropped_cols: dict[str, set[str]], + dropped_joins: dict[str, set[str]], + pk_per_model: dict[str, set[str]], + candidate_base_names: set[str], +) -> DeleteReason | None: """Decide whether a single stage of a query-backed model triggers the whole-drop. Returns a ``DeleteReason`` on the first matching trigger, or ``None`` when the stage has no fatal references. @@ -1102,13 +1131,13 @@ def _check_stage_for_whole_drop( def _query_backed_should_whole_drop( *, qb_model: SlayerModel, - dropped_cols: Dict[str, Set[str]], - dropped_joins: Dict[str, Set[str]], - whole_dropped_models: Set[str], - pk_per_model: Dict[str, Set[str]], - candidate_base_names: Optional[Set[str]] = None, - models_by_name: Optional[Dict[str, SlayerModel]] = None, -) -> Optional[DeleteReason]: + dropped_cols: dict[str, set[str]], + dropped_joins: dict[str, set[str]], + whole_dropped_models: set[str], + pk_per_model: dict[str, set[str]], + candidate_base_names: set[str] | None = None, + models_by_name: dict[str, SlayerModel] | None = None, +) -> DeleteReason | None: """Return a non-None DeleteReason when this query-backed model should be whole-dropped due to cascading from base-model drift, else None. @@ -1125,7 +1154,7 @@ def _query_backed_should_whole_drop( models_by_name = models_by_name or {} for i, stage in enumerate(stages): - prior_by_name: Dict[str, SlayerQuery] = {} + prior_by_name: dict[str, SlayerQuery] = {} for s in stages[:i]: s_name = getattr(s, "name", None) if s_name: @@ -1183,13 +1212,13 @@ class _CascadeState: def __init__( self, *, - models_by_name: Dict[str, SlayerModel], - edit_entries: Dict[str, EditModelDelete], - whole_entries: Dict[str, WholeModelDelete], - dropped_cols: Dict[str, Set[str]], - dropped_measures: Dict[str, Set[str]], - dropped_joins: Dict[str, Set[str]], - pk_per_model: Dict[str, Set[str]], + models_by_name: dict[str, SlayerModel], + edit_entries: dict[str, EditModelDelete], + whole_entries: dict[str, WholeModelDelete], + dropped_cols: dict[str, set[str]], + dropped_measures: dict[str, set[str]], + dropped_joins: dict[str, set[str]], + pk_per_model: dict[str, set[str]], ) -> None: self.models_by_name = models_by_name self.edit_entries = edit_entries @@ -1199,18 +1228,18 @@ def __init__( self.dropped_joins = dropped_joins self.pk_per_model = pk_per_model - def cascadable(self, name: str) -> Set[str]: + def cascadable(self, name: str) -> set[str]: """Cascadable column drops on ``name`` (excludes PKs — rule 7).""" return self.dropped_cols.get(name, set()) - self.pk_per_model.get(name, set()) def _column_ref_targets_dropped( *, - table_alias: Optional[str], + table_alias: str | None, ref_col: str, model: SlayerModel, state: _CascadeState, -) -> Tuple[bool, Optional[SlayerModel]]: +) -> tuple[bool, SlayerModel | None]: """Decide if a single ``(table_alias, ref_col)`` reference resolves to a dropped column. Returns ``(is_dropped, resolved_target_model)``. """ @@ -1228,7 +1257,7 @@ def _column_ref_targets_dropped( def _first_dropped_sql_column_ref( *, col: Column, model: SlayerModel, state: _CascadeState -) -> Optional[Tuple[SlayerModel, str]]: +) -> tuple[SlayerModel, str] | None: """Return ``(target_model, ref_col)`` for the first reference in ``col.sql`` that resolves to a dropped column, or ``None`` when nothing in the column's SQL references a dropped target. @@ -1275,7 +1304,7 @@ def _cascade_derived_columns( def _measure_drop_cause( *, ref: str, model: SlayerModel, state: _CascadeState -) -> Optional[str]: +) -> str | None: """If the measure ref resolves to a dropped column or measure, return a reason string; otherwise None. """ @@ -1295,10 +1324,10 @@ def _measure_drop_cause( def _first_dropped_cause( *, - refs: Set[str], + refs: set[str], model: SlayerModel, state: _CascadeState, -) -> Optional[str]: +) -> str | None: """Return the cause string for the first ref that resolves to a dropped column or measure, or ``None`` when nothing in ``refs`` is dropped. """ @@ -1441,7 +1470,7 @@ def _cascade_filters(*, model: SlayerModel, state: _CascadeState) -> bool: def _cascade_query_backed( - *, models: List[SlayerModel], state: _CascadeState + *, models: list[SlayerModel], state: _CascadeState ) -> bool: """Rule 6: query-backed model whose source_queries chain transitively references dropped state — whole-drop.""" @@ -1478,14 +1507,14 @@ def _cascade_query_backed( def _cascade_one_pass( *, - models: List[SlayerModel], - models_by_name: Dict[str, SlayerModel], - edit_entries: Dict[str, EditModelDelete], - whole_entries: Dict[str, WholeModelDelete], - dropped_cols: Dict[str, Set[str]], - dropped_measures: Dict[str, Set[str]], - dropped_joins: Dict[str, Set[str]], - pk_per_model: Dict[str, Set[str]], + models: list[SlayerModel], + models_by_name: dict[str, SlayerModel], + edit_entries: dict[str, EditModelDelete], + whole_entries: dict[str, WholeModelDelete], + dropped_cols: dict[str, set[str]], + dropped_measures: dict[str, set[str]], + dropped_joins: dict[str, set[str]], + pk_per_model: dict[str, set[str]], ) -> bool: """Run a single cascade pass; return True if anything new was added. @@ -1524,13 +1553,13 @@ def _cascade_one_pass( def _seed_one_diff_entry( *, model_name: str, - entry: Optional[ToDeleteEntry], - cols: Set[str], - edit_entries: Dict[str, EditModelDelete], - whole_entries: Dict[str, WholeModelDelete], - dropped_cols: Dict[str, Set[str]], - dropped_measures: Dict[str, Set[str]], - dropped_joins: Dict[str, Set[str]], + entry: ToDeleteEntry | None, + cols: set[str], + edit_entries: dict[str, EditModelDelete], + whole_entries: dict[str, WholeModelDelete], + dropped_cols: dict[str, set[str]], + dropped_measures: dict[str, set[str]], + dropped_joins: dict[str, set[str]], ) -> None: """Apply one ``(entry, dropped_columns)`` diff result to the cascade state dicts.""" @@ -1552,14 +1581,14 @@ def _seed_one_diff_entry( def _seed_state_from_diffs( *, - diffs_iterables: Tuple[ - Dict[str, Tuple[Optional[ToDeleteEntry], Set[str]]], ... + diffs_iterables: tuple[ + dict[str, tuple[ToDeleteEntry | None, set[str]]], ... ], - edit_entries: Dict[str, EditModelDelete], - whole_entries: Dict[str, WholeModelDelete], - dropped_cols: Dict[str, Set[str]], - dropped_measures: Dict[str, Set[str]], - dropped_joins: Dict[str, Set[str]], + edit_entries: dict[str, EditModelDelete], + whole_entries: dict[str, WholeModelDelete], + dropped_cols: dict[str, set[str]], + dropped_measures: dict[str, set[str]], + dropped_joins: dict[str, set[str]], ) -> None: """Populate the cascade state dicts from the base per-model diffs.""" for diffs in diffs_iterables: @@ -1578,13 +1607,13 @@ def _seed_state_from_diffs( def _collapse_entries( *, - edit_entries: Dict[str, EditModelDelete], - whole_entries: Dict[str, WholeModelDelete], -) -> List[ToDeleteEntry]: + edit_entries: dict[str, EditModelDelete], + whole_entries: dict[str, WholeModelDelete], +) -> list[ToDeleteEntry]: """Apply the collapse rule (whole-drop preempts edit on the same model) and return the final, name-sorted list of delete entries. """ - final: List[ToDeleteEntry] = [] + final: list[ToDeleteEntry] = [] for name in sorted(set(edit_entries.keys()) | set(whole_entries.keys())): if name in whole_entries: final.append(whole_entries[name]) @@ -1595,21 +1624,21 @@ def _collapse_entries( def compute_datasource_drops( *, - models: List[SlayerModel], - sql_table_diffs: Dict[str, Tuple[Optional[ToDeleteEntry], Set[str]]], - sql_diffs: Dict[str, Tuple[Optional[ToDeleteEntry], Set[str]]], -) -> List[ToDeleteEntry]: + models: list[SlayerModel], + sql_table_diffs: dict[str, tuple[ToDeleteEntry | None, set[str]]], + sql_diffs: dict[str, tuple[ToDeleteEntry | None, set[str]]], +) -> list[ToDeleteEntry]: """Combine per-model base diffs with cascade walking and the collapse rule. Pure: takes pre-computed diffs as input and returns the final flat list. Caller is responsible for restricting ``models`` to a single datasource — cascade walking does not cross datasource boundaries. """ - edit_entries: Dict[str, EditModelDelete] = {} - whole_entries: Dict[str, WholeModelDelete] = {} - dropped_cols: Dict[str, Set[str]] = {} - dropped_measures: Dict[str, Set[str]] = {} - dropped_joins: Dict[str, Set[str]] = {} + edit_entries: dict[str, EditModelDelete] = {} + whole_entries: dict[str, WholeModelDelete] = {} + dropped_cols: dict[str, set[str]] = {} + dropped_measures: dict[str, set[str]] = {} + dropped_joins: dict[str, set[str]] = {} _seed_state_from_diffs( diffs_iterables=(sql_table_diffs, sql_diffs), @@ -1651,19 +1680,18 @@ def compute_datasource_drops( def _live_schema_for_datasource( *, datasource: DatasourceConfig, - schema: Optional[str] = None, -) -> Dict[str, LiveTable]: + schema: str | None = None, +) -> dict[str, LiveTable]: """Return ``{table_name: LiveTable}`` for every live table in the DS, using SQLAlchemy ``Inspector`` and the same fallback path as auto-ingestion (``slayer/engine/ingestion.py``). """ - sa_engine = sa.create_engine( - datasource.resolve_env_vars().get_connection_string() - ) + from slayer.sql import engine_factory + sa_engine = engine_factory.get_engine(datasource.resolve_env_vars()) try: inspector = sa.inspect(sa_engine) table_names = list(inspector.get_table_names(schema=schema)) - out: Dict[str, LiveTable] = {} + out: dict[str, LiveTable] = {} for table_name in table_names: try: out[table_name] = _introspect_one_table( @@ -1682,6 +1710,10 @@ def _live_schema_for_datasource( ) return out finally: + # Same rationale as ``ingest_datasource``: this is a one-shot + # admin path. Disposing releases the underlying connection so + # external direct file access (e.g. ``duckdb.connect(file)``) + # in the same process isn't blocked. sa_engine.dispose() @@ -1690,7 +1722,7 @@ def _introspect_one_table( inspector: sa.engine.Inspector, sa_engine: sa.Engine, table_name: str, - schema: Optional[str], + schema: str | None, ) -> LiveTable: """Build a ``LiveTable`` for one table via the existing safe-introspection path used by ``slayer/engine/ingestion.py``. @@ -1699,7 +1731,7 @@ def _introspect_one_table( pk = _safe_get_pk_constraint(inspector, sa_engine, table_name, schema) pk_columns = set(pk.get("constrained_columns", []) or []) - columns: Dict[str, DataType] = {} + columns: dict[str, DataType] = {} for col in cols_meta: col_type = col["type"] if isinstance(col_type, DataType): @@ -1710,7 +1742,7 @@ def _introspect_one_table( # collapse to NUMBER. _ = _sa_type_is_float(col_type) - fks: List[Tuple[str, str, str]] = [] + fks: list[tuple[str, str, str]] = [] try: for fk in inspector.get_foreign_keys(table_name, schema=schema): constrained = fk.get("constrained_columns") or [] @@ -1720,8 +1752,10 @@ def _introspect_one_table( if referred_table: fks.append((src, referred_table, tgt)) except Exception: - # Some dialects (ClickHouse, BigQuery, Snowflake) don't surface FK - # metadata. Skip silently — joins are still validated by name. + # Some dialects (ClickHouse, BigQuery) don't surface FK metadata + # via Inspector. Skip silently — joins are still validated by name. + # Snowflake DOES expose declarative FK constraints; see + # docs/configuration/datasources.md. pass return LiveTable(columns=columns, pk_columns=pk_columns, fk_relationships=fks) @@ -1741,7 +1775,7 @@ async def _live_columns_for_sql_model( *, model: SlayerModel, client: SlayerSQLClient, -) -> Optional[Dict[str, DataType]]: +) -> dict[str, DataType] | None: """Trial-execute ``model.sql`` with a 0-row guard and return cursor types. Returns ``None`` when the trial-execute itself fails — callers map that @@ -1778,31 +1812,257 @@ async def _live_columns_for_sql_model( # =========================================================================== +def _strip_ident_quotes(ident: str) -> str: + """Strip surrounding double-quotes from an SQL identifier and unescape + ``""`` → ``"``. Bare identifiers pass through unchanged. + """ + ident = ident.strip() + if len(ident) >= 2 and ident[0] == '"' == ident[-1]: + return ident[1:-1].replace('""', '"') + return ident + + def _resolve_live_table( - *, sql_table: str, live_tables: Dict[str, LiveTable] -) -> Optional[LiveTable]: + *, sql_table: str, live_tables: dict[str, LiveTable] +) -> LiveTable | None: """Look up a model's ``sql_table`` in the live introspection map, falling back to the bare name when the persisted value is schema- - qualified (``schema.table``). + qualified (``schema.table``) and unquoting double-quoted identifiers + (e.g. ``prod."Company"`` for case-sensitive Postgres tables). + """ + candidates = [sql_table] + if "." in sql_table: + candidates.append(sql_table.split(".", 1)[1]) + # Materialise the snapshot before extending — a bare generator + # ``(_strip_ident_quotes(c) for c in candidates)`` would iterate the + # list lazily WHILE ``extend`` appends to it, so every appended item + # gets re-fed into the iterator and the loop never terminates. + candidates.extend([_strip_ident_quotes(c) for c in candidates]) + for name in candidates: + live = live_tables.get(name) + if live is not None: + return live + return None + + +def _is_validate_models_base_column(col: Column) -> bool: + """Same base-column predicate as the storage refinement: ``col.sql`` + is None or a single bare identifier.""" + if col.type is not DataType.INT: + return False + if col.sql is None: + return True + s = col.sql.strip() + if not s or s[0].isdigit(): + return False + return all(c.isalnum() or c == "_" for c in s) + + +def _probe_validate_models_column( + *, conn, model: SlayerModel, col: Column, table_name: str, + schema_name: str | None, +) -> DataType | None: + """Run the affinity probe for one column in a validate_models pass.""" + from slayer.sql.sqlite_introspect import probe_sqlite_integer_column + try: + return probe_sqlite_integer_column( + conn=conn, + table=table_name, + column=col.sql or col.name, + schema=schema_name, + ) + except Exception as exc: + logger.warning( + "validate_models probe raised for %s.%s; ignoring: %s", + model.name, col.name, exc, + ) + return None + + +def _drift_reason_for_probe( + *, model: SlayerModel, col: Column, verdict: DataType, +) -> DeleteReason: + return DeleteReason( + target=f"column:{col.name}", + reason=( + f"SQLite affinity probe widened {model.name}.{col.name} " + f"from INT to {verdict.value}; re-run `slayer ingest` " + f"to recreate with the correct type." + ), + ) + + +def _sqlite_probe_int_drift_for_model( + *, + model: SlayerModel, + sa_engine, + default_schema: str | None = None, +) -> list[tuple[str, DataType, DeleteReason]]: + """DEV-1538: probe-driven type-drift detection on SQLite. + + For every persisted base column with ``Column.type == DataType.INT``, + open a connection from ``sa_engine`` and run + :func:`probe_sqlite_integer_column` against the live storage. When the + probe disagrees (verdict DOUBLE or TEXT), return a list of + ``(column_name, verdict, DeleteReason)`` tuples so the caller can + merge them into the model's diff state BEFORE the cascade fixed-point + walk fires. + + ``default_schema`` (typically ``datasource.schema_name``) is used as + the SQLite schema when ``model.sql_table`` is an unqualified table + name. Without this, attached SQLite schemas would silently fall back + to ``main`` and drift would be skipped or attributed to the wrong DB. + + Probe failures (``None`` verdict — explicit failure or saturated + sample) silently skip; the helper's own WARNING covers them. + """ + if model.sql_table is None: + return [] + + if "." in model.sql_table: + schema_name, _, table_name = model.sql_table.partition(".") + schema_name = schema_name or None + else: + schema_name, table_name = default_schema or None, model.sql_table + + drifts: list[tuple[str, DataType, DeleteReason]] = [] + with sa_engine.connect() as conn: + for col in model.columns: + if not _is_validate_models_base_column(col): + continue + verdict = _probe_validate_models_column( + conn=conn, model=model, col=col, + table_name=table_name, schema_name=schema_name, + ) + if verdict is None or verdict is DataType.INT: + continue + drifts.append( + (col.name, verdict, _drift_reason_for_probe( + model=model, col=col, verdict=verdict, + )) + ) + return drifts + + +async def _sqlite_probe_drifts_for_models( + *, + datasource: DatasourceConfig, + sql_table_models: list[SlayerModel], +) -> dict[str, list[tuple[str, DataType, DeleteReason]]]: + """Run the SQLite probe for every model in one synchronous worker so + the engine + connection lifecycle is shared across the validate pass.""" + if not sql_table_models: + return {} + if (datasource.type or "").lower() != "sqlite": + return {m.name: [] for m in sql_table_models} + + def _run() -> dict[str, list[tuple[str, DataType, DeleteReason]]]: + from slayer.sql import engine_factory + sa_engine = engine_factory.get_engine(datasource.resolve_env_vars()) + try: + out: dict[str, list[tuple[str, DataType, DeleteReason]]] = {} + default_schema = datasource.schema_name or None + for m in sql_table_models: + out[m.name] = _sqlite_probe_int_drift_for_model( + model=m, + sa_engine=sa_engine, + default_schema=default_schema, + ) + return out + finally: + # Cached engine — do not dispose; engine_factory owns lifecycle. + pass + + return await asyncio.to_thread(_run) + + +def _merge_probe_drifts_into_diff( + *, + model: SlayerModel, + base_diff: tuple[ToDeleteEntry | None, set[str]], + probe_drifts: list[tuple[str, DataType, DeleteReason]], +) -> tuple[ToDeleteEntry | None, set[str]]: + """Merge ``probe_drifts`` from + :func:`_sqlite_probe_int_drift_for_model` into a model's + ``(entry, dropped_columns)`` diff so the cascade fixed-point walk in + :func:`compute_datasource_drops` treats them as regular column drops. + + Skipped when ``base_diff`` is already a :class:`WholeModelDelete` + (the whole model is going anyway) or when there are no drifts. """ - live = live_tables.get(sql_table) - if live is None and "." in sql_table: - live = live_tables.get(sql_table.split(".", 1)[1]) - return live + if not probe_drifts: + return base_diff + base_entry, dropped = base_diff + if isinstance(base_entry, WholeModelDelete): + return base_diff + + drift_cols = [name for name, _, _ in probe_drifts] + drift_reasons = [reason for _, _, reason in probe_drifts] + + if base_entry is None: + merged_entry = EditModelDelete( + model_name=model.name, + data_source=model.data_source, + remove=RemoveSpec(columns=list(drift_cols)), + reasons=list(drift_reasons), + ) + else: + # base_entry is an EditModelDelete; append probe columns + reasons + # without dropping anything that was already there. + merged_columns = list(base_entry.remove.columns) + for c in drift_cols: + if c not in merged_columns: + merged_columns.append(c) + merged_entry = base_entry.model_copy( + update={ + "remove": base_entry.remove.model_copy( + update={"columns": merged_columns} + ), + "reasons": list(base_entry.reasons) + list(drift_reasons), + } + ) + + merged_dropped = set(dropped) | set(drift_cols) + return merged_entry, merged_dropped + + +def _diff_one_sql_table_model( + *, + model: SlayerModel, + live_tables: dict[str, "LiveTable"], + available_in_ds: set[str], + probe_drifts: list[tuple[str, DataType, DeleteReason]], +) -> tuple[ToDeleteEntry | None, set[str]]: + """Per-model body of :func:`_collect_sql_table_diffs` — resolves the + live table, runs ``diff_sql_table_model``, and merges any DEV-1538 + SQLite probe drifts so the cascade walk treats them as regular drops.""" + live = _resolve_live_table( + sql_table=model.sql_table or "", live_tables=live_tables, + ) + base = diff_sql_table_model( + model=model, + live_table=live, + available_models_in_ds=available_in_ds, + ) + return _merge_probe_drifts_into_diff( + model=model, base_diff=base, probe_drifts=probe_drifts, + ) async def _collect_sql_table_diffs( *, datasource: DatasourceConfig, - sql_table_models: List[SlayerModel], - available_in_ds: Set[str], -) -> Dict[str, Tuple[Optional[ToDeleteEntry], Set[str]]]: + sql_table_models: list[SlayerModel], + available_in_ds: set[str], +) -> dict[str, tuple[ToDeleteEntry | None, set[str]]]: """Run live SQLAlchemy introspection (off the event loop) and diff each - sql_table-mode model against it. + sql_table-mode model against it. On SQLite, additionally run the + DEV-1538 affinity probe per persisted INT base column and merge any + drift entries into each model's diff so the cascade fixed-point walk + sees them as regular column drops. """ - out: Dict[str, Tuple[Optional[ToDeleteEntry], Set[str]]] = {} if not sql_table_models: - return out + return {} # Honour the datasource's configured schema_name so non-default-schema # datasources diff against the right table set; otherwise SQLAlchemy # introspects the default and produces false WholeModelDeletes. @@ -1811,31 +2071,48 @@ async def _collect_sql_table_diffs( datasource=datasource, schema=datasource.schema_name or None, ) - for m in sql_table_models: - live = _resolve_live_table( - sql_table=m.sql_table or "", live_tables=live_tables - ) - out[m.name] = diff_sql_table_model( + probe_drifts_by_model = await _sqlite_probe_drifts_for_models( + datasource=datasource, + sql_table_models=sql_table_models, + ) + return { + m.name: _diff_one_sql_table_model( model=m, - live_table=live, - available_models_in_ds=available_in_ds, + live_tables=live_tables, + available_in_ds=available_in_ds, + probe_drifts=probe_drifts_by_model.get(m.name, []), ) - return out + for m in sql_table_models + } async def _collect_sql_diffs( *, datasource: DatasourceConfig, - sql_models: List[SlayerModel], - sql_clients: Optional[Dict[str, SlayerSQLClient]], -) -> Dict[str, Tuple[Optional[ToDeleteEntry], Set[str]]]: + sql_models: list[SlayerModel], + sql_clients: dict[tuple[str, str], SlayerSQLClient] | None, +) -> dict[str, tuple[ToDeleteEntry | None, set[str]]]: """Trial-execute each sql-mode model concurrently and produce its diff.""" - out: Dict[str, Tuple[Optional[ToDeleteEntry], Set[str]]] = {} + out: dict[str, tuple[ToDeleteEntry | None, set[str]]] = {} if not sql_models: return out - client = (sql_clients or {}).get(datasource.get_connection_string()) + # DEV-1551: SlayerQueryEngine._sql_clients is tuple-keyed + # (connection_string, runtime_fingerprint) so Snowflake datasources + # sharing a connection_name but differing in warehouse/role get + # distinct clients. Mirror that key shape here via the shared + # ``_sql_client_cache_key`` helper. + from slayer.engine.query_engine import _sql_client_cache_key # noqa: PLC0415 + key = _sql_client_cache_key(datasource) + client = (sql_clients or {}).get(key) if client is None: client = SlayerSQLClient(datasource=datasource) + # DEV-1656: cache the client back into the shared engine dict so the + # asyncpg pool it opens (trial-execute of sql-mode models) is + # reachable by ``SlayerQueryEngine.aclose()`` and disposed at task + # teardown. When ``sql_clients`` is None (no engine — direct/test + # callers own the lifecycle), behaviour is unchanged. + if sql_clients is not None: + sql_clients[key] = client async def _diff_one(model: SlayerModel) -> None: live_cols = await _live_columns_for_sql_model(model=model, client=client) @@ -1848,9 +2125,9 @@ async def _diff_one(model: SlayerModel) -> None: async def validate_datasource( *, datasource: DatasourceConfig, - models: List[SlayerModel], - sql_clients: Optional[Dict[str, SlayerSQLClient]] = None, -) -> List[ToDeleteEntry]: + models: list[SlayerModel], + sql_clients: dict[tuple[str, str], SlayerSQLClient] | None = None, +) -> list[ToDeleteEntry]: """Validate every persisted model in ``models`` (all in the same DS) against the live schema of ``datasource``. Read-only. """ diff --git a/slayer/engine/stage_planner.py b/slayer/engine/stage_planner.py index 309cd2d2..5014b197 100644 --- a/slayer/engine/stage_planner.py +++ b/slayer/engine/stage_planner.py @@ -32,6 +32,7 @@ from slayer.core.errors import ( AmbiguousReferenceError, UnknownReferenceError, + UnresolvableOrderColumnError, ) from slayer.core.keys import ( AggregateKey, @@ -50,9 +51,19 @@ normalize_scalar, ) from slayer.core.models import SlayerModel -from slayer.core.query import ModelExtension, SlayerQuery, TimeDimension +from slayer.core.query import ( + ORDER_PLACEHOLDER_NAMES, + ModelExtension, + SlayerQuery, + TimeDimension, +) from slayer.core.refs import agg_kwarg_canonical_str, canonical_agg_name +from slayer.core.time_bounds import strip_frame_bounds +from slayer.core.window_duration import parse_window_duration from slayer.core.scope import ModelScope, StageColumn, StageSchema +from slayer.engine.aggregate_input_paths import ( + compute_aggregate_input_join_paths, +) from slayer.engine.binding import ( BoundExpr as BinderBoundExpr, BoundFilter, @@ -73,8 +84,10 @@ FilterPhase, OrderEntry, PlannedQuery, + SrcFilterRewrite, TransformLayer, ValueSlot, + WindowedAggregatePlan, ) from slayer.engine.planning import ( DeclaredMeasure, @@ -83,6 +96,7 @@ _iter_slot_deps, filter_referenced_slot_ids, lower_sugar_transforms, + rewrite_rank_partition_keys, ) from slayer.engine.source_bundle import ( ResolvedSourceBundle, @@ -92,6 +106,7 @@ synthetic_model_from_stage_schema, ) from slayer.engine.syntax import parse_expr, parse_filter_expr +from slayer.sql.naming import flat_name from slayer.sql.sql_expr import has_window_function from slayer.sql.sql_predicate import parse_sql_predicate @@ -171,6 +186,27 @@ def _attach_time_keys( return key +def _partition_key_display(pk: ValueKey) -> str: + """Human-readable name of a rank ``partition_by`` key for error messages + (DEV-1497). Local refs surface as the bare leaf; joined refs keep the + dotted path.""" + if isinstance(pk, ColumnKey): + return ".".join([*pk.path, pk.leaf]) + if isinstance(pk, ColumnSqlKey): + return ".".join([*pk.path, pk.column_name]) + if isinstance(pk, TimeTruncKey): + return _partition_key_display(pk.column) + return str(pk) + + +def _row_key_path(key: ValueKey) -> tuple: + """Join path of a ROW value key (``()`` for local, non-empty for joined). + Unwraps a ``TimeTruncKey`` to its underlying column.""" + if isinstance(key, TimeTruncKey): + return _row_key_path(key.column) + return tuple(getattr(key, "path", ())) + + def _find_unresolved_time_needing_op(key: ValueKey) -> Optional[str]: """Return the op name of the first time-needing TransformKey reached that has ``time_key is None``, or ``None`` if every time-needing @@ -208,6 +244,257 @@ def _find_unresolved_time_needing_op(key: ValueKey) -> Optional[str]: return None +# --------------------------------------------------------------------------- +# DEV-1714 Stage 10 — duration-windowed measures (``window='90d'``). +# --------------------------------------------------------------------------- + + +def _window_kwarg_of(key: ValueKey): + """The ``window`` kwarg value of an ``AggregateKey``, or ``None``. + + ``window`` is a globally reserved aggregation kwarg name (legacy parity — + the enrichment pipeline pops it unconditionally before dispatch), so its + presence marks a windowed measure regardless of the aggregation. + """ + if isinstance(key, AggregateKey): + for k, v in key.kwargs: + if k == "window": + return v + return None + + +def _windowed_agg_keys(vk: ValueKey) -> list: + """Every windowed ``AggregateKey`` in ``vk``'s value-key tree.""" + return [k for k in walk_value_keys(vk) if _window_kwarg_of(k) is not None] + + +def _reject_unsupported_windowed_key(key: AggregateKey) -> None: + """Per-key guards shared by selected and filter/order-referenced windowed + aggregates: sum/avg-only (G1), string duration + compact syntax (G8), and + no cross-model source (G3). Raises with the pinned-message contract.""" + if key.agg not in ("sum", "avg"): + raise ValueError( + f"Aggregation parameter 'window' is only supported for sum and avg, " + f"not '{key.agg}'." + ) + window_val = _window_kwarg_of(key) + if not isinstance(window_val, str): + raise ValueError( + f"Window duration must be a compact duration string like '90d', got " + f"{window_val!r}. Use syntax like '1y2m3w5d6h7min8s'." + ) + parse_window_duration(window_val) # G8 — raises on empty / malformed + if getattr(key.source, "path", ()): # G3 + raise NotImplementedError( + "Windowed cross-model aggregates (e.g. customers.revenue:sum(" + "window='90d')) are not yet supported (DEV-1504)." + ) + + +def _guard_windowed_measures( # NOSONAR(S3776) — one cohesive ordered guard pass (G1→G8→G3→G4→G5→G7→G6→G2) over the original value-key trees; each branch is a distinct unsupported-shape rejection sharing the windowed-key scan, and splitting would scatter the precedence contract. + *, + measure_vks: list, + filter_vks: list, + order_vks: list, + active_td_key, +) -> dict: + """Reject unsupported windowed-measure shapes at plan time and return the + cleanly-SELECTED windowed ``AggregateKey``s (the ones that get a + ``WindowedAggregatePlan``) as an insertion-ordered mapping in measure + declaration order — so the emitted ``_wm_`` CTEs and combined-SELECT columns + are DETERMINISTIC (a set made the SQL output order vary across runs, which + breaks the SQL-text cache key of DEV-1587). + + Runs on the ORIGINAL declared-measure / filter / order value-key trees — + before projection interning would hide a transform / composite dependency + slot — so the transform (G4) and composite (G5) guards win over the + hidden-slot guard (G6). Precedence: G1 → G8 → G3 → G4 → G5 → G7 → G6 → G2. + """ + all_vks = [*measure_vks, *filter_vks, *order_vks] + if not any(_windowed_agg_keys(vk) for vk in all_vks): + return {} + + # G1 / G8 / G3 — per-key validation runs FIRST (documented precedence), so a + # windowed key with an invalid aggregation / malformed duration / cross-model + # source reports its specific error even when it is also wrapped in a + # transform (G4) or composite (G5). + for vk in all_vks: + for key in _windowed_agg_keys(vk): + _reject_unsupported_windowed_key(key) + + # G4 — a windowed measure cannot coexist with (or be the input of) any + # transform. Checked before the hidden-slot guard so + # ``cumsum(revenue:sum(window='90d'))`` reports 'transform', never 'selected'. + if any(isinstance(k, TransformKey) for vk in all_vks for k in walk_value_keys(vk)): + raise NotImplementedError( + "Windowed measures (window='…') combined with transforms are not yet " + "supported (DEV-1504). Compute the windowed measure in a separate " + "query stage." + ) + + # G5 — a top-level declared measure that IS a windowed AggregateKey is + # cleanly selected; a windowed key nested in an arithmetic / scalar composite + # measure is rejected. ``dict`` (not ``set``) preserves measure order; the + # value is the slot's ``hidden`` flag (DEV-1733) — False for a declared + # measure, True for an order-only target. + selected_windowed: dict = {} + for vk in measure_vks: + if not _windowed_agg_keys(vk): + continue + if _window_kwarg_of(vk) is not None: + selected_windowed.setdefault(vk, False) + else: + raise NotImplementedError( # G5 + "Windowed measures (window='…') inside arithmetic / composite / " + "scalar expressions are not yet supported (DEV-1504)." + ) + + # G7 (mixed) then G6 (hidden) for filter-referenced windowed measures. + for vk in filter_vks: + wkeys = _windowed_agg_keys(vk) + if not wkeys: + continue + has_plain_agg = any( + isinstance(k, AggregateKey) and _window_kwarg_of(k) is None + for k in walk_value_keys(vk) + ) + if has_plain_agg: + raise NotImplementedError( # G7 + "A single filter that mixes a windowed measure (window='…') with " + "a plain aggregate is not yet supported (DEV-1504)." + ) + if any(k not in selected_windowed for k in wkeys): + raise NotImplementedError( # G6 + "Filtering on a windowed measure (window='…') requires that " + "measure to also be selected (DEV-1504)." + ) + # DEV-1733 — order-only windowed targets. This IS a reachable shape (the + # pre-DEV-1733 comment here claimed otherwise): ``OrderItem`` canonicalises + # ``revenue:sum(window='90d')`` to the column name ``revenue_sum``, but + # ``OrderItem.raw_formula`` preserves the original text and the planner + # binds from it whenever the canonical name matches no declared measure. It + # used to fall through with no ``WindowedAggregatePlan``, materialising a + # PLAIN ``SUM`` in the base and ordering by it — the window silently gone. + # + # A windowed key referenced ONLY by ORDER BY is registered here as a HIDDEN + # plan (S-a top-level, S-b nested in a composite). Registering it after the + # measure loop means an also-declared key keeps ``hidden=False``. + for vk in order_vks: + for key in _windowed_agg_keys(vk): + selected_windowed.setdefault(key, True) + + # G2 — a windowed measure needs a resolvable time dimension. + if active_td_key is None: + raise ValueError( + "Windowed measure could not resolve its time dimension. Add a single " + "time_dimensions entry, or set main_time_dimension to select among " + "multiple time dimensions." + ) + return selected_windowed + + +def _windowed_grain_partition( + *, + row_slots: list, + active_td_slot_id, +) -> Tuple[list, list, list]: + """Split the PROJECTED (non-hidden) ROW slots into the ``_wm_`` grain roles. + + Returns ``(dim_slot_ids, other_td_slot_ids, grain_slot_ids)`` — plain + dimensions render as ``_w_dim_`` in the ``_src`` subquery, non-window + time dimensions as ``_w_td_``, and ``grain_slot_ids`` is the join-back + key order (dims, then the window TD, then the other TDs). + + HIDDEN row slots are excluded by design: the window buckets at the grain + the query actually projects, so an order-only (hidden) target must not + widen or narrow it. + """ + dim_slot_ids: list = [] + other_td_slot_ids: list = [] + for rs in row_slots: + if rs.hidden: + continue + if isinstance(rs.key, TimeTruncKey): + if rs.id != active_td_slot_id: + other_td_slot_ids.append(rs.id) + else: + dim_slot_ids.append(rs.id) + grain_slot_ids = ( + dim_slot_ids + + ([active_td_slot_id] if active_td_slot_id is not None else []) + + other_td_slot_ids + ) + return dim_slot_ids, other_td_slot_ids, grain_slot_ids + + +def _build_windowed_plans( + *, + selected_windowed: dict, + registry, + row_slots: list, + active_td_key, + active_td_slot_id, +) -> Tuple[list, set]: + """Build one ``WindowedAggregatePlan`` per selected windowed measure and + return ``(plans, windowed_slot_ids)``. The window time dimension is the + query's resolved main/active TD (same one first/last/time_shift use).""" + plans: list = [] + windowed_slot_ids: set = set() + if not selected_windowed: + return plans, windowed_slot_ids + + # CR#3 / G2 (post-projection): the window time dimension must be a SELECTED + # query time dimension (interned as a row slot so it becomes part of the + # bucket grain). A model ``default_time_dimension`` the query does not select + # resolves ``active_td_key`` (so the pre-projection G2 passes) but is never + # interned — ``active_td_slot_id`` is then None. Raise the G2 message rather + # than crash on the required ``window_time_dimension_slot_id: SlotId`` field. + if active_td_slot_id is None: + raise ValueError( + "Windowed measure could not resolve its time dimension. Add a single " + "time_dimensions entry, or set main_time_dimension to select among " + "multiple time dimensions." + ) + + dim_slot_ids, other_td_slot_ids, grain_slot_ids = _windowed_grain_partition( + row_slots=row_slots, active_td_slot_id=active_td_slot_id, + ) + + for key, is_hidden in selected_windowed.items(): + sid = registry.find_by_key(key) + if sid is None: + # CR#4: the guard pass already proved this is a cleanly-selected + # top-level windowed measure, so a missing slot is planner/projection + # drift — fail loudly rather than let the measure degrade to a plain + # (non-windowed) aggregate in the base (the silent-wrong-results mode + # the guards exist to prevent). + raise RuntimeError( + f"Windowed measure {key!r} was selected but has no projection " + f"slot; planner/projection drift (DEV-1714).", + ) + window_raw = _window_kwarg_of(key) + # DEV-1733: the slot is the authority on visibility — a key registered + # hidden by the order pass may still have been promoted to public by a + # declared measure sharing it (same key -> one slot). + slot = registry.get(sid) + hidden = bool(is_hidden and slot.hidden) + plans.append(WindowedAggregatePlan( + aggregate_slot_id=sid, + agg=key.agg, + window_raw=window_raw, + window_parts=parse_window_duration(window_raw), + window_time_dimension_slot_id=active_td_slot_id, + window_granularity=active_td_key.granularity, + dimension_slot_ids=dim_slot_ids, + other_time_dimension_slot_ids=other_td_slot_ids, + grain_slot_ids=grain_slot_ids, + hidden=hidden, + public_alias=None if hidden else slot.public_name, + )) + windowed_slot_ids.add(sid) + return plans, windowed_slot_ids + + def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-1503 addition is a small trigger-predicate branch + a kwarg pass-through; the function's pre-existing complexity is owned by the multi-stage scope / bundle / projection / filter-routing wiring it orchestrates and is tracked as a separate refactor. *, query: SlayerQuery, @@ -215,7 +502,7 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 scope: Optional[Union[ModelScope, StageSchema]] = None, cross_model_planner: Optional[CrossModelPlanner] = None, stage_schemas: Optional[Dict[str, StageSchema]] = None, - disable_dev1503_isolation: bool = False, + disable_host_rooted_isolation: bool = False, ) -> PlannedQuery: """Compile one ``SlayerQuery`` into a typed ``PlannedQuery``. @@ -224,13 +511,16 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 ``stage_schemas`` is a name → StageSchema map used by ``plan_stages`` to wire multi-stage references. - ``disable_dev1503_isolation`` (DEV-1503) suppresses the trigger that - isolates cross-model-FILTERED local measures into a per-measure CTE. + ``disable_host_rooted_isolation`` (DEV-1503; renamed and widened by + DEV-1709) suppresses the HOST-ROOTED half of the Law-3 trigger — the + isolation of a LOCAL aggregate whose ``Column.filter`` or any other + input (source ``Column.sql``, positional args, kwargs) crosses a join. + It never affects target-rooted isolation (``source.path`` non-empty). Threaded through ``subplan_builder`` whenever a cross-model strategy recurses for a host-rooted (or target-rooted) nested sub-plan, so the - sub-plan's same filtered measure is rendered inline (not infinitely + sub-plan's same crossing measure is rendered inline (not infinitely re-isolated) and so re-rooting of a genuine cross-model aggregate - doesn't redundantly DEV-1503-isolate the target's own filter joins. + doesn't redundantly isolate the target's own filter joins. """ stage_schemas = stage_schemas or {} cross_model_planner = ( @@ -362,9 +652,43 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 bound_filter_texts.append(f) order_specs = [] + # Host identity for the qualifier check below — the source model for a + # ``ModelScope``, the stage relation name (``s1``) for a downstream + # ``StageSchema`` (so a self-qualified ``s1.metric`` order stays host-local, + # Codex). Same resolution ``_host_model_name`` uses everywhere else. + _order_host_name = _host_model_name(scope) for o in (query.order or []): col_name = o.column.name full_name = o.column.full_name + # DEV-1733: a placeholder ColumnRef means the item is an EXPRESSION, + # not a column reference — bind ``raw_formula`` and skip the + # declared-alias lookups below (which could otherwise match a real + # model column that happens to share the sentinel's name). BOTH the + # sentinel AND a captured ``raw_formula`` are required, so a model with + # a genuine ``_expr_pending`` column, or a hand-built / deserialized + # ``OrderItem``, still resolves through the normal path. + if col_name in ORDER_PLACEHOLDER_NAMES and o.raw_formula: + order_specs.append(OrderSpec( + bound=bind_expr( + parsed=parse_expr(o.raw_formula, allow_dunder=flat_scope), + scope=scope, + bundle=bundle, + ), + direction=o.direction, + )) + continue + # An order ref qualified with a FOREIGN model (``owners.status`` when + # the host is ``orders``) must not resolve to a same-named local column + # via the bare-leaf shortcut — otherwise a joined sort key silently + # binds to the local column and sorts by the wrong field (Codex). The + # bare-name lookups below apply only to unqualified refs or refs + # qualified with the host itself; a foreign-qualified ref falls through + # to the dotted/flattened/`bind_expr` paths, where a truly-joined ref + # is then rejected by the plan-time order validation. + _order_qualifier = getattr(o.column, "model", None) + _order_host_local = ( + _order_qualifier is None or _order_qualifier == _order_host_name + ) # Prefer declared-measure alias resolution over model-scope # binding (DEV-1450 stage 7b.8 — gap fix): aggregate canonical # aliases like ``amount_sum`` are not columns on the model, so @@ -379,7 +703,7 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 # fall back to binding the preserved colon/path ``raw_formula`` # so the order key interns onto the same cross-model aggregate # slot (P2/P4) rather than raising. - if col_name in declared_alias_to_bound: + if _order_host_local and col_name in declared_alias_to_bound: bo = declared_alias_to_bound[col_name] elif full_name in declared_alias_to_bound: bo = declared_alias_to_bound[full_name] @@ -390,7 +714,7 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 # written in dotted form must intern onto that same declared # slot rather than binding the raw column as a fresh slot. bo = declared_alias_to_bound[_flatten_dotted(full_name)] - elif f"_{col_name}" in declared_alias_to_bound: + elif _order_host_local and f"_{col_name}" in declared_alias_to_bound: # ``*:count`` surfaces as the alias ``_count`` (the ``*`` is # dropped, the leading ``_`` kept as a marker); users naturally # order by the bare ``count``. Mirror the legacy @@ -538,9 +862,104 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 for spec in order_specs ] + # DEV-1497: validate that every rank-family ``partition_by`` column resolves + # to a query dimension / time-dimension, and rewrite a time-dimension source + # column to its truncated-bucket ``TimeTruncKey`` (partition by the bucket, + # not the raw timestamp — which would silently widen the grain). Runs BEFORE + # interning so a rewritten key never leaves a stale slot behind (identity is + # only touched on the rewritten rank transform). + _dim_dms = declared_measures[:n_dims] + _td_dms = declared_measures[n_dims:n_dims + n_tds] + _dim_key_set = {dm.bound.value_key for dm in _dim_dms} + # A source column carrying two time-dimension granularities (``created_at`` + # at both month and day) maps to two distinct ``TimeTruncKey`` buckets — a + # bare ``partition_by=created_at`` is then ambiguous, so track those columns + # and reject rather than silently pick whichever bucket comes last. + _td_by_source: Dict[ValueKey, TimeTruncKey] = {} + _td_ambiguous_sources: set = set() + for dm in _td_dms: + vk = dm.bound.value_key + if not isinstance(vk, TimeTruncKey): + continue + # Ambiguous only when the SAME source column already mapped to a + # DIFFERENT bucket (a different granularity) — two identical + # ``created_at:month`` declarations resolve to one bucket, not a clash. + if vk.column in _td_by_source and _td_by_source[vk.column] != vk: + _td_ambiguous_sources.add(vk.column) + _td_by_source[vk.column] = vk + _td_key_set = set(_td_by_source.values()) + _available_dims = [dm.declared_name for dm in (*_dim_dms, *_td_dms)] + + def _validate_partition_keys(tk: TransformKey) -> frozenset: + new_pks = [] + for pk in tk.partition_keys: + if pk in _dim_key_set or pk in _td_key_set: + new_pks.append(pk) # already a query dim / td bucket + elif pk in _td_ambiguous_sources: + raise ValueError( + f"Transform '{tk.op}': partition_by column " + f"'{_partition_key_display(pk)}' is ambiguous — it is a " + f"time dimension at multiple granularities. Partition by a " + f"single query dimension instead." + ) + elif pk in _td_by_source: + new_pks.append(_td_by_source[pk]) # td source col -> bucket + else: + raise ValueError( + f"Transform '{tk.op}': partition_by column " + f"'{_partition_key_display(pk)}' is not a query dimension. " + f"Add it to dimensions/time_dimensions, or choose one of: " + f"{', '.join(_available_dims) or '(none)'}." + ) + return frozenset(new_pks) + + def _rw(vk: ValueKey) -> ValueKey: + return rewrite_rank_partition_keys(vk, rewrite_fn=_validate_partition_keys) + + declared_measures = [ + DeclaredMeasure( + bound=BinderBoundExpr(value_key=_rw(dm.bound.value_key)), + declared_name=dm.declared_name, + public_name=dm.public_name, + label=dm.label, + canonical_alias=dm.canonical_alias, + type=dm.type, + format=dm.format, + description=dm.description, + ) + for dm in declared_measures + ] + _rewritten_filters = [] + for bf in bound_filters: + bf_vk = _rw(bf.value_key) + _rewritten_filters.append(BoundFilter( + value_key=bf_vk, + phase=bf.phase, + referenced_keys=tuple(walk_value_keys(bf_vk)), + )) + bound_filters = _rewritten_filters + order_specs = [ + OrderSpec( + bound=BinderBoundExpr(value_key=_rw(spec.bound.value_key)), + direction=spec.direction, + ) + for spec in order_specs + ] + source_col_names = _source_column_names(scope) host_model_name = _host_model_name(scope) + # DEV-1714 Stage 10 — windowed-measure guards on the ORIGINAL (pre- + # projection) value-key trees. Raises on unsupported shapes (non-sum/avg, + # no time dim, cross-model, transform, composite, hidden, mixed, malformed + # duration); returns the set of cleanly-selected windowed AggregateKeys. + selected_windowed = _guard_windowed_measures( + measure_vks=[dm.bound.value_key for dm in declared_measures], + filter_vks=[bf.value_key for bf in bound_filters], + order_vks=[sp.bound.value_key for sp in order_specs], + active_td_key=active_td_key, + ) + projection = ProjectionPlanner().plan( measures=declared_measures, filters=bound_filters, @@ -553,6 +972,90 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 projection.registry.slots, ) + # DEV-1714 Stage 10 — build one WindowedAggregatePlan per selected windowed + # measure. The window time dimension is the query's resolved active TD. + active_td_slot_id = ( + projection.registry.find_by_key(active_td_key) + if active_td_key is not None + else None + ) + windowed_plans, windowed_slot_ids = _build_windowed_plans( + selected_windowed=selected_windowed, + registry=projection.registry, + row_slots=row_slots, + active_td_key=active_td_key, + active_td_slot_id=active_td_slot_id, + ) + + # DEV-1714 (Codex round 5): a filter that references a windowed measure is + # reclassified WHOLE to Phase.POST (outer WHERE on the joined-back column). + # It must therefore reference ONLY windowed measures (+ literals). Mixing a + # windowed predicate with a row column or a plain aggregate in ONE filter + # can't be cleanly split — the windowed part is POST while the row part is a + # pre-aggregation WHERE — and would emit an outer-WHERE reference to an + # unprojected ``_base`` column. Reject it (the pre-projection G7 already + # catches the windowed+plain-aggregate half with a specific message; this + # also covers windowed+row-column, which G7's aggregate-only scan misses). + if windowed_slot_ids: + for bf in bound_filters: + refs = filter_referenced_slot_ids(bf, projection.registry) + if (refs & windowed_slot_ids) and (refs - windowed_slot_ids): + raise NotImplementedError( + "A single filter that mixes a windowed measure (window='…') " + "with another predicate (a row column or a plain aggregate) " + "is not yet supported (DEV-1504). Put them in separate " + "filters." + ) + + # DEV-1712 (Law 2): plan-time classification of every ORDER BY target that + # is not a declared/public slot. Order-only AGGREGATES (local or + # cross-model) always materialise and order — never rejected. The rest: + # * joined row column -> UnresolvableOrderColumnError (the sort + # scope is relocated where the joined table is unbound); + # * local row column, grouped -> ValueError (no valid SQL — the column + # isn't in GROUP BY; add it to dims, or order by an aggregate of it); + # * local row column, ungrouped -> allowed (split emission in the + # generator, ``_apply_order_limit_from_planned``); + # * transform / composite -> allowed since DEV-1733 (materialised + # hidden, ordered at the outer wrap, stripped from the projection). + _has_grouping = bool(agg_slots) or ( + bool(query.dimensions or query.time_dimensions) + and query.distinct_dimension_values + ) + for spec in order_specs: + okey = spec.bound.value_key + osid = projection.registry.find_by_key(okey) + if osid is not None and not projection.registry.get(osid).hidden: + continue # declared / projected output — orders on a real column + if isinstance(okey, AggregateKey): + continue # hidden aggregate (local base or cross-model CTE) + if isinstance(okey, (ColumnKey, ColumnSqlKey, TimeTruncKey)): + disp = _partition_key_display(okey) + path = _row_key_path(okey) + if path: + # ``UnresolvableOrderColumnError`` formats ``qualifier.column``; + # pass the bare leaf as ``column`` and the joined path as the + # qualifier so the message reads ``customers.regions.name`` and + # not a duplicated ``customers.customers.regions.name``. + raise UnresolvableOrderColumnError( + column=disp.rsplit(".", 1)[-1], qualifier=".".join(path), + ) + if _has_grouping: + raise ValueError( + f"ORDER BY column '{disp}' is a row column that this " + f"aggregated query does not project, so it is not in the " + f"GROUP BY. Add it to dimensions/time_dimensions, or order " + f"by an aggregate of it (e.g. '{disp}:max')." + ) + continue # ungrouped local row column -> split emission + # DEV-1733: TransformKey / ArithmeticKey / ScalarCallKey — an inline + # transform or composite expression referenced only in ORDER BY. These + # materialise as hidden slots (a step CTE on the transform path, a + # trimmed base-SELECT column on the no-transform path, an inline + # combined-SELECT term when an operand is cross-model or windowed) and + # order at the outer wrap. Stage 8 rejected them here; nothing left to + # reject. + # Build filters_by_phase in legacy WHERE order: # 1. date_range bound filters (bound_filters[:n_date_range]) # 2. model.filters (text_filter_entries) @@ -560,13 +1063,24 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 # bound_filter_ids preserves the mapping back to bound_filters for # the cross-model routing pass that follows (text_filter_entries # are excluded — model filters never feed cross-model routing). + # DEV-1714 Stage 10 — a filter referencing a windowed slot is reclassified + # to Phase.POST: the windowed value is computed in the ``_wm_`` CTE and + # joined back, so the predicate must apply on the combined SELECT (outer + # WHERE), never as a HAVING on the plain base aggregate. + def _windowed_phase(bf: BoundFilter) -> Phase: + if windowed_slot_ids and ( + filter_referenced_slot_ids(bf, projection.registry) & windowed_slot_ids + ): + return Phase.POST + return bf.phase + filters_by_phase: List[FilterPhase] = [] bound_filter_ids: List[str] = [] for i, bf in enumerate(bound_filters[:n_date_range]): fid = f"f{i}" filters_by_phase.append( FilterPhase( - id=fid, phase=bf.phase, text=None, + id=fid, phase=_windowed_phase(bf), text=None, expression=PlannedBoundExpr(value_key=bf.value_key), ), ) @@ -576,7 +1090,7 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 fid = f"f{i}" filters_by_phase.append( FilterPhase( - id=fid, phase=bf.phase, text=None, + id=fid, phase=_windowed_phase(bf), text=None, expression=PlannedBoundExpr(value_key=bf.value_key), ), ) @@ -608,23 +1122,47 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 cross_model_plans = [] host_slots_for_classifier = projection.registry.slots for slot in agg_slots: + # DEV-1714 Stage 10 — a windowed slot renders via its own ``_wm_`` CTE + # (host-rooted range join), never a cross-model ``_cm_`` CTE, even when + # its ``Column.filter`` crosses a join (which would otherwise trip the + # host-rooted isolation trigger below). + if slot.id in windowed_slot_ids: + continue key = slot.key if not isinstance(key, AggregateKey): continue agg_path = getattr(key.source, "path", ()) - # DEV-1503 — extended trigger predicate. Invoke the cross-model planner - # when the aggregate's source carries a non-empty join path (cross- - # model aggregate, existing behaviour) OR when the aggregate's - # ``column_filter_key`` references a non-anchor join path (cross- - # model-FILTERED local measure — the new filtered-local isolation - # case). The typed ``referenced_join_paths`` field is computed at - # binder time by ``compute_column_filter_join_paths``. - has_cross_model_filter = ( - not disable_dev1503_isolation - and key.column_filter_key is not None + # DEV-1503 / DEV-1709 — Law-3 trigger predicate. Invoke the + # cross-model planner when the aggregate's source carries a + # non-empty join path (target-rooted, existing behaviour) OR when + # ANY other input of a LOCAL aggregate crosses a join (host-rooted + # isolation): ``Column.filter`` (typed ``referenced_join_paths`` + # from binder time — DEV-1503), source ``Column.sql``, positional + # args incl. the explicit first/last time arg, kwargs (column + # refs, user template fragments, and non-overridden model-default + # ``AggregationParam`` fragments) — DEV-1709's widened trigger, + # computed plan-time by ``compute_aggregate_input_join_paths``. + has_crossing_filter = ( + key.column_filter_key is not None and bool(key.column_filter_key.referenced_join_paths) ) - if not agg_path and not has_cross_model_filter: + has_crossing_input = ( + not disable_host_rooted_isolation + and not agg_path + and ( + has_crossing_filter + or bool(compute_aggregate_input_join_paths( + key=key, + anchor_model=bundle.source_model, + anchor_relation=( + bundle.source_model.name + if bundle.source_model is not None else "" + ), + bundle=bundle, + )) + ) + ) + if not agg_path and not has_crossing_input: continue # DEV-1450 #2: re-rooting (C1) is owned by the strategy. We hand it # the host query, the public projection, and a sub-plan builder so it @@ -634,12 +1172,14 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 # same ``plan_query`` recursion the post-hoc pass used, injected here # so cross_model_planner.py needn't import stage_planner. # - # DEV-1503 — the subplan_builder ALWAYS suppresses DEV-1503 isolation: - # for filtered-local isolation, the host-rooted sub-plan contains the - # same filtered measure and would otherwise recurse infinitely; for - # the existing cross-model re-rooting case, the sub-plan's target- - # rooted local aggregate would redundantly DEV-1503-isolate its own - # filter joins (already handled by the surrounding cross-model CTE). + # DEV-1503 / DEV-1709 — the subplan_builder ALWAYS suppresses + # host-rooted isolation: the host-rooted sub-plan contains the same + # crossing measure and would otherwise recurse infinitely; for the + # existing cross-model re-rooting case, the sub-plan's target-rooted + # local aggregate would redundantly isolate its own crossing inputs + # (already handled by the surrounding cross-model CTE). Inside the + # sub-plan, crossing inputs render INLINE (base-pull) — legal there + # because the CTE is the aggregate's own scope. reroot_enabled = ( isinstance(scope, ModelScope) and scope.source_model is not None ) @@ -658,7 +1198,7 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 subplan_builder=( (lambda q, b: plan_query( query=q, bundle=b, cross_model_planner=cross_model_planner, - disable_dev1503_isolation=True, + disable_host_rooted_isolation=True, )) if reroot_enabled else None ), @@ -668,10 +1208,25 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 order_entries = [] for spec in order_specs: sid = projection.registry.find_by_key(spec.bound.value_key) - if sid is not None: - order_entries.append( - OrderEntry(slot_id=sid, direction=spec.direction), + if sid is None: + # DEV-1733: an order target that reached here without a slot would + # be SILENTLY DROPPED — the query runs unsorted and returns wrong + # rows with no error. That was the original `change(...)` / + # scalar-call bug, and the entry-point relaxation makes new key + # shapes reachable (e.g. a top-level `IN` / `BETWEEN` predicate, + # which `_iter_slot_deps` treats as WHERE-inlined and never slots). + # Fail loudly for ANY unslotted shape rather than enumerating them, + # so this whole bug class cannot come back. + raise ValueError( + f"ORDER BY expression is not supported: " + f"{type(spec.bound.value_key).__name__} has no materialisable " + f"slot. Order by an aggregate, a transform, a composite " + f"arithmetic / scalar expression, a dimension, or declare the " + f"expression as a measure and order by its name." ) + order_entries.append( + OrderEntry(slot_id=sid, direction=spec.direction), + ) transform_layers = _emit_transform_layers(slots=projection.registry.slots) stage_schema = _emit_stage_schema( @@ -683,21 +1238,37 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 else host_model_name ) - # Stage 7b.10 — surface the active TD's slot id so the generator can - # render ``ORDER BY `` in OVER clauses without re-walking - # the model graph. ``None`` when there is no TD (validation already - # ran above; we only reach here if no time-needing transform exists). - active_td_slot_id = ( - projection.registry.find_by_key(active_td_key) - if active_td_key is not None - else None - ) + # Stage 7b.10 — the active TD's slot id (``active_td_slot_id``) is resolved + # right after projection above so the windowed-plan builder can use it. + + # DEV-1732 — the frame-bound column set: raw columns of this stage's + # NON-HIDDEN time dimensions. Computed once and carried on the plan so the + # windowed ``_src`` path (below) and the generator's ``time_shift`` + # shifted-CTE path read the SAME set. + frame_bound_columns = _frame_bound_columns(row_slots=row_slots) + + # DEV-1714 Stage 10 / DEV-1732 — the ``_wm_`` ``_src`` scope inherits + # WHERE-phase row filters (model + user) MINUS their frame bounds: the + # trailing window must reach rows before the visible frame starts. + # POST-reclassified windowed-measure filters are already excluded + # (phase != ROW). + if windowed_plans: + date_range_fids = {f"f{i}" for i in range(n_date_range)} + src_where_ids, src_rewrites = _plan_src_row_filters( + filters_by_phase=filters_by_phase, + date_range_fids=date_range_fids, + frame_bound_columns=frame_bound_columns, + ) + for wp in windowed_plans: + wp.where_filter_ids = src_where_ids + wp.src_filter_rewrites = src_rewrites return PlannedQuery( source_relation=source_relation, row_slots=row_slots, aggregate_slots=agg_slots, cross_model_aggregate_plans=cross_model_plans, + windowed_aggregate_plans=windowed_plans, combined_expression_slots=combined_slots, transform_layers=transform_layers, filters_by_phase=filters_by_phase, @@ -708,9 +1279,89 @@ def plan_query( # NOSONAR(S3776) — planner entry-point dispatcher. The DEV-15 stage_schema=stage_schema, active_time_dimension_slot_id=active_td_slot_id, render_source_model=render_source_model, + distinct_dimension_values=query.distinct_dimension_values, + frame_bound_columns=frame_bound_columns, ) +def _frame_bound_columns(*, row_slots: list) -> List[ValueKey]: + """Raw column keys of the stage's NON-HIDDEN time dimensions (DEV-1732). + + An explicit relational bound on one of these is a FRAME bound — the + caller restating what ``TimeDimension.date_range`` expresses — and is + stripped from CTEs that must read outside the frame. A bound on any other + column, temporal or not, is a population filter and is left alone. + + Hidden ``TimeTruncKey`` slots are excluded deliberately, and the exclusion + is load-bearing rather than cosmetic: ``_build_windowed_plans`` skips hidden + row slots when building ``other_time_dimension_slot_ids``, so a hidden time + axis is never equality-joined into ``_src``. Stripping a bound on one would + leave that axis wholly unconstrained — an unbounded over-count — where + keeping it merely preserves the pre-DEV-1732 result. + + Order-stable and de-duplicated: the same column carried at two + granularities contributes one entry. + """ + out: List[ValueKey] = [] + seen: set = set() + for rs in row_slots: + if rs.hidden or not isinstance(rs.key, TimeTruncKey): + continue + col = rs.key.column + if col in seen: + continue + seen.add(col) + out.append(col) + return out + + +def _plan_src_row_filters( + *, + filters_by_phase: list, + date_range_fids: set, + frame_bound_columns: List[ValueKey], +) -> "Tuple[List[str], List[SrcFilterRewrite]]": + """Partition ROW-phase filters for a windowed measure's ``_src`` scope. + + Returns ``(where_filter_ids, src_filter_rewrites)``: + + * a filter that is ENTIRELY a frame bound is omitted from the ids; + * a filter that is PARTLY one keeps its id and gains a rewrite carrying the + residual population predicate; + * everything else keeps its id with no rewrite. + + Mode-A model filters (``FilterPhase.text``, no typed expression) are exempt + by design — a model filter defines which rows EXIST rather than which frame + the query looks at, there is no ``date_range`` spelling at model level, and + analysing raw dialect SQL would make a silent mis-strip possible. + + ``date_range_fids`` is skipped up front. That is redundant with + ``strip_frame_bounds`` (which recognises ``BetweenKey`` too) and kept + deliberately: it makes a Stage-10 regression structurally impossible even if + a ``date_range`` ever binds to a shape the helper does not match. + """ + time_cols = frozenset(frame_bound_columns) + where_ids: List[str] = [] + rewrites: List[SrcFilterRewrite] = [] + for fp in filters_by_phase: + if fp.phase != Phase.ROW or fp.id in date_range_fids: + continue + if fp.expression is None: + where_ids.append(fp.id) # Mode-A model filter — exempt. + continue + residual = strip_frame_bounds( + key=fp.expression.value_key, time_columns=time_cols, + ) + if residual is None: + continue # wholly a frame bound + where_ids.append(fp.id) + if residual is not fp.expression.value_key: + rewrites.append(SrcFilterRewrite( + filter_id=fp.id, expression=PlannedBoundExpr(value_key=residual), + )) + return where_ids, rewrites + + def _coerce_extension(spec) -> ModelExtension: """Coerce a ``ModelExtension`` / dict-with-``source_name`` to a typed ``ModelExtension`` (for overlaying onto a synthetic sibling model).""" @@ -852,7 +1503,9 @@ def _format_description_for_dimension( return col.format, col.description -_COUNT_AGGREGATIONS: FrozenSet[str] = frozenset({"count", "count_distinct"}) +_COUNT_AGGREGATIONS: FrozenSet[str] = frozenset( + {"count", "count_distinct", "count_distinct_approx"} +) _FLOAT_AGGREGATIONS: FrozenSet[str] = frozenset({ "avg", "weighted_avg", "median", "stddev_samp", "stddev_pop", "var_samp", "var_pop", @@ -870,7 +1523,7 @@ def _infer_aggregated_type( ``_infer_aggregated_format`` (decision #2 of the Stage B plan): * ``*:count`` (measure_name=``"*"``) → ``INT`` - * ``count`` / ``count_distinct`` → ``INT`` + * ``count`` / ``count_distinct`` / ``count_distinct_approx`` → ``INT`` * ``avg`` / ``weighted_avg`` / ``median`` / parametric / stat aggs → ``DOUBLE`` * ``sum`` / ``min`` / ``max`` / ``first`` / ``last`` → inherit from @@ -1021,6 +1674,62 @@ def _type_for_dimension( return col.type if col is not None else None +def _opaque_dim_type( + *, + scope: Union[ModelScope, StageSchema], + full_name: str, + bundle: ResolvedSourceBundle, +) -> Optional[DataType]: + """Declared type of a query dimension, for the opaque-grouping guard only. + + Resolves BOTH a ``ModelScope`` origin (via ``_type_for_dimension``) AND a + downstream ``StageSchema`` (via its typed ``columns``). ``_type_for_dimension`` + deliberately returns ``None`` for a StageSchema — that ``None`` is load-bearing + for downstream typing (``_query_as_model`` coercion) and must not change — so + the guard needs its own resolver to catch an opaque column projected in one + stage and grouped in the next. + """ + if isinstance(scope, StageSchema): + col = scope.get(full_name) + return col.type if col is not None else None + return _type_for_dimension(scope=scope, full_name=full_name, bundle=bundle) + + +def _reject_opaque_grouping_dim( + *, + query: SlayerQuery, + scope: Union[ModelScope, StageSchema], + full_name: str, + bundle: ResolvedSourceBundle, +) -> None: + """Raise if ``full_name`` is an opaque dimension this query will GROUP BY. + + Grouping by an opaque column (``DataType.UNKNOWN`` — e.g. a PostGIS ``point`` + or any type with no equality operator) emits SQL the database rejects, so we + fail with an actionable message instead of a raw driver error. Only an + *actually grouped* dimension is refused: aggregating queries and dim-only + DISTINCT queries group, but raw-row mode (``distinct_dimension_values=False`` + with no measures, DEV-1543) projects dimensions without a top-level GROUP BY, + so an opaque column is legal there — and a downstream stage that groups such a + projected value is still caught via the StageSchema (see ``_opaque_dim_type``). + Checked on the declared type *before* ``bind_expr`` expands the column's + ``sql``, so an opaque *derived* column is caught by its type rather than + tripping the DEV-1410 cycle check first. (PR #259 "Unknown type" main-parity: + the legacy guard lived in ``enrichment._resolve_dimensions``, which the typed + pipeline bypasses.) + """ + if not (bool(query.measures) or query.distinct_dimension_values): + return + dim_type = _opaque_dim_type(scope=scope, full_name=full_name, bundle=bundle) + if dim_type is not None and dim_type.is_opaque: + raise ValueError( + f"Column '{full_name}' cannot be used as a dimension: its type does " + f"not support the GROUP BY / DISTINCT this query requires. Define a " + f"derived column that extracts a comparable value instead, e.g. " + f"sql=\"payload->>'status'\" with type TEXT." + ) + + def _saved_model_measure_type( *, scope: Union[ModelScope, StageSchema], formula: str, ) -> Optional[DataType]: @@ -1050,6 +1759,29 @@ def _saved_model_measure_type( return saved.type if saved is not None else None +def _bare_saved_measure_name( + *, scope: Union[ModelScope, StageSchema], formula: str, +) -> Optional[str]: + """The saved ``ModelMeasure.name`` when the query formula is a BARE + reference to one (DEV-1713 / DEV-1495 bare-named-measure aliasing). + + ``expand_model_measures`` rewrites ``rev_total`` to the saved measure's + underlying formula AST, so without this the measure would surface under + the formula-derived canonical (``revenue_sum``) instead of the name the + user referenced (``rev_total``). Fires ONLY for a bare identifier matching + a ``ModelMeasure.name`` on the source model — the same gate as + :func:`_saved_model_measure_type`; qualified / arithmetic / colon-suffix + formulas fall through (name-preservation is scoped to the bare form). + """ + if not isinstance(scope, ModelScope) or scope.source_model is None: + return None + bare = formula.strip() + if not bare.isidentifier(): + return None + saved = scope.source_model.get_measure(bare) + return saved.name if saved is not None else None + + def _declared_measures_from_query( *, query: SlayerQuery, @@ -1064,6 +1796,9 @@ def _declared_measures_from_query( declared: List[DeclaredMeasure] = [] for d in (query.dimensions or []): full = d.full_name + _reject_opaque_grouping_dim( + query=query, scope=scope, full_name=full, bundle=bundle, + ) bound = bind_expr( parsed=parse_expr(full, allow_dunder=flat_scope), scope=scope, @@ -1122,8 +1857,13 @@ def _declared_measures_from_query( # (DEV-1446) still holds — ``lower_sugar_transforms`` keeps the # inner ``AggregateKey`` instance unchanged. canonical = _canonical_alias_for_formula(formula, bound=bound) - declared_name = explicit_name or canonical - public_name = explicit_name or canonical + # DEV-1713: a bare reference to a saved ModelMeasure surfaces under the + # measure NAME, not the formula-derived canonical. Explicit query + # ``name`` still wins; the saved name is an implicit ``name``. + saved_name = _bare_saved_measure_name(scope=scope, formula=formula) + alias_name = explicit_name or saved_name + declared_name = alias_name or canonical + public_name = alias_name or canonical fmt, desc = _format_description_for_measure_formula( scope=scope, bound=bound, ) @@ -1147,7 +1887,10 @@ def _declared_measures_from_query( declared_name=declared_name, public_name=public_name, label=m.label, - canonical_alias=canonical if explicit_name else None, + # DEV-1443: keep the canonical alias whenever the surfaced name + # differs from it (explicit ``name`` OR an implicit saved-measure + # name) so a colon-form filter / ORDER BY still resolves. + canonical_alias=canonical if alias_name else None, type=m_type, format=fmt, description=desc, @@ -1209,7 +1952,8 @@ def _topo_sort(queries: List[SlayerQuery]) -> List[SlayerQuery]: def _flatten_dotted(name: str) -> str: - return name.replace(".", "__") + # DEV-1713: the ``__``-flatten is owned by the naming module. + return flat_name(name) def _canonical_alias_for_formula( diff --git a/slayer/engine/syntax.py b/slayer/engine/syntax.py index 5c34b019..e604d70d 100644 --- a/slayer/engine/syntax.py +++ b/slayer/engine/syntax.py @@ -159,6 +159,31 @@ class BoolOp(_BaseNode): # (Codex). Used by ``_normalize_sql_filter_operators``, ``_preprocess_colons``, # and the raw-``OVER(`` pre-scan. _PY_STRING_LITERAL_RE = re.compile(r"'(?:\\.|[^'\\])*'|\"(?:\\.|[^\"\\])*\"") +# SQL ``LIKE`` / ``NOT LIKE`` operator → the ``like(col, pattern)`` scalar the +# Mode-B DSL already accepts (DEV-1484 emits it as SQL ``LIKE``). Mirrors +# ``formula._preprocess_like`` so the typed filter parser accepts the same LIKE +# spelling as the documented Mode-B filter grammar — a pg-facade WHERE +# ``col LIKE 'p%'`` (or ``NOT LIKE``) lands here as a verbatim filter (DEV-1704). +# LHS is a bare/dotted identifier or a single scalar call; RHS a quoted pattern. +# Applied to the whole expression (the pattern is a string literal), same as +# ``formula._preprocess_like``. +_SQL_LIKE_RE = re.compile( + r"\b(\w+\([^()]*\)|(?:\w+\.)*\w+)\s+(not\s+)?like\s+('[^']*')", + flags=re.IGNORECASE, +) + + +def _rewrite_sql_like(text: str) -> str: + """``col LIKE 'p%'`` → ``like(col, 'p%')``; ``col NOT LIKE 'p%'`` → + ``not like(col, 'p%')`` — outside/inside handling matches + ``formula._preprocess_like``.""" + + def _sub(m: "re.Match[str]") -> str: + lhs, neg, pat = m.group(1), m.group(2), m.group(3) + call = f"like({lhs}, {pat})" + return f"not {call}" if neg else call + + return _SQL_LIKE_RE.sub(_sub, text) _COLON_AGG_RE = re.compile( r"(\*|[a-zA-Z_]\w*(?:\.[a-zA-Z_]\w*)*(?:\.\*)?)" # source: * / ident / dotted r":" @@ -283,10 +308,14 @@ def _normalize_sql_filter_operators(text: str) -> str: """Rewrite SQL operator spellings to Python ones outside string literals. ``NULL`` → ``None``; ``IS`` / ``NOT`` / ``AND`` / ``OR`` / ``IN`` → - lowercase; standalone ``=`` → ``==``; ``<>`` → ``!=``. Replicated from the - legacy ``slayer.core.formula._preprocess_sql_operators`` so the typed - pipeline doesn't depend on the module DEV-1452 deletes. + lowercase; standalone ``=`` → ``==``; ``<>`` → ``!=``; ``col [NOT] LIKE + 'p%'`` → ``[not ]like(col, 'p%')``. Replicated from the legacy + ``slayer.core.formula._preprocess_sql_operators`` / ``_preprocess_like`` so + the typed pipeline doesn't depend on the module DEV-1452 deletes. """ + # LIKE runs first, on the whole string: its pattern is a quoted literal, so + # it can't be rewritten per-non-literal-part like the other operators. + text = _rewrite_sql_like(text) # CR review: use the escape-aware Python-string matcher so backslash- # escaped quotes don't leak ``IS`` / ``IN`` / ``AND`` rewrites into # the string body (``"x \" IN ("``). diff --git a/slayer/engine/timing.py b/slayer/engine/timing.py new file mode 100644 index 00000000..1f6eb6df --- /dev/null +++ b/slayer/engine/timing.py @@ -0,0 +1,85 @@ +"""Opt-in per-stage timing for the query hot path (facade perf debugging). + +Set ``SLAYER_PROFILE_TIMING`` to a truthy value (``1`` / ``true`` / ``yes`` / +``on``) to enable. Disabled by default so production logs stay quiet. + +When enabled, wrap a request in :func:`open_query_profile` and bracket +sub-stages with :func:`start` / :func:`record`; one summary line is logged at +INFO on scope exit, e.g.:: + + slayer.timing query: resolve_model=2.1 resolve_datasource=1.4 enrich=0.8 \ + generate_sql=3.2 execute=774.1 connect=254.6 set_timeout=251.1 \ + query=268.4 total=782.9 ms + +Overhead when disabled: one env-var read when a scope would open, and one +``ContextVar`` read per ``start`` / ``record`` — no timestamps are taken and +nothing is logged. +""" + +from __future__ import annotations + +import contextlib +import logging +import os +import time +from contextvars import ContextVar + +logger = logging.getLogger(__name__) + +_TRUTHY = frozenset({"1", "true", "yes", "on"}) + +# Active collector: ordered ``[(stage_name, elapsed_ms), ...]``. ``None`` when +# no profile scope is open in the current context — the common (and disabled) +# case, which every ``start`` / ``record`` short-circuits on. +_collector: ContextVar[list[tuple[str, float]] | None] = ContextVar( + "slayer_timing_collector", default=None +) + + +def timing_enabled() -> bool: + """True when ``SLAYER_PROFILE_TIMING`` is set to a truthy value.""" + return os.environ.get("SLAYER_PROFILE_TIMING", "").strip().lower() in _TRUTHY + + +@contextlib.contextmanager +def open_query_profile(label: str = "query"): + """Open a timing scope; emit one summary line on exit. No-op when disabled. + + Safe to wrap an ``await`` — the ``ContextVar`` propagates to coroutines + awaited within the same task, so nested ``record`` calls in the engine and + SQL client land in this scope's collector. + """ + if not timing_enabled(): + yield + return + token = _collector.set([]) + started = time.perf_counter() + try: + yield + finally: + total_ms = (time.perf_counter() - started) * 1000.0 + stages = _collector.get() or [] + _collector.reset(token) + rendered = " ".join(f"{name}={ms:.1f}" for name, ms in stages) + logger.info("slayer.timing %s: %s total=%.1f ms", label, rendered, total_ms) + + +def start() -> float | None: + """Return a start timestamp when a profile is open, else ``None``. + + The ``None`` fast-path keeps ``start`` / ``record`` free when profiling is + off or the call site runs outside any :func:`open_query_profile` scope. + """ + return time.perf_counter() if _collector.get() is not None else None + + +def record(name: str, started: float | None) -> None: + """Record elapsed ms for ``name`` since ``started`` into the open profile. + + No-op when ``started`` is ``None`` (profiling off / no scope open). + """ + if started is None: + return + collector = _collector.get() + if collector is not None: + collector.append((name, (time.perf_counter() - started) * 1000.0)) diff --git a/slayer/facade/catalog.py b/slayer/facade/catalog.py index f105ada6..6b9bb3d4 100644 --- a/slayer/facade/catalog.py +++ b/slayer/facade/catalog.py @@ -15,18 +15,19 @@ from __future__ import annotations import logging -from typing import Dict, FrozenSet, List, Optional, Set, Tuple -from pydantic import BaseModel +from pydantic import BaseModel, ConfigDict, Field from slayer.core.enums import ( DEFAULT_AGGREGATIONS_BY_TYPE, PRIMARY_KEY_AGGREGATIONS, DataType, + JoinType, ) from slayer.core.models import ( Aggregation, Column, + ModelJoin, SlayerModel, ) from slayer.facade.datatypes import SUPPORTED_DATATYPES @@ -38,7 +39,7 @@ # expansion (§5.1 rule 3) and the custom-agg expansion (rule 4) both # skip these for built-ins. Custom aggs with non-empty ``params`` are # also skipped per rule 4 for the same reason. -_PARAMETRIC_BUILTIN_AGGS: FrozenSet[str] = frozenset({ +_PARAMETRIC_BUILTIN_AGGS: frozenset[str] = frozenset({ "weighted_avg", "percentile", "corr", "covar_samp", "covar_pop", }) @@ -48,42 +49,131 @@ class FacadeMetric(BaseModel): name: str - description: Optional[str] = None - label: Optional[str] = None - data_type: Optional[DataType] = None + description: str | None = None + label: str | None = None + data_type: DataType | None = None measure_formula: str class FacadeDimension(BaseModel): name: str - description: Optional[str] = None - label: Optional[str] = None + description: str | None = None + label: str | None = None data_type: DataType is_time: bool dimension_ref: str +class FacadeJoin(BaseModel): + """A direct (single-hop) join from a parent model to a target model. + + Used by the wire-facade translator (DEV-1565) to recognise BI-tool-emitted + LEFT JOIN-with-subquery shapes against the parent's configured joins. + Only joins whose target is a non-hidden model in the same catalog are + exposed, mirroring the BFS dim/metric filter. + """ + target_model: str + join_pairs: list[list[str]] + join_type: JoinType = JoinType.LEFT + + class FacadeTable(BaseModel): + # arbitrary_types_allowed lets `model_ref` carry the in-memory + # SlayerModel handle without Pydantic deep-copying / re-validating it. + model_config = ConfigDict(arbitrary_types_allowed=True) + name: str table_type: str - description: Optional[str] = None - metrics: List[FacadeMetric] - dimensions: List[FacadeDimension] + description: str | None = None + metrics: list[FacadeMetric] + dimensions: list[FacadeDimension] + joins: list[FacadeJoin] = Field(default_factory=list) + # In-memory handle to the underlying SlayerModel — required by the + # translator for ON-clause column validation (hidden FK/PK columns + # don't appear on `dimensions`) and for the dynamic-join lookup + # materialisation (DEV-1565). Excluded from any future serialisation + # of FacadeCatalog. + model_ref: SlayerModel | None = Field(default=None, exclude=True) class FacadeSchema(BaseModel): name: str - tables: List[FacadeTable] + tables: list[FacadeTable] class FacadeCatalog(BaseModel): catalog_name: str = CATALOG_NAME - schemas: List[FacadeSchema] + schemas: list[FacadeSchema] + + +def local_metrics(table: FacadeTable) -> list[FacadeMetric]: + """Metrics that should appear in the per-table flat-column view + (``pg_attribute`` / ``INFORMATION_SCHEMA.COLUMNS``) — saved + ``ModelMeasure`` entries only. + + DEV-1567: ``_metric_expansion`` produces three kinds of entries on + every table: + + 1. **Cross-model entries** — names like ``customers.row_count`` / + ``customers.regions.population_sum`` produced by joining sibling + models' metrics under a dotted prefix. + 2. **Synthetic same-model entries** — the ``row_count`` rule-1 + metric (``measure_formula="*:count"``), the column × built-in- + aggregation cartesian (``_`` / ``:``), and + the column × custom-aggregation cartesian. None of these are + user-authored — they're catalog fan-out so BI tools can pick + any column × any agg via colon-form resolution. + 3. **Saved measures** — ``ModelMeasure`` entries where the catalog + sets ``name == measure_formula`` (the user named them, so the + formula IS the name). + + BI tools (Metabase, dbt schema scan, pgjdbc clients) that flatten + the catalog through ``pg_attribute`` then discover every dimension + AND every metric as a projectable "column" of the parent table and + emit ``SELECT *``-style queries listing them all. Metabase's MBQL + fingerprint pass then wraps each one in ``COUNT(...)``/``MAX(...)``, + landing dotted names in ``SlayerQuery.measures[*].name`` (Pydantic + rejects them) or exploding the wire response width (the C.1 e2e + test asserts ``len(cols) == 7``, the count of ``orders``' user- + authored columns). + + Both kinds (1) and (2) are stripped here. Saved measures (kind 3) + stay because the user named them — they ARE the model's queryable + surface. + + The raw ``table.metrics`` list keeps all three kinds so: + * ``INFORMATION_SCHEMA.METRICS`` still exposes them as the + catalog-namespaced answer to "what can I aggregate?"; + * the catalog-SQL fingerprint hash still tracks them for cache + invalidation; + * the translator's ``metrics_by_name`` / ``metrics_by_formula`` + lookups still resolve hand-written cross-model SQL (rejected + there by the translator-side guard) and same-model aggregate + refs like ``MAX(total)``. + + The "dot in name" cross-model predicate is safe because every + catalog-side name source forbids dots: ``Column.name``, + ``ModelMeasure.name``, and (DEV-1567) ``Aggregation.name`` all + enforce ``[a-zA-Z_][a-zA-Z0-9_]*``. + """ + return [ + m for m in table.metrics + if "." not in m.name and m.measure_formula == m.name + ] + + +def local_dimensions(table: FacadeTable) -> list[FacadeDimension]: + """Mirror of :func:`local_metrics` for dimensions: drop cross-model + entries (single-hop and multi-hop joined dimensions). Unlike + metrics, dimensions don't have a synthetic-vs-user-authored split — + every dimension IS a ``Column`` the user defined on the model. + See :func:`local_metrics` for the leak-path rationale.""" + return [d for d in table.dimensions if "." not in d.name] def build_catalog( *, - models_by_datasource: Dict[str, List[SlayerModel]], + models_by_datasource: dict[str, list[SlayerModel]], bfs_depth: int = DEFAULT_BFS_DEPTH, ) -> FacadeCatalog: """Build a ``FacadeCatalog`` snapshot. @@ -93,10 +183,10 @@ def build_catalog( list_models(data_source=...)`` so cross-datasource joins are naturally constrained (SLayer doesn't auto-mirror joins across datasources). """ - schemas: List[FacadeSchema] = [] + schemas: list[FacadeSchema] = [] for datasource, models in models_by_datasource.items(): - by_name: Dict[str, SlayerModel] = {m.name: m for m in models} - tables: List[FacadeTable] = [] + by_name: dict[str, SlayerModel] = {m.name: m for m in models} + tables: list[FacadeTable] = [] for model in models: if model.hidden: continue @@ -115,13 +205,85 @@ def build_catalog( return FacadeCatalog(catalog_name=CATALOG_NAME, schemas=schemas) +DEFAULT_PG_SCHEMA = "public" + + +def build_catalog_grouped_by_schema( + *, + models_by_datasource: dict[str, list[SlayerModel]], + schema_by_datasource: dict[str, str] | None = None, + datasource_priority: list[str] | None = None, + default_schema: str = DEFAULT_PG_SCHEMA, + bfs_depth: int = DEFAULT_BFS_DEPTH, +) -> FacadeCatalog: + """Build a catalog spanning many datasources, grouped into Postgres schemas. + + Each datasource's tables are built independently (so join BFS stays scoped + to one datasource — merging never fabricates cross-datasource joins), then + re-grouped into ``FacadeSchema``s named by each datasource's + ``schema_by_datasource`` entry (default ``"public"``). When two datasources + map to the same schema and share a model name, ``datasource_priority`` + decides the winner (earlier = higher priority); the loser is shadowed (a + Postgres schema can only expose one table of a given name) and logged. + """ + schema_by_datasource = schema_by_datasource or {} + priority = datasource_priority or [] + + def _priority_index(datasource: str) -> int: + try: + return priority.index(datasource) + except ValueError: + return len(priority) + + # Per-datasource build keeps join scoping correct (one FacadeSchema each). + per_datasource = build_catalog( + models_by_datasource=models_by_datasource, bfs_depth=bfs_depth, + ) + + # target schema -> table name -> (priority_index, datasource, table) + grouped: dict[str, dict[str, tuple[int, str, FacadeTable]]] = {} + for source_schema in per_datasource.schemas: + datasource = source_schema.name + target = schema_by_datasource.get(datasource, default_schema) + bucket = grouped.setdefault(target, {}) + for table in source_schema.tables: + incoming = (_priority_index(datasource), datasource, table) + existing = bucket.get(table.name) + if existing is None or incoming[0] < existing[0]: + if existing is not None: + logger.warning( + "Facade catalog: model %r exists in both datasource %r " + "and %r under schema %r; keeping %r (higher priority), " + "shadowing %r. Set distinct postgres_schema to expose " + "both.", + table.name, existing[1], datasource, target, + datasource, existing[1], + ) + bucket[table.name] = incoming + elif existing is not None: + logger.warning( + "Facade catalog: model %r exists in both datasource %r and " + "%r under schema %r; keeping %r (higher priority), shadowing " + "%r. Set distinct postgres_schema to expose both.", + table.name, existing[1], datasource, target, + existing[1], datasource, + ) + + schemas = [ + FacadeSchema(name=name, tables=[entry[2] for entry in bucket.values()]) + for name, bucket in grouped.items() + ] + return FacadeCatalog(catalog_name=CATALOG_NAME, schemas=schemas) + + def _column_types_supported(*, model: SlayerModel) -> bool: """Reject the whole model if any non-hidden column has a Column.type - outside the six base types (§12 gotcha #7). DataType is a StrEnum so the - pydantic field is already constrained to the six values — but a future - extension that adds a new variant would silently surface here as + outside ``SUPPORTED_DATATYPES`` (§12 gotcha #7). DataType is a StrEnum so + the pydantic field is already constrained to the known values — but a + future extension that adds a new variant would silently surface here as unmappable, which we'd rather catch with a clear warning than emit a - half-typed catalog.""" + half-typed catalog. Opaque ``UNKNOWN`` columns ARE supported: they map to + VARCHAR and carry no aggregations (see DEFAULT_AGGREGATIONS_BY_TYPE).""" supported = set(SUPPORTED_DATATYPES) for col in model.columns: if col.hidden: @@ -140,7 +302,7 @@ def _column_types_supported(*, model: SlayerModel) -> bool: def _build_table( *, model: SlayerModel, - models_by_name: Dict[str, SlayerModel], + models_by_name: dict[str, SlayerModel], bfs_depth: int, ) -> FacadeTable: table_type = _table_type(model=model) @@ -149,12 +311,39 @@ def _build_table( ) metrics = _metric_expansion(model=model, reachable=reachable) dimensions = _dimension_expansion(model=model, reachable=reachable) + joins = _facade_joins_for(model=model, models_by_name=models_by_name) return FacadeTable( name=model.name, table_type=table_type, description=model.description, metrics=metrics, dimensions=dimensions, + joins=joins, + model_ref=model, + ) + + +def _facade_joins_for( + *, model: SlayerModel, models_by_name: dict[str, SlayerModel], +) -> list[FacadeJoin]: + """Expose every direct (single-hop) join whose target is a non-hidden + model in the same catalog. Mirrors the BFS filter so the translator's + existence check never matches a join that isn't otherwise addressable + (DEV-1565).""" + out: list[FacadeJoin] = [] + for j in model.joins: + target = models_by_name.get(j.target_model) + if target is None or target.hidden: + continue + out.append(_facade_join_from(join=j)) + return out + + +def _facade_join_from(*, join: ModelJoin) -> FacadeJoin: + return FacadeJoin( + target_model=join.target_model, + join_pairs=[list(pair) for pair in join.join_pairs], + join_type=join.join_type, ) @@ -167,9 +356,9 @@ def _table_type(*, model: SlayerModel) -> str: def _walk_join_paths( *, root: SlayerModel, - models_by_name: Dict[str, SlayerModel], + models_by_name: dict[str, SlayerModel], max_depth: int, -) -> List[Tuple[List[str], SlayerModel]]: +) -> list[tuple[list[str], SlayerModel]]: """BFS the join graph from ``root`` up to ``max_depth`` hops. Returns a list of (path, target_model) tuples where ``path`` is the @@ -181,10 +370,10 @@ def _walk_join_paths( ``A→B→A`` revisit is allowed (a legitimate query shape when the join columns differ); past ``max_depth`` the BFS terminates. """ - out: List[Tuple[List[str], SlayerModel]] = [] + out: list[tuple[list[str], SlayerModel]] = [] if max_depth <= 0: return out - queue: List[Tuple[SlayerModel, List[str]]] = [(root, [])] + queue: list[tuple[SlayerModel, list[str]]] = [(root, [])] while queue: current, path = queue.pop(0) if len(path) >= max_depth: @@ -199,7 +388,7 @@ def _walk_join_paths( return out -def _path_dotted(path: List[str]) -> str: +def _path_dotted(path: list[str]) -> str: """Convert a join path to its dotted reference form. Used uniformly for both the catalog-facing metric / dimension ``name`` @@ -211,8 +400,39 @@ def _path_dotted(path: List[str]) -> str: return ".".join(path) -def _eligible_aggregations(*, column: Column) -> Set[str]: - """Per §5.1.3: default-by-type ∩ explicit whitelist, with PK clamp.""" +# first / last are SLayer's "value at earliest / latest time" aggregations — +# the engine resolves them against the query's time dimension (or the +# model's default). Exposing the corresponding ``_first`` / ``_last`` +# pseudo-columns through pg_attribute on a model with no time dimension +# leaves BI tools (Metabase fingerprint, dbt schema scan) discovering +# them as queryable columns whose execution then fails with +# "Aggregation 'first' on measure '' requires a time column". +_TIME_DEPENDENT_AGGREGATIONS: frozenset[str] = frozenset({"first", "last"}) + + +def _model_has_resolvable_time_dimension(model: SlayerModel) -> bool: + """True if the engine can resolve ``first`` / ``last`` aggregations + on this model without an explicit time-dimension argument. + + The engine auto-picks a time dimension only when the QUERY already + includes one. For ``:first`` referenced as a flat metric (which + is how every BI tool's flat ``SELECT _first FROM `` + fingerprint scan emits it), the engine falls back to + ``model.default_time_dimension`` and errors if it's unset. So the + facade only exposes ``_first`` / ``_last`` when the model + declares ``default_time_dimension`` explicitly.""" + return bool(model.default_time_dimension) + + +def _eligible_aggregations( + *, column: Column, model: SlayerModel | None = None, +) -> set[str]: + """Per §5.1.3: default-by-type ∩ explicit whitelist, with PK clamp. + + When ``model`` is given AND it has no time dimension, time-dependent + aggregations (``first``, ``last``) are dropped — they would expose + pseudo-columns the engine then refuses to execute against. + """ if column.primary_key: base = set(PRIMARY_KEY_AGGREGATIONS) else: @@ -220,10 +440,13 @@ def _eligible_aggregations(*, column: Column) -> Set[str]: if column.allowed_aggregations is not None: base &= set(column.allowed_aggregations) # Strip parametric built-ins — they need named args (§5.1.3). - return base - _PARAMETRIC_BUILTIN_AGGS + base -= _PARAMETRIC_BUILTIN_AGGS + if model is not None and not _model_has_resolvable_time_dimension(model): + base -= _TIME_DEPENDENT_AGGREGATIONS + return base -def _eligible_custom_aggregations(*, model: SlayerModel) -> List[Aggregation]: +def _eligible_custom_aggregations(*, model: SlayerModel) -> list[Aggregation]: """Per §5.1.4: custom aggs that use only ``{value}`` (no extra params).""" return [agg for agg in model.aggregations if not agg.params] @@ -231,8 +454,8 @@ def _eligible_custom_aggregations(*, model: SlayerModel) -> List[Aggregation]: def _metric_expansion( *, model: SlayerModel, - reachable: List[Tuple[List[str], SlayerModel]], -) -> List[FacadeMetric]: + reachable: list[tuple[list[str], SlayerModel]], +) -> list[FacadeMetric]: local = _local_metrics_for(model=model) out = list(local) # Apply BFS-derived joined metrics. Rules 1-4 are computed on ``J`` @@ -281,7 +504,7 @@ def _synthetic_row_count(model: SlayerModel) -> FacadeMetric: ) -def _saved_model_measures(model: SlayerModel) -> List[FacadeMetric]: +def _saved_model_measures(model: SlayerModel) -> list[FacadeMetric]: """Rule 2: every saved ``ModelMeasure`` with a name.""" return [ FacadeMetric( @@ -296,7 +519,7 @@ def _saved_model_measures(model: SlayerModel) -> List[FacadeMetric]: ] -def _column_x_builtin_aggs(model: SlayerModel) -> List[FacadeMetric]: +def _column_x_builtin_aggs(model: SlayerModel) -> list[FacadeMetric]: """Rule 3: column × eligible-builtin-agg cartesian.""" return [ FacadeMetric( @@ -308,11 +531,11 @@ def _column_x_builtin_aggs(model: SlayerModel) -> List[FacadeMetric]: ) for col in model.columns if not col.hidden - for agg in sorted(_eligible_aggregations(column=col)) + for agg in sorted(_eligible_aggregations(column=col, model=model)) ] -def _column_x_custom_aggs(model: SlayerModel) -> List[FacadeMetric]: +def _column_x_custom_aggs(model: SlayerModel) -> list[FacadeMetric]: """Rule 4: column × parameterless custom aggs. Custom aggs are not gated by ``DEFAULT_AGGREGATIONS_BY_TYPE``, so we expose them on every non-hidden column. Custom-agg output type is opaque.""" @@ -333,7 +556,7 @@ def _column_x_custom_aggs(model: SlayerModel) -> List[FacadeMetric]: ] -def _local_metrics_for(*, model: SlayerModel) -> List[FacadeMetric]: +def _local_metrics_for(*, model: SlayerModel) -> list[FacadeMetric]: """Apply rules 1-4 to a single model in isolation (no join walk).""" return [ _synthetic_row_count(model), @@ -343,20 +566,49 @@ def _local_metrics_for(*, model: SlayerModel) -> List[FacadeMetric]: ] -def _describe_column_agg(*, column: Column, agg: str) -> Optional[str]: +def _local_dimensions_for(*, model: SlayerModel) -> list[FacadeDimension]: + """Bare-column dims for a single model in isolation (no join walk).""" + out: list[FacadeDimension] = [] + for col in model.columns: + if col.hidden: + continue + out.append(FacadeDimension( + name=col.name, + description=col.description, + label=col.label, + data_type=col.type, + is_time=col.type in {DataType.DATE, DataType.TIMESTAMP}, + dimension_ref=col.name, + )) + return out + + +def build_local_view( + model: SlayerModel, +) -> tuple[list[FacadeDimension], list[FacadeMetric]]: + """Build the bare-column dims + col×agg metrics for a single model + in isolation (no join walk). Used by the translator's dynamic-join + lookup materialisation (DEV-1565) so a join the catalog's BFS didn't + pre-expand can still resolve `.` / `.:` + refs. + """ + return _local_dimensions_for(model=model), _local_metrics_for(model=model) + + +def _describe_column_agg(*, column: Column, agg: str) -> str | None: if column.description: return f"{column.description} ({agg})" return None -def _agg_output_type(*, column: Column, agg: str) -> Optional[DataType]: +def _agg_output_type(*, column: Column, agg: str) -> DataType | None: """Coarse-grained output-type inference for column × agg pairs. Used only to populate ``INFORMATION_SCHEMA.METRICS.data_type``; the wire schema is always derived from the actual ``LIMIT 0`` execution (§5.3), so any inference here is informational. """ - if agg in {"count", "count_distinct"}: + if agg in {"count", "count_distinct", "count_distinct_approx"}: return DataType.INT if agg in {"sum"}: # SUM(INT) → INT for SQLite/Postgres; SUM(DOUBLE) → DOUBLE. @@ -376,9 +628,9 @@ def _agg_output_type(*, column: Column, agg: str) -> Optional[DataType]: def _dimension_expansion( *, model: SlayerModel, - reachable: List[Tuple[List[str], SlayerModel]], -) -> List[FacadeDimension]: - out: List[FacadeDimension] = [] + reachable: list[tuple[list[str], SlayerModel]], +) -> list[FacadeDimension]: + out: list[FacadeDimension] = [] for col in model.columns: if col.hidden: continue diff --git a/slayer/facade/catalog_sql.py b/slayer/facade/catalog_sql.py new file mode 100644 index 00000000..9de40d20 --- /dev/null +++ b/slayer/facade/catalog_sql.py @@ -0,0 +1,2459 @@ +"""DuckDB-backed catalog SQL executor (DEV-1558). + +Replaces the canned-row ``match_pg_catalog`` and (for the Postgres facade) +``match_info_schema`` matching with arbitrary SQL execution against an +in-memory DuckDB that materialises the catalog corpus as flat tables under +the ``main`` schema. The translator routes catalog-shaped queries here when +``catalog_sql_executor`` is provided; Flight's path keeps the canned +``match_info_schema`` answer unchanged. + +The pipeline: + +1. Pre-rewrite the parsed AST: strip ``pg_catalog.`` and + ``information_schema.`` qualifiers (info-schema tables rewrite to + ``_is_``); rewrite ``::regclass`` casts (literal → OID, dynamic → + ``slayer_regclass_oid(...)`` UDF); rewrite ``::regproc``/``::regtype`` to + ``0``; substitute ``current_database``/``current_catalog``/ + ``current_user``/``session_user``/``current_role`` with stored literals; + short-circuit ``current_schemas(true)[1]`` to ``'public'``; rewrite + Postgres regex operators (``~``/``!~``/``~*``/``!~*``) to + ``regexp_matches``; AST-rename every stub function (``format_type``, + ``obj_description`` …) to a private ``_slayer_*`` name so the macros + can't be shadowed by DuckDB built-ins. +2. Transpile postgres → duckdb via sqlglot. +3. Execute synchronously. +4. Map DuckDB cursor description → ``RowBatch`` with the six coarse + ``DataType``s. + +Errors → ``TranslationError`` after a WARNING log carrying the offending +SQL. + +Caching: ``executor_for(catalog)`` returns a process-cached executor keyed +by a SHA-256 fingerprint of a compact ``FacadeCatalog`` summary; FIFO +eviction at 4 entries. No lock — single-threaded asyncio + sync execute. +""" + +from __future__ import annotations + +import collections +import hashlib +import json +import logging +import zlib +from collections.abc import Iterable +from typing import Any + +import duckdb +import sqlglot +import sqlglot.expressions as exp +from pydantic import BaseModel, ConfigDict + +from slayer.core.enums import DataType +from slayer.facade.catalog import ( + CATALOG_NAME, + FacadeCatalog, + FacadeTable, + local_dimensions, + local_metrics, +) +from slayer.facade.rows import FacadeColumn, RowBatch +from slayer.pg_facade.types import datatype_to_oid + +logger = logging.getLogger(__name__) + + +# --- OID constants (moved from slayer/pg_facade/pg_catalog.py) --------------- + +PUBLIC_NAMESPACE_OID = 2200 +PG_CATALOG_NAMESPACE_OID = 11 +DEFAULT_OWNER_OID = 10 + +# Well-known Postgres system catalog OIDs. Hardcoded so ``'pg_class'::regclass`` +# resolves to ``1259`` (matching ``pg_description.classoid``) and Metabase's +# get-tables JOIN works end-to-end. +KNOWN_SYSTEM_OIDS: dict[str, int] = { + "pg_class": 1259, + "pg_namespace": 2615, + "pg_attribute": 1249, + "pg_type": 1247, + "pg_proc": 1255, + "pg_description": 2609, + "pg_constraint": 2606, + "pg_index": 2610, + "pg_attrdef": 2604, + # psql backslash-command coverage stubs. + "pg_am": 2601, + "pg_database": 1262, + "pg_authid": 1260, +} + +# Postgres heap access-method OID. Hardcoded to match real Postgres so +# ``pg_class.relam = 2`` round-trips against any tool that knows the +# canonical value. +PG_AM_HEAP_OID = 2 +# Single synthetic role for ``\du`` / ``pg_get_userbyid``. The facade's +# auth model is shared-token, so there's exactly one principal. +PG_SLAYER_ROLE_OID = 10 +PG_SLAYER_ROLE_NAME = "slayer" +# UTF8 is the only encoding the facade ever advertises. +PG_ENCODING_UTF8 = 6 + +# Per-OID metadata (typname, typlen, typcategory) for the six wire types. +from slayer.pg_facade.protocol import ( # noqa: E402 — wire OIDs co-located here + OID_BOOL, + OID_DATE, + OID_FLOAT8, + OID_INT8, + OID_TEXT, + OID_TIMESTAMP, +) + +_TYPE_META: dict[int, tuple[str, int, str]] = { + OID_BOOL: ("bool", 1, "B"), + OID_INT8: ("int8", 8, "N"), + OID_TEXT: ("text", -1, "S"), + OID_FLOAT8: ("float8", 8, "N"), + OID_DATE: ("date", 4, "D"), + OID_TIMESTAMP: ("timestamp", 8, "D"), +} + +# Inverse of _TYPE_META: type name → OID. Used to resolve ``::regtype`` +# casts to the underlying ``pg_type.oid`` so catalog queries like +# ``WHERE oid = 'int8'::regtype`` work. +_KNOWN_TYPE_OIDS: dict[str, int] = { + typname: oid for oid, (typname, _len, _cat) in _TYPE_META.items() +} + + +_UDT_NAME_BY_DATATYPE: dict[DataType, str] = { + DataType.BOOLEAN: "bool", + DataType.INT: "int8", + DataType.TEXT: "text", + DataType.DOUBLE: "float8", + DataType.DATE: "date", + DataType.TIMESTAMP: "timestamp", +} + + +def stable_oid(*parts: str) -> int: + """Deterministic positive 31-bit OID from a namespaced identifier.""" + key = ".".join(parts).encode("utf-8") + return zlib.crc32(key) & 0x7FFFFFFF + + +# --- CatalogRelation Pydantic ------------------------------------------------ + + +class CatalogRelation(BaseModel): + """One catalog table's content — facade-neutral. + + ``columns`` is a list of typed FacadeColumns; ``rows`` is a list of + ``{column_name: value}`` dicts. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + name: str + columns: list[FacadeColumn] + rows: list[dict[str, Any]] + + +# --- corpus builder --------------------------------------------------------- + + +def build_catalog_relations( + catalog: FacadeCatalog, + datasource: str | None = None, + *, + extra_relations: Iterable[CatalogRelation] | None = None, +) -> list[CatalogRelation]: + """Build every catalog table from ``catalog``. + + ``datasource`` is used as the ``catalog_name`` / ``table_catalog`` + value in the ``information_schema.*`` relations so queries that + filter by ``current_database()`` (which the AST rewrite substitutes + with the connection's datasource) see consistent rows. When omitted, + falls back to the catalog's static name (``slayer``) for backward + compatibility. + + ``extra_relations`` is the extensibility hook for embedders: each + ``CatalogRelation`` provided either **replaces** the default builder's + output for that table name (override case — e.g. project real per-tenant + rows into ``pg_roles``) or **adds** a new relation the default doesn't + know about. Override is by table-name match; order of the returned + list places overrides where the default originally appeared, with + additions appended at the end. + """ + ds = datasource or catalog.catalog_name + out: list[CatalogRelation] = [] + out.append(_build_pg_namespace(catalog)) + out.append(_build_pg_class(catalog)) + out.append(_build_pg_attribute(catalog)) + out.append(_build_pg_type()) + out.append(_build_pg_proc()) + out.append(_build_pg_settings()) + out.append(_build_pg_description(catalog)) + out.append(_build_pg_stat_user_tables(catalog)) + out.append(_build_pg_enum()) + out.append(_build_pg_tables(catalog)) + out.append(_build_pg_views(catalog)) + out.append(_build_pg_matviews()) + out.append(_build_pg_constraint()) + out.append(_build_pg_index()) + out.append(_build_pg_attrdef()) + out.append(_build_pg_am()) + out.append(_build_pg_database(catalog, ds)) + out.append(_build_pg_roles()) + out.append(_build_pg_collation()) + out.append(_build_pg_policy()) + # Empty stubs for catalog tables psql's ``\d`` variants touch + # (triggers / inheritance / publications / partitioning / views + # rewrite / extensions) — all legitimately empty for SLayer. + out.append(_build_pg_trigger()) + out.append(_build_pg_inherits()) + out.append(_build_pg_publication()) + out.append(_build_pg_publication_rel()) + out.append(_build_pg_publication_tables()) + out.append(_build_pg_partitioned_table()) + out.append(_build_pg_rewrite()) + out.append(_build_pg_extension()) + out.append(_build_is_columns(catalog, ds)) + out.append(_build_is_table_constraints()) + out.append(_build_is_key_column_usage()) + out.append(_build_is_schemata(catalog, ds)) + out.append(_build_is_tables(catalog, ds)) + out.append(_build_is_metrics(catalog, ds)) + out.append(_build_is_dimensions(catalog, ds)) + if extra_relations is not None: + extras = {r.name: r for r in extra_relations} + out = [extras.pop(r.name, r) for r in out] + out.extend(extras.values()) + return out + + +def _all_tables(catalog: FacadeCatalog): + """Yield ``(datasource, FacadeTable)`` for every table in the catalog.""" + for sch in catalog.schemas: + for tbl in sch.tables: + yield sch.name, tbl + + +def _table_oid(datasource: str, table: FacadeTable) -> int: + return stable_oid(datasource, table.name) + + +def _column_specs(table: FacadeTable): + """Yield ``(name, DataType)`` for every projectable column (dims + + metrics). + + DEV-1567: cross-model entries are excluded so ``pg_attribute`` / + ``pg_class.relnatts`` advertise only the same-model column list. BI + tools that flatten the catalog into a column view (Metabase, dbt + schema scan) discover only base columns and don't issue fingerprint + queries that would lead to dotted ``SlayerQuery.measures[*].name``. + """ + for d in local_dimensions(table): + yield d.name, d.data_type + for m in local_metrics(table): + yield m.name, m.data_type if m.data_type is not None else DataType.TEXT + + +def _namespace_oid(schema: str) -> int: + """Stable namespace OID for a facade schema. ``public`` keeps Postgres's + well-known 2200 so clients that hardcode it keep working; other + ``postgres_schema`` values get a deterministic derived OID.""" + if schema == "public": + return PUBLIC_NAMESPACE_OID + return stable_oid("namespace", schema) + + +def _user_schema_names(catalog: FacadeCatalog) -> list[str]: + """Distinct facade schema names in catalog order (no pg_catalog / info).""" + seen: dict[str, None] = {} + for sch in catalog.schemas: + seen.setdefault(sch.name, None) + seen.setdefault("public", None) # always advertise public, even if empty + return list(seen) + + +def _build_pg_namespace(catalog: FacadeCatalog) -> CatalogRelation: + columns = [ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="nspname", type=DataType.TEXT), + FacadeColumn(name="nspowner", type=DataType.INT), + FacadeColumn(name="nspacl", type=DataType.TEXT), + ] + rows = [ + {"oid": _namespace_oid(name), "nspname": name, + "nspowner": DEFAULT_OWNER_OID, "nspacl": None} + for name in _user_schema_names(catalog) + ] + rows.append( + {"oid": PG_CATALOG_NAMESPACE_OID, "nspname": "pg_catalog", + "nspowner": DEFAULT_OWNER_OID, "nspacl": None} + ) + return CatalogRelation(name="pg_namespace", columns=columns, rows=rows) + + +def _build_pg_class(catalog: FacadeCatalog) -> CatalogRelation: + columns = [ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="relname", type=DataType.TEXT), + FacadeColumn(name="relnamespace", type=DataType.INT), + FacadeColumn(name="reltype", type=DataType.INT), + FacadeColumn(name="relowner", type=DataType.INT), + FacadeColumn(name="relkind", type=DataType.TEXT), + FacadeColumn(name="relnatts", type=DataType.INT), + FacadeColumn(name="relhasindex", type=DataType.BOOLEAN), + FacadeColumn(name="relpersistence", type=DataType.TEXT), + FacadeColumn(name="relpages", type=DataType.INT), + FacadeColumn(name="reltuples", type=DataType.DOUBLE), + FacadeColumn(name="relhasrules", type=DataType.BOOLEAN), + FacadeColumn(name="relhastriggers", type=DataType.BOOLEAN), + FacadeColumn(name="relrowsecurity", type=DataType.BOOLEAN), + FacadeColumn(name="relispartition", type=DataType.BOOLEAN), + # Access-method OID — points at ``pg_am.oid``. psql's ``\d`` LEFT + # JOINs on this; ``heap`` is the only access method we advertise. + FacadeColumn(name="relam", type=DataType.INT), + # psql's ``\d
`` reads these too. Sensible defaults: no + # CHECK constraints, no TOAST, no forced RLS, default tablespace + # / replica identity, not a typed table. + FacadeColumn(name="relchecks", type=DataType.INT), + FacadeColumn(name="relforcerowsecurity", type=DataType.BOOLEAN), + FacadeColumn(name="reloftype", type=DataType.INT), + FacadeColumn(name="reltablespace", type=DataType.INT), + FacadeColumn(name="reltoastrelid", type=DataType.INT), + FacadeColumn(name="relreplident", type=DataType.TEXT), + ] + rows = [] + seen_oids: dict[int, str] = {} + for ds, tbl in _all_tables(catalog): + oid = _table_oid(ds, tbl) + _check_collision(seen_oids, oid, f"{ds}.{tbl.name}") + natts = sum(1 for _ in _column_specs(tbl)) + # SQL-backed models are advertised as views (relkind='v') so + # ``pg_views`` discovery and JDBC view-specific paths see them; + # ``sql_table``-mode models stay as regular tables ('r'). + relkind = "v" if tbl.table_type == "VIEW" else "r" + rows.append({ + "oid": oid, "relname": tbl.name, + "relnamespace": _namespace_oid(ds), "reltype": 0, + "relowner": DEFAULT_OWNER_OID, "relkind": relkind, + "relnatts": natts, "relhasindex": False, "relpersistence": "p", + "relpages": 0, "reltuples": -1.0, + "relhasrules": False, "relhastriggers": False, + "relrowsecurity": False, "relispartition": False, + "relam": PG_AM_HEAP_OID, + "relchecks": 0, "relforcerowsecurity": False, + "reloftype": 0, "reltablespace": 0, "reltoastrelid": 0, + "relreplident": "d", # 'd' = default replica identity + }) + return CatalogRelation(name="pg_class", columns=columns, rows=rows) + + +def _build_pg_attribute(catalog: FacadeCatalog) -> CatalogRelation: + columns = [ + FacadeColumn(name="attrelid", type=DataType.INT), + FacadeColumn(name="attname", type=DataType.TEXT), + FacadeColumn(name="atttypid", type=DataType.INT), + FacadeColumn(name="attnum", type=DataType.INT), + FacadeColumn(name="attlen", type=DataType.INT), + FacadeColumn(name="atttypmod", type=DataType.INT), + FacadeColumn(name="attnotnull", type=DataType.BOOLEAN), + FacadeColumn(name="atthasdef", type=DataType.BOOLEAN), + FacadeColumn(name="attisdropped", type=DataType.BOOLEAN), + FacadeColumn(name="attidentity", type=DataType.TEXT), + FacadeColumn(name="attgenerated", type=DataType.TEXT), + # Collation OID — psql's ``\d
`` LEFT JOINs pg_collation on + # this. Zero (== no non-default collation) is fine for all our + # ASCII columns; the join returns no rows and the whole + # ``attcollation`` sub-select is NULL — the exact ``\d`` fast path. + FacadeColumn(name="attcollation", type=DataType.INT), + ] + rows = [] + for ds, tbl in _all_tables(catalog): + attrelid = _table_oid(ds, tbl) + attnum = 1 + for name, data_type in _column_specs(tbl): + oid = datatype_to_oid(data_type) + rows.append({ + "attrelid": attrelid, "attname": name, "atttypid": oid, + "attnum": attnum, "attlen": _TYPE_META[oid][1], + "atttypmod": -1, "attnotnull": False, "atthasdef": False, + "attisdropped": False, "attidentity": "", "attgenerated": "", + "attcollation": 0, + }) + attnum += 1 + return CatalogRelation(name="pg_attribute", columns=columns, rows=rows) + + +def _build_pg_type() -> CatalogRelation: + columns = [ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="typname", type=DataType.TEXT), + FacadeColumn(name="typnamespace", type=DataType.INT), + FacadeColumn(name="typlen", type=DataType.INT), + FacadeColumn(name="typtype", type=DataType.TEXT), + FacadeColumn(name="typcategory", type=DataType.TEXT), + FacadeColumn(name="typisdefined", type=DataType.BOOLEAN), + FacadeColumn(name="typdelim", type=DataType.TEXT), + FacadeColumn(name="typrelid", type=DataType.INT), + FacadeColumn(name="typelem", type=DataType.INT), + FacadeColumn(name="typarray", type=DataType.INT), + FacadeColumn(name="typnotnull", type=DataType.BOOLEAN), + FacadeColumn(name="typbasetype", type=DataType.INT), + FacadeColumn(name="typtypmod", type=DataType.INT), + # Default collation OID. psql's ``\d
`` compares this to + # ``pg_attribute.attcollation``; matching zero (== default + # collation) means the ``attcollation`` sub-select returns NULL, + # which is the correct behavior for columns using the DB default. + FacadeColumn(name="typcollation", type=DataType.INT), + ] + rows = [] + for oid, (typname, typlen, typcategory) in _TYPE_META.items(): + rows.append({ + "oid": oid, "typname": typname, + "typnamespace": PG_CATALOG_NAMESPACE_OID, + "typlen": typlen, "typtype": "b", "typcategory": typcategory, + "typisdefined": True, "typdelim": ",", "typrelid": 0, + "typelem": 0, "typarray": 0, "typnotnull": False, + "typbasetype": 0, "typtypmod": -1, + "typcollation": 0, + }) + return CatalogRelation(name="pg_type", columns=columns, rows=rows) + + +def _build_pg_proc() -> CatalogRelation: + return CatalogRelation(name="pg_proc", columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="proname", type=DataType.TEXT), + FacadeColumn(name="pronamespace", type=DataType.INT), + FacadeColumn(name="prorettype", type=DataType.INT), + ], rows=[]) + + +def _build_pg_settings() -> CatalogRelation: + from slayer.pg_facade.identity import PG_SERVER_VERSION + columns = [ + FacadeColumn(name="name", type=DataType.TEXT), + FacadeColumn(name="setting", type=DataType.TEXT), + FacadeColumn(name="category", type=DataType.TEXT), + FacadeColumn(name="unit", type=DataType.TEXT), + FacadeColumn(name="source", type=DataType.TEXT), + FacadeColumn(name="vartype", type=DataType.TEXT), + FacadeColumn(name="context", type=DataType.TEXT), + FacadeColumn(name="min_val", type=DataType.TEXT), + FacadeColumn(name="max_val", type=DataType.TEXT), + ] + settings = [ + ("server_version", PG_SERVER_VERSION), + ("client_encoding", "UTF8"), + ("server_encoding", "UTF8"), + ("DateStyle", "ISO, MDY"), + ("IntervalStyle", "postgres"), + ("TimeZone", "UTC"), + ("standard_conforming_strings", "on"), + ("integer_datetimes", "on"), + ("max_index_keys", "32"), + ("block_size", "8192"), + ] + rows = [{"name": name, "setting": value, "category": "Preset Options", + "unit": None, "source": "default", "vartype": "string", + "context": "user", "min_val": None, "max_val": None} + for name, value in settings] + return CatalogRelation(name="pg_settings", columns=columns, rows=rows) + + +def _build_pg_description(catalog: FacadeCatalog) -> CatalogRelation: + columns = [ + FacadeColumn(name="objoid", type=DataType.INT), + FacadeColumn(name="classoid", type=DataType.INT), + FacadeColumn(name="objsubid", type=DataType.INT), + FacadeColumn(name="description", type=DataType.TEXT), + ] + rows: list[dict[str, Any]] = [] + for ds, tbl in _all_tables(catalog): + oid = _table_oid(ds, tbl) + if tbl.description: + rows.append({"objoid": oid, "classoid": KNOWN_SYSTEM_OIDS["pg_class"], + "objsubid": 0, "description": tbl.description}) + attnum = 1 + # DEV-1567: stay within the (filtered) pg_attribute attnum space so + # ``objsubid`` always matches a real pg_attribute row. + for d in local_dimensions(tbl): + if d.description: + rows.append({"objoid": oid, "classoid": KNOWN_SYSTEM_OIDS["pg_class"], + "objsubid": attnum, "description": d.description}) + attnum += 1 + for m in local_metrics(tbl): + if m.description: + rows.append({"objoid": oid, "classoid": KNOWN_SYSTEM_OIDS["pg_class"], + "objsubid": attnum, "description": m.description}) + attnum += 1 + return CatalogRelation(name="pg_description", columns=columns, rows=rows) + + +def _build_pg_stat_user_tables(catalog: FacadeCatalog) -> CatalogRelation: + columns = [ + FacadeColumn(name="schemaname", type=DataType.TEXT), + FacadeColumn(name="relname", type=DataType.TEXT), + FacadeColumn(name="n_live_tup", type=DataType.INT), + ] + rows = [{"schemaname": ds, "relname": tbl.name, "n_live_tup": None} + for ds, tbl in _all_tables(catalog)] + return CatalogRelation(name="pg_stat_user_tables", columns=columns, rows=rows) + + +def _build_pg_enum() -> CatalogRelation: + return CatalogRelation(name="pg_enum", columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="enumtypid", type=DataType.INT), + FacadeColumn(name="enumlabel", type=DataType.TEXT), + ], rows=[]) + + +def _build_pg_tables(catalog: FacadeCatalog) -> CatalogRelation: + columns = [ + FacadeColumn(name="schemaname", type=DataType.TEXT), + FacadeColumn(name="tablename", type=DataType.TEXT), + FacadeColumn(name="tableowner", type=DataType.TEXT), + FacadeColumn(name="tablespace", type=DataType.TEXT), + FacadeColumn(name="hasindexes", type=DataType.BOOLEAN), + FacadeColumn(name="hasrules", type=DataType.BOOLEAN), + FacadeColumn(name="hastriggers", type=DataType.BOOLEAN), + FacadeColumn(name="rowsecurity", type=DataType.BOOLEAN), + ] + rows = [] + for ds, tbl in _all_tables(catalog): + # M1 — VIEW-typed models are excluded from pg_tables. + if tbl.table_type != "TABLE": + continue + rows.append({ + "schemaname": ds, "tablename": tbl.name, + "tableowner": "slayer", "tablespace": None, + "hasindexes": False, "hasrules": False, "hastriggers": False, + "rowsecurity": False, + }) + return CatalogRelation(name="pg_tables", columns=columns, rows=rows) + + +def _build_pg_views(catalog: FacadeCatalog) -> CatalogRelation: + columns = [ + FacadeColumn(name="schemaname", type=DataType.TEXT), + FacadeColumn(name="viewname", type=DataType.TEXT), + FacadeColumn(name="viewowner", type=DataType.TEXT), + FacadeColumn(name="definition", type=DataType.TEXT), + ] + # SQL-backed models surface here so view-aware clients + # (pgAdmin, dbeaver view category, JDBC ``getTables`` with + # ``types=['VIEW']``) discover them as views. We do NOT include the + # underlying SQL definition — it's a SLayer abstraction detail and + # exposing it would leak datasource SQL through the facade. + rows = [] + for ds, tbl in _all_tables(catalog): + if tbl.table_type != "VIEW": + continue + rows.append({ + "schemaname": ds, + "viewname": tbl.name, + "viewowner": "slayer", + "definition": None, + }) + return CatalogRelation(name="pg_views", columns=columns, rows=rows) + + +def _build_pg_matviews() -> CatalogRelation: + return CatalogRelation(name="pg_matviews", columns=[ + FacadeColumn(name="schemaname", type=DataType.TEXT), + FacadeColumn(name="matviewname", type=DataType.TEXT), + FacadeColumn(name="matviewowner", type=DataType.TEXT), + FacadeColumn(name="tablespace", type=DataType.TEXT), + FacadeColumn(name="hasindexes", type=DataType.BOOLEAN), + FacadeColumn(name="ispopulated", type=DataType.BOOLEAN), + FacadeColumn(name="definition", type=DataType.TEXT), + ], rows=[]) + + +def _build_pg_constraint() -> CatalogRelation: + return CatalogRelation(name="pg_constraint", columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="conname", type=DataType.TEXT), + FacadeColumn(name="contype", type=DataType.TEXT), + FacadeColumn(name="conrelid", type=DataType.INT), + FacadeColumn(name="confrelid", type=DataType.INT), + FacadeColumn(name="connamespace", type=DataType.INT), + FacadeColumn(name="conkey", type=DataType.TEXT), + FacadeColumn(name="confkey", type=DataType.TEXT), + ], rows=[]) + + +def _build_pg_index() -> CatalogRelation: + return CatalogRelation(name="pg_index", columns=[ + FacadeColumn(name="indexrelid", type=DataType.INT), + FacadeColumn(name="indrelid", type=DataType.INT), + FacadeColumn(name="indnatts", type=DataType.INT), + FacadeColumn(name="indnkeyatts", type=DataType.INT), + FacadeColumn(name="indisunique", type=DataType.BOOLEAN), + FacadeColumn(name="indisprimary", type=DataType.BOOLEAN), + FacadeColumn(name="indkey", type=DataType.TEXT), + ], rows=[]) + + +def _build_pg_attrdef() -> CatalogRelation: + return CatalogRelation(name="pg_attrdef", columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="adrelid", type=DataType.INT), + FacadeColumn(name="adnum", type=DataType.INT), + FacadeColumn(name="adbin", type=DataType.TEXT), + ], rows=[]) + + +def _build_pg_am() -> CatalogRelation: + """Stub access-method table. Real Postgres exposes heap/btree/hash/gist/gin/ + brin; for the facade only ``heap`` matters (every ``pg_class.relam`` we emit + points at it). psql's ``\\d`` LEFT JOINs ``pg_class.relam = pg_am.oid``; + without ``heap`` the JOIN would yield NULL for every relation.""" + return CatalogRelation( + name="pg_am", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="amname", type=DataType.TEXT), + FacadeColumn(name="amhandler", type=DataType.INT), + FacadeColumn(name="amtype", type=DataType.TEXT), + ], + rows=[ + {"oid": PG_AM_HEAP_OID, "amname": "heap", + "amhandler": 0, "amtype": "t"}, + ], + ) + + +def _build_pg_policy() -> CatalogRelation: + """Stub RLS-policies table — empty. psql's ``\\d
`` queries + this per relation to render the "Policies:" section; an empty + ``pg_policy`` correctly yields zero rows, matching real Postgres + behavior for a table with no row-level security policies.""" + return CatalogRelation( + name="pg_policy", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="polname", type=DataType.TEXT), + FacadeColumn(name="polrelid", type=DataType.INT), + FacadeColumn(name="polcmd", type=DataType.TEXT), + FacadeColumn(name="polpermissive", type=DataType.BOOLEAN), + FacadeColumn(name="polroles", type=DataType.TEXT), + FacadeColumn(name="polqual", type=DataType.TEXT), + FacadeColumn(name="polwithcheck", type=DataType.TEXT), + ], + rows=[], + ) + + +# --- Empty stubs for catalog tables psql's ``\d`` variants touch. +# +# SLayer has no triggers, no inheritance, no publications, no partitioning, +# no extensions by construction — every one of these tables is legitimately +# empty. Real Postgres returns zero rows too. Each stub declares the full +# column set psql's queries reference so DuckDB doesn't error on unknown +# columns; the ``_PG_CATALOG_NAMES`` allowlist entry is what stops the +# "Unknown schema: 'pg_catalog'" from the routing decision. + + +def _build_pg_trigger() -> CatalogRelation: + """Triggers — empty. psql's ``\\d
`` reads this for the + "Triggers:" section per relation.""" + return CatalogRelation( + name="pg_trigger", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="tgrelid", type=DataType.INT), + FacadeColumn(name="tgparentid", type=DataType.INT), + FacadeColumn(name="tgname", type=DataType.TEXT), + FacadeColumn(name="tgfoid", type=DataType.INT), + FacadeColumn(name="tgtype", type=DataType.INT), + FacadeColumn(name="tgenabled", type=DataType.TEXT), + FacadeColumn(name="tgisinternal", type=DataType.BOOLEAN), + FacadeColumn(name="tgconstrrelid", type=DataType.INT), + FacadeColumn(name="tgconstrindid", type=DataType.INT), + FacadeColumn(name="tgconstraint", type=DataType.INT), + FacadeColumn(name="tgdeferrable", type=DataType.BOOLEAN), + FacadeColumn(name="tginitdeferred", type=DataType.BOOLEAN), + FacadeColumn(name="tgnargs", type=DataType.INT), + FacadeColumn(name="tgargs", type=DataType.TEXT), + FacadeColumn(name="tgqual", type=DataType.TEXT), + FacadeColumn(name="tgoldtable", type=DataType.TEXT), + FacadeColumn(name="tgnewtable", type=DataType.TEXT), + ], + rows=[], + ) + + +def _build_pg_inherits() -> CatalogRelation: + """Inheritance — empty. Read by ``\\d`` to render parent/child + relationships. SLayer models are flat.""" + return CatalogRelation( + name="pg_inherits", + columns=[ + FacadeColumn(name="inhrelid", type=DataType.INT), + FacadeColumn(name="inhparent", type=DataType.INT), + FacadeColumn(name="inhseqno", type=DataType.INT), + FacadeColumn(name="inhdetachpending", type=DataType.BOOLEAN), + ], + rows=[], + ) + + +def _build_pg_publication() -> CatalogRelation: + """Logical-replication publications — empty. ``\\dRp`` and ``\\d`` + query this.""" + return CatalogRelation( + name="pg_publication", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="pubname", type=DataType.TEXT), + FacadeColumn(name="pubowner", type=DataType.INT), + FacadeColumn(name="puballtables", type=DataType.BOOLEAN), + FacadeColumn(name="pubinsert", type=DataType.BOOLEAN), + FacadeColumn(name="pubupdate", type=DataType.BOOLEAN), + FacadeColumn(name="pubdelete", type=DataType.BOOLEAN), + FacadeColumn(name="pubtruncate", type=DataType.BOOLEAN), + FacadeColumn(name="pubviaroot", type=DataType.BOOLEAN), + ], + rows=[], + ) + + +def _build_pg_publication_rel() -> CatalogRelation: + """Publication membership — empty.""" + return CatalogRelation( + name="pg_publication_rel", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="prpubid", type=DataType.INT), + FacadeColumn(name="prrelid", type=DataType.INT), + ], + rows=[], + ) + + +def _build_pg_publication_tables() -> CatalogRelation: + """Publication → table denormalized view — empty.""" + return CatalogRelation( + name="pg_publication_tables", + columns=[ + FacadeColumn(name="pubname", type=DataType.TEXT), + FacadeColumn(name="schemaname", type=DataType.TEXT), + FacadeColumn(name="tablename", type=DataType.TEXT), + ], + rows=[], + ) + + +def _build_pg_partitioned_table() -> CatalogRelation: + """Partitioning metadata — empty. SLayer has no partitioned models.""" + return CatalogRelation( + name="pg_partitioned_table", + columns=[ + FacadeColumn(name="partrelid", type=DataType.INT), + FacadeColumn(name="partstrat", type=DataType.TEXT), + FacadeColumn(name="partnatts", type=DataType.INT), + FacadeColumn(name="partdefid", type=DataType.INT), + FacadeColumn(name="partattrs", type=DataType.TEXT), + FacadeColumn(name="partclass", type=DataType.TEXT), + FacadeColumn(name="partcollation", type=DataType.TEXT), + FacadeColumn(name="partexprs", type=DataType.TEXT), + ], + rows=[], + ) + + +def _build_pg_rewrite() -> CatalogRelation: + """Rewrite rules (used by views) — empty. SLayer views are surfaced + via ``pg_class.relkind = 'v'`` but the rewrite text isn't tracked.""" + return CatalogRelation( + name="pg_rewrite", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="rulename", type=DataType.TEXT), + FacadeColumn(name="ev_class", type=DataType.INT), + FacadeColumn(name="ev_type", type=DataType.TEXT), + FacadeColumn(name="ev_enabled", type=DataType.TEXT), + FacadeColumn(name="is_instead", type=DataType.BOOLEAN), + FacadeColumn(name="ev_qual", type=DataType.TEXT), + FacadeColumn(name="ev_action", type=DataType.TEXT), + ], + rows=[], + ) + + +def _build_pg_extension() -> CatalogRelation: + """Installed extensions — empty. SLayer doesn't advertise any.""" + return CatalogRelation( + name="pg_extension", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="extname", type=DataType.TEXT), + FacadeColumn(name="extowner", type=DataType.INT), + FacadeColumn(name="extnamespace", type=DataType.INT), + FacadeColumn(name="extrelocatable", type=DataType.BOOLEAN), + FacadeColumn(name="extversion", type=DataType.TEXT), + FacadeColumn(name="extconfig", type=DataType.TEXT), + FacadeColumn(name="extcondition", type=DataType.TEXT), + ], + rows=[], + ) + + +def _build_pg_collation() -> CatalogRelation: + """Stub collation table — empty. psql's ``\\d
`` LEFT JOINs + ``pg_collation`` on ``pg_attribute.attcollation``; with every column + reporting ``attcollation = 0`` (default collation), the join returns + zero rows and the whole ``attcollation`` sub-select in the ``\\d`` + query evaluates to NULL — matching real Postgres behavior for + columns using the database default.""" + return CatalogRelation( + name="pg_collation", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="collname", type=DataType.TEXT), + FacadeColumn(name="collnamespace", type=DataType.INT), + FacadeColumn(name="collencoding", type=DataType.INT), + FacadeColumn(name="collcollate", type=DataType.TEXT), + FacadeColumn(name="collctype", type=DataType.TEXT), + ], + rows=[], + ) + + +def _build_pg_roles() -> CatalogRelation: + """Stub roles table. SLayer's auth is shared-token (one principal at a + time); ``\\du`` gets one synthetic row so the listing renders. Embedders + can override via the ``extra_relations`` hook on ``build_catalog_relations`` + to project real per-tenant principals.""" + return CatalogRelation( + name="pg_roles", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="rolname", type=DataType.TEXT), + FacadeColumn(name="rolsuper", type=DataType.BOOLEAN), + FacadeColumn(name="rolinherit", type=DataType.BOOLEAN), + FacadeColumn(name="rolcreaterole", type=DataType.BOOLEAN), + FacadeColumn(name="rolcreatedb", type=DataType.BOOLEAN), + FacadeColumn(name="rolcanlogin", type=DataType.BOOLEAN), + FacadeColumn(name="rolreplication", type=DataType.BOOLEAN), + FacadeColumn(name="rolconnlimit", type=DataType.INT), + FacadeColumn(name="rolvaliduntil", type=DataType.TIMESTAMP), + FacadeColumn(name="rolbypassrls", type=DataType.BOOLEAN), + ], + rows=[{ + "oid": PG_SLAYER_ROLE_OID, "rolname": PG_SLAYER_ROLE_NAME, + "rolsuper": False, "rolinherit": True, + "rolcreaterole": False, "rolcreatedb": False, + "rolcanlogin": True, "rolreplication": False, + "rolconnlimit": -1, "rolvaliduntil": None, "rolbypassrls": False, + }], + ) + + +def _build_pg_database(catalog: FacadeCatalog, datasource: str) -> CatalogRelation: + """Stub databases table. The facade scopes one connection to one SLayer + datasource (= one Postgres ``database``); ``\\l`` returns a single row for + the connected datasource. Embedders that want to enumerate every available + datasource (multi-tenant management UIs, etc.) override this builder.""" + return CatalogRelation( + name="pg_database", + columns=[ + FacadeColumn(name="oid", type=DataType.INT), + FacadeColumn(name="datname", type=DataType.TEXT), + FacadeColumn(name="datdba", type=DataType.INT), + FacadeColumn(name="encoding", type=DataType.INT), + FacadeColumn(name="datcollate", type=DataType.TEXT), + FacadeColumn(name="datctype", type=DataType.TEXT), + FacadeColumn(name="datistemplate", type=DataType.BOOLEAN), + FacadeColumn(name="datallowconn", type=DataType.BOOLEAN), + FacadeColumn(name="datconnlimit", type=DataType.INT), + FacadeColumn(name="dattablespace", type=DataType.INT), + FacadeColumn(name="datacl", type=DataType.TEXT), + ], + rows=[{ + "oid": stable_oid("database", datasource), + "datname": datasource, + "datdba": PG_SLAYER_ROLE_OID, + "encoding": PG_ENCODING_UTF8, + "datcollate": "en_US.UTF-8", + "datctype": "en_US.UTF-8", + "datistemplate": False, "datallowconn": True, + "datconnlimit": -1, "dattablespace": 1663, + "datacl": None, + }], + ) + + +def _build_is_columns(catalog: FacadeCatalog, datasource: str) -> CatalogRelation: + """Postgres-shape information_schema.columns with SLayer extension fields. + + Materialised as ``_is_columns`` in DuckDB; the AST rewrite pass strips + the ``information_schema.`` qualifier and rewrites the table name. + """ + columns = [ + FacadeColumn(name="table_catalog", type=DataType.TEXT), + FacadeColumn(name="table_schema", type=DataType.TEXT), + FacadeColumn(name="table_name", type=DataType.TEXT), + FacadeColumn(name="column_name", type=DataType.TEXT), + FacadeColumn(name="ordinal_position", type=DataType.INT), + FacadeColumn(name="column_default", type=DataType.TEXT), + FacadeColumn(name="is_nullable", type=DataType.TEXT), + FacadeColumn(name="data_type", type=DataType.TEXT), + FacadeColumn(name="udt_schema", type=DataType.TEXT), + FacadeColumn(name="udt_name", type=DataType.TEXT), + FacadeColumn(name="is_identity", type=DataType.TEXT), + FacadeColumn(name="is_generated", type=DataType.TEXT), + # SLayer extension columns (Q3 sub-b-ii). + FacadeColumn(name="column_kind", type=DataType.TEXT), + FacadeColumn(name="description", type=DataType.TEXT), + FacadeColumn(name="label", type=DataType.TEXT), + ] + rows: list[dict[str, Any]] = [] + # DEV-1567: exclude cross-model entries — they leak as dotted "columns" + # that Metabase fingerprint scans then project (see local_metrics + # docstring in slayer/facade/catalog.py). + for ds, tbl in _all_tables(catalog): + position = 1 + for d in local_dimensions(tbl): + udt = _UDT_NAME_BY_DATATYPE.get(d.data_type, "text") + rows.append({ + "table_catalog": datasource, "table_schema": ds, + "table_name": tbl.name, "column_name": d.name, + "ordinal_position": position, "column_default": None, + "is_nullable": "YES", "data_type": udt, + "udt_schema": "pg_catalog", "udt_name": udt, + "is_identity": "NO", "is_generated": "NEVER", + "column_kind": "DIMENSION", "description": d.description, + "label": d.label, + }) + position += 1 + for m in local_metrics(tbl): + udt = _UDT_NAME_BY_DATATYPE.get(m.data_type, "text") if m.data_type else "text" + rows.append({ + "table_catalog": datasource, "table_schema": ds, + "table_name": tbl.name, "column_name": m.name, + "ordinal_position": position, "column_default": None, + "is_nullable": "YES", "data_type": udt, + "udt_schema": "pg_catalog", "udt_name": udt, + "is_identity": "NO", "is_generated": "NEVER", + "column_kind": "METRIC", "description": m.description, + "label": m.label, + }) + position += 1 + return CatalogRelation(name="_is_columns", columns=columns, rows=rows) + + +def _build_is_table_constraints() -> CatalogRelation: + return CatalogRelation(name="_is_table_constraints", columns=[ + FacadeColumn(name="constraint_catalog", type=DataType.TEXT), + FacadeColumn(name="constraint_schema", type=DataType.TEXT), + FacadeColumn(name="constraint_name", type=DataType.TEXT), + FacadeColumn(name="table_catalog", type=DataType.TEXT), + FacadeColumn(name="table_schema", type=DataType.TEXT), + FacadeColumn(name="table_name", type=DataType.TEXT), + FacadeColumn(name="constraint_type", type=DataType.TEXT), + FacadeColumn(name="is_deferrable", type=DataType.TEXT), + FacadeColumn(name="initially_deferred", type=DataType.TEXT), + ], rows=[]) + + +def _build_is_key_column_usage() -> CatalogRelation: + return CatalogRelation(name="_is_key_column_usage", columns=[ + FacadeColumn(name="constraint_catalog", type=DataType.TEXT), + FacadeColumn(name="constraint_schema", type=DataType.TEXT), + FacadeColumn(name="constraint_name", type=DataType.TEXT), + FacadeColumn(name="table_catalog", type=DataType.TEXT), + FacadeColumn(name="table_schema", type=DataType.TEXT), + FacadeColumn(name="table_name", type=DataType.TEXT), + FacadeColumn(name="column_name", type=DataType.TEXT), + FacadeColumn(name="ordinal_position", type=DataType.INT), + ], rows=[]) + + +def _build_is_schemata( + catalog: FacadeCatalog, datasource: str, +) -> CatalogRelation: + """INFORMATION_SCHEMA.SCHEMATA — one row per facade schema. Datasources + map to schemas via ``postgres_schema`` (default ``public``); multiple + datasources sharing a schema collapse to one row.""" + columns = [ + FacadeColumn(name="catalog_name", type=DataType.TEXT), + FacadeColumn(name="schema_name", type=DataType.TEXT), + ] + rows = [ + {"catalog_name": datasource, "schema_name": name} + for name in _user_schema_names(catalog) + ] + return CatalogRelation(name="_is_schemata", columns=columns, rows=rows) + + +def _build_is_tables( + catalog: FacadeCatalog, datasource: str, +) -> CatalogRelation: + columns = [ + FacadeColumn(name="table_catalog", type=DataType.TEXT), + FacadeColumn(name="table_schema", type=DataType.TEXT), + FacadeColumn(name="table_name", type=DataType.TEXT), + FacadeColumn(name="table_type", type=DataType.TEXT), + ] + rows = [{"table_catalog": datasource, "table_schema": sch.name, + "table_name": tbl.name, "table_type": tbl.table_type} + for sch in catalog.schemas for tbl in sch.tables] + return CatalogRelation(name="_is_tables", columns=columns, rows=rows) + + +def _build_is_metrics( + catalog: FacadeCatalog, datasource: str, +) -> CatalogRelation: + """SLayer's INFORMATION_SCHEMA.METRICS extension — JDBC-style type + names (``DOUBLE`` / ``BIGINT`` / ``TIMESTAMP``) to match the contract + the canned ``match_info_schema._serve_metrics`` previously emitted.""" + from slayer.facade.datatypes import datatype_to_jdbc + columns = [ + FacadeColumn(name="catalog_name", type=DataType.TEXT), + FacadeColumn(name="schema_name", type=DataType.TEXT), + FacadeColumn(name="table_name", type=DataType.TEXT), + FacadeColumn(name="metric_name", type=DataType.TEXT), + FacadeColumn(name="description", type=DataType.TEXT), + FacadeColumn(name="data_type", type=DataType.TEXT), + FacadeColumn(name="label", type=DataType.TEXT), + ] + rows: list[dict[str, Any]] = [] + for sch in catalog.schemas: + for tbl in sch.tables: + for m in tbl.metrics: + rows.append({ + "catalog_name": datasource, + "schema_name": sch.name, "table_name": tbl.name, + "metric_name": m.name, "description": m.description, + "data_type": ( + datatype_to_jdbc(m.data_type) if m.data_type else None + ), + "label": m.label, + }) + return CatalogRelation(name="_is_metrics", columns=columns, rows=rows) + + +def _build_is_dimensions( + catalog: FacadeCatalog, datasource: str, +) -> CatalogRelation: + """SLayer's INFORMATION_SCHEMA.DIMENSIONS extension — JDBC-style type + names to match the contract the canned ``_serve_dimensions`` + previously emitted.""" + from slayer.facade.datatypes import datatype_to_jdbc + columns = [ + FacadeColumn(name="catalog_name", type=DataType.TEXT), + FacadeColumn(name="schema_name", type=DataType.TEXT), + FacadeColumn(name="table_name", type=DataType.TEXT), + FacadeColumn(name="dimension_name", type=DataType.TEXT), + FacadeColumn(name="description", type=DataType.TEXT), + FacadeColumn(name="data_type", type=DataType.TEXT), + FacadeColumn(name="label", type=DataType.TEXT), + FacadeColumn(name="is_time", type=DataType.BOOLEAN), + ] + rows: list[dict[str, Any]] = [] + for sch in catalog.schemas: + for tbl in sch.tables: + for d in tbl.dimensions: + rows.append({ + "catalog_name": datasource, + "schema_name": sch.name, "table_name": tbl.name, + "dimension_name": d.name, "description": d.description, + "data_type": datatype_to_jdbc(d.data_type), + "label": d.label, "is_time": d.is_time, + }) + return CatalogRelation(name="_is_dimensions", columns=columns, rows=rows) + + +def _check_collision(seen: dict[int, str], oid: int, key: str) -> None: + prior = seen.get(oid) + if prior is not None and prior != key: + raise ValueError( + f"pg_catalog OID collision: {key!r} and {prior!r} both hash to {oid}" + ) + seen[oid] = key + + +# --- is_catalog_only -------------------------------------------------------- + +# The set of known catalog relation names (bare). Both `pg_catalog.X` and +# `information_schema.X` schema qualifiers are stripped before lookup; the +# pre-rewrite pass aliases information_schema names to `_is_` so those go +# under their alias forms too. +# +# Must stay in lockstep with the relations built by ``build_catalog_relations`` +# above — every ``out.append(_build_pg_(...))`` needs a corresponding +# entry here, or the routing decision misclassifies the catalog query as a +# user-table reference and falls through to "Unknown schema: 'pg_catalog'". +_PG_CATALOG_NAMES = frozenset({ + "pg_namespace", "pg_class", "pg_attribute", "pg_type", "pg_proc", + "pg_settings", "pg_description", "pg_stat_user_tables", "pg_enum", + "pg_tables", "pg_views", "pg_matviews", "pg_constraint", "pg_index", + "pg_attrdef", + # psql backslash-command coverage. Every one is legitimately empty + # for SLayer (no roles / policies / triggers / publications / etc.). + "pg_am", "pg_roles", "pg_database", "pg_collation", "pg_policy", + "pg_trigger", "pg_inherits", + "pg_publication", "pg_publication_rel", "pg_publication_tables", + "pg_partitioned_table", "pg_rewrite", "pg_extension", +}) + +_INFO_SCHEMA_NAMES = frozenset({ + "columns", "table_constraints", "key_column_usage", + "schemata", "tables", "metrics", "dimensions", +}) + + +def _is_known_catalog_table(tbl: exp.Table) -> bool: + """True if ``tbl`` resolves to a catalog relation the executor should + handle — either one we've explicitly cataloged, or any table under + ``pg_catalog`` / ``information_schema`` (unknown ones are synthesized + as empty relations at execute time; see + ``_synthesize_missing_catalog_tables``). + + Bare table names still require an explicit allowlist match — bare + ``columns`` / ``tables`` from ``information_schema`` would otherwise + hijack user models with the same name. + """ + name = str(tbl.name).lower() + schema_part = tbl.args.get("db") + schema = None + if schema_part is not None: + schema = (str(schema_part.this) if hasattr(schema_part, "this") else str(schema_part)).lower() + catalog_part = tbl.args.get("catalog") + catalog = None + if catalog_part is not None: + catalog = ( + str(catalog_part.this) if hasattr(catalog_part, "this") else str(catalog_part) + ).lower() + # Three-part catalog-qualified refs must name the SLayer catalog. + # Anything else (a foreign catalog) is never our catalog SQL. + if catalog != CATALOG_NAME.lower(): + return False + if schema == "information_schema": + # Explicit info_schema refs route to the executor — unknown ones + # get synthesized as empty at execute time. + return True + if schema == "pg_catalog": + # Any explicit pg_catalog. ref routes to the executor. Unknown + # ones get synthesized as empty at execute time. + return True + if schema is None: + # Bare names — only well-known pg_catalog relations. Unknown bare + # names would shadow user models with the same name, so they + # never route to the catalog executor. + return name in _PG_CATALOG_NAMES + return False + + +def is_catalog_only(parsed: exp.Expression) -> bool: + """True iff ``parsed`` references at least one known catalog relation + or qualified catalog function, AND every Table node it walks resolves + to a known catalog relation (or a same-statement CTE). + + Tableless SELECTs are NOT auto-routed to the executor (CR review + feedback): the probe matcher already handles the standard tableless + probes (``SELECT 1``, ``SELECT current_database()``, ``SHOW …``), + and routing unknown tableless SQL through DuckDB would expand the + facade's accepted SQL surface with DuckDB semantics. Only tableless + SELECTs that explicitly reference catalog functions + (``::regclass``, ``pg_catalog.``, ``information_schema.``) + are accepted here. + """ + cte_names = { + str(cte.alias).lower() for cte in parsed.find_all(exp.CTE) + if cte.alias + } + saw_catalog_table = False + for tbl in parsed.find_all(exp.Table): + name = str(tbl.name).lower() + if name in cte_names: + continue + if not _is_known_catalog_table(tbl): + return False + saw_catalog_table = True + if saw_catalog_table: + return True + # Tableless: only catalog-only if the statement references a catalog + # function explicitly (regclass cast, qualified pg_catalog/info_schema + # function call, or a known stub function name). + return _references_catalog_function(parsed) + + +_CATALOG_FUNCTION_NAMES = frozenset({ + # Underscored spellings (Anonymous + bare-word Column refs). + "current_database", "current_catalog", "current_user", "session_user", + "current_role", "current_schemas", + "format_type", "obj_description", "col_description", "pg_get_userbyid", + "pg_table_is_visible", "pg_get_expr", "pg_total_relation_size", + "pg_encoding_to_char", + "has_table_privilege", "has_any_column_privilege", "has_schema_privilege", + "_pg_expandarray", + # psql / BI-tool helper stubs — see ``_CONSTANT_STUB_LITERALS`` for + # their return values (mostly NULL for features SLayer doesn't advertise). + "pg_get_constraintdef", "pg_get_indexdef", "pg_get_triggerdef", + "pg_get_viewdef", "pg_get_partkeydef", "pg_get_ruledef", + "pg_get_functiondef", "pg_get_statisticsobjdef", + "pg_get_statisticsobjdef_columns", "pg_get_statisticsobjdef_expressions", + "pg_get_serial_sequence", "pg_column_is_updatable", + "pg_get_object_address", "pg_identify_object", "pg_size_pretty", + "pg_relation_size", "pg_indexes_size", "pg_table_size", + "pg_database_size", "pg_tablespace_size", + # sqlglot's class-key forms for the dedicated Func subclasses + # (``type(node).key`` returns e.g. ``currentdatabase`` without the + # underscore — they appear here via ``_function_name_lower``). + "currentdatabase", "currentcatalog", "currentuser", "sessionuser", + "currentrole", "currentschema", "currentschemas", +}) + + +def _references_catalog_function(parsed: exp.Expression) -> bool: + """True iff ``parsed`` contains a ``::regclass``/``::regproc``/ + ``::regtype`` cast, a ``pg_catalog.`` or + ``information_schema.`` qualified function call, or a bare known + stub function name. Used by ``is_catalog_only`` to admit tableless + SELECTs only when they explicitly target catalog metadata.""" + return ( + _has_catalog_cast(parsed) + or _has_catalog_qualified_dot(parsed) + or _has_catalog_function_call(parsed) + ) + + +_CATALOG_CAST_KINDS = frozenset({"regclass", "regproc", "regtype"}) +_CATALOG_DOT_SCHEMAS = frozenset({"pg_catalog", "information_schema"}) + + +def _has_catalog_cast(parsed: exp.Expression) -> bool: + for cast in parsed.find_all(exp.Cast): + to = cast.args.get("to") + kind = getattr(to, "this", None) if to is not None else None + if kind is not None and str(kind).lower() in _CATALOG_CAST_KINDS: + return True + return False + + +def _has_catalog_qualified_dot(parsed: exp.Expression) -> bool: + for dot in parsed.find_all(exp.Dot): + lhs = dot.this + if isinstance(lhs, exp.Identifier) and str(lhs.this).lower() in _CATALOG_DOT_SCHEMAS: + return True + return False + + +def _has_catalog_function_call(parsed: exp.Expression) -> bool: + for node in parsed.walk(): + if _function_name_lower(node) in _CATALOG_FUNCTION_NAMES: + return True + # sqlglot parses bareword niladic context functions + # (``current_role``, ``current_user``, ``current_catalog`` …) as + # an unqualified ``Column``. Treat those as catalog-only too so + # ``SELECT current_role`` routes to the executor instead of + # falling through to the SLayer model-query path (which then + # errors with "no FROM clause"). CR/Codex review. + if isinstance(node, exp.Column) and not node.table: + ident = node.this + if isinstance(ident, exp.Identifier): + name = str(ident.this).lower() + if name in _CATALOG_FUNCTION_NAMES: + return True + return False + + +# --- AST pre-rewrite pass --------------------------------------------------- + + +# Constant-return stubs are AST-rewritten directly to literals (no DuckDB +# macros). This sidesteps DuckDB's lack of macro arity overloading — the +# corpus has 2-arg and 3-arg variants of has_*_privilege. +_CONSTANT_STUB_LITERALS: dict[str, Any] = { + "has_table_privilege": True, + "has_any_column_privilege": True, + "has_schema_privilege": True, + "pg_get_userbyid": "slayer", + "pg_table_is_visible": True, + "pg_total_relation_size": 0, + "pg_get_expr": None, + # SLayer always emits UTF8; ``\l`` calls this with ``d.encoding``. + "pg_encoding_to_char": "UTF8", + # Postgres helper functions psql's ``\d`` and BI tools call for + # definitions of things SLayer doesn't advertise (constraints / + # indexes / triggers / views / partitions / rules / stats objects / + # functions / serial-sequence / column-updatable). NULL is the right + # answer for absence — matches real Postgres behavior for a table + # that has none of these features. + "pg_get_constraintdef": None, + "pg_get_indexdef": None, + "pg_get_triggerdef": None, + "pg_get_viewdef": None, + "pg_get_partkeydef": None, + "pg_get_ruledef": None, + "pg_get_functiondef": None, + "pg_get_statisticsobjdef": None, + "pg_get_statisticsobjdef_columns": None, + "pg_get_statisticsobjdef_expressions": None, + "pg_get_serial_sequence": None, + "pg_column_is_updatable": False, + # ACL / policy-role helpers — return NULL / empty for absence. + "pg_get_object_address": None, + "pg_identify_object": None, + "pg_size_pretty": "0 bytes", + "pg_relation_size": 0, + "pg_indexes_size": 0, + "pg_table_size": 0, + "pg_database_size": 0, + "pg_tablespace_size": 0, +} + +# Data-lookup stubs are AST-renamed to private names; the macros are +# registered with a single arity per name. ``obj_description`` has two +# Postgres arities; the 2-arg form is normalised to the 1-arg by dropping +# the second argument at rewrite time. +_LOOKUP_STUB_NAMES: dict[str, str] = { + "format_type": "_slayer_format_type", + "obj_description": "_slayer_obj_description", + "col_description": "_slayer_col_description", + "_pg_expandarray": "_slayer_pg_expandarray", +} + + +def _function_name_lower(node: exp.Expression) -> str | None: + """Return the lowercased function name for any kind of function node, + or None if ``node`` isn't a function call.""" + if isinstance(node, exp.Anonymous): + n = node.args.get("this") + if n is not None: + return str(n).lower() + if isinstance(node, exp.Func): + # exp.Func sub-classes carry their name in `sql_name()` (CamelCase + # class -> snake_case). We use the class's `key` attribute. + return type(node).key.lower() + return None + + +def _to_literal(value: Any) -> exp.Expression: + """Build a sqlglot Literal for ``value``.""" + if value is None: + return exp.Null() + if isinstance(value, bool): + return exp.Boolean(this=value) + if isinstance(value, int): + return exp.Literal.number(value) + return exp.Literal.string(str(value)) + + +def _unwrap_qualified_stub_call(node: exp.Expression) -> exp.Expression | None: + """Detect ``information_schema.(args)`` (and the same for + ``pg_catalog.``) and rewrite the qualified call as a bare private-name + Anonymous so DuckDB resolves the macro. + + sqlglot parses ``information_schema._pg_expandarray(arr)`` as + ``Dot(this=Identifier("information_schema"), expression=Anonymous("_pg_expandarray", [arr]))``. + Returns the rewritten Anonymous, or None if no match. + """ + if not isinstance(node, exp.Dot): + return None + lhs = node.this + rhs = node.expression + if not isinstance(lhs, exp.Identifier): + return None + schema = str(lhs.this).lower() + if schema not in {"information_schema", "pg_catalog"}: + return None + name = _function_name_lower(rhs) + if name is None: + return None + if name in _CONSTANT_STUB_LITERALS: + return _to_literal(_CONSTANT_STUB_LITERALS[name]) + if name in _LOOKUP_STUB_NAMES: + private = _LOOKUP_STUB_NAMES[name] + args = list(rhs.args.get("expressions") or []) + if name == "obj_description" and len(args) > 1: + args = args[:1] + return exp.Anonymous(this=private, expressions=args) + # Unknown ``pg_catalog.`` / ``information_schema.`` — strip the + # schema qualifier and emit a bare call. Many such names (``array_to_string``, + # ``string_agg``, …) are real functions in DuckDB and resolve cleanly + # without the qualifier. Leaving the Dot node in place makes DuckDB + # interpret ``pg_catalog`` as a column reference and surface a + # misleading "column not found" Binder error; baring the call yields + # either a working resolution or a sensible "function not found". + args = list(rhs.args.get("expressions") or []) + return exp.Anonymous(this=name, expressions=args) + + +class _AstRewriter: + """Encapsulates the pre-rewrite pass. + + The pass walks the AST top-down and applies, in order: + + 1. Schema-qualifier strip on every Table node: + ``pg_catalog.X`` → ``X``; ``information_schema.X`` → ``_is_X``. + 2. ``current_schemas(...)[1]`` → ``'public'`` (short-circuit). + 3. ``::regclass`` casts (literal + dynamic). + 4. ``::regproc`` / ``::regtype`` → ``0``. + 5. Substitute zero-arg ``current_database``/``current_catalog``/ + ``current_user``/``session_user``/``current_role`` with stored + literals. + 6. Rename Postgres-only stubs (``format_type``, ``obj_description``…) + to private ``_slayer_*`` names. + 7. Rewrite Postgres regex operators (``~``/``!~``/``~*``/``!~*``) + to ``regexp_matches``/``NOT regexp_matches`` with the case flag. + 8. Unwrap Postgres schema-qualified operator syntax — psql emits + ``x OPERATOR(pg_catalog.~) y`` for ``\\du `` etc. + 9. Strip ``COLLATE pg_catalog.default`` (no DuckDB equivalent; + ASCII byte-compare is the right semantic for catalog names). + """ + + def __init__(self, *, datasource: str, + regclass_map: dict[str, int] | None = None) -> None: + self.datasource = datasource + self.regclass_map: dict[str, int] = regclass_map or {} + + def rewrite(self, parsed: exp.Expression) -> exp.Expression: + # Strip schema qualifiers AND rewrite information_schema names first + # so subsequent passes see the canonical bare names. + parsed = parsed.transform(self._strip_schema_qualifiers) + parsed = parsed.transform(self._strip_column_schema_qualifiers) + parsed = parsed.transform(self._rewrite_current_schemas_indexed) + parsed = parsed.transform(self._rewrite_current_schemas_bare) + parsed = parsed.transform(self._rewrite_pg_format_quoted_ident) + # Normalise schema-qualified CAST target types BEFORE the reg* + # rewriters below — ``pg_catalog.regclass`` becomes a bare + # ``regclass`` DataType so ``_rewrite_regclass_casts`` still + # catches it and performs the ``'foo'::regclass → OID`` lookup. + parsed = parsed.transform(self._rewrite_schema_qualified_cast) + parsed = parsed.transform(self._rewrite_regclass_casts) + parsed = parsed.transform(self._rewrite_regproc_regtype_casts) + parsed = parsed.transform(self._substitute_context_functions) + parsed = parsed.transform(self._rename_stub_functions) + parsed = parsed.transform(self._rewrite_regex_operators) + parsed = parsed.transform(self._rewrite_schema_qualified_operator) + parsed = parsed.transform(self._strip_collate_clause) + parsed = parsed.transform(self._rewrite_pg_any_array) + # Fallback for any lingering unknown ``pg_*`` function call — + # becomes NULL. Runs last so all known stubs claim their calls + # first. Semantically "SLayer doesn't advertise this feature", + # matching real Postgres for empty catalogs. + parsed = parsed.transform(self._stub_unknown_pg_functions) + return parsed + + # ----- 1. schema qualifier strip ---------------------------------------- + + @staticmethod + def _strip_schema_qualifiers(node: exp.Expression) -> exp.Expression: + if not isinstance(node, exp.Table): + return node + schema_part = node.args.get("db") + if schema_part is None: + return node + schema_name = ( + str(schema_part.this) if hasattr(schema_part, "this") else str(schema_part) + ).lower() + if schema_name == "pg_catalog": + new = node.copy() + new.set("db", None) + # Drop any outer catalog qualifier too (e.g. + # ``slayer.pg_catalog.pg_class`` → ``pg_class``). + new.set("catalog", None) + return new + if schema_name == "information_schema": + # Rewrite the table name itself: information_schema.X → _is_X. + new = node.copy() + new.set("db", None) + new.set("catalog", None) + inner_name = str(new.this.this) if hasattr(new.this, "this") else str(new.this) + new.set("this", exp.Identifier(this=f"_is_{inner_name.lower()}", quoted=False)) + return new + return node + + # ----- 1b. FORMAT('%I.%I', a, b) → CONCAT(a, '.', b) -------------------- + + @staticmethod + def _rewrite_pg_format_quoted_ident(node: exp.Expression) -> exp.Expression: + """Postgres' ``FORMAT('%I.%I', a, b)`` quotes both arguments and + returns ``"a"."b"``. DuckDB's ``FORMAT`` is printf-style and treats + ``%I`` literally. Rewrite the schema-qualified ident pattern to a + plain ``CONCAT(a, '.', b)`` so the regclass UDF sees ``public.orders`` + instead of an unfilled format string. Same shape applies to the + single-argument ``FORMAT('%I', a)`` → ``a``. + """ + if not isinstance(node, exp.Format): + return node + # sqlglot puts the spec at .this and the args under .expressions. + fmt = node.this + if not (isinstance(fmt, exp.Literal) and fmt.is_string): + return node + spec = str(fmt.this) + rest = list(node.args.get("expressions") or []) + if spec == "%I.%I" and len(rest) == 2: + return exp.Anonymous( + this="concat", + expressions=[rest[0], exp.Literal.string("."), rest[1]], + ) + if spec == "%I" and len(rest) == 1: + return rest[0] + return node + + # ----- 2. current_schemas(...)[1] → 'public' ---------------------------- + + @staticmethod + def _strip_column_schema_qualifiers(node: exp.Expression) -> exp.Expression: + """Rewrite ``pg_catalog..`` / ``information_schema..`` + column refs so they match the underlying DuckDB table names + after the FROM-side strip (Codex review). + + sqlglot's ``exp.Column`` represents the dotted qualifiers as + ``this`` (leaf), ``table`` (the table-qualifier), ``db`` (the + schema-qualifier), ``catalog`` (the catalog-qualifier — only + for 4-part refs). + + * For ``pg_catalog`` the underlying table keeps the same name + (``pg_namespace.nspname``); we drop ``db`` so the column + resolves as a 2-part ``pg_namespace.nspname`` ref. + * For ``information_schema`` the table is renamed to ``_is_`` + by ``_strip_schema_qualifiers``; here we apply the same + rename to the column's ``table`` qualifier so + ``information_schema.columns.column_name`` resolves to + ``_is_columns.column_name``. + """ + if not isinstance(node, exp.Column): + return node + db_part = node.args.get("db") + table_part = node.args.get("table") + if db_part is None: + return node + schema = ( + str(db_part.this) if hasattr(db_part, "this") else str(db_part) + ).lower() + if schema not in {"pg_catalog", "information_schema"}: + return node + new = node.copy() + new.set("db", None) + # Drop any outer catalog qualifier too (e.g. + # ``slayer.pg_catalog.pg_class.oid``) — symmetric with the + # FROM-side strip in ``_strip_schema_qualifiers``. Otherwise the + # 4-part column ref keeps a stale ``slayer.pg_class.oid`` form + # that fails to bind against DuckDB's ``main.pg_class`` (Codex + # round-20 follow-up). + new.set("catalog", None) + if schema == "information_schema" and table_part is not None: + tbl_name = ( + str(table_part.this) if hasattr(table_part, "this") else str(table_part) + ).lower() + new.set("table", exp.Identifier(this=f"_is_{tbl_name}", quoted=False)) + return new + + @staticmethod + def _rewrite_current_schemas_bare(node: exp.Expression) -> exp.Expression: + """Rewrite a bare ``current_schemas(...)`` call (no bracket index) + to ``['public']`` — the facade only advertises one schema, so + the unindexed call must return the single-element list rather + than fall through to DuckDB's internal schema list (CR/Codex + review). Handles bare and ``pg_catalog.current_schemas(...)`` + qualified forms; the indexed form is rewritten separately by + ``_rewrite_current_schemas_indexed``.""" + if _AstRewriter._is_current_schemas(node): + return exp.Anonymous( + this="list_value", expressions=[exp.Literal.string("public")], + ) + return node + + @staticmethod + def _is_current_schemas(node: exp.Expression) -> bool: + if isinstance(node, exp.CurrentSchemas): + return True + if _function_name_lower(node) == "current_schemas": + return True + # Qualified ``pg_catalog.current_schemas(...)`` form. + if isinstance(node, exp.Dot): + lhs = node.this + rhs = node.expression + if isinstance(lhs, exp.Identifier) and str(lhs.this).lower() == "pg_catalog": + if isinstance(rhs, exp.CurrentSchemas): + return True + if _function_name_lower(rhs) == "current_schemas": + return True + return False + + @staticmethod + def _rewrite_current_schemas_indexed(node: exp.Expression) -> exp.Expression: + # Match Bracket(this=Paren?(CurrentSchemas|Anonymous("current_schemas"))) + # AND a single literal index ``[1]`` — other indices are not + # safely collapsible to 'public' because the facade only advertises + # one schema (cf. CR review feedback). sqlglot normalises 1-based + # SQL array indices to 0-based internally, so user-level ``[1]`` + # arrives as ``Literal(0)``. Handles bare and + # ``pg_catalog.current_schemas(...)`` qualified forms. + if not isinstance(node, exp.Bracket): + return node + inner = node.this + if isinstance(inner, exp.Paren): + inner = inner.this + if not _AstRewriter._is_current_schemas(inner): + return node + indices = node.args.get("expressions") or [] + if len(indices) != 1: + return node + index = indices[0] + if not (isinstance(index, exp.Literal) and not index.is_string): + return node + try: + if int(str(index.this)) != 0: # 0-based — user's [1] + return node + except ValueError: + return node + return exp.Literal.string("public") + + # ----- 3. regclass casts ------------------------------------------------ + + def _rewrite_regclass_casts(self, node: exp.Expression) -> exp.Expression: + if not isinstance(node, exp.Cast): + return node + target_kind = self._cast_target_kind(node) + if target_kind != "regclass": + return node + inner = node.this + # Static: CAST('foo' AS REGCLASS) → integer OID literal. + if isinstance(inner, exp.Literal) and inner.is_string: + text = inner.this # raw string value (no quotes) + return exp.Literal.number(self._lookup_static_regclass(text)) + # Numeric-typed inner (column ref or number literal): pass through + # unchanged. Column refs are OID columns like ``c.oid``, and + # ``CAST(0 AS regclass)`` is Postgres's InvalidOid sentinel. Both + # would fail against the VARCHAR-only UDF, and the numeric value + # itself is the most useful answer for SLayer's minimal catalog. + if isinstance(inner, exp.Column): + return inner + if isinstance(inner, exp.Literal) and not inner.is_string: + return inner + # Everything else — function calls, nested casts, expressions — + # routes through the UDF lookup. Metabase's real query uses + # ``CAST(FORMAT('%I.%I', schema, table) AS regclass)`` to look up + # a class OID by its qualified name; that's the shape the UDF was + # built for. + return exp.Anonymous(this="slayer_regclass_oid", expressions=[inner]) + + @staticmethod + def _cast_target_kind(node: exp.Cast) -> str | None: + to = node.args.get("to") + if to is None: + return None + kind_attr = getattr(to, "this", None) + if kind_attr is not None: + return str(kind_attr).lower() + return None + + def _lookup_static_regclass(self, text: str) -> int: + # Check the full schema-qualified name, then the bare leaf name. + # The map carries both forms for user tables ('public.orders' AND + # 'orders') plus the well-known system OIDs. + return ( + self.regclass_map.get(text) + or self.regclass_map.get(text.lower()) + or self.regclass_map.get(text.split(".")[-1].lower(), 0) + ) + + # ----- 4. regproc / regtype casts --------------------------------------- + + def _rewrite_regproc_regtype_casts(self, node: exp.Expression) -> exp.Expression: + """``::regproc`` / ``::regoper`` / ``::regoperator`` / ``::regprocedure`` + → ``0`` (their catalog tables are empty in this facade). + ``::regtype`` → ``pg_type.oid`` lookup for known type names, ``0`` + otherwise — so ``WHERE oid = 'int8'::regtype`` matches the int8 + row in pg_type.""" + if not isinstance(node, exp.Cast): + return node + target_kind = self._cast_target_kind(node) + if target_kind in {"regproc", "regoper", "regoperator", "regprocedure"}: + return exp.Literal.number(0) + if target_kind == "regtype": + inner = node.this + if isinstance(inner, exp.Literal) and inner.is_string: + return exp.Literal.number(_KNOWN_TYPE_OIDS.get( + str(inner.this).lower(), 0, + )) + return exp.Literal.number(0) + return node + + # ----- 5. context function substitution --------------------------------- + + def _substitute_context_functions(self, node: exp.Expression) -> exp.Expression: + # Try each substitution branch in order; first hit wins. + substituted = ( + self._substitute_qualified_context_call(node) + or self._substitute_qualified_context_column(node) + or self._substitute_dedicated_func(node) + or self._substitute_bareword_column(node) + or self._substitute_anonymous_function(node) + ) + return substituted if substituted is not None else node + + def _substitute_qualified_context_call( + self, node: exp.Expression, + ) -> exp.Expression | None: + """Replace ``pg_catalog.`` (and ``pg_catalog.()``) + as a whole so the outer ``Dot`` doesn't end up wrapping a string + literal (``pg_catalog.'jaffle'`` — invalid SQL). + + sqlglot parses ``pg_catalog.current_database()`` as + ``Dot(Identifier('pg_catalog'), CurrentDatabase(...))``; without + this branch the inner-node rewrite would leave the Dot intact. + """ + if not isinstance(node, exp.Dot): + return None + lhs = node.this + if not isinstance(lhs, exp.Identifier): + return None + if str(lhs.this).lower() != "pg_catalog": + return None + rhs = node.expression + return ( + self._substitute_dedicated_func(rhs) + or self._substitute_bareword_column(rhs) + or self._substitute_anonymous_function(rhs) + ) + + def _substitute_qualified_context_column( + self, node: exp.Expression, + ) -> exp.Expression | None: + """Replace ``pg_catalog.`` where sqlglot parses the + whole thing as ``Column(this=, table='pg_catalog')`` — the + no-parens shape (``pg_catalog.current_user``, + ``pg_catalog.current_catalog``). The Dot-shaped variant + (``pg_catalog.current_database()``) is handled by + ``_substitute_qualified_context_call``. + """ + if not isinstance(node, exp.Column): + return None + table = node.args.get("table") + if table is None: + return None + table_name = ( + str(table.this) if hasattr(table, "this") else str(table) + ).lower() + if table_name != "pg_catalog": + return None + ident = node.this + if not isinstance(ident, exp.Identifier): + return None + return self._literal_for_context_name(str(ident.this).lower()) + + def _substitute_dedicated_func(self, node: exp.Expression) -> exp.Expression | None: + """Dedicated sqlglot Func subclasses (typed nodes for niladic ctx fns).""" + if isinstance(node, (exp.CurrentDatabase, getattr(exp, "CurrentCatalog", exp.CurrentDatabase))): + return exp.Literal.string(self.datasource) + if isinstance(node, (exp.CurrentUser, exp.SessionUser)): + return exp.Literal.string("slayer") + if isinstance(node, exp.CurrentSchema): + return exp.Literal.string("public") + return None + + def _substitute_bareword_column(self, node: exp.Expression) -> exp.Expression | None: + """sqlglot parses ``current_role`` (no parens) as a Column reference. + Treat single-token unqualified Column refs naming a known niladic ctx + function as that function. ``node.table`` is ``""`` (not None) for an + unqualified column.""" + if not (isinstance(node, exp.Column) and not node.table): + return None + ident = node.this + if not isinstance(ident, exp.Identifier): + return None + return self._literal_for_context_name(str(ident.this).lower()) + + def _substitute_anonymous_function(self, node: exp.Expression) -> exp.Expression | None: + """Less-common Anonymous function spellings — fallback path.""" + name = _function_name_lower(node) + if name is None: + return None + return self._literal_for_context_name(name) + + def _literal_for_context_name(self, name: str) -> exp.Expression | None: + if name in {"current_database", "current_catalog", + "currentdatabase", "currentcatalog"}: + return exp.Literal.string(self.datasource) + if name in {"current_user", "session_user", "current_role", + "currentuser", "sessionuser", "currentrole"}: + return exp.Literal.string("slayer") + # current_schema() / current_schema → 'public' (the single schema + # the pg facade advertises). DuckDB has its own current_schema + # (returning 'main'), so we must rewrite explicitly even inside + # catalog SQL. + if name in {"current_schema", "currentschema"}: + return exp.Literal.string("public") + return None + + # ----- 6. rename stub functions to private names ------------------------ + + @staticmethod + def _rename_stub_functions(node: exp.Expression) -> exp.Expression: + # Strip qualified function calls of the form + # ``Dot(this=Paren?(Dot(Identifier(), + # expression=Anonymous((args)))), expression=)``. + # The full pattern arises from corpus #14's + # ``(information_schema._pg_expandarray(i.indkey)).n`` — we want to + # rewrite the inner Anonymous to the private name in place and let + # the outer Dot continue to extract the field. + unwrapped = _unwrap_qualified_stub_call(node) + if unwrapped is not None: + return unwrapped + name = _function_name_lower(node) + if name is None: + return node + # Constant-return stubs collapse straight to a literal regardless of + # arity (sidesteps DuckDB's no-overload-by-arity macro limitation). + if name in _CONSTANT_STUB_LITERALS: + return _to_literal(_CONSTANT_STUB_LITERALS[name]) + if name in _LOOKUP_STUB_NAMES: + private = _LOOKUP_STUB_NAMES[name] + args = list(node.args.get("expressions") or []) + # Postgres obj_description has 2-arg form (oid, catname); the + # macro is single-arg, so drop the second. + if name == "obj_description" and len(args) > 1: + args = args[:1] + return exp.Anonymous(this=private, expressions=args) + return node + + # ----- 8. = ANY() → FALSE ----------------------------------- + + @staticmethod + def _rewrite_pg_any_array(node: exp.Expression) -> exp.Expression: + """Postgres' `` = ANY()`` performs array-membership testing. + DuckDB needs typed array columns to evaluate this; declaring + ``pg_constraint.conkey`` as ``BIGINT[]`` would work but couples + the wire-type abstraction. Since the relations that carry array + columns (``pg_constraint``, ``pg_index``) are always empty, the + WHERE result is always false anyway — rewrite the comparison to a + literal ``FALSE`` so the bind step succeeds. + """ + if not isinstance(node, exp.EQ): + return node + rhs = node.expression + if not isinstance(rhs, exp.Any): + return node + # ``ANY()`` wraps the inner in a Paren; unwrap before the + # column check. + inner = rhs.this + if isinstance(inner, exp.Paren): + inner = inner.this + if isinstance(inner, exp.Column): + return exp.Boolean(this=False) + return node + + # ----- Unknown pg_* function fallback ----------------------------------- + + # DuckDB-native ``pg_*`` names we must NOT rewrite. Extend as DuckDB + # adds new Postgres compatibility helpers. + _DUCKDB_NATIVE_PG_FUNCTIONS: frozenset[str] = frozenset() + + @classmethod + def _stub_unknown_pg_functions(cls, node: exp.Expression) -> exp.Expression: + """Fallback for unknown ``pg_*`` function calls — return NULL. + + Any Anonymous call whose name starts with ``pg_`` and isn't a known + catalog stub or DuckDB native gets replaced with a NULL literal + + WARN log. Semantically "SLayer doesn't advertise this feature" — + the same answer real Postgres gives when the underlying catalog + table is empty. + + User-defined SQL functions (no ``pg_`` prefix) are left alone — + typos in user queries still surface loudly. + """ + if not isinstance(node, exp.Anonymous): + return node + name = str(node.args.get("this", "")).lower() + if not name.startswith("pg_"): + return node + # Already-known stubs get resolved by earlier passes; if we still + # see a ``pg_*`` Anonymous here it's genuinely unknown. + if name in _CATALOG_FUNCTION_NAMES: + return node + if name in cls._DUCKDB_NATIVE_PG_FUNCTIONS: + return node + # The AST-renamed private stubs (``_slayer_*``) don't start with + # ``pg_``, so they can't hit this branch — safe. + logger.warning( + "pg-facade: stubbed unknown Postgres helper function %s(...) → NULL — " + "consider adding it to _CONSTANT_STUB_LITERALS if this recurs.", + name, + ) + return exp.Null() + + # ----- 7. regex operator rewrites --------------------------------------- + + @staticmethod + def _rewrite_regex_operators(node: exp.Expression) -> exp.Expression: + # Postgres parses these as exp.Binary subclasses. sqlglot maps: + # x ~ y → exp.RegexpLike(this=x, expression=y) + # x ~* y → exp.RegexpILike (or RegexpLike with flag=i) + # x !~ y → exp.Not(this=RegexpLike(...)) + # The cleanest approach: rebuild as Anonymous regexp_matches calls. + if isinstance(node, exp.RegexpLike): + return exp.Anonymous( + this="regexp_matches", + expressions=[node.this, node.expression], + ) + if isinstance(node, exp.RegexpILike): + return exp.Anonymous( + this="regexp_matches", + expressions=[node.this, node.expression, exp.Literal.string("i")], + ) + return node + + # Postgres's ``OPERATOR(.)`` calls a schema-qualified operator + # — psql emits ``x OPERATOR(pg_catalog.~) y`` for ``\du `` / + # ``\dt `` etc. sqlglot parses this as ``exp.Operator(this=lhs, + # operator="pg_catalog.~", expression=rhs)``. DuckDB has no OPERATOR() + # syntax; strip the schema qualifier and rebuild the equivalent + # expression the previous pass understands (RegexpLike / RegexpILike or + # a plain binary op). + _PG_REGEX_OPS = {"~", "~*", "!~", "!~*"} + + @staticmethod + def _rewrite_schema_qualified_operator(node: exp.Expression) -> exp.Expression: + if not isinstance(node, exp.Operator): + return node + raw = str(node.args.get("operator", "")) + # ``pg_catalog.~`` → ``~``; a plain ``=`` etc. stays unchanged. + op = raw.rsplit(".", 1)[-1] + lhs = node.this + rhs = node.expression + if op == "~": + return exp.Anonymous(this="regexp_matches", expressions=[lhs, rhs]) + if op == "~*": + return exp.Anonymous( + this="regexp_matches", + expressions=[lhs, rhs, exp.Literal.string("i")], + ) + if op == "!~": + return exp.Not(this=exp.Anonymous( + this="regexp_matches", expressions=[lhs, rhs], + )) + if op == "!~*": + return exp.Not(this=exp.Anonymous( + this="regexp_matches", + expressions=[lhs, rhs, exp.Literal.string("i")], + )) + # Unknown / plain operator: strip the schema qualifier and re-emit + # as a bare binary operation. Preserves generality without adding + # per-operator branches. + return exp.condition(f"({lhs.sql()}) {op} ({rhs.sql()})") + + # ``COLLATE pg_catalog."default"`` (psql emits this next to the regex + # match on \du etc.) has no DuckDB equivalent. Strip the whole Collate + # wrapper — matching is done byte-wise, which is the right semantic + # against ASCII role / table names. + @staticmethod + def _strip_collate_clause(node: exp.Expression) -> exp.Expression: + if isinstance(node, exp.Collate): + return node.this + return node + + # ``CAST(x AS pg_catalog.text)`` — psql's ``\d`` emits schema-qualified + # type names. sqlglot parses these as ``DataType(USERDEFINED, + # kind=Dot(pg_catalog, ))``. DuckDB doesn't accept them; map + # each Postgres type to its DuckDB equivalent. Postgres OID-family + # types (regtype, regclass, regproc, …) all become BIGINT since OIDs + # are integers on the wire. Unknown types fall back to VARCHAR. + _PG_TO_DUCKDB_TYPES: dict[str, str] = { + "text": "VARCHAR", "varchar": "VARCHAR", "char": "VARCHAR", + "name": "VARCHAR", + "int2": "SMALLINT", "int4": "INTEGER", "int8": "BIGINT", + "oid": "BIGINT", + "float4": "REAL", "float8": "DOUBLE", "numeric": "DECIMAL", + "bool": "BOOLEAN", "date": "DATE", + "timestamp": "TIMESTAMP", "timestamptz": "TIMESTAMP", + "bytea": "BLOB", + } + # Reg-family types need OID-lookup semantics (``'foo'::regclass → + # pg_class.oid``), not plain integer coercion. Normalise them to + # bare DataType names so the existing ``_rewrite_regclass_casts`` / + # ``_rewrite_regproc_regtype_casts`` passes catch them next. + _PG_REG_TYPES = frozenset({ + "regclass", "regproc", "regtype", + "regoper", "regoperator", "regprocedure", + }) + + @classmethod + def _rewrite_schema_qualified_cast(cls, node: exp.Expression) -> exp.Expression: + if not isinstance(node, exp.DataType): + return node + # USERDEFINED with a dotted ``kind`` is how sqlglot represents + # ``pg_catalog.`` on the postgres dialect. + if node.args.get("this") != exp.DataType.Type.USERDEFINED: + return node + kind = node.args.get("kind") + if not isinstance(kind, exp.Dot): + return node + lhs = kind.this + rhs = kind.expression + if not (isinstance(lhs, exp.Identifier) and str(lhs.this).lower() == "pg_catalog"): + return node + if not isinstance(rhs, exp.Identifier): + return node + pg_type = str(rhs.this).lower() + if pg_type in cls._PG_REG_TYPES: + # Rebuild as a bare DataType so the reg* rewriters below can + # dispatch on ``_cast_target_kind() == 'regclass'`` etc. + return exp.DataType.build(pg_type.upper(), dialect="postgres") + duckdb_type = cls._PG_TO_DUCKDB_TYPES.get(pg_type, "VARCHAR") + return exp.DataType.build(duckdb_type) + + +# --- DuckDB type mapping ---------------------------------------------------- + + +_DUCKDB_TO_DATATYPE: dict[str, DataType] = { + # ints + "TINYINT": DataType.INT, "SMALLINT": DataType.INT, + "INTEGER": DataType.INT, "BIGINT": DataType.INT, "HUGEINT": DataType.INT, + "UTINYINT": DataType.INT, "USMALLINT": DataType.INT, + "UINTEGER": DataType.INT, "UBIGINT": DataType.INT, + "INT1": DataType.INT, "INT2": DataType.INT, "INT4": DataType.INT, + "INT8": DataType.INT, + "BIGINT[]": DataType.TEXT, # arrays fall back to TEXT + # floats + "REAL": DataType.DOUBLE, "FLOAT": DataType.DOUBLE, + "DOUBLE": DataType.DOUBLE, "DECIMAL": DataType.DOUBLE, + "NUMERIC": DataType.DOUBLE, "FLOAT4": DataType.DOUBLE, + "FLOAT8": DataType.DOUBLE, + # bool / date / timestamp / text + "BOOLEAN": DataType.BOOLEAN, "BOOL": DataType.BOOLEAN, + "DATE": DataType.DATE, + "TIMESTAMP": DataType.TIMESTAMP, + "TIMESTAMP_NS": DataType.TIMESTAMP, "TIMESTAMP_MS": DataType.TIMESTAMP, + "TIMESTAMP_S": DataType.TIMESTAMP, + "DATETIME": DataType.TIMESTAMP, + "VARCHAR": DataType.TEXT, "TEXT": DataType.TEXT, "STRING": DataType.TEXT, + "CHAR": DataType.TEXT, "BPCHAR": DataType.TEXT, +} + + +def _duckdb_typename_to_datatype(typename: str) -> DataType: + """Map a DuckDB column type name to one of the six coarse SLayer + ``DataType``s. Anything unmapped falls back to ``TEXT`` so wire encoding + has a safe path.""" + base = str(typename).split("(")[0].split("[")[0].strip().upper() + return _DUCKDB_TO_DATATYPE.get(base, DataType.TEXT) + + +# --- DuckDB type mapping for table columns at creation time ---------------- + + +_DATATYPE_TO_DUCKDB_CREATE: dict[DataType, str] = { + DataType.INT: "BIGINT", + DataType.DOUBLE: "DOUBLE", + DataType.TEXT: "VARCHAR", + DataType.BOOLEAN: "BOOLEAN", + DataType.DATE: "DATE", + DataType.TIMESTAMP: "TIMESTAMP", +} + + +# --- CatalogSqlExecutor ----------------------------------------------------- + + +class CatalogSqlExecutor: + """Owns one in-memory DuckDB connection with the catalog materialised. + + Construction is expensive (table creates + bulk inserts + macro defs); + cache via ``executor_for(catalog)``. + """ + + def __init__( + self, + *, + catalog: FacadeCatalog, + datasource: str, + extra_relations: Iterable[CatalogRelation] | None = None, + ) -> None: + self._datasource = datasource + self._conn = duckdb.connect(":memory:") + # DEV-1558 security hardening (Codex round 9): lock down the + # DuckDB instance so a catalog-shaped query can't pivot through + # built-ins like ``read_text('/etc/hostname')`` to exfiltrate + # local files. ``enable_external_access=false`` is DuckDB's + # one-shot kill switch — it blocks all filesystem / HTTP / S3 + # readers at bind time (BinderException). The setting cannot be + # re-enabled within the same connection by an authenticated + # client. ``lock_configuration`` (DuckDB 1.x) further prevents + # any later SET from re-opening the door if a future code path + # registers something risky. + self._conn.execute("SET enable_external_access = false") + try: + self._conn.execute("SET lock_configuration = true") + except duckdb.Error: + # Older DuckDB versions don't have lock_configuration; + # enable_external_access alone is still binding. + pass + # OID lookup for the regclass UDF: maps schema-qualified and bare + # names to OIDs (system catalogs + user tables). Three forms are + # registered per table so Metabase's COL_DESCRIPTION etc. resolve + # whichever spelling psql / a BI tool emits: + # - ``.
`` — the authoritative form for custom + # postgres_schema datasources (without this, COL_DESCRIPTION + # silently returns OID 0 and drops descriptions); + # - ``public.
`` — the back-compat alias the facade always + # advertised before custom schemas existed; + # - ``
`` — bare, for clients that don't qualify. + self._regclass_map: dict[str, int] = dict(KNOWN_SYSTEM_OIDS) + for ds, tbl in _all_tables(catalog): + oid = _table_oid(ds, tbl) + self._regclass_map[f"{ds}.{tbl.name}"] = oid + self._regclass_map[f"public.{tbl.name}"] = oid + self._regclass_map[tbl.name] = oid + self._rewriter = _AstRewriter( + datasource=datasource, regclass_map=self._regclass_map, + ) + relations = build_catalog_relations( + catalog, datasource, extra_relations=extra_relations, + ) + self._registered_tables: set[str] = set() + for relation in relations: + self._register_relation(relation) + self._register_stubs() + + def _register_relation(self, relation: CatalogRelation) -> None: + cols_ddl = ", ".join( + f'"{c.name}" {_DATATYPE_TO_DUCKDB_CREATE[c.type]}' + for c in relation.columns + ) + self._conn.execute(f'CREATE TABLE "{relation.name}" ({cols_ddl})') + self._registered_tables.add(relation.name.lower()) + if not relation.rows: + return + # Bulk insert via a single statement with a row-list expansion. + col_names = [c.name for c in relation.columns] + placeholders = "(" + ", ".join("?" for _ in col_names) + ")" + params = [tuple(row.get(name) for name in col_names) for row in relation.rows] + sql = ( + f'INSERT INTO "{relation.name}" ' + f'({", ".join(f"{chr(34)}{n}{chr(34)}" for n in col_names)}) VALUES {placeholders}' + ) + self._conn.executemany(sql, params) + + def _register_stubs(self) -> None: + # Only the data-aware stubs need DuckDB macros — constant-return + # stubs collapse to literals in the AST rewrite pass. + macros = [ + "CREATE MACRO _slayer_format_type(p_oid, p_typmod) AS " + " COALESCE((SELECT typname FROM pg_type WHERE oid = p_oid LIMIT 1), 'text')", + "CREATE MACRO _slayer_obj_description(p_oid) AS " + " (SELECT description FROM pg_description " + " WHERE objoid = p_oid AND objsubid = 0 LIMIT 1)", + "CREATE MACRO _slayer_col_description(p_oid, p_attnum) AS " + " (SELECT description FROM pg_description " + " WHERE objoid = p_oid AND objsubid = p_attnum LIMIT 1)", + # _pg_expandarray returns a struct with x and n fields so + # `(stub).x` / `(stub).n` field-access parses; combined with + # empty pg_index, corpus #14 yields zero rows. + "CREATE MACRO _slayer_pg_expandarray(p_arr) AS " + " STRUCT_PACK(x := CAST(NULL AS INTEGER), n := CAST(NULL AS INTEGER))", + ] + for sql in macros: + self._conn.execute(sql) + # Python UDF for the dynamic-regclass path. + self._conn.create_function( + "slayer_regclass_oid", self._regclass_oid, + ["VARCHAR"], "INTEGER", + ) + + def _regclass_oid(self, text: str | None) -> int: + if text is None: + return 0 + # Both schema-qualified and bare lookups go through the same map. + return self._regclass_map.get(text, self._regclass_map.get(text.lower(), 0)) + + _SYNTHETIC_SCHEMAS: frozenset[str] = frozenset({ + "pg_catalog", "information_schema", + }) + + def _synthesize_missing_catalog_tables( + self, root: exp.Expression, + ) -> exp.Expression: + """Replace ``pg_catalog.`` / ``information_schema.`` Table + refs whose ```` isn't in ``self._registered_tables`` with an + inline ``(SELECT NULL::VARCHAR AS c1, ... WHERE FALSE) AS `` + subquery. Columns come from scanning the same query for column + refs against the Table's alias (or bare-name qualifier). NEVER + fires for user-schema references — those keep the loud "unknown + table" error. + + Fallback for obscure catalog tables (``pg_statistic_ext``, + ``pg_hba_file_rules``, …) SLayer doesn't advertise. Empty result + matches real Postgres for tables with no rows. Emits a WARN log + per synthesis so recurring hits are visible. + """ + replacements: list[tuple[exp.Table, exp.Subquery]] = [] + for tbl in root.find_all(exp.Table): + # Runs BEFORE strip-schema-qualifier, so ``db`` still carries + # the original schema qualifier. Only synthesize when it's + # explicitly ``pg_catalog`` / ``information_schema`` — bare + # names might be user tables or aliases, and never fabricating + # for those keeps typo diagnostics loud. + db_part = tbl.args.get("db") + if db_part is None: + continue + schema = ( + str(db_part.this) if hasattr(db_part, "this") else str(db_part) + ).lower() + if schema not in self._SYNTHETIC_SCHEMAS: + continue + name = str(tbl.name).lower() + if name in self._registered_tables: + continue + # Info-schema tables are registered under the ``_is_`` + # alias in DuckDB (see the builder-name remap in + # ``_index_catalog_relations``); check that form too. + if schema == "information_schema" and f"_is_{name}" in self._registered_tables: + continue + # Discover column names the same query references off this + # Table's alias (or bare name). Column discovery matches on the + # user-visible handle (still ``pg_publication`` / ``character_sets`` + # at this point — the strip pass hasn't run). + handle = tbl.alias or tbl.name + columns = self._discover_referenced_columns(root, handle) + if not columns: + # No column refs — the query probably does ``SELECT *``; + # give it a single throwaway ``oid`` column so DuckDB has + # something to project. Real queries always reference some + # column, so this is a rare path. + columns = ["oid"] + # For information_schema tables without a user-supplied alias, + # the subquery must ALIAS AS ``_is_`` — that's what the + # downstream ``_strip_column_schema_qualifiers`` pass rewrites + # bare ``information_schema.character_sets.col`` refs to. If we + # aliased the synthetic subquery as ``character_sets``, the + # binder would miss the column. + output_alias = handle + if schema == "information_schema" and not tbl.alias: + output_alias = f"_is_{name}" + subq = self._build_empty_subquery(columns, alias=output_alias) + replacements.append((tbl, subq)) + logger.warning( + "pg-facade: synthesized empty relation for unknown catalog " + "table %r (columns: %s) — consider adding a stub in " + "build_catalog_relations if this recurs.", + (schema or "pg_catalog") + "." + name, ", ".join(columns), + ) + for old, new in replacements: + old.replace(new) + return root + + @staticmethod + def _discover_referenced_columns( + root: exp.Expression, handle: str, + ) -> list[str]: + """Scan ``root`` for Column refs that plausibly belong to the + Table being synthesized: + + - qualified refs (``handle.col``): keep — unambiguous. + - bare refs (``col`` with no ``table=`` qualifier): keep too. A + bare col could technically belong to any FROM table, but when + we synthesize an empty-relation stand-in the columns type as + VARCHAR NULL — adding one that "shouldn't" belong is harmless + (the query never reads from the synthetic row set anyway). + The alternative would need sqlglot Scope resolution across all + FROM tables, which is complex for negligible correctness gain. + + Returns distinct column names in first-seen order.""" + seen: dict[str, None] = {} + handle_lower = handle.lower() + for col in root.find_all(exp.Column): + tbl_ident = col.args.get("table") + if tbl_ident is not None: + tbl_name = ( + str(tbl_ident.this) if hasattr(tbl_ident, "this") + else str(tbl_ident) + ).lower() + if tbl_name != handle_lower: + continue + # tbl_ident is None (bare) OR matches handle — take it. + colname = str(col.name) + if colname and colname not in seen: + seen[colname] = None + return list(seen.keys()) + + @staticmethod + def _build_empty_subquery( + columns: list[str], *, alias: str, + ) -> exp.Subquery: + """Emit ``(SELECT NULL::VARCHAR AS c1, ... WHERE FALSE) AS ``. + Every column types as VARCHAR — NULL literals coerce freely to any + downstream type, and the ``WHERE FALSE`` guarantees zero rows so + the type choice is only for column-shape validation. + + Column names come from client-controlled SQL identifiers; escape + the ``"`` character (SQL identifier delimiter) before embedding + so a valid quoted-identifier column ref can't malform the SQL we + parse. Double the quote character per the SQL-standard escape. + """ + def _quote_ident(name: str) -> str: + return '"' + name.replace('"', '""') + '"' + + select_cols = ", ".join( + f"NULL::VARCHAR AS {_quote_ident(c)}" for c in columns + ) + sub_sql = f"SELECT {select_cols} WHERE FALSE" + parsed_sub = sqlglot.parse_one(sub_sql, read="duckdb") + return exp.Subquery( + this=parsed_sub, + alias=exp.TableAlias(this=exp.to_identifier(alias)), + ) + + def execute(self, *, parsed: exp.Expression, sql: str) -> RowBatch: + # Synthesize empties for unknown ``pg_catalog.`` refs BEFORE + # the rewriter's strip-schema-qualifier pass — after that runs, + # the ``db=pg_catalog`` qualifier is gone and we can't distinguish + # "originally schema-qualified" from "bare name that happens to + # shadow a user table". + parsed = self._synthesize_missing_catalog_tables(parsed.copy()) + rewritten = self._rewriter.rewrite(parsed) + try: + duckdb_sql = rewritten.sql(dialect="duckdb") + cursor = self._conn.execute(duckdb_sql) + description = cursor.description + data_rows = cursor.fetchall() + except Exception as exc: # noqa: BLE001 — every DuckDB failure surfaces + from slayer.facade.translator import TranslationError + logger.warning("catalog-sql exec failed: %s\nSQL: %s", exc, sql) + raise TranslationError(str(exc)) from exc + if description is None: + return RowBatch(columns=[], rows=[]) + columns: list[FacadeColumn] = [] + col_keys: list[str] = [] + seen_keys: dict[str, int] = {} + for col in description: + name = col[0] + typename = col[1] if len(col) > 1 else "VARCHAR" + # CR/Codex review: Postgres allows duplicate output names + # (``SELECT oid AS x, relname AS x``); the row-as-dict shape + # would collapse them. Keep the user-visible ``name`` on the + # FacadeColumn (so the wire RowDescription still reports the + # duplicate name) but disambiguate the per-row dict key by + # appending ``__`` to the second and later occurrences. + base_key = name + n = seen_keys.get(base_key, 0) + seen_keys[base_key] = n + 1 + key = base_key if n == 0 else f"{base_key}__{n + 1}" + columns.append(FacadeColumn( + name=name, type=_duckdb_typename_to_datatype(str(typename)), + )) + col_keys.append(key) + rows = [ + {col_keys[i]: value for i, value in enumerate(row)} + for row in data_rows + ] + # Stash the row-key list on the batch so wire emitters can look + # up values by position-aware key even when names duplicate. + batch = RowBatch(columns=columns, rows=rows) + # Pydantic v2 model_extra lets us attach a non-schema attribute; + # consumers that don't care about duplicates still read via name. + object.__setattr__(batch, "_row_keys", col_keys) + return batch + + +# --- caching ---------------------------------------------------------------- + + +_EXECUTOR_CACHE_LIMIT = 4 +_EXECUTOR_CACHE: "collections.OrderedDict[str, CatalogSqlExecutor]" = collections.OrderedDict() + + +def _fingerprint(catalog: FacadeCatalog, datasource: str) -> str: + """Stable cache key for an executor. + + Tables are sorted for cross-build determinism (table order doesn't + affect any catalog row content). Dimensions and metrics WITHIN each + table are NOT sorted because their position drives ``attnum``, + ``ordinal_position``, and ``pg_description.objsubid`` — reordering + columns under the same name set is a real catalog change that the + fingerprint must distinguish. + """ + summary = [ + catalog.catalog_name, + datasource, + sorted([( + sch.name, tbl.name, tbl.table_type, tbl.description, + [ + (d.name, d.data_type.value, d.description, d.label, d.is_time) + for d in tbl.dimensions + ], + [ + (m.name, m.data_type.value if m.data_type else None, + m.description, m.label) + for m in tbl.metrics + ], + ) for sch in catalog.schemas for tbl in sch.tables]), + ] + payload = json.dumps(summary, sort_keys=True, default=str) + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + +def executor_for( + catalog: FacadeCatalog, + datasource: str | None = None, + *, + extra_relations: Iterable[CatalogRelation] | None = None, +) -> CatalogSqlExecutor: + """Return a process-cached ``CatalogSqlExecutor`` for ``catalog``. + + ``datasource`` scopes ``current_database()`` / ``current_catalog`` / + ``current_user`` literal substitution. When omitted, falls back to the + catalog's first schema name (or the catalog name for empty catalogs). + The pg facade always passes the real datasource explicitly because its + catalog schema is the literal ``public``. + + ``extra_relations`` is the per-call extensibility hook. Passing it + BYPASSES the cache (the cache key would otherwise need to digest the + relations' row data, which is the expensive part we're trying to skip). + Embedders that want fast repeat introspection should call this once at + setup and hold the returned executor. + + Cache (default path) is keyed by a stable SHA-256 of (catalog, datasource). + FIFO eviction at 4 entries. Single-threaded asyncio + sync execute, so no + lock is needed. + """ + if datasource is None: + datasource = ( + catalog.schemas[0].name if catalog.schemas else catalog.catalog_name + ) + if extra_relations is not None: + # Hot path is the default-relations cache; embedders with extras + # opt out of caching to avoid digesting row payloads in the key. + return CatalogSqlExecutor( + catalog=catalog, datasource=datasource, + extra_relations=extra_relations, + ) + fp = _fingerprint(catalog, datasource) + cached = _EXECUTOR_CACHE.get(fp) + if cached is not None: + return cached + executor = CatalogSqlExecutor(catalog=catalog, datasource=datasource) + _EXECUTOR_CACHE[fp] = executor + if len(_EXECUTOR_CACHE) > _EXECUTOR_CACHE_LIMIT: + # FIFO eviction. + _EXECUTOR_CACHE.popitem(last=False) + return executor + + +__all__ = [ + "CatalogRelation", + "CatalogSqlExecutor", + "KNOWN_SYSTEM_OIDS", + "build_catalog_relations", + "executor_for", + "is_catalog_only", + "stable_oid", +] + + +# Trigger Pydantic to fully construct the forward-ref classes (so the +# translator's imports don't run into a stale reference when this module +# is the entry point). +_ = sqlglot # quiet linters about the conditional import diff --git a/slayer/facade/datatypes.py b/slayer/facade/datatypes.py index 0aa16243..df1a8d5d 100644 --- a/slayer/facade/datatypes.py +++ b/slayer/facade/datatypes.py @@ -18,6 +18,9 @@ DataType.BOOLEAN: "BOOLEAN", DataType.DATE: "DATE", DataType.TIMESTAMP: "TIMESTAMP", + # Opaque columns travel as text over the wire — the value is whatever the + # driver stringified, and no client-side type claim would be honest. + DataType.UNKNOWN: "VARCHAR", } diff --git a/slayer/facade/info_schema.py b/slayer/facade/info_schema.py index 5d0d3d73..599865e1 100644 --- a/slayer/facade/info_schema.py +++ b/slayer/facade/info_schema.py @@ -23,12 +23,16 @@ from __future__ import annotations -from typing import List, Optional import sqlglot.expressions as exp from slayer.core.enums import DataType -from slayer.facade.catalog import CATALOG_NAME, FacadeCatalog +from slayer.facade.catalog import ( + CATALOG_NAME, + FacadeCatalog, + local_dimensions, + local_metrics, +) from slayer.facade.datatypes import datatype_to_jdbc from slayer.facade.rows import FacadeColumn, RowBatch @@ -44,7 +48,7 @@ _CATALOG_NAME_LOWER = CATALOG_NAME.lower() -def _is_information_schema_from(node: exp.Expression) -> Optional[str]: +def _is_information_schema_from(node: exp.Expression) -> str | None: """If ``node`` is ``SELECT ... FROM information_schema.
``, return the uppercased table name; else ``None``. @@ -89,7 +93,7 @@ def _is_information_schema_from(node: exp.Expression) -> Optional[str]: def match_info_schema( *, parsed: exp.Expression, catalog: FacadeCatalog, -) -> Optional[RowBatch]: +) -> RowBatch | None: """Return the canned ``INFORMATION_SCHEMA.
`` answer or ``None``.""" table_name = _is_information_schema_from(parsed) if table_name is None: @@ -121,7 +125,7 @@ def _serve_metrics(*, catalog: FacadeCatalog) -> RowBatch: FacadeColumn(name="data_type", type=DataType.TEXT), FacadeColumn(name="label", type=DataType.TEXT), ] - rows: List[dict] = [] + rows: list[dict] = [] for sch in catalog.schemas: for tbl in sch.tables: for m in tbl.metrics: @@ -148,7 +152,7 @@ def _serve_dimensions(*, catalog: FacadeCatalog) -> RowBatch: FacadeColumn(name="label", type=DataType.TEXT), FacadeColumn(name="is_time", type=DataType.BOOLEAN), ] - rows: List[dict] = [] + rows: list[dict] = [] for sch in catalog.schemas: for tbl in sch.tables: for d in tbl.dimensions: @@ -184,7 +188,7 @@ def _serve_tables(*, catalog: FacadeCatalog) -> RowBatch: FacadeColumn(name="table_name", type=DataType.TEXT), FacadeColumn(name="table_type", type=DataType.TEXT), ] - rows: List[dict] = [] + rows: list[dict] = [] for sch in catalog.schemas: for tbl in sch.tables: rows.append({ @@ -201,6 +205,13 @@ def _serve_columns(*, catalog: FacadeCatalog) -> RowBatch: into the JDBC ``COLUMNS`` shape. BI tools introspecting a "table" via the wire-facade driver see this as the column list of the underlying semantic model. + + DEV-1567: cross-model entries are excluded — they leak as dotted + "columns" that Metabase / dbt fingerprint scans then project, landing + a dotted name in ``SlayerQuery.measures[*].name`` (Pydantic rejects + the dot). Catalog-namespaced surfaces (``INFORMATION_SCHEMA.METRICS`` + / ``.DIMENSIONS``) and the catalog SQL fingerprint hash continue to + use the raw ``tbl.metrics`` / ``tbl.dimensions``. """ columns = [ FacadeColumn(name="table_catalog", type=DataType.TEXT), @@ -212,11 +223,11 @@ def _serve_columns(*, catalog: FacadeCatalog) -> RowBatch: FacadeColumn(name="is_nullable", type=DataType.TEXT), # Postgres YES/NO FacadeColumn(name="column_kind", type=DataType.TEXT), # METRIC / DIMENSION ] - rows: List[dict] = [] + rows: list[dict] = [] for sch in catalog.schemas: for tbl in sch.tables: position = 1 - for d in tbl.dimensions: + for d in local_dimensions(tbl): rows.append({ "table_catalog": catalog.catalog_name, "table_schema": sch.name, @@ -228,7 +239,7 @@ def _serve_columns(*, catalog: FacadeCatalog) -> RowBatch: "column_kind": "DIMENSION", }) position += 1 - for m in tbl.metrics: + for m in local_metrics(tbl): rows.append({ "table_catalog": catalog.catalog_name, "table_schema": sch.name, diff --git a/slayer/facade/probe_queries.py b/slayer/facade/probe_queries.py index a551a03b..d3c394bc 100644 --- a/slayer/facade/probe_queries.py +++ b/slayer/facade/probe_queries.py @@ -13,7 +13,6 @@ from __future__ import annotations -from typing import Optional import sqlglot.expressions as exp @@ -135,7 +134,7 @@ def _matches_select_current_database(node: exp.Expression) -> bool: return False -def match_probe(parsed: exp.Expression) -> Optional[RowBatch]: +def match_probe(parsed: exp.Expression) -> RowBatch | None: """Return the canned ``RowBatch`` for a matching probe, else ``None``.""" if _matches_select_one(parsed): return _batch_select_one() diff --git a/slayer/facade/rows.py b/slayer/facade/rows.py index 6d297f58..b6bf2494 100644 --- a/slayer/facade/rows.py +++ b/slayer/facade/rows.py @@ -11,7 +11,7 @@ from __future__ import annotations -from typing import Any, Dict, List +from typing import Any from pydantic import BaseModel, ConfigDict @@ -26,5 +26,5 @@ class FacadeColumn(BaseModel): class RowBatch(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) - columns: List[FacadeColumn] - rows: List[Dict[str, Any]] + columns: list[FacadeColumn] + rows: list[dict[str, Any]] diff --git a/slayer/facade/translator.py b/slayer/facade/translator.py index da57e3b1..8c16d70c 100644 --- a/slayer/facade/translator.py +++ b/slayer/facade/translator.py @@ -7,18 +7,21 @@ ``TranslationError`` on user-visible failures (parse error, unknown table, ``SELECT *``, DML/DDL, etc.). -The pipeline (see §6 of DEV-1390): +The pipeline (see §6 of DEV-1390; DEV-1558 swapped catalog-matchers for a +DuckDB-backed executor on the Postgres path): 1. Parse with sqlglot (optionally with a dialect). 2. Probe-query whitelist → canned ``RowBatch``. 3. Classify AST root → reject DML/DDL, no-op SET/SHOW/BEGIN/COMMIT (carrying a ``command_tag`` so the Postgres facade can drive its transaction state machine), continue on SELECT. -4. INFORMATION_SCHEMA dispatch → canned ``RowBatch``. -5. Injected ``catalog_matchers`` (e.g. the Postgres ``pg_catalog`` builder) - → ``PgCatalogResult``. -6. ``SELECT *`` rejection (on real models). -7. SLayer-table translation → ``SlayerQuery`` + column-name mapping. +4. If ``catalog_sql_executor`` is provided (Postgres facade) and + ``is_catalog_only(parsed)`` is True, execute the SQL against the + in-memory DuckDB and return a ``PgCatalogResult``. Otherwise + (Flight facade) dispatch INFORMATION_SCHEMA queries to the canned + ``match_info_schema`` builder. +5. ``SELECT *`` rejection (on real models). +6. SLayer-table translation → ``SlayerQuery`` + column-name mapping. The translator never touches the engine or storage — it produces a ``SlayerQuery`` description and lets the handler decide when to call @@ -32,16 +35,20 @@ from __future__ import annotations import logging -from typing import Callable, Dict, List, Optional, Sequence, Tuple +import re +from contextvars import ContextVar +from collections.abc import Callable, Sequence import sqlglot import sqlglot.errors import sqlglot.expressions as exp from pydantic import BaseModel, ConfigDict -from slayer.core.enums import DataType, TimeGranularity +from slayer.core.enums import DataType, JoinType, TimeGranularity +from slayer.core.models import ModelJoin, SlayerModel from slayer.core.query import ( ColumnRef, + ModelExtension, OrderItem, SlayerQuery, TimeDimension, @@ -52,6 +59,7 @@ FacadeDimension, FacadeMetric, FacadeTable, + build_local_view, ) from slayer.facade.info_schema import match_info_schema from slayer.facade.probe_queries import match_probe @@ -59,14 +67,66 @@ logger = logging.getLogger(__name__) +_IN_FACADE_PARSE: ContextVar[bool] = ContextVar("slayer_facade_parse", default=False) +_COMMAND_FALLBACK_MARKER = "Falling back to parsing as a 'Command'" -# A probe matcher takes the parsed statement and returns a canned RowBatch or -# None. The Flight facade uses the default ``match_probe``; the Postgres facade -# injects its own (datasource-aware version()/current_database()/SHOW/etc.). -ProbeMatcher = Callable[[exp.Expression], Optional[RowBatch]] -# A catalog matcher takes (parsed, catalog) and returns a canned RowBatch or -# None. The Postgres facade injects its ``pg_catalog`` matcher here. -CatalogMatcher = Callable[[exp.Expression, FacadeCatalog], Optional[RowBatch]] + +class _SuppressCommandFallbackWarning(logging.Filter): + """sqlglot warns whenever a statement parses to the generic ``Command`` + node. For facade traffic that fallback is the expected, handled path + (``SHOW TRANSACTION ISOLATION LEVEL`` etc. — one warning per BI + connection), so it is suppressed while ``translate`` is parsing. + Engine/user parse paths keep the warning.""" + + def filter(self, record: logging.LogRecord) -> bool: + return not ( + _IN_FACADE_PARSE.get() and _COMMAND_FALLBACK_MARKER in record.getMessage() + ) + + +logging.getLogger("sqlglot").addFilter(_SuppressCommandFallbackWarning()) + + +# --- session-setting capture (DEV-1569) ------------------------------------- + + +class SetSettingOp(BaseModel): + """A single ``SET = `` capture or ``set_config(name, value, ...)`` + mutation. Names are lowercased on capture. Consumed by the Postgres + facade to mutate its per-connection session-settings map. The Flight + facade ignores these. + """ + + name: str + value: str + + +class ResetSettingOp(BaseModel): + """``RESET `` (``name`` set) or ``RESET ALL`` (``reset_all=True``).""" + + name: str | None = None + reset_all: bool = False + + +# A probe matcher takes the parsed statement and returns either a canned +# ``RowBatch`` (Flight default) or a ``ProbeMatcherOutcome`` (Postgres facade, +# used to tunnel ``set_config`` mutation hints through to the connection). The +# translator unwraps both into a ``ProbeResult``. +class ProbeMatcherOutcome(BaseModel): + """Postgres-facade-only wrapper carrying a probe's row batch alongside + any pending session-setting mutation (e.g. from ``set_config(...)``). + The Flight facade's default matcher returns the bare ``RowBatch``; the + translator accepts either.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + batch: RowBatch + settings_mutation: SetSettingOp | None = None + + +ProbeMatcher = Callable[ + [exp.Expression], RowBatch | ProbeMatcherOutcome | None, +] # --- result types (tagged union via subclassing) ----------------------------- @@ -79,9 +139,15 @@ class TranslatorResult(BaseModel): class ProbeResult(TranslatorResult): - """One of the whitelisted connection probes matched.""" + """One of the whitelisted connection probes matched. + + ``settings_mutation`` carries a ``set_config(name, value, ...)`` + mutation hint when the matcher returned a ``ProbeMatcherOutcome``. + The Postgres facade applies it on Execute (but not Describe). Flight + matchers return ``None`` here.""" batch: RowBatch + settings_mutation: SetSettingOp | None = None class InfoSchemaResult(TranslatorResult): @@ -104,9 +170,16 @@ class NoOpResult(TranslatorResult): ``"START TRANSACTION"``). The Postgres facade uses it to drive its transaction state machine and pick the ``CommandComplete`` tag; the Flight facade ignores it. + + ``set_setting`` carries the captured ``(name, value)`` pair when the + root parsed as ``exp.Set`` with a single ``name = value`` SetItem. + ``reset_setting`` carries the parsed ``RESET `` or ``RESET ALL`` + intent. Both are PG-facade-only; the Flight facade ignores them. """ - command_tag: Optional[str] = None + command_tag: str | None = None + set_setting: SetSettingOp | None = None + reset_setting: ResetSettingOp | None = None class QueryResult(TranslatorResult): @@ -125,10 +198,14 @@ class QueryResult(TranslatorResult): """ query: SlayerQuery - column_name_mapping: List[Tuple[str, str]] + column_name_mapping: list[tuple[str, str]] facade_table: FacadeTable schema_name: str - projection_types: "List[Optional['DataType']]" + projection_types: "list[DataType | None]" + # Datasource the engine should execute against, resolved from the FROM + # (and any joined) table's model. ``None`` for catalogs whose models carry + # no datasource; the facade falls back to its connection datasource. + data_source: str | None = None @property def flight_table(self) -> FacadeTable: @@ -151,10 +228,47 @@ def __init__(self, message: str, *, status: str = "INVALID_ARGUMENT") -> None: READ_ONLY_MESSAGE = "SLayer wire facade is read-only" SELECT_STAR_MESSAGE = ( - "SELECT * not supported; project specific metric or dimension names. " - "Use 'SELECT * FROM INFORMATION_SCHEMA.METRICS WHERE table_name=...' " - "to discover available names." + "SELECT * with aggregates is not supported; project specific metric or " + "dimension names. Use 'SELECT * FROM INFORMATION_SCHEMA.METRICS " + "WHERE table_name=...' to discover available names." ) + + +def _is_browse_mode_select(parsed: exp.Select, proj_exprs: list[exp.Expression]) -> bool: + """``SELECT * FROM t`` browse-mode predicate: no GROUP BY, no HAVING, + no aggregate function anywhere in the projection list (incl. COUNT(*), + SUM/AVG/MIN/MAX, etc.). When this holds, ``*`` expands to every + non-hidden column of ``t``; otherwise it stays rejected so the + user-facing 'project specific names' hint fires for the cases where + it's actually useful.""" + if parsed.args.get("group") is not None: + return False + if parsed.args.get("having") is not None: + return False + return not any( + isinstance(n, exp.AggFunc) + for proj in proj_exprs + for n in proj.walk() + ) + + +def _expand_select_star( + proj_exprs: list[exp.Expression], table: "FacadeTable", +) -> list[exp.Expression]: + """Replace each top-level ``exp.Star`` with a sequence of column + references — one per non-hidden column on ``table`` — preserving the + relative order of any non-Star projections. Used by browse-mode + ``SELECT *`` only (see ``_is_browse_mode_select``).""" + column_names = [d.name for d in table.dimensions] + out: list[exp.Expression] = [] + for expr in proj_exprs: + if isinstance(expr, exp.Star): + out.extend( + exp.Column(this=exp.to_identifier(n)) for n in column_names + ) + else: + out.append(expr) + return out # DEV-1493: aggregating over a saved measure or a non-column expression needs # a multi-stage rewrite, out of scope for the current facade aggregate mapping. AGG_OVER_MEASURE_MESSAGE = ( @@ -167,7 +281,7 @@ def __init__(self, message: str, *, status: str = "INVALID_ARGUMENT") -> None: # --- AST helpers ------------------------------------------------------------- -_TIME_GRAIN_NAMES: Dict[str, TimeGranularity] = { +_TIME_GRAIN_NAMES: dict[str, TimeGranularity] = { "year": TimeGranularity.YEAR, "quarter": TimeGranularity.QUARTER, "month": TimeGranularity.MONTH, @@ -180,7 +294,7 @@ def __init__(self, message: str, *, status: str = "INVALID_ARGUMENT") -> None: # sqlglot represents the unwrapped one-arg time functions as dedicated nodes # (exp.Month, exp.Year, …). date_trunc is exp.DateTrunc with a literal unit. -_TIME_GRAIN_CLASSES: Dict[type, TimeGranularity] = { +_TIME_GRAIN_CLASSES: dict[type, TimeGranularity] = { exp.Year: TimeGranularity.YEAR, exp.Quarter: TimeGranularity.QUARTER, exp.Month: TimeGranularity.MONTH, @@ -192,14 +306,14 @@ def __init__(self, message: str, *, status: str = "INVALID_ARGUMENT") -> None: # sqlglot aggregate-function AST classes → SLayer aggregation names. COUNT is # handled separately (it has *-arg and DISTINCT variants). -_AGG_CLASS_TO_NAME: Dict[type, str] = { +_AGG_CLASS_TO_NAME: dict[type, str] = { exp.Sum: "sum", exp.Avg: "avg", exp.Min: "min", exp.Max: "max", } -_COMPARATOR_SQL: Dict[type, str] = { +_COMPARATOR_SQL: dict[type, str] = { exp.GT: ">", exp.GTE: ">=", exp.LT: "<", @@ -208,15 +322,149 @@ def __init__(self, message: str, *, status: str = "INVALID_ARGUMENT") -> None: exp.NEQ: "<>", } +# DEV-1566: sqlglot DataType.Type → SLayer DataType for CAST(AS ) +# projection support. Aliases sqlglot canonicalises at parse time (STRING→TEXT, +# INTEGER→INT, NUMERIC→DECIMAL, BOOL→BOOLEAN, FLOAT→DOUBLE) need no entry; +# REAL parses to exp.DataType.Type.FLOAT (not normalised), so DOUBLE coverage +# routes through here. Parameterised forms (VARCHAR(255), DECIMAL(10,2)) +# collapse onto the same base member at parse time — precision is dropped at +# the SLayer boundary because SLayer wire types don't carry it. +# +# CAST is a COARSE wire-OID hint, not a precision-preserving conversion. The +# SLayer engine projects the bare column unchanged; the pg-facade encoder is +# OID-driven, so the wire bytes always match the OID we advertise. The only +# "mismatch" exposed by the coarsenings below is that the advertised OID is +# broader than what the user typed: +# +# * DECIMAL/NUMERIC → DataType.DOUBLE (OID 701 `float8`, not 1700 `numeric`). +# * INTEGER / SMALLINT / TINYINT / MEDIUMINT → DataType.INT +# (OID 20 `int8`, not 23/21). +# * TIMESTAMPTZ / TIMESTAMP WITH TIME ZONE / TIMESTAMPLTZ → DataType.TIMESTAMP +# (OID 1114 `timestamp`, no TZ semantics). +# +# Callers needing exact NUMERIC precision, narrow integer wire widths, or +# TZ-aware timestamps must compute upstream. See pg-facade.md +# §"CAST coarse-OID mapping" for the user-facing table. +_SQLGLOT_TYPE_TO_DATATYPE: dict[exp.DataType.Type, DataType] = { + exp.DataType.Type.TEXT: DataType.TEXT, + exp.DataType.Type.VARCHAR: DataType.TEXT, + exp.DataType.Type.CHAR: DataType.TEXT, + exp.DataType.Type.NCHAR: DataType.TEXT, + exp.DataType.Type.NVARCHAR: DataType.TEXT, + exp.DataType.Type.INT: DataType.INT, + exp.DataType.Type.BIGINT: DataType.INT, + exp.DataType.Type.SMALLINT: DataType.INT, + exp.DataType.Type.TINYINT: DataType.INT, + exp.DataType.Type.MEDIUMINT: DataType.INT, + exp.DataType.Type.DOUBLE: DataType.DOUBLE, + exp.DataType.Type.FLOAT: DataType.DOUBLE, + exp.DataType.Type.DECIMAL: DataType.DOUBLE, + exp.DataType.Type.BOOLEAN: DataType.BOOLEAN, + exp.DataType.Type.DATE: DataType.DATE, + exp.DataType.Type.TIMESTAMP: DataType.TIMESTAMP, + exp.DataType.Type.DATETIME: DataType.TIMESTAMP, + exp.DataType.Type.TIMESTAMPTZ: DataType.TIMESTAMP, + exp.DataType.Type.TIMESTAMPLTZ: DataType.TIMESTAMP, +} + +# DEV-1566: per-pair allowlist of CAST coercions admitted in projection. +# Each pair is one the wire encoders in slayer/pg_facade/types.py handle +# losslessly. The unknown-source (None) case admits ONLY TEXT — for every +# other target we'd push the failure into value_to_text / value_to_binary at +# response time, surfacing as an opaque connection error instead of a clean +# translation error. Identity (X→X) is always admitted; computed implicitly. +_CAST_ADMITTED_TARGETS: dict[DataType, frozenset] = { + DataType.DATE: frozenset({DataType.DATE, DataType.TIMESTAMP, DataType.TEXT}), + DataType.TIMESTAMP: frozenset({DataType.TIMESTAMP, DataType.DATE, DataType.TEXT}), + DataType.INT: frozenset({DataType.INT, DataType.DOUBLE, DataType.TEXT}), + # DOUBLE → INT intentionally dropped (Python int(x) truncates toward zero; + # Postgres rounds half-to-even). Round-2 Codex review. + DataType.DOUBLE: frozenset({DataType.DOUBLE, DataType.TEXT}), + DataType.BOOLEAN: frozenset({DataType.BOOLEAN, DataType.TEXT}), + DataType.TEXT: frozenset({DataType.TEXT}), +} -def _column_to_dotted(col: exp.Column) -> str: +# DEV-1566 — Codex round 3: pairs whose ORDER BY / GROUP BY semantics under +# Postgres differ from what the bare-column engine projection would produce. +# A query that references a CAST-projected item (via alias or canonical form) +# in ORDER BY / GROUP BY is rejected when the pair is in the corresponding +# set — workaround: order/group by the bare column, or wait for a follow-up +# ticket that pushes CAST into the engine SQL. +# +# ORDER BY lossy: every X→TEXT pair (lex sort ≠ engine's natural sort). +# Identity X→X, INT→DOUBLE, DATE↔TIMESTAMP all preserve relative order so +# they stay admitted. +_LOSSY_ORDER_BY_CAST_PAIRS: frozenset = frozenset({ + (DataType.INT, DataType.TEXT), + (DataType.DOUBLE, DataType.TEXT), + (DataType.BOOLEAN, DataType.TEXT), + (DataType.DATE, DataType.TEXT), + (DataType.TIMESTAMP, DataType.TEXT), +}) +# GROUP BY lossy: many-to-one casts where the engine-column grouping +# returns MORE groups than the casted column would. +# - TIMESTAMP → DATE: multiple timestamps per date. +# - INT → DOUBLE: int64 has precise range ±2^53 in IEEE 754 float64; +# larger bigints lose precision so distinct ints can collapse to the +# same double under Postgres's GROUP BY semantics, but the facade +# groups by the bare int and over-reports groups. +# Every other admitted pair is a 1:1 / identity mapping within the +# supported value range. +_LOSSY_GROUP_BY_CAST_PAIRS: frozenset = frozenset({ + (DataType.TIMESTAMP, DataType.DATE), + (DataType.INT, DataType.DOUBLE), +}) + +# Single source of truth for the GROUP BY string used in lossy-CAST error +# messages and the LEFT-JOIN-subquery forbidden-clauses list. +_GROUP_BY_KIND = "GROUP BY" + + +def _sqlglot_type_to_datatype(node: exp.DataType) -> DataType | None: + """Map a sqlglot ``DataType`` node to a SLayer ``DataType``. + + Returns ``None`` when the type member isn't in the mapping (UUID, JSON, + ARRAY, STRUCT, …) — caller falls through to the existing + 'Unsupported projection expression' error. + """ + return _SQLGLOT_TYPE_TO_DATATYPE.get(node.this) + + +def _is_admitted_cast(source: DataType | None, target: DataType) -> bool: + """Strict per-pair admission. ``None`` source admits only ``TEXT``.""" + if source is None: + return target is DataType.TEXT + if source == target: + return True + return target in _CAST_ADMITTED_TARGETS.get(source, frozenset()) + + +def _column_to_dotted( + col: exp.Column, + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> str: """Reconstruct the dotted reference from a sqlglot ``Column``. ``customers.regions.name`` (3-part) → ``"customers.regions.name"`` ``customers.row_count`` (2-part) → ``"customers.row_count"`` ``revenue_sum`` (bare) → ``"revenue_sum"`` + + DEV-1558 B5: when ``strip_prefix=(schema, table)`` is given and the + leading qualifiers on the column reference are exactly + ``schema.table.`` (case-insensitive, or with ``schema='public'`` since + the pg facade always exposes models under ``public``), drop them so + three-part refs like ``"public"."orders"."customer_id"`` resolve to the + bare ``customer_id`` dimension. + + DEV-1565: when ``alias_map`` is given and the column's leading table + qualifier matches an alias entry (case-insensitive), the alias is + rewritten to the target SLayer model name BEFORE the dotted form is + built — so ``"Stores"."name"`` with ``alias_map={"Stores": "stores"}`` + yields ``"stores.name"`` (the SLayer cross-model dotted form). """ - parts: List[str] = [] + parts: list[str] = [] for key in ("catalog", "db", "table"): node = col.args.get(key) if node is None: @@ -224,15 +472,88 @@ def _column_to_dotted(col: exp.Column) -> str: parts.append(str(node.this) if hasattr(node, "this") else str(node)) leaf = col.this parts.append(str(leaf.this) if hasattr(leaf, "this") else str(leaf)) - return ".".join(parts) + parts = _apply_alias_remap(parts, alias_map) + return ".".join(_apply_strip_prefix(parts, strip_prefix)) + + +def _apply_alias_remap( + parts: list[str], alias_map: dict[str, str] | None, +) -> list[str]: + """If ``parts`` is at least 2-part and the table-qualifier (second-to- + last element) matches an entry in ``alias_map`` (case-insensitive), + rewrite it to the target model name. Otherwise return ``parts`` + unchanged. + + DEV-1565: maps ``.`` refs (e.g. Metabase's + ``"Stores"."name"``) to ``.`` (``"stores"."name"``). + """ + if not alias_map or len(parts) < 2: + return parts + qual = parts[-2] + target = alias_map.get(qual) + if target is None: + # case-insensitive fallback for quoted-identifier mismatches + lower = qual.lower() + for k, v in alias_map.items(): + if k.lower() == lower: + target = v + break + if target is None: + return parts + new = list(parts) + new[-2] = target + return new + + +def _apply_strip_prefix( + parts: list[str], strip_prefix: tuple[str, str] | None, +) -> list[str]: + """Drop the leading ``schema.table.`` qualifier from ``parts`` when it + matches ``strip_prefix``. Four-part catalog-qualified refs drop the + leading 3; three-part drops the leading 2; two-part drops the + leading 1. Bare and unrelated refs pass through unchanged. + + For 4-part refs, the leading catalog must match the SLayer catalog + name (``slayer``) — otherwise we leave the ref alone (a foreign + catalog reference is not addressable here). + """ + if strip_prefix is None: + return parts + schema_p, table_p = strip_prefix + if len(parts) >= 4: + c = parts[-4].lower() + s = parts[-3].lower() + t = parts[-2].lower() + if (c == CATALOG_NAME.lower() + and t == table_p.lower() + and s in {"public", schema_p.lower()}): + return parts[:-4] + parts[-1:] + if len(parts) >= 3: + s = parts[-3].lower() + t = parts[-2].lower() + if t == table_p.lower() and s in {"public", schema_p.lower()}: + return parts[:-3] + parts[-1:] + if len(parts) == 2 and parts[0].lower() == table_p.lower(): + return parts[1:] + return parts def _detect_time_grain_date_trunc( node: exp.Expression, -) -> Optional[Tuple[TimeGranularity, exp.Column]]: +) -> tuple[TimeGranularity, exp.Column] | None: + """Plain ``DATE_TRUNC(, )`` detector. + + Does NOT unwrap any day offsets on the column side — a bare + ``DATE_TRUNC('week', col + INTERVAL '1 day')`` is a user-written + shifted bucket, not the Metabase Sunday-week wrapper, and must + fall through to the regular "unsupported projection" error. + The full Sunday-week pattern is handled by + ``_detect_sunday_week_wrapper`` which requires BOTH the outer ``-1 + day`` shift and the inner ``+1 day`` shift to be present together. + """ unit = node.args.get("unit") col = node.this - if unit is None or not isinstance(col, exp.Column): + if unit is None: return None # The unit is a string literal under the dialect-less parse and a bare # identifier (``exp.Var``) under the Postgres dialect's TIMESTAMP_TRUNC. @@ -243,12 +564,143 @@ def _detect_time_grain_date_trunc( grain = _TIME_GRAIN_NAMES.get(unit_str) if grain is None: return None + if isinstance(col, exp.Cast): + col = col.this + if not isinstance(col, exp.Column): + return None return grain, col +def _date_trunc_unit(node: exp.Expression) -> str | None: + """Return the lowercase unit string of a ``DATE_TRUNC``/``TIMESTAMP_TRUNC`` + node, or ``None`` if not a trunc call. Used by the Sunday-week wrapper + detector below.""" + if not isinstance(node, (exp.DateTrunc, exp.TimestampTrunc)): + return None + unit = node.args.get("unit") + if unit is None: + return None + if isinstance(unit, (exp.Literal, exp.Var)): + return str(unit.this).lower() + return str(unit).lower() + + +def _detect_sunday_week_wrapper( + node: exp.Expression, +) -> tuple[TimeGranularity, exp.Column] | None: + """Recognise the complete Metabase Sunday-week wrapper as a single shape. + + Full pattern (the one Metabase emits for a week breakout on a DATE + column):: + + (CAST(DATE_TRUNC('week', + INTERVAL '1 day') AS DATE) + + INTERVAL '-1 day') + + Metabase rounds to Sunday by shifting the input forward one day, + applying Postgres' Monday-based ``DATE_TRUNC('week', ...)``, then + shifting the result back one day. SLayer models this as + ``TimeGranularity.WEEK_SUNDAY`` (DEV-1572), whose per-dialect SQL + generation reproduces the same Sunday-anchored bucketing — so the + result matches what Metabase asked for instead of being rejected. + + Both halves must be present together — a bare ``DATE_TRUNC`` with a + ``+1``-day inner offset and no outer wrapper, OR an outer ``-1``-day + wrapper around a grain other than WEEK, are NOT Sunday-week and must + NOT be silently collapsed (they're either user intent we must preserve + or other translator gaps that should keep raising). Each shift leg must + be exactly one day. Returns ``(WEEK_SUNDAY, col)`` on match, ``None`` + otherwise. The outer ``CAST`` is already peeled by ``_detect_time_grain``. + """ + inner = _unwrap_signed_day_offset(node, expected_sign=-1) + if inner is node: + return None # no outer -1-day shift + if isinstance(inner, exp.Cast): + inner = inner.this + if isinstance(inner, exp.Paren): + inner = inner.this + if _TIME_GRAIN_NAMES.get(_date_trunc_unit(inner) or "") != TimeGranularity.WEEK: + return None + col = inner.this + if isinstance(col, exp.Cast): + col = col.this + unwrapped_col = _unwrap_signed_day_offset(col, expected_sign=1) + if unwrapped_col is col: + return None # no inner +1-day shift — partial wrapper, not Sunday-week + if not isinstance(unwrapped_col, exp.Column): + return None + return TimeGranularity.WEEK_SUNDAY, unwrapped_col + + +def _day_interval_sign(node: exp.Expression) -> int | None: + """Return ``+1`` for ``INTERVAL '1 day'``, ``-1`` for ``INTERVAL '-1 day'``, + or ``None`` if ``node`` isn't a one-day interval at all. + + Recognised forms: + * Dialect-less parse: ``INTERVAL '1 day'`` / ``INTERVAL '-1 day'`` — the + literal carries the unit string. + * Postgres dialect: ``INTERVAL '1' DAY`` — literal is the magnitude, + unit is a separate ``DAY`` node. + """ + if not isinstance(node, exp.Interval): + return None + val = node.this + if not isinstance(val, exp.Literal): + return None + s = str(val.this).strip().lower().replace("'", "") + unit = node.args.get("unit") + unit_str = "" + if unit is not None: + unit_str = str(unit.this if hasattr(unit, "this") else unit).lower() + if s in {"1", "-1"}: + if not unit_str.startswith("day"): + return None + return 1 if s == "1" else -1 + if s == "1 day": + return 1 + if s == "-1 day": + return -1 + return None + + +def _unwrap_signed_day_offset( + node: exp.Expression, *, expected_sign: int, +) -> exp.Expression: + """If ``node`` shifts by exactly ``expected_sign`` days (``+1`` or ``-1``) + via a single ADD/SUB of ``INTERVAL '1 day'`` / ``INTERVAL '-1 day'``, + return the inner expression. Otherwise return ``node`` unchanged. + + Direction matters: ``expected_sign=-1`` matches Metabase's outer + Sunday-week wrapper (`` + INTERVAL '-1 day'`` or + `` - INTERVAL '1 day'``) but NOT the inverse, so a legitimate + user-written ``DATE_TRUNC('week', x + INTERVAL '1 day')`` outside the + Sunday-week wrapper stays preserved (would not match + ``expected_sign=-1``). ``expected_sign=+1`` matches the inner + column-side shift (``+ INTERVAL '1 day'`` or + ``- INTERVAL '-1 day'``). + """ + if isinstance(node, exp.Paren): + inner = _unwrap_signed_day_offset(node.this, expected_sign=expected_sign) + if inner is not node.this: + return inner + if isinstance(node, exp.Add): + # Adding +1 day → net +1; adding -1 day → net -1. + right_sign = _day_interval_sign(node.expression) + if right_sign is not None and right_sign == expected_sign: + return node.this + left_sign = _day_interval_sign(node.this) + if left_sign is not None and left_sign == expected_sign: + return node.expression + if isinstance(node, exp.Sub): + # Subtracting +1 day → net -1; subtracting -1 day → net +1. + right_sign = _day_interval_sign(node.expression) + if right_sign is not None and -right_sign == expected_sign: + return node.this + return node + + def _detect_time_grain_single_arg( node: exp.Expression, -) -> Optional[Tuple[TimeGranularity, exp.Column]]: +) -> tuple[TimeGranularity, exp.Column] | None: """Dedicated AST classes like ``exp.Month`` / ``exp.Year``.""" for cls, grain in _TIME_GRAIN_CLASSES.items(): if isinstance(node, cls): @@ -261,7 +713,7 @@ def _detect_time_grain_single_arg( def _detect_time_grain_anonymous( node: exp.Anonymous, -) -> Optional[Tuple[TimeGranularity, exp.Column]]: +) -> tuple[TimeGranularity, exp.Column] | None: """``hour(col)`` / ``minute(col)`` / ``second(col)`` come through here.""" grain = _TIME_GRAIN_NAMES.get(str(node.this).lower()) if grain is None: @@ -272,10 +724,35 @@ def _detect_time_grain_anonymous( return None -def _detect_time_grain(node: exp.Expression) -> Optional[Tuple[TimeGranularity, exp.Column]]: +def _detect_time_grain(node: exp.Expression) -> tuple[TimeGranularity, exp.Column] | None: """If ``node`` is ``()`` or ``date_trunc('', )``, return ``(granularity, column)``. Otherwise ``None``. + + Also unwraps an outer ``CAST(...)`` — Metabase emits + ``CAST(TIMESTAMP_TRUNC(col, MONTH) AS DATE)`` when the column is + DATE-typed (the truncation function widens to TIMESTAMP and Metabase + casts back). The cast is irrelevant to the semantic time-grain + classification. """ + if isinstance(node, exp.Cast): + # Unwrap the cast and recurse — the inner expression is what + # carries the time-grain semantics. + unwrapped = _detect_time_grain(node.this) + if unwrapped is not None: + return unwrapped + # DEV-1562 / DEV-1558 round-20 follow-up: Metabase emits Sunday-based + # week truncation as the full wrapper + # ``CAST((CAST(DATE_TRUNC('week', col + INTERVAL '1 day') AS DATE) + + # INTERVAL '-1 day') AS DATE)``. Detect the complete pattern as a single + # match — partial wrappers (just the outer -1d, or just the inner +1d) + # are NOT Sunday-week and must keep raising as unsupported projections. + sunday_week = _detect_sunday_week_wrapper(node) + if sunday_week is not None: + return sunday_week + if isinstance(node, exp.Paren): + recur = _detect_time_grain(node.this) + if recur is not None: + return recur if isinstance(node, (exp.DateTrunc, exp.TimestampTrunc)): match = _detect_time_grain_date_trunc(node) if match is not None: @@ -288,13 +765,18 @@ def _detect_time_grain(node: exp.Expression) -> Optional[Tuple[TimeGranularity, return None -def _alias_for_time_grain(grain: TimeGranularity, col: exp.Column) -> str: +def _alias_for_time_grain( + grain: TimeGranularity, col: exp.Column, + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> str: """The flat projection name we expose for ``month(ordered_at)`` etc. Format: ``"()"`` lowercased so it round-trips cleanly through GROUP BY / ORDER BY equality checks. """ - return f"{grain.value}({_column_to_dotted(col)})" + return f"{grain.value}({_column_to_dotted(col, strip_prefix=strip_prefix, alias_map=alias_map)})" # --- aggregate-call detection (DEV-1486 decision 21) ------------------------- @@ -306,19 +788,35 @@ class _AggCall(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) agg: str # SLayer aggregation name: sum/avg/min/max/count/count_distinct - inner_ref: Optional[str] = None # dotted column ref, or None for COUNT(*) + inner_ref: str | None = None # dotted column ref, or None for COUNT(*) inner_is_column: bool = False # False → COUNT(*) or non-column arg is_count_star: bool = False # True only for the literal COUNT(*) -def _detect_aggregate(node: exp.Expression) -> Optional[_AggCall]: # NOSONAR(S3776) — flat per-aggregate-kind dispatch; splitting hides the shape +def _detect_aggregate( # NOSONAR(S3776) — flat per-aggregate-kind dispatch; splitting hides the shape + node: exp.Expression, + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> _AggCall | None: """If ``node`` is a SQL aggregate call, classify it; else ``None``. ``COUNT(*)`` → ``count`` / ``inner_ref=None``. ``COUNT(DISTINCT col)`` → ``count_distinct``. ``COUNT(col)`` → ``count``. ``SUM/AVG/MIN/MAX(col)`` → the matching agg. An aggregate over a non-column argument sets ``inner_is_column=False`` so the caller can raise the DEV-1493 error. + + DEV-1558 B5: ``strip_prefix`` drops the FROM-table's ``schema.table.`` + qualifier from the inner column ref so ``SUM("public"."orders"."revenue")`` + resolves to the metric ``revenue:sum`` instead of + ``public.orders.revenue:sum``. + + DEV-1565: ``alias_map`` rewrites join-alias qualifiers (e.g. + ``AVG("Stores"."tax_rate")``) to the SLayer cross-model dotted form + (``stores.tax_rate:avg``). """ + def _dot(col: exp.Column) -> str: + return _column_to_dotted(col, strip_prefix=strip_prefix, alias_map=alias_map) if isinstance(node, exp.Count): inner = node.this if isinstance(inner, exp.Star): @@ -328,13 +826,13 @@ def _detect_aggregate(node: exp.Expression) -> Optional[_AggCall]: # NOSONAR(S3 if len(exprs) == 1 and isinstance(exprs[0], exp.Column): return _AggCall( agg="count_distinct", - inner_ref=_column_to_dotted(exprs[0]), + inner_ref=_dot(exprs[0]), inner_is_column=True, ) return _AggCall(agg="count_distinct", inner_is_column=False) if isinstance(inner, exp.Column): return _AggCall( - agg="count", inner_ref=_column_to_dotted(inner), inner_is_column=True, + agg="count", inner_ref=_dot(inner), inner_is_column=True, ) # COUNT() — not the row-count star. return _AggCall(agg="count", inner_is_column=False) @@ -343,7 +841,7 @@ def _detect_aggregate(node: exp.Expression) -> Optional[_AggCall]: # NOSONAR(S3 inner = node.this if isinstance(inner, exp.Column): return _AggCall( - agg=name, inner_ref=_column_to_dotted(inner), inner_is_column=True, + agg=name, inner_ref=_dot(inner), inner_is_column=True, ) return _AggCall(agg=name, inner_ref=None, inner_is_column=False) return None @@ -361,8 +859,35 @@ def _saved_measure_names(table: FacadeTable) -> set[str]: return {m.name for m in table.metrics if m.measure_formula == m.name} +def _assert_local_metric(metric: FacadeMetric) -> None: + """DEV-1567: reject cross-model metric projection at the resolver. + + The catalog pre-expands cross-model metrics into entries with dotted + names (``customers.row_count``, ``customers.regions.population_sum``). + The flat-column probes (``pg_attribute`` / + ``INFORMATION_SCHEMA.COLUMNS``) hide them so BI tools don't discover + them as projectable, but a hand-written SQL ref still resolves + through ``metrics_by_name`` / ``metrics_by_formula``. Letting it + flow into ``SlayerQuery.measures[*].name`` would either trip the + Pydantic validator (29 errors per query) or — with a non-dotted user + alias — produce a SlayerQuery whose engine alias mis-keys against + the engine result (DEV-1448 preserves the cross-model hop path on + the engine side, so ``orders.cr`` against ``orders.customers.cr``). + + Guard predicate matches the catalog-flatten filter + (``slayer/facade/catalog.py:local_metrics``); see DEV-1493 for the + multi-stage rewrite path. + """ + if "." in metric.name: + raise TranslationError( + f"Cross-model metric {metric.name!r} cannot be projected in " + f"a flat SELECT. Use a saved metric or rewrite as a multi-" + f"stage query (DEV-1493)." + ) + + def _metric_for_aggregate( - call: _AggCall, table: FacadeTable, metrics_by_formula: Dict[str, FacadeMetric], + call: _AggCall, table: FacadeTable, metrics_by_formula: dict[str, FacadeMetric], ) -> FacadeMetric: """Resolve an aggregate call to its catalog ``FacadeMetric``. @@ -379,6 +904,7 @@ def _metric_for_aggregate( formula = _agg_formula(call) metric = metrics_by_formula.get(formula) if metric is not None: + _assert_local_metric(metric) return metric # Not eligible / unknown. Distinguish the saved-measure case for a # clearer message pointing at the follow-up ticket. @@ -393,16 +919,16 @@ def _metric_for_aggregate( # --- table resolution -------------------------------------------------------- -def _flatten_catalog(catalog: FacadeCatalog) -> Dict[str, List[Tuple[str, FacadeTable]]]: +def _flatten_catalog(catalog: FacadeCatalog) -> dict[str, list[tuple[str, FacadeTable]]]: """Build a (model_name → [(schema, table), …]) index for bare-name lookup.""" - by_name: Dict[str, List[Tuple[str, FacadeTable]]] = {} + by_name: dict[str, list[tuple[str, FacadeTable]]] = {} for sch in catalog.schemas: for tbl in sch.tables: by_name.setdefault(tbl.name, []).append((sch.name, tbl)) return by_name -def _unwrap_identifier(node: Optional[exp.Expression]) -> Optional[str]: +def _unwrap_identifier(node: exp.Expression | None) -> str | None: """Pull the string value out of a sqlglot identifier-ish node.""" if node is None: return None @@ -411,7 +937,10 @@ def _unwrap_identifier(node: Optional[exp.Expression]) -> Optional[str]: def _resolve_qualified_table( *, schema_str: str, table_name: str, catalog: FacadeCatalog, -) -> Tuple[str, FacadeTable]: +) -> tuple[str, FacadeTable]: + # Try the exact schema match first — if a catalog actually carries a + # ``public`` schema (or whatever name the user passed), honour the + # explicit qualifier. for sch in catalog.schemas: if sch.name != schema_str: continue @@ -421,12 +950,28 @@ def _resolve_qualified_table( raise TranslationError( f"Unknown table {table_name!r} in schema {schema_str!r}" ) + # Pg-facade alias fall-back. The Postgres facade always advertises + # ``public`` as its single schema (cf. ``pg_namespace`` row); when no + # real ``public`` schema exists in the catalog (e.g. when the catalog + # is keyed by the actual datasource name), accept ``public`` as a + # synonym so Metabase's ``"public"."orders"`` keeps working. + # + # Only fall back when the catalog presents a single schema — the + # "no user-customized postgres_schema anywhere" case where mapping + # ``public`` is unambiguous. With multiple schemas present (custom + # postgres_schema in use), ``public.
`` would silently cross + # schema isolation; reject with "Unknown schema" instead. Real BI + # clients address tables via the actual schema name (read from + # pg_namespace); this only ever bites hand-written SQL that + # hard-codes ``public``. + if schema_str.lower() == "public" and len(catalog.schemas) == 1: + return _resolve_bare_table(table_name=table_name, catalog=catalog) raise TranslationError(f"Unknown schema: {schema_str!r}") def _resolve_bare_table( *, table_name: str, catalog: FacadeCatalog, -) -> Tuple[str, FacadeTable]: +) -> tuple[str, FacadeTable]: matches = _flatten_catalog(catalog).get(table_name, []) if not matches: raise TranslationError(f"Unknown table: {table_name!r}") @@ -439,9 +984,37 @@ def _resolve_bare_table( return matches[0] +def _resolve_query_datasource( + *, table: FacadeTable, join_plan: "_JoinPlan | None", +) -> str | None: + """Datasource the engine should execute the query against, or raise. + + SLayer cannot execute a query spanning datasources, so when the FROM table + and an explicitly joined table resolve to different datasources we reject + with a clear message instead of letting the engine fail opaquely. Catalog + BFS joins are scoped per datasource, so only explicit cross-schema JOINs + can trigger this. + """ + sources: set[str] = set() + if table.model_ref is not None and table.model_ref.data_source: + sources.add(table.model_ref.data_source) + if ( + join_plan is not None + and join_plan.target_table.model_ref is not None + and join_plan.target_table.model_ref.data_source + ): + sources.add(join_plan.target_table.model_ref.data_source) + if len(sources) > 1: + raise TranslationError( + "cross-datasource queries are not supported (referenced " + f"datasources: {sorted(sources)})" + ) + return next(iter(sources), None) + + def _resolve_table( from_clause: exp.From, catalog: FacadeCatalog, -) -> Tuple[str, FacadeTable]: +) -> tuple[str, FacadeTable]: """Resolve a SELECT's FROM into ``(schema_name, FacadeTable)``. Handles the three qualification forms (§6.1): @@ -484,21 +1057,27 @@ class _ProjectionItem(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) projected_name: str # what the BI tool sees (alias or natural name) - metric: Optional[FacadeMetric] = None - dimension: Optional[FacadeDimension] = None - time_grain: Optional[TimeGranularity] = None - time_grain_underlying: Optional[FacadeDimension] = None + metric: FacadeMetric | None = None + dimension: FacadeDimension | None = None + time_grain: TimeGranularity | None = None + time_grain_underlying: FacadeDimension | None = None + # DEV-1566: target type when the projection was CAST( AS ). + # Overrides projection_types[i] at _record_metric / _record_dimension time, + # leaving engine_alias and the SlayerQuery measure/dimension untouched. + cast_target: DataType | None = None def _resolve_time_grain_projection( *, grain: TimeGranularity, col: exp.Column, - alias_name: Optional[str], + alias_name: str | None, table: FacadeTable, - dims_by_name: Dict[str, FacadeDimension], + dims_by_name: dict[str, FacadeDimension], + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, ) -> _ProjectionItem: - dotted = _column_to_dotted(col) + dotted = _column_to_dotted(col, strip_prefix=strip_prefix, alias_map=alias_map) dim = dims_by_name.get(dotted) if dim is None: raise TranslationError( @@ -511,7 +1090,9 @@ def _resolve_time_grain_projection( f"in {grain.value}()" ) return _ProjectionItem( - projected_name=alias_name or _alias_for_time_grain(grain, col), + projected_name=alias_name or _alias_for_time_grain( + grain, col, strip_prefix=strip_prefix, alias_map=alias_map, + ), dimension=dim, time_grain=grain, time_grain_underlying=dim, @@ -521,9 +1102,9 @@ def _resolve_time_grain_projection( def _resolve_aggregate_projection( *, call: _AggCall, - alias_name: Optional[str], + alias_name: str | None, table: FacadeTable, - metrics_by_formula: Dict[str, FacadeMetric], + metrics_by_formula: dict[str, FacadeMetric], ) -> _ProjectionItem: """Map a SQL aggregate call to the same projection item a bare metric name would produce (DEV-1486 decision 21).""" @@ -539,16 +1120,20 @@ def _resolve_aggregate_projection( def _resolve_column_projection( *, body: exp.Column, - alias_name: Optional[str], + alias_name: str | None, table: FacadeTable, - metrics_by_name: Dict[str, FacadeMetric], - dims_by_name: Dict[str, FacadeDimension], + metrics_by_name: dict[str, FacadeMetric], + dims_by_name: dict[str, FacadeDimension], + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, ) -> _ProjectionItem: - dotted = _column_to_dotted(body) + dotted = _column_to_dotted(body, strip_prefix=strip_prefix, alias_map=alias_map) if dotted in metrics_by_name: + metric = metrics_by_name[dotted] + _assert_local_metric(metric) return _ProjectionItem( projected_name=alias_name or dotted, - metric=metrics_by_name[dotted], + metric=metric, ) if dotted in dims_by_name: return _ProjectionItem( @@ -560,20 +1145,209 @@ def _resolve_column_projection( ) -def _resolve_projection( - expressions: Sequence[exp.Expression], table: FacadeTable, -) -> List[_ProjectionItem]: - """Walk the projection list, classifying each item against the table.""" +def _detect_column_cast( + body: exp.Expression, +) -> tuple[exp.Column, DataType] | None: + """If ``body`` is ``CAST( AS )``, + return ``(inner_col, target_data_type)``. Otherwise ``None``. + + The detector REQUIRES ``body.this`` to be an ``exp.Column`` — aggregates, + hygiene wrappers, function calls, and arithmetic stay outside scope. + ``exp.TryCast`` is rejected (Postgres has no native TRY_CAST equivalent). + """ + if not isinstance(body, exp.Cast) or isinstance(body, exp.TryCast): + return None + inner = body.this + if not isinstance(inner, exp.Column): + return None + target = _sqlglot_type_to_datatype(body.to) + if target is None: + return None + return inner, target + + +def _resolve_column_cast_projection( + *, + body: exp.Column, + cast_target: DataType, + alias_name: str | None, + table: FacadeTable, + metrics_by_name: dict[str, FacadeMetric], + dims_by_name: dict[str, FacadeDimension], + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> _ProjectionItem: + """Resolve ``CAST( AS )`` to a projection item that runs + the bare column through the engine and overrides the wire OID via + ``cast_target``. Enforces the strict per-pair coercion allowlist. + + DEV-1565: ``alias_map`` carries the LEFT-JOIN subquery's + ``"Stores"."name"`` → catalog ``stores.name`` rewrite so Metabase-style + joined CAST refs resolve through the same dimension-lookup path as + non-casted joined refs. + """ + inner = _resolve_column_projection( + body=body, alias_name=alias_name, table=table, + metrics_by_name=metrics_by_name, dims_by_name=dims_by_name, + strip_prefix=strip_prefix, alias_map=alias_map, + ) + source = _item_cast_source_type(inner) + if not _is_admitted_cast(source=source, target=cast_target): + offending = exp.Cast(this=body, to=exp.DataType.build(cast_target.value)).sql() + raise TranslationError( + f"Unsupported CAST: cannot project {source!s} column as " + f"{cast_target!s} ({offending!r}). Admitted coercions: see " + f"docs/interfaces/pg-facade.md." + ) + return inner.model_copy(update={"cast_target": cast_target}) + + +# DEV-1558 B5: hygiene-scalar wrappers Metabase uses for fingerprint queries. +# Each maps a sqlglot AST class to a human-readable name for the WARNING log. +_HYGIENE_FUNC_CLASSES: dict[type, str] = { + exp.Substring: "SUBSTRING", + exp.Upper: "UPPER", + exp.Lower: "LOWER", + exp.Trim: "TRIM", + exp.Length: "LENGTH", + # exp.Left / exp.Right exist for some dialects; check at runtime. +} +for _cls_name in ("Left", "Right"): + _cls = getattr(exp, _cls_name, None) + if _cls is not None: + _HYGIENE_FUNC_CLASSES[_cls] = _cls_name.upper() + +# Anonymous-form hygiene calls (sqlglot doesn't always lift them to a Func +# subclass) — names match `_function_name_lower` output. +_HYGIENE_ANONYMOUS_NAMES = {"substr", "substring", "left", "right", + "upper", "lower", "trim", "length"} + + +def _detect_hygiene_wrapper(body: exp.Expression) -> tuple[str, exp.Column] | None: + """If ``body`` is a hygiene-scalar wrapper around exactly one column + reference, return ``(printable_func_name, inner_col)``. Otherwise None. + """ + for cls, name in _HYGIENE_FUNC_CLASSES.items(): + if isinstance(body, cls): + inner = body.this + if isinstance(inner, exp.Column): + return name, inner + return None + if isinstance(body, exp.Anonymous): + fname = str(body.this).lower() + if fname in _HYGIENE_ANONYMOUS_NAMES: + args = body.args.get("expressions") or [] + if args and isinstance(args[0], exp.Column): + return fname.upper(), args[0] + return None + + +def _is_fingerprint_shape_wrap( + inner_col: exp.Column, *, strip_prefix: tuple[str, str] | None, +) -> bool: + """True iff ``inner_col`` is shaped like Metabase's fingerprint + projection — a 3-part qualified column reference whose + ``.
.`` prefix matches the FROM-table prefix. + + The full pattern (e.g. ``SUBSTRING("public"."customers"."name", 1, + 1234)``) is exclusive to Metabase's field-value rescan: hand-written + SQL like ``LENGTH(name)`` or ``UPPER(orders.status)`` does NOT match + and stays an error so the user notices the unsupported projection + instead of silently getting bare column values back. + """ + if strip_prefix is None: + return False + # Count qualifier parts on the column ref: catalog + db + table + leaf. + qualifier_parts = sum( + 1 for k in ("catalog", "db", "table") if inner_col.args.get(k) is not None + ) + if qualifier_parts < 2: + return False # not a 3-part (or deeper) qualified ref + # Confirm the leading schema.table prefix matches the FROM table by + # checking that strip_prefix would actually drop something. + raw = _raw_column_parts(inner_col) + stripped = _apply_strip_prefix(list(raw), strip_prefix) + return len(stripped) < len(raw) + + +def _raw_column_parts(col: exp.Column) -> list[str]: + """The qualifier parts plus leaf identifier of ``col``, in order.""" + parts: list[str] = [] + for key in ("catalog", "db", "table"): + node = col.args.get(key) + if node is None: + continue + parts.append(str(node.this) if hasattr(node, "this") else str(node)) + leaf = col.this + parts.append(str(leaf.this) if hasattr(leaf, "this") else str(leaf)) + return parts + + +def _build_projection_lookups( + table: FacadeTable, + *, + extra_dims_by_name: dict[str, FacadeDimension] | None = None, + extra_metrics_by_name: dict[str, FacadeMetric] | None = None, + extra_metrics_by_formula: dict[str, FacadeMetric] | None = None, +) -> tuple[dict[str, FacadeMetric], dict[str, FacadeMetric], dict[str, FacadeDimension]]: + """Assemble the (metrics_by_name, metrics_by_formula, dims_by_name) + lookup dicts from the catalog's table metadata, overlaid with the + DEV-1565 dynamic-join extras when supplied.""" metrics_by_name = {m.name: m for m in table.metrics} metrics_by_formula = {m.measure_formula: m for m in table.metrics} dims_by_name = {d.name: d for d in table.dimensions} + if extra_dims_by_name: + dims_by_name.update(extra_dims_by_name) + if extra_metrics_by_name: + metrics_by_name.update(extra_metrics_by_name) + if extra_metrics_by_formula: + metrics_by_formula.update(extra_metrics_by_formula) + return metrics_by_name, metrics_by_formula, dims_by_name + - out: List[_ProjectionItem] = [] +def _resolve_projection( # NOSONAR(S3776) — flat dispatch over projection-expression shapes; one branch per shape by design + expressions: Sequence[exp.Expression], table: FacadeTable, + *, + schema_name: str | None = None, + alias_map: dict[str, str] | None = None, + extra_dims_by_name: dict[str, FacadeDimension] | None = None, + extra_metrics_by_name: dict[str, FacadeMetric] | None = None, + extra_metrics_by_formula: dict[str, FacadeMetric] | None = None, + allow_column_cast: bool = True, +) -> list[_ProjectionItem]: + """Walk the projection list, classifying each item against the table. + + DEV-1565: when a LEFT JOIN to a dynamic-fallback target is in play, + ``extra_dims_by_name`` / ``extra_metrics_by_*`` carry the materialised + ``.`` lookups so the joined dotted refs resolve without + needing a configured catalog join. + + DEV-1566: ``allow_column_cast`` gates the ``CAST( AS )`` + projection branch. The pg-facade passes ``True`` (default) — its wire + encoders coerce values to the declared OID. The Flight facade passes + ``False`` because its ``pa.Table.from_pylist`` schema-binding rejects + values whose Python type doesn't match the declared Arrow type and the + facade has no value-coercion pass. + """ + metrics_by_name, metrics_by_formula, dims_by_name = _build_projection_lookups( + table, + extra_dims_by_name=extra_dims_by_name, + extra_metrics_by_name=extra_metrics_by_name, + extra_metrics_by_formula=extra_metrics_by_formula, + ) + # DEV-1558 B5: when the FROM was schema-qualified (e.g. "public"."orders"), + # let column refs that lead with the same `schema.table.` prefix resolve + # to the bare column. + strip_prefix: tuple[str, str] | None = ( + (schema_name, table.name) if schema_name else None + ) + + out: list[_ProjectionItem] = [] for expr in expressions: if isinstance(expr, exp.Star): raise TranslationError(SELECT_STAR_MESSAGE) - alias_name: Optional[str] = None + alias_name: str | None = None body: exp.Expression = expr if isinstance(expr, exp.Alias): alias_name = str(expr.alias) @@ -585,10 +1359,11 @@ def _resolve_projection( out.append(_resolve_time_grain_projection( grain=grain, col=col, alias_name=alias_name, table=table, dims_by_name=dims_by_name, + strip_prefix=strip_prefix, alias_map=alias_map, )) continue - agg_call = _detect_aggregate(body) + agg_call = _detect_aggregate(body, strip_prefix=strip_prefix, alias_map=alias_map) if agg_call is not None: out.append(_resolve_aggregate_projection( call=agg_call, alias_name=alias_name, table=table, @@ -596,10 +1371,51 @@ def _resolve_projection( )) continue + # DEV-1566: CAST( AS ) projection. Runs AFTER the + # time-grain unwrap (preserves Metabase's CAST(DATE_TRUNC(...) AS DATE)) + # and AFTER the aggregate detector (CAST(SUM(...) AS T) is out of + # scope — the detector requires the inner to be a bare Column). + # Gated by allow_column_cast so the Flight facade falls through to + # the "Unsupported projection expression" error instead of producing + # a wire schema its row materialiser can't fulfil. + if allow_column_cast: + cast_match = _detect_column_cast(body) + if cast_match is not None: + inner_col, cast_target = cast_match + out.append(_resolve_column_cast_projection( + body=inner_col, cast_target=cast_target, + alias_name=alias_name, table=table, + metrics_by_name=metrics_by_name, dims_by_name=dims_by_name, + strip_prefix=strip_prefix, alias_map=alias_map, + )) + continue + if isinstance(body, exp.Column): out.append(_resolve_column_projection( body=body, alias_name=alias_name, table=table, metrics_by_name=metrics_by_name, dims_by_name=dims_by_name, + strip_prefix=strip_prefix, alias_map=alias_map, + )) + continue + + # DEV-1558 B5: hygiene-scalar projection wrappers. Metabase's + # field-value rescan emits SUBSTRING("public"."customers"."name", + # 1, 1234) AS "..." — we drop the wrapper and project the bare + # column under the user's alias. + hygiene = _detect_hygiene_wrapper(body) + if hygiene is not None and _is_fingerprint_shape_wrap( + hygiene[1], strip_prefix=strip_prefix, + ): + func_name, inner_col = hygiene + logger.debug( + "hygiene-scalar wrapper %r dropped for fingerprint projection " + "(column %r)", + func_name, _column_to_dotted(inner_col, strip_prefix=strip_prefix), + ) + out.append(_resolve_column_projection( + body=inner_col, alias_name=alias_name, table=table, + metrics_by_name=metrics_by_name, dims_by_name=dims_by_name, + strip_prefix=strip_prefix, alias_map=alias_map, )) continue @@ -612,9 +1428,9 @@ def _resolve_projection( # --- WHERE translation ------------------------------------------------------- -def _split_and_chain(node: exp.Expression) -> List[exp.Expression]: +def _split_and_chain(node: exp.Expression) -> list[exp.Expression]: """Flatten a top-level AND chain into its conjuncts.""" - out: List[exp.Expression] = [] + out: list[exp.Expression] = [] stack = [node] while stack: cur = stack.pop() @@ -628,11 +1444,14 @@ def _split_and_chain(node: exp.Expression) -> List[exp.Expression]: def _lift_time_between( conj: exp.Between, time_dim_names: set[str], -) -> Optional[Tuple[str, Optional[str], Optional[str]]]: + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> tuple[str, str | None, str | None] | None: col = conj.this if not isinstance(col, exp.Column): return None - dotted = _column_to_dotted(col) + dotted = _column_to_dotted(col, strip_prefix=strip_prefix, alias_map=alias_map) if dotted not in time_dim_names: return None lo = _literal_str(conj.args.get("low")) @@ -644,11 +1463,14 @@ def _lift_time_between( def _lift_time_comparator( conj: exp.Expression, time_dim_names: set[str], -) -> Optional[Tuple[str, Optional[str], Optional[str]]]: + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> tuple[str, str | None, str | None] | None: col = conj.this if not isinstance(col, exp.Column): return None - dotted = _column_to_dotted(col) + dotted = _column_to_dotted(col, strip_prefix=strip_prefix, alias_map=alias_map) if dotted not in time_dim_names: return None val = _literal_str(conj.expression) @@ -659,27 +1481,149 @@ def _lift_time_comparator( return dotted, None, val +def _aggregate_alias_for_column( + side: exp.Expression, + items_by_projected_name: dict[str, "_ProjectionItem"], + *, strip_prefix: tuple[str, str] | None = None, +) -> "_ProjectionItem | None": + """If ``side`` is a column reference whose dotted (strip_prefix-aware) + name matches an aggregate projection's ``projected_name``, return that + projection item; else ``None``. + + Used by ``_try_aggregate_alias_filter`` to route Metabase-style + ``WHERE "" > `` and ``HAVING "" > `` + conjuncts to colon-form filters (DEV-1568). The ``item.metric`` guard + keeps dimension projections that happen to share a name with what + *would* be an aggregate alias from being treated as aggregates. + """ + if not isinstance(side, exp.Column): + return None + name = _column_to_dotted(side, strip_prefix=strip_prefix) + item = items_by_projected_name.get(name) + if item is None or item.metric is None: + return None + return item + + +def _try_aggregate_alias_filter( + conj: exp.Expression, + items_by_projected_name: dict[str, "_ProjectionItem"], + *, strip_prefix: tuple[str, str] | None = None, +) -> str | None: + """If ``conj`` is `` `` (in either + order), return the colon-form filter string. If one side matches an + aggregate alias but the other side is not a literal, raise. Return + ``None`` if neither side matches an aggregate-alias column ref. + + DEV-1568: Metabase compiles MBQL ``["aggregation", N]`` post-aggregation + references to alias-bearing SQL — ``WHERE "count" > 1`` for filters and + ``ORDER BY "count" DESC`` for sorts. This detector rewrites the filter + shapes to colon-form (``*:count > 1``) so the engine classifies them + correctly as HAVING; ``_translate_order_by`` handles the ORDER BY side. + """ + op_sql = _COMPARATOR_SQL.get(type(conj)) + if op_sql is None: + return None + lhs, rhs = conj.this, conj.expression + alias_lhs = _aggregate_alias_for_column(lhs, items_by_projected_name, strip_prefix=strip_prefix) + alias_rhs = _aggregate_alias_for_column(rhs, items_by_projected_name, strip_prefix=strip_prefix) + if alias_lhs is None and alias_rhs is None: + return None + if alias_lhs is not None and alias_rhs is None and isinstance(rhs, exp.Literal): + assert alias_lhs.metric is not None + return f"{alias_lhs.metric.measure_formula} {op_sql} {rhs.sql()}" + if alias_rhs is not None and alias_lhs is None and isinstance(lhs, exp.Literal): + assert alias_rhs.metric is not None + return f"{alias_rhs.metric.measure_formula} {_flip_comparator(op_sql)} {lhs.sql()}" + raise TranslationError( + f"Aggregate-alias filter expects a literal on the other side; " + f"got: {conj.sql()!r}" + ) + + def _classify_where_conjunct( conj: exp.Expression, time_dim_names: set[str], -) -> Tuple[Optional[Tuple[str, Optional[str], Optional[str]]], Optional[str]]: + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> tuple[tuple[str, str | None, str | None] | None, str | None]: """Classify a single conjunct. Returns ``((time_dim, date_range_lo, date_range_hi), None)`` if this is a time-dim filter that should lift to ``time_dimensions[*].date_range``. Returns ``(None, verbatim_sql)`` for the everything-else case. + + DEV-1558 B5: the engine's Mode-B DSL parses ``SlayerQuery.filters`` + and only accepts single-dot dotted paths. Before serialising the + verbatim fallback, normalise any ``exp.Column`` nodes in the + predicate via ``strip_prefix`` so a WHERE clause like + ``"public"."orders"."total" > 0`` lands as ``"total" > 0`` in + SlayerQuery.filters, not as the original 3-part-qualified form that + the DSL parser would reject. """ if isinstance(conj, exp.Between): - lifted = _lift_time_between(conj, time_dim_names) + lifted = _lift_time_between( + conj, time_dim_names, strip_prefix=strip_prefix, alias_map=alias_map, + ) if lifted is not None: return lifted, None if isinstance(conj, (exp.GTE, exp.GT, exp.LTE, exp.LT)): - lifted = _lift_time_comparator(conj, time_dim_names) + lifted = _lift_time_comparator( + conj, time_dim_names, strip_prefix=strip_prefix, alias_map=alias_map, + ) if lifted is not None: return lifted, None - return None, _rewrite_neq(conj.sql()) + normalised = _normalise_predicate_columns( + conj, strip_prefix=strip_prefix, alias_map=alias_map, + ) + return None, _rewrite_neq(normalised.sql()) + +def _normalise_predicate_columns( + node: exp.Expression, + *, + strip_prefix: tuple[str, str] | None, + alias_map: dict[str, str] | None = None, +) -> exp.Expression: + """Walk ``node`` and rewrite every ``exp.Column``: apply ``alias_map`` + first (so ``.`` becomes ``.``), then + ``strip_prefix`` (so the parent table's ``schema.table.`` qualifier + drops), then ALWAYS emit the identifiers UNQUOTED. + + The un-quoting is load-bearing, not cosmetic. ``SlayerQuery.filters`` + is parsed by the Mode B (Python-AST) DSL, where a double-quoted + ``"merchantId"`` is a STRING LITERAL identical to ``'merchantId'`` — + NOT a column reference (Python doesn't distinguish quote styles). + Emitting the quoted form silently rewrites ``WHERE "merchantId" = 'x'`` + into the constant-vs-constant comparison ``'merchantId' = 'x'``, which + matches no rows. Bare identifiers resolve as columns. We un-quote every + column regardless of whether alias_map / strip_prefix changed anything, + so this fires for the common ``FROM "Table" WHERE "col" = 'x'`` shape + that has neither a schema prefix nor a join alias. + """ -def _literal_str(node: Optional[exp.Expression]) -> Optional[str]: + def rewrite(child: exp.Expression) -> exp.Expression: + if not isinstance(child, exp.Column): + return child + original = _raw_column_parts(child) + parts = _apply_alias_remap(list(original), alias_map) + parts = _apply_strip_prefix(parts, strip_prefix) + # Rebuild the Column with UNQUOTED identifiers. Even when the parts + # are unchanged, the original may carry ``quoted=True`` that must be + # dropped for the DSL to see a column rather than a string literal. + new = exp.Column(this=exp.Identifier(this=parts[-1], quoted=False)) + if len(parts) >= 2: + new.set("table", exp.Identifier(this=parts[-2], quoted=False)) + if len(parts) >= 3: + new.set("db", exp.Identifier(this=parts[-3], quoted=False)) + if len(parts) >= 4: + new.set("catalog", exp.Identifier(this=parts[-4], quoted=False)) + return new + + return node.transform(rewrite) + + +def _literal_str(node: exp.Expression | None) -> str | None: if node is None: return None if isinstance(node, exp.Literal): @@ -692,17 +1636,67 @@ def _rewrite_neq(sql: str) -> str: return sql.replace("!=", "<>") +# SQL reserved words Metabase v0.62 uses as unquoted column aliases in its +# captured corpus (e.g. ``AS select``, ``AS update``, ``AS delete`` in the +# table-privileges CTE). sqlglot rejects these without quotes; we +# preprocess by quoting them on a parse-fail retry. +_KEYWORD_ALIAS_QUOTE = re.compile( + r"\b(AS)\s+(select|update|insert|delete|create|drop|alter|grant|revoke)\b", + re.IGNORECASE, +) + + +def _quote_keyword_aliases(sql: str) -> str: + """Quote unquoted SQL-keyword aliases. ``AS select`` → ``AS "select"``.""" + return _KEYWORD_ALIAS_QUOTE.sub( + lambda m: f'{m.group(1)} "{m.group(2)}"', sql, + ) + + +def _parse_with_keyword_alias_fallback( + sql: str, *, dialect: str | None, +) -> exp.Expression: + """Parse ``sql`` with sqlglot. On failure, retry once with unquoted + SQL keyword aliases auto-quoted — Metabase's table-privileges CTE + (corpus #8) uses ``AS select`` / ``AS update`` / ``AS delete`` which + sqlglot rejects, but Postgres accepts. Raises ``TranslationError`` + with the original parse error if both attempts fail.""" + try: + return sqlglot.parse_one(sql, dialect=dialect) + except sqlglot.errors.ParseError as primary: + retry_sql = _quote_keyword_aliases(sql) + if retry_sql == sql: + raise TranslationError(f"SQL parse error: {primary}") from primary + try: + return sqlglot.parse_one(retry_sql, dialect=dialect) + except sqlglot.errors.ParseError: + raise TranslationError(f"SQL parse error: {primary}") from primary + + def _apply_where( - where: Optional[exp.Where], - time_dims_built: Dict[str, TimeDimension], - filters_out: List[str], + where: exp.Where | None, + time_dims_built: dict[str, TimeDimension], + items_by_projected_name: dict[str, "_ProjectionItem"], + filters_out: list[str], + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, ) -> None: - """Walk the WHERE chain; lift time-dim filters, append verbatim rest.""" + """Walk the WHERE chain; route aggregate-alias refs to colon-form + filters (DEV-1568), lift time-dim filters, append verbatim rest.""" if where is None: return time_dim_names = set(time_dims_built.keys()) for conj in _split_and_chain(where.this): - lifted, verbatim = _classify_where_conjunct(conj, time_dim_names) + colon_form = _try_aggregate_alias_filter( + conj, items_by_projected_name, strip_prefix=strip_prefix, + ) + if colon_form is not None: + filters_out.append(colon_form) + continue + lifted, verbatim = _classify_where_conjunct( + conj, time_dim_names, strip_prefix=strip_prefix, alias_map=alias_map, + ) if lifted is not None: name, lo, hi = lifted td = time_dims_built[name] @@ -717,17 +1711,37 @@ def _apply_where( def _apply_having( - having: Optional[exp.Having], + having: exp.Having | None, table: FacadeTable, - filters_out: List[str], + items_by_projected_name: dict[str, "_ProjectionItem"], + filters_out: list[str], + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, + extra_metrics_by_formula: dict[str, FacadeMetric] | None = None, ) -> None: - """Map ``HAVING `` conjuncts to colon-form - filters (DEV-1486 decision 21). The engine classifies colon-form - aggregate filters as HAVING.""" + """Map ``HAVING `` (DEV-1486 decision 21) and + ``HAVING `` (DEV-1568) conjuncts to + colon-form filters. The engine classifies colon-form aggregate filters + as HAVING. + + DEV-1565: ``extra_metrics_by_formula`` overlay covers dynamic-join + targets whose metrics aren't yet in the catalog's projection. + """ if having is None: return metrics_by_formula = {m.measure_formula: m for m in table.metrics} + if extra_metrics_by_formula: + metrics_by_formula.update(extra_metrics_by_formula) for conj in _split_and_chain(having.this): + # DEV-1568: alias-ref form (``HAVING "rev" > 1000``). Symmetric with + # the WHERE pre-pass; tries first so it wins before the agg-call form. + colon_form = _try_aggregate_alias_filter( + conj, items_by_projected_name, strip_prefix=strip_prefix, + ) + if colon_form is not None: + filters_out.append(colon_form) + continue op_sql = _COMPARATOR_SQL.get(type(conj)) if op_sql is None: raise TranslationError( @@ -735,8 +1749,8 @@ def _apply_having( f" ." ) lhs, rhs = conj.this, conj.expression - agg_lhs = _detect_aggregate(lhs) - agg_rhs = _detect_aggregate(rhs) + agg_lhs = _detect_aggregate(lhs, strip_prefix=strip_prefix, alias_map=alias_map) + agg_rhs = _detect_aggregate(rhs, strip_prefix=strip_prefix, alias_map=alias_map) if agg_lhs is not None and agg_rhs is None and isinstance(rhs, exp.Literal): _metric_for_aggregate(call=agg_lhs, table=table, metrics_by_formula=metrics_by_formula) filters_out.append(f"{_agg_formula(agg_lhs)} {op_sql} {rhs.sql()}") @@ -759,26 +1773,147 @@ def _flip_comparator(op_sql: str) -> str: # --- ORDER BY / GROUP BY ----------------------------------------------------- +def _item_cast_source_type(item: _ProjectionItem) -> DataType | None: + """Return the underlying source DataType of a CAST-projected item, or + None when the source type is unknown (custom metric without a declared + ``data_type``). Caller MUST gate on ``item.cast_target is not None`` + before calling this — the helper's None return is intentionally + overloaded with the unknown-source case so the lossy-pair check can + treat unknown-source casts as lossy by default.""" + if item.metric is not None: + return item.metric.data_type + if item.dimension is not None: + return item.dimension.data_type + return None + + +def _cast_pair_is_lossy( + *, source: DataType | None, target: DataType, + lossy_pairs: frozenset, +) -> bool: + """A cast is lossy when either: (a) the source is known and (source, + target) is in ``lossy_pairs``, or (b) the source is unknown — the only + admitted unknown-source target is TEXT (see ``_is_admitted_cast``) + which is always lossy for ORDER BY (lex vs engine's natural sort). + For GROUP BY the unknown-source path is admitted because the engine's + bare-value grouping preserves whatever per-value distinction TEXT + would expose.""" + if source is None: + return target is DataType.TEXT + return (source, target) in lossy_pairs + + +def _lossy_cast_error_message( + *, kind: str, name: str, source: DataType | None, target: DataType, +) -> str: + """One-line user-facing message for the lossy-CAST-pair rejection. The + ``kind`` is ``ORDER BY`` or ``GROUP BY``; ``name`` is the user-visible + identifier the message points at (typically the alias). ``source`` is + None when the underlying metric/dimension has no declared data_type + (custom metric path), which is treated as lossy by default.""" + source_str = str(source) if source is not None else "unknown" + return ( + f"{kind} on CAST projection {name!r} with lossy pair " + f"{source_str}→{target!s} is unsupported: the engine query projects " + f"the bare column, so the underlying sort/group semantics don't " + f"match the casted type. Use the bare column, or wait for engine-" + f"side CAST pushdown." + ) + + +def _reject_lossy_cast_or_pass( + *, kind: str, name: str, item: _ProjectionItem, lossy_pairs: frozenset, +) -> None: + """If ``item`` is a CAST projection AND its (source, target) pair is + lossy under bare-column ORDER BY / GROUP BY semantics, raise. Otherwise + no-op. Unknown-source casts (custom metrics without a declared + data_type) are admitted as None→TEXT only; ``_cast_pair_is_lossy`` + treats that case as lossy for ORDER BY by default.""" + if item.cast_target is None: + return + source = _item_cast_source_type(item) + if not _cast_pair_is_lossy( + source=source, target=item.cast_target, lossy_pairs=lossy_pairs, + ): + return + raise TranslationError(_lossy_cast_error_message( + kind=kind, name=name, source=source, target=item.cast_target, + )) + + +def _resolve_order_by_item( + body: exp.Expression, + item_by_projected_name: dict[str, _ProjectionItem], + metric_item_by_formula: dict[str, _ProjectionItem], + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> tuple[_ProjectionItem, str]: + """Look up the projection item an ORDER BY term resolves to. + + Tries the bare-name lookup first (alias or canonical metric name from + ``_order_by_name``); on miss, DEV-1568 fallback maps a literal aggregate + call (``ORDER BY SUM(revenue)``) to the projection registered under a + different alias (``SELECT SUM(revenue) AS "rev"``) by matching on the + aggregate's canonical ``measure_formula``. Returns ``(item, name)`` so + the DEV-1566 lossy-CAST check downstream has both the resolved item and + the user-visible name for error messaging. Raises ``TranslationError`` + if neither path resolves. + """ + name = _order_by_name(body, strip_prefix=strip_prefix, alias_map=alias_map) + item = item_by_projected_name.get(name) + if item is None: + agg = _detect_aggregate(body, strip_prefix=strip_prefix, alias_map=alias_map) + if agg is not None: + item = metric_item_by_formula.get(_agg_formula(agg)) + if item is None: + raise TranslationError( + f"ORDER BY column {name!r} is not in the projection list" + ) + return item, name + + def _translate_order_by( - order: Optional[exp.Order], - item_by_projected_name: Dict[str, _ProjectionItem], -) -> List[OrderItem]: + order: exp.Order | None, + item_by_projected_name: dict[str, _ProjectionItem], + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> list[OrderItem]: if order is None: return [] - out: List[OrderItem] = [] + # DEV-1568: when a SELECT aliases an aggregate (``SUM(revenue) AS "rev"``) + # and the ORDER BY repeats the call literally (``ORDER BY SUM(revenue)``), + # ``_order_by_name`` returns the canonical metric name (``"revenue_sum"``) + # — but ``item_by_projected_name`` is keyed on the user alias (``"rev"``). + # Build a fallback map keyed by ``metric.measure_formula`` so a repeated + # aggregate call resolves to the same projection the alias does. + metric_item_by_formula: dict[str, _ProjectionItem] = { + item.metric.measure_formula: item + for item in item_by_projected_name.values() + if item.metric is not None + } + out: list[OrderItem] = [] for ord_expr in order.args.get("expressions") or []: if not isinstance(ord_expr, exp.Ordered): continue - body = ord_expr.this direction = "desc" if ord_expr.args.get("desc") else "asc" - name = _order_by_name(body) - if name not in item_by_projected_name: - raise TranslationError( - f"ORDER BY column {name!r} is not in the projection list" - ) - item = item_by_projected_name[name] + item, name = _resolve_order_by_item( + ord_expr.this, item_by_projected_name, metric_item_by_formula, + strip_prefix=strip_prefix, alias_map=alias_map, + ) + _reject_lossy_cast_or_pass( + kind="ORDER BY", name=name, item=item, + lossy_pairs=_LOSSY_ORDER_BY_CAST_PAIRS, + ) if item.metric is not None: - ref = ColumnRef(name=item.metric.name) + # DEV-1568: use the projection's SELECT alias, not the catalog + # ``FacadeMetric.name``. ``_record_metric`` registered the SLayer + # measure under ``projected_name``, so the OrderItem must point + # at the same name. Pre-fix the engine saw ``ColumnRef("row_count")`` + # against a measure named ``"count"`` and DuckDB rejected with + # ``Referenced column "orders.row_count" not found``. + ref = ColumnRef(name=item.projected_name) else: assert item.dimension is not None ref = ColumnRef.from_string(item.dimension.dimension_ref) @@ -786,7 +1921,12 @@ def _translate_order_by( return out -def _order_by_name(body: exp.Expression) -> str: +def _order_by_name( + body: exp.Expression, + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, +) -> str: """Resolve an ORDER BY term to its projected name. A bare column / alias resolves by name. An aggregate term @@ -795,34 +1935,51 @@ def _order_by_name(body: exp.Expression) -> str: item registered for ``SUM(amount)``. """ if isinstance(body, exp.Column): - return _column_to_dotted(body) - agg = _detect_aggregate(body) + return _column_to_dotted(body, strip_prefix=strip_prefix, alias_map=alias_map) + agg = _detect_aggregate(body, strip_prefix=strip_prefix, alias_map=alias_map) if agg is not None and agg.inner_is_column: return f"{agg.inner_ref}_{agg.agg}" if agg is not None and agg.is_count_star: return "row_count" + grain_match = _detect_time_grain(body) + if grain_match is not None: + grain, col = grain_match + return _alias_for_time_grain(grain, col, strip_prefix=strip_prefix, alias_map=alias_map) return body.sql() def _validate_group_by( # NOSONAR(S3776) — single GROUP BY validation pass; extraction adds indirection - group: Optional[exp.Group], - derived: List[str], + group: exp.Group | None, + derived: list[str], + item_by_projected_name: dict[str, _ProjectionItem], + *, + strip_prefix: tuple[str, str] | None = None, + alias_map: dict[str, str] | None = None, ) -> None: - """Apply the strict-on-extras / lenient-on-omissions policy (§6.1).""" + """Apply the strict-on-extras / lenient-on-omissions policy (§6.1). + + ``item_by_projected_name`` is consulted only to look up the projection + item behind a user-facing name so the DEV-1566 lossy-CAST-pair rejection + can fire. Membership validation still runs against the ``derived`` list + (which carries time-grain canonical forms in addition to the projected + names) to preserve the existing strict-on-extras behaviour. + """ if group is None: return derived_set = set(derived) - user_items: List[str] = [] + user_items: list[str] = [] for g in group.args.get("expressions") or []: if isinstance(g, exp.Column): - user_items.append(_column_to_dotted(g)) + user_items.append(_column_to_dotted(g, strip_prefix=strip_prefix, alias_map=alias_map)) else: grain_match = _detect_time_grain(g) if grain_match is not None: grain, col = grain_match - user_items.append(_alias_for_time_grain(grain, col)) - else: - user_items.append(g.sql()) + user_items.append(_alias_for_time_grain( + grain, col, strip_prefix=strip_prefix, alias_map=alias_map, + )) + continue + user_items.append(g.sql()) for u in user_items: if u not in derived_set: # GROUP BY positional refs (GROUP BY 1) and aggregate terms are @@ -834,6 +1991,18 @@ def _validate_group_by( # NOSONAR(S3776) — single GROUP BY validation pass; e f"GROUP BY item {u!r} is not in the projection's derived " f"dimension set ({sorted(derived_set)})" ) + # DEV-1566 — Codex round 3: reject GROUP BY on CAST projections + # whose (source, target) pair is lossy under bare-column grouping. + # Unknown-source casts (only admitted as None→TEXT) are admitted + # here — the engine's per-value grouping already collapses to the + # same set of TEXT representations. + item = item_by_projected_name.get(u) + if item is None: + continue + _reject_lossy_cast_or_pass( + kind=_GROUP_BY_KIND, name=u, item=item, + lossy_pairs=_LOSSY_GROUP_BY_CAST_PAIRS, + ) def _is_ignorable_group_item(item: str) -> bool: @@ -842,6 +2011,32 @@ def _is_ignorable_group_item(item: str) -> bool: return item.isdigit() +def _reject_lossy_cast_in_implicit_grouping( + group: exp.Group | None, items: Sequence[_ProjectionItem], +) -> None: + """DEV-1566 — Codex round 12: when the user omits an explicit GROUP BY + but projects at least one dimension, SLayer auto-groups (dim-only-dedup + when there are no measures, mandatory dim-group when there are + measures). The auto-grouping is by the bare engine column, so a + CAST-projected dimension with a lossy (source, target) pair has the + same correctness gap as an explicit ``GROUP BY `` would. Run + the same lossy check the explicit-GROUP-BY path runs, on each + dim-shaped CAST item. + + No-op when ``group is not None``: the explicit path's + ``_validate_group_by`` already covers that case. + """ + if group is not None: + return + for item in items: + if item.dimension is None: + continue + _reject_lossy_cast_or_pass( + kind=_GROUP_BY_KIND, name=item.projected_name, item=item, + lossy_pairs=_LOSSY_GROUP_BY_CAST_PAIRS, + ) + + # --- main entry point -------------------------------------------------------- @@ -860,9 +2055,169 @@ def _is_start_transaction(node: exp.Expression) -> bool: return body_name == "START" and alias_name == "TRANSACTION" -def _classify_noop_root(parsed: exp.Expression) -> Optional[NoOpResult]: +def _extract_set_setting(parsed: exp.Set) -> SetSettingOp | None: + """Extract a ``SetSettingOp`` from a clean ``SET = `` / + ``SET TO `` AST. Returns ``None`` for multi-item SETs or + SetItem shapes whose body isn't a single ``EQ(Column(Identifier), rhs)`` + pair — those are silently no-op'd by the caller (DEV-1569). + + DEV-1569 / Codex round 2 F2: dotted custom names (``SET myapp.user_id + = '42'``) parse as ``Column(table='myapp', name='user_id')``; + reconstruct the dotted form so ``SHOW myapp.user_id`` round-trips. + """ + items = parsed.args.get("expressions") or [] + if len(items) != 1: + return None + item = items[0] + body = item.this if hasattr(item, "this") else None + if not isinstance(body, exp.EQ): + return None + lhs = body.this + rhs = body.expression + if not isinstance(lhs, exp.Column): + return None + # LHS is the setting name — Postgres GUC names are case-insensitive; we + # normalise to lowercase. For multi-part `a.b.c.name` shapes (custom + # GUCs like `myapp.user_id` or `my.app.user_id`) reconstruct the full + # dotted form via lhs.parts (Codex round 4 F2). Bare names have a + # single part and produce the same string as `lhs.name`. + name = ".".join(part.name for part in lhs.parts).lower() + value = _extract_setting_value(rhs) + if value is None: + return None + return SetSettingOp(name=name, value=value) + + +def _extract_setting_value(rhs: exp.Expression | None) -> str | None: + """Return the string value of a ``SET`` rhs node, robust across the + parser's literal vs. identifier vs. variable encodings: + + - ``Literal(this='UTF8', is_string=True)`` → ``'UTF8'`` + - ``Var(this='UTF8')`` → ``'UTF8'`` (unquoted form + pgjdbc emits for `SET client_encoding TO UTF8`) + - ``Var(this='DEFAULT')`` → ``'DEFAULT'`` (treated as + literal string; users wanting reset semantics use ``RESET``) + - ``Column`` / ``Identifier`` → name + - ``Cast(this=)`` → recurse one level (Codex + round 4 F3: after extended-protocol bind substitution the rhs may + arrive as ``'foo'::text``) + + Returns ``None`` for shapes we don't recognise so the caller can decide + to silently no-op the SET rather than store a malformed value. + """ + if rhs is None: + return None + if isinstance(rhs, exp.Cast): + return _extract_setting_value(rhs.this) + # Signed numeric literals — `SET extra_float_digits = -1` parses as + # `Neg(Literal(1))` (Codex round 5 F2). Recurse and prefix the sign. + if isinstance(rhs, exp.Neg): + inner = _extract_setting_value(rhs.this) + return f"-{inner}" if inner is not None else None + if isinstance(rhs, exp.Literal): + return str(rhs.this) + if isinstance(rhs, exp.Var): + return str(rhs.this) + if isinstance(rhs, exp.Identifier): + return str(rhs.this) + if isinstance(rhs, exp.Column): + return rhs.name + return None + + +def _extract_reset_setting(parsed: exp.Command) -> ResetSettingOp | None: + """Extract ``RESET `` or ``RESET ALL`` intent from an ``exp.Command`` + root. Returns ``None`` for the bare ``RESET`` spelling — defensive + fallback (drivers don't emit this in practice). + + Multi-word names (``RESET TIME ZONE`` / ``RESET SESSION AUTHORIZATION``) + are returned with internal whitespace preserved and lowercased; the + Postgres facade's ``_apply_reset_setting`` alias-resolves them via the + same ``_SHOW_ALIASES`` table that ``SHOW`` consults. + """ + expr = parsed.expression + if expr is None: + return None + # Mirror the pattern used by ``_show_setting_name`` for robustness across + # sqlglot's various expression shapes. + raw = expr.this if hasattr(expr, "this") else expr + name = str(raw).strip().strip("'\"") + if name.upper() == "ALL": + return ResetSettingOp(reset_all=True) + # Collapse multi-word names into a single-space lowercase form so the + # facade can apply the alias table (`"TIME ZONE"` → `"time zone"` → + # `"timezone"`). + name = re.sub(r"\s+", " ", name).lower() + return ResetSettingOp(name=name) + + +# Single-identifier-or-dotted-chain validation regex (single quantifier on +# identifier chars; the optional dotted-tail uses one bounded `*` segment — +# safe from ReDoS / S5852). Used below to validate the name half of a +# Command-form SET after the value has been peeled off by plain string ops. +# Accepts both `application_name` and `myapp.user_id` shapes to stay +# consistent with the exp.Set path's lhs.parts reconstruction (Codex +# round 5 F3). +_COMMAND_SET_NAME_RE = re.compile(r"[A-Za-z_]\w*(?:\.[A-Za-z_]\w*)*") + + +def _command_expression_text(expr) -> str: + """Extract raw text from a Command's ``expression`` slot (which sqlglot + sometimes carries as a bare ``str`` and sometimes as a node).""" + if isinstance(expr, str): + return expr + if hasattr(expr, "this"): + return str(expr.this) + return str(expr) + + +def _extract_command_form_set(parsed: exp.Command) -> SetSettingOp | None: + """For an ``exp.Command(this='SET', expression='')`` root, try to + parse the raw expression text into a ``(name, value)`` pair. + + Returns ``None`` for shapes the spec cuts: multi-word names + (``SET TIME ZONE 'UTC'``), ``SET SESSION CHARACTERISTICS …``, no + value, etc. Implementation uses plain string operations rather than + a multi-segment regex to keep clear of catastrophic backtracking + (Sonar S5852 / DEV-1569 round 2). + """ + expr = parsed.expression + if expr is None: + return None + raw = _command_expression_text(expr).strip().rstrip(";").strip() + if not raw: + return None + # Find the first separator on the original raw so internal whitespace + # inside captured values is preserved (Codex round 4 F1). `=` wins over + # ` TO ` so `SET x = TO_DATE(...)` never accidentally splits on TO. + # The TO match uses `\sTO\s` with case-insensitive matching: each `\s` + # is a single-char (bounded) match — no ReDoS risk under S5852, and it + # accepts tab / newline separators the round-2 regex did. + eq_idx = raw.find("=") + if eq_idx != -1: + name = raw[:eq_idx].strip() + value = raw[eq_idx + 1:].strip() + else: + to_match = re.search(r"\sTO\s", raw, flags=re.IGNORECASE) + if to_match is None: + return None + name = raw[:to_match.start()].strip() + value = raw[to_match.end():].strip() + if not name or not value: + return None + if not _COMMAND_SET_NAME_RE.fullmatch(name): + return None + return SetSettingOp(name=name.lower(), value=value) + + +def _classify_noop_root(parsed: exp.Expression) -> NoOpResult | None: """Classify SET/SHOW/BEGIN/COMMIT/ROLLBACK roots into a NoOpResult with a - facade-neutral ``command_tag``; ``None`` if not a no-op root.""" + facade-neutral ``command_tag``; ``None`` if not a no-op root. + + DEV-1569: also extracts the parsed-out ``SET = `` and + ``RESET `` / ``RESET ALL`` shapes onto ``set_setting`` / + ``reset_setting`` for the Postgres facade to apply per-connection. + """ if isinstance(parsed, exp.Transaction): return NoOpResult(command_tag="BEGIN") if isinstance(parsed, exp.Commit): @@ -870,45 +2225,143 @@ def _classify_noop_root(parsed: exp.Expression) -> Optional[NoOpResult]: if isinstance(parsed, exp.Rollback): return NoOpResult(command_tag="ROLLBACK") if isinstance(parsed, exp.Set): - return NoOpResult(command_tag="SET") + return NoOpResult( + command_tag="SET", set_setting=_extract_set_setting(parsed), + ) if _is_start_transaction(parsed): return NoOpResult(command_tag="START TRANSACTION") if isinstance(parsed, exp.Command): verb = str(parsed.this).upper() if parsed.this else "" - if verb in {"SHOW", "USE", "RESET"}: + # "SET" covers spellings sqlglot cannot parse into exp.Set, e.g. + # pgjdbc's setTransactionIsolation() emitting `SET SESSION + # CHARACTERISTICS AS TRANSACTION ISOLATION LEVEL ...`. We + # acknowledge but do not capture (set_setting stays None) since + # the rhs isn't a clean (name, value) pair. + if verb == "RESET": + return NoOpResult( + command_tag="RESET", + reset_setting=_extract_reset_setting(parsed), + ) + if verb == "SET": + # DEV-1569 / Codex F1: also try to extract a (name, value) pair + # from the Command-form fallback so spellings like + # `SET search_path = public, extensions` round-trip through + # SHOW. Multi-word names (`SET TIME ZONE 'UTC'`) and + # session-characteristics spellings deliberately don't match + # the regex and remain silent no-ops. + return NoOpResult( + command_tag="SET", + set_setting=_extract_command_form_set(parsed), + ) + if verb in {"SHOW", "USE"}: return NoOpResult(command_tag=verb) return None +def _unwrap_probe( + probe: "RowBatch | ProbeMatcherOutcome", +) -> ProbeResult: + """Normalise a probe matcher's return value into a ``ProbeResult``. + + ``ProbeMatcher`` may return either a bare ``RowBatch`` (Flight default) + or a ``ProbeMatcherOutcome`` (Postgres facade, carrying an optional + session-setting mutation hint from ``set_config``). Both shapes + collapse into ``ProbeResult`` here so the dispatcher's branch count + stays low (DEV-1569 / Sonar S3776). + """ + if isinstance(probe, ProbeMatcherOutcome): + return ProbeResult( + batch=probe.batch, settings_mutation=probe.settings_mutation, + ) + return ProbeResult(batch=probe) + + +# Transaction-open statements. sqlglot's postgres dialect parses bare ``BEGIN`` +# / ``START TRANSACTION`` but REJECTS the characteristic forms Metabase emits +# (``BEGIN READ ONLY``, ``START TRANSACTION ISOLATION LEVEL …``). The facade is +# read-only, so every variant is a no-op that just opens a transaction; we +# recognise them BEFORE the parse to avoid the parse error entirely. +# +# The characteristic run uses a POSSESSIVE quantifier (``[^;]*+``) so the +# engine can't backtrack into it when the trailing ``;?\s*$`` fails to match. +# The prior ``(?:\s+[^;]*?)?\s*;?\s*$`` form had ``\s`` matched by both the +# lazy inner class and the trailing ``\s*``, so a crafted long whitespace run +# before a mid-string ``;`` drove O(n²) backtracking on the asyncio loop — +# stalling every connection (SonarCloud ReDoS finding, PR #221). +_TX_START_RE = re.compile( + r"^\s*START\s+TRANSACTION\b[^;]*+;?\s*$", + re.IGNORECASE | re.DOTALL, +) +_TX_BEGIN_RE = re.compile( + r"^\s*BEGIN\b(?:\s+(?:WORK|TRANSACTION))?[^;]*+;?\s*$", + re.IGNORECASE | re.DOTALL, +) + + +def _classify_transaction_open(sql: str) -> NoOpResult | None: + """Recognise ``BEGIN`` / ``START TRANSACTION`` (with optional + characteristics like ``READ ONLY`` / ``ISOLATION LEVEL …``) pre-parse. + + Returns a ``NoOpResult`` opening a transaction, or ``None`` when the + statement isn't a transaction-open. ``COMMIT`` / ``ROLLBACK`` / ``END`` + parse fine on their own and are handled by ``_classify_noop_root``. + """ + if _TX_START_RE.match(sql): + return NoOpResult(command_tag="START TRANSACTION") + if _TX_BEGIN_RE.match(sql): + return NoOpResult(command_tag="BEGIN") + return None + + def translate( sql: str, catalog: FacadeCatalog, *, - dialect: Optional[str] = None, - probe_matcher: Optional[ProbeMatcher] = None, - catalog_matchers: Sequence[CatalogMatcher] = (), + dialect: str | None = None, + probe_matcher: ProbeMatcher | None = None, + catalog_sql_executor: ( + "CatalogSqlExecutorProtocol | Callable[[], CatalogSqlExecutorProtocol] | None" + ) = None, + allow_column_cast: bool = True, + expand_star_in_browse_mode: bool = False, ) -> TranslatorResult: """Translate a SQL string into a TranslatorResult. ``dialect`` is passed to the sqlglot parser only. ``probe_matcher`` overrides the default Flight probe whitelist (the Postgres facade - injects a datasource-aware one). ``catalog_matchers`` are extra canned- - table matchers tried after INFORMATION_SCHEMA (the Postgres facade - injects its ``pg_catalog`` matcher here). + injects a datasource-aware one). ``catalog_sql_executor`` routes + catalog SQL through an in-memory DuckDB (the Postgres facade does + this; Flight passes ``None`` and falls back to ``match_info_schema``). + + ``expand_star_in_browse_mode``: when True, a ``SELECT *`` with no + GROUP BY / HAVING / aggregate expands to every non-hidden column + of the table (pg-facade convenience for interactive psql sessions). + When False (default — Flight's contract), ``SELECT *`` always + raises, since Flight's clients (dbt-SL JDBC, Tableau, etc.) project + explicit metric/dim names by construction and a bare ``*`` from + them almost always indicates a query-builder bug worth surfacing. Raises ``TranslationError`` on user-visible failures. """ + # Step 0 — transaction-open shim (before parse): sqlglot rejects the + # characteristic forms (``BEGIN READ ONLY`` etc.) that BI tools wrap reads + # in, so recognise them here rather than letting the parse fail. + tx_open = _classify_transaction_open(sql) + if tx_open is not None: + return tx_open + + token = _IN_FACADE_PARSE.set(True) try: - parsed = sqlglot.parse_one(sql, dialect=dialect) - except sqlglot.errors.ParseError as exc: - raise TranslationError(f"SQL parse error: {exc}") from exc + parsed = _parse_with_keyword_alias_fallback(sql, dialect=dialect) + finally: + _IN_FACADE_PARSE.reset(token) # Step 2 — probe-query whitelist (runs first so facade-specific probes, # e.g. SHOW for Postgres, win before generic root classification). pm = probe_matcher or match_probe probe = pm(parsed) if probe is not None: - return ProbeResult(batch=probe) + return _unwrap_probe(probe) # Step 3 — AST root classification. if isinstance(parsed, (exp.Insert, exp.Update, exp.Delete, exp.Merge, @@ -919,24 +2372,59 @@ def translate( noop = _classify_noop_root(parsed) if noop is not None: return noop + + # Step 4 — DuckDB catalog executor (Postgres facade) OR info-schema + # dispatch (Flight facade). The executor handles BOTH pg_catalog AND + # information_schema queries via materialised tables; when it's not + # provided we keep the canned info-schema answer for Flight. + # + # This check runs BEFORE the "must be exp.Select" gate so catalog + # queries that aren't a plain Select — UNION/UNION ALL (Metabase + # corpus #12), set-ops, WITH-only constructs — route to the + # executor when every Table node resolves to a catalog relation. + # Non-catalog Selects continue to the SLayer-table translation + # below; non-catalog UNIONs etc. surface the unsupported-statement + # error from the gate. + if catalog_sql_executor is not None: + from slayer.facade.catalog_sql import is_catalog_only + if is_catalog_only(parsed): + # ``catalog_sql_executor`` accepts either the executor itself + # or a zero-arg factory — lazy construction lets the pg + # facade skip the DuckDB materialisation cost on + # non-catalog (model) queries (Codex round 16). Resolve the + # factory only inside this branch. + executor = ( + catalog_sql_executor() + if callable(catalog_sql_executor) + else catalog_sql_executor + ) + return PgCatalogResult(batch=executor.execute(parsed=parsed, sql=sql)) + elif isinstance(parsed, exp.Select): + info = match_info_schema(parsed=parsed, catalog=catalog) + if info is not None: + return InfoSchemaResult(batch=info) + if not isinstance(parsed, exp.Select): raise TranslationError( f"Unsupported statement: {type(parsed).__name__}" ) - # Step 4 — INFORMATION_SCHEMA dispatch. - info = match_info_schema(parsed=parsed, catalog=catalog) - if info is not None: - return InfoSchemaResult(batch=info) + # Step 5 / 6 — SLayer-table translation. + return _translate_slayer_select( + parsed, catalog, allow_column_cast=allow_column_cast, + expand_star_in_browse_mode=expand_star_in_browse_mode, + ) + - # Step 5 — injected catalog matchers (e.g. pg_catalog). - for matcher in catalog_matchers: - matched = matcher(parsed, catalog) - if matched is not None: - return PgCatalogResult(batch=matched) +# Lightweight Protocol so the translator doesn't pull catalog_sql at import +# time (which would create a duckdb-at-import dependency for Flight). +class CatalogSqlExecutorProtocol: + """Protocol the catalog SQL executor must satisfy. Defined here so + ``translate`` can type-annotate its parameter without importing + catalog_sql (which imports duckdb).""" - # Step 6 / 7 — SLayer-table translation. - return _translate_slayer_select(parsed, catalog) + def execute(self, *, parsed: exp.Expression, sql: str) -> RowBatch: + raise NotImplementedError class _ProjectionPlan(BaseModel): @@ -944,13 +2432,13 @@ class _ProjectionPlan(BaseModel): model_config = ConfigDict(arbitrary_types_allowed=True) - measures: List[dict] - dimension_refs: List[ColumnRef] - time_dims: List[TimeDimension] - time_dim_by_name: Dict[str, TimeDimension] - derived_dims: List[str] - column_name_mapping: List[Tuple[str, str]] - projection_types: List[Optional[DataType]] + measures: list[dict] + dimension_refs: list[ColumnRef] + time_dims: list[TimeDimension] + time_dim_by_name: dict[str, TimeDimension] + derived_dims: list[str] + column_name_mapping: list[tuple[str, str]] + projection_types: list[DataType | None] def _record_metric( @@ -963,7 +2451,8 @@ def _record_metric( }) engine_alias = f"{table.name}.{item.projected_name}" plan.column_name_mapping.append((engine_alias, item.projected_name)) - plan.projection_types.append(item.metric.data_type) + # DEV-1566: CAST( AS T) overrides the declared metric type. + plan.projection_types.append(item.cast_target or item.metric.data_type) def _record_time_grain( @@ -978,6 +2467,15 @@ def _record_time_grain( plan.time_dims.append(td) plan.time_dim_by_name[dotted] = td plan.derived_dims.append(item.projected_name) + # When the projection aliases the time-grain expression (Metabase emits + # ``SELECT CAST(DATE_TRUNC('month', ordered_at) AS DATE) AS "ordered_at"`` + # together with ``GROUP BY CAST(DATE_TRUNC('month', ordered_at) AS DATE)``) + # the GROUP BY validator computes the canonical ``month(ordered_at)`` form + # for the unaliased GROUP BY expression. Register both forms so either one + # validates against the projection's derived dimension set. + canonical = f"{item.time_grain.value}({dotted})" + if canonical != item.projected_name: + plan.derived_dims.append(canonical) engine_alias = f"{table.name}.{dotted}" plan.column_name_mapping.append((engine_alias, item.projected_name)) plan.projection_types.append(item.time_grain_underlying.data_type) @@ -989,9 +2487,17 @@ def _record_dimension( assert item.dimension is not None plan.dimension_refs.append(ColumnRef.from_string(item.dimension.dimension_ref)) plan.derived_dims.append(item.projected_name) + # DEV-1565 (mirrors the time-grain canonical-alias trick): when the + # projection aliases a joined-col dim (e.g. ``"Stores"."name" AS + # "Stores__name"``) the GROUP BY / ORDER BY may reference the alias- + # qualified form, which alias-remap rewrites to the dotted SLayer + # form (``stores.name``). Register that form too so validation finds it. + if item.dimension.dimension_ref != item.projected_name: + plan.derived_dims.append(item.dimension.dimension_ref) engine_alias = f"{table.name}.{item.dimension.dimension_ref}" plan.column_name_mapping.append((engine_alias, item.projected_name)) - plan.projection_types.append(item.dimension.data_type) + # DEV-1566: CAST( AS T) overrides the declared dim type. + plan.projection_types.append(item.cast_target or item.dimension.data_type) def _build_projection_plan( @@ -1011,7 +2517,37 @@ def _build_projection_plan( return plan -def _parse_int_literal(node: Optional[exp.Expression]) -> Optional[int]: +def _index_items_by_canonical_form( + items: Sequence[_ProjectionItem], +) -> dict[str, _ProjectionItem]: + """Index projection items by alias plus canonical forms (time-grain only). + + Time-grain items are also keyed by ``grain(col)`` so an unaliased + Metabase-style GROUP BY / ORDER BY (``ORDER BY CAST(DATE_TRUNC('month', + ordered_at) AS DATE)``) resolves against the aliased projection. + + Column-CAST items are intentionally NOT registered here: DEV-1566's CAST + projection is a wire-type override that leaves the engine query + projecting the bare column, so ``ORDER BY CAST(col AS T)`` would sort by + the engine column's natural type (numeric/temporal) instead of the + casted type's semantics (e.g. lex for TEXT), and + ``GROUP BY CAST(ts AS DATE)`` would group per-timestamp instead of + per-date. Both are silently wrong. We reject these queries with the + existing "ORDER BY column not in projection list" / GROUP BY + strict-on-extras error and the user aliases the CAST projection to + reference it (``SELECT CAST(c AS T) AS x ... ORDER BY x``). + """ + by_name: dict[str, _ProjectionItem] = {item.projected_name: item for item in items} + for item in items: + if item.time_grain is not None and item.time_grain_underlying is not None: + canonical = ( + f"{item.time_grain.value}({item.time_grain_underlying.dimension_ref})" + ) + by_name.setdefault(canonical, item) + return by_name + + +def _parse_int_literal(node: exp.Expression | None) -> int | None: """Pull an int out of ``LIMIT N`` / ``OFFSET N`` style nodes.""" if node is None or not isinstance(node.expression, exp.Literal): return None @@ -1021,8 +2557,459 @@ def _parse_int_literal(node: Optional[exp.Expression]) -> Optional[int]: return None +# --- DEV-1565: LEFT JOIN-with-subquery recognition --------------------------- + + +class _JoinPlan(BaseModel): + """Result of recognising Metabase's single LEFT-JOIN-with-subquery shape.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + alias: str + target_table: FacadeTable + target_schema: str + source_col: str + target_col: str + is_dynamic: bool + warn_cardinality: bool = False + + +def _parse_left_join( + *, + parsed_joins: list[exp.Join], + parent_table: FacadeTable, + catalog: FacadeCatalog, +) -> _JoinPlan | None: + """Validate and parse the single LEFT JOIN on the SELECT, if any. + + Returns ``None`` when no joins are present. Raises ``TranslationError`` + for any shape outside the Phase-1 scope (multiple joins, non-LEFT type, + bare-table right side, exotic subquery body, malformed ON clause).""" + if not parsed_joins: + return None + if len(parsed_joins) > 1: + raise TranslationError( + f"Multiple JOINs in one query are not supported (Phase 1 — " + f"DEV-1565); got {len(parsed_joins)}. Use one LEFT JOIN." + ) + join = parsed_joins[0] + _reject_non_left_join(join) + target_table, target_schema, alias = _resolve_join_subquery_target(join, catalog) + source_col, target_col = _parse_on_clause( + on=join.args.get("on"), + parent_table=parent_table, + target_table=target_table, + alias=alias, + ) + is_dynamic, warn_cardinality = _classify_against_parent_joins( + parent_table=parent_table, + target_name=target_table.name, + source_col=source_col, + target_col=target_col, + ) + return _JoinPlan( + alias=alias, + target_table=target_table, + target_schema=target_schema, + source_col=source_col, + target_col=target_col, + is_dynamic=is_dynamic, + warn_cardinality=warn_cardinality, + ) + + +def _reject_non_left_join(join: exp.Join) -> None: + """Phase 1 accepts only LEFT JOIN (sqlglot: side='LEFT', kind in (None, + 'OUTER')). Every other shape — INNER / RIGHT / FULL / CROSS / plain + JOIN — is rejected with a single error surface.""" + side = join.args.get("side") + kind = join.args.get("kind") + side_upper = (side or "").upper() + kind_upper = (kind or "").upper() + if side_upper == "LEFT" and kind_upper in ("", "OUTER"): + return + raise TranslationError( + f"Only LEFT JOIN is supported (Phase 1 — DEV-1565); got " + f"{(side_upper + ' ' + kind_upper).strip() or 'JOIN'}." + ) + + +def _resolve_join_subquery_target( + join: exp.Join, catalog: FacadeCatalog, +) -> tuple[FacadeTable, str, str]: + """Validate the right-side subquery shape and resolve its FROM table. + + Returns ``(target_facade_table, target_schema, join_alias)``. Raises + on bare-table right side, non-Select subquery body, missing FROM, + inner WHERE/HAVING/GROUP/JOIN/CTE.""" + right = join.this + if not isinstance(right, exp.Subquery): + raise TranslationError( + "LEFT JOIN right side must be a subquery '(SELECT … FROM " + "
) AS ' (Phase 1 — DEV-1565); got a bare table " + "reference." + ) + alias = right.alias + if not alias: + raise TranslationError( + "LEFT JOIN subquery must have an alias: '(SELECT … FROM " + "
) AS '." + ) + inner = right.this + if not isinstance(inner, exp.Select): + raise TranslationError( + "LEFT JOIN subquery body must be a SELECT statement (Phase 1 " + "— DEV-1565); set-ops (UNION/INTERSECT/EXCEPT) not accepted." + ) + for forbidden, label in ( + ("where", "WHERE"), + ("group", _GROUP_BY_KIND), + ("having", "HAVING"), + ("joins", "JOIN"), + ("with_", "WITH (CTE)"), + # DEV-1565 (Codex round 1+2): DISTINCT / LIMIT / OFFSET inside + # the subquery change the joined row set in ways the translator + # would silently drop (the configured/dynamic join treats the + # right side as the full target table). Postgres also accepts + # OFFSET without LIMIT — handle both so the cardinality change + # always surfaces. + ("distinct", "DISTINCT"), + ("limit", "LIMIT"), + ("offset", "OFFSET"), + ): + if inner.args.get(forbidden): + raise TranslationError( + f"LEFT JOIN subquery body must be 'SELECT … FROM ' (Phase 1 — DEV-1565); inner {label} not accepted." + ) + inner_from = inner.args.get("from_") + if inner_from is None: + raise TranslationError( + "LEFT JOIN subquery body must have a FROM clause naming a " + "single SLayer model (Phase 1 — DEV-1565)." + ) + # Reject inner comma-join shape (FROM a, b). + inner_table = inner_from.this + if not isinstance(inner_table, exp.Table): + raise TranslationError( + "LEFT JOIN subquery body must reference exactly one table in " + "its FROM (Phase 1 — DEV-1565); got " + f"{type(inner_table).__name__}." + ) + # _resolve_table accepts an exp.From wrapper, so feed the inner FROM. + schema_name, target_table = _resolve_table(inner_from, catalog) + return target_table, schema_name, alias + + +def _parse_on_clause( + *, + on: exp.Expression | None, + parent_table: FacadeTable, + target_table: FacadeTable, + alias: str, +) -> tuple[str, str]: + """Parse the ON clause as exactly one equality between a parent-table- + qualified column and a join-alias-qualified column. + + Returns ``(source_col, target_col)`` in the canonical case used by + the underlying ``SlayerModel.columns[]`` (case-insensitive ON-column + lookup catches hand-written UPPERCASE refs that Postgres folds to + lowercase). Hidden columns counted. + """ + if on is None or not isinstance(on, exp.EQ): + raise TranslationError( + "LEFT JOIN ON clause must be a single equality " + "'.= .' (Phase 1 — DEV-1565); got " + f"{type(on).__name__ if on is not None else 'no ON clause'}." + ) + lhs, rhs = on.this, on.expression + if not isinstance(lhs, exp.Column) or not isinstance(rhs, exp.Column): + raise TranslationError( + "LEFT JOIN ON clause must compare two simple column refs " + "(Phase 1 — DEV-1565); function calls and expressions not " + "accepted." + ) + lhs_qual = _on_qualifier(lhs) + rhs_qual = _on_qualifier(rhs) + parent_match_l = _matches_parent_qualifier(lhs_qual, parent_table.name) + parent_match_r = _matches_parent_qualifier(rhs_qual, parent_table.name) + alias_match_l = _ci_eq(lhs_qual, alias) + alias_match_r = _ci_eq(rhs_qual, alias) + if parent_match_l and alias_match_r and not (parent_match_r and alias_match_l): + source_raw, target_raw = _leaf(lhs), _leaf(rhs) + elif parent_match_r and alias_match_l and not (parent_match_l and alias_match_r): + source_raw, target_raw = _leaf(rhs), _leaf(lhs) + else: + raise TranslationError( + f"LEFT JOIN ON clause must have one side qualified by the " + f"parent table ({parent_table.name!r}) and the other by the " + f"join alias ({alias!r}); got {on.sql()!r}." + ) + return ( + _canonical_column_or_raise(parent_table, source_raw, role="parent"), + _canonical_column_or_raise(target_table, target_raw, role="target"), + ) + + +def _on_qualifier(col: exp.Column) -> str | None: + """The table-qualifier of an ON-clause column ref, or None if bare.""" + table_node = col.args.get("table") + if table_node is None: + return None + return str(table_node.this) if hasattr(table_node, "this") else str(table_node) + + +def _matches_parent_qualifier( + qual: str | None, parent_name: str, +) -> bool: + """True if the qualifier names the parent table (case-insensitive). + Schema-prefix form is handled by ``_column_to_dotted``'s strip path, + so this only matches the table-name segment.""" + return qual is not None and _ci_eq(qual, parent_name) + + +def _ci_eq(a: str | None, b: str | None) -> bool: + return a is not None and b is not None and a.lower() == b.lower() + + +def _leaf(col: exp.Column) -> str: + leaf = col.this + return str(leaf.this) if hasattr(leaf, "this") else str(leaf) + + +def _canonical_column_or_raise( + table: FacadeTable, col_name: str, *, role: str, +) -> str: + """Lookup ``col_name`` against the underlying ``SlayerModel.columns[]`` + (hidden cols counted). Returns the canonical column name; raises + ``TranslationError`` on no match OR on case-insensitive ambiguity. + + Resolution order (Codex round 2): + 1. Exact (case-sensitive) match — preferred so user intent wins + when the model has two columns differing only by case + (``store_id`` and ``Store_ID``). + 2. Case-insensitive match — used when no exact match exists. + Raises if multiple columns match case-insensitively (Postgres + folds unquoted identifiers; sqlglot preserves what's written, + so a hand-written ``ON ... = STORE_ID`` should pick the lower- + case canonical column when that's the only match, but error + loudly if the model carries both cases). + + DEV-1565: Postgres folds unquoted identifiers to lowercase but + sqlglot preserves what was written, so hand-written SQL like + ``ON ORDERS.STORE_ID = STORES.ID`` would otherwise fail to match + against a model whose ``Column.name`` is ``store_id``. The canonical + return propagates the model-side casing to every downstream + comparison (the configured-join existence check, the dynamic-join + ``ModelExtension`` build). + """ + if table.model_ref is None: + return col_name + columns = table.model_ref.columns + for c in columns: + if c.name == col_name: + return c.name + needle = col_name.lower() + matches = [c.name for c in columns if c.name.lower() == needle] + if len(matches) == 1: + return matches[0] + if len(matches) > 1: + raise TranslationError( + f"LEFT JOIN ON {role}-side column {col_name!r} is " + f"case-insensitively ambiguous on table {table.name!r} " + f"(matches: {sorted(matches)}); quote the exact column name." + ) + raise TranslationError( + f"LEFT JOIN ON {role}-side column {col_name!r} does not exist " + f"on table {table.name!r}." + ) + + +def _classify_against_parent_joins( + *, + parent_table: FacadeTable, + target_name: str, + source_col: str, + target_col: str, +) -> tuple[bool, bool]: + """Match the emitted (target_model, join_pairs) against the parent's + configured joins. + + Returns ``(is_dynamic, warn_cardinality)``: + - existing LEFT match → (False, False). + - existing INNER match on same pairs → (False, True) (warn about + SQL LEFT vs configured INNER cardinality divergence). + - no entry for target_model → (True, False) (dynamic fallback). + - entry for target_model but DIFFERENT join_pairs → raise (an + additive ModelExtension would produce a duplicate join). + """ + same_target = [j for j in parent_table.joins if j.target_model == target_name] + if not same_target: + return True, False + matching_pairs = [ + j for j in same_target if list(j.join_pairs) == [[source_col, target_col]] + ] + if not matching_pairs: + configured = same_target[0].join_pairs + raise TranslationError( + f"LEFT JOIN to {target_name!r} uses ON columns " + f"({source_col!r}, {target_col!r}) which do not match the " + f"configured join_pairs ({configured}) on {parent_table.name!r}. " + f"ModelExtension cannot override an existing join — adjust the " + f"emitted SQL to match the configured join, or update the model " + f"join_pairs." + ) + j = matching_pairs[0] + warn_cardinality = j.join_type != JoinType.LEFT + return False, warn_cardinality + + +def _materialise_dynamic_join_lookups( + *, + target_model: SlayerModel, + extra_dims_by_name: dict[str, FacadeDimension], + extra_metrics_by_name: dict[str, FacadeMetric], + extra_metrics_by_formula: dict[str, FacadeMetric], +) -> None: + """For a dynamic-join target (no configured BFS expansion in the + catalog), build the bare-col dims and col×agg metrics keyed by + ``.`` / ``.:`` so the projection / + aggregate / WHERE / HAVING resolution paths find the joined refs. + """ + local_dims, local_metrics = build_local_view(target_model) + prefix = target_model.name + for dim in local_dims: + ref = f"{prefix}.{dim.dimension_ref}" + extra_dims_by_name[ref] = FacadeDimension( + name=ref, + description=dim.description, + label=dim.label, + data_type=dim.data_type, + is_time=dim.is_time, + dimension_ref=ref, + ) + for m in local_metrics: + if m.measure_formula == "*:count": + formula = f"{prefix}.*:count" + else: + formula = f"{prefix}.{m.measure_formula}" + new_metric = FacadeMetric( + name=f"{prefix}.{m.name}", + description=m.description, + label=m.label, + data_type=m.data_type, + measure_formula=formula, + ) + extra_metrics_by_name[f"{prefix}.{m.name}"] = new_metric + extra_metrics_by_formula[formula] = new_metric + + +def _build_source_model_from_join( + *, parent_name: str, plan: _JoinPlan, +) -> object: + """``SlayerQuery.source_model`` value derived from the join plan: + the parent's bare name when an existing configured join matched, or a + ``ModelExtension`` carrying the dynamically-built ``ModelJoin`` when + not. + """ + if not plan.is_dynamic: + return parent_name + return ModelExtension( + source_name=parent_name, + joins=[ModelJoin( + target_model=plan.target_table.name, + join_pairs=[[plan.source_col, plan.target_col]], + join_type=JoinType.LEFT, + )], + ) + + +def _emit_join_warnings(plan: _JoinPlan, parent_name: str) -> None: + if plan.is_dynamic: + logger.warning( + "pg-facade: dynamic join from %r to %r on %r=%r — no configured " + "join matched in parent.joins[]; using a ModelExtension to honor " + "the emitted ON clause (DEV-1565).", + parent_name, plan.target_table.name, plan.source_col, plan.target_col, + ) + elif plan.warn_cardinality: + logger.warning( + "pg-facade: LEFT JOIN to %r matched a configured non-LEFT join " + "(join_type=INNER on %r.joins) — using the configured join_type " + "but cardinality semantics differ from the emitted SQL (DEV-1565).", + plan.target_table.name, parent_name, + ) + + +class _JoinOverlays(BaseModel): + """Translator-state derived from a parsed ``_JoinPlan``: the alias + rewrite map and the on-demand projection-lookup overlays for the + dynamic-fallback case.""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + alias_map: dict[str, str] | None = None + extra_dims_by_name: dict[str, FacadeDimension] = {} + extra_metrics_by_name: dict[str, FacadeMetric] = {} + extra_metrics_by_formula: dict[str, FacadeMetric] = {} + + +def _prepare_join_overlays( + join_plan: _JoinPlan | None, parent_name: str, +) -> _JoinOverlays: + """Build the alias map + (for dynamic joins) the materialised lookup + overlays, and emit the join-related warnings. Extracted from + ``_translate_slayer_select`` to keep its cognitive complexity in check.""" + if join_plan is None: + return _JoinOverlays() + overlays = _JoinOverlays( + alias_map={join_plan.alias: join_plan.target_table.name}, + ) + if join_plan.is_dynamic and join_plan.target_table.model_ref is not None: + _materialise_dynamic_join_lookups( + target_model=join_plan.target_table.model_ref, + extra_dims_by_name=overlays.extra_dims_by_name, + extra_metrics_by_name=overlays.extra_metrics_by_name, + extra_metrics_by_formula=overlays.extra_metrics_by_formula, + ) + _emit_join_warnings(join_plan, parent_name) + return overlays + + +def _build_item_index(items: list[_ProjectionItem]) -> dict[str, _ProjectionItem]: + """Map every projection item by its user-facing name PLUS its canonical + secondary key (time-grain canonical form for `CAST(date_trunc(...))` + GROUP BY matches; dimension_ref dotted form for joined-col aliased + projections — see DEV-1565). ``setdefault`` preserves the primary + projected_name entry when the secondary key collides with it. + + DEV-1566: CAST-projected items are NOT registered under their + ``dimension_ref`` secondary key — that would route ``ORDER BY `` + on a query like ``SELECT CAST(id AS TEXT) AS x ... ORDER BY id`` to the + cast item and trip the lossy-pair rejection, even though the user + explicitly named the bare column (and the engine ordering already + matches that intent). + """ + out: dict[str, _ProjectionItem] = {item.projected_name: item for item in items} + for item in items: + if item.time_grain is not None and item.time_grain_underlying is not None: + canonical = ( + f"{item.time_grain.value}({item.time_grain_underlying.dimension_ref})" + ) + out.setdefault(canonical, item) + elif ( + item.dimension is not None + and item.cast_target is None + and item.dimension.dimension_ref != item.projected_name + ): + out.setdefault(item.dimension.dimension_ref, item) + return out + + def _translate_slayer_select( parsed: exp.Select, catalog: FacadeCatalog, + *, allow_column_cast: bool = True, + expand_star_in_browse_mode: bool = False, ) -> QueryResult: from_clause = parsed.args.get("from_") if from_clause is None: @@ -1033,25 +3020,78 @@ def _translate_slayer_select( schema_name, table = _resolve_table(from_clause, catalog) proj_exprs = parsed.args.get("expressions") or [] - # Reject SELECT * before catalog lookup so we get the named error - # instead of "Unknown projection item '*'". + # SELECT * handling. When ``expand_star_in_browse_mode`` is set (pg-facade + # only — see translate's docstring), a ``SELECT *`` with no GROUP BY / + # HAVING / aggregate in the projection list expands to every non-hidden + # column of the table. Flight's clients project explicit names by + # construction, so it leaves the flag default-False and ``*`` always + # raises there. Mixed ``*`` + aggregate cases always reject (the + # explicit "project specific names" hint is more useful guidance). if any(isinstance(e, exp.Star) for e in proj_exprs): - raise TranslationError(SELECT_STAR_MESSAGE) + if expand_star_in_browse_mode and _is_browse_mode_select(parsed, proj_exprs): + proj_exprs = _expand_select_star(proj_exprs, table) + else: + raise TranslationError(SELECT_STAR_MESSAGE) - items = _resolve_projection(proj_exprs, table) + # DEV-1558 B5: every helper that resolves a column ref needs the same + # ``(schema, table)`` prefix-strip context as ``_resolve_projection``. + strip_prefix: tuple[str, str] | None = ( + (schema_name, table.name) if schema_name else None + ) + + join_plan = _parse_left_join( + parsed_joins=parsed.args.get("joins") or [], + parent_table=table, + catalog=catalog, + ) + data_source = _resolve_query_datasource(table=table, join_plan=join_plan) + overlays = _prepare_join_overlays(join_plan, table.name) + + items = _resolve_projection( + proj_exprs, table, + schema_name=schema_name, + alias_map=overlays.alias_map, + extra_dims_by_name=overlays.extra_dims_by_name or None, + extra_metrics_by_name=overlays.extra_metrics_by_name or None, + extra_metrics_by_formula=overlays.extra_metrics_by_formula or None, + allow_column_cast=allow_column_cast, + ) plan = _build_projection_plan(items, table) + item_by_projected_name = _index_items_by_canonical_form(items) - _validate_group_by(parsed.args.get("group"), plan.derived_dims) + group = parsed.args.get("group") + _validate_group_by( + group, plan.derived_dims, item_by_projected_name, + strip_prefix=strip_prefix, alias_map=overlays.alias_map, + ) + _reject_lossy_cast_in_implicit_grouping(group, items) + + filters: list[str] = [] + _apply_where( + parsed.args.get("where"), plan.time_dim_by_name, + item_by_projected_name, filters, + strip_prefix=strip_prefix, alias_map=overlays.alias_map, + ) + _apply_having( + parsed.args.get("having"), table, + item_by_projected_name, filters, + strip_prefix=strip_prefix, alias_map=overlays.alias_map, + extra_metrics_by_formula=overlays.extra_metrics_by_formula or None, + ) - filters: List[str] = [] - _apply_where(parsed.args.get("where"), plan.time_dim_by_name, filters) - _apply_having(parsed.args.get("having"), table, filters) + order_items = _translate_order_by( + parsed.args.get("order"), _build_item_index(items), + strip_prefix=strip_prefix, alias_map=overlays.alias_map, + ) - item_by_projected_name = {item.projected_name: item for item in items} - order_items = _translate_order_by(parsed.args.get("order"), item_by_projected_name) + source_model: object = ( + _build_source_model_from_join(parent_name=table.name, plan=join_plan) + if join_plan is not None + else table.name + ) query = SlayerQuery( - source_model=table.name, + source_model=source_model, measures=plan.measures or None, dimensions=plan.dimension_refs or None, time_dimensions=plan.time_dims or None, @@ -1067,4 +3107,5 @@ def _translate_slayer_select( facade_table=table, schema_name=schema_name, projection_types=plan.projection_types, + data_source=data_source, ) diff --git a/slayer/flight/_capture_stub.py b/slayer/flight/_capture_stub.py index 27da282f..82d62173 100644 --- a/slayer/flight/_capture_stub.py +++ b/slayer/flight/_capture_stub.py @@ -21,7 +21,7 @@ import json import time from pathlib import Path -from typing import Any, Optional +from typing import Any import pyarrow as pa import pyarrow.flight as fl @@ -47,7 +47,7 @@ def _log(self, *, rpc: str, **payload: Any) -> None: f.write(json.dumps(record, default=str) + "\n") @staticmethod - def _b64(b: Optional[bytes]) -> Optional[str]: + def _b64(b: bytes | None) -> str | None: return base64.b64encode(b).decode("ascii") if b else None @staticmethod @@ -146,7 +146,7 @@ def list_actions(self, context: fl.ServerCallContext): return [] def do_action(self, context: fl.ServerCallContext, action: fl.Action): - body_bytes: Optional[bytes] = None + body_bytes: bytes | None = None if action.body is not None: body_bytes = action.body.to_pybytes() self._log( @@ -172,7 +172,7 @@ def _log(self, *, rpc: str, **payload: Any) -> None: f.write(json.dumps(record, default=str) + "\n") @staticmethod - def _b64(b: Optional[bytes]) -> Optional[str]: + def _b64(b: bytes | None) -> str | None: return base64.b64encode(b).decode("ascii") if b else None @staticmethod @@ -205,9 +205,9 @@ def __init__( location: str, handlers: FlightHandlers, log_path: Path, - token: Optional[str] = None, - tls_cert: Optional[str] = None, - tls_key: Optional[str] = None, + token: str | None = None, + tls_cert: str | None = None, + tls_key: str | None = None, ) -> None: super().__init__( location=location, @@ -244,7 +244,7 @@ def do_get(self, context: fl.ServerCallContext, ticket: fl.Ticket): return super().do_get(context, ticket) def do_action(self, context: fl.ServerCallContext, action: fl.Action): - body_bytes: Optional[bytes] = ( + body_bytes: bytes | None = ( action.body.to_pybytes() if action.body is not None else None ) self._recorder._log( diff --git a/slayer/flight/auth.py b/slayer/flight/auth.py index 6ed1d0e1..ce8c62bc 100644 --- a/slayer/flight/auth.py +++ b/slayer/flight/auth.py @@ -20,7 +20,6 @@ import hmac import ipaddress import logging -from typing import Optional import pyarrow.flight as fl @@ -52,7 +51,7 @@ def _is_loopback(host: str) -> bool: return False -def validate_bind_address(*, host: str, token: Optional[str]) -> None: +def validate_bind_address(*, host: str, token: str | None) -> None: """Raise ``ValueError`` if the server is about to bind a non-loopback address without a configured token (§4.3 / §7.1). """ @@ -66,7 +65,7 @@ def validate_bind_address(*, host: str, token: Optional[str]) -> None: ) -def validate_tls_pair(*, cert: Optional[str], key: Optional[str]) -> None: +def validate_tls_pair(*, cert: str | None, key: str | None) -> None: """TLS cert/key must be supplied together or not at all (§4.4).""" if (cert is None) != (key is None): raise ValueError( @@ -78,10 +77,10 @@ def validate_tls_pair(*, cert: Optional[str], key: Optional[str]) -> None: class _BearerTokenMiddleware(fl.ServerMiddleware): """No-op once-per-call middleware; auth check happened in the factory.""" - def __init__(self, *, environment_id: Optional[str] = None) -> None: + def __init__(self, *, environment_id: str | None = None) -> None: self._environment_id = environment_id - def call_completed(self, exception: Optional[BaseException]) -> None: + def call_completed(self, exception: BaseException | None) -> None: if exception is not None and self._environment_id is not None: logger.debug( "Flight SQL call (environmentId=%s) failed: %r", @@ -107,12 +106,12 @@ class BearerTokenMiddlewareFactory(fl.ServerMiddlewareFactory): handler-layer recheck would be possible if we ever want one.) """ - def __init__(self, *, token: Optional[str]) -> None: + def __init__(self, *, token: str | None) -> None: self._expected = token def start_call( self, info: fl.CallInfo, headers: dict - ) -> Optional[fl.ServerMiddleware]: + ) -> fl.ServerMiddleware | None: # Extract and lowercase header keys (gRPC standardises to lowercase # but client implementations differ). normalised = { @@ -121,7 +120,7 @@ def start_call( for k, v in (headers or {}).items() } env_id_raw = normalised.get("environmentid") - environment_id: Optional[str] = None + environment_id: str | None = None if isinstance(env_id_raw, (bytes, bytearray)): environment_id = env_id_raw.decode("utf-8", errors="replace") elif isinstance(env_id_raw, str): @@ -130,7 +129,7 @@ def start_call( logger.info("Flight SQL request environmentId=%s", environment_id) auth_raw = normalised.get("authorization") - provided: Optional[str] = None + provided: str | None = None if isinstance(auth_raw, (bytes, bytearray)): auth_raw = auth_raw.decode("utf-8", errors="replace") if isinstance(auth_raw, str) and auth_raw.lower().startswith("bearer "): diff --git a/slayer/flight/cli.py b/slayer/flight/cli.py index 14de3754..d472d008 100644 --- a/slayer/flight/cli.py +++ b/slayer/flight/cli.py @@ -10,7 +10,6 @@ import logging import os import sys -from typing import Optional logger = logging.getLogger(__name__) @@ -92,7 +91,7 @@ def run_flight_serve(args, *, resolve_storage, prepare_demo) -> None: engine = SlayerQueryEngine(storage=storage) handlers = FlightHandlers(engine=engine, storage=storage) - token: Optional[str] = args.token or os.environ.get("SLAYER_FLIGHT_TOKEN") + token: str | None = args.token or os.environ.get("SLAYER_FLIGHT_TOKEN") host = _resolve_host(host_arg=args.host, demo=args.demo, token=token) @@ -112,7 +111,7 @@ def run_flight_serve(args, *, resolve_storage, prepare_demo) -> None: server.serve() -def _resolve_host(*, host_arg: Optional[str], demo: bool, token: Optional[str]) -> str: +def _resolve_host(*, host_arg: str | None, demo: bool, token: str | None) -> str: """Apply the §7.1 demo-loopback default. If --host is not explicitly given AND --demo is set AND no token is diff --git a/slayer/flight/handlers.py b/slayer/flight/handlers.py index e8bf1fbc..aa9b514e 100644 --- a/slayer/flight/handlers.py +++ b/slayer/flight/handlers.py @@ -13,7 +13,6 @@ import decimal import logging from collections import defaultdict -from typing import Dict, List, Tuple import pyarrow as pa import pyarrow.flight as fl @@ -45,7 +44,7 @@ _TYPE_URL_PREFIX = "type.googleapis.com/arrow.flight.protocol.sql." -_COMMAND_BY_TYPE_URL: Dict[str, type] = { +_COMMAND_BY_TYPE_URL: dict[str, type] = { f"{_TYPE_URL_PREFIX}CommandStatementQuery": fsql_pb.CommandStatementQuery, f"{_TYPE_URL_PREFIX}CommandPreparedStatementQuery": fsql_pb.CommandPreparedStatementQuery, f"{_TYPE_URL_PREFIX}CommandGetCatalogs": fsql_pb.CommandGetCatalogs, @@ -67,7 +66,7 @@ } -def _decode_any(buf: bytes) -> Tuple[str, object]: +def _decode_any(buf: bytes) -> tuple[str, object]: """Decode an Any-wrapped Flight SQL command. Returns ``(type_url, message)``.""" any_msg = PbAny() any_msg.ParseFromString(buf) @@ -185,10 +184,10 @@ def _build_catalog(self) -> FlightCatalog: models_by_ds = self._fetch_models_by_datasource() return build_catalog(models_by_datasource=models_by_ds) - def _fetch_models_by_datasource(self) -> Dict[str, List[SlayerModel]]: - async def fetch() -> Dict[str, List[SlayerModel]]: + def _fetch_models_by_datasource(self) -> dict[str, list[SlayerModel]]: + async def fetch() -> dict[str, list[SlayerModel]]: datasources = await self._storage.list_datasources() - out: Dict[str, List[SlayerModel]] = defaultdict(list) + out: dict[str, list[SlayerModel]] = defaultdict(list) for ds in datasources: model_names = await self._storage.list_models(data_source=ds) for name in model_names: @@ -401,7 +400,7 @@ def _build_schema(result: "QueryResult") -> pa.Schema: @staticmethod def _rewrite_row( - row: dict, mapping: List[Tuple[str, str]], + row: dict, mapping: list[tuple[str, str]], ) -> dict: """Rewrite an engine row's keys into projected names + coerce Decimals.""" out: dict = {} @@ -439,12 +438,12 @@ def _build_flight_info( # --- top-level dispatch ------------------------------------------------------- -def decode_command(buf: bytes) -> Tuple[str, object]: +def decode_command(buf: bytes) -> tuple[str, object]: """Public re-export for tests / the server.""" return _decode_any(buf) -def decode_ticket(buf: bytes) -> Tuple[str, object]: +def decode_ticket(buf: bytes) -> tuple[str, object]: """Tickets are also Any-wrapped (TicketStatementQuery / CommandPreparedStatementQuery).""" return _decode_any(buf) diff --git a/slayer/flight/info_schema.py b/slayer/flight/info_schema.py index 3313e204..533d560f 100644 --- a/slayer/flight/info_schema.py +++ b/slayer/flight/info_schema.py @@ -7,7 +7,6 @@ from __future__ import annotations -from typing import Optional import pyarrow as pa import sqlglot.expressions as exp @@ -23,7 +22,7 @@ def match_info_schema( *, parsed: exp.Expression, catalog: FlightCatalog, -) -> Optional[pa.Table]: +) -> pa.Table | None: """Return the canned ``INFORMATION_SCHEMA.
`` answer as a ``pyarrow.Table`` or ``None``.""" batch = _shared_match_info_schema(parsed=parsed, catalog=catalog) diff --git a/slayer/flight/probe_queries.py b/slayer/flight/probe_queries.py index ebf22778..e963d038 100644 --- a/slayer/flight/probe_queries.py +++ b/slayer/flight/probe_queries.py @@ -6,7 +6,6 @@ from __future__ import annotations -from typing import Optional import pyarrow as pa import sqlglot.expressions as exp @@ -15,7 +14,7 @@ from slayer.flight.types import row_batch_to_arrow -def match_probe(parsed: exp.Expression) -> Optional[pa.Table]: +def match_probe(parsed: exp.Expression) -> pa.Table | None: """Return the canned ``pyarrow.Table`` for a matching probe, else ``None``.""" batch = _shared_match_probe(parsed) if batch is None: diff --git a/slayer/flight/server.py b/slayer/flight/server.py index b012e272..c3106bd1 100644 --- a/slayer/flight/server.py +++ b/slayer/flight/server.py @@ -11,7 +11,7 @@ import logging from pathlib import Path -from typing import Iterator, Optional +from collections.abc import Iterator import pyarrow.flight as fl @@ -89,9 +89,9 @@ def __init__( *, location: str, handlers: FlightHandlers, - token: Optional[str] = None, - tls_cert: Optional[str] = None, - tls_key: Optional[str] = None, + token: str | None = None, + tls_cert: str | None = None, + tls_key: str | None = None, ) -> None: tls_certificates = [] if tls_cert is not None and tls_key is not None: @@ -251,9 +251,9 @@ def build_server( host: str, port: int, handlers: FlightHandlers, - token: Optional[str] = None, - tls_cert: Optional[str] = None, - tls_key: Optional[str] = None, + token: str | None = None, + tls_cert: str | None = None, + tls_key: str | None = None, ) -> FlightSqlServer: """Factory wrapping ``FlightSqlServer`` with startup-time validation. diff --git a/slayer/flight/translator.py b/slayer/flight/translator.py index 2a4ca4c2..8ee4fb91 100644 --- a/slayer/flight/translator.py +++ b/slayer/flight/translator.py @@ -50,8 +50,18 @@ class InfoSchemaResult(TranslatorResult): def translate(sql: str, catalog: FacadeCatalog) -> TranslatorResult: """Translate ``sql`` for the Flight facade (``dialect=None``), converting - the shared ``RowBatch`` results into Arrow-shaped ones.""" - result = _shared_translate(sql, catalog, dialect=None) + the shared ``RowBatch`` results into Arrow-shaped ones. + + ``allow_column_cast=False`` — DEV-1566 ``CAST(AS )`` projections + are gated to the pg-facade. The Flight handler materialises rows via + ``pa.Table.from_pylist`` against a catalog-typed schema, which rejects + values whose Python type doesn't match the declared Arrow type + (e.g. ``datetime.date`` against ``pa.timestamp``); Flight has no + value-coercion pass to bridge the gap. Rejecting at translate time + surfaces a clean ``TranslationError`` instead of an opaque + ``ArrowTypeError`` at materialisation. + """ + result = _shared_translate(sql, catalog, dialect=None, allow_column_cast=False) if isinstance(result, _SharedProbeResult): return ProbeResult(table=row_batch_to_arrow(result.batch)) if isinstance(result, _SharedInfoSchemaResult): diff --git a/slayer/flight/types.py b/slayer/flight/types.py index d3a2a926..0835229e 100644 --- a/slayer/flight/types.py +++ b/slayer/flight/types.py @@ -4,19 +4,21 @@ (``SUPPORTED_DATATYPES`` + ``datatype_to_jdbc``) lives in ``slayer.facade.datatypes`` and is re-exported here for backward compat. -* SLayer's ``DataType`` (``slayer.core.enums``) — six canonical values. +* SLayer's ``DataType`` (``slayer.core.enums``) — six operable values plus + the opaque ``UNKNOWN`` marker. * Apache Arrow ``DataType`` — the wire encoding the Flight SQL gRPC server emits to clients. -The forward direction (``DataType → Arrow``) is total over the six -supported values. The reverse (``Arrow → DataType``) collapses Arrow's -wider type space onto the six SLayer types; ``arrow_to_datatype`` -returns ``None`` for genuinely unmappable Arrow types. +The forward direction (``DataType → Arrow``) is total over every value. +The reverse (``Arrow → DataType``) collapses Arrow's wider type space onto +the six operable SLayer types; ``arrow_to_datatype`` returns ``None`` for +genuinely unmappable Arrow types. The round trip is lossy for ``UNKNOWN`` +(it goes out as ``utf8`` and comes back as ``TEXT``) — Arrow has no +"unclassified" type to preserve it. """ from __future__ import annotations -from typing import Optional import pyarrow as pa @@ -31,6 +33,10 @@ DataType.BOOLEAN: pa.bool_(), DataType.DATE: pa.date32(), DataType.TIMESTAMP: pa.timestamp("us"), + # Opaque columns travel as Arrow strings — see DataType.is_opaque. The + # forward map is intentionally lossy here: UNKNOWN -> utf8 -> TEXT does + # not round-trip back to UNKNOWN. + DataType.UNKNOWN: pa.utf8(), } @@ -39,7 +45,7 @@ def datatype_to_arrow(dt: DataType) -> pa.DataType: return _DATATYPE_TO_ARROW[dt] -def arrow_to_datatype(at: pa.DataType) -> Optional[DataType]: +def arrow_to_datatype(at: pa.DataType) -> DataType | None: """Best-effort reverse map. Returns ``None`` if ``at`` cannot be coerced into one of the six diff --git a/slayer/help/__init__.py b/slayer/help/__init__.py deleted file mode 100644 index 68bb9d13..00000000 --- a/slayer/help/__init__.py +++ /dev/null @@ -1,109 +0,0 @@ -"""Conceptual help for SLayer. - -Content is authored as ``.md`` files under :mod:`slayer.help.topics`. Topic -names are discovered at module import time by scanning that directory, so -adding a new topic is a matter of dropping a new ``NN_name.md`` file in — -no Python changes needed. - -Public API: - -* :func:`render_help` — return the intro when called with no topic, or the - content of the requested topic. Returns a friendly error string (never - raises) for unknown topics. -* :func:`available_topic_names` — ordered tuple of topic keys (excluding the - intro). -* :data:`TOPIC_SUMMARY_LINE` — one-line string listing every topic, reused by - the MCP tool description and the CLI subparser epilog. - -Filenames in ``topics/`` use an ``NN_name.md`` convention (e.g. -``01_queries.md``) so that sorted filesystem iteration gives a stable -teaching order. The ``NN_`` prefix is stripped to form the topic key. -``00_intro.md`` is treated specially as the intro body returned when the -caller asks for no topic in particular. -""" - -from __future__ import annotations - -from importlib.resources import files -from typing import Optional - -__all__ = ( - "render_help", - "available_topic_names", - "TOPIC_SUMMARY_LINE", - "INTRO_KEY", -) - -INTRO_KEY = "intro" - -_TOPICS_SUBDIR = "topics" - - -def _strip_numeric_prefix(stem: str) -> str: - """Turn ``"01_queries"`` into ``"queries"``; leave other stems unchanged.""" - if len(stem) >= 3 and stem[0].isdigit() and stem[1].isdigit() and stem[2] == "_": - return stem[3:] - return stem - - -def _discover() -> tuple[str, dict[str, str], tuple[str, ...]]: - """Scan ``topics/*.md`` once at module load. - - Returns ``(intro_body, topic_bodies, ordered_topic_keys)``. ``topic_bodies`` - does not contain the intro. ``ordered_topic_keys`` preserves filesystem - sort order (driven by the ``NN_`` prefix). - """ - topics_dir = files(__name__) / _TOPICS_SUBDIR - intro_body = "" - bodies: dict[str, str] = {} - order: list[str] = [] - - entries = sorted(topics_dir.iterdir(), key=lambda e: e.name) - for entry in entries: - if not entry.is_file() or not entry.name.endswith(".md"): - continue - stem = entry.name[: -len(".md")] - key = _strip_numeric_prefix(stem) - body = entry.read_text(encoding="utf-8").rstrip() + "\n" - if key == INTRO_KEY: - intro_body = body - else: - bodies[key] = body - order.append(key) - - return intro_body, bodies, tuple(order) - - -_INTRO_BODY, _TOPIC_BODIES, _TOPIC_ORDER = _discover() - -TOPIC_SUMMARY_LINE = ( - "Available help topics: " + ", ".join(_TOPIC_ORDER) + "." - if _TOPIC_ORDER - else "No help topics are installed." -) - - -def available_topic_names() -> tuple[str, ...]: - """Return the ordered tuple of topic keys (intro excluded).""" - return _TOPIC_ORDER - - -def render_help(topic: Optional[str] = None) -> str: - """Render help content. - - * No topic (``None``, empty, or whitespace-only): return the intro body. - * Known topic (case-insensitive, leading/trailing whitespace ignored): - return that topic's body. - * Unknown topic: return a friendly ``"Unknown topic 'X'. Available: ..."`` - string. Never raises. - """ - if topic is None or not str(topic).strip(): - return _INTRO_BODY - key = str(topic).strip().lower() - body = _TOPIC_BODIES.get(key) - if body is not None: - return body - return ( - f"Unknown help topic '{topic}'. " - f"Available topics: {', '.join(_TOPIC_ORDER)}." - ) diff --git a/slayer/ingest_report.py b/slayer/ingest_report.py new file mode 100644 index 00000000..28cc1fc0 --- /dev/null +++ b/slayer/ingest_report.py @@ -0,0 +1,73 @@ +"""Shared conversion-report types for semantic-layer importers (DEV-1643). + +``ConversionWarning`` / ``ConversionResult`` were originally defined in +``slayer.dbt.converter``; they are extracted here so both the dbt importer and +the OSI importer (``slayer.osi.converter``) can reuse them without importing +each other. ``slayer.dbt.converter`` re-exports them for back-compat, so the +class objects are shared (identity-equal) across both import paths. +""" + +from collections import defaultdict +from typing import Literal + +from pydantic import BaseModel, Field + +from slayer.core.models import SlayerModel + + +class ConversionWarning(BaseModel): + """A structured entry in a conversion report. + + ``category`` groups entries in ``render_report``; ``severity`` is one of + ``"unconverted"`` (tried to convert, couldn't), ``"dropped"`` (intentional + clean-fail of an inexpressible construct), or ``"info"`` (a caveat — the + construct imports but has a runtime limitation). ``suggestion`` carries the + documented workaround. + """ + model_name: str | None = None + metric_name: str | None = None + message: str + category: str = "general" + severity: Literal["unconverted", "dropped", "info"] = "unconverted" + suggestion: str | None = None + + +class ConversionResult(BaseModel): + """Result of converting a semantic-layer project into SLayer models.""" + models: list[SlayerModel] = Field(default_factory=list) + unconverted_metrics: list[ConversionWarning] = Field(default_factory=list) + warnings: list[ConversionWarning] = Field(default_factory=list) + + def _all_entries(self) -> list[ConversionWarning]: + return list(self.unconverted_metrics) + list(self.warnings) + + def render_report(self) -> str: + """Render the conversion report grouped by category. + + Each category becomes a heading with a count; each entry lists its + entity, severity, reason, and (when present) the documented workaround. + """ + entries = self._all_entries() + if not entries: + return "No conversion issues." + by_cat: dict[str, list[ConversionWarning]] = defaultdict(list) + for e in entries: + by_cat[e.category or "general"].append(e) + lines: list[str] = [] + for cat in sorted(by_cat): + items = by_cat[cat] + lines.append(f"## {cat} ({len(items)})") + for e in items: + entity = e.metric_name or e.model_name or "general" + lines.append(f" - [{e.severity}] {entity}: {e.message}") + if e.suggestion: + lines.append(f" workaround: {e.suggestion}") + lines.append("") + return "\n".join(lines).rstrip() + + def tally(self) -> tuple[int, int]: + """``(unconverted, dropped)`` counts by severity for the CLI summary.""" + entries = self._all_entries() + unconverted = sum(1 for e in entries if e.severity == "unconverted") + dropped = sum(1 for e in entries if e.severity == "dropped") + return unconverted, dropped diff --git a/slayer/inspect/__init__.py b/slayer/inspect/__init__.py new file mode 100644 index 00000000..a0ba21fe --- /dev/null +++ b/slayer/inspect/__init__.py @@ -0,0 +1,8 @@ +"""DEV-1588: shared single-entity inspection core. + +Exposes :class:`slayer.inspect.service.InspectService` (the shared core +behind the MCP ``inspect`` tool + REST/CLI/SlayerClient surfaces) and the +model-render helpers extracted out of ``slayer/mcp/server.py`` so the +``inspect`` surfaces and the legacy ``inspect_model`` MCP tool share one +implementation. +""" diff --git a/slayer/inspect/collection_render.py b/slayer/inspect/collection_render.py new file mode 100644 index 00000000..37198980 --- /dev/null +++ b/slayer/inspect/collection_render.py @@ -0,0 +1,334 @@ +"""DEV-1667: shared renderers for the ``inspect`` collection views. + +A null/omitted ``reference`` on ``inspect`` renders the *collection* at an +``entity_type``. These pure renderers are the single code path shared by the +``inspect`` collection dispatch AND the ``models_summary`` / ``list_datasources`` +MCP tools (kept as thin aliases), guaranteeing byte-identical output. + +``slayer.inspect`` must NOT import ``slayer.mcp`` (cycle avoidance); the shared +markdown/skeleton helpers live in ``slayer.inspect.model_render``. +""" + +from __future__ import annotations + +import json +from typing import Any + +from slayer.core.models import SlayerModel +from slayer.inspect.model_render import ( + _markdown_table, + _truncate_description, + model_skeleton_fields, +) + +# markdown rule separating per-datasource blocks in compact=False collections +# (same rule the DEV-1612 batch view uses between per-id blocks). +BLOCK_SEP = "\n\n---\n\n" + +_NO_DATASOURCES = ( + "No datasources configured. Use create_datasource to add a database " + "connection." +) + + +def _visible_column_count(model: SlayerModel) -> int: + return sum(1 for c in model.columns if not c.hidden) + + +def _join_targets(model: SlayerModel) -> list[str]: + return sorted({j.target_model for j in model.joins}) + + +# --------------------------------------------------------------------------- +# models_summary — extracted verbatim from the MCP tool (byte-identical) +# --------------------------------------------------------------------------- + + +def render_models_summary( + *, + datasource_name: str, + models: list[SlayerModel], + fmt: str, + compact: bool, + descriptions_max_chars: int | None = None, +) -> str: + """Render a datasource's (already hidden-filtered + name-sorted) models. + + Extracted from ``mcp/server.py::models_summary`` so both the tool and the + ``inspect`` model collection (compact=False) render through one path. With + ``descriptions_max_chars=None`` the output is byte-identical to the tool. + """ + if not models: + # An empty datasource must still emit valid JSON under fmt="json" + # (a plain-text sentinel would break json.loads for the caller). + if fmt == "json": + return json.dumps( + {"datasource_name": datasource_name, "model_count": 0, "models": []}, + indent=2, + ) + return f"Datasource '{datasource_name}' has no models." + desc = _desc_fn(descriptions_max_chars) + if fmt == "json": + return _models_summary_json( + datasource_name=datasource_name, models=models, compact=compact, + desc=desc, + ) + return _models_summary_markdown( + datasource_name=datasource_name, models=models, compact=compact, + desc=desc, + ) + + +def _desc_fn(descriptions_max_chars: int | None): + def _desc(text: str | None) -> str | None: + return _truncate_description(text, descriptions_max_chars) + return _desc + + +def _models_summary_json( + *, datasource_name: str, models: list[SlayerModel], compact: bool, desc, +) -> str: + if compact: + model_payload = [ + { + "name": m.name, + "description": desc(m.description), + "column_count": _visible_column_count(m), + "measure_names": [mm.name for mm in m.measures], + "joins_to": _join_targets(m), + } + for m in models + ] + else: + model_payload = [ + { + "name": m.name, + "description": desc(m.description), + "columns": [ + { + "name": c.name, + "type": str(c.type), + "description": desc(c.description), + } + for c in m.columns if not c.hidden + ], + "measures": [ + { + "name": mm.name, + "formula": mm.formula, + "description": desc(mm.description), + } + for mm in m.measures + ], + "joins_to": _join_targets(m), + } + for m in models + ] + return json.dumps( + { + "datasource_name": datasource_name, + "model_count": len(models), + "models": model_payload, + }, + indent=2, + ) + + +def _models_summary_markdown( + *, datasource_name: str, models: list[SlayerModel], compact: bool, desc, +) -> str: + sections: list[str] = [ + f"# Datasource: `{datasource_name}` — {len(models)} model(s)" + ] + for m in models: + model_lines: list[str] = [f"## `{m.name}`"] + if m.description: + model_lines.append(desc(m.description) or "") + if compact: + _append_compact_model_lines(model_lines=model_lines, m=m) + else: + _append_verbose_model_lines(model_lines=model_lines, m=m, desc=desc) + sections.append("\n".join(model_lines)) + return "\n\n".join(sections) + + +def _append_compact_model_lines(*, model_lines: list[str], m: SlayerModel) -> None: + model_lines.append(f"Columns: {_visible_column_count(m)}") + measure_names = ", ".join(mm.name for mm in m.measures if mm.name is not None) + model_lines.append(f"Measures: {measure_names}") + if m.joins: + rendered = ", ".join(f"`{t}`" for t in _join_targets(m)) + model_lines.append(f"Joins to: {rendered}") + else: + model_lines.append("Joins to: _(none)_") + + +def _append_verbose_model_lines( + *, model_lines: list[str], m: SlayerModel, desc, +) -> None: + col_rows = [ + {"name": c.name, "type": str(c.type), "description": desc(c.description)} + for c in m.columns if not c.hidden + ] + model_lines.append(f"**Columns ({len(col_rows)}):**") + model_lines.append("") + model_lines.append( + _markdown_table(rows=col_rows, columns=["name", "type", "description"]) + ) + model_lines.append("") + + measure_rows = [ + {"name": mm.name, "formula": mm.formula, "description": desc(mm.description)} + for mm in m.measures + ] + model_lines.append(f"**Measures ({len(measure_rows)}):**") + model_lines.append("") + model_lines.append( + _markdown_table(rows=measure_rows, columns=["name", "formula", "description"]) + ) + model_lines.append("") + + if m.joins: + rendered = ", ".join(f"`{t}`" for t in _join_targets(m)) + model_lines.append(f"**Joins to:** {rendered}") + else: + model_lines.append("**Joins to:** _(none)_") + + +# --------------------------------------------------------------------------- +# Model collection — compact=True one-liner index +# --------------------------------------------------------------------------- + +# A per-DS group is (data_source, models) where ``models is None`` marks a +# datasource whose config failed to load (invalid-config tolerance). +ModelGroup = tuple[str, list[SlayerModel] | None] + + +def render_model_oneliner_index( + *, + groups: list[ModelGroup], + fmt: str, + warnings: list[str], +) -> str: + """The compact=True model collection: one terse line per model, grouped by + datasource. Deliberately terser than ``models_summary`` (scales to large + catalogs).""" + if fmt == "json": + return _oneliner_index_json(groups=groups, warnings=warnings) + return _oneliner_index_markdown(groups=groups, warnings=warnings) + + +def _oneliner_index_json( + *, groups: list[ModelGroup], warnings: list[str], +) -> str: + entries: list[dict[str, Any]] = [] + for ds, models in groups: + if models is None: + entries.append( + {"data_source": ds, "error": "invalid config", "models": []} + ) + continue + entries.append({ + "data_source": ds, + "model_count": len(models), + "models": [ + { + "name": m.name, + "column_count": _visible_column_count(m), + "joins_to": _join_targets(m), + } + for m in models + ], + }) + return json.dumps({ + "entity_type": "model", + "collection": True, + "datasources": entries, + "warnings": warnings, + }, indent=2, default=str) + + +def _oneliner_index_markdown( + *, groups: list[ModelGroup], warnings: list[str], +) -> str: + blocks: list[str] = [] + for ds, models in groups: + if models is None: + blocks.append(f"# Datasource: `{ds}` — (ERROR: invalid config)") + continue + lines = [f"# Datasource: `{ds}` — {len(models)} model(s)"] + for m in models: + joins = _join_targets(m) + joins_str = ( + ", ".join(f"`{t}`" for t in joins) if joins else "_(none)_" + ) + lines.append( + f"- `{m.name}` ({_visible_column_count(m)} cols; " + f"joins: {joins_str})" + ) + blocks.append("\n".join(lines)) + body = "\n\n".join(blocks) + if warnings: + warn_block = "\n".join(f"> Warning: {w}" for w in warnings) + return f"{body}\n\n{warn_block}" if body else warn_block + return body + + +# --------------------------------------------------------------------------- +# Datasource collection — compact=True listing (list_datasources alias) +# --------------------------------------------------------------------------- + +# A datasource pair is (name, type) where ``type is None`` marks an +# invalid-config datasource. +DatasourcePair = tuple[str, str | None] + + +def render_datasource_list( + *, + pairs: list[DatasourcePair], + fmt: str, + warnings: list[str] | None = None, +) -> str: + """The compact=True datasource collection. In markdown this is byte-identical + to the ``list_datasources`` tool (the tool delegates here).""" + if fmt == "json": + entries: list[dict[str, Any]] = [] + for name, ds_type in pairs: + if ds_type is None: + entries.append({"name": name, "error": "invalid config"}) + else: + entries.append({"name": name, "type": ds_type}) + return json.dumps({ + "entity_type": "datasource", + "collection": True, + "datasources": entries, + "warnings": warnings or [], + }, indent=2) + + if not pairs: + return _NO_DATASOURCES + lines = [ + f"- {name} ({ds_type})" if ds_type is not None + else f"- {name} (ERROR: invalid datasource config)" + for name, ds_type in pairs + ] + return "\n".join(lines) + + +def datasource_skeleton_fields( + *, + name: str, + description: str | None, + models: list[SlayerModel], + descriptions_max_chars: int | None, +) -> dict[str, Any]: + """The datasource compact=False JSON per-DS element: name + description + + per-model skeletons.""" + return { + "name": name, + "description": _truncate_description(description, descriptions_max_chars), + "models": [ + model_skeleton_fields(model=m, max_chars=descriptions_max_chars) + for m in models + ], + } diff --git a/slayer/inspect/model_render.py b/slayer/inspect/model_render.py new file mode 100644 index 00000000..5c48a89d --- /dev/null +++ b/slayer/inspect/model_render.py @@ -0,0 +1,1326 @@ +"""DEV-1588: model-render core extracted from ``slayer/mcp/server.py``. + +This module owns the helpers + the full ``render_model_inspection`` body +that the legacy MCP ``inspect_model`` tool used to inline. Both the +``inspect`` surfaces (via :class:`slayer.inspect.service.InspectService`) +and the kept-but-deprecated ``inspect_model`` tool now delegate here, so +there is a single source of truth for the model render. + +IMPORTANT: this module must NOT import ``slayer.mcp`` — ``mcp/server.py`` +imports from here, so the reverse would be a circular import. +""" + +from __future__ import annotations + +import json +import logging +from typing import Any + +import sqlalchemy as sa + +from slayer.core.enums import DataType +from slayer.core.models import Column, SlayerModel +from slayer.core.query import SlayerQuery +from slayer.engine.ingestion import _friendly_db_error +from slayer.engine.profiling import ( + _is_sample_cached, + _profile_numeric_temporal_columns, + ensure_column_sample_fresh, +) +from slayer.engine.query_engine import SlayerQueryEngine +from slayer.search.render import compact_description_from_learning +from slayer.storage.base import StorageBackend + +logger = logging.getLogger(__name__) + +# Aggregations that are safe for sample-data extraction: zero extra args, +# no time-column context needed. +_SAFE_SAMPLE_AGGS = frozenset({"avg", "sum", "min", "max", "count", "count_distinct", "median"}) + +# The one failure the Data Profile's count-only retry is designed to recover +# from: the database cannot group / deduplicate one of the column types. Kept +# deliberately narrow — every other failure (permission, connection, syntax, +# validation) must keep its own cause rather than being relabeled as a type +# problem, which would both mislead the caller and hide the real error. +_UNSUPPORTED_GROUPING_SIGNATURES = ( + "could not identify an equality operator", + "could not identify a comparison function", +) +# Postgres SQLSTATE 42883 (undefined_function) is what the missing equality / +# comparison operator behind GROUP BY / DISTINCT actually raises. +_UNSUPPORTED_GROUPING_SQLSTATES = frozenset({"42883"}) + + +def _is_unsupported_grouping_error(exc: BaseException) -> bool: + """True when ``exc`` says the database can't group/deduplicate a column type. + + Checks the driver SQLSTATE first (precise) and falls back to the message + text. Anything not matched is treated as an unrelated failure, so it + propagates with its own cause instead of being retried and mislabeled. + """ + for err in (exc, getattr(exc, "orig", None)): + if err is None: + continue + code = getattr(err, "sqlstate", None) or getattr(err, "pgcode", None) + if code in _UNSUPPORTED_GROUPING_SQLSTATES: + return True + text = str(exc).lower() + orig = getattr(exc, "orig", None) + if orig is not None: + text = f"{text} {str(orig).lower()}" + return any(sig in text for sig in _UNSUPPORTED_GROUPING_SIGNATURES) + +# Section-level budgeting for inspect_model output. +# columns/measures/aggregations/joins fall back to a names-only CSV when the +# caller drops the section from `sections`; samples/learnings are fully +# omitted (they have no natural "names" to list). +_INSPECT_SECTIONS_NAMES_ONLY = ("columns", "measures", "aggregations", "joins") +_INSPECT_SECTIONS_OMITTABLE = ("samples", "learnings") +_VALID_INSPECT_SECTIONS = _INSPECT_SECTIONS_NAMES_ONLY + _INSPECT_SECTIONS_OMITTABLE +_TRUNCATION_MARKER = " ... [truncated]" +# Placeholder rendered for an empty section / pruned markdown table. +_NONE_PLACEHOLDER = "_(none)_" + + +def _escape_md_cell(value: Any) -> str: + """Escape a value for inclusion in a markdown table cell. + + Pipes become ``\\|``, carriage returns and newlines collapse to a single + space, and ``None``/empty renders as an em-dash so empty columns stay + aligned in the rendered table. + """ + if value is None: + return "—" + s = str(value).replace("|", "\\|").replace("\r\n", " ").replace("\r", " ").replace("\n", " ").strip() + return s if s else "—" + + +def _md_code_span(value: Any) -> str: + """Wrap *value* in a CommonMark inline code span, safe for any content. + + The fence is chosen to be one backtick longer than the longest contiguous + run of backticks inside the value, so embedded backticks never break the + span. Per the CommonMark spec, a space is added inside the fence when the + content starts or ends with a backtick. + """ + text = str(value).replace("|", "\\|").replace("\r\n", " ").replace("\r", " ").replace("\n", " ").strip() + if not text: + return "` `" + # Find the longest run of consecutive backticks + max_run = 0 + run = 0 + for ch in text: + if ch == "`": + run += 1 + if run > max_run: + max_run = run + else: + run = 0 + fence = "`" * (max_run + 1) + # CommonMark: space padding needed when content starts or ends with backtick + if text.startswith("`") or text.endswith("`"): + return f"{fence} {text} {fence}" + return f"{fence}{text}{fence}" + + +def _cell_is_present(value: Any) -> bool: + """A cell is 'present' when it carries information: not None, and not an + empty (or whitespace-only) string. Every other value counts as present.""" + if value is None: + return False + if isinstance(value, str): + return bool(value.strip()) + return True + + +def _truncate_description(text: str | None, max_chars: int | None) -> str | None: + """Trim a description to ``max_chars`` and append the truncation marker. + + Returns the input unchanged when ``max_chars`` is ``None`` or the text is + already short enough. ``max_chars=0`` is allowed and yields just the + marker for any non-empty input. + """ + if text is None or max_chars is None: + return text + if len(text) <= max_chars: + return text + return text[:max_chars] + _TRUNCATION_MARKER + + +def _format_meta(meta: dict[str, Any] | None) -> str | None: + """Compact JSON for the ``inspect_model`` meta cell. + + Returns ``None`` when ``meta`` is ``None`` so ``_markdown_table``'s + all-empty-column pruner hides the meta column when no row has meta set. + """ + if meta is None: + return None + return json.dumps(meta, sort_keys=True, default=str) + + +def _resolve_inspect_sections( + sections: list[str] | None, +) -> tuple[list[str], list[str]]: + """Validate and normalise the ``sections`` argument for ``inspect_model``. + + Returns ``(resolved, unknown)`` where ``resolved`` is the list of valid + section names to render (preserving the canonical order, not the caller's + order) and ``unknown`` is the unrecognised entries (in caller order) for + the warning line. + + ``sections=None`` and ``sections=[]`` both resolve to all six valid + sections — that's the documented "I want everything" path. + + A non-empty list of *only* unknown names resolves to ``[]`` (not all six): + "all sections" is reserved for the explicit None/[] forms so a typo like + ``sections=["sample"]`` can't silently trigger the full expensive payload. + The footer warns about the unknown names and lists what was dropped, so + the caller can correct and re-call. + """ + if not sections: + return list(_VALID_INSPECT_SECTIONS), [] + valid_set = {s for s in sections if s in _VALID_INSPECT_SECTIONS} + unknown = [s for s in sections if s not in _VALID_INSPECT_SECTIONS] + # Canonical order so output is stable regardless of caller's order + resolved = [s for s in _VALID_INSPECT_SECTIONS if s in valid_set] + return resolved, unknown + + +def _render_inspect_footer( + *, + included: list[str], + names_only: list[str], + omitted: list[str], + unknown: list[str], +) -> str | None: + """Build the per-call truncation footer for ``inspect_model``. + + Returns ``None`` when there is nothing to report (no trimming, no + unknown names). Otherwise returns a quoted-markdown block. + """ + if not (names_only or omitted or unknown): + return None + lines: list[str] = [] + if unknown: + # repr() escapes newlines / quote chars so a caller-supplied value + # like "foo\n> evil" can't forge additional footer lines. + quoted = ", ".join(repr(u) for u in unknown) + lines.append( + f"> Warning: ignored unknown sections: {quoted}. " + f"Valid: {', '.join(_VALID_INSPECT_SECTIONS)}." + ) + if names_only or omitted: + lines.append(f"> Sections shown: {', '.join(included) if included else '(none)'}.") + if names_only: + lines.append(f"> Names-only: {', '.join(names_only)}.") + if omitted: + lines.append(f"> Omitted: {', '.join(omitted)}.") + lines.append("> Re-call inspect_model with `sections=[...]` to fetch.") + return "\n".join(lines) if lines else None + + +def _markdown_table(rows: list[dict[str, Any]], columns: list[str]) -> str: + """Render a list of row dicts as a GitHub-flavored markdown table. + + Columns with no present cell across every row are dropped automatically so + uninformative all-empty columns don't clutter the output. The degenerate + cases collapse: + + - ``rows`` is empty, or every column gets pruned → ``"_(none)_"``. + - Exactly one column survives pruning → a comma-separated, backtick-wrapped + list of its values, much denser than a one-column table. + + Otherwise a normal markdown table is produced over the surviving columns. + """ + if not rows: + return _NONE_PLACEHOLDER + + kept = [c for c in columns if any(_cell_is_present(r.get(c)) for r in rows)] + if not kept: + return _NONE_PLACEHOLDER + + if len(kept) == 1: + col = kept[0] + rendered = [] + for r in rows: + v = r.get(col) + if not _cell_is_present(v): + continue + rendered.append(_md_code_span(v)) + return ", ".join(rendered) + + header = "| " + " | ".join(kept) + " |" + sep = "| " + " | ".join("---" for _ in kept) + " |" + body = [ + "| " + " | ".join(_escape_md_cell(r.get(c)) for c in kept) + " |" + for r in rows + ] + return "\n".join([header, sep] + body) + + +def _render_column_type(column: Column) -> str: + """Render the ``type`` cell of the Columns table. + + Opaque (``UNKNOWN``) columns are still shown — SLayer stores and displays + them — but annotated with their raw database type and a marker saying they + can't be queried, so an agent doesn't try to group or aggregate on them. + """ + if not column.type.is_opaque: + return str(column.type) + detail = column.db_type or "unrecognized DB type" + return f"{column.type} ({detail}; not queryable)" + + +def _choose_sample_dims( + model: SlayerModel, +) -> tuple[list[dict[str, str]], set]: + """Pick up to two categorical (TEXT/BOOLEAN) non-hidden, non-PK columns to + group the sample by, so they aren't also aggregated as measures + (count_distinct(status) grouped by status is always 1).""" + dims: list[dict[str, str]] = [] + dim_names: set = set() + for c in model.columns: + if c.hidden or c.primary_key: + continue + # DEV-1361: TEXT/BOOLEAN are the categorical-shaped types. This filter + # also excludes opaque (UNKNOWN) columns, which cannot be GROUP BY'd. + if c.type not in (DataType.TEXT, DataType.BOOLEAN): + continue + dims.append({"name": c.name}) + dim_names.add(c.name) + if len(dims) >= 2: + break + return dims, dim_names + + +def _choose_sample_agg( + column: Column, + *, + measure_types: dict[str, str], +) -> str | None: + """Pick a sample aggregation for ``column``, or ``None`` to skip it. + + - With a restricted ``allowed_aggregations`` that excludes ``avg``: prefer + the first zero-arg-safe built-in (``_SAFE_SAMPLE_AGGS``); if none, fall + back to the first allowed entry (even if it needs extra context — an + intentional, tested behavior). Empty list → skip. + - Otherwise (``avg`` permitted): prefer ``avg`` for numeric columns, else + ``count_distinct`` (type inferred from ``measure_types`` — the lowercase + ``engine.get_column_types`` contract — or the column's own ``type``). + - Opaque (``UNKNOWN``) columns are always skipped: the DB has no equality + operator for their underlying type, so ``count_distinct``/``min``/``max`` + would fail and take the whole Data Profile query down with them. + """ + if column.type.is_opaque: + return None + allowed = column.allowed_aggregations + if allowed is not None and "avg" not in allowed: + if not allowed: + return None + safe = next((a for a in allowed if a in _SAFE_SAMPLE_AGGS), None) + return safe if safe else allowed[0] + inferred = measure_types.get(column.name) + inferred_norm = inferred.strip().lower() if isinstance(inferred, str) else None + if inferred_norm and inferred_norm != "number": + return "count_distinct" + if column.type not in (DataType.INT, DataType.DOUBLE): + return "count_distinct" + return "avg" + + +def _build_sample_query_args( + model: SlayerModel, + num_rows: int, + measure_types: dict[str, str] | None = None, +) -> dict[str, Any]: + """Build the ``SlayerQuery`` payload for ``inspect_model``'s sample data. + + First measure is always ``*:count``; then one aggregation per non-hidden, + non-primary-key, non-grouped column (see :func:`_choose_sample_agg`). + """ + measure_types = measure_types or {} + dims, dim_names = _choose_sample_dims(model) + + measures: list[dict[str, str]] = [{"formula": "*:count"}] + for c in model.columns: + if c.hidden or c.primary_key or c.name in dim_names: + continue + agg = _choose_sample_agg(c, measure_types=measure_types) + if agg is None: + continue + measures.append({"formula": f"{c.name}:{agg}"}) + + return { + "source_model": model.name, + "measures": measures, + "dimensions": dims, + "limit": num_rows, + } + + +def _strip_model_prefix( + columns: list[str], + data: list[dict[str, Any]], + model_name: str, +) -> tuple[list[str], list[dict[str, Any]]]: + """Drop the redundant ``{model_name}.`` prefix from sample-data column keys. + + Keeps the markdown table compact (the model name already appears in the + ``# Model: X`` heading above the sample). + """ + prefix = f"{model_name}." + + def _strip(key: str) -> str: + return key[len(prefix):] if key.startswith(prefix) else key + + new_cols = [_strip(c) for c in columns] + new_data = [{_strip(k): v for k, v in row.items()} for row in data] + return new_cols, new_data + + +async def _get_row_count( + model: SlayerModel, engine: SlayerQueryEngine, +) -> int | None: + """Return the total row count of ``model``'s underlying table, or ``None`` + on any failure. Uses a bare ``*:count`` query — the same aggregation a user + would run to ask for the count. + + The result column is read positionally (the query has exactly one field) + rather than by name, because SLayer's column-naming convention for the + bare-count-no-dimensions case is ``{model}._count`` rather than the + with-dimensions ``{model}.count``. + """ + try: + q = SlayerQuery.model_validate({ + "source_model": model.name, + "measures": [{"formula": "*:count"}], + }) + r = await engine.execute(query=q, data_source=model.data_source or None) + except Exception: + return None + if not r.data or not r.columns: + return None + val = r.data[0].get(r.columns[0]) + if val is None: + return None + try: + return int(val) + except (TypeError, ValueError): + return None + + +async def _collect_measure_profile( + model: SlayerModel, + engine: SlayerQueryEngine, +) -> dict[str, str]: + """Probe min/max for each non-hidden, non-primary-key NUMERIC/TEMPORAL + column via a single batched query. + + Returns ``{column_name: "min .. max"}`` for columns with data, or + ``{column_name: "all NULL"}`` for columns where both min and max are NULL. + Skips primary-key columns (their values are identifiers, not values to + profile). + + DEV-1480: text/boolean columns are excluded here so they are served + exclusively by the categorical dim profile (which populates both + ``Column.sampled`` and ``Column.sampled_values``). Mixing the two + paths for the same column would leave ``sampled_values=None`` while + ``sampled`` is set, which ``_is_sample_cached`` correctly treats as a + cache miss — leading to permanent re-profile every ``inspect_model`` + call. + """ + _NUMERIC_TEMPORAL = ( + DataType.INT, DataType.DOUBLE, DataType.DATE, DataType.TIMESTAMP, + ) + columns = [ + c for c in model.columns + if not c.hidden and not c.primary_key + and c.type in _NUMERIC_TEMPORAL + ] + if not columns: + return {} + + # Use ModelExtension with inline columns to bypass allowed_aggregations + ext_columns = [ + {"name": f"_slayer_probe_{c.name}", "sql": c.sql if c.sql else c.name, + "type": str(c.type)} + for c in columns + ] + measures_payload: list[dict[str, str]] = [] + for c in columns: + measures_payload.append({"formula": f"_slayer_probe_{c.name}:min"}) + measures_payload.append({"formula": f"_slayer_probe_{c.name}:max"}) + + try: + q = SlayerQuery.model_validate({ + "source_model": {"source_name": model.name, "columns": ext_columns}, + "measures": measures_payload, + }) + r = await engine.execute(query=q, data_source=model.data_source or None) + row = r.data[0] if r.data else {} + except Exception: + return {} + + result: dict[str, str] = {} + for c in columns: + mn = row.get(f"{model.name}._slayer_probe_{c.name}_min") + mx = row.get(f"{model.name}._slayer_probe_{c.name}_max") + if mn is None and mx is None: + result[c.name] = "all NULL" + else: + result[c.name] = f"{mn} .. {mx}" + return result + + +def _build_backing_query_info(model: SlayerModel) -> dict | None: + """Build the ``backing_query`` block for inspect_model output. + + Returns ``None`` for non-query-backed models. For query-backed models, + returns ``{variables, required_variables, stages}`` where: + + - ``variables``: ``model.query_variables`` (defaults). + - ``required_variables``: placeholder names that have no default. + - ``stages``: each stage dumped as a dict, ready for JSON output. + """ + if not model.source_queries: + return None + from slayer.core.query import extract_placeholder_names + + all_placeholders: set = set() + stage_dicts: list[dict] = [] + # A placeholder is "required" only if it has no default at any layer the + # engine consults: model.query_variables OR the stage's own variables. + defaulted: set = set(model.query_variables.keys()) + for q in model.source_queries: + all_placeholders |= extract_placeholder_names(q) + if q.variables: + defaulted |= set(q.variables.keys()) + stage_dicts.append(q.model_dump(mode="json", exclude_none=True)) + required = sorted(all_placeholders - defaulted) + return { + "variables": dict(model.query_variables), + "required_variables": required, + "stages": stage_dicts, + } + + +def _render_field_value(v: Any) -> str: + """Pick the most descriptive label out of a query-stage field value. + + Stage list entries can be plain strings, simple `{name}` dicts, formula + dicts, or wrapper dicts like `{"dimension": {"name": ...}}`. Try each + shape in priority order and fall back to `str(v)` if nothing matches. + """ + if not isinstance(v, dict): + return str(v) + name = v.get("name") + if name: + return str(name) + formula = v.get("formula") + if formula: + return str(formula) + inner = v.get("dimension") + if isinstance(inner, dict): + inner_name = inner.get("name") + if inner_name: + return str(inner_name) + return str(v) + + +def _render_stage_field_list(key: str, val: list) -> str: + """Render a stage's field list (dimensions / measures / filters / etc.).""" + if key == "filters": + return "; ".join(f"`{f}`" for f in val) + return "; ".join(_render_field_value(v) for v in val) + + +def _render_source_model(src: Any) -> str | None: + """Render a stage's ``source_model`` (str or ModelExtension dict).""" + if isinstance(src, str): + return f"- source_model: `{src}`" + if isinstance(src, dict): + sn = src.get("source_name") or src.get("name") + if sn: + return f"- source_model: `{sn}` (extension)" + return None + + +def _render_stage(i: int, stage: dict, total: int) -> list[str]: + """Render one stage's markdown lines.""" + title = stage.get("name") or ("final" if i == total else f"stage {i}") + out: list[str] = [f"\n**{i}. {title}**"] + src_line = _render_source_model(stage.get("source_model")) + if src_line: + out.append(src_line) + for key in ("dimensions", "time_dimensions", "measures", "filters"): + val = stage.get(key) + if not val: + continue + out.append(f"- {key}: {_render_stage_field_list(key, val)}") + return out + + +def _backing_query_markdown_section(info: dict) -> str: + """Format the ``backing_query`` info as a markdown section.""" + lines: list[str] = ["## Backing Query"] + stages = info.get("stages") or [] + for i, stage in enumerate(stages, start=1): + lines.extend(_render_stage(i, stage, len(stages))) + variables = info.get("variables") or {} + required = info.get("required_variables") or [] + if variables or required: + lines.append("\n**Variables:**") + for k, v in variables.items(): + lines.append(f"- `{k}`: default `{v}`") + for k in required: + lines.append(f"- `{k}`: required") + return "\n".join(lines) + + +def _source_type_for(model: SlayerModel) -> str: + """Classify a model's source mode for summary/inspect output.""" + if model.source_queries: + return "query" + if model.sql_table: + return "table" + if model.sql: + return "sql" + return "unknown" + + +# --------------------------------------------------------------------------- +# Model schema skeleton (DEV-1588 follow-up) +# --------------------------------------------------------------------------- + +def model_skeleton_fields( + *, model: SlayerModel, max_chars: int | None = None, +) -> dict[str, Any]: + """Cheap, DB-free structured skeleton of a model. + + Shape: ``{name, canonical_id, description, column_names, measure_names, + aggregation_names, joins_to}``. Used by ``inspect(model, compact=True)`` + JSON and by each entry of ``inspect(datasource, compact=False)``'s + ``models`` list (DEV-1588). ``description`` is truncated by ``max_chars``; + ``canonical_id`` falls back to the bare name when ``data_source`` is unset + (e.g. a not-yet-refined query-backed model). + """ + canonical_id = ( + f"{model.data_source}.{model.name}" if model.data_source else model.name + ) + return { + "name": model.name, + "canonical_id": canonical_id, + "description": _truncate_description(model.description, max_chars), + "column_names": [c.name for c in model.columns if not c.hidden], + "measure_names": [m.name for m in model.measures if m.name is not None], + "aggregation_names": [a.name for a in model.aggregations], + "joins_to": sorted({j.target_model for j in model.joins}), + } + + +def _skeleton_csv(names: list[str]) -> str: + return ", ".join(names) if names else _NONE_PLACEHOLDER + + +def render_model_skeleton( + *, model: SlayerModel, max_chars: int | None = None, +) -> str: + """Heading-less markdown schema skeleton (DB-free). + + An optional truncated description line (only when set), then four lines — + ``Columns`` / ``Measures`` / ``Aggregations`` / ``Joins to`` — always + present, each empty value rendered ``_(none)_`` (aligned to + ``models_summary(compact)``). The caller prepends the ``#``/``##`` heading. + """ + fields = model_skeleton_fields(model=model, max_chars=max_chars) + lines: list[str] = [] + if fields["description"]: + lines.append(fields["description"]) + lines.append(f"Columns: {_skeleton_csv(fields['column_names'])}") + lines.append(f"Measures: {_skeleton_csv(fields['measure_names'])}") + lines.append(f"Aggregations: {_skeleton_csv(fields['aggregation_names'])}") + lines.append(f"Joins to: {_skeleton_csv(fields['joins_to'])}") + return "\n".join(lines) + + +async def render_model_inspection( # NOSONAR(S3776) — faithful extraction of the inspect_model tool body; the section-gating + cache-miss + dual markdown/json render is intentionally a single linear pass + *, + model: SlayerModel, + storage: StorageBackend, + engine: SlayerQueryEngine | None, + num_rows: int = 3, + show_sql: bool = False, + format: str = "markdown", + sections: list[str] | None = None, + descriptions_max_chars: int | None = None, + compact: bool = True, +) -> str: + """Render a complete-yet-compact view of an already-resolved model. + + This is the verbatim body of the legacy ``inspect_model`` MCP tool, + extracted (DEV-1588) so the new ``inspect`` surfaces and the kept + ``inspect_model`` tool share one implementation. + + ``engine=None`` contract: when no engine is supplied, the DB-hitting + blocks (row count, live profiling, sample data) are skipped and the + rest of the render proceeds without raising. + """ + fmt = format.lower().strip() + if fmt not in ("markdown", "json"): + raise ValueError( + f"Invalid format '{format}' for inspect_model. Must be 'markdown' or 'json'." + ) + if descriptions_max_chars is not None and descriptions_max_chars < 0: + raise ValueError( + f"descriptions_max_chars must be >= 0, got {descriptions_max_chars}." + ) + + # Resolve section gating up front so we can short-circuit DB calls + # for parts the caller doesn't want. + included, unknown = _resolve_inspect_sections(sections) + included_set = set(included) + + # Categorise non-included sections into "names-only" (still listed, + # just collapsed to CSV) vs "fully omitted" (no heading at all). + names_only_sections = [ + s for s in _INSPECT_SECTIONS_NAMES_ONLY if s not in included_set + ] + omitted_sections = [ + s for s in _INSPECT_SECTIONS_OMITTABLE if s not in included_set + ] + + truncated_model_desc = _truncate_description(model.description, descriptions_max_chars) + out_sections: list[str] = [f"# Model: `{model.name}`"] + if truncated_model_desc: + out_sections.append(truncated_model_desc) + + # Metadata bullets (incl. row_count from a cheap *:count query) + meta: list[str] = [] + if model.data_source: + meta.append(f"- **data_source:** `{model.data_source}`") + if model.sql_table: + meta.append(f"- **sql_table:** `{model.sql_table}`") + if model.default_time_dimension: + meta.append( + f"- **default_time_dimension:** `{model.default_time_dimension}`" + ) + if model.hidden: + meta.append("- **hidden:** true") + if model.meta is not None: + meta.append(f"- **meta:** {json.dumps(model.meta, sort_keys=True, default=str)}") + row_count: int | None = None + if engine is not None: + row_count = await _get_row_count(model=model, engine=engine) + if row_count is not None: + meta.append(f"- **row_count:** {row_count:,}") + if meta: + out_sections.append("\n".join(meta)) + + if show_sql and model.sql: + out_sections.append(f"## SQL\n\n```sql\n{model.sql}\n```") + + if show_sql and model.filters: + filter_lines = "\n".join(f"- `{f}`" for f in model.filters) + out_sections.append(f"## Filters (model-level)\n\n{filter_lines}") + + # Backing-query section (query-backed models only). Structure is + # always-on (it's the model's identity for query-backed models, like + # `sql_table` is for table-backed); only the SQL cache is gated by + # show_sql. + backing_info = _build_backing_query_info(model) + if backing_info is not None: + out_sections.append(_backing_query_markdown_section(backing_info)) + if show_sql and model.backing_query_sql: + out_sections.append( + f"## Backing Query SQL\n\n```sql\n{model.backing_query_sql}\n```" + ) + + # ------------------------------------------------------------------ + # DB-hitting computations — skip when their consumers aren't requested + # (and when no engine is available, DEV-1588). + # ------------------------------------------------------------------ + profile_by_name: dict[str, str] = {} + profile_values_by_name: dict[str, list[str] | None] = {} + distinct_count_by_name: dict[str, int | None] = {} + measure_profile: dict[str, str] = {} + if engine is not None and "columns" in included_set: + uncached_columns: list[Column] = [] + for c in model.columns: + if c.hidden or c.primary_key: + continue + # DEV-1480 cache validity: categorical needs + # ``sampled_values`` to be present (the structured field + # is authoritative); numeric/temporal needs ``sampled``. + if _is_sample_cached(c): + if c.sampled is not None: + profile_by_name[c.name] = c.sampled + profile_values_by_name[c.name] = c.sampled_values + distinct_count_by_name[c.name] = c.distinct_count + else: + # v6-upgrade fallback: a categorical column may have + # legacy ``sampled`` text but no ``sampled_values`` + # yet. Surface the legacy text in case the live + # re-profile below fails for transient reasons — + # ``profile_column`` will overwrite on success. + if c.sampled is not None: + profile_by_name[c.name] = c.sampled + uncached_columns.append(c) + if uncached_columns: + # DEV-1480: split the live profile into two paths so we + # preserve the pre-DEV-1480 batching for numeric/temporal + # columns. Categorical columns fire a top-values query + # (and a secondary count_distinct on overflow) per column — + # there's no efficient cross-column batching for those. + # Numeric/temporal columns share one batched min/max query. + _CATEGORICAL = (DataType.TEXT, DataType.BOOLEAN) + _NUMERIC_TEMPORAL = ( + DataType.INT, DataType.DOUBLE, + DataType.DATE, DataType.TIMESTAMP, + ) + cat_uncached = [ + c for c in uncached_columns if c.type in _CATEGORICAL + ] + num_uncached = [ + c for c in uncached_columns if c.type in _NUMERIC_TEMPORAL + ] + + async def _persist_sample( + *, col_name: str, + sampled: str | None, + sampled_values: list[str] | None, + distinct_count: int | None, + ) -> None: + try: + await storage.update_column_sampled( + data_source=model.data_source, + model_name=model.name, + column_name=col_name, + sampled=sampled, + sampled_values=sampled_values, + distinct_count=distinct_count, + ) + except Exception as exc: + logger.warning( + "inspect_model: failed to persist sampled value for " + "%s.%s.%s: %s", + model.data_source, model.name, col_name, exc, + ) + + # Categorical: one top-values query per column (+ optional + # count_distinct on overflow). DEV-1516: delegates to the + # shared ``ensure_column_sample_fresh`` helper so the + # cache-miss + persist + render-dict-population pattern is + # owned by exactly one place (also used by the search + # service's post-fusion column-hit hook). + for col in cat_uncached: + refreshed = await ensure_column_sample_fresh( + model=model, column=col, + engine=engine, storage=storage, + ) + # On any failure (profile raise / None / persist raise) + # the helper returns the INPUT column. Legacy ``sampled`` + # text on the input still feeds the markdown cell — the + # pre-pass above has already populated + # ``profile_by_name[col.name]`` from ``col.sampled``, + # so we only overwrite when we actually have something + # fresher (avoids clobbering the legacy fallback with + # ``None`` and producing an empty cell). + if refreshed.sampled is not None: + profile_by_name[col.name] = refreshed.sampled + profile_values_by_name[col.name] = refreshed.sampled_values + distinct_count_by_name[col.name] = refreshed.distinct_count + + # Numeric/temporal: one batched min/max query for all of + # them at once (restores the pre-DEV-1480 batching for + # wide models). + if num_uncached: + num_entries = await _profile_numeric_temporal_columns( + model=model, columns=num_uncached, engine=engine, + ) + for col in num_uncached: + entry = num_entries.get(col.name) + if entry is None: + continue + if entry.min_value is None and entry.max_value is None: + continue + sampled_text = f"{entry.min_value} .. {entry.max_value}" + profile_by_name[col.name] = sampled_text + # Numeric/temporal columns carry no structured list + # and no distinct_count per the DEV-1480 contract. + profile_values_by_name[col.name] = None + distinct_count_by_name[col.name] = None + await _persist_sample( + col_name=col.name, + sampled=sampled_text, + sampled_values=None, + distinct_count=None, + ) + measure_profile = await _collect_measure_profile(model=model, engine=engine) + # Persist any measure-side (numeric/temporal) profile + # values to ``Column.sampled`` so subsequent + # ``inspect_model`` / search calls hit the cache + # instead of re-running the live profile query. + for col in uncached_columns: + sampled_value = measure_profile.get(col.name) + if sampled_value is None or col.name in profile_by_name: + # Either no measure-side value for this column + # (already covered by dim profile above), or + # the dim profile already won the cache slot. + continue + profile_by_name[col.name] = sampled_value + try: + await storage.update_column_sampled( + data_source=model.data_source, + model_name=model.name, + column_name=col.name, + sampled=sampled_value, + sampled_values=None, + distinct_count=None, + ) + except Exception as exc: + logger.warning( + "inspect_model: failed to persist sampled value for " + "%s.%s.%s: %s", + model.data_source, model.name, col.name, exc, + ) + + # ``measure_types`` informs the sample query's choice of avg vs + # count_distinct. Only needed when ``samples`` is in the included set. + measure_types: dict[str, str] = {} + if engine is not None and "samples" in included_set: + measure_types = await engine.get_column_types( + model_name=model.name, + data_source=model.data_source or None, + ) + + # ------------------------------------------------------------------ + # Columns section + # ------------------------------------------------------------------ + visible_columns = [c for c in model.columns if not c.hidden] + if "columns" in included_set: + col_rows: list[dict[str, Any]] = [] + for c in visible_columns: + aggs = ", ".join(c.allowed_aggregations) if c.allowed_aggregations else "all" + # DEV-1480: key-presence check (not ``or`` truthiness) so an + # all-NULL categorical column's ``sampled=""`` doesn't + # silently fall through to the measure_profile fallback's + # ``"all NULL"`` text. + if c.name in profile_by_name: + sampled_cell = profile_by_name[c.name] + else: + sampled_cell = measure_profile.get(c.name) + col_rows.append({ + "name": c.name, + "type": _render_column_type(c), + "primary_key": "yes" if c.primary_key else "", + "sql": c.sql if c.sql else c.name, + "allowed_aggregations": aggs, + "filter": c.filter, + "label": c.label, + "description": _truncate_description(c.description, descriptions_max_chars), + "meta": _format_meta(c.meta), + "sampled": sampled_cell, + }) + col_columns = [ + "name", "type", "primary_key", "sql", "allowed_aggregations", + "filter", "label", "description", "meta", "sampled", + ] + if not show_sql: + col_columns = [c for c in col_columns if c not in ("sql", "filter")] + out_sections.append( + f"## Columns ({len(col_rows)})\n\n" + + _markdown_table(rows=col_rows, columns=col_columns) + ) + elif visible_columns: + csv = ", ".join(_md_code_span(c.name) for c in visible_columns) + out_sections.append( + f"## Columns ({len(visible_columns)} — names only)\n\n{csv}" + ) + + # ------------------------------------------------------------------ + # Measures section + # ------------------------------------------------------------------ + if "measures" in included_set: + measure_rows: list[dict[str, Any]] = [] + for mm in model.measures: + measure_rows.append({ + "name": mm.name, + "formula": mm.formula, + "label": mm.label, + "description": _truncate_description(mm.description, descriptions_max_chars), + "meta": _format_meta(mm.meta), + }) + out_sections.append( + f"## Measures ({len(measure_rows)})\n\n" + + _markdown_table( + rows=measure_rows, + columns=["name", "formula", "label", "description", "meta"], + ) + ) + elif model.measures: + csv = ", ".join(_md_code_span(mm.name) for mm in model.measures) + out_sections.append( + f"## Measures ({len(model.measures)} — names only)\n\n{csv}" + ) + + # ------------------------------------------------------------------ + # Aggregations section + # ------------------------------------------------------------------ + if "aggregations" in included_set: + if model.aggregations: + agg_rows: list[dict[str, Any]] = [] + for a in model.aggregations: + if a.params: + if show_sql: + params = "; ".join(f"{p.name}={p.sql}" for p in a.params) + else: + params = ", ".join(p.name for p in a.params) + else: + params = None + agg_rows.append({ + "name": a.name, + "formula": a.formula or "(built-in override)", + "params": params, + "description": _truncate_description( + a.description, descriptions_max_chars, + ), + "meta": _format_meta(a.meta), + }) + agg_columns = ["name", "formula", "params", "description", "meta"] + if not show_sql: + agg_columns = [c for c in agg_columns if c != "formula"] + out_sections.append( + f"## Aggregations ({len(agg_rows)})\n\n" + + _markdown_table(rows=agg_rows, columns=agg_columns) + ) + elif model.aggregations: + csv = ", ".join(_md_code_span(a.name) for a in model.aggregations) + out_sections.append( + f"## Aggregations ({len(model.aggregations)} — names only)\n\n{csv}" + ) + + # ------------------------------------------------------------------ + # Joins section + # ------------------------------------------------------------------ + if "joins" in included_set: + join_rows: list[dict[str, Any]] = [] + for j in model.joins: + pairs = "; ".join(f"{src} = {tgt}" for src, tgt in j.join_pairs) + join_rows.append({ + "target_model": j.target_model, + "join_pairs": pairs, + }) + out_sections.append( + f"## Joins ({len(join_rows)})\n\n" + + _markdown_table( + rows=join_rows, + columns=["target_model", "join_pairs"], + ) + ) + elif model.joins: + csv = ", ".join(_md_code_span(j.target_model) for j in model.joins) + out_sections.append( + f"## Joins ({len(model.joins)} — names only)\n\n{csv}" + ) + + # ------------------------------------------------------------------ + # Sample data (fully omitted when not in sections / no engine) + # ------------------------------------------------------------------ + sample_sql: str | None = None + sample_data: dict[str, Any] | None = None + sample_error: str | None = None + # Set when the profile fell back to a row count so JSON callers can tell a + # reduced profile from a complete one (the markdown note is not machine + # readable). + sample_reduced_reason: str | None = None + if engine is not None and "samples" in included_set: + query_args = _build_sample_query_args( + model=model, num_rows=num_rows, measure_types=measure_types, + ) + # A column whose underlying DB type has no equality operator (point, + # json, xml — all coarsed to TEXT before opaque classification existed, + # so still undetectable on older models) makes the grouped/DISTINCT + # profile fail. Retry once with a row-count-only profile so one exotic + # column can't sink the section — but ONLY for that specific failure. + # Any other error (permission, connection, validation, syntax) must + # surface with its own cause instead of being relabeled as a type + # problem; the outer handler still degrades the section gracefully. + note = "" + try: + try: + sample_query = SlayerQuery.model_validate(query_args) + sample_result = await engine.execute( + query=sample_query, data_source=model.data_source or None + ) + except Exception as exc: + if not _is_unsupported_grouping_error(exc): + raise + minimal_args = dict(query_args) + minimal_args["measures"] = [{"formula": "*:count"}] + minimal_args["dimensions"] = [] + sample_query = SlayerQuery.model_validate(minimal_args) + try: + sample_result = await engine.execute( + query=sample_query, data_source=model.data_source or None + ) + except Exception: + # The reduced profile failed too — report the original + # cause, not this second failure. + raise exc + sample_reduced_reason = ( + "at least one column's type does not support the " + "grouping/DISTINCT this profile uses" + ) + note = f"\n\n_Reduced to a row count: {sample_reduced_reason}._" + sample_sql = sample_result.sql + cols, data = _strip_model_prefix( + columns=sample_result.columns, + data=sample_result.data, + model_name=model.name, + ) + sample_data = {"columns": cols, "rows": data} + sample_result.columns = cols + sample_result.data = data + sample_section = f"## Data Profile\n\n{sample_result.to_markdown()}{note}" + if show_sql and sample_sql: + sample_section = ( + f"## Data Profile SQL\n\n```sql\n{sample_sql}\n```\n\n" + + sample_section + ) + out_sections.append(sample_section) + except Exception as e: + if isinstance(e, (sa.exc.OperationalError, sa.exc.DatabaseError)): + err = _friendly_db_error(e) + else: + err = str(e) + sample_error = err + sample_section = f"## Data Profile\n\n_Error fetching data profile: {err}_" + if show_sql and sample_sql: + sample_section = ( + f"## Data Profile SQL\n\n```sql\n{sample_sql}\n```\n\n" + + sample_section + ) + out_sections.append(sample_section) + + # ------------------------------------------------------------------ + # Learnings (DEV-1357 v2) — surfaces only memories where ``query`` is + # ``None``; query-bearing memories are recall-only. Auto-pruned when + # no learning-shaped memory matches. + # ------------------------------------------------------------------ + relevant_learnings: list[Any] = [] + wanted: list[str] = [] + if "learnings" in included_set: + ds = model.data_source + wanted = [f"{ds}.{model.name}"] + wanted.extend(f"{ds}.{model.name}.{c.name}" for c in model.columns) + wanted.extend( + f"{ds}.{model.name}.{m.name}" + for m in model.measures + if m.name is not None + ) + wanted.extend( + f"{ds}.{model.name}.{a.name}" for a in model.aggregations + ) + candidates = await storage.list_memories(entities=wanted) + relevant_learnings = [m for m in candidates if m.query is None] + if relevant_learnings: + lines = [f"## Learnings ({len(relevant_learnings)})", ""] + for memory in relevant_learnings: + matched = sorted(set(wanted) & set(memory.entities)) + matched_md = ", ".join(f"`{e}`" for e in matched) + # DEV-1549: compact mode emits Memory.description (or + # the first-paragraph fallback computed from learning); + # verbose dumps the full learning body. + if compact: + body = ( + memory.description + if memory.description + else compact_description_from_learning(memory.learning) + ) + else: + body = memory.learning + lines.append( + f"- **M{memory.id}** ({matched_md}): {body}" + ) + out_sections.append("\n".join(lines)) + + # ------------------------------------------------------------------ + # Per-call truncation footer (only when something was trimmed or an + # unknown section name was supplied). + # ------------------------------------------------------------------ + footer = _render_inspect_footer( + included=included, + names_only=names_only_sections, + omitted=omitted_sections, + unknown=unknown, + ) + + if fmt == "json": + payload: dict[str, Any] = { + "model_name": model.name, + "description": truncated_model_desc, + "data_source": model.data_source, + "source_type": _source_type_for(model), + } + if show_sql: + payload["sql_table"] = model.sql_table + payload["sql"] = model.sql + if backing_info is not None: + payload["backing_query"] = backing_info + if show_sql and model.backing_query_sql: + payload["backing_query_sql"] = model.backing_query_sql + payload["default_time_dimension"] = model.default_time_dimension + payload["hidden"] = model.hidden + payload["meta"] = model.meta + payload["row_count"] = row_count + if show_sql: + payload["filters"] = model.filters + + # Columns + if "columns" in included_set: + col_payloads: list[dict[str, Any]] = [] + for c in visible_columns: + # DEV-1480 key-presence (not ``or`` truthiness) so empty + # string ``sampled=""`` (all-NULL categorical) survives. + if c.name in profile_by_name: + sampled_cell = profile_by_name[c.name] + else: + sampled_cell = measure_profile.get(c.name) + col_payloads.append({ + "name": c.name, + "type": str(c.type), + # Opaque columns only: the raw DB type plus an explicit + # not-queryable marker (see _render_column_type). + **( + {"db_type": c.db_type, "queryable": False} + if c.type.is_opaque else {} + ), + "primary_key": c.primary_key, + **({"sql": c.sql} if show_sql else {}), + "allowed_aggregations": c.allowed_aggregations, + **({"filter": c.filter} if show_sql else {}), + "label": c.label, + "description": _truncate_description( + c.description, descriptions_max_chars, + ), + "meta": c.meta, + "sampled": sampled_cell, + # DEV-1480: structured top-50 list + true cardinality, + # surfaced only in the JSON shape (the markdown table + # text format is unchanged per the issue). + "sampled_values": profile_values_by_name.get(c.name), + "distinct_count": distinct_count_by_name.get(c.name), + }) + payload["columns"] = col_payloads + elif visible_columns: + payload["columns_names"] = [c.name for c in visible_columns] + + # Measures + if "measures" in included_set: + payload["measures"] = [ + { + "name": mm.name, + "formula": mm.formula, + "label": mm.label, + "description": _truncate_description( + mm.description, descriptions_max_chars, + ), + "meta": mm.meta, + } + for mm in model.measures + ] + elif model.measures: + payload["measures_names"] = [mm.name for mm in model.measures] + + # Aggregations + if "aggregations" in included_set: + payload["aggregations"] = [ + { + "name": a.name, + **({"formula": a.formula} if show_sql else {}), + "params": [ + ({"name": p.name, "sql": p.sql} if show_sql else {"name": p.name}) + for p in (a.params or []) + ], + "description": _truncate_description( + a.description, descriptions_max_chars, + ), + "meta": a.meta, + } + for a in model.aggregations + ] + elif model.aggregations: + payload["aggregations_names"] = [a.name for a in model.aggregations] + + # Joins + if "joins" in included_set: + payload["joins"] = [ + { + "target_model": j.target_model, + "join_pairs": j.join_pairs, + } + for j in model.joins + ] + elif model.joins: + payload["joins_names"] = [j.target_model for j in model.joins] + + # Samples + if "samples" in included_set: + payload["sample_data"] = sample_data + payload["sample_data_error"] = sample_error + payload["sample_data_reduced"] = sample_reduced_reason is not None + payload["sample_data_reduced_reason"] = sample_reduced_reason + if show_sql and sample_sql: + payload["sample_sql"] = sample_sql + + # Learnings (DEV-1357 v2) — Memory carries ``learning``, + # not ``body``; reading ``.body`` here would AttributeError + # the moment a memory matches and the caller asked for JSON + # output. + if "learnings" in included_set and relevant_learnings: + # DEV-1549: compact JSON Learnings drops ``learning`` and + # surfaces ``description`` (Memory.description or the + # first-paragraph fallback). Verbose JSON keeps the full + # learning key as today. + if compact: + payload["learnings"] = [ + { + "id": memory.id, + "description": ( + memory.description + if memory.description + else compact_description_from_learning( + memory.learning, + ) + ), + "matched_entities": sorted( + set(wanted) & set(memory.entities) + ), + } + for memory in relevant_learnings + ] + else: + payload["learnings"] = [ + { + "id": memory.id, + "learning": memory.learning, + "matched_entities": sorted( + set(wanted) & set(memory.entities) + ), + } + for memory in relevant_learnings + ] + + # Top-level gating-state arrays (only when non-empty) + if names_only_sections: + payload["names_only_sections"] = names_only_sections + if omitted_sections: + payload["omitted_sections"] = omitted_sections + if unknown: + payload["unknown_sections"] = unknown + + return json.dumps(payload, indent=2, default=str) + + if footer: + out_sections.append(footer) + return "\n\n".join(out_sections) diff --git a/slayer/inspect/service.py b/slayer/inspect/service.py new file mode 100644 index 00000000..e3e526a0 --- /dev/null +++ b/slayer/inspect/service.py @@ -0,0 +1,1014 @@ +"""DEV-1588: shared single-entity inspection service. + +``InspectService.inspect(reference, entity_type, ...)`` returns the +rendered detail for EXACTLY one entity — no RRF / fusion / cypher / +bundled memories. ``entity_type`` is required and disambiguates the +3-part canonical collision (a name shared by, e.g., a column and an +aggregation). + +Exposed on four surfaces: the MCP ``inspect`` tool, REST ``POST +/inspect``, CLI ``slayer inspect``, and ``SlayerClient.inspect`` / +``inspect_sync``. +""" + +from __future__ import annotations + +import json +from typing import Any, NamedTuple + +from slayer.core.errors import ( + AmbiguousModelError, + EntityResolutionError, + MemoryNotFoundError, +) +from slayer.core.models import SlayerModel +from slayer.engine.profiling import ensure_column_sample_fresh +from slayer.inspect.collection_render import ( + BLOCK_SEP, + datasource_skeleton_fields, + render_datasource_list, + render_model_oneliner_index, + render_models_summary, +) +from slayer.inspect.model_render import ( + _TRUNCATION_MARKER, + _truncate_description, + model_skeleton_fields, + render_model_inspection, + render_model_skeleton, +) +from slayer.memories.resolver import resolve_entity +from slayer.search.render import ( + collect_model_entity_pairs, + compact_description_from_learning, + render_memory_text, +) +from slayer.storage.base import StorageBackend + +try: # SlayerQueryEngine is only needed for the model sample-data path. + from slayer.engine.query_engine import SlayerQueryEngine +except Exception: # pragma: no cover - engine import always succeeds in-repo + SlayerQueryEngine = None # type: ignore[assignment, misc] + +VALID_ENTITY_TYPES = { + "datasource", "model", "column", "measure", "aggregation", "memory", +} +_VALID_FORMATS = {"markdown", "json"} + +# Kinds for which the leaf-lookup canonical form is the 3-part id. +_LEAF_KINDS = {"column", "measure", "aggregation"} + +# DEV-1667: kinds for which a null/empty reference renders the collection. +_COLLECTION_KINDS = {"model", "datasource"} +_COLLECTION_UNSUPPORTED = ( + "Collection view (null reference) is only supported for entity_type " + "'model' or 'datasource'." +) + +_DESCRIPTION_PREFIX = "Description: " + +# DEV-1612: markdown batch blocks are separated by this rule so per-id block +# boundaries are unambiguous even when a body carries its own ``##`` headings +# (e.g. a datasource compact=False render lists models under ``## `model```). +_BATCH_BLOCK_SEP = "\n\n---\n\n" + + +class _OneResult(NamedTuple): + """The outcome of inspecting a SINGLE id (DEV-1612). + + ``serialized`` is the exact per-kind output the single-id path returns + byte-for-byte (markdown body or JSON string). ``canonical_id`` is the + resolved id when available (used for the markdown batch header). + ``is_error`` is set explicitly on every error branch — never inferred + from ``canonical_id`` or from whether ``serialized`` parses as JSON. + """ + + canonical_id: str | None + is_error: bool + serialized: str + + +def _warn_line(*, arg: str, entity_type: str) -> str: + """A model-only-arg warning message (plain text, no ``> Warning:`` + prefix — that is added at markdown render time).""" + return ( + f"'{arg}' is ignored for entity_type " + f"'{entity_type}' (only applies to models)." + ) + + +class InspectService: + """Shared single-entity point-lookup core (DEV-1588).""" + + def __init__( + self, + *, + storage: StorageBackend, + engine: SlayerQueryEngine | None = None, + ) -> None: + self._storage = storage + self._engine = engine + + # ------------------------------------------------------------------ + # Public API + # ------------------------------------------------------------------ + + async def inspect( + self, + *, + reference: str | list[str] | None, + entity_type: str, + compact: bool = True, + format: str = "markdown", + num_rows: int = 3, + show_sql: bool = False, + sections: list[str] | None = None, + descriptions_max_chars: int | None = None, + ) -> str: + """Inspect EXACTLY one entity, a homogeneous-kind BATCH when + ``reference`` is a list (DEV-1612), or — DEV-1667 — the whole COLLECTION + at a kind when ``reference`` is ``None`` / ``[]``. + + A ``str`` keeps its single-id behaviour and output byte-for-byte. A + non-empty ``list`` returns one rendered block per id, in input order, + each echoing its resolved canonical id; per-id resolution errors are + isolated. ``None`` or ``[]`` (identical) renders the collection at + ``entity_type`` — supported only for ``model`` / ``datasource``. + """ + # 1. Global argument validation (raise ValueError). Applies once to + # the whole call for the str, list, and collection shapes. + if entity_type not in VALID_ENTITY_TYPES: + raise ValueError( + f"Invalid entity_type '{entity_type}'. Must be one of: " + f"{', '.join(sorted(VALID_ENTITY_TYPES))}." + ) + fmt = format.lower().strip() + if fmt not in _VALID_FORMATS: + raise ValueError( + f"Invalid format '{format}'. Must be 'markdown' or 'json'." + ) + if descriptions_max_chars is not None and descriptions_max_chars < 0: + raise ValueError( + f"descriptions_max_chars must be >= 0, got " + f"{descriptions_max_chars}." + ) + + # 2. Collection detection (DEV-1667): ``None`` OR ``[]`` → collection. + # ``[]`` is normalized to ``None`` here, so it produces the SAME + # behaviour as ``None`` (the old empty-list raise is removed). + if reference is None or reference == []: + if entity_type not in _COLLECTION_KINDS: + raise ValueError(_COLLECTION_UNSUPPORTED) + if entity_type == "model": + return await self._inspect_collection_model( + compact=compact, fmt=fmt, + descriptions_max_chars=descriptions_max_chars, + ) + return await self._inspect_collection_datasource( + compact=compact, fmt=fmt, + descriptions_max_chars=descriptions_max_chars, + ) + + # Non-collection: str single / non-empty list batch. + if isinstance(reference, list): + if any(not isinstance(ref, str) for ref in reference): + raise ValueError("reference list must contain only strings.") + elif not isinstance(reference, str): + raise ValueError("reference must be a string or a list of strings.") + + # 4. Model-only-arg warnings (skip entirely for model entity_type). + # These are global-arg warnings, so the SAME base list seeds every + # id in a batch; each id appends its own resolver warnings to a copy. + warnings: list[str] = self._model_only_arg_warnings( + entity_type=entity_type, + num_rows=num_rows, + show_sql=show_sql, + sections=sections, + ) + + # 5. Single id → byte-for-byte single output. List → batch framing. + if isinstance(reference, str): + result = await self._inspect_one( + reference=reference, entity_type=entity_type, compact=compact, + fmt=fmt, num_rows=num_rows, show_sql=show_sql, + sections=sections, descriptions_max_chars=descriptions_max_chars, + warnings=warnings, + ) + return result.serialized + return await self._inspect_batch( + references=reference, entity_type=entity_type, compact=compact, + fmt=fmt, num_rows=num_rows, show_sql=show_sql, sections=sections, + descriptions_max_chars=descriptions_max_chars, warnings=warnings, + ) + + async def _inspect_one( # NOSONAR(S3776) — single linear dispatch over the six entity kinds; per-kind helpers would obscure the shared output-assembly flow + self, + *, + reference: str, + entity_type: str, + compact: bool, + fmt: str, + num_rows: int, + show_sql: bool, + sections: list[str] | None, + descriptions_max_chars: int | None, + warnings: list[str], + ) -> _OneResult: + """Dispatch a SINGLE id to its per-kind helper. Returns the structured + :class:`_OneResult` so the batch path can frame success vs error + explicitly (no inference).""" + if entity_type == "model": + return await self._inspect_model( + reference=reference, compact=compact, fmt=fmt, + num_rows=num_rows, show_sql=show_sql, sections=sections, + descriptions_max_chars=descriptions_max_chars, + warnings=warnings, + ) + if entity_type == "memory": + return await self._inspect_memory( + reference=reference, compact=compact, fmt=fmt, + descriptions_max_chars=descriptions_max_chars, + warnings=warnings, + ) + if entity_type == "datasource": + return await self._inspect_datasource( + reference=reference, compact=compact, fmt=fmt, + descriptions_max_chars=descriptions_max_chars, + warnings=warnings, + ) + # column / measure / aggregation + return await self._inspect_leaf( + reference=reference, entity_type=entity_type, compact=compact, + fmt=fmt, descriptions_max_chars=descriptions_max_chars, + warnings=warnings, + ) + + async def _inspect_batch( + self, + *, + references: list[str], + entity_type: str, + compact: bool, + fmt: str, + num_rows: int, + show_sql: bool, + sections: list[str] | None, + descriptions_max_chars: int | None, + warnings: list[str], + ) -> str: + """DEV-1612: render a homogeneous-kind batch. Order preserved, no + dedup, per-id errors isolated.""" + results: list[tuple[str, _OneResult]] = [] + for ref in references: + r = await self._inspect_one( + reference=ref, entity_type=entity_type, compact=compact, + fmt=fmt, num_rows=num_rows, show_sql=show_sql, + sections=sections, + descriptions_max_chars=descriptions_max_chars, + warnings=warnings, + ) + results.append((ref, r)) + + if fmt == "json": + elements: list[Any] = [] + for ref, r in results: + if r.is_error: + # Error elements are objects keyed by the INPUT ref so a + # batch JSON array stays homogeneous (objects only). + elements.append({"reference": ref, "error": r.serialized}) + else: + # ``serialized`` is our own freshly-emitted JSON object → + # round-trips safely; default=str re-applies at the array + # layer for any non-JSON-native value. + elements.append(json.loads(r.serialized)) + return json.dumps(elements, default=str) + + # Markdown: one ``##
`` block per id, joined by the rule. + blocks: list[str] = [] + for ref, r in results: + header = ref if r.is_error else (r.canonical_id or ref) + blocks.append(f"## {header}\n{r.serialized}") + return _BATCH_BLOCK_SEP.join(blocks) + + # ------------------------------------------------------------------ + # Warnings + # ------------------------------------------------------------------ + + @staticmethod + def _model_only_arg_warnings( + *, + entity_type: str, + num_rows: int, + show_sql: bool, + sections: list[str] | None, + ) -> list[str]: + if entity_type == "model": + return [] + out: list[str] = [] + # num_rows: warns for all non-model kinds when != default. + if num_rows != 3: + out.append(_warn_line(arg="num_rows", entity_type=entity_type)) + # sections: warns for all non-model kinds when set. + if sections: + out.append(_warn_line(arg="sections", entity_type=entity_type)) + # show_sql: no-op (no warn) for leaf kinds; warns for ds / memory. + if show_sql and entity_type in ("datasource", "memory"): + out.append(_warn_line(arg="show_sql", entity_type=entity_type)) + # descriptions_max_chars applies to every kind (never warns). + return out + + # ------------------------------------------------------------------ + # Output assembly helpers + # ------------------------------------------------------------------ + + @staticmethod + def _truncate_description_field( + text: str, max_chars: int | None, + ) -> str: + """Truncate only the ``Description: `` line(s) of a rendered + entity blob — NOT the whole render. Mirrors ``inspect_model``'s + per-field truncation semantics so the structural lines (Type, SQL, + sample values, …) of a column/measure/aggregation/datasource render + survive a small ``descriptions_max_chars``.""" + if max_chars is None: + return text + out: list[str] = [] + for line in text.split("\n"): + if line.startswith(_DESCRIPTION_PREFIX): + value = line[len(_DESCRIPTION_PREFIX):] + if len(value) > max_chars: + line = ( + _DESCRIPTION_PREFIX + + value[:max_chars] + + _TRUNCATION_MARKER + ) + out.append(line) + return "\n".join(out) + + @staticmethod + def _markdown_with_warnings(body: str, warnings: list[str]) -> str: + if not warnings: + return body + warn_block = "\n".join(f"> Warning: {w}" for w in warnings) + if body: + return f"{body}\n\n{warn_block}" + return warn_block + + # ------------------------------------------------------------------ + # Collection views (DEV-1667) — null / [] reference + # ------------------------------------------------------------------ + + async def _load_visible_models(self, ds_name: str) -> list[SlayerModel]: + """Hidden-filtered, name-sorted models for one datasource (matches + ``models_summary``). Individual load failures skip that model.""" + models: list[SlayerModel] = [] + for name in await self._storage.list_models(data_source=ds_name): + try: + m = await self._storage.get_model(name, data_source=ds_name) + except Exception: # noqa: BLE001 — one bad model must not sink the DS + continue + if m is not None and not m.hidden: + models.append(m) + models.sort(key=lambda m: m.name) + return models + + async def _inspect_collection_model( + self, + *, + compact: bool, + fmt: str, + descriptions_max_chars: int | None, + ) -> str: + ds_names = await self._storage.list_datasources() + if not ds_names: + if fmt == "json": + return json.dumps({ + "entity_type": "model", + "collection": True, + "datasources": [], + "warnings": [], + }, indent=2) + return "No models found." + + # Build per-DS groups; ``models is None`` marks an invalid-config DS. + groups: list[tuple[str, list[SlayerModel] | None]] = [] + for ds in ds_names: + try: + await self._storage.get_datasource(ds) + except Exception: # noqa: BLE001 — invalid config: mark + continue + groups.append((ds, None)) + continue + groups.append((ds, await self._load_visible_models(ds))) + + if compact: + return render_model_oneliner_index( + groups=groups, fmt=fmt, warnings=[], + ) + # compact=False: full models_summary block per DS. + if fmt == "json": + return self._collection_model_verbose_json( + groups=groups, descriptions_max_chars=descriptions_max_chars, + ) + return self._collection_model_verbose_markdown( + groups=groups, descriptions_max_chars=descriptions_max_chars, + ) + + @staticmethod + def _collection_model_verbose_json( + *, + groups: list[tuple[str, list[SlayerModel] | None]], + descriptions_max_chars: int | None, + ) -> str: + entries: list[dict[str, Any]] = [] + for ds, models in groups: + if models is None: + entries.append( + {"data_source": ds, "error": "invalid config", "models": []} + ) + else: + # render_models_summary now returns valid JSON for empty + # datasources too (model_count 0, models []) — one consistent + # ``datasource_name`` shape for every non-error entry. + entries.append(json.loads(render_models_summary( + datasource_name=ds, models=models, fmt="json", + compact=False, descriptions_max_chars=descriptions_max_chars, + ))) + return json.dumps({ + "entity_type": "model", + "collection": True, + "datasources": entries, + "warnings": [], + }, indent=2, default=str) + + @staticmethod + def _collection_model_verbose_markdown( + *, + groups: list[tuple[str, list[SlayerModel] | None]], + descriptions_max_chars: int | None, + ) -> str: + blocks: list[str] = [] + for ds, models in groups: + if models is None: + blocks.append(f"Datasource '{ds}' has an invalid config.") + continue + blocks.append(render_models_summary( + datasource_name=ds, models=models, fmt="markdown", + compact=False, descriptions_max_chars=descriptions_max_chars, + )) + return BLOCK_SEP.join(blocks) + + async def _inspect_collection_datasource( + self, + *, + compact: bool, + fmt: str, + descriptions_max_chars: int | None, + ) -> str: + ds_names = await self._storage.list_datasources() + if not ds_names: + # Empty-state parity with the model collection: the list renderer + # emits the "No datasources configured" message (markdown) / the + # empty envelope (json) for both compact modes. + return render_datasource_list(pairs=[], fmt=fmt, warnings=[]) + + if compact: + pairs: list[tuple[str, str | None]] = [] + for name in ds_names: + try: + cfg = await self._storage.get_datasource(name) + pairs.append((name, cfg.type if cfg is not None else "unknown")) + except Exception: # noqa: BLE001 — invalid config sentinel + pairs.append((name, None)) + return render_datasource_list(pairs=pairs, fmt=fmt, warnings=[]) + + # compact=False: per-DS name + description + model skeleton. + if fmt == "json": + return await self._collection_datasource_verbose_json( + ds_names=ds_names, descriptions_max_chars=descriptions_max_chars, + ) + return await self._collection_datasource_verbose_markdown( + ds_names=ds_names, descriptions_max_chars=descriptions_max_chars, + ) + + async def _collection_datasource_verbose_json( + self, + *, + ds_names: list[str], + descriptions_max_chars: int | None, + ) -> str: + entries: list[dict[str, Any]] = [] + for ds in ds_names: + try: + cfg = await self._storage.get_datasource(ds) + except Exception: # noqa: BLE001 — invalid config: error entry + entries.append({"name": ds, "error": "invalid config"}) + continue + entries.append(datasource_skeleton_fields( + name=ds, + description=cfg.description if cfg is not None else None, + models=await self._load_visible_models(ds), + descriptions_max_chars=descriptions_max_chars, + )) + return json.dumps({ + "entity_type": "datasource", + "collection": True, + "datasources": entries, + "warnings": [], + }, indent=2, default=str) + + async def _collection_datasource_verbose_markdown( + self, + *, + ds_names: list[str], + descriptions_max_chars: int | None, + ) -> str: + blocks: list[str] = [] + for ds in ds_names: + try: + await self._storage.get_datasource(ds) + except Exception: # noqa: BLE001 — invalid config: error block + blocks.append(f"Datasource: {ds}\nERROR: invalid config") + continue + blocks.append(await self._render_datasource( + ds_name=ds, compact=False, fmt="markdown", + descriptions_max_chars=descriptions_max_chars, warnings=[], + )) + return BLOCK_SEP.join(blocks) + + # ------------------------------------------------------------------ + # Memory + # ------------------------------------------------------------------ + + async def _inspect_memory( + self, + *, + reference: str, + compact: bool, + fmt: str, + descriptions_max_chars: int | None, + warnings: list[str], + ) -> _OneResult: + if not reference.startswith("memory:"): + return _OneResult(None, True, ( + f"entity_type='memory' requires a 'memory:' reference; " + f"got '{reference}'. Memory references must start with " + f"'memory:'." + )) + memory_id = reference[len("memory:"):] + try: + mem = await self._storage.get_memory(memory_id) + except MemoryNotFoundError: + return _OneResult(None, True, ( + f"No memory with id '{memory_id}' found " + f"(reference '{reference}')." + )) + + description = ( + mem.description + if mem.description + else compact_description_from_learning(mem.learning) + ) + description = _truncate_description( + text=description, max_chars=descriptions_max_chars, + ) + if compact: + full_text = "" + else: + mem_for_render = mem + if descriptions_max_chars is not None: + # Truncate the learning body only — keep the tagged-entities + # line intact (mirrors per-field truncation elsewhere). + truncated_learning = _truncate_description( + text=mem.learning, max_chars=descriptions_max_chars, + ) or "" + mem_for_render = mem.model_copy( + update={"learning": truncated_learning}, + ) + full_text = render_memory_text(memory=mem_for_render) + + canonical = f"memory:{mem.id}" + if fmt == "json": + payload = { + "canonical_id": canonical, + "entity_type": "memory", + "description": description, + } + # ``text`` present iff non-empty (DEV-1588 follow-up): compact mode + # leaves ``full_text`` empty, so the key is omitted. + if full_text: + payload["text"] = full_text + payload["warnings"] = warnings + return _OneResult(canonical, False, json.dumps(payload)) + body = description if compact else full_text + return _OneResult( + canonical, False, + self._markdown_with_warnings(body or "", warnings), + ) + + # ------------------------------------------------------------------ + # Datasource + # ------------------------------------------------------------------ + + async def _resolve_single_canonical( + self, *, reference: str, warnings: list[str], + ) -> tuple[str, list[str]] | _OneResult: + """Resolve ``reference`` to its single canonical form for the + datasource / leaf paths. Returns ``(canonical, warnings)`` on success, + or an error ``_OneResult`` (the resolver raised, or the reference did + not resolve to exactly one canonical id).""" + try: + res = await resolve_entity( + reference, storage=self._storage, source_model=None, + ) + except (EntityResolutionError, AmbiguousModelError) as exc: + # AmbiguousModelError (a SlayerError sibling, NOT a subclass of + # EntityResolutionError) escapes resolve_entity's bare-name model + # leg; surface its message instead of crashing the surface. + return _OneResult(None, True, str(exc)) + warnings = warnings + list(res.warnings) + if len(res.canonical_forms) != 1: + return _OneResult(None, True, ( + f"Internal error: reference '{reference}' resolved to " + f"{len(res.canonical_forms)} canonical forms; expected 1." + )) + return res.canonical_forms[0], warnings + + async def _inspect_datasource( + self, + *, + reference: str, + compact: bool, + fmt: str, + descriptions_max_chars: int | None, + warnings: list[str], + ) -> _OneResult: + resolved = await self._resolve_single_canonical( + reference=reference, warnings=warnings, + ) + if isinstance(resolved, _OneResult): + return resolved + canonical, warnings = resolved + + known = set(await self._storage.list_datasources()) + ds_name: str | None = None + if "." not in canonical and canonical in known: + ds_name = canonical + elif reference in known: + ds_name = reference + if ds_name is None: + return _OneResult(None, True, ( + f"'{reference}' is not a datasource (resolved to " + f"'{canonical}'). Known datasources: " + f"{', '.join(sorted(known))}." + )) + body = await self._render_datasource( + ds_name=ds_name, compact=compact, fmt=fmt, + descriptions_max_chars=descriptions_max_chars, warnings=warnings, + ) + return _OneResult(ds_name, False, body) + + async def _render_datasource( + self, + *, + ds_name: str, + compact: bool, + fmt: str, + descriptions_max_chars: int | None, + warnings: list[str], + ) -> str: + cfg = await self._storage.get_datasource(ds_name) + description = cfg.description if cfg is not None else None + trunc_desc = _truncate_description( + text=description, max_chars=descriptions_max_chars, + ) + + # compact=True: datasource description only (DB-free); ``text`` is + # omitted entirely (present iff non-empty, DEV-1588 follow-up). + if compact: + if fmt == "json": + return json.dumps({ + "canonical_id": ds_name, + "entity_type": "datasource", + "description": trunc_desc, + "warnings": warnings, + }) + return self._markdown_with_warnings(trunc_desc or "", warnings) + + # compact=False: a per-model schema skeleton for each VISIBLE model, + # sorted by name (matches models_summary), still DB-free. Uses the + # shared resilient loader so one malformed model file is skipped rather + # than sinking the whole render (parity with the collection JSON path). + models = await self._load_visible_models(ds_name) + + if fmt == "json": + return json.dumps({ + "canonical_id": ds_name, + "entity_type": "datasource", + "description": trunc_desc, + "models": [ + model_skeleton_fields( + model=m, max_chars=descriptions_max_chars, + ) + for m in models + ], + "warnings": warnings, + }, indent=2, default=str) + + md_lines: list[str] = [f"Datasource: {ds_name}"] + if trunc_desc: + md_lines.append(f"Description: {trunc_desc}") + for m in models: + md_lines.append(f"\n## `{m.name}`") + md_lines.append( + render_model_skeleton( + model=m, max_chars=descriptions_max_chars, + ) + ) + return self._markdown_with_warnings("\n".join(md_lines), warnings) + + # ------------------------------------------------------------------ + # Model + # ------------------------------------------------------------------ + + async def _inspect_model( + self, + *, + reference: str, + compact: bool, + fmt: str, + num_rows: int, + show_sql: bool, + sections: list[str] | None, + descriptions_max_chars: int | None, + warnings: list[str], + ) -> _OneResult: + try: + canonical = await self._resolve_model_canonical(reference) + except AmbiguousModelError as exc: + # A bare model name present in ≥2 datasources with no priority + # winner — surface the actionable message, not an uncaught raise. + return _OneResult(None, True, str(exc)) + if canonical is None: + return _OneResult(None, True, ( + f"'{reference}' does not resolve to a model. Pass a " + f"datasource-qualified model id (e.g. '.') or a " + f"bare model name." + )) + ds_name, model_name = canonical.split(".", 1) + model = await self._storage.get_model(model_name, data_source=ds_name) + if model is None: + return _OneResult(None, True, ( + f"Model '{canonical}' not found " + f"(reference '{reference}')." + )) + if compact: + # Schema skeleton (DEV-1588 follow-up): column / measure / + # aggregation NAMES + join targets, zero DB calls — short-circuit + # before the full renderer (which can run row-count / profiling / + # sample-data DB work). compact=False returns the full model view + # (sections / samples / SQL). + if fmt == "json": + payload = dict(model_skeleton_fields( + model=model, max_chars=descriptions_max_chars, + )) + # The resolved id is authoritative (echoes the normalized + # reference, like every other inspect JSON shape). + payload["canonical_id"] = canonical + payload["entity_type"] = "model" + payload["warnings"] = warnings + return _OneResult( + canonical, False, json.dumps(payload, indent=2, default=str), + ) + body = render_model_skeleton( + model=model, max_chars=descriptions_max_chars, + ) + return _OneResult(canonical, False, self._markdown_with_warnings( + f"# `{model.name}`\n{body}", warnings, + )) + rendered = await render_model_inspection( + model=model, + storage=self._storage, + engine=self._engine, + num_rows=num_rows, + show_sql=show_sql, + format=fmt, + sections=sections, + descriptions_max_chars=descriptions_max_chars, + compact=compact, + ) + if fmt == "json": + payload = json.loads(rendered) + payload["canonical_id"] = canonical + payload["warnings"] = warnings + return _OneResult( + canonical, False, json.dumps(payload, indent=2, default=str), + ) + return _OneResult( + canonical, False, self._markdown_with_warnings(rendered, warnings), + ) + + async def _resolve_model_canonical(self, reference: str) -> str | None: + """Resolve ``reference`` to a 2-part ``.`` canonical id, + applying the Case-D entity_type=model override (a resolver that + picked a datasource for a name that is also a model).""" + try: + res = await resolve_entity( + reference, storage=self._storage, source_model=None, + ) + except AmbiguousModelError: + # Bare ambiguous model name: let the caller surface the message. + raise + except EntityResolutionError: + res = None + if res is not None and len(res.canonical_forms) == 1: + canonical = res.canonical_forms[0] + if canonical.count(".") == 1: + return canonical + # Case D fallback: a *bare* name the resolver mapped to a datasource + # (1-seg) that is ALSO a model elsewhere. Only the reference itself is + # a valid model-identity candidate — never the last segment of a + # dotted reference. A dotted reference that resolved to a leaf (or + # didn't resolve to a 2-seg model) is a kind mismatch, not a model: + # collapsing `ds.orders.amount` to `amount` could return an unrelated + # model named `amount`. + try: + ident = await self._storage.resolve_model_identity(reference) + except AmbiguousModelError: + raise + except Exception: + ident = None + if ident is not None: + return f"{ident[0]}.{ident[1]}" + return None + + # ------------------------------------------------------------------ + # Leaf (column / measure / aggregation) + # ------------------------------------------------------------------ + + async def _inspect_leaf( + self, + *, + reference: str, + entity_type: str, + compact: bool, + fmt: str, + descriptions_max_chars: int | None, + warnings: list[str], + ) -> _OneResult: + resolved = await self._resolve_single_canonical( + reference=reference, warnings=warnings, + ) + if isinstance(resolved, _OneResult): + return resolved + canonical, warnings = resolved + if canonical.count(".") != 2: + return _OneResult(canonical, True, ( + f"'{reference}' resolved to '{canonical}', which is not a " + f"{entity_type} (expected a '..' id)." + )) + ds_name, model_name, leaf = canonical.split(".", 2) + model = await self._storage.get_model(model_name, data_source=ds_name) + if model is None: + return _OneResult(None, True, ( + f"Model '{ds_name}.{model_name}' not found " + f"(reference '{reference}')." + )) + + # DEV-1615: lazily back-fill the column's sample values before render. + model = await self._maybe_refresh_leaf_sample( + model=model, entity_type=entity_type, compact=compact, leaf=leaf, + ) + + pairs = collect_model_entity_pairs(model=model, include_hidden=True) + matches = [ + p for p in pairs + if p.canonical_id == canonical and p.kind == entity_type + ] + if len(matches) == 1: + body = self._render_leaf_entry( + entry=matches[0], canonical=canonical, entity_type=entity_type, + compact=compact, fmt=fmt, + descriptions_max_chars=descriptions_max_chars, + warnings=warnings, + ) + return _OneResult(canonical, False, body) + return _OneResult(canonical, True, self._leaf_lookup_error( + canonical=canonical, entity_type=entity_type, leaf=leaf, + ds_name=ds_name, model_name=model_name, pairs=pairs, + match_count=len(matches), + )) + + async def _maybe_refresh_leaf_sample( + self, + *, + model: SlayerModel, + entity_type: str, + compact: bool, + leaf: str, + ) -> SlayerModel: + """DEV-1615: lazily back-fill a column's missing/stale sample values on + read — same shared helper + cache-aware semantics inspect_model / + search use — so this point-lookup is no longer a regression vs the + tools it replaced. + + Gated to ``entity_type="column"`` (measures / aggregations have no + sample concept) and to ``compact=False``: the compact leaf render is + description-only and never shows "Sample values:", so refreshing there + would add a profiling DB query to a deliberately cheap lookup. Engine- + guarded (no-op without an engine, like search's hook). Hidden columns + are rendered but never back-filled — the helper's ``_is_sample_cached`` + treats hidden/PK as cached (system-wide convention, parity with + inspect_model). + + Returns the input model unchanged when no refresh applies; otherwise a + ``model_copy`` with the refreshed column substituted, so the render + (``collect_model_entity_pairs``) reflects the fresh sample with no + change to the downstream render logic. + """ + if entity_type != "column" or compact or self._engine is None: + return model + col = model.get_column(leaf) + if col is None: + return model + refreshed = await ensure_column_sample_fresh( + model=model, column=col, + engine=self._engine, storage=self._storage, + ) + if refreshed is col: + return model + return model.model_copy(update={ + "columns": [ + refreshed if c.name == col.name else c + for c in model.columns + ], + }) + + def _render_leaf_entry( + self, + *, + entry, + canonical: str, + entity_type: str, + compact: bool, + fmt: str, + descriptions_max_chars: int | None, + warnings: list[str], + ) -> str: + trunc_desc = _truncate_description( + text=entry.description, max_chars=descriptions_max_chars, + ) + full_text = self._truncate_description_field( + text=entry.text, max_chars=descriptions_max_chars, + ) + # Measures / aggregations carry no verbose sample data — their full text + # (formula / params / label / type) is essential, so include it even in + # compact mode. Columns stay gated: their text can carry sampled values + # (a DB read / verbose output). + show_full = bool(full_text) and ( + not compact or entity_type in {"measure", "aggregation"} + ) + if fmt == "json": + payload = { + "canonical_id": canonical, + "entity_type": entity_type, + "description": trunc_desc, + } + # ``text`` present iff non-empty (DEV-1588 follow-up): included in + # full mode, and for measures/aggregations in compact mode too. + if show_full: + payload["text"] = full_text + payload["warnings"] = warnings + return json.dumps(payload) + body = full_text if show_full else (trunc_desc or "") + return self._markdown_with_warnings(body, warnings) + + @staticmethod + def _leaf_lookup_error( + *, + canonical: str, + entity_type: str, + leaf: str, + ds_name: str, + model_name: str, + pairs, + match_count: int, + ) -> str: + if match_count > 1: + return ( + f"'{canonical}' matches {match_count} {entity_type}s on " + f"model '{ds_name}.{model_name}'; cannot uniquely identify " + f"which to inspect." + ) + # Zero matches of the requested kind. Name the available kind(s). + other_kinds = sorted({ + p.kind for p in pairs if p.canonical_id == canonical + }) + if other_kinds: + return ( + f"'{canonical}' is a {', '.join(other_kinds)}, not a " + f"{entity_type}. Available here: {', '.join(other_kinds)}." + ) + return ( + f"No {entity_type} '{leaf}' found on model " + f"'{ds_name}.{model_name}'." + ) diff --git a/slayer/mcp/server.py b/slayer/mcp/server.py index 9e33fa8a..1fa09ee6 100644 --- a/slayer/mcp/server.py +++ b/slayer/mcp/server.py @@ -2,15 +2,15 @@ import json import logging -from typing import Any, Dict, List, Optional, Tuple +from typing import Any import sqlalchemy as sa -from slayer.core.enums import DataType from slayer.core.errors import ( AmbiguousModelError, EntityResolutionError, MemoryNotFoundError, + SlayerError, ) from slayer.core.models import ( Aggregation, @@ -21,15 +21,31 @@ SlayerModel, ) from slayer.core.query import ModelExtension, SlayerQuery +from slayer.core.recommend import render_recommendation_markdown from slayer.engine.ingestion import _friendly_db_error -from slayer.engine.profiling import ( - _is_sample_cached, - _profile_numeric_temporal_columns, - handle_edit_refresh, - profile_column, -) +from slayer.engine.profiling import handle_edit_refresh from slayer.engine.query_engine import SlayerQueryEngine, SlayerResponse -from slayer.help import TOPIC_SUMMARY_LINE, render_help +from slayer.memories.help_seed import seed_help_memories +from slayer.inspect.model_render import ( # noqa: F401 — re-exported for backward-compat (tests + other modules import these names from slayer.mcp.server) + _build_sample_query_args, + _collect_measure_profile, + _escape_md_cell, + _format_meta, + _get_row_count, + _markdown_table, + _md_code_span, + _render_inspect_footer, + _resolve_inspect_sections, + _source_type_for, + _strip_model_prefix, + _truncate_description, + render_model_inspection, +) +from slayer.inspect.collection_render import ( + render_datasource_list, + render_models_summary, +) +from slayer.inspect.service import InspectService from slayer.memories.service import MemoryService from slayer.search.service import SearchService from slayer.storage.base import StorageBackend @@ -39,20 +55,6 @@ VALID_DIMENSION_TYPES = {"string", "time", "date", "boolean", "number"} _UNSET = object() # Sentinel to distinguish "not provided" from "explicitly set to None" -# Aggregations that are safe for sample-data extraction: zero extra args, -# no time-column context needed. -_SAFE_SAMPLE_AGGS = frozenset({"avg", "sum", "min", "max", "count", "count_distinct", "median"}) - -# Section-level budgeting for inspect_model output. -# columns/measures/aggregations/joins fall back to a names-only CSV when the -# caller drops the section from `sections`; reachable_fields/samples are fully -# omitted (they have no natural "names" to list). -_INSPECT_SECTIONS_NAMES_ONLY = ("columns", "measures", "aggregations", "joins") -_INSPECT_SECTIONS_OMITTABLE = ("reachable_fields", "samples", "learnings") -_VALID_INSPECT_SECTIONS = _INSPECT_SECTIONS_NAMES_ONLY + _INSPECT_SECTIONS_OMITTABLE -_TRUNCATION_MARKER = " ... [truncated]" -_MAX_REACHABLE_FIELDS_DEPTH = 20 - def _ambiguous_with_mcp_hint(exc: AmbiguousModelError) -> str: """Render an ``AmbiguousModelError`` for the MCP surface. @@ -70,11 +72,11 @@ def _ambiguous_with_mcp_hint(exc: AmbiguousModelError) -> str: def _test_connection(ds: DatasourceConfig) -> tuple[bool, str]: """Test a datasource connection. Returns (success, message).""" try: - conn_str = ds.resolve_env_vars().get_connection_string() - engine = sa.create_engine(conn_str) + from slayer.sql import engine_factory + engine = engine_factory.get_engine(ds.resolve_env_vars()) with engine.connect() as conn: conn.execute(sa.text("SELECT 1")) - engine.dispose() + # Cached engine — engine_factory owns lifecycle; don't dispose. return True, "Connection successful." except Exception as e: return False, _friendly_db_error(e) @@ -83,30 +85,28 @@ def _test_connection(ds: DatasourceConfig) -> tuple[bool, str]: def _get_schemas(ds: DatasourceConfig) -> list[str]: """List available schemas for a datasource.""" try: - conn_str = ds.resolve_env_vars().get_connection_string() - engine = sa.create_engine(conn_str) + from slayer.sql import engine_factory + engine = engine_factory.get_engine(ds.resolve_env_vars()) inspector = sa.inspect(engine) schemas = inspector.get_schema_names() - engine.dispose() return schemas except Exception: return [] def _fetch_tables( - ds: DatasourceConfig, schema_name: Optional[str] = None, -) -> Tuple[Optional[List[str]], Optional[str]]: + ds: DatasourceConfig, schema_name: str | None = None, +) -> tuple[list[str] | None, str | None]: """Inspect a datasource's table names. Returns ``(tables, None)`` on success or ``(None, friendly_error_message)`` on failure. ``schema_name=None`` uses the dialect's default schema. """ try: - conn_str = ds.resolve_env_vars().get_connection_string() - sa_engine = sa.create_engine(conn_str) + from slayer.sql import engine_factory + sa_engine = engine_factory.get_engine(ds.resolve_env_vars()) inspector = sa.inspect(sa_engine) tables = inspector.get_table_names(schema=schema_name) - sa_engine.dispose() return sorted(tables), None except Exception as e: if isinstance(e, (sa.exc.OperationalError, sa.exc.DatabaseError)): @@ -114,110 +114,6 @@ def _fetch_tables( return None, str(e) -def _escape_md_cell(value: Any) -> str: - """Escape a value for inclusion in a markdown table cell. - - Pipes become ``\\|``, carriage returns and newlines collapse to a single - space, and ``None``/empty renders as an em-dash so empty columns stay - aligned in the rendered table. - """ - if value is None: - return "—" - s = str(value).replace("|", "\\|").replace("\r\n", " ").replace("\r", " ").replace("\n", " ").strip() - return s if s else "—" - - -def _md_code_span(value: Any) -> str: - """Wrap *value* in a CommonMark inline code span, safe for any content. - - The fence is chosen to be one backtick longer than the longest contiguous - run of backticks inside the value, so embedded backticks never break the - span. Per the CommonMark spec, a space is added inside the fence when the - content starts or ends with a backtick. - """ - text = str(value).replace("|", "\\|").replace("\r\n", " ").replace("\r", " ").replace("\n", " ").strip() - if not text: - return "` `" - # Find the longest run of consecutive backticks - max_run = 0 - run = 0 - for ch in text: - if ch == "`": - run += 1 - if run > max_run: - max_run = run - else: - run = 0 - fence = "`" * (max_run + 1) - # CommonMark: space padding needed when content starts or ends with backtick - if text.startswith("`") or text.endswith("`"): - return f"{fence} {text} {fence}" - return f"{fence}{text}{fence}" - - -def _cell_is_present(value: Any) -> bool: - """A cell is 'present' when it carries information: not None, and not an - empty (or whitespace-only) string. Every other value counts as present.""" - if value is None: - return False - if isinstance(value, str): - return bool(value.strip()) - return True - - -def _truncate_description(text: Optional[str], max_chars: Optional[int]) -> Optional[str]: - """Trim a description to ``max_chars`` and append the truncation marker. - - Returns the input unchanged when ``max_chars`` is ``None`` or the text is - already short enough. ``max_chars=0`` is allowed and yields just the - marker for any non-empty input. - """ - if text is None or max_chars is None: - return text - if len(text) <= max_chars: - return text - return text[:max_chars] + _TRUNCATION_MARKER - - -def _format_meta(meta: Optional[Dict[str, Any]]) -> Optional[str]: - """Compact JSON for the ``inspect_model`` meta cell. - - Returns ``None`` when ``meta`` is ``None`` so ``_markdown_table``'s - all-empty-column pruner hides the meta column when no row has meta set. - """ - if meta is None: - return None - return json.dumps(meta, sort_keys=True, default=str) - - -def _resolve_inspect_sections( - sections: Optional[List[str]], -) -> Tuple[List[str], List[str]]: - """Validate and normalise the ``sections`` argument for ``inspect_model``. - - Returns ``(resolved, unknown)`` where ``resolved`` is the list of valid - section names to render (preserving the canonical order, not the caller's - order) and ``unknown`` is the unrecognised entries (in caller order) for - the warning line. - - ``sections=None`` and ``sections=[]`` both resolve to all six valid - sections — that's the documented "I want everything" path. - - A non-empty list of *only* unknown names resolves to ``[]`` (not all six): - "all sections" is reserved for the explicit None/[] forms so a typo like - ``sections=["sample"]`` can't silently trigger the full expensive payload. - The footer warns about the unknown names and lists what was dropped, so - the caller can correct and re-call. - """ - if not sections: - return list(_VALID_INSPECT_SECTIONS), [] - valid_set = {s for s in sections if s in _VALID_INSPECT_SECTIONS} - unknown = [s for s in sections if s not in _VALID_INSPECT_SECTIONS] - # Canonical order so output is stable regardless of caller's order - resolved = [s for s in _VALID_INSPECT_SECTIONS if s in valid_set] - return resolved, unknown - - def _empty_ingest_message(*, schema_name: str, ds: DatasourceConfig) -> str: schema_label = f" in schema '{schema_name}'" if schema_name else "" lines = [f"No tables found{schema_label}."] @@ -230,7 +126,7 @@ def _empty_ingest_message(*, schema_name: str, ds: DatasourceConfig) -> str: return "\n".join(lines) -def _render_new_models_section(new_models: List[Any]) -> List[str]: +def _render_new_models_section(new_models: list[Any]) -> list[str]: if not new_models: return [] lines = [f"Created {len(new_models)} new model(s):"] @@ -241,7 +137,7 @@ def _render_new_models_section(new_models: List[Any]) -> List[str]: return lines -def _render_updated_section(updated: List[Any]) -> List[str]: +def _render_updated_section(updated: list[Any]) -> list[str]: if not updated: return [] lines = [f"Updated {len(updated)} existing model(s):"] @@ -255,7 +151,7 @@ def _render_updated_section(updated: List[Any]) -> List[str]: return lines -def _render_unchanged_section(unchanged: List[Any]) -> List[str]: +def _render_unchanged_section(unchanged: list[Any]) -> list[str]: if not unchanged: return [] return [ @@ -264,7 +160,7 @@ def _render_unchanged_section(unchanged: List[Any]) -> List[str]: ] -def _render_drift_section(to_delete: List[Any]) -> List[str]: +def _render_drift_section(to_delete: list[Any]) -> list[str]: if not to_delete: return [] out = ["", "Pending drift (run validate_models / apply manually):"] @@ -272,7 +168,7 @@ def _render_drift_section(to_delete: List[Any]) -> List[str]: return out -def _render_errors_section(errors: List[Any]) -> List[str]: +def _render_errors_section(errors: list[Any]) -> list[str]: if not errors: return [] out = ["", f"Errors ({len(errors)}):"] @@ -308,7 +204,7 @@ def _render_ingest_result( if not a.created and not a.new_columns and not a.new_joins ] - lines: List[str] = [] + lines: list[str] = [] lines.extend(_render_new_models_section(new_models)) lines.extend(_render_updated_section(updated)) lines.extend(_render_unchanged_section(unchanged)) @@ -319,441 +215,6 @@ def _render_ingest_result( return "\n".join(lines) -def _render_inspect_footer( - *, - included: List[str], - names_only: List[str], - omitted: List[str], - unknown: List[str], -) -> Optional[str]: - """Build the per-call truncation footer for ``inspect_model``. - - Returns ``None`` when there is nothing to report (no trimming, no - unknown names). Otherwise returns a quoted-markdown block. - """ - if not (names_only or omitted or unknown): - return None - lines: List[str] = [] - if unknown: - # repr() escapes newlines / quote chars so a caller-supplied value - # like "foo\n> evil" can't forge additional footer lines. - quoted = ", ".join(repr(u) for u in unknown) - lines.append( - f"> Warning: ignored unknown sections: {quoted}. " - f"Valid: {', '.join(_VALID_INSPECT_SECTIONS)}." - ) - if names_only or omitted: - lines.append(f"> Sections shown: {', '.join(included) if included else '(none)'}.") - if names_only: - lines.append(f"> Names-only: {', '.join(names_only)}.") - if omitted: - lines.append(f"> Omitted: {', '.join(omitted)}.") - lines.append("> Re-call inspect_model with `sections=[...]` to fetch.") - return "\n".join(lines) if lines else None - - -def _markdown_table(rows: List[Dict[str, Any]], columns: List[str]) -> str: - """Render a list of row dicts as a GitHub-flavored markdown table. - - Columns with no present cell across every row are dropped automatically so - uninformative all-empty columns don't clutter the output. The degenerate - cases collapse: - - - ``rows`` is empty, or every column gets pruned → ``"_(none)_"``. - - Exactly one column survives pruning → a comma-separated, backtick-wrapped - list of its values, much denser than a one-column table. - - Otherwise a normal markdown table is produced over the surviving columns. - """ - if not rows: - return "_(none)_" - - kept = [c for c in columns if any(_cell_is_present(r.get(c)) for r in rows)] - if not kept: - return "_(none)_" - - if len(kept) == 1: - col = kept[0] - rendered = [] - for r in rows: - v = r.get(col) - if not _cell_is_present(v): - continue - rendered.append(_md_code_span(v)) - return ", ".join(rendered) - - header = "| " + " | ".join(kept) + " |" - sep = "| " + " | ".join("---" for _ in kept) + " |" - body = [ - "| " + " | ".join(_escape_md_cell(r.get(c)) for c in kept) + " |" - for r in rows - ] - return "\n".join([header, sep] + body) - - -def _build_sample_query_args( - model: SlayerModel, - num_rows: int, - measure_types: Optional[Dict[str, str]] = None, -) -> Dict[str, Any]: - """Build the ``SlayerQuery`` payload for ``inspect_model``'s sample data. - - - First entry is always ``*:count``. - - For each non-hidden, non-primary-key column: - - If ``allowed_aggregations`` is restricted and doesn't include ``avg``, - use the first safe entry (or skip if empty). - - Else (avg is permitted): prefer ``avg``, but fall back to - ``count_distinct`` for non-numeric columns (inferred from - ``measure_types`` or the column's own ``type``). - - Groups by up to two non-primary-key, non-hidden columns of non-numeric - type so the sample shows variation without exploding table width. - """ - measure_types = measure_types or {} - - # Pick up to two categorical columns to group by first, so we don't also - # aggregate them as measures (count_distinct(status) grouped by status is - # always 1, which isn't useful sample data). - dims: List[Dict[str, str]] = [] - dim_names: set[str] = set() - for c in model.columns: - if c.hidden or c.primary_key: - continue - # DEV-1361: TEXT/BOOLEAN are the categorical-shaped types. - if c.type not in (DataType.TEXT, DataType.BOOLEAN): - continue - dims.append({"name": c.name}) - dim_names.add(c.name) - if len(dims) >= 2: - break - - measures: List[Dict[str, str]] = [{"formula": "*:count"}] - for c in model.columns: - if c.hidden or c.primary_key or c.name in dim_names: - continue - allowed = c.allowed_aggregations - if allowed is not None and "avg" not in allowed: - if not allowed: - continue - safe = next((a for a in allowed if a in _SAFE_SAMPLE_AGGS), None) - agg = safe if safe else allowed[0] - else: - # DEV-1361: numeric columns (INT/DOUBLE) are avg-able; everything - # else falls back to count_distinct. ``measure_types`` comes from - # ``engine.get_column_types`` whose contract is the lowercase - # category set {"number","string","time","boolean"}; normalize - # before comparing in case the contract widens later. - inferred = measure_types.get(c.name) - inferred_norm = inferred.strip().lower() if isinstance(inferred, str) else None - if inferred_norm and inferred_norm != "number": - agg = "count_distinct" - elif c.type not in (DataType.INT, DataType.DOUBLE): - agg = "count_distinct" - else: - agg = "avg" - measures.append({"formula": f"{c.name}:{agg}"}) - - return { - "source_model": model.name, - "measures": measures, - "dimensions": dims, - "limit": num_rows, - } - - -def _strip_model_prefix( - columns: List[str], - data: List[Dict[str, Any]], - model_name: str, -) -> Tuple[List[str], List[Dict[str, Any]]]: - """Drop the redundant ``{model_name}.`` prefix from sample-data column keys. - - Keeps the markdown table compact (the model name already appears in the - ``# Model: X`` heading above the sample). - """ - prefix = f"{model_name}." - - def _strip(key: str) -> str: - return key[len(prefix):] if key.startswith(prefix) else key - - new_cols = [_strip(c) for c in columns] - new_data = [{_strip(k): v for k, v in row.items()} for row in data] - return new_cols, new_data - - -async def _get_row_count( - model: SlayerModel, engine: SlayerQueryEngine, -) -> Optional[int]: - """Return the total row count of ``model``'s underlying table, or ``None`` - on any failure. Uses a bare ``*:count`` query — the same aggregation a user - would run to ask for the count. - - The result column is read positionally (the query has exactly one field) - rather than by name, because SLayer's column-naming convention for the - bare-count-no-dimensions case is ``{model}._count`` rather than the - with-dimensions ``{model}.count``. - """ - try: - q = SlayerQuery.model_validate({ - "source_model": model.name, - "measures": [{"formula": "*:count"}], - }) - r = await engine.execute(query=q, data_source=model.data_source or None) - except Exception: - return None - if not r.data or not r.columns: - return None - val = r.data[0].get(r.columns[0]) - if val is None: - return None - try: - return int(val) - except (TypeError, ValueError): - return None - - -async def _collect_measure_profile( - model: SlayerModel, - engine: SlayerQueryEngine, -) -> Dict[str, str]: - """Probe min/max for each non-hidden, non-primary-key NUMERIC/TEMPORAL - column via a single batched query. - - Returns ``{column_name: "min .. max"}`` for columns with data, or - ``{column_name: "all NULL"}`` for columns where both min and max are NULL. - Skips primary-key columns (their values are identifiers, not values to - profile). - - DEV-1480: text/boolean columns are excluded here so they are served - exclusively by the categorical dim profile (which populates both - ``Column.sampled`` and ``Column.sampled_values``). Mixing the two - paths for the same column would leave ``sampled_values=None`` while - ``sampled`` is set, which ``_is_sample_cached`` correctly treats as a - cache miss — leading to permanent re-profile every ``inspect_model`` - call. - """ - _NUMERIC_TEMPORAL = ( - DataType.INT, DataType.DOUBLE, DataType.DATE, DataType.TIMESTAMP, - ) - columns = [ - c for c in model.columns - if not c.hidden and not c.primary_key - and c.type in _NUMERIC_TEMPORAL - ] - if not columns: - return {} - - # Use ModelExtension with inline columns to bypass allowed_aggregations - ext_columns = [ - {"name": f"_slayer_probe_{c.name}", "sql": c.sql if c.sql else c.name, - "type": str(c.type)} - for c in columns - ] - measures_payload: List[Dict[str, str]] = [] - for c in columns: - measures_payload.append({"formula": f"_slayer_probe_{c.name}:min"}) - measures_payload.append({"formula": f"_slayer_probe_{c.name}:max"}) - - try: - q = SlayerQuery.model_validate({ - "source_model": {"source_name": model.name, "columns": ext_columns}, - "measures": measures_payload, - }) - r = await engine.execute(query=q, data_source=model.data_source or None) - row = r.data[0] if r.data else {} - except Exception: - return {} - - result: Dict[str, str] = {} - for c in columns: - mn = row.get(f"{model.name}._slayer_probe_{c.name}_min") - mx = row.get(f"{model.name}._slayer_probe_{c.name}_max") - if mn is None and mx is None: - result[c.name] = "all NULL" - else: - result[c.name] = f"{mn} .. {mx}" - return result - - -async def _collect_reachable_fields( - model: SlayerModel, - storage: StorageBackend, - *, - max_depth: int = 5, -) -> Tuple[List[str], List[str]]: - """BFS the join graph from ``model``; return sorted fully-qualified dotted - paths for every reachable non-hidden, non-pk dimension and non-hidden - measure (excluding the root model's own fields — those live in the main - Dimensions/Measures tables). Depth is measured in path segments and capped - at ``max_depth``. Cycles are broken by a visited-path set. - """ - reachable_dims: set[str] = set() - reachable_measures: set[str] = set() - visited: set[str] = set() - queue: List[Tuple[str, str]] = [] # (full_path, target_model_name) - - def _derive_path(base: str, join: ModelJoin) -> str: - if base: - return f"{base}.{join.target_model}" - return join.target_model - - for j in model.joins: - path = _derive_path("", j) - if path not in visited: - queue.append((path, j.target_model)) - - while queue: - path, target_name = queue.pop(0) - if path in visited: - continue - visited.add(path) - if path.count(".") + 1 > max_depth: - continue - # v4 (DEV-1330): walk the join graph within the *root* model's - # data_source. Cross-datasource joins aren't auto-mirrored, so any - # bare-name resolution that crosses a datasource boundary would be - # picking up a sibling model that isn't actually reachable from - # ``model``. - try: - target = await storage.get_model(target_name, data_source=model.data_source or None) - except Exception: # noqa: BLE001 — AmbiguousModelError or storage misses - target = None - if target is None: - continue - for c in target.columns: - if c.hidden: - continue - if not c.primary_key: - reachable_dims.add(f"{path}.{c.name}") - reachable_measures.add(f"{path}.{c.name}") - for j in target.joins: - sub_path = _derive_path(path, j) - # Per-path cycle check: don't revisit any model already on this - # path (prevents bounce-backs from peer joins while preserving - # diamond joins where the same model is reached via independent paths). - path_models = set(path.split(".")) - path_models.add(model.name) # include root - if sub_path not in visited and j.target_model not in path_models: - queue.append((sub_path, j.target_model)) - - return sorted(reachable_dims), sorted(reachable_measures) - - -def _build_backing_query_info(model: SlayerModel) -> Optional[dict]: - """Build the ``backing_query`` block for inspect_model output. - - Returns ``None`` for non-query-backed models. For query-backed models, - returns ``{variables, required_variables, stages}`` where: - - - ``variables``: ``model.query_variables`` (defaults). - - ``required_variables``: placeholder names that have no default. - - ``stages``: each stage dumped as a dict, ready for JSON output. - """ - if not model.source_queries: - return None - from slayer.core.query import extract_placeholder_names - - all_placeholders: set = set() - stage_dicts: List[dict] = [] - # A placeholder is "required" only if it has no default at any layer the - # engine consults: model.query_variables OR the stage's own variables. - defaulted: set = set(model.query_variables.keys()) - for q in model.source_queries: - all_placeholders |= extract_placeholder_names(q) - if q.variables: - defaulted |= set(q.variables.keys()) - stage_dicts.append(q.model_dump(mode="json", exclude_none=True)) - required = sorted(all_placeholders - defaulted) - return { - "variables": dict(model.query_variables), - "required_variables": required, - "stages": stage_dicts, - } - - -def _render_field_value(v: Any) -> str: - """Pick the most descriptive label out of a query-stage field value. - - Stage list entries can be plain strings, simple `{name}` dicts, formula - dicts, or wrapper dicts like `{"dimension": {"name": ...}}`. Try each - shape in priority order and fall back to `str(v)` if nothing matches. - """ - if not isinstance(v, dict): - return str(v) - name = v.get("name") - if name: - return str(name) - formula = v.get("formula") - if formula: - return str(formula) - inner = v.get("dimension") - if isinstance(inner, dict): - inner_name = inner.get("name") - if inner_name: - return str(inner_name) - return str(v) - - -def _render_stage_field_list(key: str, val: list) -> str: - """Render a stage's field list (dimensions / measures / filters / etc.).""" - if key == "filters": - return "; ".join(f"`{f}`" for f in val) - return "; ".join(_render_field_value(v) for v in val) - - -def _render_source_model(src: Any) -> Optional[str]: - """Render a stage's ``source_model`` (str or ModelExtension dict).""" - if isinstance(src, str): - return f"- source_model: `{src}`" - if isinstance(src, dict): - sn = src.get("source_name") or src.get("name") - if sn: - return f"- source_model: `{sn}` (extension)" - return None - - -def _render_stage(i: int, stage: dict, total: int) -> List[str]: - """Render one stage's markdown lines.""" - title = stage.get("name") or ("final" if i == total else f"stage {i}") - out: List[str] = [f"\n**{i}. {title}**"] - src_line = _render_source_model(stage.get("source_model")) - if src_line: - out.append(src_line) - for key in ("dimensions", "time_dimensions", "measures", "filters"): - val = stage.get(key) - if not val: - continue - out.append(f"- {key}: {_render_stage_field_list(key, val)}") - return out - - -def _backing_query_markdown_section(info: dict) -> str: - """Format the ``backing_query`` info as a markdown section.""" - lines: List[str] = ["## Backing Query"] - stages = info.get("stages") or [] - for i, stage in enumerate(stages, start=1): - lines.extend(_render_stage(i, stage, len(stages))) - variables = info.get("variables") or {} - required = info.get("required_variables") or [] - if variables or required: - lines.append("\n**Variables:**") - for k, v in variables.items(): - lines.append(f"- `{k}`: default `{v}`") - for k in required: - lines.append(f"- `{k}`: required") - return "\n".join(lines) - - -def _source_type_for(model: SlayerModel) -> str: - """Classify a model's source mode for summary/inspect output.""" - if model.source_queries: - return "query" - if model.sql_table: - return "table" - if model.sql: - return "sql" - return "unknown" - - def _model_to_summary(model: SlayerModel) -> dict: """Convert a SlayerModel to a summary dict.""" columns = [] @@ -795,11 +256,32 @@ def create_mcp_server( # NOSONAR(S3776) — FastMCP tool-registration factory; storage: StorageBackend, *, ingest_on_startup: bool = False, + _seed_help: bool = True, ): + from slayer.async_utils import run_sync + + # DEV-1658: seed the conceptual-help memories (help.intro …). ``_seed_help`` + # is False when embedded in create_app (which seeds once itself), so the + # pass never fires twice. Idempotent / skip-if-unchanged, so a warm store + # is a cheap no-op. + # + # DEV-1669: seeding is a convenience side-effect and must never crash server + # construction. Skip silently for a ``None`` / non-``StorageBackend`` arg — + # metadata-only builds (reading advertised tool names / a tool's JSON + # schema) need no storage at all. When a real backend is given, treat a + # genuine seed failure (nested-loop ``run_sync``, embedding/DB error) as + # best-effort: warn and continue rather than abort the build. + if _seed_help and isinstance(storage, StorageBackend): + try: + run_sync(seed_help_memories(storage=storage)) + except Exception as exc: # noqa: BLE001 — seeding must not abort the build + logger.warning( + "SLayer help-memory seeding skipped: %s", exc, exc_info=True + ) + if ingest_on_startup: import sys - from slayer.async_utils import run_sync from slayer.engine.ingestion import ingest_all_datasources_idempotent run_sync( @@ -815,43 +297,44 @@ def create_mcp_server( # NOSONAR(S3776) — FastMCP tool-registration factory; instructions=( "SLayer is a semantic layer for querying databases. " "Instead of writing SQL, describe what data you want using models, measures, dimensions, and filters. " - "Call help() for an overview of SLayer concepts, and help(topic='...') for deep dives on specific topics. " - "Typical workflow: list_datasources → models_summary → inspect_model → query. " + "New to SLayer? Start with inspect(reference='memory:help.intro', entity_type='memory') for an overview of core concepts and the query shape — it lists the deep-dive topics you can inspect the same way. " + "Use search(question='...') to find relevant concepts, models, and saved learnings. " + "Typical workflow: inspect(memory:help.intro) → search → inspect → query. " "To connect a new database: create_datasource → describe_datasource (verify + list tables) → ingest_datasource_models → models_summary." ), ) engine = SlayerQueryEngine(storage=storage) - - _help_description = ( - "Return conceptual help on SLayer. " - "Call without a topic for the intro (what SLayer is, core entities, the query shape). " - "Pass a topic name for a deep dive. " - f"{TOPIC_SUMMARY_LINE} " - "Args: topic (optional) — the topic name. Unknown topics return a friendly error listing the valid ones." - ) - - @mcp.tool(description=_help_description) - async def help(topic: Optional[str] = None) -> str: # noqa: A001 — intentional shadow of builtin inside factory - return render_help(topic=topic) + # DEV-1656: expose the closure engine so callers (bird-interact-agents on + # the cloud Ray runner, where one actor process is reused across many + # tasks) can dispose its per-task asyncpg pools at task teardown: + # engine = getattr(mcp, "_slayer_engine", None) + # if engine is not None: + # await engine.aclose() # loop-bound; run before the task loop closes + # aclose() is idempotent and leaves the engine reusable (a later execute + # lazily recreates the async engine). The read-only introspection tools + # (validate_models / recommend_root_model) reuse this same engine so a + # single engine holds every cached SQL client for the server's lifetime. + mcp._slayer_engine = engine @mcp.tool() async def query( # NOSONAR S107 — FastMCP introspects this signature to expose each query option as a typed MCP tool argument; collapsing into a dict would degrade the agent-facing schema source_model: str | ModelExtension | SlayerModel, - measures: Optional[List[Dict[str, str]]] = None, - dimensions: Optional[List[str]] = None, - filters: Optional[List[str]] = None, - time_dimensions: Optional[List[Dict[str, Any]]] = None, - order: Optional[List[Dict[str, str]]] = None, - limit: Optional[int] = None, - offset: Optional[int] = None, + measures: list[dict[str, str]] | None = None, + dimensions: list[str] | None = None, + filters: list[str] | None = None, + time_dimensions: list[dict[str, Any]] | None = None, + order: list[dict[str, str]] | None = None, + limit: int | None = None, + offset: int | None = None, whole_periods_only: bool = False, show_sql: bool = False, dry_run: bool = False, explain: bool = False, format: str = "markdown", - variables: Optional[Dict[str, Any]] = None, + variables: dict[str, Any] | None = None, + distinct_dimension_values: bool = True, ) -> str: - """Query data from a semantic model. Call inspect_model first to see available columns and measures. + """Query data from a semantic model. Call inspect(reference=".", entity_type="model") first to see available columns and measures. Args: source_model: One of three forms: @@ -863,11 +346,21 @@ async def query( # NOSONAR S107 — FastMCP introspects this signature to expos ``{"name": "ad_hoc", "sql_table": "things", "data_source": "test", "columns": [...]}``. measures: Aggregated values to return. Each is a formula: {"formula": "*:count"}, {"formula": "revenue:sum / *:count", "name": "aov"} (arithmetic), - {"formula": "cumsum(revenue:sum)"} (cumulative sum), {"formula": "change(revenue:sum)"} (diff from previous row), - {"formula": "change_pct(revenue:sum)"} (% change), {"formula": "time_shift(revenue:sum, -1)"} (previous period via self-join), - {"formula": "time_shift(revenue:sum, -1, 'year')"} (year-over-year), {"formula": "lag(revenue:sum, 1)"} (previous row via window function), + {"formula": "cumsum(revenue:sum)"} (cumulative sum), + {"formula": "change(revenue:sum)"} (period-over-period difference), + {"formula": "change_pct(revenue:sum)"} (period-over-period % change, e.g. month-over-month growth), + {"formula": "time_shift(revenue:sum, -1)"} (the shifted value itself, one time bucket back), + {"formula": "time_shift(revenue:sum, -1, 'year')"} (value from one year earlier, for custom arithmetic), + {"formula": "lag(revenue:sum, 1)"} (previous row via window function; shifts by row position, NULL at edges), {"formula": "lead(revenue:sum, 1)"} (next row via window function), {"formula": "last(revenue:sum)"} (most recent), {"formula": "rank(revenue:sum)"} (ranking). A bare name like {"formula": "aov"} resolves to a saved ModelMeasure on the model. + change / change_pct / time_shift are calendar-aware and partition-safe: change and change_pct compare + each row against the prior time bucket (one step back at the query's own granularity), while time_shift + compares at its explicitly requested offset and granularity. All three join on the same non-time + dimension values, so per-group series reset cleanly — safe for grouped queries like month-over-month + revenue by store. + For period-over-period growth, prefer change_pct (or change for the absolute delta); use time_shift + only when you need the shifted value itself as a term in your own arithmetic. dimensions: List of dimension names to group by, e.g. ["status", "region"]. filters: Filter conditions as formula strings. Examples: "status == 'completed'", "amount > 100", "status in ('a', 'b')", "status is None", @@ -884,12 +377,13 @@ async def query( # NOSONAR S107 — FastMCP introspects this signature to expos dry_run: When true, generate and return the SQL without executing it. explain: When true, run EXPLAIN ANALYZE and return the query plan. format: Output format — "markdown" (default, compact and LLM-friendly), "json" (structured), or "csv" (most compact). Case-insensitive. + distinct_dimension_values: Default True (Cube.js-style auto-dedup for dim-only queries — emits GROUP BY ). Set False to emit raw rows: no top-level GROUP BY, just SELECT with the usual WHERE/ORDER BY/LIMIT. Any measure reference (in measures, filters, or order) raises an error in this mode. Example: query(source_model="orders", measures=[{"formula": "*:count"}], dimensions=["status"], filters=["status == 'completed'"]) Before calling this tool, run ``search`` first, supplying the entities you're thinking of using (and/or the query itself via the ``query`` arg, or a free-text ``question``). Read the returned memories and consider any matching example queries before formulating the final query. """ - data: Dict[str, Any] = {"source_model": source_model} + data: dict[str, Any] = {"source_model": source_model} if dimensions: data["dimensions"] = list(dimensions) if filters: @@ -908,6 +402,9 @@ async def query( # NOSONAR S107 — FastMCP introspects this signature to expos data["measures"] = measures if variables: data["variables"] = dict(variables) + # DEV-1543: only emit when non-default so tool calls stay compact. + if distinct_dimension_values is False: + data["distinct_dimension_values"] = False try: fmt = format.lower().strip() if fmt not in ("json", "csv", "markdown"): @@ -927,6 +424,9 @@ async def query( # NOSONAR S107 — FastMCP introspects this signature to expos and not time_dimensions and not order and limit is None and offset is None and not whole_periods_only + # DEV-1543: explicit ``False`` is a real override; default + # ``True`` falls through. + and distinct_dimension_values ) if isinstance(source_model, str) and no_overrides: model_name = source_model @@ -974,8 +474,8 @@ async def query( # NOSONAR S107 — FastMCP introspects this signature to expos @mcp.tool() async def query_nested( - queries: List[Dict[str, Any]], - variables: Optional[Dict[str, Any]] = None, + queries: list[dict[str, Any]], + variables: dict[str, Any] | None = None, show_sql: bool = False, dry_run: bool = False, explain: bool = False, @@ -1058,21 +558,28 @@ async def query_nested( # ----------------------------------------------------------------------- @mcp.tool() - async def models_summary(datasource_name: str, format: str = "markdown") -> str: + async def models_summary( + datasource_name: str, + format: str = "markdown", + compact: bool = True, + ) -> str: """Brief summary of all (non-hidden) models in a datasource. - For each model: name, description, a table of its columns (name + - type + description), a table of its named-formula measures (name - + formula + description), and a comma-separated list of the model - names it joins to. No distinct values, no sample data, and no - expansion of joined models' fields — call inspect_model for any - of that. + DEV-1549: compact-by-default rendering. Under ``compact=True`` + each model section emits its name, description, the column count + (``Columns: N``), the comma-separated measure NAMES + (``Measures: a, b, c``) and the ``Joins to:`` list — no + per-column table, no per-measure formula block. Pass + ``compact=False`` to restore the verbose markdown / JSON shape + with full column and measure payloads. Args: datasource_name: Name of the datasource (from list_datasources). format: Output format — "markdown" (default, compact and LLM-friendly) or "json" (structured array of model summaries). Case-insensitive. + compact: Default True — drop per-column / per-measure detail. + Set False to surface the full per-model tables. """ fmt = format.lower().strip() if fmt not in ("markdown", "json"): @@ -1089,7 +596,7 @@ async def models_summary(datasource_name: str, format: str = "markdown") -> str: return f"Datasource '{datasource_name}' not found." all_names = await storage.list_models(data_source=datasource_name) - matched: List[SlayerModel] = [] + matched: list[SlayerModel] = [] for n in all_names: try: m = await storage.get_model(n, data_source=datasource_name) @@ -1100,74 +607,14 @@ async def models_summary(datasource_name: str, format: str = "markdown") -> str: matched.append(m) matched.sort(key=lambda m: m.name) - if not matched: - return f"Datasource '{datasource_name}' has no models." - - if fmt == "json": - return json.dumps( - { - "datasource_name": datasource_name, - "model_count": len(matched), - "models": [ - { - "name": m.name, - "description": m.description, - "columns": [ - {"name": c.name, "type": str(c.type), "description": c.description} - for c in m.columns if not c.hidden - ], - "measures": [ - {"name": mm.name, "formula": mm.formula, "description": mm.description} - for mm in m.measures - ], - "joins_to": sorted({j.target_model for j in m.joins}), - } - for m in matched - ], - }, - indent=2, - ) - - sections: List[str] = [ - f"# Datasource: `{datasource_name}` — {len(matched)} model(s)" - ] - for m in matched: - model_lines: List[str] = [f"## `{m.name}`"] - if m.description: - model_lines.append(m.description) - - col_rows = [ - {"name": c.name, "type": str(c.type), "description": c.description} - for c in m.columns if not c.hidden - ] - model_lines.append(f"**Columns ({len(col_rows)}):**") - model_lines.append("") - model_lines.append( - _markdown_table(rows=col_rows, columns=["name", "type", "description"]) - ) - model_lines.append("") - - measure_rows = [ - {"name": mm.name, "formula": mm.formula, "description": mm.description} - for mm in m.measures - ] - model_lines.append(f"**Measures ({len(measure_rows)}):**") - model_lines.append("") - model_lines.append( - _markdown_table(rows=measure_rows, columns=["name", "formula", "description"]) - ) - model_lines.append("") - - if m.joins: - targets = sorted({j.target_model for j in m.joins}) - rendered = ", ".join(f"`{t}`" for t in targets) - model_lines.append(f"**Joins to:** {rendered}") - else: - model_lines.append("**Joins to:** _(none)_") - - sections.append("\n".join(model_lines)) - - return "\n\n".join(sections) + # DEV-1667: rendering delegates to the shared renderer (also used by + # the ``inspect`` model collection view) — one code path, no drift. + return render_models_summary( + datasource_name=datasource_name, + models=matched, + fmt=fmt, + compact=compact, + ) @mcp.tool() async def inspect_model( @@ -1175,12 +622,12 @@ async def inspect_model( num_rows: int = 3, show_sql: bool = False, format: str = "markdown", - sections: Optional[List[str]] = None, - descriptions_max_chars: Optional[int] = None, - reachable_fields_depth: int = 5, - data_source: Optional[str] = None, + sections: list[str] | None = None, + descriptions_max_chars: int | None = None, + data_source: str | None = None, + compact: bool = True, ) -> str: - """Return a complete-yet-compact view of a semantic model. + """DEPRECATED: use the ``inspect`` tool. Return a complete-yet-compact view of a semantic model. Always emitted (regardless of ``sections``): model header + description, metadata bullets (data_source, sql_table, default_time_dimension, @@ -1199,13 +646,14 @@ async def inspect_model( column and the ``sql`` field of each ``params[]`` entry are gated by ``show_sql``. - ``joins`` — join definitions. - - ``reachable_fields`` — BFS-walked fields reachable via joins. - ``samples`` — live sample-data query (``COUNT(*)`` plus one aggregation per column). + - ``learnings`` — learning-only memories whose canonical entities + reference this model. When a section is omitted from ``sections``: ``columns``, ``measures``, ``aggregations`` and ``joins`` collapse to a one-line backticked CSV - of names; ``reachable_fields`` and ``samples`` are dropped entirely. + of names; ``samples`` and ``learnings`` are dropped entirely. A footer at the end of the response lists what was trimmed and how to fetch more. @@ -1218,7 +666,7 @@ async def inspect_model( format: Output format — ``"markdown"`` (default) or ``"json"``. Case-insensitive. sections: Subset of ``["columns", "measures", "aggregations", - "joins", "reachable_fields", "samples"]``. Default (``None`` + "joins", "samples", "learnings"]``. Default (``None`` or empty list) renders all six. Unknown names are ignored with a warning line at the end of the response. A non-empty list of *only* unknown names resolves to no sections (not @@ -1228,26 +676,7 @@ async def inspect_model( column, measure, aggregation) longer than this is truncated with a ``... [truncated]`` suffix. Must be ``>= 0``. ``None`` (default) means no truncation. - reachable_fields_depth: Max BFS depth (in path segments) for the - reachable-fields walk. Default 5; allowed range - ``[0, 20]``. Ignored when ``reachable_fields`` is not in - ``sections``. """ - fmt = format.lower().strip() - if fmt not in ("markdown", "json"): - raise ValueError( - f"Invalid format '{format}' for inspect_model. Must be 'markdown' or 'json'." - ) - if descriptions_max_chars is not None and descriptions_max_chars < 0: - raise ValueError( - f"descriptions_max_chars must be >= 0, got {descriptions_max_chars}." - ) - if reachable_fields_depth < 0 or reachable_fields_depth > _MAX_REACHABLE_FIELDS_DEPTH: - raise ValueError( - f"reachable_fields_depth must be between 0 and {_MAX_REACHABLE_FIELDS_DEPTH}, " - f"got {reachable_fields_depth}." - ) - try: model = await storage.get_model(model_name, data_source=data_source) except AmbiguousModelError as exc: @@ -1261,610 +690,94 @@ async def inspect_model( available.append(f"{ds_name}.{n}") available.sort() return f"Model '{model_name}' not found. Available models: {', '.join(available)}" - - # Resolve section gating up front so we can short-circuit DB calls - # for parts the caller doesn't want. - included, unknown = _resolve_inspect_sections(sections) - included_set = set(included) - - # Categorise non-included sections into "names-only" (still listed, - # just collapsed to CSV) vs "fully omitted" (no heading at all). - names_only_sections = [ - s for s in _INSPECT_SECTIONS_NAMES_ONLY if s not in included_set - ] - omitted_sections = [ - s for s in _INSPECT_SECTIONS_OMITTABLE if s not in included_set - ] - - truncated_model_desc = _truncate_description(model.description, descriptions_max_chars) - out_sections: List[str] = [f"# Model: `{model.name}`"] - if truncated_model_desc: - out_sections.append(truncated_model_desc) - - # Metadata bullets (incl. row_count from a cheap *:count query) - meta: List[str] = [] - if model.data_source: - meta.append(f"- **data_source:** `{model.data_source}`") - if model.sql_table: - meta.append(f"- **sql_table:** `{model.sql_table}`") - if model.default_time_dimension: - meta.append( - f"- **default_time_dimension:** `{model.default_time_dimension}`" - ) - if model.hidden: - meta.append("- **hidden:** true") - if model.meta is not None: - meta.append(f"- **meta:** {json.dumps(model.meta, sort_keys=True, default=str)}") - row_count = await _get_row_count(model=model, engine=engine) - if row_count is not None: - meta.append(f"- **row_count:** {row_count:,}") - if meta: - out_sections.append("\n".join(meta)) - - if show_sql and model.sql: - out_sections.append(f"## SQL\n\n```sql\n{model.sql}\n```") - - if show_sql and model.filters: - filter_lines = "\n".join(f"- `{f}`" for f in model.filters) - out_sections.append(f"## Filters (model-level)\n\n{filter_lines}") - - # Backing-query section (query-backed models only). Structure is - # always-on (it's the model's identity for query-backed models, like - # `sql_table` is for table-backed); only the SQL cache is gated by - # show_sql. - backing_info = _build_backing_query_info(model) - if backing_info is not None: - out_sections.append(_backing_query_markdown_section(backing_info)) - if show_sql and model.backing_query_sql: - out_sections.append( - f"## Backing Query SQL\n\n```sql\n{model.backing_query_sql}\n```" - ) - - # ------------------------------------------------------------------ - # DB-hitting computations — skip when their consumers aren't requested. - # ------------------------------------------------------------------ - # Dimension/measure profile populates the ``sampled`` / ``sampled_values`` - # / ``distinct_count`` columns of the row. Read the cache first; on - # miss, profile live and write back via storage so subsequent calls - # (and any search) hit the cache for free (DEV-1375 + DEV-1480). - profile_by_name: Dict[str, str] = {} - profile_values_by_name: Dict[str, Optional[List[str]]] = {} - distinct_count_by_name: Dict[str, Optional[int]] = {} - measure_profile: Dict[str, str] = {} - if "columns" in included_set: - uncached_columns: List[Column] = [] - for c in model.columns: - if c.hidden or c.primary_key: - continue - # DEV-1480 cache validity: categorical needs - # ``sampled_values`` to be present (the structured field - # is authoritative); numeric/temporal needs ``sampled``. - if _is_sample_cached(c): - if c.sampled is not None: - profile_by_name[c.name] = c.sampled - profile_values_by_name[c.name] = c.sampled_values - distinct_count_by_name[c.name] = c.distinct_count - else: - # v6-upgrade fallback: a categorical column may have - # legacy ``sampled`` text but no ``sampled_values`` - # yet. Surface the legacy text in case the live - # re-profile below fails for transient reasons — - # ``profile_column`` will overwrite on success. - if c.sampled is not None: - profile_by_name[c.name] = c.sampled - uncached_columns.append(c) - if uncached_columns: - # DEV-1480: split the live profile into two paths so we - # preserve the pre-DEV-1480 batching for numeric/temporal - # columns. Categorical columns fire a top-values query - # (and a secondary count_distinct on overflow) per column — - # there's no efficient cross-column batching for those. - # Numeric/temporal columns share one batched min/max query. - _CATEGORICAL = (DataType.TEXT, DataType.BOOLEAN) - _NUMERIC_TEMPORAL = ( - DataType.INT, DataType.DOUBLE, - DataType.DATE, DataType.TIMESTAMP, - ) - cat_uncached = [ - c for c in uncached_columns if c.type in _CATEGORICAL - ] - num_uncached = [ - c for c in uncached_columns if c.type in _NUMERIC_TEMPORAL - ] - - async def _persist_sample( - *, col_name: str, - sampled: Optional[str], - sampled_values: Optional[List[str]], - distinct_count: Optional[int], - ) -> None: - try: - await storage.update_column_sampled( - data_source=model.data_source, - model_name=model.name, - column_name=col_name, - sampled=sampled, - sampled_values=sampled_values, - distinct_count=distinct_count, - ) - except Exception as exc: - logger.warning( - "inspect_model: failed to persist sampled value for " - "%s.%s.%s: %s", - model.data_source, model.name, col_name, exc, - ) - - # Categorical: one top-values query per column (+ optional - # count_distinct on overflow). - for col in cat_uncached: - try: - sample = await profile_column( - model=model, column=col, engine=engine, - ) - except Exception as exc: - logger.warning( - "inspect_model: failed to profile %s.%s.%s: %s", - model.data_source, model.name, col.name, exc, - ) - sample = None - if sample is None: - continue - if sample.sampled is not None: - profile_by_name[col.name] = sample.sampled - profile_values_by_name[col.name] = sample.sampled_values - distinct_count_by_name[col.name] = sample.distinct_count - await _persist_sample( - col_name=col.name, - sampled=sample.sampled, - sampled_values=sample.sampled_values, - distinct_count=sample.distinct_count, - ) - - # Numeric/temporal: one batched min/max query for all of - # them at once (restores the pre-DEV-1480 batching for - # wide models). - if num_uncached: - num_entries = await _profile_numeric_temporal_columns( - model=model, columns=num_uncached, engine=engine, - ) - for col in num_uncached: - entry = num_entries.get(col.name) - if entry is None: - continue - if entry.min_value is None and entry.max_value is None: - continue - sampled_text = f"{entry.min_value} .. {entry.max_value}" - profile_by_name[col.name] = sampled_text - # Numeric/temporal columns carry no structured list - # and no distinct_count per the DEV-1480 contract. - profile_values_by_name[col.name] = None - distinct_count_by_name[col.name] = None - await _persist_sample( - col_name=col.name, - sampled=sampled_text, - sampled_values=None, - distinct_count=None, - ) - measure_profile = await _collect_measure_profile(model=model, engine=engine) - # Persist any measure-side (numeric/temporal) profile - # values to ``Column.sampled`` so subsequent - # ``inspect_model`` / search calls hit the cache - # instead of re-running the live profile query. - for col in uncached_columns: - sampled_value = measure_profile.get(col.name) - if sampled_value is None or col.name in profile_by_name: - # Either no measure-side value for this column - # (already covered by dim profile above), or - # the dim profile already won the cache slot. - continue - profile_by_name[col.name] = sampled_value - try: - await storage.update_column_sampled( - data_source=model.data_source, - model_name=model.name, - column_name=col.name, - sampled=sampled_value, - sampled_values=None, - distinct_count=None, - ) - except Exception as exc: - logger.warning( - "inspect_model: failed to persist sampled value for " - "%s.%s.%s: %s", - model.data_source, model.name, col.name, exc, - ) - - # ``measure_types`` informs the sample query's choice of avg vs - # count_distinct. Only needed when ``samples`` is in the included set. - measure_types: Dict[str, str] = {} - if "samples" in included_set: - measure_types = await engine.get_column_types( - model_name=model.name, - data_source=model.data_source or None, - ) - - # ------------------------------------------------------------------ - # Columns section - # ------------------------------------------------------------------ - visible_columns = [c for c in model.columns if not c.hidden] - if "columns" in included_set: - col_rows: List[Dict[str, Any]] = [] - for c in visible_columns: - aggs = ", ".join(c.allowed_aggregations) if c.allowed_aggregations else "all" - # DEV-1480: key-presence check (not ``or`` truthiness) so an - # all-NULL categorical column's ``sampled=""`` doesn't - # silently fall through to the measure_profile fallback's - # ``"all NULL"`` text. - if c.name in profile_by_name: - sampled_cell = profile_by_name[c.name] - else: - sampled_cell = measure_profile.get(c.name) - col_rows.append({ - "name": c.name, - "type": str(c.type), - "primary_key": "yes" if c.primary_key else "", - "sql": c.sql if c.sql else c.name, - "allowed_aggregations": aggs, - "filter": c.filter, - "label": c.label, - "description": _truncate_description(c.description, descriptions_max_chars), - "meta": _format_meta(c.meta), - "sampled": sampled_cell, - }) - col_columns = [ - "name", "type", "primary_key", "sql", "allowed_aggregations", - "filter", "label", "description", "meta", "sampled", - ] - if not show_sql: - col_columns = [c for c in col_columns if c not in ("sql", "filter")] - out_sections.append( - f"## Columns ({len(col_rows)})\n\n" - + _markdown_table(rows=col_rows, columns=col_columns) - ) - elif visible_columns: - csv = ", ".join(_md_code_span(c.name) for c in visible_columns) - out_sections.append( - f"## Columns ({len(visible_columns)} — names only)\n\n{csv}" - ) - - # ------------------------------------------------------------------ - # Measures section - # ------------------------------------------------------------------ - if "measures" in included_set: - measure_rows: List[Dict[str, Any]] = [] - for mm in model.measures: - measure_rows.append({ - "name": mm.name, - "formula": mm.formula, - "label": mm.label, - "description": _truncate_description(mm.description, descriptions_max_chars), - "meta": _format_meta(mm.meta), - }) - out_sections.append( - f"## Measures ({len(measure_rows)})\n\n" - + _markdown_table( - rows=measure_rows, - columns=["name", "formula", "label", "description", "meta"], - ) - ) - elif model.measures: - csv = ", ".join(_md_code_span(mm.name) for mm in model.measures) - out_sections.append( - f"## Measures ({len(model.measures)} — names only)\n\n{csv}" - ) - - # ------------------------------------------------------------------ - # Aggregations section - # ------------------------------------------------------------------ - if "aggregations" in included_set: - if model.aggregations: - agg_rows: List[Dict[str, Any]] = [] - for a in model.aggregations: - if a.params: - if show_sql: - params = "; ".join(f"{p.name}={p.sql}" for p in a.params) - else: - params = ", ".join(p.name for p in a.params) - else: - params = None - agg_rows.append({ - "name": a.name, - "formula": a.formula or "(built-in override)", - "params": params, - "description": _truncate_description( - a.description, descriptions_max_chars, - ), - "meta": _format_meta(a.meta), - }) - agg_columns = ["name", "formula", "params", "description", "meta"] - if not show_sql: - agg_columns = [c for c in agg_columns if c != "formula"] - out_sections.append( - f"## Aggregations ({len(agg_rows)})\n\n" - + _markdown_table(rows=agg_rows, columns=agg_columns) - ) - elif model.aggregations: - csv = ", ".join(_md_code_span(a.name) for a in model.aggregations) - out_sections.append( - f"## Aggregations ({len(model.aggregations)} — names only)\n\n{csv}" - ) - - # ------------------------------------------------------------------ - # Joins section - # ------------------------------------------------------------------ - if "joins" in included_set: - join_rows: List[Dict[str, Any]] = [] - for j in model.joins: - pairs = "; ".join(f"{src} = {tgt}" for src, tgt in j.join_pairs) - join_rows.append({ - "target_model": j.target_model, - "join_pairs": pairs, - }) - out_sections.append( - f"## Joins ({len(join_rows)})\n\n" - + _markdown_table( - rows=join_rows, - columns=["target_model", "join_pairs"], - ) - ) - elif model.joins: - csv = ", ".join(_md_code_span(j.target_model) for j in model.joins) - out_sections.append( - f"## Joins ({len(model.joins)} — names only)\n\n{csv}" - ) - - # ------------------------------------------------------------------ - # Reachable via joins (fully omitted when not in sections) - # ------------------------------------------------------------------ - reach_dims: List[str] = [] - reach_measures: List[str] = [] - if "reachable_fields" in included_set: - reach_dims, reach_measures = await _collect_reachable_fields( - model=model, storage=storage, max_depth=reachable_fields_depth, - ) - if reach_dims or reach_measures: - lines = [ - f"## Reachable via joins (max depth: {reachable_fields_depth})", "", - ] - if reach_dims: - rendered = ", ".join(f"`{d}`" for d in reach_dims) - lines.append(f"**Dimensions ({len(reach_dims)}):** {rendered}") - if reach_measures: - rendered = ", ".join(f"`{m}`" for m in reach_measures) - lines.append(f"**Measures ({len(reach_measures)}):** {rendered}") - out_sections.append("\n".join(lines)) - - # ------------------------------------------------------------------ - # Sample data (fully omitted when not in sections) - # ------------------------------------------------------------------ - sample_sql: Optional[str] = None - sample_data: Optional[Dict[str, Any]] = None - sample_error: Optional[str] = None - if "samples" in included_set: - query_args = _build_sample_query_args( - model=model, num_rows=num_rows, measure_types=measure_types, - ) - try: - sample_query = SlayerQuery.model_validate(query_args) - sample_result = await engine.execute( - query=sample_query, data_source=model.data_source or None - ) - sample_sql = sample_result.sql - cols, data = _strip_model_prefix( - columns=sample_result.columns, - data=sample_result.data, - model_name=model.name, - ) - sample_data = {"columns": cols, "rows": data} - sample_result.columns = cols - sample_result.data = data - sample_section = f"## Sample Data\n\n{sample_result.to_markdown()}" - if show_sql and sample_sql: - sample_section = ( - f"## Sample Data SQL\n\n```sql\n{sample_sql}\n```\n\n" - + sample_section - ) - out_sections.append(sample_section) - except Exception as e: - if isinstance(e, (sa.exc.OperationalError, sa.exc.DatabaseError)): - err = _friendly_db_error(e) - else: - err = str(e) - sample_error = err - sample_section = f"## Sample Data\n\n_Error fetching sample data: {err}_" - if show_sql and sample_sql: - sample_section = ( - f"## Sample Data SQL\n\n```sql\n{sample_sql}\n```\n\n" - + sample_section - ) - out_sections.append(sample_section) - - # ------------------------------------------------------------------ - # Learnings (DEV-1357 v2) — surfaces only memories where ``query`` is - # ``None``; query-bearing memories are recall-only. Auto-pruned when - # no learning-shaped memory matches. - # ------------------------------------------------------------------ - relevant_learnings: List[Any] = [] - wanted: List[str] = [] - if "learnings" in included_set: - ds = model.data_source - wanted = [f"{ds}.{model.name}"] - wanted.extend(f"{ds}.{model.name}.{c.name}" for c in model.columns) - wanted.extend( - f"{ds}.{model.name}.{m.name}" - for m in model.measures - if m.name is not None - ) - wanted.extend( - f"{ds}.{model.name}.{a.name}" for a in model.aggregations - ) - candidates = await storage.list_memories(entities=wanted) - relevant_learnings = [m for m in candidates if m.query is None] - if relevant_learnings: - lines = [f"## Learnings ({len(relevant_learnings)})", ""] - for memory in relevant_learnings: - matched = sorted(set(wanted) & set(memory.entities)) - matched_md = ", ".join(f"`{e}`" for e in matched) - lines.append( - f"- **M{memory.id}** ({matched_md}): {memory.learning}" - ) - out_sections.append("\n".join(lines)) - - # ------------------------------------------------------------------ - # Per-call truncation footer (only when something was trimmed or an - # unknown section name was supplied). - # ------------------------------------------------------------------ - footer = _render_inspect_footer( - included=included, - names_only=names_only_sections, - omitted=omitted_sections, - unknown=unknown, + return await render_model_inspection( + model=model, + storage=storage, + engine=engine, + num_rows=num_rows, + show_sql=show_sql, + format=format, + sections=sections, + descriptions_max_chars=descriptions_max_chars, + compact=compact, ) - if fmt == "json": - payload: Dict[str, Any] = { - "model_name": model.name, - "description": truncated_model_desc, - "data_source": model.data_source, - "source_type": _source_type_for(model), - } - if show_sql: - payload["sql_table"] = model.sql_table - payload["sql"] = model.sql - if backing_info is not None: - payload["backing_query"] = backing_info - if show_sql and model.backing_query_sql: - payload["backing_query_sql"] = model.backing_query_sql - payload["default_time_dimension"] = model.default_time_dimension - payload["hidden"] = model.hidden - payload["meta"] = model.meta - payload["row_count"] = row_count - if show_sql: - payload["filters"] = model.filters - - # Columns - if "columns" in included_set: - col_payloads: List[Dict[str, Any]] = [] - for c in visible_columns: - # DEV-1480 key-presence (not ``or`` truthiness) so empty - # string ``sampled=""`` (all-NULL categorical) survives. - if c.name in profile_by_name: - sampled_cell = profile_by_name[c.name] - else: - sampled_cell = measure_profile.get(c.name) - col_payloads.append({ - "name": c.name, - "type": str(c.type), - "primary_key": c.primary_key, - **({"sql": c.sql} if show_sql else {}), - "allowed_aggregations": c.allowed_aggregations, - **({"filter": c.filter} if show_sql else {}), - "label": c.label, - "description": _truncate_description( - c.description, descriptions_max_chars, - ), - "meta": c.meta, - "sampled": sampled_cell, - # DEV-1480: structured top-50 list + true cardinality, - # surfaced only in the JSON shape (the markdown table - # text format is unchanged per the issue). - "sampled_values": profile_values_by_name.get(c.name), - "distinct_count": distinct_count_by_name.get(c.name), - }) - payload["columns"] = col_payloads - elif visible_columns: - payload["columns_names"] = [c.name for c in visible_columns] - - # Measures - if "measures" in included_set: - payload["measures"] = [ - { - "name": mm.name, - "formula": mm.formula, - "label": mm.label, - "description": _truncate_description( - mm.description, descriptions_max_chars, - ), - "meta": mm.meta, - } - for mm in model.measures - ] - elif model.measures: - payload["measures_names"] = [mm.name for mm in model.measures] - - # Aggregations - if "aggregations" in included_set: - payload["aggregations"] = [ - { - "name": a.name, - **({"formula": a.formula} if show_sql else {}), - "params": [ - ({"name": p.name, "sql": p.sql} if show_sql else {"name": p.name}) - for p in (a.params or []) - ], - "description": _truncate_description( - a.description, descriptions_max_chars, - ), - "meta": a.meta, - } - for a in model.aggregations - ] - elif model.aggregations: - payload["aggregations_names"] = [a.name for a in model.aggregations] - - # Joins - if "joins" in included_set: - payload["joins"] = [ - { - "target_model": j.target_model, - "join_pairs": j.join_pairs, - } - for j in model.joins - ] - elif model.joins: - payload["joins_names"] = [j.target_model for j in model.joins] - - # Reachable fields - if "reachable_fields" in included_set: - payload["reachable_dimensions"] = reach_dims - payload["reachable_measures"] = reach_measures - - # Samples - if "samples" in included_set: - payload["sample_data"] = sample_data - payload["sample_data_error"] = sample_error - if show_sql and sample_sql: - payload["sample_sql"] = sample_sql - - # Learnings (DEV-1357 v2) — Memory carries ``learning``, - # not ``body``; reading ``.body`` here would AttributeError - # the moment a memory matches and the caller asked for JSON - # output. - if "learnings" in included_set and relevant_learnings: - payload["learnings"] = [ - { - "id": memory.id, - "learning": memory.learning, - "matched_entities": sorted( - set(wanted) & set(memory.entities) - ), - } - for memory in relevant_learnings - ] - - # Top-level gating-state arrays (only when non-empty) - if names_only_sections: - payload["names_only_sections"] = names_only_sections - if omitted_sections: - payload["omitted_sections"] = omitted_sections - if unknown: - payload["unknown_sections"] = unknown - - return json.dumps(payload, indent=2, default=str) + @mcp.tool() + async def inspect( + entity_type: str, + reference: str | list[str] | None = None, + compact: bool = True, + format: str = "markdown", + num_rows: int = 3, + show_sql: bool = False, + sections: list[str] | None = None, + descriptions_max_chars: int | None = None, + ) -> str: + """Inspect EXACTLY one entity by reference and kind, a homogeneous + BATCH when ``reference`` is a list — or the whole COLLECTION at a kind + when ``reference`` is omitted / ``None``. + + A clean point-lookup: no fusion / ranking / cypher, and no bundled + memories. Use ``search`` instead when you want an entity surfaced *in + context* (with related memories and ranked neighbours). + + Collection (DEV-1667): omit ``reference`` (or pass ``None`` / ``[]``) + to list a whole kind. ``entity_type="model"`` lists all models grouped + by datasource (compact=True: one terse line per model; compact=False: + the full per-model tables). ``entity_type="datasource"`` lists all + datasources. Only ``model`` / ``datasource`` support the collection + view; other kinds raise. This subsumes ``models_summary`` / + ``list_datasources``. + + Batch (DEV-1612): pass a ``list`` of references that all share the one + ``entity_type``. Returns one rendered block per id, in input order, + each echoing its resolved canonical id (a ``## `` header in + markdown; a JSON array under ``format="json"``). Per-id resolution + errors are isolated — one bad id does not sink the batch (in JSON it + becomes a ``{"reference": ..., "error": ...}`` element). A single + ``str`` keeps its byte-for-byte single output; a one-element list is + still batch-framed. - if footer: - out_sections.append(footer) - return "\n\n".join(out_sections) + Args: + reference: The entity reference, or a list of references (batch). + Accepts canonical forms (``mydb``, ``mydb.orders``, + ``mydb.orders.amount``), bare names, join paths + (``orders.customers.region`` → resolved to the owning model), + and ``memory:`` for memories. Normalised via the shared + resolver; the normalised canonical id is echoed in the JSON + shape. + entity_type: REQUIRED. One of ``datasource``, ``model``, + ``column``, ``measure``, ``aggregation``, ``memory``. + Disambiguates the 3-part canonical collision (a name + shared by, e.g., a column and an aggregation) and asserts + the resolved kind — a mismatch returns a detailed error. + compact: When true (default): description-only for + column/measure/aggregation/datasource/memory; for + ``entity_type="model"`` a cheap schema skeleton (column / + measure / aggregation names + join targets, zero DB calls). + False returns the full render (and, for the datasource kind, + a per-model skeleton for each visible model). + format: ``"markdown"`` (default) or ``"json"``. + num_rows: Sample-data rows for ``entity_type="model"``. Ignored + (with a warning) for other kinds. + show_sql: Include generated SQL for ``entity_type="model"``. + Ignored (with a warning) for datasource/memory; a silent + no-op for column/measure/aggregation. + sections: Section subset for ``entity_type="model"``. Ignored + (with a warning) for other kinds. + descriptions_max_chars: Truncate description fields to this many + characters. Applies to every kind. + """ + return await InspectService(storage=storage, engine=engine).inspect( + reference=reference, + entity_type=entity_type, + compact=compact, + format=format, + num_rows=num_rows, + show_sql=show_sql, + sections=sections, + descriptions_max_chars=descriptions_max_chars, + ) # ----------------------------------------------------------------------- # Model creation and editing @@ -1873,14 +786,14 @@ async def _persist_sample( @mcp.tool() async def create_model( name: str, - sql_table: Optional[str] = None, - sql: Optional[str] = None, - data_source: Optional[str] = None, - description: Optional[str] = None, - columns: Optional[List[Dict[str, Any]]] = None, - measures: Optional[List[Dict[str, Any]]] = None, - query: Optional[Any] = None, - variables: Optional[Dict[str, Any]] = None, + sql_table: str | None = None, + sql: str | None = None, + data_source: str | None = None, + description: str | None = None, + columns: list[dict[str, Any]] | None = None, + measures: list[dict[str, Any]] | None = None, + query: Any | None = None, + variables: dict[str, Any] | None = None, ) -> str: """Create a new semantic model, either from a database table or from a query. @@ -1979,7 +892,7 @@ def _upsert_entity( id_field: str, changes: list, label: str, - ) -> Optional[str]: + ) -> str | None: """Upsert a named entity in *entity_list*. Returns an error string on validation failure, ``None`` on success. @@ -2014,23 +927,23 @@ def _upsert_entity( @mcp.tool() async def edit_model( model_name: str, - description: Optional[str] = None, - data_source: Optional[str] = None, - new_data_source: Optional[str] = None, - default_time_dimension: Optional[str] = None, - sql_table: Optional[str] = None, - sql: Optional[str] = None, - source_queries: Optional[List[Dict[str, Any]]] = None, + description: str | None = None, + data_source: str | None = None, + new_data_source: str | None = None, + default_time_dimension: str | None = None, + sql_table: str | None = None, + sql: str | None = None, + source_queries: list[dict[str, Any]] | None = None, query_variables: Any = _UNSET, - hidden: Optional[bool] = None, - columns: Optional[List[Dict[str, Any]]] = None, - measures: Optional[List[Dict[str, Any]]] = None, - aggregations: Optional[List[Dict[str, Any]]] = None, - joins: Optional[List[Dict[str, Any]]] = None, - add_filters: Optional[List[str]] = None, - remove_filters: Optional[List[str]] = None, - remove: Optional[Dict[str, List[str]]] = None, - meta: Optional[Dict[str, Any]] = _UNSET, + hidden: bool | None = None, + columns: list[dict[str, Any]] | None = None, + measures: list[dict[str, Any]] | None = None, + aggregations: list[dict[str, Any]] | None = None, + joins: list[dict[str, Any]] | None = None, + add_filters: list[str] | None = None, + remove_filters: list[str] | None = None, + remove: dict[str, list[str]] | None = None, + meta: dict[str, Any] | None = _UNSET, ) -> str: """Edit an existing model in a single call — update metadata, upsert columns/measures/aggregations/joins, manage filters, and remove entities. @@ -2097,7 +1010,7 @@ async def edit_model( return f"Model '{model_name}' not found." original_data_source = model.data_source - changes: List[str] = [] + changes: list[str] = [] # DEV-1375: track refresh-triggering changes so the post-save hook # knows whether to refresh just the touched columns or every # column on the model. @@ -2353,7 +1266,7 @@ async def edit_model( # changed the indexed text. Best-effort: any raise here is # captured into ``refresh_warnings`` so the save's success # status survives a flaky embedding API. - refresh_warnings: List[str] = [] + refresh_warnings: list[str] = [] if changed_columns or model_level_change or model_doc_changed: try: refresh_warnings = await handle_edit_refresh( @@ -2390,13 +1303,13 @@ async def edit_model( async def create_datasource( name: str, type: str, - host: Optional[str] = None, - port: Optional[int] = None, - database: Optional[str] = None, - username: Optional[str] = None, - password: Optional[str] = None, - connection_string: Optional[str] = None, - schema_name: Optional[str] = None, + host: str | None = None, + port: int | None = None, + database: str | None = None, + username: str | None = None, + password: str | None = None, + connection_string: str | None = None, + schema_name: str | None = None, auto_ingest: bool = True, ) -> str: """Create a database connection, verify it, and auto-ingest models. Use ${ENV_VAR} syntax in credentials to reference environment variables. @@ -2430,7 +1343,10 @@ async def create_datasource( ) ds = DatasourceConfig.model_validate(data) existed = await storage.get_datasource(name) is not None - await storage.save_datasource(ds) + try: + await storage.save_datasource(ds) + except ValueError as exc: + return f"Cannot create datasource: {exc}" verb = "replaced" if existed else "created" ok, msg = _test_connection(ds) @@ -2451,20 +1367,32 @@ async def create_datasource( return "\n".join(lines) raise + save_errors: list[str] = [] + saved_models = [] for model in models: - await storage.save_model(model) - - if not models: + try: + await storage.save_model(model) + saved_models.append(model) + except ValueError as exc: + # e.g. quoted case-variant tables — report and continue. + save_errors.append(f"- {model.name}: {exc}") + models = saved_models + + if not models and not save_errors: lines.append("No tables found to ingest.") schemas = _get_schemas(ds) if schemas: lines.append(f"Available schemas: {', '.join(schemas)}") - else: + elif models: lines.append(f"Ingested {len(models)} model(s):") for m in models: lines.append(f"- {m.name} ({len(m.columns)} columns, {len(m.measures)} measures)") lines.append("") - lines.append("Use models_summary and inspect_model to explore, then query to fetch data.") + lines.append("Use models_summary and inspect to explore, then query to fetch data.") + + if save_errors: + lines.append(f"Failed to save {len(save_errors)} model(s):") + lines.extend(save_errors) return "\n".join(lines) @@ -2472,18 +1400,17 @@ async def create_datasource( async def list_datasources() -> str: """List all configured database connections (names and types only, credentials are not shown). Use describe_datasource for connection details and status.""" names = await storage.list_datasources() - if not names: - return "No datasources configured. Use create_datasource to add a database connection." - lines = [] + # DEV-1667: rendering delegates to the shared renderer (also used by + # the ``inspect`` datasource collection view) — one code path. + pairs: list[tuple[str, str | None]] = [] for name in names: try: ds = await storage.get_datasource(name) - ds_type = ds.type if ds else "unknown" - lines.append(f"- {name} ({ds_type})") + pairs.append((name, ds.type if ds else "unknown")) except Exception as exc: logger.warning("Failed to load datasource '%s': %s", name, exc) - lines.append(f"- {name} (ERROR: invalid datasource config)") - return "\n".join(lines) + pairs.append((name, None)) + return render_datasource_list(pairs=pairs, fmt="markdown") @mcp.tool() async def describe_datasource( @@ -2556,7 +1483,7 @@ async def describe_datasource( @mcp.tool() async def edit_datasource( name: str, - description: Optional[str] = None, + description: str | None = None, ) -> str: """Update a datasource's metadata. @@ -2568,10 +1495,45 @@ async def edit_datasource( if ds is None: return f"Datasource '{name}' not found." + old_description = ds.description if description is not None: ds.description = description await storage.save_datasource(ds) + + # DEV-1549: the datasource embedding text now includes + # ``DatasourceConfig.description``, so an edit to the + # description must refresh the embedding inline — otherwise the + # persisted row stays stale until the next ``slayer ingest`` + # and description-only semantic matches silently miss. + # + # The save is already committed at this point. Per CodeRabbit + # round-7 review: the refresh is post-save and best-effort — + # log a warning if it raises and surface a partial-success + # message rather than telling the agent the save itself failed. + refresh_warning: str | None = None + if description is not None and description != old_description: + models_in_ds: list[SlayerModel] = [] + for model_name in await storage.list_models(data_source=name): + m = await storage.get_model(model_name, data_source=name) + if m is not None: + models_in_ds.append(m) + try: + await search_service.refresh_datasource( + name=name, + models=models_in_ds, + description=ds.description, + ) + except Exception as exc: # noqa: BLE001 — best-effort post-save refresh + logger.warning( + "edit_datasource refresh failed for %r: %s", name, exc, + ) + refresh_warning = str(exc) + if refresh_warning: + return ( + f"Datasource '{name}' updated. " + f"Warning: embedding refresh failed: {refresh_warning}" + ) return f"Datasource '{name}' updated." # ----------------------------------------------------------------------- @@ -2579,7 +1541,7 @@ async def edit_datasource( # ----------------------------------------------------------------------- @mcp.tool() - async def delete_model(name: str, data_source: Optional[str] = None) -> str: + async def delete_model(name: str, data_source: str | None = None) -> str: """Delete a semantic model. Args: @@ -2597,7 +1559,7 @@ async def delete_model(name: str, data_source: Optional[str] = None) -> str: return f"Model '{name}' not found." @mcp.tool() - async def validate_models(data_source: Optional[str] = None) -> str: + async def validate_models(data_source: str | None = None) -> str: """Diff persisted SLayer models against the live database schema(s). Returns a JSON-serialized list of pending delete operations @@ -2617,13 +1579,70 @@ async def validate_models(data_source: Optional[str] = None) -> str: ds = await storage.get_datasource(data_source) if ds is None: return f"Datasource '{data_source}' not found." - engine = SlayerQueryEngine(storage=storage) + # DEV-1656: reuse the closure engine (not a fresh per-call engine) so + # the schema-drift SQL client it opens is cached on the server's + # engine and disposed by ``mcp._slayer_engine.aclose()`` at teardown. try: entries = await engine.validate_models(data_source=data_source) except (sa.exc.OperationalError, sa.exc.DatabaseError) as exc: return _friendly_db_error(exc) return json.dumps([e.model_dump(mode="json") for e in entries], indent=2) + @mcp.tool() + async def recommend_root_model( + items: list[str], data_source: str | None = None, + root_hint: str | None = None, format: str = "markdown" # noqa: A002 + ) -> str: + """Recommend the root model (query ``source_model``) for a set of + ``model.column`` / ``model.metric`` items, and give each item's + join-qualified reference path from that root. + + Introspects the join graph and picks the model from which every + requested item is reachable (LEFT joins are directional; INNER + joins traverse both ways), minimizing total join hops. The returned + paths are ready to drop into a query whose ``source_model`` is the + recommended root — e.g. a joined column comes back as + ``customers.regions.name`` and a root-owned one as ``status``; + aggregation suffixes (``:sum``) are preserved. + + When no single model reaches everything, ``root_model`` is null and + ``coverage`` lists the best partial roots so you can split the + request into a multi-stage query. + + Args: + items: entity references (``orders.revenue``, ``customers.name``, + ``orders.revenue:sum``, bare ``aov`` for a saved metric...). + data_source: optional datasource scope; when omitted, names + resolve via the datasource-priority list. All items must + resolve to a single datasource. + root_hint: optional intended root — a bare model name or + ``.`` within the resolved datasource. + Honored when it reaches every item (overriding the min-hops + pick, so you can force a bridge model that owns none of the + items); otherwise the auto-pick is used and a warning + explains why. Resolved after the datasource is determined, + so it cannot pick the datasource. + format: ``"markdown"`` (default) or ``"json"``. + """ + fmt = format.lower().strip() + if fmt not in ("markdown", "json"): + return ( + f"recommend_root_model failed: unknown format '{format}'. " + f"Use 'markdown' or 'json'." + ) + # DEV-1656: reuse the closure engine (see validate_models above). + try: + rec = await engine.recommend_root_model( + items, data_source=data_source, root_hint=root_hint + ) + except AmbiguousModelError as exc: + return _ambiguous_with_mcp_hint(exc) + except (ValueError, EntityResolutionError) as exc: + return f"recommend_root_model failed: {exc}" + if fmt == "json": + return json.dumps(rec.model_dump(mode="json"), indent=2) + return render_recommendation_markdown(rec) + @mcp.tool() async def delete_datasource(name: str) -> str: """Delete a datasource configuration. @@ -2677,7 +1696,7 @@ async def ingest_datasource_models(datasource_name: str, include_tables: str = " ) @mcp.tool() - async def set_datasource_priority(priority: List[str]) -> str: + async def set_datasource_priority(priority: list[str]) -> str: """Configure how SLayer disambiguates bare model names that exist in multiple datasources. @@ -2723,7 +1742,8 @@ def _format_resolution_error(exc: Exception) -> str: async def save_memory( learning: str, linked_entities: Any, - id: Optional[str] = None, # noqa: A002 — MCP arg name + id: str | None = None, # noqa: A002 — MCP arg name + description: str | None = None, ) -> str: """Save an agent memory: a free-form note plus the SLayer entities it concerns. @@ -2789,6 +1809,7 @@ async def save_memory( learning=learning, linked_entities=linked_entities, id=id, + description=description, ) except ( EntityResolutionError, @@ -2825,17 +1846,19 @@ async def forget_memory(id: Any) -> str: # noqa: A002 — MCP arg name # ---------- DEV-1375: semantic search ----------------------------- - search_service = SearchService(storage=storage) + # DEV-1516: pass the engine so the search service's post-fusion + # column-hit hook can auto-refresh stale categorical columns. + search_service = SearchService(storage=storage, engine=engine) @mcp.tool() async def search( - entities: Optional[List[str]] = None, + entities: list[str] | None = None, query: Any = None, - question: Optional[str] = None, - datasource: Optional[str] = None, - max_memories: int = 5, - max_example_queries: int = 2, - max_entities: int = 5, + question: str | None = None, + datasource: str | None = None, + max_results: int = 10, + cypher_filter: str | None = None, + compact: bool = True, ) -> str: """Up to three-channel semantic search over memories + canonical entities. @@ -2853,23 +1876,19 @@ async def search( non-hidden column / named measure / aggregation). Channel 3 (dense embedding similarity, optional): runs when - ``question`` is supplied AND the ``embedding_search`` extra is + ``question`` is supplied AND the ``advanced_search`` extra is installed AND a provider API key is configured for the active embedding model. Cosine similarity between the question embedding and persisted entity/memory embeddings. Skipped with a single warning into ``SearchResponse.warnings`` when any precondition fails — tantivy + BM25 continue to work. - Memory rankings from every active channel and entity rankings - from channels 2 and 3 are fused via Reciprocal Rank Fusion - (k=60). Query-bearing memories (those saved with an attached - ``SlayerQuery``) are partitioned into ``example_queries`` and - capped independently from learning-only ``memories`` so bulky - example queries cannot crowd out small notes. + All hits (memories, example queries, entities) are fused via + Reciprocal Rank Fusion (k=60) into a single ranked + ``results`` list capped at ``max_results``. Empty input (no entities, no query, no question) returns the - newest ``max_memories`` learning-only memories and the newest - ``max_example_queries`` query-bearing memories, with a warning. + newest memories capped at ``max_results``, with a warning. Args: entities: Canonical entity reference strings. @@ -2884,11 +1903,15 @@ async def search( a memory spanning multiple datasources surfaces from each. BM25 / IDF stats reflect only the filtered subset. Unknown datasource raises ``ValueError``. - max_memories: Cap on returned learning-only memory hits - (default 5). - max_example_queries: Cap on returned query-bearing memory - hits (default 2 — they're bulky). - max_entities: Cap on returned entity hits (default 5). + max_results: Maximum total number of hits to return (default 10). + cypher_filter: Optional openCypher MATCH query returning + ``… AS id`` that pre-filters all three channels to the + returned canonical IDs. When ``advanced_search`` is not + installed, only simple + ``MATCH (n:Label1:Label2) RETURN n.id AS id`` patterns are + supported as a kind filter (multi-label uses union + semantics; allowed labels: Memory, Datasource, Model, + Column, Measure, Aggregation). """ try: response = await search_service.search( @@ -2896,27 +1919,23 @@ async def search( query=query, question=question, datasource=datasource, - max_memories=max_memories, - max_example_queries=max_example_queries, - max_entities=max_entities, + max_results=max_results, + cypher_filter=cypher_filter, + compact=compact, ) - except ( - EntityResolutionError, - AmbiguousModelError, - ValueError, - ) as exc: + except (SlayerError, ValueError) as exc: return _format_resolution_error(exc) return response.model_dump_json(indent=2) return mcp -def _build_dict(**kwargs: Any) -> Dict[str, Any]: +def _build_dict(**kwargs: Any) -> dict[str, Any]: """Build a dict from keyword arguments, excluding None values.""" return {k: v for k, v in kwargs.items() if v is not None} -def _format_table(data: List[Dict[str, Any]], columns: List[str], max_rows: int = 50) -> str: +def _format_table(data: list[dict[str, Any]], columns: list[str], max_rows: int = 50) -> str: """Format data as a pipe-separated table (used for sample data display).""" if not data: return "No results." @@ -2936,14 +1955,14 @@ def _format_table(data: List[Dict[str, Any]], columns: List[str], max_rows: int return result -def _format_json(data: List[Dict[str, Any]], columns: List[str]) -> str: +def _format_json(data: list[dict[str, Any]], columns: list[str]) -> str: """Format data as JSON array.""" import json return json.dumps(data, default=str) -def _format_csv(data: List[Dict[str, Any]], columns: List[str]) -> str: +def _format_csv(data: list[dict[str, Any]], columns: list[str]) -> str: """Format data as CSV.""" if not data: return "" @@ -2968,7 +1987,7 @@ def _format_output(result: SlayerResponse, fmt: str) -> str: return _format_json(data=result.data, columns=result.columns) -def _format_field_meta(entries: Dict[str, Any]) -> List[str]: +def _format_field_meta(entries: dict[str, Any]) -> list[str]: """Format a dict of field metadata entries into lines.""" lines = [] for col, fm in entries.items(): diff --git a/slayer/help/topics/00_intro.md b/slayer/memories/help_content/00_intro.md similarity index 84% rename from slayer/help/topics/00_intro.md rename to slayer/memories/help_content/00_intro.md index e8d474f7..3bca48ad 100644 --- a/slayer/help/topics/00_intro.md +++ b/slayer/memories/help_content/00_intro.md @@ -74,7 +74,8 @@ in `measures`: 5. It's critically important to choose the right source_model for a query. Put EXTRA THOUGHT into that. 6. When picking a measure for a query, MAKE SURE to consider the underlying values range - shown under "values" in inspect_model. If that's all NULL, maybe that's not the measure you want. + shown under "values" in `inspect(reference="", entity_type="model")`. If that's all + NULL, maybe that's not the measure you want. 7. **`time_shift`, `change`, `change_pct` can only wrap aggregated measures** — e.g. `time_shift(revenue:sum, -1)`, `change(amount:avg)`. They cannot wrap @@ -84,9 +85,13 @@ in `measures`: ## Deep dives -Call `help(topic='...')` for detail pages on specific subjects. -Available topics: `queries`, `formulas`, `aggregations`, `transforms`, -`time`, `filters`, `joins`, `models`, `extending`, `workflow`. +Each subject below is its own help memory. Read one with +`inspect(reference="memory:help.", entity_type="memory")`, e.g. +`inspect(reference="memory:help.queries", entity_type="memory")`. Available topics: +`memory:help.queries`, `memory:help.formulas`, `memory:help.aggregations`, +`memory:help.transforms`, `memory:help.time`, `memory:help.filters`, +`memory:help.joins`, `memory:help.models`, `memory:help.extending`, +`memory:help.workflow`. -Recommended starting order for an unfamiliar agent: `help(topic='workflow')` for -tool-chaining, then `help(topic='queries')` for the query model. +Recommended starting order for an unfamiliar agent: `memory:help.workflow` for +tool-chaining, then `memory:help.queries` for the query model. diff --git a/slayer/help/topics/01_queries.md b/slayer/memories/help_content/01_queries.md similarity index 84% rename from slayer/help/topics/01_queries.md rename to slayer/memories/help_content/01_queries.md index 41e7a304..0acba267 100644 --- a/slayer/help/topics/01_queries.md +++ b/slayer/memories/help_content/01_queries.md @@ -16,6 +16,7 @@ see the `query` tool's own arg documentation. | `limit` / `offset` | Row slicing on the final result. | | `main_time_dimension` | Which time dim drives transforms when 2+ are present. | | `whole_periods_only` | Snap `date_range` to bucket edges; drop incomplete current bucket. | +| `distinct_dimension_values` | Default `true` — auto-dedup dim-only queries (emit `GROUP BY `). Set `false` to emit raw rows (no top-level GROUP BY); rejects any measure reference in `measures` / `filters` / `order`. | ## Evaluation order (the SQL the generator builds) @@ -27,7 +28,7 @@ see the `query` tool's own arg documentation. 6. ORDER BY → LIMIT / OFFSET. Knowing which stage your filter lands in is why the auto-routing works. See -`help(topic='filters')`. +`memory:help.filters`. ## Dimensions vs time_dimensions on the same column @@ -81,7 +82,7 @@ query; `customers.regions.name` walks `orders → customers → regions`; ## See also -- `help(topic='formulas')` — the colon syntax and arithmetic that power `measures`. -- `help(topic='filters')` — operators, WHERE vs HAVING, post-filters. -- `help(topic='time')` — granularities, whole_periods_only, `last()` distinctions. -- `help(topic='extending')` — `source_model` as a `ModelExtension` or a query name. +- `memory:help.formulas` — the colon syntax and arithmetic that power `measures`. +- `memory:help.filters` — operators, WHERE vs HAVING, post-filters. +- `memory:help.time` — granularities, whole_periods_only, `last()` distinctions. +- `memory:help.extending` — `source_model` as a `ModelExtension` or a query name. diff --git a/slayer/help/topics/02_formulas.md b/slayer/memories/help_content/02_formulas.md similarity index 76% rename from slayer/help/topics/02_formulas.md rename to slayer/memories/help_content/02_formulas.md index b190af80..38ada90b 100644 --- a/slayer/help/topics/02_formulas.md +++ b/slayer/memories/help_content/02_formulas.md @@ -43,14 +43,16 @@ Inside a field, use a dict to name the result: ## Nesting -Transforms (see `help(topic='transforms')`) can wrap measures, arithmetic, or -each other. Arbitrary nesting is allowed: +Transforms (see `memory:help.transforms`) can wrap measures, arithmetic, or +each other, but they follow compatibility rules. In particular `change`, +`change_pct`, and `time_shift` must wrap an aggregated measure directly; window +transforms such as `cumsum` can wrap those (the reverse is rejected): ```json { "source_model": "orders", "measures": [ - {"formula": "change(cumsum(revenue:sum))", "name": "cumsum_delta"}, + {"formula": "cumsum(change(revenue:sum))", "name": "cumulative_change"}, {"formula": "cumsum(revenue:sum / *:count)", "name": "running_aov"} ], "time_dimensions": [{"dimension": "created_at", "granularity": "month"}] @@ -92,18 +94,18 @@ transforms (`cumsum`, `change`, `time_shift`, …) are rejected at model save. The same parser powers `filters`. Left and right of an operator can be a dimension, a measure with `:agg`, or a transform expression. See -`help(topic='filters')` for operators and routing. +`memory:help.filters` for operators and routing. ## Gotchas -- Bare measure renames (`{"formula": "*:count", "name": "n"}`) cannot be - referenced by `n` in `filters` — reference the original `*:count` instead. +- Bare measure renames (`{"formula": "*:count", "name": "n"}`) can be + referenced by either `n` or `*:count` in `filters`. - Formulas validate measure names against the source model at query time. - If you get "measure not found", call `inspect_model` and check the actual - measure list. + If you get "measure not found", call `inspect(reference="", entity_type="model")` + and check the actual measure list. ## See also -- `help(topic='aggregations')` — the full list of `:agg` options. -- `help(topic='transforms')` — `cumsum`, `change`, `time_shift`, etc. -- `help(topic='joins')` — dotted paths like `customers.score`. +- `memory:help.aggregations` — the full list of `:agg` options. +- `memory:help.transforms` — `cumsum`, `change`, `time_shift`, etc. +- `memory:help.joins` — dotted paths like `customers.score`. diff --git a/slayer/help/topics/03_aggregations.md b/slayer/memories/help_content/03_aggregations.md similarity index 92% rename from slayer/help/topics/03_aggregations.md rename to slayer/memories/help_content/03_aggregations.md index 9b46ad5b..06688dfb 100644 --- a/slayer/help/topics/03_aggregations.md +++ b/slayer/memories/help_content/03_aggregations.md @@ -44,7 +44,7 @@ Don't confuse: - `:first`/`:last` aggregation — per-group record's earliest/latest value. - `first(x)`/`last(x)` transform — broadcasts the earliest/most recent bucket's - aggregated value to every row. See `help(topic='transforms')`. + aggregated value to every row. See `memory:help.transforms`. ## Windowed sum and average @@ -117,6 +117,6 @@ arg, and `price:weighted_avg(weight=revenue)` overrides. ## See also -- `help(topic='formulas')` — where `:agg` fits in the broader formula language. -- `help(topic='transforms')` — `first()`/`last()` transforms vs `:first`/`:last` aggregations. -- `help(topic='models')` — declaring measures and their `allowed_aggregations`. +- `memory:help.formulas` — where `:agg` fits in the broader formula language. +- `memory:help.transforms` — `first()`/`last()` transforms vs `:first`/`:last` aggregations. +- `memory:help.models` — declaring measures and their `allowed_aggregations`. diff --git a/slayer/help/topics/04_transforms.md b/slayer/memories/help_content/04_transforms.md similarity index 77% rename from slayer/help/topics/04_transforms.md rename to slayer/memories/help_content/04_transforms.md index bbc86461..7d91e38a 100644 --- a/slayer/help/topics/04_transforms.md +++ b/slayer/memories/help_content/04_transforms.md @@ -9,10 +9,10 @@ becomes an extra CTE in the generated SQL. | Transform | Purpose | SQL strategy | |-----------|---------|--------------| | `cumsum(x)` | Running total over time | Window: `SUM(x) OVER (PARTITION BY dims ORDER BY time)` | -| `time_shift(x, n)` | Value N periods back/ahead | Self-join CTE with INTERVAL offset | +| `time_shift(x, n)` | Value N time buckets back/ahead (calendar-aware) | Self-join CTE with INTERVAL offset | | `time_shift(x, n, 'year')` | Value at a different granularity offset (e.g. YoY) | Self-join CTE with INTERVAL offset | -| `change(x)` | `x − previous(x)` | Desugars to `x − time_shift(x, -1)` | -| `change_pct(x)` | `(x − previous) / previous` | Desugars to `(x − ts) / ts` where `ts = time_shift(x, -1)` | +| `change(x)` | Period-over-period difference (partition-safe, resets per group) | Desugars to `x − time_shift(x, -1)` | +| `change_pct(x)` | Period-over-period % change, e.g. month-over-month growth (partition-safe; NULL when the prior period's value is 0 or missing) | Desugars to `CASE WHEN ts != 0 THEN (x − ts) / ts END` where `ts = time_shift(x, -1)` | | `lag(x, n)` / `lead(x, n)` | N rows back / ahead | `LAG` / `LEAD` window fn, partitioned by dimensions | | `consecutive_periods(predicate)` | Current trailing run length where predicate is true | Staged window CTEs with reset groups | | `rank(x[, partition_by=...])` | Rank by x, descending; ties skip ranks | `RANK() OVER ([PARTITION BY ...] ORDER BY x DESC)` | @@ -32,6 +32,10 @@ the previous/next value. Consequences: - No NULLs at the first / last rows when the database actually has the data. - Handles **gaps** in the time series correctly — shifts by calendar, not by row. +- **Partition-safe**: the join matches on all non-time dimensions as well as the + shifted time column (`ON base.month = shifted.month AND base.store = + shifted.store`), so each group's series is compared only against itself and + resets cleanly per group. - Slightly heavier SQL. `lag` and `lead` use SQL `LAG` / `LEAD`: @@ -43,6 +47,17 @@ the previous/next value. Consequences: Use `time_shift`, `change`, `change_pct` unless you have a specific reason to prefer `lag` / `lead`. +**Intent recipes:** + +- Month-over-month / period-over-period growth → `change_pct(revenue:sum)` + with a `time_dimensions` entry at the desired granularity. Prefer this over + hand-building the ratio from `time_shift` — same partition-safe self-join, + cleaner SQL. +- Absolute period-over-period delta → `change(revenue:sum)`. +- Comparing against a *different* grain than the query's (e.g. year-over-year + on a monthly series), or using the shifted value as a term in custom + arithmetic → `time_shift(revenue:sum, -1, 'year')`. + ## Time dimension requirement All time-ordered transforms (`cumsum`, `time_shift`, `change`, `change_pct`, @@ -139,10 +154,12 @@ dotted paths work too: `partition_by=customers.region`. Raw `OVER (...)` SQL inside a `ModelMeasure.formula` or filter string is rejected with an actionable error pointing at the rank-family / `first()` / -`last()` / `lag()` / `lead()` transforms. For non-standard window expressions, -define a `Column` whose `sql` is the window expression and filter on the -column — SLayer auto-promotes the predicate to a post-aggregation outer -`WHERE`. +`last()` / `lag()` / `lead()` transforms. As an advanced escape hatch for a +non-standard window expression, define a `Column` whose `sql` is that window +expression and filter on the column — {{product}} auto-promotes the predicate to a +post-aggregation outer `WHERE`. This is the one documented exception to the +row-level `Column.sql` rule (see `memory:help.models`); prefer the built-in +transforms whenever they cover the need. ## first() and last() — broadcast transforms @@ -165,6 +182,6 @@ Useful for filtering on trend: `"filters": ["last(change(revenue:sum)) < 0"]`. ## See also -- `help(topic='time')` — granularity, whole_periods_only, main_time_dimension. -- `help(topic='filters')` — filtering on transform outputs. -- `help(topic='aggregations')` — `:first`/`:last` aggregation vs `first()`/`last()` transform. +- `memory:help.time` — granularity, whole_periods_only, main_time_dimension. +- `memory:help.filters` — filtering on transform outputs. +- `memory:help.aggregations` — `:first`/`:last` aggregation vs `first()`/`last()` transform. diff --git a/slayer/help/topics/05_time.md b/slayer/memories/help_content/05_time.md similarity index 73% rename from slayer/help/topics/05_time.md rename to slayer/memories/help_content/05_time.md index 9f222c57..26627152 100644 --- a/slayer/help/topics/05_time.md +++ b/slayer/memories/help_content/05_time.md @@ -1,6 +1,6 @@ # Time -Time is the most load-bearing dimension in most analytical queries. SLayer +Time is the most load-bearing dimension in most analytical queries. {{product}} treats it specially in a few places. ## time_dimensions vs plain dimensions on the same column @@ -39,7 +39,7 @@ first/last bucket may be partial) — use `whole_periods_only` for that. ## whole_periods_only -When `true`, SLayer snaps the `date_range` to the granularity's bucket edges +When `true`, {{product}} snaps the `date_range` to the granularity's bucket edges and drops the current incomplete bucket. Useful when a dashboard should not show "this month is half-done, the bar looks tiny": @@ -67,16 +67,16 @@ Without any `time_dimensions` entry, transforms will error. Set ## The three meanings of "last" — don't mix them up -SLayer has **three** distinct things named `last`: +{{product}} has **three** distinct things named `last`: 1. `:last(time_col)` — the **aggregation**. Per group, returns the value from - the record with the latest `time_col`. See `help(topic='aggregations')`. + the record with the latest `time_col`. See `memory:help.aggregations`. 2. `last(x)` — the **transform**. Broadcasts the aggregated value from the - most recent time bucket to every row. See `help(topic='transforms')`. + most recent time bucket to every row. See `memory:help.transforms`. 3. `last(…)` inside a `filters` string (e.g. `"last(change(revenue:sum)) < 0"`) - — a post-filter on the transform output. See `help(topic='filters')`. + — a post-filter on the transform output. See `memory:help.filters`. They all concern "latest something" but operate on different levels: record / bucket / filter. Pick the one that matches your question. @@ -89,8 +89,7 @@ bucket / filter. Pick the one that matches your question. "measures": [ "revenue:sum", {"formula": "time_shift(revenue:sum, -1, 'year')", "name": "prev_year"}, - {"formula": "revenue:sum / time_shift(revenue:sum, -1, 'year') - 1", - "name": "yoy_growth"} + {"formula": "change_pct(revenue:sum, -1, 'year')", "name": "yoy_growth"} ], "time_dimensions": [{ "dimension": "created_at", "granularity": "month", @@ -99,8 +98,14 @@ bucket / filter. Pick the one that matches your question. } ``` +`change_pct(revenue:sum, -1, 'year')` is the safe way to express YoY % growth: it +desugars to the same prior-year `time_shift` ratio but guards the denominator +(it returns `NULL` when the prior-year value is `0` or missing, instead of erroring). +Writing the ratio by hand as `revenue:sum / time_shift(revenue:sum, -1, 'year') - 1` +has no such guard. + ## See also -- `help(topic='transforms')` — the transform family. -- `help(topic='aggregations')` — `:first` and `:last` aggregations. -- `help(topic='queries')` — where `time_dimensions` and `main_time_dimension` sit. +- `memory:help.transforms` — the transform family. +- `memory:help.aggregations` — `:first` and `:last` aggregations. +- `memory:help.queries` — where `time_dimensions` and `main_time_dimension` sit. diff --git a/slayer/help/topics/06_filters.md b/slayer/memories/help_content/06_filters.md similarity index 86% rename from slayer/help/topics/06_filters.md rename to slayer/memories/help_content/06_filters.md index 51a369d5..07b1f56b 100644 --- a/slayer/help/topics/06_filters.md +++ b/slayer/memories/help_content/06_filters.md @@ -1,7 +1,7 @@ # Filters Filters are formula strings. They go in the query's `filters` list and/or on -a model's `filters` list. SLayer routes them to the right SQL stage +a model's `filters` list. {{product}} routes them to the right SQL stage automatically — there is no explicit HAVING keyword. ## Operators @@ -45,7 +45,7 @@ Multiple entries in the `filters` list are AND-ed: - Filter references a transform or computed field (e.g. `change(revenue:sum) > 0`) → **post-filter** on an outer wrapper. -Inner and outer filters can mix in one query — SLayer splits them. +Inner and outer filters can mix in one query — {{product}} splits them. ## Filtering on computed measures @@ -85,9 +85,9 @@ canonical alias literally shadows a source column on the same model is also rejected (the colon-form filter would otherwise be ambiguous). Cross-model agg-ref filters with rename (`customers.revenue:sum >= 100`) are NOT yet auto-resolved in any form — neither the colon syntax nor the -user alias resolves. As a workaround until DEV-1445 lands, restructure -as a multi-stage `source_queries` model so the cross-model measure -becomes local in the downstream stage. +user alias resolves. As a workaround, restructure as a multi-stage +`source_queries` model so the cross-model measure becomes local in the +downstream stage. ## Filtered columns — CASE WHEN inside an aggregate @@ -122,6 +122,6 @@ These are WHERE-only. They do not reference measures or transforms. ## See also -- `help(topic='formulas')` — parsing rules shared with `measures`. -- `help(topic='transforms')` — the transforms you can wrap in a filter. -- `help(topic='queries')` — where filters sit in the evaluation order. +- `memory:help.formulas` — parsing rules shared with `measures`. +- `memory:help.transforms` — the transforms you can wrap in a filter. +- `memory:help.queries` — where filters sit in the evaluation order. diff --git a/slayer/help/topics/07_joins.md b/slayer/memories/help_content/07_joins.md similarity index 76% rename from slayer/help/topics/07_joins.md rename to slayer/memories/help_content/07_joins.md index 6a39b3ce..68ce4fd7 100644 --- a/slayer/help/topics/07_joins.md +++ b/slayer/memories/help_content/07_joins.md @@ -1,6 +1,6 @@ # Joins -SLayer models relate to each other via **joins**. Only LEFT JOIN is supported +{{product}} models relate to each other via **joins**. Only LEFT JOIN is supported — joins are used for enrichment, not set operations. ## Declaring joins @@ -29,7 +29,7 @@ In **queries**, use dots: - Measure: `{"measures": ["customers.*:count"]}` - Cross-model transform: `{"measures": [{"formula": "cumsum(customers.score:avg)"}]}` -SLayer walks the join graph via BFS and inserts the LEFT JOINs. +{{product}} walks the join graph via BFS and inserts the LEFT JOINs. In **SQL snippets** (dimension `sql`, measure `sql`, model `filters`), use `__` instead of dots, because dots aren't valid SQL: @@ -48,7 +48,7 @@ A dimension from a joined model is just another column to GROUP BY — no cardinality issue. A **measure** from a joined model is different: a LEFT JOIN can duplicate rows, so aggregating after the join would double-count. -SLayer splits any query containing a cross-model measure: it evaluates that +{{product}} splits any query containing a cross-model measure: it evaluates that measure in a scoped sub-query (same dimensions, scoped to the joined model), then LEFT-JOINs the result back on the shared dimensions. @@ -62,7 +62,10 @@ Upshot: } ``` -gives exactly the same answer as: +gives the same answer as the following — but only for customers that appear on +at least one order. Because the first query is rooted at `orders` and enriches +via LEFT JOIN, it omits customers with no orders; the `customers`-rooted query +below includes them (with a zero count), so the row sets can differ: ```json { @@ -99,6 +102,6 @@ filters: ## See also -- `help(topic='models')` — declaring joins and the `__` alias convention. -- `help(topic='extending')` — adding ad-hoc joins via `ModelExtension`. -- `help(topic='queries')` — dotted dimensions and time dimensions. +- `memory:help.models` — declaring joins and the `__` alias convention. +- `memory:help.extending` — adding ad-hoc joins via `ModelExtension`. +- `memory:help.queries` — dotted dimensions and time dimensions. diff --git a/slayer/help/topics/08_models.md b/slayer/memories/help_content/08_models.md similarity index 86% rename from slayer/help/topics/08_models.md rename to slayer/memories/help_content/08_models.md index 1069c45c..7b30e6a0 100644 --- a/slayer/help/topics/08_models.md +++ b/slayer/memories/help_content/08_models.md @@ -31,9 +31,10 @@ columns: ``` Types: `string`, `number`, `boolean`, `time`, `date`. `label` is optional and -propagates to query result metadata. A column's `sql` is a **row-level** -expression, not an aggregate. Plain column names are fine; for complex -expressions prefix with the model name: +propagates to query result metadata. A column's `sql` is normally a +**row-level** expression, not an aggregate (the one exception is the advanced +window-expression escape hatch in `memory:help.transforms`). Plain column names +are fine; for complex expressions prefix with the model name: ```yaml columns: @@ -105,10 +106,10 @@ model use `__` to encode the original join path: | `customers.regions.name` | `customers__regions__name` | | `revenue:sum` | `revenue_sum` | -See `help(topic='extending')` for multi-stage queries using this. +See `memory:help.extending` for multi-stage queries using this. ## See also -- `help(topic='joins')` — `joins` list and the `__` SQL alias convention. -- `help(topic='extending')` — inline model extension for one-off dims/filters. -- `help(topic='filters')` — model-level `filters` and filtered measures. +- `memory:help.joins` — `joins` list and the `__` SQL alias convention. +- `memory:help.extending` — inline model extension for one-off dims/filters. +- `memory:help.filters` — model-level `filters` and filtered measures. diff --git a/slayer/help/topics/09_extending.md b/slayer/memories/help_content/09_extending.md similarity index 92% rename from slayer/help/topics/09_extending.md rename to slayer/memories/help_content/09_extending.md index 3e6eeb65..96567aeb 100644 --- a/slayer/help/topics/09_extending.md +++ b/slayer/memories/help_content/09_extending.md @@ -85,6 +85,6 @@ refreshed only when you save the model again, never during execution. ## See also -- `help(topic='models')` — permanent model shape; `hidden: true` for internal ones. -- `help(topic='joins')` — the `__` vs dot convention for joined paths. -- `help(topic='workflow')` — when to extend inline vs edit the stored model. +- `memory:help.models` — permanent model shape; `hidden: true` for internal ones. +- `memory:help.joins` — the `__` vs dot convention for joined paths. +- `memory:help.workflow` — when to extend inline vs edit the stored model. diff --git a/slayer/help/topics/10_workflow.md b/slayer/memories/help_content/10_workflow.md similarity index 68% rename from slayer/help/topics/10_workflow.md rename to slayer/memories/help_content/10_workflow.md index a35ab95f..98589e0e 100644 --- a/slayer/help/topics/10_workflow.md +++ b/slayer/memories/help_content/10_workflow.md @@ -6,17 +6,17 @@ tool-by-tool documentation, which covers what each one does in isolation. ## Discovery — "what data is here?" ```text -1. list_datasources() # pick a datasource -2. models_summary(datasource_name="mydb") # brief list of its models -3. inspect_model(model_name="orders") # dimensions, measures, sample rows, SQL +1. list_datasources() # pick a datasource +2. models_summary(datasource_name="mydb") # brief list of its models +3. inspect(reference="orders", entity_type="model") # columns, measures, sample rows, SQL ``` `models_summary` gives one line per model with just names + descriptions of its columns and measures and the list of joined models — pick the right one without the -weight of a full `inspect_model` call. +weight of a full `inspect(entity_type="model")` call. -`inspect_model` with `num_rows` returns live sample data — helpful for guessing -what values a column actually holds before writing a filter. +`inspect(reference="", entity_type="model")` with `num_rows` returns live sample +data — helpful for guessing what values a column actually holds before writing a filter. ## Building a query @@ -55,7 +55,7 @@ Two paths. - Missing a saved aggregated formula? `edit_model` with a `measures` upsert. Example: `measures=[{"name": "avg_margin", "formula": "margin:sum / *:count"}]`. - One-off concept for a single query? Use `ModelExtension` inside - `source_model` instead of editing the model — see `help(topic='extending')`. + `source_model` instead of editing the model — see `memory:help.extending`. - Multi-stage result you'd like to reuse? `create_model` with a `query` parameter persists the computed shape as a new model. @@ -63,21 +63,24 @@ Two paths. | Error message fragment | What to check | |------------------------|--------------| -| "Measure X not found" | `inspect_model` — spelled right, or on a joined model? | -| "Aggregation Y not allowed on measure X" | `allowed_aggregations` whitelist — see `help(topic='aggregations')`. | +| "Measure X not found" | `inspect(reference="", entity_type="model")` — spelled right, or on a joined model? | +| "Aggregation Y not allowed on measure X" | `allowed_aggregations` whitelist — see `memory:help.aggregations`. | | "Unresolvable dot path" | Missing `joins` entry or a typo in the target_model. | | "Time dimension required" | Transform needs a time dim — set `time_dimensions` or `main_time_dimension`. | | "Datasource 'X' not found" | `list_datasources`. | | Database connection errors | `describe_datasource(name=...)` runs a test query and surfaces the error. | -## When to reach for help() +## When to reach for the concept topics + +Each topic below is a help memory — read it with +`inspect(reference="memory:help.", entity_type="memory")`. - Unfamiliar colon/aggregation/transform output in a tool arg doc → - `help(topic='aggregations')` or `help(topic='transforms')`. -- Wondering why a filter didn't do what you expected → `help(topic='filters')`. -- Need to compose queries or bucket an aggregate → `help(topic='extending')`. + `memory:help.aggregations` or `memory:help.transforms`. +- Wondering why a filter didn't do what you expected → `memory:help.filters`. +- Need to compose queries or bucket an aggregate → `memory:help.extending`. ## See also -- `help(topic='queries')` — the anatomy of a single query. -- `help(topic='extending')` — multi-stage queries and inline model extension. +- `memory:help.queries` — the anatomy of a single query. +- `memory:help.extending` — multi-stage queries and inline model extension. diff --git a/slayer/memories/help_seed.py b/slayer/memories/help_seed.py new file mode 100644 index 00000000..906d3140 --- /dev/null +++ b/slayer/memories/help_seed.py @@ -0,0 +1,227 @@ +"""DEV-1658: SLayer's conceptual help, seeded as predefined memories. + +The old standalone ``help()`` tool/subcommand duplicated the memory system with +a fixed content set. Instead, the topic bodies under ``help_content/*.md`` are +seeded as real memories with fixed ids (``help.intro`` … ``help.workflow``) and +retrieved through the ordinary ``inspect(entity_type="memory")`` / ``search`` +surfaces. + +``seed_help_memories(storage)`` is idempotent: upsert-always, but it skips the +write (and the embedding fan-out) when the stored ``learning`` + ``description`` +already match the shipped content, so a warm store is a cheap no-op. Seeded +memories carry **no entities**, so they never surface in a model's Learnings +section (that section filters by entity overlap). + +Content lives in ``help_content/NN_name.md``; the ``NN_`` prefix fixes the +teaching order and is stripped to form the topic key. ``00_intro`` is the entry +point that lists the deep-dive topics. +""" + +from __future__ import annotations + +import re +from collections import Counter +from collections.abc import Mapping, Sequence +from importlib.resources import files + +from pydantic import BaseModel + +from slayer.storage.base import StorageBackend + +_CONTENT_SUBDIR = "help_content" +_ID_PREFIX = "help." + +#: Authored one-line previews (<=500 chars) surfaced by search(compact=True) +#: and inspect(compact=True). Keyed by the topic key (``NN_`` prefix stripped). +_DESCRIPTIONS: dict[str, str] = { + "intro": "What {{product}} is, the core entities, the query shape, and the biggest gotchas.", + "queries": "Anatomy of a SlayerQuery: source_model, measures, dimensions, filters, order, limit.", + "formulas": "Writing measure formulas: colon aggregations, arithmetic, and saved measures.", + "aggregations": "Built-in and custom aggregations, colon syntax, *:count, and allowed_aggregations.", + "transforms": "cumsum, time_shift, change, the rank family, lag/lead, and their wrapping rules.", + "time": "Time dimensions, granularities, and time-ordered formula resolution.", + "filters": "WHERE vs HAVING routing, filters on measures/transforms, and {variable} placeholders.", + "joins": "Reaching joined data via dotted paths and how joins auto-resolve.", + "models": "What a model is: columns, measures, source modes, and model-level filters.", + "extending": "Ad hoc columns/measures/joins via ModelExtension and saving queries as models.", + "workflow": "Recommended tool-chaining order for an agent: inspect -> search -> inspect -> query.", +} + + +class HelpTopic(BaseModel): + """One seeded help memory: a fixed id, the migrated topic body, and an + authored one-line preview.""" + + id: str + learning: str + description: str + + +def _strip_numeric_prefix(stem: str) -> str: + """``"01_queries"`` -> ``"queries"``; leave other stems unchanged.""" + if len(stem) >= 3 and stem[0].isdigit() and stem[1].isdigit() and stem[2] == "_": + return stem[3:] + return stem + + +#: Host-substitutable tokens in the shipped content. An embedding host (e.g. a +#: hosted SLayer that renames the query tool) overrides these instead of forking +#: the markdown. Written ``{{name}}`` — deliberately NOT ``str.format`` / +#: ``string.Template`` syntax, because the content is full of single-brace JSON +#: examples (``{"source_model": "orders"}``) and ``'$'`` currency symbols. +DEFAULT_HELP_CONTEXT: dict[str, str] = { + "product": "SLayer", +} + +_PLACEHOLDER_RE = re.compile(r"\{\{(\w+)\}\}") + + +def _render(text: str, context: Mapping[str, str]) -> str: + """Substitute ``{{name}}`` tokens from ``context``. + + An unknown token raises rather than rendering literally — a typo in the + shipped content should fail loudly at load, not ship ``{{prodcut}}`` to an + agent. + """ + def _sub(match: re.Match[str]) -> str: + key = match.group(1) + if key not in context: + raise KeyError( + f"help content references unknown placeholder '{{{{{key}}}}}'; " + f"known tokens: {', '.join(sorted(context))}" + ) + return context[key] + + return _PLACEHOLDER_RE.sub(_sub, text) + + +def load_help_topics( + *, context: Mapping[str, str] | None = None, +) -> tuple[HelpTopic, ...]: + """SLayer's built-in help topics, in teaching (``NN_``) order. + + ``context`` overrides :data:`DEFAULT_HELP_CONTEXT` so a host can rename the + product or the query tool without copying the markdown. Pair with + :func:`merge_help_topics` to replace or extend individual topics. + """ + ctx = {**DEFAULT_HELP_CONTEXT, **(context or {})} + content_dir = files(__package__) / _CONTENT_SUBDIR + topics: list[HelpTopic] = [] + for entry in sorted(content_dir.iterdir(), key=lambda e: e.name): + if not entry.name.endswith(".md"): + continue + key = _strip_numeric_prefix(entry.name[: -len(".md")]) + description = _DESCRIPTIONS.get(key) + if description is None: + raise ValueError( + f"help topic {key!r} has no authored description in " + f"_DESCRIPTIONS; add one." + ) + topics.append(HelpTopic( + id=f"{_ID_PREFIX}{key}", + learning=_render(entry.read_text(encoding="utf-8"), ctx), + description=_render(description, ctx), + )) + return tuple(topics) + + +def merge_help_topics( + base: Sequence[HelpTopic], + *, + override: Mapping[str, HelpTopic] | None = None, + extra: Sequence[HelpTopic] = (), +) -> tuple[HelpTopic, ...]: + """Compose a host's topic set from SLayer's. + + ``override`` replaces topics by id, keeping ``base``'s teaching order, so a + host only ships the bodies that genuinely differ. ``extra`` appends + host-specific topics — give those a namespaced id (e.g. ``help.motley.x``) + so they can't collide with a future built-in. + + Raises when an ``override`` id isn't in ``base``: that means the built-in was + renamed or removed upstream and the host's copy is silently dead. + """ + override = dict(override or {}) + unknown = sorted(set(override) - {topic.id for topic in base}) + if unknown: + raise ValueError( + f"override targets no built-in help topic: {', '.join(unknown)}. " + f"Known ids: {', '.join(topic.id for topic in base)}." + ) + # A value whose own id differs from its key replaces the built-in with a + # topic seeded under that other id — so the topic it was meant to replace + # silently stops being served (e.g. keyed help.workflow, id help.workflows + # removes help.workflow from the set entirely). + mismatched = sorted( + f"{key} -> {topic.id}" for key, topic in override.items() if topic.id != key + ) + if mismatched: + raise ValueError( + f"override topic id must equal its key: {', '.join(mismatched)}." + ) + merged = [override.get(topic.id, topic) for topic in base] + merged.extend(extra) + # Two topics sharing an id would seed last-write-wins, so one body is lost + # with no error. Usually an ``extra`` that collides with a built-in. + duplicates = sorted( + topic_id + for topic_id, count in Counter(topic.id for topic in merged).items() + if count > 1 + ) + if duplicates: + raise ValueError( + f"duplicate help topic ids after merge: {', '.join(duplicates)}. " + f"Namespace host-specific topics (e.g. 'help.motley.x')." + ) + return tuple(merged) + + +def _load_topics() -> tuple[HelpTopic, ...]: + """Back-compat alias for :func:`load_help_topics` with default context.""" + return load_help_topics() + + +HELP_TOPICS: tuple[HelpTopic, ...] = load_help_topics() + + +async def seed_help_memories( + storage: StorageBackend, *, topics: Sequence[HelpTopic] | None = None, +) -> int: + """Idempotently seed the help topics as memories. Returns the number of + rows actually written (0 on a warm, unchanged store). + + Upsert-always with skip-if-unchanged: an existing ``help.*`` row whose + ``learning`` + ``description`` already match the shipped content is left + untouched (no write, no embedding refresh). Changed/absent rows are saved + with empty ``entities`` (so they never pollute Learnings sections), and the + embedding channel is refreshed via ``SearchService.upsert_memory`` — the + storage layer does not embed on its own. + """ + written = 0 + for topic in (HELP_TOPICS if topics is None else topics): + existing = await storage.get_memory_row(topic.id) + if ( + existing is not None + and existing.learning == topic.learning + and existing.description == topic.description + # Also require the invariant metadata to already hold — otherwise a + # help.* id someone tagged with entities / a query (but with matching + # text) would skip the rewrite and keep polluting Learnings / recall. + and existing.entities == [] + and existing.query is None + ): + continue + memory = await storage.save_memory( + id=topic.id, + learning=topic.learning, + description=topic.description, + entities=[], + ) + # Embedding/retriever fan-out (DEV-1658 / Codex): storage.save_memory + # only persists the row. Local import mirrors MemoryService.save_memory + # — keeps the search module off the critical-path import graph. + from slayer.search.service import SearchService + + await SearchService(storage=storage).upsert_memory(memory) + written += 1 + return written diff --git a/slayer/memories/models.py b/slayer/memories/models.py index 1b3718b8..718bf7aa 100644 --- a/slayer/memories/models.py +++ b/slayer/memories/models.py @@ -19,7 +19,7 @@ """ from datetime import datetime, timezone -from typing import Any, List, Optional +from typing import Any from pydantic import BaseModel, Field, field_validator, model_validator @@ -31,7 +31,9 @@ def _utcnow() -> datetime: return datetime.now(timezone.utc) -_FORBIDDEN_ID_CHARS = (":", "/", "?", "#") +# DEV-1658: ``\`` is forbidden too so a memory id is a safe single path +# segment on every platform (YAMLStorage writes ``memories/.md``). +_FORBIDDEN_ID_CHARS = (":", "/", "?", "#", "\\") #: Canonical-id prefix for cross-memory references (`memory:`). #: Re-exported from this module so the resolver, search service, and @@ -74,6 +76,12 @@ def is_valid_memory_id(value: str) -> bool: return True +#: DEV-1549: hard cap on Memory.description length. The compact-mode +#: first-paragraph fallback of learning shares the same cap so the two +#: code paths never disagree on payload size. +MEMORY_DESCRIPTION_MAX_CHARS = 500 + + class Memory(BaseModel): """A single agent memory: a note plus its canonical entity tags, optionally bundled with a ``SlayerQuery`` example.""" @@ -81,8 +89,9 @@ class Memory(BaseModel): version: int = 2 id: str = "" learning: str - entities: List[str] = Field(default_factory=list) - query: Optional[SlayerQuery] = None + description: str | None = None + entities: list[str] = Field(default_factory=list) + query: SlayerQuery | None = None created_at: datetime = Field(default_factory=_utcnow) @model_validator(mode="before") @@ -111,6 +120,41 @@ def _check_id_charset(cls, value: str) -> str: _validate_memory_id_charset(value) return value + @field_validator("learning") + @classmethod + def _check_learning_non_whitespace(cls, value: str) -> str: + """DEV-1549 Codex#4: reject whitespace-only learning at the model + layer so direct construction can never persist an unusable + memory.""" + if not value.strip(): + raise ValueError("learning must be a non-empty string.") + return value + + @field_validator("description", mode="before") + @classmethod + def _normalise_description(cls, value: Any) -> Any: + """DEV-1549 Codex#1: empty / whitespace-only ``description`` is + not a deliberate empty preview — coerce to ``None`` so the + downstream compact-mode renderer falls back to the first + paragraph of ``learning``.""" + if value is None: + return None + if isinstance(value, str) and not value.strip(): + return None + return value + + @field_validator("description") + @classmethod + def _check_description_length(cls, value: str | None) -> str | None: + """DEV-1549: hard cap so a single memory hit can never balloon + the search payload.""" + if value is not None and len(value) > MEMORY_DESCRIPTION_MAX_CHARS: + raise ValueError( + f"description must be <= {MEMORY_DESCRIPTION_MAX_CHARS} " + f"chars; got {len(value)}." + ) + return value + # --------------------------------------------------------------------------- # Tool / endpoint response models @@ -119,8 +163,8 @@ def _check_id_charset(cls, value: str) -> str: class SaveMemoryResponse(BaseModel): memory_id: str - resolved_entities: List[str] - warnings: List[str] = Field(default_factory=list) + resolved_entities: list[str] + warnings: list[str] = Field(default_factory=list) class ForgetMemoryResponse(BaseModel): diff --git a/slayer/memories/ranker.py b/slayer/memories/ranker.py index 5702abfd..f7d46f8a 100644 --- a/slayer/memories/ranker.py +++ b/slayer/memories/ranker.py @@ -29,7 +29,6 @@ from __future__ import annotations -from typing import List, Tuple from rank_bm25 import BM25Plus @@ -37,9 +36,9 @@ def bm25_rank( - memories: List[Memory], - query_entities: List[str], -) -> List[Tuple[Memory, float]]: + memories: list[Memory], + query_entities: list[str], +) -> list[tuple[Memory, float]]: """Rank ``memories`` against ``query_entities`` using BM25Plus. Returns ``(memory, score)`` pairs sorted by score descending. @@ -68,7 +67,7 @@ def bm25_rank( bm25 = BM25Plus(tokenised) scores = bm25.get_scores(list(query_set)) - paired: List[Tuple[Memory, float]] = [ + paired: list[tuple[Memory, float]] = [ (memories[i], float(scores[i])) for i in eligible ] paired.sort(key=lambda pair: pair[1], reverse=True) diff --git a/slayer/memories/resolver.py b/slayer/memories/resolver.py index 832d0183..bf3fe29b 100644 --- a/slayer/memories/resolver.py +++ b/slayer/memories/resolver.py @@ -29,11 +29,12 @@ from __future__ import annotations import re -from typing import Iterable, List, Optional, Set, Tuple +from collections.abc import Iterable from pydantic import BaseModel from slayer.core.errors import ( + AmbiguousModelError, EntityResolutionError, UnknownFunctionError, ) @@ -108,8 +109,8 @@ class EntityResolution(BaseModel): datasource-vs-model collision; fatal failures raise instead. """ - canonical_forms: List[str] - warnings: List[str] = [] + canonical_forms: list[str] + warnings: list[str] = [] def _model_has_leaf(model: SlayerModel, leaf: str) -> bool: @@ -126,9 +127,9 @@ def _model_has_leaf(model: SlayerModel, leaf: str) -> bool: async def _all_models_in_datasource( storage: StorageBackend, data_source: str -) -> List[SlayerModel]: +) -> list[SlayerModel]: identities = await storage._list_all_model_identities() - out: List[SlayerModel] = [] + out: list[SlayerModel] = [] for ds, name in identities: if ds != data_source: continue @@ -140,7 +141,7 @@ async def _all_models_in_datasource( async def _find_leaf_in_priority_winner( storage: StorageBackend, leaf: str -) -> Tuple[Optional[str], List[SlayerModel]]: +) -> tuple[str | None, list[SlayerModel]]: """Walk the priority list; return ``(data_source, matches)`` for the first datasource that has ≥1 model carrying ``leaf`` as a column / measure / custom aggregation. ``matches`` may have multiple models @@ -160,7 +161,7 @@ async def _find_leaf_in_priority_winner( async def _resolve_join_path( storage: StorageBackend, starting_model: SlayerModel, - path: List[str], + path: list[str], ) -> SlayerModel: """Walk a chain of join targets, returning the leaf model. @@ -192,7 +193,7 @@ async def _resolve_join_path( async def _resolve_dotted_against_model( storage: StorageBackend, starting_model: SlayerModel, - rest: List[str], + rest: list[str], ) -> str: """Apply the leaf rule to a path ``[hop, hop, ..., leaf?]`` rooted at ``starting_model``. Returns the canonical form.""" @@ -227,7 +228,7 @@ async def resolve_entity( # NOSONAR(S3776) — single linear dispatch matching raw: str, *, storage: StorageBackend, - source_model: Optional[SlayerModel] = None, + source_model: SlayerModel | None = None, ) -> EntityResolution: """Resolve a single entity reference. @@ -299,19 +300,30 @@ async def resolve_entity( # NOSONAR(S3776) — single linear dispatch matching ) segments = prefix.split(".") - if not all(re.match(r"^[a-zA-Z_]\w*$", s) for s in segments): + # Hyphens are permitted in a segment: a datasource name may embed a UUID + # (e.g. ``source_019f4323-cb59-77c3-...``), so the datasource segment can + # contain hyphens. Model/leaf segments that don't match a stored entity + # simply resolve to not-found below, so allowing the character is safe. + if not all(re.match(r"^[a-zA-Z_][\w-]*$", s) for s in segments): raise EntityResolutionError( f"'{raw}' contains an invalid identifier segment." ) - warnings: List[str] = [] + warnings: list[str] = [] known_dses = set(await storage.list_datasources()) # ----- step 3: datasource-prefix detection --------------------------- if segments[0] in known_dses: ds = segments[0] - # Case D: same name is also a model in some other datasource. - ds_as_model = await storage.resolve_model_identity(ds) + # Case D: same name is also a model in some other datasource. This + # probe is best-effort (it only drives a warning) — an *ambiguous* + # model leg (same name in ≥2 datasources, no priority winner) must + # NOT abort a perfectly valid datasource-prefixed reference, so we + # treat the ambiguity as "yes, also a model" and keep going. + try: + ds_as_model = await storage.resolve_model_identity(ds) + except AmbiguousModelError: + ds_as_model = (ds, ds) if ds_as_model is not None: warnings.append( f"'{ds}' is both a datasource and a model; interpreted " @@ -452,7 +464,7 @@ def _formula_entity_tokens(parsed: ParsedExpr) -> Iterable[str]: _FILTER_TOKEN_RE = re.compile(r"[a-zA-Z_]\w*(?:\.[a-zA-Z_]\w*)*") -def _extract_filter_tokens(filter_text: str) -> List[str]: +def _extract_filter_tokens(filter_text: str) -> list[str]: """Extract identifier-shaped tokens from a filter expression that might be entity references. @@ -463,7 +475,7 @@ def _extract_filter_tokens(filter_text: str) -> List[str]: cleaned = _FILTER_AGG_SUFFIX_RE.sub("", filter_text) cleaned = _FILTER_LITERAL_RE.sub("", cleaned) cleaned = _FILTER_VAR_RE.sub("", cleaned) - out: List[str] = [] + out: list[str] = [] for m in _FILTER_TOKEN_RE.finditer(cleaned): token = m.group(0) # Skip identifiers immediately followed by '(' — they're SQL @@ -495,9 +507,9 @@ async def extract_entities_from_query( # NOSONAR(S3776) — straight-line walk always tagged, even if no field references it explicitly. Resolution failures bubble up unchanged. """ - canonical: List[str] = [] - warnings: List[str] = [] - seen: Set[str] = set() + canonical: list[str] = [] + warnings: list[str] = [] + seen: set[str] = set() def _add(forms: Iterable[str]) -> None: for f in forms: @@ -621,7 +633,7 @@ async def _resolve_join_target_for_resolver( warnings.extend(result.warnings) # Deduplicate warnings while preserving order. - seen_warn: Set[str] = set() + seen_warn: set[str] = set() deduped_warnings = [] for w in warnings: if w not in seen_warn: diff --git a/slayer/memories/service.py b/slayer/memories/service.py index e2e729c2..f856ad09 100644 --- a/slayer/memories/service.py +++ b/slayer/memories/service.py @@ -20,7 +20,6 @@ from __future__ import annotations -from typing import List, Optional, Union from slayer.core.query import SlayerQuery from slayer.memories.models import ( @@ -35,8 +34,8 @@ from slayer.storage.base import StorageBackend -QueryInput = Union[SlayerQuery, dict] -LinkedEntities = Union[List[str], SlayerQuery, dict] +QueryInput = SlayerQuery | dict +LinkedEntities = list[str] | SlayerQuery | dict def _coerce_query(query: QueryInput) -> SlayerQuery: @@ -57,7 +56,7 @@ def _coerce_query(query: QueryInput) -> SlayerQuery: ) -def _coerce_memory_id(identifier: Union[int, str]) -> str: +def _coerce_memory_id(identifier: int | str) -> str: """DEV-1428: accept native ``str`` (canonical form) or legacy ``int`` (back-compat: stringify decimally). Validates the result through :func:`_validate_memory_id_charset` so the surface layer @@ -79,9 +78,9 @@ def _coerce_memory_id(identifier: Union[int, str]) -> str: return value -def _dedup(items: List[str]) -> List[str]: +def _dedup(items: list[str]) -> list[str]: seen: set[str] = set() - out: List[str] = [] + out: list[str] = [] for x in items: if x not in seen: seen.add(x) @@ -104,16 +103,17 @@ async def save_memory( *, learning: str, linked_entities: LinkedEntities, - id: Optional[str] = None, # noqa: A002 — public kwarg + id: str | None = None, # noqa: A002 — public kwarg + description: str | None = None, ) -> SaveMemoryResponse: if not learning or not learning.strip(): raise ValueError("learning text must be a non-empty string.") if id is not None: _validate_memory_id_charset(id) - canonical: List[str] = [] - warnings: List[str] = [] - attached_query: Optional[SlayerQuery] = None + canonical: list[str] = [] + warnings: list[str] = [] + attached_query: SlayerQuery | None = None if isinstance(linked_entities, list): if not linked_entities: @@ -145,19 +145,19 @@ async def save_memory( entities=canonical, query=attached_query, id=id, + description=description, ) - # DEV-1386: best-effort embedding refresh for this single - # memory. Local import keeps the embeddings module off the - # critical-path import graph; failures are surfaced as warnings, - # never aborting the save. - from slayer.embeddings.service import EmbeddingService - - try: - embed_warnings = await EmbeddingService( - storage=self._storage, - ).refresh_memory(memory) - except Exception as exc: # noqa: BLE001 — best-effort - embed_warnings = [f"embedding refresh failed: {exc}"] + # DEV-1514: fan out the upsert through SearchService so every + # registered retriever gets a chance to react. SearchService + # isolates per-retriever exceptions as prefixed warnings, so + # this site no longer needs its own try/except. + # Local import keeps the search module off the critical-path + # import graph. + from slayer.search.service import SearchService + + embed_warnings = await SearchService( + storage=self._storage, + ).upsert_memory(memory) warnings = _dedup(warnings + embed_warnings) return SaveMemoryResponse( memory_id=memory.id, @@ -168,7 +168,7 @@ async def save_memory( # ---- forget_memory ------------------------------------------------- async def forget_memory( - self, *, identifier: Union[int, str] + self, *, identifier: int | str ) -> ForgetMemoryResponse: memory_id = _coerce_memory_id(identifier) await self._storage.delete_memory(memory_id) diff --git a/slayer/osi/__init__.py b/slayer/osi/__init__.py new file mode 100644 index 00000000..2746bbe5 --- /dev/null +++ b/slayer/osi/__init__.py @@ -0,0 +1 @@ +"""OSI (Open Semantic Interchange) importer for SLayer (DEV-1643).""" diff --git a/slayer/osi/converter.py b/slayer/osi/converter.py new file mode 100644 index 00000000..a22f5940 --- /dev/null +++ b/slayer/osi/converter.py @@ -0,0 +1,929 @@ +"""Convert parsed OSI documents into SLayer models (DEV-1643). + +Each OSI dataset becomes one ``SlayerModel``: its physical table is introspected +live (real column types + PK) and OSI semantic metadata (labels, descriptions, +is_time, ai_context, primary keys) is overlaid on top. OSI relationships become +``ModelJoin`` entries; OSI metrics become ``ModelMeasure`` formulas, anchored on +the model that reaches every dataset the metric references (via the shared +``recommend_root_model`` selection core). Constructs that cannot be expressed +exactly are clean-failed to a ``ConversionResult`` report, never silently lost. +""" + +from __future__ import annotations + +import logging +from typing import Any, Optional + +import sqlalchemy as sa +import sqlglot +import sqlglot.expressions as exp + +from slayer.core.enums import DataType +from slayer.core.formula import parse_formula +from slayer.core.models import Column, ModelJoin, ModelMeasure, SlayerModel +from slayer.core.refs import IDENTIFIER_RE as _IDENTIFIER_RE +from slayer.engine.column_expansion import _root_scope_column_ids +from slayer.engine.ingestion import introspect_table_to_model +from slayer.engine.join_graph import JoinGraph, min_hops_root +from slayer.ingest_report import ConversionResult, ConversionWarning +from slayer.sql.client import get_column_types_sync +from slayer.osi.expression import SQL_DIALECTS, convert_expression +from slayer.osi.models import ( + OSIAIContext, + OSICustomExtension, + OSIDataset, + OSIDocument, + OSIExpression, + OSIField, + OSIMetric, + OSIRelationship, + OSISemanticModel, + ai_context_to_dict, +) +from slayer.osi.source import parse_source, resolve_datasource + +logger = logging.getLogger(__name__) + +# Dialects lacking a GROUP BY percentile/median aggregate (mirrors the dbt +# converter's caveat set). +_NO_PERCENTILE_DIALECTS = frozenset({"mysql", "tsql", "mssql", "sqlserver"}) + +# OSI SQL dialect -> sqlglot read dialect for normalizing an expression to +# default SQL. ANSI_SQL maps to None (already default-compatible). +_SQLGLOT_DIALECT = {"SNOWFLAKE": "snowflake", "DATABRICKS": "databricks"} + + +class OsiConversionError(Exception): + """Raised when an OSI import set cannot be converted (e.g. duplicate names).""" + + +# Characters/shapes that are unsafe in a SLayer model name — notably path +# separators and NUL, since a model name becomes a filename in YAML storage +# (``/.yaml``); an absolute/traversal name would escape the tree. +_UNSAFE_MODEL_NAME_CHARS = ("__", ".", ":", "/", "\\", "\x00") + + +def _legal_model_name(name: str) -> bool: + if not name or name.strip() != name or not name.strip(): + return False + return not any(ch in name for ch in _UNSAFE_MODEL_NAME_CHARS) + + +def _legal_column_name(name: str) -> bool: + return "." not in name and ":" not in name + + +def _legal_measure_name(name: str) -> bool: + return bool(_IDENTIFIER_RE.match(name)) + + +def _as_bare_column(sql: str) -> str | None: + """The unquoted column name if ``sql`` is a single unqualified column + reference (bare or double-quoted), else ``None``. + + Lets a case-sensitive quoted identifier (e.g. ``"legalEntityType"``) be + treated as a base-column reference rather than a derived expression. + """ + stripped = sql.strip() + if _IDENTIFIER_RE.match(stripped): + return stripped + try: + tree = sqlglot.parse_one(stripped) + except sqlglot.errors.ParseError: + return None + if (isinstance(tree, exp.Column) and not tree.table + and not tree.args.get("db") and not tree.args.get("catalog")): + return tree.name + return None + + +def _missing_expr_columns( + sql: str, available: set[str], self_name: str +) -> list[str] | None: + """Unqualified and self-qualified column names in ``sql`` absent from + ``available``. Returns ``None`` when ``sql`` cannot be parsed. + + Self-qualified references (``.col``) are validated too; genuinely + cross-model references (a different qualifier) are left to query-time join + resolution, matching how ``Column.sql`` expansion treats join aliases. + """ + try: + tree = sqlglot.parse_one(sql) + except sqlglot.errors.ParseError: + return None + missing = [] + for col in tree.find_all(exp.Column): + if not col.table: + if col.name not in available: + missing.append(col.name) + elif col.table == self_name and col.name not in available: + missing.append(f"{col.table}.{col.name}") + return missing + + +def _render_description(explicit: Optional[str], ctx: Optional[OSIAIContext]) -> Optional[str]: + """Description = explicit OSI description (lead) + ai_context instructions + + synonyms.""" + parts: list[str] = [] + if explicit: + parts.append(explicit) + ctx_dict = ai_context_to_dict(ctx) + if ctx_dict: + instructions = ctx_dict.get("instructions") + if instructions and instructions != explicit: + parts.append(instructions) + synonyms = ctx_dict.get("synonyms") + if synonyms: + parts.append("Synonyms: " + ", ".join(synonyms)) + return "\n".join(parts) or None + + +def _build_meta( + ctx: Optional[OSIAIContext], + custom_extensions: Optional[list[OSICustomExtension]], + extra: Optional[dict[str, Any]] = None, +) -> Optional[dict[str, Any]]: + meta: dict[str, Any] = dict(extra or {}) + ctx_dict = ai_context_to_dict(ctx) + if ctx_dict: + meta["osi_ai_context"] = ctx_dict + if custom_extensions: + meta["osi_custom_extensions"] = [e.model_dump() for e in custom_extensions] + return meta or None + + +class OsiToSlayerConverter: + """Convert OSI documents into SLayer models.""" + + def __init__( + self, + documents: list[OSIDocument], + data_source: str, + sa_engine: sa.Engine, + *, + dialect: str = "ANSI_SQL", + target_dialect: str | None = None, + ) -> None: + self.documents = documents + self.data_source = data_source + self.sa_engine = sa_engine + self.dialect = dialect + self.target_dialect = target_dialect + self._models: dict[str, SlayerModel] = {} + self._warnings: list[ConversionWarning] = [] + self._unconverted: list[ConversionWarning] = [] + # (model, column) -> the introspected Column a derived overlay replaced, + # so an invalidated overlay can be reverted to the physical column + # instead of deleting a real column. + self._shadowed: dict[tuple[str, str], Column] = {} + + # ---- report helpers ---- + + def _warn(self, message: str, *, model_name: str | None = None, + metric_name: str | None = None, category: str = "general", + severity: str = "dropped", suggestion: str | None = None) -> None: + self._warnings.append(ConversionWarning( + model_name=model_name, metric_name=metric_name, message=message, + category=category, severity=severity, suggestion=suggestion, + )) + + def _unconv(self, message: str, *, metric_name: str, category: str = "metric", + suggestion: str | None = None) -> None: + self._unconverted.append(ConversionWarning( + metric_name=metric_name, message=message, category=category, + severity="unconverted", suggestion=suggestion, + )) + + # ---- top-level ---- + + def convert(self) -> ConversionResult: + inspector = sa.inspect(self.sa_engine) + semantic_models = [sm for doc in self.documents for sm in doc.semantic_model] + self._check_duplicate_dataset_names(semantic_models) + + for sm in semantic_models: + for ds in sm.datasets: + self._build_model(ds=ds, sm=sm, inspector=inspector) + for sm in semantic_models: + for rel in sm.relationships or []: + self._build_join(rel) + + self._validate_cross_model_field_refs() + self._type_cross_model_columns() + + graph = JoinGraph.build_from_models(list(self._models.values())) + for sm in semantic_models: + self._build_measures_for(sm, graph) + + return ConversionResult( + models=list(self._models.values()), + unconverted_metrics=self._unconverted, + warnings=self._warnings, + ) + + def _build_measures_for(self, sm: OSISemanticModel, graph: JoinGraph) -> None: + sm_model_names = [d.name for d in sm.datasets if d.name in self._models] + for metric in sm.metrics or []: + self._build_measure(metric=metric, sm_model_names=sm_model_names, graph=graph) + + def _check_duplicate_dataset_names(self, sms: list[OSISemanticModel]) -> None: + seen: set[str] = set() + dupes: set[str] = set() + for sm in sms: + for ds in sm.datasets: + if ds.name in seen: + dupes.add(ds.name) + seen.add(ds.name) + if dupes: + raise OsiConversionError( + f"Duplicate dataset names across the OSI import set: " + f"{sorted(dupes)}. Dataset names map to SLayer model names and " + f"must be unique within a datasource." + ) + + # ---- datasets -> models ---- + + def _build_model(self, ds: OSIDataset, sm: OSISemanticModel, + inspector: sa.engine.Inspector) -> None: + if not _legal_model_name(ds.name): + self._warn( + f"Dataset name {ds.name!r} is not a safe SLayer model name " + f"(empty, surrounding whitespace, or a forbidden character such " + f"as '__', '.', ':', '/', '\\\\', NUL); skipping.", + model_name=ds.name, category="illegal_name", + ) + return + + parsed = parse_source(ds.source) + resolve_datasource(parsed.database, self.data_source) # stubbed routing + + try: + if parsed.is_query: + base = self._build_sql_mode_model(ds, parsed.query) + else: + base = introspect_table_to_model( + sa_engine=self.sa_engine, inspector=inspector, + table_name=parsed.table, schema=parsed.schema_name, + data_source=self.data_source, model_name=ds.name, + ) + except Exception as exc: # noqa: BLE001 — per-dataset isolation + self._warn( + f"Failed to introspect dataset {ds.name!r} (source {ds.source!r}): " + f"{exc}; skipping.", + model_name=ds.name, category="introspection", + ) + return + + self._overlay_fields(base, ds) + self._apply_dataset_metadata(base, ds, sm) + self._models[ds.name] = base + + def _build_sql_mode_model(self, ds: OSIDataset, query: str) -> SlayerModel: + # Query source: introspect the query's output columns live (LIMIT-0 / + # cursor-metadata probe), exactly as table sources are introspected. + # ``target_dialect`` is the datasource type, used for the dialect-correct + # probe (LIMIT 0 vs SELECT TOP vs SQLite's LIMIT-1 fallback). + types = get_column_types_sync( + sql=query, engine=self.sa_engine, db_type=self.target_dialect, + ) + columns = [Column(name=name, type=category) for name, category in types.items()] + return SlayerModel(name=ds.name, sql=query, data_source=self.data_source, + columns=columns) + + def _overlay_fields(self, model: SlayerModel, ds: OSIDataset) -> None: + by_name = {c.name: c for c in model.columns} + introspected = set(by_name) + first_time_dim: str | None = None + + for field in ds.fields or []: + time_col = self._overlay_one_field(field=field, model=model, by_name=by_name, introspected=introspected, ds=ds) + if time_col and first_time_dim is None: + first_time_dim = time_col + + # OSI primary_key is authoritative: when set, it fully REPLACES the + # introspected primary key (clearing physical PK flags that OSI omits); + # when unset, the introspected PK is kept. If any listed key column is + # missing (typo), the override is unsafe (it would leave the model with + # no PK) — report and keep the introspected PK instead. + if ds.primary_key: + present = {c.name for c in model.columns} + missing_pk = [k for k in ds.primary_key if k not in present] + if missing_pk: + self._warn( + f"Dataset {ds.name!r} primary_key references unknown " + f"column(s) {missing_pk}; keeping the introspected primary key.", + model_name=ds.name, category="primary_key", + ) + else: + osi_pk = set(ds.primary_key) + for col in model.columns: + col.primary_key = col.name in osi_pk + + if first_time_dim and not model.default_time_dimension: + model.default_time_dimension = first_time_dim + + def _overlay_one_field(self, field: OSIField, model: SlayerModel, + by_name: dict[str, Column], introspected: set[str], + ds: OSIDataset) -> str | None: + """Overlay one OSI field onto the model. Returns the column name if the + field is a time dimension, else None (clean-fails are reported).""" + if not _legal_column_name(field.name): + self._warn( + f"Field name {field.name!r} on dataset {ds.name!r} contains " + f"'.'/':'; skipping the field.", + model_name=ds.name, category="illegal_name", + ) + return None + + sql = self._resolve_expression(field.expression) + if sql is None: + self._warn( + f"Field {field.name!r} on {ds.name!r} has no SQL-dialect " + f"expression; skipping.", + model_name=ds.name, category="dialect", + ) + return None + + col = self._resolve_field_column(field=field, sql=sql, by_name=by_name, introspected=introspected, ds=ds, model=model) + if col is None: + return None + + col.label = field.label or col.label + col.description = _render_description(explicit=field.description, ctx=field.ai_context) \ + or col.description + meta = _build_meta(ctx=field.ai_context, custom_extensions=field.custom_extensions) + if meta: + col.meta = {**(col.meta or {}), **meta} + + is_time = bool(field.dimension and field.dimension.is_time) + if is_time and col.type not in (DataType.DATE, DataType.TIMESTAMP): + col.type = DataType.TIMESTAMP + + # Bare overlays return the existing column object (no-op here); an + # aliased/derived field that shadows an existing column REPLACES it so + # its expression isn't silently dropped by an append-only path. + existing = by_name.get(col.name) + if existing is None: + model.columns.append(col) + by_name[col.name] = col + elif existing is not col: + # Preserve the shadowed column's primary-key flag across the + # redefinition — an introspected PK must survive a derived overlay + # unless OSI's primary_key explicitly overrides it later. Remember + # the original so an overlay later invalidated by cross-model + # validation can revert to the physical column (not delete it). + col.primary_key = existing.primary_key + self._shadowed.setdefault((model.name, col.name), existing) + model.columns[model.columns.index(existing)] = col + by_name[col.name] = col + return col.name if is_time else None + + def _resolve_field_column(self, field: OSIField, sql: str, + by_name: dict[str, Column], introspected: set[str], + ds: OSIDataset, model: SlayerModel) -> Column | None: + """Return the Column (existing or new) this field maps to, or None on a + clean-fail (already reported).""" + bare = _as_bare_column(sql) + if bare is not None: + if bare == field.name and field.name in by_name: + return by_name[field.name] # overlay existing column + if bare in introspected: + # aliased/renamed reference to a real column. Keep the ORIGINAL + # expression (preserving any identifier quoting, which matters on + # case-folding dialects); use the unquoted name only for lookup. + return Column(name=field.name, sql=sql, type=by_name[bare].type) + self._warn( + f"Field {field.name!r} on {ds.name!r} references column {bare!r} " + f"which is not present in the table; skipping.", + model_name=ds.name, category="missing_column", + ) + return None + # derived expression. Validate its column references exist on the table + # (consistent with the bare-field / metric / relationship checks); a + # collision with an existing column is handled by the replace-or-append + # logic in _overlay_one_field. + missing = _missing_expr_columns(sql=sql, available=introspected, self_name=ds.name) + if missing is None: + self._warn( + f"Field {field.name!r} on {ds.name!r} has an unparseable " + f"expression {sql!r}; skipping.", + model_name=ds.name, category="expression", + ) + return None + if missing: + self._warn( + f"Field {field.name!r} on {ds.name!r} references unknown " + f"column(s) {missing}; skipping.", + model_name=ds.name, category="missing_column", + ) + return None + is_time = bool(field.dimension and field.dimension.is_time) + if is_time: + dtype = DataType.TIMESTAMP + elif field.name in by_name: + # A derived field that redefines an existing column inherits that + # column's known type (e.g. LOWER(status) stays TEXT). + dtype = by_name[field.name].type + else: + # A genuinely new derived column: live-probe the expression's real + # type so a non-numeric expression (UPPER(x), concat, ...) isn't + # mis-cast as DOUBLE by the SQL generator. Fall back to DOUBLE if the + # probe is unavailable. + dtype = self._probe_expression_type(model=model, expr=sql) or DataType.DOUBLE + return Column(name=field.name, sql=sql, type=dtype) + + def _probe_expression_type(self, model: SlayerModel, expr: str) -> DataType | None: + """Infer a derived expression's SLayer type by probing it against the + model's physical source (LIMIT-0 cursor metadata). Returns None if the + model has no probeable source or the probe fails.""" + if model.sql_table: + from_clause = model.sql_table + elif model.sql: + from_clause = f"({model.sql}) AS _osi_sub" + else: + return None + probe = f"SELECT ({expr}) AS _osi_probe FROM {from_clause}" + try: + types = get_column_types_sync( + sql=probe, engine=self.sa_engine, db_type=self.target_dialect, + ) + except Exception: # noqa: BLE001 — probe is best-effort; fall back + return None + category = next(iter(types.values()), None) + if category is None: + return None + # Reuse Column's category->DataType coercion (as _build_sql_mode_model does). + return Column(name="_osi_probe", type=category).type + + def _apply_dataset_metadata(self, model: SlayerModel, ds: OSIDataset, + sm: OSISemanticModel) -> None: + model.description = _render_description(explicit=ds.description, ctx=ds.ai_context) \ + or model.description + extra: dict[str, Any] = {} + if ds.unique_keys: + extra["osi_unique_keys"] = ds.unique_keys + sm_ctx = ai_context_to_dict(sm.ai_context) + if sm_ctx or sm.description: + extra["osi_semantic_model"] = { + "name": sm.name, + "description": sm.description, + "ai_context": sm_ctx, + } + meta = _build_meta(ctx=ds.ai_context, custom_extensions=ds.custom_extensions, extra=extra) + if meta: + model.meta = {**(model.meta or {}), **meta} + + # ---- relationships -> joins ---- + + def _build_join(self, rel: OSIRelationship) -> None: + src = rel.from_dataset + if src not in self._models: + self._warn( + f"Relationship {rel.name!r} references unknown source dataset " + f"{src!r}; skipping.", + category="relationship", + ) + return + if rel.to not in self._models: + self._warn( + f"Relationship {rel.name!r} targets unknown dataset {rel.to!r}; " + f"skipping.", + model_name=src, category="relationship", + ) + return + if len(rel.from_columns) != len(rel.to_columns): + self._warn( + f"Relationship {rel.name!r} has mismatched key lengths " + f"({len(rel.from_columns)} vs {len(rel.to_columns)}); skipping.", + model_name=src, category="relationship", + ) + return + + missing = self._missing_join_columns(rel) + if missing: + self._warn( + f"Relationship {rel.name!r} references join columns not present " + f"on their models: {missing}; skipping.", + model_name=src, category="relationship", + ) + return + + # SLayer's ModelJoin keys only on target_model and runtime join-walking + # picks the first match, so a second relationship to the same target + # (e.g. a distinct role) would be unreachable and could bind refs to the + # wrong join. Keep the first; report and skip duplicates. + if any(j.target_model == rel.to for j in self._models[src].joins): + self._warn( + f"Relationship {rel.name!r} is a second join from {src!r} to " + f"{rel.to!r}; SLayer cannot disambiguate multiple joins to one " + f"model (no join aliases). Keeping the first; skipping this one.", + model_name=src, category="relationship", + ) + return + + pairs = [[f, t] for f, t in zip(rel.from_columns, rel.to_columns)] + self._models[src].joins.append(ModelJoin( + target_model=rel.to, + join_pairs=pairs, + description=_render_description(explicit=None, ctx=rel.ai_context), + meta=_build_meta(ctx=rel.ai_context, custom_extensions=rel.custom_extensions), + )) + + def _validate_cross_model_field_refs(self) -> None: + """Post-join pass: derived columns may reference joined models via + ``.
`` / ``__.``. Resolve each such ref through the + join graph and drop (with a report) any column whose cross-model ref + names a model with no join path or a nonexistent target column — so a + typo clean-fails at import instead of erroring at query time. + """ + # Fixed-point: dropping a column can invalidate another column that + # referenced it, and dropping a column used as a join key invalidates + # that join (which can in turn invalidate more columns). Re-run column + # and join pruning until a pass changes nothing. + while True: + if not (self._drop_invalid_cross_model_columns() + or self._drop_stale_joins()): + break + + def _drop_invalid_cross_model_columns(self) -> bool: + dropped = False + for model in self._models.values(): + for col, bad in self._invalid_cross_model_columns(model): + self._revert_or_drop_column(model=model, col=col, bad=bad) + dropped = True + return dropped + + def _invalid_cross_model_columns( + self, model: SlayerModel + ) -> list[tuple[Column, list[str]]]: + result = [] + for col in model.columns: + if col.sql: + bad = self._unresolvable_cross_model_refs(model=model, sql=col.sql) + if bad: + result.append((col, bad)) + return result + + def _revert_or_drop_column( + self, model: SlayerModel, col: Column, bad: list[str] + ) -> None: + original = self._shadowed.pop((model.name, col.name), None) + if original is not None: + # The invalid column was a derived overlay of a physical column — + # revert to the physical column, don't delete it. + model.columns[model.columns.index(col)] = original + self._warn( + f"Derived overlay for {col.name!r} on {model.name!r} references " + f"unresolvable cross-model column(s) {bad}; reverting to the " + f"physical column.", + model_name=model.name, category="missing_column", + ) + else: + model.columns.remove(col) + if model.default_time_dimension == col.name: + # Don't leave the default pointing at a dropped column. + model.default_time_dimension = next( + (c.name for c in model.columns + if c.type in (DataType.DATE, DataType.TIMESTAMP)), + None, + ) + self._warn( + f"Column {col.name!r} on {model.name!r} references unresolvable " + f"cross-model column(s) {bad}; dropping.", + model_name=model.name, category="missing_column", + ) + + def _drop_stale_joins(self) -> bool: + """Drop joins whose key columns were removed by column validation, so a + join never references a column that no longer exists.""" + dropped = False + for model in self._models.values(): + stale = [j for j in model.joins if self._join_columns_missing(model=model, join=j)] + for join in stale: + model.joins.remove(join) + dropped = True + self._warn( + f"Join from {model.name!r} to {join.target_model!r} " + f"references a column removed during validation; dropping.", + model_name=model.name, category="relationship", + ) + return dropped + + def _type_cross_model_columns(self) -> None: + """After joins exist, fix the type of a derived column that is exactly a + single cross-model column ref (e.g. ``customers.segment``) — the local + probe couldn't resolve it, so it was left as the DOUBLE fallback and + would mis-cast a joined text/temporal column at query time.""" + for model in self._models.values(): + for col in model.columns: + if col.sql: + target_type = self._single_cross_model_ref_type(model=model, sql=col.sql) + if target_type is not None: + col.type = target_type + + def _single_cross_model_ref_type( + self, model: SlayerModel, sql: str + ) -> DataType | None: + try: + tree = sqlglot.parse_one(sql) + except sqlglot.errors.ParseError: + return None + if not isinstance(tree, exp.Column) or not tree.table or tree.table == model.name: + return None + target = self._walk_join_alias(host=model, alias=tree.table) + if target is None: + return None + tcol = next((c for c in target.columns if c.name == tree.name), None) + return tcol.type if tcol is not None else None + + def _join_columns_missing(self, model: SlayerModel, join: ModelJoin) -> bool: + target = self._models.get(join.target_model) + for src_col, tgt_col in join.join_pairs: + if not self._model_has_column(model_name=model.name, column=src_col): + return True + if target is None or not any(c.name == tgt_col for c in target.columns): + return True + return False + + def _unresolvable_cross_model_refs(self, model: SlayerModel, sql: str) -> list[str]: + """Cross-model (non-self, qualified) column refs in ``sql`` that don't + resolve to a joined model + existing column. Mirrors runtime scope rules: + only root-scope refs are checked (nested subquery/CTE aliases are left + alone), and catalog/db-qualified physical refs are skipped. Self / + unqualified refs were validated at field-overlay time.""" + try: + tree = sqlglot.parse_one(sql) + except sqlglot.errors.ParseError: + return [] + root_ids = _root_scope_column_ids(parsed=tree) + bad = [] + for col in tree.find_all(exp.Column): + if id(col) not in root_ids: + continue # nested-scope alias (CTE / sub-query) — not a join ref + if col.args.get("db") or col.args.get("catalog"): + continue # catalog/db-qualified physical ref — outside SLayer's contract + if not col.table or col.table == model.name: + continue + target = self._walk_join_alias(host=model, alias=col.table) + if target is None or not any(c.name == col.name for c in target.columns): + bad.append(f"{col.table}.{col.name}") + return bad + + def _walk_join_alias(self, host: SlayerModel, alias: str) -> SlayerModel | None: + """Resolve a ``__``-delimited join alias (e.g. ``customers__regions``) + to the terminal joined model by walking ``host``'s join chain, or None + if any hop is not a declared join.""" + current = host + for hop in (alias.split("__") if "__" in alias else [alias]): + join = next((j for j in current.joins if j.target_model == hop), None) + if join is None: + return None + current = self._models.get(hop) + if current is None: + return None + return current + + def _model_has_column(self, model_name: str, column: str) -> bool: + model = self._models.get(model_name) + return model is not None and any(c.name == column for c in model.columns) + + def _model_has_name(self, model_name: str, name: str) -> bool: + """Whether ``name`` is taken by a column OR measure on the model — + SLayer requires the two to share one namespace, so a materialized + hidden-column name must avoid both.""" + model = self._models.get(model_name) + if model is None: + return False + return (any(c.name == name for c in model.columns) + or any(m.name == name for m in model.measures)) + + def _missing_join_columns(self, rel: OSIRelationship) -> list[str]: + """Qualified names of relationship join columns absent from their + model, so a typo clean-fails instead of emitting a broken join.""" + missing = [f"{rel.from_dataset}.{c}" for c in rel.from_columns + if not self._model_has_column(model_name=rel.from_dataset, column=c)] + missing += [f"{rel.to}.{c}" for c in rel.to_columns + if not self._model_has_column(model_name=rel.to, column=c)] + return missing + + # ---- metrics -> measures ---- + + def _build_measure(self, metric: OSIMetric, sm_model_names: list[str], + graph: JoinGraph) -> None: + if not _legal_measure_name(metric.name): + self._unconv( + f"Metric name {metric.name!r} is not a valid SLayer identifier.", + metric_name=metric.name, + ) + return + + expr = self._resolve_expression(metric.expression) + if expr is None: + self._unconv( + f"Metric {metric.name!r} has no SQL-dialect expression.", + metric_name=metric.name, category="dialect", + ) + return + + owner_of = self._make_owner_of(sm_model_names) + anchor = self._select_anchor(metric_name=metric.name, expr=expr, sm_model_names=sm_model_names, owner_of=owner_of, graph=graph) + if anchor is None: + return # already reported + + # Enforce SLayer's namespace invariants before the post-construction + # append (which bypasses SlayerModel's validators): a measure name must + # be unique and must not collide with a column on the same model. + anchor_model = self._models[anchor] + if any(m.name == metric.name for m in anchor_model.measures): + self._unconv( + f"Metric {metric.name!r} duplicates an existing measure on " + f"model {anchor!r}.", + metric_name=metric.name, category="duplicate_measure", + ) + return + if self._model_has_column(model_name=anchor, column=metric.name): + self._unconv( + f"Metric {metric.name!r} collides with a column name on model " + f"{anchor!r}.", + metric_name=metric.name, category="name_collision", + ) + return + + ref_of = self._make_ref_of(graph=graph, anchor=anchor) + percentile_unsupported = ( + self.target_dialect is not None + and self.target_dialect.lower() in _NO_PERCENTILE_DIALECTS + ) + result = convert_expression( + expr, entity_name=metric.name, owner_of=owner_of, ref_of=ref_of, + percentile_unsupported=percentile_unsupported, + name_taken=self._model_has_name, + ) + if not result.ok: + self._unconv( + f"Metric {metric.name!r}: {result.reason}", + metric_name=metric.name, + ) + return + + try: + parse_formula(result.formula) + except Exception as exc: # noqa: BLE001 — reject un-parseable emission + self._unconv( + f"Metric {metric.name!r}: emitted formula {result.formula!r} is " + f"not a valid SLayer formula ({exc}).", + metric_name=metric.name, + ) + return + + # Build the measure first so a construction failure (e.g. a metric named + # after a reserved transform like ``cumsum``) clean-fails instead of + # crashing the import — and before materializing columns, so a rejected + # metric leaves no orphan hidden columns. + try: + measure = ModelMeasure( + formula=result.formula, + name=metric.name, + description=_render_description(explicit=metric.description, ctx=metric.ai_context), + meta=_build_meta(ctx=metric.ai_context, custom_extensions=metric.custom_extensions), + ) + except Exception as exc: # noqa: BLE001 — any validation error -> report + self._unconv( + f"Metric {metric.name!r} cannot be expressed as a SLayer measure " + f"({exc}).", + metric_name=metric.name, + ) + return + + self._materialize_columns(result.materialized) + self._models[anchor].measures.append(measure) + for w in result.warnings: + self._warn(f"Metric {metric.name!r}: {w}", metric_name=metric.name, + category="dialect", severity="info") + + def _select_anchor(self, metric_name: str, expr: str, sm_model_names: list[str], + owner_of, graph: JoinGraph) -> str | None: + owners = self._referenced_owners(expr, owner_of) + if owners: + anchor = min_hops_root(graph=graph, candidates=sm_model_names, mentioned=owners) + if anchor is None: + self._unconv( + f"Metric {metric_name!r} references models {sorted(owners)} " + f"with no single model reaching all of them via joins.", + metric_name=metric_name, category="no_join_path", + suggestion="Add the required relationship(s), or split the " + "metric across a multi-stage query.", + ) + return anchor + # No column references (e.g. COUNT(*)). Attribute it to the semantic + # model's unique fact table (the one dataset that is never a join + # target). When there is no unique fact table, the metric is an orphan + # with no determinable grain -> error rather than guess. + anchor = self._fact_root(sm_model_names) + if anchor is None: + self._unconv( + f"Metric {metric_name!r} has no column references and the " + f"semantic model has no unique fact table to attribute it to; " + f"its grain is ambiguous.", + metric_name=metric_name, category="orphan_metric", + suggestion="Aggregate over an explicit column, or ensure the " + "semantic model has a single fact dataset.", + ) + return anchor + + def _referenced_owners(self, expr: str, owner_of) -> set[str]: + try: + tree = sqlglot.parse_one(expr) + except sqlglot.errors.ParseError: + return set() + owners: set[str] = set() + for col in tree.find_all(exp.Column): + owner = owner_of(col.table or None, col.name) + if owner: + owners.add(owner) + return owners + + def _fact_root(self, sm_model_names: list[str]) -> str | None: + """The unique dataset that is never a join target (the fact table), or + ``None`` when there is no unique fact table (0 or >1 candidates).""" + if not sm_model_names: + return None + targets: set[str] = set() + for name in sm_model_names: + for j in self._models[name].joins: + targets.add(j.target_model) + non_targets = [n for n in sm_model_names if n not in targets] + return non_targets[0] if len(non_targets) == 1 else None + + def _make_owner_of(self, sm_model_names: list[str]): + def has_column(model_name: str, column: str) -> bool: + model = self._models.get(model_name) + return model is not None and any(c.name == column for c in model.columns) + + def owner_of(qualifier: Optional[str], column: str) -> Optional[str]: + if qualifier is not None: + # Verify the column actually exists on the qualified model — + # otherwise a metric like SUM(orders.no_such_col) would import + # as a measure that fails at query time. + return qualifier if has_column(qualifier, column) else None + # Unqualified: resolve only when exactly one dataset owns the column. + # Ambiguity (the same column name on multiple datasets) returns None + # so the metric clean-fails instead of binding by dataset order. + matches = [name for name in sm_model_names if has_column(name, column)] + return matches[0] if len(matches) == 1 else None + return owner_of + + def _make_ref_of(self, graph: JoinGraph, anchor: str): + def ref_of(model: str, column: str) -> Optional[str]: + path = graph.shortest_path(anchor, model) + if path is None: + return None + return ".".join([*path, column]) + return ref_of + + def _materialize_columns(self, materialized) -> None: + for mc in materialized: + model = self._models.get(mc.owning_model) + if model is None: + continue + if any(c.name == mc.name for c in model.columns): + continue + model.columns.append(Column( + name=mc.name, sql=mc.sql, type=DataType.DOUBLE, hidden=True, + )) + + # ---- dialect selection ---- + + def _resolve_expression(self, osi_expr: OSIExpression) -> str | None: + """Pick the expression for the requested dialect, else fall back among + SQL-compatible dialects. Non-SQL-only expressions return None. + + The requested dialect is honored only when it is itself SQL-compatible — + a non-SQL request (e.g. ``--dialect MDX``) must not feed non-SQL syntax + into the SQL conversion path; it falls back to an available SQL dialect. + """ + by_dialect = {de.dialect.value: de.expression for de in osi_expr.dialects} + resolved = None + if self.dialect in by_dialect and self.dialect in SQL_DIALECTS: + resolved = self.dialect + else: + resolved = next((n for n in by_dialect if n in SQL_DIALECTS), None) + if resolved is None: + return None + return self._normalize_sql(raw=by_dialect[resolved], dialect_name=resolved) + + @staticmethod + def _normalize_sql(raw: str, dialect_name: str) -> str: + """Normalize a dialect-specific expression to default-dialect SQL so all + downstream sqlglot parsing (which uses the default dialect) succeeds — a + Databricks/Snowflake expression like ``SUM(`amount`)`` would otherwise + fail to parse as default SQL. ANSI_SQL is already default-compatible and + is returned verbatim; an unparseable expression is returned as-is so the + downstream parse clean-fails with a clear reason.""" + source = _SQLGLOT_DIALECT.get(dialect_name) + if source is None: + return raw + try: + return sqlglot.parse_one(raw, read=source).sql() + except sqlglot.errors.ParseError: + return raw diff --git a/slayer/osi/expression.py b/slayer/osi/expression.py new file mode 100644 index 00000000..101f4954 --- /dev/null +++ b/slayer/osi/expression.py @@ -0,0 +1,329 @@ +"""OSI metric/field SQL expression -> SLayer formula transform (DEV-1643). + +An OSI metric carries a raw SQL aggregation expression (e.g. ``SUM(amount)``, +``(SUM(a)) / (COUNT(*))``, ``SUM(quantity * amount)``). SLayer measures use +colon syntax (``amount:sum``, ``*:count``) with arithmetic over aggregated refs. + +``convert_expression`` walks the sqlglot AST, replaces each *outermost* +aggregate subtree with a sentinel, lets sqlglot render the surrounding +arithmetic / scalar-function structure, then substitutes the SLayer ref for each +sentinel. Non-bare aggregate operands (arithmetic / scalar / CASE) are +materialized as hidden derived Columns on the operand's owning model. Anything +inexpressible is clean-failed (``ok=False``) with a reason. + +Model-awareness is injected via two callbacks so the transform is unit-testable +in isolation: +- ``owner_of(qualifier, column) -> model | None`` — which dataset owns a column. +- ``ref_of(model, column) -> anchor-relative dotted ref | None`` — the ref to + emit for a column on ``model`` (``None`` = unreachable from the anchor). +""" + +from __future__ import annotations + +import math +import re +from typing import Callable, Optional + +import sqlglot +import sqlglot.expressions as exp +from pydantic import BaseModel + +from slayer.core.formula import SCALAR_PASSTHROUGH + +# Dialects whose expressions are SQL and can be fed to sqlglot / Column.sql. +SQL_DIALECTS = frozenset({"ANSI_SQL", "SNOWFLAKE", "DATABRICKS"}) + +_SENTINEL_PREFIX = "SLAYERTOKEN" +_SIMPLE_AGG = {exp.Sum: "sum", exp.Avg: "avg", exp.Min: "min", exp.Max: "max"} + +OwnerOf = Callable[[Optional[str], str], Optional[str]] +RefOf = Callable[[str, str], Optional[str]] +NameTaken = Callable[[str, str], bool] # (owning_model, name) -> already exists? + + +class MaterializedColumn(BaseModel): + """A hidden derived Column the converter must create on ``owning_model``.""" + + owning_model: str + name: str + sql: str + + +class ExprResult(BaseModel): + """Outcome of converting one OSI expression to a SLayer formula.""" + + ok: bool + formula: str | None = None + reason: str | None = None + materialized: list[MaterializedColumn] = [] + warnings: list[str] = [] + + +class _Unconvertible(Exception): + """Internal control-flow signal carrying a clean-fail reason.""" + + +class _Converter: + def __init__(self, entity_name: str, owner_of: OwnerOf, ref_of: RefOf, + percentile_unsupported: bool, name_taken: NameTaken) -> None: + self.entity_name = entity_name + self.owner_of = owner_of + self.ref_of = ref_of + self.percentile_unsupported = percentile_unsupported + self.name_taken = name_taken + self.materialized: list[MaterializedColumn] = [] + self.warnings: list[str] = [] + self._dedup: dict[tuple[str, str], str] = {} + self._counter = 0 + + # ---- ref building for a single aggregate ---- + + def _column_ref(self, qualifier: Optional[str], column: str) -> str: + owner = self.owner_of(qualifier or None, column) + if owner is None: + raise _Unconvertible(f"cannot resolve owning dataset for column {column!r}") + ref = self.ref_of(owner, column) + if ref is None: + raise _Unconvertible( + f"column {column!r} on model {owner!r} is not reachable from the anchor" + ) + return ref + + def _materialize(self, operand: exp.Expression) -> str: + """Create/reuse a hidden derived column for a non-bare operand; return + its anchor-relative ref.""" + if operand.find(exp.AggFunc, exp.Window, exp.WithinGroup): + raise _Unconvertible("nested aggregate in aggregate operand") + columns = list(operand.find_all(exp.Column)) + resolved = [self.owner_of(c.table or None, c.name) for c in columns] + # An unresolved column (unknown, or ambiguous) must fail the whole + # operand — discarding it would materialize SQL referencing a column + # that does not exist. + if not columns or any(owner is None for owner in resolved): + raise _Unconvertible( + "aggregate operand references a column that cannot be resolved" + ) + owners = set(resolved) + if len(owners) != 1: + raise _Unconvertible("aggregate operand spans multiple datasets") + owner = owners.pop() + operand_sql = operand.sql() + key = (owner, operand_sql) + name = self._dedup.get(key) + if name is None: + name = self._fresh_column_name(owner) + self._dedup[key] = name + self.materialized.append( + MaterializedColumn(owning_model=owner, name=name, sql=operand_sql) + ) + ref = self.ref_of(owner, name) + if ref is None: + raise _Unconvertible("materialized operand is not reachable from the anchor") + return ref + + def _fresh_column_name(self, owner: str) -> str: + """A hidden-column name that collides with no existing column on + ``owner`` (the formula references this name verbatim, so a collision + would silently aggregate the wrong column).""" + while True: + name = f"_{self.entity_name}_{self._counter}" + self._counter += 1 + if not self.name_taken(owner, name): + return name + + def _operand_ref(self, operand: exp.Expression) -> str: + if isinstance(operand, exp.Column): + return self._column_ref(operand.table or None, operand.name) + return self._materialize(operand) + + def _agg_ref(self, node: exp.Expression) -> str: + """Return the ``ref:agg`` token for one outermost aggregate node.""" + # PERCENTILE_CONT/DISC(...) WITHIN GROUP (ORDER BY col) + if isinstance(node, exp.WithinGroup): + return self._percentile_ref(node) + + if type(node) in _SIMPLE_AGG: + agg = _SIMPLE_AGG[type(node)] + # SLayer supports DISTINCT only for COUNT (handled below); SUM/AVG/ + # MIN/MAX over DISTINCT have no colon-syntax representation. + if isinstance(node.this, exp.Distinct): + raise _Unconvertible(f"{agg.upper()}(DISTINCT ...) is not supported") + return f"{self._operand_ref(node.this)}:{agg}" + + if isinstance(node, exp.Count): + inner = node.this + if inner is None or isinstance(inner, exp.Star): + return "*:count" + if isinstance(inner, exp.Distinct): + exprs = inner.expressions + if len(exprs) != 1: + raise _Unconvertible("COUNT(DISTINCT ...) with multiple columns") + return f"{self._operand_ref(exprs[0])}:count_distinct" + return f"{self._operand_ref(inner)}:count" + + raise _Unconvertible(f"unsupported aggregation {node.sql_name()!r}") + + def _percentile_ref(self, node: exp.WithinGroup) -> str: + inner = node.this + if not isinstance(inner, (exp.PercentileCont, exp.PercentileDisc)): + raise _Unconvertible("unsupported WITHIN GROUP aggregate") + p_node = inner.this + if not (isinstance(p_node, exp.Literal) and p_node.is_number): + raise _Unconvertible("percentile fraction must be a numeric literal") + try: + p_val = float(p_node.name) + except ValueError as exc: # pragma: no cover - defensive + raise _Unconvertible("percentile fraction is not numeric") from exc + if not 0.0 <= p_val <= 1.0: + raise _Unconvertible("percentile fraction must be between 0 and 1") + + order = node.args.get("expression") + if not (isinstance(order, exp.Order) and order.expressions): + raise _Unconvertible("percentile WITHIN GROUP missing ORDER BY") + ordered = order.expressions[0] + col_node = ordered.this if isinstance(ordered, exp.Ordered) else ordered + if not isinstance(col_node, exp.Column): + raise _Unconvertible("percentile ORDER BY must be a bare column") + ref = self._column_ref(col_node.table or None, col_node.name) + + if self.percentile_unsupported: + self.warnings.append( + "percentile/median has no GROUP BY aggregate on the target dialect; " + "the measure imports but fails at query time there." + ) + if isinstance(inner, exp.PercentileCont) and math.isclose(p_val, 0.5): + return f"{ref}:median" + return f"{ref}:percentile(p={p_node.name})" + + # ---- residual validation ---- + + @staticmethod + def _is_sentinel(node: exp.Expression) -> bool: + return isinstance(node, exp.Column) and node.name.startswith(_SENTINEL_PREFIX) + + def _validate_residual(self, root: exp.Expression) -> None: + for node in root.walk(): + reason = self._residual_violation(node) + if reason: + raise _Unconvertible(reason) + + def _residual_violation(self, node: exp.Expression) -> str | None: + """Return the clean-fail reason for a single residual node, or None.""" + if isinstance(node, (exp.AggFunc, exp.Window, exp.WithinGroup)): + return "window/aggregate function not expressible" + if isinstance(node, exp.Case): + return "CASE outside an aggregate is not expressible" + if isinstance(node, exp.Column) and not self._is_sentinel(node): + return f"bare column {node.name!r} must appear inside an aggregation" + if isinstance(node, exp.Literal) and node.is_string: + return "string literal is not expressible in a measure" + if isinstance(node, exp.Func) and not isinstance(node, exp.Count): + if node.sql_name().lower() not in SCALAR_PASSTHROUGH: + return f"function {node.sql_name()!r} is not allowed" + return None + + # ---- top-level ---- + + def convert(self, expr: str) -> ExprResult: + try: + tree = sqlglot.parse_one(expr) + except sqlglot.errors.ParseError as exc: + return self._fail(f"could not parse expression: {exc}") + + if tree.find(exp.Window): + return self._fail("window functions are not expressible; use a transform") + + try: + outermost = self._find_outermost_aggregates(tree) + replacements: list[tuple[exp.Expression, exp.Column]] = [] + for i, node in enumerate(outermost): + ref = self._agg_ref(node) + sentinel = exp.column(f"{_SENTINEL_PREFIX}{i}") + sentinel.meta["slayer_ref"] = ref + replacements.append((node, sentinel)) + + new_root = tree + ref_by_token: dict[str, str] = {} + for node, sentinel in replacements: + ref_by_token[sentinel.name] = sentinel.meta["slayer_ref"] + if node is new_root: + new_root = sentinel + else: + node.replace(sentinel) + + new_root = self._strip_redundant_parens(new_root) + self._validate_residual(new_root) + rendered = new_root.sql(normalize_functions="lower") + formula = self._substitute(rendered, ref_by_token) + except _Unconvertible as exc: + return self._fail(str(exc)) + + return ExprResult( + ok=True, formula=formula, + materialized=self.materialized, warnings=self.warnings, + ) + + @staticmethod + def _find_outermost_aggregates(tree: exp.Expression) -> list[exp.Expression]: + agg_types = (exp.AggFunc, exp.WithinGroup) + result: list[exp.Expression] = [] + for node in tree.find_all(*agg_types): + ancestor = node.parent + nested = False + while ancestor is not None: + if isinstance(ancestor, agg_types): + nested = True + break + ancestor = ancestor.parent + if not nested: + result.append(node) + return result + + @staticmethod + def _strip_redundant_parens(root: exp.Expression) -> exp.Expression: + """Drop parentheses that wrap a single atom (a sentinel ref or literal), + so ``(SUM(a)) / (COUNT(*))`` renders as ``a:sum / *:count``.""" + # Materialize the matches first (a list comprehension, not list(gen)) — + # the tree is mutated in place below, so we can't iterate it lazily. + atom_parens = [ + p for p in root.find_all(exp.Paren) + if isinstance(p.this, (exp.Column, exp.Literal)) + ] + for paren in atom_parens: + if paren is root: + root = paren.this + else: + paren.replace(paren.this) + return root + + @staticmethod + def _substitute(rendered: str, ref_by_token: dict[str, str]) -> str: + out = rendered + for token, ref in ref_by_token.items(): + out = re.sub(rf"\b{re.escape(token)}\b", ref, out) + return out + + def _fail(self, reason: str) -> ExprResult: + return ExprResult(ok=False, formula=None, reason=reason, + materialized=[], warnings=self.warnings) + + +def convert_expression( + expr: str, + *, + entity_name: str, + owner_of: OwnerOf, + ref_of: RefOf, + percentile_unsupported: bool = False, + name_taken: NameTaken = lambda model, name: False, +) -> ExprResult: + """Convert an OSI SQL aggregation expression into a SLayer formula. + + ``name_taken(owning_model, name)`` lets the caller reserve hidden + derived-column names against existing columns so a materialized operand + never collides with a real column. + """ + return _Converter( + entity_name=entity_name, owner_of=owner_of, ref_of=ref_of, + percentile_unsupported=percentile_unsupported, name_taken=name_taken, + ).convert(expr) diff --git a/slayer/osi/models.py b/slayer/osi/models.py new file mode 100644 index 00000000..828a7b2b --- /dev/null +++ b/slayer/osi/models.py @@ -0,0 +1,175 @@ +"""Pydantic v2 models for OSI (Open Semantic Interchange) documents. + +Ported from the OSI reference package (``open-semantic-interchange/OSI``, +``python/src/osi/models.py``, Apache-2.0). The schema is stable across OSI spec +versions 1.0 / 0.1.0 / 0.1.1 / 0.2.0.dev0 — the only differences are the version +string and two optional document-level enum arrays, both absorbed by +``extra="ignore"``. + +Deviations from the reference package: +- ``extra="ignore"`` everywhere (forward/back-compat across spec versions). +- ``vendor_name`` is a free string (0.2.0 widened it from an enum). +- Models are not frozen (the importer does not mutate them, but leaving them + mutable avoids friction with Pydantic ``model_validate`` round-trips in tests). +""" + +from enum import Enum +from typing import Any, Optional, Union + +from pydantic import BaseModel, ConfigDict, Field, field_validator + + +class OSIDialect(str, Enum): + """Supported SQL and expression language dialects.""" + + ANSI_SQL = "ANSI_SQL" + SNOWFLAKE = "SNOWFLAKE" + MDX = "MDX" + MAQL = "MAQL" + TABLEAU = "TABLEAU" + DATABRICKS = "DATABRICKS" + + +class OSIAIContextObject(BaseModel): + """Structured AI context with instructions, synonyms, and examples.""" + + model_config = ConfigDict(extra="allow") + + instructions: Optional[str] = None + synonyms: Optional[list[str]] = None + examples: Optional[list[str]] = None + + +# ai_context is either a plain string or the structured object above. +OSIAIContext = Union[str, OSIAIContextObject] + + +class OSICustomExtension(BaseModel): + """Vendor-specific metadata as a serialized JSON string.""" + + model_config = ConfigDict(extra="ignore") + + vendor_name: str + data: str + + +class OSIDialectExpression(BaseModel): + """Expression in a specific dialect.""" + + model_config = ConfigDict(extra="ignore") + + dialect: OSIDialect + expression: str + + +class OSIExpression(BaseModel): + """Expression definition with multi-dialect support.""" + + model_config = ConfigDict(extra="ignore") + + dialects: list[OSIDialectExpression] + + +class OSIDimension(BaseModel): + """Dimension metadata on a field.""" + + model_config = ConfigDict(extra="ignore") + + is_time: Optional[bool] = None + + +class OSIField(BaseModel): + """Row-level attribute for grouping, filtering, and metric expressions.""" + + model_config = ConfigDict(extra="ignore") + + name: str + expression: OSIExpression + dimension: Optional[OSIDimension] = None + label: Optional[str] = None + description: Optional[str] = None + ai_context: Optional[OSIAIContext] = None + custom_extensions: Optional[list[OSICustomExtension]] = None + + +class OSIDataset(BaseModel): + """Logical dataset representing a business entity (fact or dimension table).""" + + model_config = ConfigDict(extra="ignore") + + name: str + source: str + primary_key: Optional[list[str]] = None + unique_keys: Optional[list[list[str]]] = None + description: Optional[str] = None + ai_context: Optional[OSIAIContext] = None + fields: Optional[list[OSIField]] = None + custom_extensions: Optional[list[OSICustomExtension]] = None + + +class OSIRelationship(BaseModel): + """Foreign key relationship between datasets (``from`` = many, ``to`` = one).""" + + model_config = ConfigDict(extra="ignore", populate_by_name=True) + + name: str + from_dataset: str = Field(..., alias="from") + to: str + from_columns: list[str] + to_columns: list[str] + ai_context: Optional[OSIAIContext] = None + custom_extensions: Optional[list[OSICustomExtension]] = None + + +class OSIMetric(BaseModel): + """Quantitative measure defined on business data (raw SQL aggregation).""" + + model_config = ConfigDict(extra="ignore") + + name: str + expression: OSIExpression + description: Optional[str] = None + ai_context: Optional[OSIAIContext] = None + custom_extensions: Optional[list[OSICustomExtension]] = None + + +class OSISemanticModel(BaseModel): + """Top-level container representing a complete semantic model.""" + + model_config = ConfigDict(extra="ignore") + + name: str + description: Optional[str] = None + ai_context: Optional[OSIAIContext] = None + datasets: list[OSIDataset] + relationships: Optional[list[OSIRelationship]] = None + metrics: Optional[list[OSIMetric]] = None + custom_extensions: Optional[list[OSICustomExtension]] = None + + +class OSIDocument(BaseModel): + """Root OSI document.""" + + model_config = ConfigDict(extra="ignore") + + version: str = "0.2.0.dev0" + semantic_model: list[OSISemanticModel] + + @field_validator("version", mode="before") + @classmethod + def _coerce_version_to_str(cls, v: Any) -> Any: + # YAML parses an unquoted ``version: 1.0`` as a float and ``version: 1`` + # as an int; OSI spec versions are strings. Coerce numeric scalars so a + # valid-but-unquoted version doesn't fail validation (and get skipped). + if isinstance(v, (int, float)): + return str(v) + return v + + +def ai_context_to_dict(ctx: Optional[OSIAIContext]) -> Optional[dict[str, Any]]: + """Normalize an ai_context (string or object) into a plain dict, or None.""" + if ctx is None: + return None + if isinstance(ctx, str): + return {"instructions": ctx} + return ctx.model_dump(exclude_none=True) diff --git a/slayer/osi/parser.py b/slayer/osi/parser.py new file mode 100644 index 00000000..79726be0 --- /dev/null +++ b/slayer/osi/parser.py @@ -0,0 +1,98 @@ +"""Parse OSI config files (YAML or JSON) into ``OSIDocument`` objects. + +``parse_osi_path`` accepts a single file or a directory (walked recursively). +Per-file parse/validation failures are logged and skipped (mirroring the dbt +parser's leniency). Known OSI spec versions parse silently; unknown versions +warn but are still attempted (the schema is stable across versions). +""" + +import json +import logging +import os +from pathlib import Path + +import yaml + +from slayer.osi.models import OSIDocument + +logger = logging.getLogger(__name__) + +# All OSI spec versions are structurally identical (verified via git diff of +# core-spec/osi-schema.json); only the version const and two optional top-level +# enum arrays differ. So every known version parses through the same models. +KNOWN_OSI_VERSIONS = frozenset({"1.0", "0.1.0", "0.1.1", "0.2.0.dev0"}) + +_SUFFIXES = (".yaml", ".yml", ".json") + + +def _collect_files(path: Path) -> list[Path]: + if path.is_file(): + # Apply the same suffix policy as directory scanning. + return [path] if path.name.endswith(_SUFFIXES) else [] + files: list[Path] = [] + for root, dirs, names in os.walk(path): + dirs[:] = [d for d in dirs if not d.startswith(".")] + for name in sorted(names): + if name.startswith("."): + continue + if name.endswith(_SUFFIXES): + files.append(Path(root) / name) + return files + + +def parse_osi_file(path: Path) -> OSIDocument | None: + """Parse a single OSI file into an ``OSIDocument`` (or ``None`` on failure).""" + try: + text = path.read_text(encoding="utf-8") + except OSError as exc: + logger.warning("Failed to read OSI file %s: %s", path, exc) + return None + + try: + if path.suffix == ".json": + data = json.loads(text) + else: + data = yaml.safe_load(text) + except (yaml.YAMLError, json.JSONDecodeError) as exc: + logger.warning("Failed to parse OSI file %s: %s", path, exc) + return None + + if not isinstance(data, dict): + logger.warning("OSI file %s is not a mapping; skipping", path) + return None + + version = data.get("version") + # Coerce for the known-version check: YAML parses an unquoted ``1.0`` as a + # float, which would never match the string set (OSIDocument coerces it too). + if version is not None and str(version) not in KNOWN_OSI_VERSIONS: + logger.warning( + "OSI file %s declares unknown spec version %r (known: %s); " + "attempting to parse anyway.", + path, version, ", ".join(sorted(KNOWN_OSI_VERSIONS)), + ) + + try: + return OSIDocument.model_validate(data) + except Exception as exc: # noqa: BLE001 — any validation error -> skip + logger.warning("Failed to validate OSI document in %s: %s", path, exc) + return None + + +def parse_osi_path(path: str | Path) -> list[OSIDocument]: + """Parse an OSI file or directory into a list of ``OSIDocument`` objects.""" + # Canonicalize the caller-supplied path before touching the filesystem so + # every downstream read works off a resolved, symlink-free base. + root = Path(path).resolve() + if not root.exists(): + raise FileNotFoundError(f"OSI path does not exist: {root}") + + files = _collect_files(root) + if not files: + logger.warning("No OSI files (.yaml/.yml/.json) found in %s", root) + + docs: list[OSIDocument] = [] + for f in files: + doc = parse_osi_file(f) + if doc is not None: + docs.append(doc) + return docs diff --git a/slayer/osi/source.py b/slayer/osi/source.py new file mode 100644 index 00000000..d925c4c6 --- /dev/null +++ b/slayer/osi/source.py @@ -0,0 +1,96 @@ +"""Parse an OSI ``Dataset.source`` into its physical components (DEV-1643). + +``source`` is either a dotted physical identifier (``[catalog.]db.schema.table``, +optionally double-quoted per segment) or a raw SQL query. The identifier form is +split table-last / schema-second-last / database-the-rest; the query form is +carried through to SLayer sql-mode. + +The parsed ``database`` is currently dropped — every dataset binds to the +importer's ``--datasource``. ``resolve_datasource`` is the stubbed extension +point for future per-database routing. +""" + +from __future__ import annotations + +import re + +from pydantic import BaseModel + +_SELECT_RE = re.compile(r"\bselect\b", re.IGNORECASE) + + +class ParsedSource(BaseModel): + """A parsed OSI dataset source.""" + + database: str | None = None + schema_name: str | None = None + table: str | None = None + query: str | None = None + is_query: bool = False + + +def _has_top_level_space(s: str) -> bool: + """True if ``s`` contains whitespace outside of double-quoted spans.""" + in_quote = False + for ch in s: + if ch == '"': + in_quote = not in_quote + elif ch.isspace() and not in_quote: + return True + return False + + +def _looks_like_query(s: str) -> bool: + stripped = s.strip() + # Run the SELECT check on the text OUTSIDE double-quoted spans so a valid + # quoted identifier segment like "My Select" isn't mistaken for a query. + unquoted = re.sub(r'"[^"]*"', "", stripped) + return ( + stripped.startswith("(") + or _has_top_level_space(stripped) + or bool(_SELECT_RE.search(unquoted)) + ) + + +def _split_identifier(s: str) -> list[str]: + """Split a dotted identifier on unquoted dots, stripping double-quotes.""" + parts: list[str] = [] + cur: list[str] = [] + in_quote = False + for ch in s: + if ch == '"': + in_quote = not in_quote + continue + if ch == "." and not in_quote: + parts.append("".join(cur)) + cur = [] + else: + cur.append(ch) + parts.append("".join(cur)) + return parts + + +def parse_source(source: str) -> ParsedSource: + """Parse an OSI ``Dataset.source`` string.""" + if _looks_like_query(source): + return ParsedSource(is_query=True, query=source.strip()) + + parts = [p for p in _split_identifier(source.strip()) if p != ""] + if not parts: + return ParsedSource(is_query=True, query=source.strip()) + + table = parts[-1] + schema_name = parts[-2] if len(parts) >= 2 else None + database = ".".join(parts[:-2]) if len(parts) >= 3 else None + return ParsedSource(database=database, schema_name=schema_name, table=table) + + +def resolve_datasource(database: str | None, default: str) -> str: + """Map an OSI dataset's ``database`` to a SLayer datasource name. + + Stubbed extension point (DEV-1643): for now the parsed database is dropped + and every dataset binds to ``default`` (the importer's ``--datasource``). A + future version can route different OSI databases to different SLayer + datasources here without reworking the converter. + """ + return default diff --git a/slayer/pg_facade/auth.py b/slayer/pg_facade/auth.py index 33666d81..5847948b 100644 --- a/slayer/pg_facade/auth.py +++ b/slayer/pg_facade/auth.py @@ -16,7 +16,8 @@ import hmac import ipaddress import logging -from typing import Optional +from dataclasses import dataclass +from typing import Protocol, runtime_checkable logger = logging.getLogger(__name__) @@ -41,9 +42,15 @@ def _is_loopback(host: str) -> bool: return any(ip in net for net in _LOOPBACK_NETWORKS) -def validate_bind_address(*, host: str, token: Optional[str]) -> None: - """Raise ``ValueError`` if binding a non-loopback address without a token.""" - if token: +def validate_bind_address( + *, host: str, token: str | None, authenticated: bool = False +) -> None: + """Raise ``ValueError`` if binding a non-loopback address without auth. + + ``authenticated`` is set by callers that supply a custom password + authenticator instead of a static token. + """ + if token or authenticated: return if _is_loopback(host): return @@ -53,7 +60,7 @@ def validate_bind_address(*, host: str, token: Optional[str]) -> None: ) -def validate_tls_pair(*, cert: Optional[str], key: Optional[str]) -> None: +def validate_tls_pair(*, cert: str | None, key: str | None) -> None: """TLS cert/key must be supplied together or not at all.""" if (cert is None) != (key is None): raise ValueError( @@ -62,7 +69,7 @@ def validate_tls_pair(*, cert: Optional[str], key: Optional[str]) -> None: ) -def verify_password(client_password: str, expected: Optional[str]) -> bool: +def verify_password(client_password: str, expected: str | None) -> bool: """Constant-time cleartext-password check. When no token is configured (``expected is None``) any non-empty password @@ -73,3 +80,77 @@ def verify_password(client_password: str, expected: Optional[str]) -> bool: if expected is None: return True return hmac.compare_digest(client_password, expected) + + +# --------------------------------------------------------------------------- +# Pluggable authentication +# --------------------------------------------------------------------------- +# +# The facade ships with the static-token check above, but a host application +# (e.g. Motley Storyline) needs to validate the cleartext password against its +# own identity store and scope the connection to a tenant. ``Authenticator`` +# is the seam for that: the connection hands over the startup ``user`` / +# ``database`` parameters plus the cleartext password and gets back an +# ``AuthOutcome`` whose opaque ``principal`` it carries for the rest of the +# session (datasource scoping, RLS, logging). + + +@dataclass +class AuthOutcome: + """Result of an authentication attempt. + + ``principal`` is opaque to the facade — a host-defined object (tenant id, + user, allowed-datasource set) attached to the connection on success. + ``message`` is surfaced to the client only on failure and should stay + generic (don't leak which factor failed). + """ + + ok: bool + principal: object | None = None + message: str = "password authentication failed" + + +@runtime_checkable +class Authenticator(Protocol): + """Validates a Postgres-facade login. + + ``requires_password`` controls the wire handshake: when False the facade + skips the ``AuthenticationCleartextPassword`` exchange and sends + ``AuthenticationOk`` directly (loopback dev mode), still calling + ``authenticate`` with ``password=None`` so the hook can veto. + """ + + @property + def requires_password(self) -> bool: ... + + async def authenticate( + self, *, username: str | None, password: str | None, database: str | None + ) -> AuthOutcome: ... + + +class StaticTokenAuthenticator: + """Default ``Authenticator``: the legacy single-shared-token behaviour. + + Wraps :func:`verify_password` so existing deployments and tests are + unchanged. With no token configured it accepts any non-empty password + (and, with ``requires_password`` False, skips the prompt entirely). + """ + + def __init__(self, token: str | None) -> None: + self._token = token + + @property + def requires_password(self) -> bool: + return self._token is not None + + async def authenticate( # NOSONAR(S7503,S1172) — Authenticator protocol conformance: async is required so callers can `await`, and username/database are part of the protocol signature for richer authenticators (LDAP, RLS-aware) even though the static-token impl is identity-blind + self, *, username: str | None, password: str | None, database: str | None, + ) -> AuthOutcome: + del username, database # static-token auth is identity-blind + # requires_password is False here, so the facade passed password=None; + # treat the no-token loopback case as an unconditional accept. + if self._token is None: + return AuthOutcome(ok=True) + if password is not None and verify_password(password, self._token): + return AuthOutcome(ok=True) + return AuthOutcome(ok=False) diff --git a/slayer/pg_facade/connection.py b/slayer/pg_facade/connection.py index 32320b32..420af3a9 100644 --- a/slayer/pg_facade/connection.py +++ b/slayer/pg_facade/connection.py @@ -18,15 +18,23 @@ import logging import re import struct -from typing import Dict, Iterator, List, Optional, Tuple +import time +from collections.abc import Awaitable, Callable, Iterable +from typing import Iterator, List, Optional, Tuple import sqlglot import sqlglot.errors import sqlglot.expressions as exp from pydantic import BaseModel, ConfigDict +from sqlglot.optimizer.scope import traverse_scope +from slayer.core.enums import DataType from slayer.core.models import SlayerModel -from slayer.facade.catalog import FacadeCatalog, build_catalog +from slayer.engine import timing +from slayer.facade.catalog import ( + FacadeCatalog, + build_catalog_grouped_by_schema, +) from slayer.facade.probe_queries import match_probe as facade_match_probe from slayer.facade.rows import RowBatch from slayer.facade.translator import ( @@ -36,14 +44,20 @@ ProbeResult, QueryResult, READ_ONLY_MESSAGE, + ResetSettingOp, + SetSettingOp, TranslationError, translate, ) +from slayer.facade.catalog_sql import build_catalog_relations, executor_for from slayer.pg_facade import protocol as proto -from slayer.pg_facade.auth import verify_password +from slayer.pg_facade.auth import Authenticator, StaticTokenAuthenticator from slayer.pg_facade.identity import parameter_status_defaults, version_string -from slayer.pg_facade.pg_catalog import match_pg_catalog -from slayer.pg_facade.probes import match_pg_probe +from slayer.pg_facade.probes import ( + SESSION_SETTING_SEED, + SHOW_ALIASES, + match_pg_probe_with_mutation, +) from slayer.pg_facade.types import ( datatype_to_oid, literal_for_substitution, @@ -205,18 +219,80 @@ def _iter_param_placeholders(sql: str) -> Iterator[Tuple[int, int, int]]: yield ph else: i += 1 -# The single schema the facade advertises (matches pg_namespace / current_schema). + + +# Strips characteristics off a statement-initial ``BEGIN`` / ``START +# TRANSACTION`` (``READ ONLY``, ``ISOLATION LEVEL …``, ``DEFERRABLE`` …) so the +# sqlglot-based simple-query splitter — which rejects those forms — can still +# parse the statement list. Anchored to statement start (^ or after ``;``) so a +# ``begin`` column reference elsewhere is never touched. The stripped statement +# stays a plain transaction-open, handled as a no-op downstream (DEV-1594). +_TX_OPEN_STRIP_RE = re.compile( + r"(?P^|;)(?P\s*)" + r"(?PBEGIN(?:\s+WORK|\s+TRANSACTION)?|START\s+TRANSACTION)\b[^;]*", + re.IGNORECASE, +) + + +def _strip_tx_open_characteristics(sql: str) -> str: + return _TX_OPEN_STRIP_RE.sub( + lambda m: f"{m.group('lead')}{m.group('ws')}{m.group('verb')}", sql + ) +# The default schema the facade advertises (matches pg_namespace / +# current_schema). Datasources without an explicit ``postgres_schema`` land here. PUBLIC_SCHEMA = "public" +# Fallback logical-database name (``current_database()`` / ``table_catalog``) +# when the client sends no ``database`` startup parameter. +DEFAULT_DATABASE = "slayer" + +# Per-connection scoping seams: resolve a storage from the authenticated +# principal, and an engine from that storage. +StorageProvider = Callable[[object], Awaitable[object]] +EngineFactory = Callable[[object], object] + + +def _default_engine_factory(storage: object) -> object: + from slayer.engine.query_engine import SlayerQueryEngine + + return SlayerQueryEngine(storage=storage) + +# DEV-1569: GUC_REPORT-class settings. After a successful SET / set_config / +# RESET of one of these, the server pushes a ``ParameterStatus`` message so +# drivers (asyncpg, pgjdbc, c3p0, …) see the new value out-of-band. The +# lowercase key maps to the canonical Postgres wire-case name — real +# Postgres emits ``DateStyle`` / ``TimeZone`` / ``IntervalStyle`` in +# camel-case on the wire even though SQL identifiers are case-insensitive. +# ``integer_datetimes``, ``is_superuser``, ``in_hot_standby``, +# ``default_transaction_read_only`` are GUC_REPORT in real Postgres too but +# the facade doesn't expose them as settable, so we don't list them here. +_GUC_REPORT_NAMES: dict[str, str] = { + "application_name": "application_name", + "client_encoding": "client_encoding", + "datestyle": "DateStyle", + "intervalstyle": "IntervalStyle", + "server_encoding": "server_encoding", + "server_version": "server_version", + "session_authorization": "session_authorization", + "standard_conforming_strings": "standard_conforming_strings", + "timezone": "TimeZone", +} + +# DEV-1570 type aliases — populated by ``_build_column_type_index`` below and +# cached per-connection. Declared here so the ``__init__`` annotation can +# reference them without a forward-ref dance. +ColumnTypeKey = tuple[str, str, str] # (schema_lower, table_lower, column_lower) +ColumnTypeIndex = dict[ColumnTypeKey, DataType] + class _PreparedStatement(BaseModel): sql: str - parameter_oids: List[int] + parameter_oids: list[int] class _Portal(BaseModel): sql: str - result_format_codes: List[int] + result_format_codes: list[int] class _Done(Exception): @@ -231,25 +307,79 @@ def __init__( reader: asyncio.StreamReader, writer: asyncio.StreamWriter, *, - engine, - storage, - token: Optional[str], + engine=None, + storage=None, + token: str | None = None, + authenticator: Authenticator | None = None, + storage_provider: "StorageProvider | None" = None, + engine_factory: "EngineFactory | None" = None, tls_ctx=None, + catalog_extra_relations=None, + catalog_ttl_seconds: float | None = None, ) -> None: self._reader = reader self._writer = writer self._engine = engine self._storage = storage - self._token = token + # Optional per-connection scoping: when ``storage_provider`` is given the + # storage (and engine, via ``engine_factory``) is resolved from the + # authenticated principal after auth — e.g. a tenant-scoped store. + self._storage_provider = storage_provider + self._engine_factory = engine_factory + # Tracks whether ``_resolve_scope`` actually built per-connection + # storage + engine objects (so teardown disposes only what this + # connection owns; an auth failure before the swap mustn't touch + # statically-provided storage / engine the host wants to keep alive). + self._owns_scoped_resources: bool = False + # ``authenticator`` wins; ``token`` is kept for back-compat and wraps + # into the default static-token authenticator. + # Explicit ``is None`` check — a custom authenticator whose + # ``__bool__``/``__len__`` is falsey must not silently fall back + # to the static-token path. + self._authenticator: Authenticator = ( + authenticator if authenticator is not None + else StaticTokenAuthenticator(token) + ) + # Opaque host-defined principal set on successful auth (tenant/user). + self._principal: object | None = None self._tls_ctx = tls_ctx + # Extensibility hook for embedders to override / extend the pg_catalog + # tables — see ``build_catalog_relations(..., extra_relations=...)``. + self._catalog_extras = catalog_extra_relations self._tx_state: bytes = proto.TX_IDLE - self._datasource: Optional[str] = None - self._catalog: Optional[FacadeCatalog] = None - self._statements: Dict[str, _PreparedStatement] = {} - self._portals: Dict[str, _Portal] = {} + # Logical database name (``current_database()`` / ``table_catalog``), + # taken from the ``database`` startup parameter. NOT a model-resolution + # datasource — execution routes per query (see ``QueryResult.data_source``). + self._database: str = DEFAULT_DATABASE + self._catalog: FacadeCatalog | None = None + # On-demand catalog refresh: when a TTL is set, an idle connection + # re-checks storage at most once per window and rebuilds the catalog + # only if the cheap ``graph_fingerprint`` actually moved (see + # ``_maybe_refresh_catalog``). ``None`` keeps the catalog static for + # the connection's lifetime (the historical behavior). + self._catalog_ttl_seconds: float | None = catalog_ttl_seconds + self._catalog_checked_at: float = 0.0 + self._catalog_fingerprint: str | None = None + self._statements: dict[str, _PreparedStatement] = {} + self._portals: dict[str, _Portal] = {} + # Lazily-built (schema, table, column) -> DataType lookup, used by the + # DEV-1570 empty-string-vs-non-text Bind rewrite. Built once per + # connection on first need; ``None`` until then so connections that + # never bind candidates pay zero cost. + self._column_type_index: ColumnTypeIndex | None = None # Extended protocol: after an error the backend discards every message # until the next Sync, then resumes with ReadyForQuery. self._skip_until_sync = False + # DEV-1569: per-connection session-settings mailbox. Captures SET / + # set_config writes; consulted by SHOW / current_setting reads. Seeded + # from the module-level SESSION_SETTING_SEED via dict(...) so each + # connection owns its own copy (never aliasing the seed). + self._session_settings: dict[str, str] = dict(SESSION_SETTING_SEED) + # DEV-1569: when True, ``_describe_sql`` is in flight — translator + # calls must remain pure. Suppresses application of any session- + # setting mutation hints surfaced by the probe matcher during a + # Describe. + self._in_describe = False # ----- lifecycle -------------------------------------------------------- @@ -258,21 +388,49 @@ async def run(self) -> None: startup = await self._handle_startup() if startup is None: return - if not await self._authenticate(): - return - if not await self._resolve_datasource(startup.parameters.get("database")): + if not await self._authenticate(startup): return + await self._resolve_scope(startup.parameters.get("database")) self._catalog = await self._build_catalog() + self._catalog_checked_at = time.monotonic() + self._catalog_fingerprint = await self._read_fingerprint() await self._send_startup_complete() await self._main_loop() except _Done: return except (asyncio.IncompleteReadError, ConnectionResetError): return + finally: + await self._close_scoped_storage() + + async def _close_scoped_storage(self) -> None: + """Release the per-connection storage + engine that ``_resolve_scope`` + built from ``storage_provider`` / ``engine_factory``. + + Gated on ``_owns_scoped_resources`` so an auth failure before the + swap leaves any static storage/engine the host wants to keep alive + untouched. Disposes the engine too (``SlayerQueryEngine.aclose`` + releases the async SQL-client pools — without this a long-lived + facade with ``storage_provider`` leaks one engine per session). + """ + if not self._owns_scoped_resources: + return + engine_aclose = getattr(self._engine, "aclose", None) + if engine_aclose is not None: + try: + await engine_aclose() + except Exception: # noqa: BLE001 — teardown best-effort + logger.exception("pg facade: scoped engine close failed") + storage_aclose = getattr(self._storage, "aclose", None) + if storage_aclose is not None: + try: + await storage_aclose() + except Exception: # noqa: BLE001 — teardown best-effort + logger.exception("pg facade: scoped storage close failed") # ----- startup ---------------------------------------------------------- - async def _read_startup_frame(self) -> Optional[bytes]: + async def _read_startup_frame(self) -> bytes | None: """Read a startup-style frame (no type byte). Returns the body (starting with the 4-byte code) or ``None`` on EOF / malformed length.""" try: @@ -288,7 +446,7 @@ async def _read_startup_frame(self) -> Optional[bytes]: except asyncio.IncompleteReadError: return None - async def _handle_startup(self) -> Optional[proto.StartupMessage]: + async def _handle_startup(self) -> proto.StartupMessage | None: while True: body = await self._read_startup_frame() if body is None: @@ -316,9 +474,9 @@ async def _handle_startup(self) -> Optional[proto.StartupMessage]: return proto.decode_startup(body) async def _perform_tls_upgrade(self) -> None: - """Upgrade the plaintext transport to TLS (best-effort). + """Upgrade the plaintext transport to TLS. - Real TLS is exercised by integration testing; unit tests monkeypatch + Not covered by end-to-end handshake tests yet; unit tests monkeypatch this. asyncio's ``start_tls`` requires the running loop + transport. """ loop = asyncio.get_running_loop() @@ -332,65 +490,158 @@ async def _perform_tls_upgrade(self) -> None: # ----- auth ------------------------------------------------------------- - async def _authenticate(self) -> bool: - if self._token is None: - self._writer.write(proto.encode_authentication_ok()) + async def _authenticate(self, startup: proto.StartupMessage) -> bool: + username = startup.parameters.get("user") + database = startup.parameters.get("database") + + password: str | None = None + if self._authenticator.requires_password: + self._writer.write(proto.encode_authentication_cleartext_password()) await self._flush() - return True - self._writer.write(proto.encode_authentication_cleartext_password()) - await self._flush() - msg = await self._read_message() - if msg is None: - return False - type_char, body = msg - if type_char != "p": - await self._send_error( - code=proto.SQLSTATE_INVALID_AUTHORIZATION, - message="expected password message", - severity="FATAL", - ) - return False - password = proto.decode_password(body) - if not verify_password(password, self._token): + msg = await self._read_message() + if msg is None: + return False + type_char, body = msg + if type_char != "p": + await self._send_error( + code=proto.SQLSTATE_INVALID_AUTHORIZATION, + message="expected password message", + severity="FATAL", + ) + return False + try: + password = proto.decode_password(body) + except (ValueError, struct.error): + # Keep auth-phase malformed input on the wire-protocol + # path; without this, the client gets a silent disconnect + # instead of a Postgres error response. + await self._send_error( + code=proto.SQLSTATE_PROTOCOL_VIOLATION, + message="malformed password message", + severity="FATAL", + ) + return False + + outcome = await self._authenticator.authenticate( + username=username, password=password, database=database + ) + if not outcome.ok: await self._send_error( code=proto.SQLSTATE_INVALID_PASSWORD, - message="password authentication failed", + message=outcome.message, severity="FATAL", ) return False + + self._principal = outcome.principal self._writer.write(proto.encode_authentication_ok()) await self._flush() return True - # ----- datasource resolution ------------------------------------------- + # ----- scope resolution ------------------------------------------------ - async def _resolve_datasource(self, database: Optional[str]) -> bool: - datasources = await self._storage.list_datasources() - if database and database in datasources: - self._datasource = database - return True - name = database if database else "(none)" - await self._send_error( - code=proto.SQLSTATE_UNDEFINED_DATABASE, - message=f'database "{name}" does not exist', - severity="FATAL", - ) - return False + async def _resolve_scope(self, database: str | None) -> None: + """Resolve per-connection scope after auth. + + The ``database`` startup parameter is the logical database name (one + emulated DB per instance/tenant), not a datasource selector — every + datasource the storage exposes appears as a schema. When a + ``storage_provider`` is configured, the storage (and engine) is + re-resolved from the authenticated principal so a host can scope the + connection to one tenant. + """ + self._database = database or DEFAULT_DATABASE + if self._storage_provider is not None: + self._storage = await self._storage_provider(self._principal) + factory = self._engine_factory or _default_engine_factory + self._engine = factory(self._storage) + # Mark only after BOTH constructions succeed — partial state + # would leave teardown unsure which side to dispose. + self._owns_scoped_resources = True async def _build_catalog(self) -> FacadeCatalog: - assert self._datasource is not None - models: List[SlayerModel] = [] - names = await self._storage.list_models(data_source=self._datasource) - for name in names: - model = await self._storage.get_model(name=name, data_source=self._datasource) - if model is not None: - models.append(model) - # The Postgres facade advertises a single schema `public` (matching - # pg_namespace / current_schema()), so the catalog's schema is named - # `public` — this keeps qualified `public.
` resolution working. - # The real datasource is carried separately (self._datasource) and - # passed to the engine as the execution hint. - return build_catalog(models_by_datasource={PUBLIC_SCHEMA: models}) + models_by_datasource: dict[str, list[SlayerModel]] = {} + schema_by_datasource: dict[str, str] = {} + for datasource in await self._storage.list_datasources(): + names = await self._storage.list_models(data_source=datasource) + models = [ + model + for name in names + if (model := await self._storage.get_model( + name=name, data_source=datasource, + )) is not None + ] + models_by_datasource[datasource] = models + config = await self._storage.get_datasource(datasource) + if config is not None and config.postgres_schema: + schema_by_datasource[datasource] = config.postgres_schema + priority = await self._storage.get_datasource_priority() + return build_catalog_grouped_by_schema( + models_by_datasource=models_by_datasource, + schema_by_datasource=schema_by_datasource, + datasource_priority=priority, + default_schema=PUBLIC_SCHEMA, + ) + + async def _read_fingerprint(self) -> str | None: + """Cheap storage staleness token, or ``None`` when unavailable. + + Only consulted when a catalog TTL is configured. ``OSError`` (e.g. a + file backend caught mid-write) is treated as "unknown" so the next + check forces a rebuild, matching the search-graph convention. + """ + if self._catalog_ttl_seconds is None: + return None + try: + return await self._storage.graph_fingerprint() + except OSError: + return None + + async def _maybe_refresh_catalog(self) -> None: + """On-demand, TTL-throttled, change-gated catalog rebuild. + + Called at statement entry. Rebuilds the per-connection catalog only + when (a) a TTL is configured, (b) the connection is idle — never + mid-transaction, to avoid a catalog shift inside a txn, (c) the TTL + window has elapsed since the last check, and (d) the storage + fingerprint has actually changed. When nothing changed the cost is a + single ``graph_fingerprint`` read per window. Backends that don't + implement a real fingerprint report a constant, so they never rebuild + and behave exactly as before. + + Best-effort: a transient storage failure during the fingerprint read + or the rebuild must not propagate out of ``_run_statement`` and tear + down the client connection. On failure we keep the existing (possibly + stale) catalog and retry on the next TTL window. + """ + if self._catalog_ttl_seconds is None or self._catalog is None: + return + if self._tx_state != proto.TX_IDLE: + return + now = time.monotonic() + if now - self._catalog_checked_at < self._catalog_ttl_seconds: + return + # Stamp the check time up front so a failing refresh retries no sooner + # than the next window rather than hammering storage every statement. + self._catalog_checked_at = now + try: + fingerprint = await self._read_fingerprint() + if fingerprint is not None and fingerprint == self._catalog_fingerprint: + return + # Build into a local and swap only on success, so a failed rebuild + # never leaves the connection with a half-built or ``None`` catalog. + catalog = await self._build_catalog() + # Refresh is best-effort: keep the old catalog on any storage failure. + except Exception: # noqa: BLE001 + logger.warning( + "pg facade: catalog refresh failed; keeping current catalog", + exc_info=True, + ) + return + self._catalog = catalog + self._catalog_fingerprint = fingerprint + # Derived from the catalog; drop it so it rebuilds against the new one. + self._column_type_index = None async def _send_startup_complete(self) -> None: for name, value in parameter_status_defaults(): @@ -479,12 +730,24 @@ async def _handle_simple_query(self, sql: str) -> None: try: statements = [s for s in sqlglot.parse(sql, dialect="postgres") if s is not None] except sqlglot.errors.ParseError as exc: - await self._send_error( - code=proto.SQLSTATE_SYNTAX_ERROR, message=f"SQL parse error: {exc}", - ) - self._fail_tx() - await self._send_ready() - return + # BI tools (Metabase) wrap reads in ``BEGIN READ ONLY`` etc., which + # sqlglot can't parse. Strip the transaction characteristics and + # retry once before surfacing a syntax error. + stripped = _strip_tx_open_characteristics(sql) + try: + statements = [ + s for s in sqlglot.parse(stripped, dialect="postgres") if s is not None + ] if stripped != sql else None + except sqlglot.errors.ParseError: + statements = None + if statements is None: + logger.warning("pg facade: cannot parse simple query %r: %s", sql, exc) + await self._send_error( + code=proto.SQLSTATE_SYNTAX_ERROR, message=f"SQL parse error: {exc}", + ) + self._fail_tx() + await self._send_ready() + return if not statements: self._writer.write(proto.encode_empty_query_response()) await self._send_ready() @@ -549,11 +812,19 @@ def _substitute_params(self, stmt: _PreparedStatement, bind: proto.BindMessage) return stmt.sql formats = proto.parse_result_format_codes(bind.parameter_format_codes, n) oids = list(resolved) - literals: List[str] = [] - for raw, fmt, oid in zip(bind.parameter_values, formats, oids): + empty_string_null_params = self._empty_string_null_params_for_bind( + sql=stmt.sql, raw_values=bind.parameter_values, oids=oids, + ) + literals: list[str] = [] + for i, (raw, fmt, oid) in enumerate( + zip(bind.parameter_values, formats, oids), start=1, + ): if raw is None: literals.append("NULL") continue + if i in empty_string_null_params: + literals.append("NULL") + continue value = ( value_from_text(raw, oid) if fmt == proto.FORMAT_TEXT else value_from_binary(raw, oid) @@ -574,7 +845,38 @@ def _substitute_params(self, stmt: _PreparedStatement, bind: proto.BindMessage) parts.append(stmt.sql[last:]) return "".join(parts) + def _empty_string_null_params_for_bind( + self, *, sql: str, raw_values, oids: list[int], + ) -> set[int]: + # DEV-1570: pre-classify $N indices whose bound value is an empty + # text-OID payload AND whose AST occurrence targets a non-TEXT catalog + # column. Those positions emit ``NULL`` rather than ``''`` so DuckDB + # doesn't trip a ``Could not convert string '' to INT64`` at Execute. + # The column-type index is built lazily on first need. + candidates = [ + i + 1 for i, (raw, oid) in enumerate(zip(raw_values, oids)) + if oid == proto.OID_TEXT and raw == b"" + ] + if not candidates or self._catalog is None: + return set() + if self._column_type_index is None: + self._column_type_index = _build_column_type_index( + catalog=self._catalog, datasource=self._database, + extra_relations=self._catalog_extras, + ) + return _classify_empty_string_param_targets( + sql=sql, + column_type_index=self._column_type_index, + candidate_param_indices=candidates, + ) + async def _handle_describe(self, msg: proto.DescribeMessage) -> None: + # Refresh here too, not just at Execute: Describe advertises the + # RowDescription, and it stamps the TTL check so the following Execute + # stays in the same window and won't shift the catalog underneath it — + # otherwise a mid-window edit could make the rows disagree with the + # already-sent RowDescription. + await self._maybe_refresh_catalog() if msg.kind == "S": stmt = self._statements.get(msg.name) if stmt is None: @@ -583,8 +885,11 @@ async def _handle_describe(self, msg: proto.DescribeMessage) -> None: message=f"prepared statement {msg.name!r} does not exist", ) return - self._writer.write(proto.encode_parameter_description(_resolve_param_oids(stmt))) - self._describe_sql(stmt.sql, result_formats=None) + param_oids = _resolve_param_oids(stmt) + self._writer.write(proto.encode_parameter_description(param_oids)) + self._describe_sql( + stmt.sql, result_formats=None, param_oids=param_oids, + ) else: portal = self._portals.get(msg.name) if portal is None: @@ -593,16 +898,45 @@ async def _handle_describe(self, msg: proto.DescribeMessage) -> None: message=f"portal {msg.name!r} does not exist", ) return - self._describe_sql(portal.sql, result_formats=portal.result_format_codes) + # Portal-describe: the bound values have already been + # substituted into portal.sql by _handle_bind, so no $N + # remain to typed-sentinel. + self._describe_sql( + portal.sql, result_formats=portal.result_format_codes, + ) - def _describe_sql(self, sql: str, *, result_formats: Optional[List[int]]) -> None: + def _describe_sql( + self, sql: str, *, result_formats: list[int] | None, + param_oids: list[int] | None = None, + ) -> None: + # DEV-1558 fix: the catalog executor's Describe path runs the SQL + # against DuckDB to obtain the cursor's column description. When the + # prepared-statement form still has ``$N`` placeholders (asyncpg + # sends Parse + Describe-Statement BEFORE Bind), DuckDB raises a + # bind-parameter error. Substitute each ``$N`` with a TYPED + # sentinel literal derived from the parameter's declared OID so + # the resulting RowDescription advertises the correct projection + # types even when ``$N`` appears in the projection itself. The + # real value substitution still happens in ``_handle_bind`` for + # Execute (Codex round 13 review). + describe_sql = _substitute_typed_sentinels(sql, param_oids or []) + # DEV-1569: ``_in_describe`` suppresses application of any + # session-setting mutations surfaced by the translator during a + # Describe pass. The Execute path applies them. + self._in_describe = True try: - result = self._translate(sql) - except TranslationError: - # Describe must not raise to the wire here; the subsequent Execute - # surfaces the error. Report NoData so the client can proceed. - self._writer.write(proto.encode_no_data()) - return + try: + result = self._translate(describe_sql) + except TranslationError as exc: + # Describe must not raise to the wire here; the subsequent + # Execute surfaces the error. Report NoData so the client + # can proceed. Log it (debug) so the silent Describe path is + # still greppable when a client swallows the later error. + logger.debug("pg facade: cannot describe %r: %s", describe_sql, exc) + self._writer.write(proto.encode_no_data()) + return + finally: + self._in_describe = False fields = self._fields_for_result(result, result_formats) if fields is None: self._writer.write(proto.encode_no_data()) @@ -669,35 +1003,74 @@ def _translate(self, sql: str): self._catalog, dialect="postgres", probe_matcher=self._probe_matcher, - catalog_matchers=[match_pg_catalog], + # Pass a lazy factory so the DuckDB executor is only + # materialised when ``is_catalog_only(parsed)`` is True + # (Codex round 16). Non-catalog model queries skip the + # construction cost entirely. + catalog_sql_executor=lambda: executor_for( + self._catalog, self._database, + extra_relations=self._catalog_extras, + ), + # Convenience for interactive psql sessions: ``SELECT * FROM t`` + # in browse mode (no GROUP BY / HAVING / aggregate) expands to + # every non-hidden column. Flight stays strict (its clients + # always project explicit names). + expand_star_in_browse_mode=True, ) - def _probe_matcher(self, parsed: exp.Expression) -> Optional[RowBatch]: - assert self._datasource is not None - pg = match_pg_probe( - parsed, datasource=self._datasource, version_str=version_string(), + def _probe_matcher(self, parsed: exp.Expression): + """Wraps the PG-facade and shared probe matchers. + + DEV-1569: ``SHOW`` / ``current_setting`` consult the per-connection + ``self._session_settings``. For ``set_config`` matches, the returned + ``ProbeMatcherOutcome`` carries a mutation hint that + ``_run_statement`` applies on Execute (Describe leaves it pending). + """ + pg = match_pg_probe_with_mutation( + parsed, datasource=self._database, version_str=version_string(), + session_settings=self._session_settings, ) if pg is not None: return pg return facade_match_probe(parsed) async def _run_statement( - self, sql: str, *, result_formats: Optional[List[int]], send_row_description: bool, + self, sql: str, *, result_formats: list[int] | None, send_row_description: bool, ) -> bool: """Translate + respond. Returns False if an error was sent.""" + # Refresh the catalog before translating so an idle connection picks + # up model/schema edits within the TTL window (no-op when disabled or + # mid-transaction). + await self._maybe_refresh_catalog() try: result = self._translate(sql) except TranslationError as exc: + # Clients (BI pools especially) often swallow the statement that + # failed; log it server-side so unsupported-SQL gaps are visible. + logger.warning("pg facade: cannot translate %r: %s", sql, exc) await self._send_error(code=_sqlstate_for(exc), message=str(exc)) self._fail_tx() return False if isinstance(result, (ProbeResult, InfoSchemaResult, PgCatalogResult)): self._emit_row_batch(result.batch, result_formats, send_row_description) + # DEV-1569: set_config(...) mutation hint surfaces on ProbeResult. + # Apply ONLY in the Execute path (not Describe). Pushes + # ParameterStatus for reportable settings after CommandComplete. + if isinstance(result, ProbeResult) and result.settings_mutation is not None: + self._apply_set_setting(result.settings_mutation) return True if isinstance(result, NoOpResult): self._apply_tx_command(result.command_tag) self._writer.write(proto.encode_command_complete(_command_tag(result.command_tag))) + # DEV-1569: apply SET / RESET captures AFTER CommandComplete so + # the post-SET ParameterStatus push lands between CC and + # ReadyForQuery (any-time ordering OK per PG protocol). Skipped + # during Describe per _describe_sql / _in_describe. + if result.set_setting is not None: + self._apply_set_setting(result.set_setting) + if result.reset_setting is not None: + self._apply_reset_setting(result.reset_setting) return True if isinstance(result, QueryResult): return await self._run_query(result, result_formats, send_row_description) @@ -709,7 +1082,7 @@ async def _run_statement( return False def _emit_row_batch( - self, batch: RowBatch, result_formats: Optional[List[int]], send_row_description: bool, + self, batch: RowBatch, result_formats: list[int] | None, send_row_description: bool, ) -> None: formats = proto.parse_result_format_codes(result_formats or [], len(batch.columns)) if send_row_description: @@ -722,23 +1095,37 @@ def _emit_row_batch( for i, col in enumerate(batch.columns) ] self._writer.write(proto.encode_row_description(fields)) + # The catalog SQL executor stashes a position-aware key list on + # the batch so duplicate output column names (Postgres allows + # ``SELECT oid AS x, relname AS x``) don't collapse to a single + # dict entry. Fall back to ``col.name`` for batches built by the + # canned probe / info-schema paths where duplicates can't arise. + row_keys = getattr(batch, "_row_keys", None) or [c.name for c in batch.columns] for row in batch.rows: values = [ - _encode_value(row.get(col.name), datatype_to_oid(col.type), formats[i]) - for i, col in enumerate(batch.columns) + _encode_value(row.get(row_keys[i]), datatype_to_oid(batch.columns[i].type), formats[i]) + for i in range(len(batch.columns)) ] self._writer.write(proto.encode_data_row(values)) self._writer.write(proto.encode_command_complete(f"SELECT {len(batch.rows)}")) async def _run_query( - self, result: QueryResult, result_formats: Optional[List[int]], send_row_description: bool, + self, result: QueryResult, result_formats: list[int] | None, send_row_description: bool, ) -> bool: try: - response = await self._engine.execute( - query=result.query, data_source=self._datasource, - ) + # The translator resolves the per-query datasource from the + # referenced model(s) and rejects cross-datasource joins. A model + # query always carries one; guard the impossible None rather than + # passing it to the engine. + if result.data_source is None: + raise ValueError("could not resolve a datasource for the query") + with timing.open_query_profile(): + response = await self._engine.execute( + query=result.query, data_source=result.data_source, + ) except Exception as exc: # noqa: BLE001 — surface any engine error to the client - await self._send_error(code=proto.SQLSTATE_INTERNAL_ERROR, message=str(exc)) + code, message = _engine_error_fields(exc) + await self._send_error(code=code, message=message) self._fail_tx() return False mapping = result.column_name_mapping @@ -764,8 +1151,8 @@ async def _run_query( return True def _fields_for_result( - self, result, result_formats: Optional[List[int]], - ) -> Optional[List[proto.FieldDescription]]: + self, result, result_formats: list[int] | None, + ) -> list[proto.FieldDescription] | None: if isinstance(result, (ProbeResult, InfoSchemaResult, PgCatalogResult)): cols = result.batch.columns formats = proto.parse_result_format_codes(result_formats or [], len(cols)) @@ -789,7 +1176,7 @@ def _fields_for_result( # ----- transaction state ------------------------------------------------- - def _apply_tx_command(self, command_tag: Optional[str]) -> None: + def _apply_tx_command(self, command_tag: str | None) -> None: if command_tag in ("BEGIN", "START TRANSACTION"): self._tx_state = proto.TX_IN_TRANSACTION elif command_tag in ("COMMIT", "ROLLBACK", "END"): @@ -799,6 +1186,62 @@ def _fail_tx(self) -> None: if self._tx_state == proto.TX_IN_TRANSACTION: self._tx_state = proto.TX_FAILED + # ----- DEV-1569: per-connection session-settings application ----------- + + def _apply_set_setting(self, op: SetSettingOp) -> None: + """Mutate the per-connection session-settings map and (for reportable + names) push a ``ParameterStatus`` message to the client. + + Skipped during ``_describe_sql`` (Describe must remain pure — the + same prepared statement may be Executed later, at which point + mutation happens). + """ + if self._in_describe: + return + self._session_settings[op.name] = op.value + self._push_parameter_status_if_reportable(op.name, op.value) + + def _apply_reset_setting(self, op: ResetSettingOp) -> None: + """Restore the per-connection session-settings map per the RESET + intent. Pushes ``ParameterStatus`` for each reportable name whose + value changed back to seed. + + DEV-1569 / Codex F2: multi-word names (``RESET TIME ZONE``, + ``RESET SESSION AUTHORIZATION``) are alias-resolved via the same + ``SHOW_ALIASES`` table that ``SHOW`` consults; without the + resolution the lookup against ``SESSION_SETTING_SEED`` would + silently miss. + """ + if self._in_describe: + return + if op.reset_all: + # Restore every name to its seed value; push ParameterStatus for + # the seeded value of every reportable name (drivers latch onto + # the post-RESET-ALL pushes to invalidate caches). + self._session_settings = dict(SESSION_SETTING_SEED) + for lower, _wire_name in _GUC_REPORT_NAMES.items(): + value = self._session_settings.get(lower, "") + self._push_parameter_status_if_reportable(lower, value) + return + # RESET : alias-resolve (multi-word names), then revert to + # seed (if seeded) or drop the override. + name = SHOW_ALIASES.get(op.name or "", op.name or "") + if name in SESSION_SETTING_SEED: + self._session_settings[name] = SESSION_SETTING_SEED[name] + self._push_parameter_status_if_reportable( + name, self._session_settings[name], + ) + else: + self._session_settings.pop(name, None) + # Non-seeded names are by definition not reportable (the + # _GUC_REPORT_NAMES set is a subset of the seed), so no push. + + def _push_parameter_status_if_reportable(self, name: str, value: str) -> None: + wire_name = _GUC_REPORT_NAMES.get(name) + if wire_name is None: + return + self._writer.write(proto.encode_parameter_status(wire_name, value)) + # ----- IO helpers -------------------------------------------------------- async def _send_ready(self) -> None: @@ -818,7 +1261,373 @@ async def _flush(self) -> None: # --- module-level helpers ---------------------------------------------------- -def _resolve_param_oids(stmt: _PreparedStatement) -> List[int]: +# Typed sentinel literal per parameter OID. Used by Describe-Statement to +# substitute ``$N`` placeholders BEFORE Bind so DuckDB can produce a +# valid RowDescription whose column types reflect the projection's +# dependence on the parameter (Codex round 13). +# +# Each sentinel is a typed NULL (``CAST(NULL AS )``) rather than a +# concrete literal: NULL is universally comparable (``col = CAST(NULL AS +# TEXT)`` always returns NULL/FALSE under DuckDB's standard SQL +# semantics), so we never trigger a conversion error like +# ``Conversion Error: Could not convert string '' to INT64`` when the +# parameter appears in a comparison against a column of a different +# type than the pgjdbc-declared OID — Metabase corpus #9 had pgjdbc +# declaring text OIDs for parameters that compared against int columns +# (``objsubid = $N``), which the literal ``''`` sentinel turned into +# an unanswerable text-vs-int comparison. +_TYPED_SENTINEL_BY_OID: dict[int, str] = { + proto.OID_TEXT: "CAST(NULL AS VARCHAR)", + proto.OID_INT8: "CAST(NULL AS BIGINT)", + proto.OID_FLOAT8: "CAST(NULL AS DOUBLE)", + proto.OID_BOOL: "CAST(NULL AS BOOLEAN)", + proto.OID_DATE: "CAST(NULL AS DATE)", + proto.OID_TIMESTAMP: "CAST(NULL AS TIMESTAMP)", +} + + +def _substitute_typed_sentinels(sql: str, param_oids: list[int]) -> str: + """Replace each ``$N`` placeholder with a typed sentinel literal + derived from ``param_oids[N-1]``. Falls back to bare ``NULL`` when + the OID is unknown (e.g. extra placeholders past the declared list) + so DuckDB picks the most permissive coercion path.""" + def repl(match): + idx = int(match.group(1)) - 1 + if 0 <= idx < len(param_oids): + return _TYPED_SENTINEL_BY_OID.get(param_oids[idx], "NULL") + return "NULL" + return _PARAM_PLACEHOLDER.sub(repl, sql) + + +# DEV-1570: empty-string-vs-non-text Bind rewrite ----------------------------- +# +# Symmetric with the Describe-side typed-NULL substitution above. pgjdbc / +# Metabase binds an empty string (``b""``) for "null" against text-OID +# parameters; when that parameter targets a non-TEXT catalog column the +# resulting ``WHERE int_col = ''`` previously tripped DuckDB's +# ``Conversion Error: Could not convert string '' to INT64`` at Execute. +# The classifier below identifies $N indices whose AST occurrences land in a +# comparison / IN / BETWEEN predicate against a column that resolves via the +# FacadeCatalog to a non-TEXT DataType, so ``_substitute_params`` can swap +# their literal to NULL for those positions. + +# Mirrors slayer.facade.catalog_sql._PG_CATALOG_NAMES — duplicating here keeps +# the classifier independent of catalog_sql internals. Bare names in user SQL +# resolve to pg_catalog only when they match a known relation (per the +# catalog executor's convention at slayer/facade/catalog_sql.py:712). +_PG_CATALOG_RELATIONS: frozenset = frozenset({ + "pg_namespace", "pg_class", "pg_attribute", "pg_type", "pg_proc", + "pg_settings", "pg_description", "pg_stat_user_tables", "pg_enum", + "pg_tables", "pg_views", "pg_matviews", "pg_constraint", "pg_index", + "pg_attrdef", +}) + +# Binary-comparison sqlglot node classes the classifier walks. Excludes LIKE / +# ILIKE (text-only operators; collision with the empty-string-vs-non-text bug +# is not possible). +_COMPARISON_NODE_TYPES: tuple = ( + exp.EQ, exp.NEQ, exp.LT, exp.LTE, exp.GT, exp.GTE, + exp.NullSafeEQ, exp.NullSafeNEQ, +) + + +def _build_column_type_index( + *, catalog: FacadeCatalog, datasource: str, + extra_relations=None, +) -> ColumnTypeIndex: + """Build the (schema_lower, table_lower, column_lower) -> DataType lookup + used by the Bind-time empty-string-to-NULL rewrite (DEV-1570). + + Covers pg_catalog.*, information_schema.* (via the ``_is_`` + builder-name remap), and user-model tables under ``PUBLIC_SCHEMA``. + """ + out: ColumnTypeIndex = {} + _index_catalog_relations( + out=out, catalog=catalog, datasource=datasource, + extra_relations=extra_relations, + ) + _index_user_tables(out=out, catalog=catalog) + return out + + +def _index_catalog_relations( + *, out: ColumnTypeIndex, catalog: FacadeCatalog, datasource: str, + extra_relations=None, +) -> None: + """Populate ``out`` with pg_catalog / information_schema column types + materialised by ``build_catalog_relations``. The ``_is_`` builder + convention is remapped to the SQL-visible ``information_schema.``.""" + for rel in build_catalog_relations( + catalog=catalog, datasource=datasource, + extra_relations=extra_relations, + ): + if rel.name.startswith("_is_"): + schema = "information_schema" + table = rel.name[len("_is_"):] + else: + schema = "pg_catalog" + table = rel.name + table_lower = table.lower() + for col in rel.columns: + _record_column_type( + out=out, key=(schema, table_lower, col.name.lower()), + dt=col.type, + ) + + +def _index_user_tables(*, out: ColumnTypeIndex, catalog: FacadeCatalog) -> None: + """Populate ``out`` with user-model column types under PUBLIC_SCHEMA.""" + for sch in catalog.schemas: + schema_lower = sch.name.lower() + for tbl in sch.tables: + tbl_lower = tbl.name.lower() + for d in tbl.dimensions: + _record_column_type( + out=out, key=(schema_lower, tbl_lower, d.name.lower()), + dt=d.data_type, + ) + for m in tbl.metrics: + dt = m.data_type if m.data_type is not None else DataType.TEXT + _record_column_type( + out=out, key=(schema_lower, tbl_lower, m.name.lower()), + dt=dt, + ) + + +def _record_column_type( + *, out: ColumnTypeIndex, key: ColumnTypeKey, dt: DataType, +) -> None: + """Insert (key → dt) into ``out`` unless the key already exists with a + different type. Case-distinct quoted identifiers (e.g. ``id`` INT and + ``"ID"`` TEXT in the same model) would otherwise collide because the + index lower-cases column names — last-writer-wins on the previous + implementation could rewrite an empty-string text comparison to NULL. + Skip the second occurrence and warn once. Same-type collisions are + silently ignored (no behaviour change). DEV-1570 / Codex CX-2.""" + existing = out.get(key) + if existing is None: + out[key] = dt + return + if existing != dt: + logger.warning( + "DEV-1570 column-type index: case-distinct collision on %s " + "(existing=%s, new=%s); keeping existing. Empty-string rewrite " + "may behave incorrectly if the case-distinct column is queried.", + key, existing, dt, + ) + + +def _classify_empty_string_param_targets( + *, sql: str, + column_type_index: ColumnTypeIndex, + candidate_param_indices: Iterable[int], +) -> set[int]: + """Return the subset of ``candidate_param_indices`` whose AST occurrences + appear in a comparison / IN / BETWEEN predicate against a column that + resolves via ``column_type_index`` to a non-TEXT ``DataType``. + + Whole-parameter granularity: if ANY occurrence of $N targets a non-text + column, $N is in the result set so ``_substitute_params`` substitutes + ``NULL`` everywhere $N appears. + + Returns the empty set on parse failure or unexpected AST shape; the + helper never raises (per Codex round 1, finding #1). + """ + candidates = set(candidate_param_indices) + if not candidates: + return set() + # sqlglot 30.4.3's tokenizer rejects PostgreSQL placeholders adjacent to + # punctuation in some compact shapes (e.g. ``IN ($1,$2)`` no-whitespace + # form trips a TokenError, while ``IN ($1, $2)`` parses cleanly). + # Pad each $N with surrounding whitespace before parsing — this only + # affects the AST we walk for classification; the literal substitution + # downstream still uses the original ``stmt.sql``. + normalised_sql = _PARAM_PLACEHOLDER.sub(r" $\1 ", sql) + try: + parsed = sqlglot.parse_one(sql=normalised_sql, dialect="postgres") + except sqlglot.errors.SqlglotError: + return set() + if parsed is None: + return set() + try: + column_to_table = _map_column_tables( + parsed=parsed, column_type_index=column_type_index, + ) + return _collect_non_text_params( + parsed=parsed, candidates=candidates, + column_to_table=column_to_table, + column_type_index=column_type_index, + ) + except Exception: # NOSONAR(S110) — defensive: never raise out of bind path + logger.debug("DEV-1570 classifier defensive catch", exc_info=True) + return set() + + +def _map_column_tables( + *, parsed: exp.Expression, column_type_index: ColumnTypeIndex, +) -> dict[int, tuple[str, str] | None]: + """Resolve every Column node to its owning scope's (schema, table).""" + column_to_table: dict[int, tuple[str, str] | None] = {} + for scope in traverse_scope(parsed): + sources = _resolved_table_sources(scope.sources) + for col in scope.find_all(exp.Column): + column_to_table[id(col)] = _resolve_column_table( + col=col, scope_sources=sources, + column_type_index=column_type_index, + ) + return column_to_table + + +def _collect_non_text_params( + *, parsed: exp.Expression, + candidates: set[int], + column_to_table: dict[int, tuple[str, str] | None], + column_type_index: ColumnTypeIndex, +) -> set[int]: + """Walk comparison / IN / BETWEEN nodes; collect $N indices whose paired + Column resolves to a non-TEXT ``DataType``.""" + result: set[int] = set() + for node in parsed.walk(): + for col, param_idx in _column_param_pairs_from_node(node): + if param_idx not in candidates: + continue + table = column_to_table.get(id(col)) + if table is None: + continue + key: ColumnTypeKey = (table[0], table[1], col.name.lower()) + dt = column_type_index.get(key) + if dt is not None and dt != DataType.TEXT: + result.add(param_idx) + return result + + +def _column_param_pairs_from_node(node) -> Iterable[tuple[exp.Column, int]]: + """Yield (Column, param_index) pairs for each comparison-shape node.""" + if isinstance(node, _COMPARISON_NODE_TYPES): + yield from _pair_column_and_param(node.this, node.expression) + return + if isinstance(node, exp.Between): + value = node.this + for bound_key in ("low", "high"): + bound = node.args.get(bound_key) + if bound is not None: + yield from _pair_column_and_param(value, bound) + return + if isinstance(node, exp.In): + value = node.this + for el in (node.expressions or []): + yield from _pair_column_and_param(value, el) + + +def _try_extract_column(node) -> exp.Column | None: + """Return the underlying ``exp.Column`` if ``node`` is one (optionally + wrapped in semantically-transparent ``exp.Paren`` layers). CAST / function + / arithmetic wrappers around a column are documented out of scope — + pin tests `test_cast_wrapped_column_not_classified`, + `test_arithmetic_wrapped_column_not_classified`, + `test_function_wrapped_column_not_classified`.""" + while isinstance(node, exp.Paren): + node = node.this + return node if isinstance(node, exp.Column) else None + + +def _try_extract_param_index(node) -> int | None: + """Return the 1-based ``$N`` index if ``node`` is an ``exp.Parameter`` + (optionally wrapped in ``exp.Paren`` and/or ``exp.Cast`` layers). + + Unwrapping ``exp.Cast`` is parameter-side-only (Codex CX-3): a user + writing ``objsubid = $1::int`` or ``objsubid = CAST($1 AS INT)`` has + explicitly cast the parameter, and the empty-string-to-NULL rewrite + still applies — ``CAST(NULL AS INT)`` is a harmless typed null, while + leaving ``$1`` as ``''`` would still trip DuckDB's INT conversion. + Asymmetric with ``_try_extract_column``, which keeps CAST-wrapped + columns out of scope (their user-supplied cast is the documented + boundary at which we stop classifying).""" + while isinstance(node, (exp.Paren, exp.Cast)): + node = node.this + if not isinstance(node, exp.Parameter): + return None + try: + return int(node.name) + except (ValueError, AttributeError, TypeError): + return None + + +def _pair_column_and_param(left, right) -> Iterable[tuple[exp.Column, int]]: + """Two operands where one is a bare Column and the other is a $N + Parameter -> yield (Column, $N). Returns nothing if both sides are + the same kind or if neither is a Column / Parameter.""" + col_l, col_r = _try_extract_column(left), _try_extract_column(right) + param_l, param_r = _try_extract_param_index(left), _try_extract_param_index(right) + col = col_l if col_l is not None else col_r + param_idx = param_l if param_l is not None else param_r + # Need exactly one column and exactly one parameter — reject if both sides + # match the same kind (col = col, param = param) or neither is matchable. + if col_l is not None and col_r is not None: + return + if param_l is not None and param_r is not None: + return + if col is not None and param_idx is not None: + yield col, param_idx + + +def _resolved_table_sources(sources) -> dict[str, tuple[str, str]]: + """Given ``Scope.sources``, return ``{alias_lower: (schema_lower, table_lower)}``. + + Sources whose value is another ``Scope`` (CTE / derived subquery) + are skipped — they lose physical-column lineage so the classifier + can't resolve their column types. + + Bare table names (no schema qualifier) are inferred to ``pg_catalog`` + when the name matches a known relation, otherwise to ``PUBLIC_SCHEMA``. + information_schema requires an explicit qualifier — bare names like + ``columns`` never resolve there (Codex round 1, finding #4). + """ + result: dict[str, tuple[str, str]] = {} + for alias, src in sources.items(): + if not isinstance(src, exp.Table): + continue + tbl_name = src.name.lower() + db_part = src.args.get("db") + schema: str | None = None + if db_part is not None: + schema_raw = db_part.name if hasattr(db_part, "name") else str(db_part) + schema = schema_raw.lower() + if schema is None: + schema = ( + "pg_catalog" if tbl_name in _PG_CATALOG_RELATIONS else PUBLIC_SCHEMA + ) + result[alias.lower()] = (schema, tbl_name) + return result + + +def _resolve_column_table( + *, col: exp.Column, + scope_sources: dict[str, tuple[str, str]], + column_type_index: ColumnTypeIndex, +) -> tuple[str, str] | None: + """Resolve a Column node to ``(schema, table_name)`` via its owning + scope's sources. Returns ``None`` if unresolvable (no scope match, + ambiguous bare name, or table not in scope).""" + table_q = (col.table or "").lower() + db_q = (col.db or "").lower() + if table_q: + if db_q: + return (db_q, table_q) + if table_q in scope_sources: + return scope_sources[table_q] + return None + name_lower = col.name.lower() + matches: list[tuple[str, str]] = [] + for _alias, (schema, tbl) in scope_sources.items(): + if (schema, tbl, name_lower) in column_type_index: + matches.append((schema, tbl)) + if len(matches) == 1: + return matches[0] + return None + + +def _resolve_param_oids(stmt: _PreparedStatement) -> list[int]: """The parameter OIDs to report in ParameterDescription. asyncpg leaves ``Parse`` parameter OIDs empty and relies on the server to @@ -846,7 +1655,7 @@ def _is_tx_end(stmt: exp.Expression) -> bool: return False -def _command_tag(command_tag: Optional[str]) -> str: +def _command_tag(command_tag: str | None) -> str: if command_tag in ("BEGIN", "START TRANSACTION"): return "BEGIN" if command_tag is None: @@ -854,6 +1663,32 @@ def _command_tag(command_tag: Optional[str]) -> str: return command_tag +def _engine_error_fields(exc: BaseException) -> tuple[str, str]: + """Extract a Postgres SQLSTATE + terse message from an engine execution error. + + SQLAlchemy wraps the driver error (asyncpg/psycopg) in a ``DBAPIError`` + whose ``.orig`` carries the real ``sqlstate`` and the bare server message. + Surfacing those lets a client see e.g. ``permission denied for table Item`` + (42501) instead of the full ``(sqlalchemy...) : ... [SQL: ...]`` + Python repr. Walks ``.orig`` / ``__cause__`` / ``__context__`` and returns + the first driver error exposing a 5-char SQLSTATE; falls back to XX000 plus + ``str(exc)`` when none is found. + """ + seen: set[int] = set() + stack: list[BaseException | None] = [exc] + while stack: + cur = stack.pop() + if cur is None or id(cur) in seen: + continue + seen.add(id(cur)) + code = getattr(cur, "sqlstate", None) or getattr(cur, "pgcode", None) + if isinstance(code, str) and len(code) == 5: + message = getattr(cur, "message", None) or str(cur) + return code, message + stack.extend([getattr(cur, "orig", None), cur.__cause__, cur.__context__]) + return proto.SQLSTATE_INTERNAL_ERROR, str(exc) + + def _sqlstate_for(exc: TranslationError) -> str: msg = str(exc) if READ_ONLY_MESSAGE in msg: @@ -865,7 +1700,7 @@ def _sqlstate_for(exc: TranslationError) -> str: return proto.SQLSTATE_FEATURE_NOT_SUPPORTED -def _encode_value(value, oid: int, fmt: int) -> Optional[bytes]: +def _encode_value(value, oid: int, fmt: int) -> bytes | None: if fmt == proto.FORMAT_BINARY: return value_to_binary(value, oid) - return value_to_text(value) + return value_to_text(value, oid) diff --git a/slayer/pg_facade/identity.py b/slayer/pg_facade/identity.py index 051b38dc..ea54bc2d 100644 --- a/slayer/pg_facade/identity.py +++ b/slayer/pg_facade/identity.py @@ -8,7 +8,6 @@ from __future__ import annotations -from typing import List, Tuple import slayer @@ -23,7 +22,7 @@ def version_string() -> str: ) -def parameter_status_defaults() -> List[Tuple[str, str]]: +def parameter_status_defaults() -> list[tuple[str, str]]: """The ParameterStatus burst sent after auth, before the first ReadyForQuery. UTC + UTF8 keep timestamp / encoding handling unambiguous.""" return [ diff --git a/slayer/pg_facade/pg_catalog.py b/slayer/pg_facade/pg_catalog.py deleted file mode 100644 index 4ce4722d..00000000 --- a/slayer/pg_facade/pg_catalog.py +++ /dev/null @@ -1,336 +0,0 @@ -"""pg_catalog.* responses for the Postgres facade (DEV-1486). - -Phase 1 implements the minimum-viable set BI tools query while enumerating -schemas / tables / columns / types: ``pg_namespace``, ``pg_class``, -``pg_attribute``, ``pg_type``, ``pg_proc``, ``pg_settings``. Both -``pg_catalog.
`` and the bare ``
`` (search_path) forms resolve. - -Phase 1 ignores ``WHERE`` (returns every row; the client filters in memory) -and ignores the SELECT projection (returns every column), mirroring the -shared INFORMATION_SCHEMA approach. Only the six built-in type OIDs are ever -emitted, so a client never has to introspect an unknown type. - -OIDs are deterministic — ``zlib.crc32`` over a namespaced ``.`` / -``..`` string — so they're stable across server restarts -(unlike the per-process-salted builtin ``hash``). A collision check runs at -build time. -""" - -from __future__ import annotations - -import zlib -from typing import Dict, Optional, Tuple - -import sqlglot.expressions as exp - -from slayer.core.enums import DataType -from slayer.facade.catalog import FacadeCatalog, FacadeTable -from slayer.facade.rows import FacadeColumn, RowBatch -from slayer.pg_facade.identity import PG_SERVER_VERSION -from slayer.pg_facade.types import datatype_to_oid -from slayer.pg_facade.protocol import ( - OID_BOOL, - OID_DATE, - OID_FLOAT8, - OID_INT8, - OID_TEXT, - OID_TIMESTAMP, -) - -# The fixed OID for the single exposed namespace, matching Postgres's -# well-known `public` schema OID (2200) closely enough for BI introspection. -PUBLIC_NAMESPACE_OID = 2200 -PG_CATALOG_NAMESPACE_OID = 11 -DEFAULT_OWNER_OID = 10 - -SUPPORTED_PG_CATALOG_TABLES = frozenset({ - "pg_namespace", - "pg_class", - "pg_attribute", - "pg_type", - "pg_proc", - "pg_settings", -}) - -# Per-OID metadata for pg_type / pg_attribute: (typname, typlen, typcategory). -_TYPE_META: Dict[int, Tuple[str, int, str]] = { - OID_BOOL: ("bool", 1, "B"), - OID_INT8: ("int8", 8, "N"), - OID_TEXT: ("text", -1, "S"), - OID_FLOAT8: ("float8", 8, "N"), - OID_DATE: ("date", 4, "D"), - OID_TIMESTAMP: ("timestamp", 8, "D"), -} - - -def stable_oid(*parts: str) -> int: - """Deterministic positive 31-bit OID from a namespaced identifier.""" - key = ".".join(parts).encode("utf-8") - return zlib.crc32(key) & 0x7FFFFFFF - - -def _pg_catalog_table(node: exp.Expression) -> Optional[str]: - """If ``node`` is ``SELECT ... FROM [pg_catalog.]``, return the - lowercased table name; else ``None``.""" - if not isinstance(node, exp.Select): - return None - from_clause = node.args.get("from_") - if from_clause is None: - return None - table = from_clause.this - if not isinstance(table, exp.Table): - return None - schema_part = table.args.get("db") - if schema_part is not None: - schema_name = ( - str(schema_part.this) if hasattr(schema_part, "this") else str(schema_part) - ) - if schema_name.lower() != "pg_catalog": - return None - name = str(table.this.this) if hasattr(table.this, "this") else str(table.this) - name_lower = name.lower() - if name_lower not in SUPPORTED_PG_CATALOG_TABLES: - return None - return name_lower - - -def match_pg_catalog(parsed: exp.Expression, catalog: FacadeCatalog) -> Optional[RowBatch]: - """Return the canned ``pg_catalog.
`` answer or ``None``. - - Signature matches the translator's ``CatalogMatcher`` protocol so it can be - injected via ``translate(..., catalog_matchers=[match_pg_catalog])``. - """ - table_name = _pg_catalog_table(parsed) - if table_name is None: - return None - builders = { - "pg_namespace": _serve_pg_namespace, - "pg_class": _serve_pg_class, - "pg_attribute": _serve_pg_attribute, - "pg_type": _serve_pg_type, - "pg_proc": _serve_pg_proc, - "pg_settings": _serve_pg_settings, - } - return builders[table_name](catalog) - - -def _all_tables(catalog: FacadeCatalog): - """Yield ``(datasource, FacadeTable)`` for every table in the catalog.""" - for sch in catalog.schemas: - for tbl in sch.tables: - yield sch.name, tbl - - -def _table_oid(datasource: str, table: FacadeTable) -> int: - return stable_oid(datasource, table.name) - - -def _column_specs(table: FacadeTable): - """Yield ``(name, DataType)`` for every projectable column (dims + metrics).""" - for d in table.dimensions: - yield d.name, d.data_type - for m in table.metrics: - yield m.name, m.data_type if m.data_type is not None else DataType.TEXT - - -def _serve_pg_namespace(catalog: FacadeCatalog) -> RowBatch: # noqa: ARG001 - columns = [ - FacadeColumn(name="oid", type=DataType.INT), - FacadeColumn(name="nspname", type=DataType.TEXT), - FacadeColumn(name="nspowner", type=DataType.INT), - FacadeColumn(name="nspacl", type=DataType.TEXT), - ] - rows = [ - { - "oid": PUBLIC_NAMESPACE_OID, - "nspname": "public", - "nspowner": DEFAULT_OWNER_OID, - "nspacl": None, - }, - { - # Builtin types live here (pg_type.typnamespace == 11); without - # this row a join from pg_type back to pg_namespace dangles. - "oid": PG_CATALOG_NAMESPACE_OID, - "nspname": "pg_catalog", - "nspowner": DEFAULT_OWNER_OID, - "nspacl": None, - }, - ] - return RowBatch(columns=columns, rows=rows) - - -def _serve_pg_class(catalog: FacadeCatalog) -> RowBatch: - columns = [ - FacadeColumn(name="oid", type=DataType.INT), - FacadeColumn(name="relname", type=DataType.TEXT), - FacadeColumn(name="relnamespace", type=DataType.INT), - FacadeColumn(name="reltype", type=DataType.INT), - FacadeColumn(name="relowner", type=DataType.INT), - FacadeColumn(name="relkind", type=DataType.TEXT), - FacadeColumn(name="relnatts", type=DataType.INT), - FacadeColumn(name="relhasindex", type=DataType.BOOLEAN), - FacadeColumn(name="relpersistence", type=DataType.TEXT), - FacadeColumn(name="relpages", type=DataType.INT), - FacadeColumn(name="reltuples", type=DataType.DOUBLE), - FacadeColumn(name="relhasrules", type=DataType.BOOLEAN), - FacadeColumn(name="relhastriggers", type=DataType.BOOLEAN), - FacadeColumn(name="relrowsecurity", type=DataType.BOOLEAN), - FacadeColumn(name="relispartition", type=DataType.BOOLEAN), - ] - rows = [] - seen_oids: Dict[int, str] = {} - for ds, tbl in _all_tables(catalog): - oid = _table_oid(ds, tbl) - _check_collision(seen_oids, oid, f"{ds}.{tbl.name}") - natts = sum(1 for _ in _column_specs(tbl)) - rows.append({ - "oid": oid, - "relname": tbl.name, - "relnamespace": PUBLIC_NAMESPACE_OID, - "reltype": 0, - "relowner": DEFAULT_OWNER_OID, - "relkind": "r", - "relnatts": natts, - "relhasindex": False, - "relpersistence": "p", - "relpages": 0, - "reltuples": -1.0, - "relhasrules": False, - "relhastriggers": False, - "relrowsecurity": False, - "relispartition": False, - }) - return RowBatch(columns=columns, rows=rows) - - -def _serve_pg_attribute(catalog: FacadeCatalog) -> RowBatch: - columns = [ - FacadeColumn(name="attrelid", type=DataType.INT), - FacadeColumn(name="attname", type=DataType.TEXT), - FacadeColumn(name="atttypid", type=DataType.INT), - FacadeColumn(name="attnum", type=DataType.INT), - FacadeColumn(name="attlen", type=DataType.INT), - FacadeColumn(name="atttypmod", type=DataType.INT), - FacadeColumn(name="attnotnull", type=DataType.BOOLEAN), - FacadeColumn(name="atthasdef", type=DataType.BOOLEAN), - FacadeColumn(name="attisdropped", type=DataType.BOOLEAN), - FacadeColumn(name="attidentity", type=DataType.TEXT), - FacadeColumn(name="attgenerated", type=DataType.TEXT), - ] - rows = [] - for ds, tbl in _all_tables(catalog): - attrelid = _table_oid(ds, tbl) - attnum = 1 - for name, data_type in _column_specs(tbl): - oid = datatype_to_oid(data_type) - rows.append({ - "attrelid": attrelid, - "attname": name, - "atttypid": oid, - "attnum": attnum, - "attlen": _TYPE_META[oid][1], - "atttypmod": -1, - "attnotnull": False, - "atthasdef": False, - "attisdropped": False, - "attidentity": "", - "attgenerated": "", - }) - attnum += 1 - return RowBatch(columns=columns, rows=rows) - - -def _serve_pg_type(catalog: FacadeCatalog) -> RowBatch: # noqa: ARG001 - columns = [ - FacadeColumn(name="oid", type=DataType.INT), - FacadeColumn(name="typname", type=DataType.TEXT), - FacadeColumn(name="typnamespace", type=DataType.INT), - FacadeColumn(name="typlen", type=DataType.INT), - FacadeColumn(name="typtype", type=DataType.TEXT), - FacadeColumn(name="typcategory", type=DataType.TEXT), - FacadeColumn(name="typisdefined", type=DataType.BOOLEAN), - FacadeColumn(name="typdelim", type=DataType.TEXT), - FacadeColumn(name="typrelid", type=DataType.INT), - FacadeColumn(name="typelem", type=DataType.INT), - FacadeColumn(name="typarray", type=DataType.INT), - ] - rows = [] - for oid, (typname, typlen, typcategory) in _TYPE_META.items(): - rows.append({ - "oid": oid, - "typname": typname, - "typnamespace": PG_CATALOG_NAMESPACE_OID, - "typlen": typlen, - "typtype": "b", - "typcategory": typcategory, - "typisdefined": True, - "typdelim": ",", - "typrelid": 0, - "typelem": 0, - "typarray": 0, - }) - return RowBatch(columns=columns, rows=rows) - - -def _serve_pg_proc(catalog: FacadeCatalog) -> RowBatch: # noqa: ARG001 - columns = [ - FacadeColumn(name="oid", type=DataType.INT), - FacadeColumn(name="proname", type=DataType.TEXT), - FacadeColumn(name="pronamespace", type=DataType.INT), - FacadeColumn(name="prorettype", type=DataType.INT), - ] - return RowBatch(columns=columns, rows=[]) - - -def _serve_pg_settings(catalog: FacadeCatalog) -> RowBatch: # noqa: ARG001 - columns = [ - FacadeColumn(name="name", type=DataType.TEXT), - FacadeColumn(name="setting", type=DataType.TEXT), - FacadeColumn(name="category", type=DataType.TEXT), - FacadeColumn(name="unit", type=DataType.TEXT), - FacadeColumn(name="source", type=DataType.TEXT), - FacadeColumn(name="vartype", type=DataType.TEXT), - FacadeColumn(name="context", type=DataType.TEXT), - FacadeColumn(name="min_val", type=DataType.TEXT), - FacadeColumn(name="max_val", type=DataType.TEXT), - ] - settings = [ - ("server_version", PG_SERVER_VERSION), - ("client_encoding", "UTF8"), - ("server_encoding", "UTF8"), - ("DateStyle", "ISO, MDY"), - ("IntervalStyle", "postgres"), - ("TimeZone", "UTC"), - ("standard_conforming_strings", "on"), - ("integer_datetimes", "on"), - ("max_index_keys", "32"), - ("block_size", "8192"), - ] - rows = [{ - "name": name, - "setting": value, - "category": "Preset Options", - "unit": None, - "source": "default", - "vartype": "string", - "context": "user", - "min_val": None, - "max_val": None, - } for name, value in settings] - return RowBatch(columns=columns, rows=rows) - - -def _check_collision(seen: Dict[int, str], oid: int, key: str) -> None: - prior = seen.get(oid) - if prior is not None and prior != key: - raise ValueError( - f"pg_catalog OID collision: {key!r} and {prior!r} both hash to {oid}" - ) - seen[oid] = key - - -__all__ = [ - "SUPPORTED_PG_CATALOG_TABLES", - "match_pg_catalog", - "stable_oid", -] diff --git a/slayer/pg_facade/probes.py b/slayer/pg_facade/probes.py index ffbe79fc..de8e4dce 100644 --- a/slayer/pg_facade/probes.py +++ b/slayer/pg_facade/probes.py @@ -1,47 +1,75 @@ -"""Postgres-facade connection probes (DEV-1486). +"""Postgres-facade connection probes (DEV-1486, DEV-1569). Datasource-aware canned answers for the connect-time pings Postgres clients and BI drivers issue: ``version()``, ``current_database()``, -``current_schema()``, ``SHOW ``, and the ``current_setting('jit')`` / -``set_config('jit', …)`` JIT probes asyncpg runs on server_version ≥ 11. +``current_schema()``, ``SHOW ``, and ``current_setting('')`` / +``set_config('', '', …)`` GUC-style probes. These differ from the Flight facade's generic probes (datasource-specific ``current_database()``, PostgreSQL-shaped ``version()``), so the Postgres facade injects ``match_pg_probe`` as the translator's ``probe_matcher`` and falls back to the shared ``match_probe`` for the truly generic ones (``SELECT 1`` / ``SELECT NULL WHERE 1=0``). + +DEV-1569: ``SHOW`` / ``current_setting`` consult a per-connection +``session_settings`` dict (passed in by the connection) so that +``SET application_name = 'foo'`` followed by ``SHOW application_name`` +round-trips correctly. ``set_config(...)`` is purely read-only inside +this module — it returns the requested value via the matched ``RowBatch`` +but never mutates the dict in place. The connection applies the mutation +on Execute (but not Describe) via ``match_pg_probe_with_mutation``, which +returns a ``ProbeMatcherOutcome`` carrying both the row batch and an +optional ``SetSettingOp`` hint. """ from __future__ import annotations -from typing import Optional import sqlglot.expressions as exp from slayer.core.enums import DataType from slayer.facade.rows import FacadeColumn, RowBatch +from slayer.facade.translator import ProbeMatcherOutcome, SetSettingOp from slayer.pg_facade.identity import PG_SERVER_VERSION -# Canned values for common SHOW settings. Unknown settings return "". -_SHOW_DEFAULTS = { +# Default per-connection session settings, seeded into every fresh +# ``PgConnection._session_settings``. Lowercase keys (Postgres GUC names +# are case-insensitive). Values align with what the startup +# ``ParameterStatus`` burst (identity.py) advertises for the same setting, +# so a client SHOW immediately after connect agrees with the burst. +SESSION_SETTING_SEED: dict[str, str] = { "search_path": '"$user", public', "transaction_isolation": "read committed", "standard_conforming_strings": "on", "server_version": PG_SERVER_VERSION, "client_encoding": "UTF8", + "server_encoding": "UTF8", + "intervalstyle": "postgres", "datestyle": "ISO, MDY", "timezone": "UTC", + "session_authorization": "slayer", + "application_name": "", + "jit": "off", +} + +# Multi-word SHOW spellings → the canonical setting they report. pgjdbc's +# Connection.getTransactionIsolation() issues `SHOW TRANSACTION ISOLATION +# LEVEL` on every pooled connection (c3p0 caches it at pool-init). +SHOW_ALIASES = { + "transaction isolation level": "transaction_isolation", + "time zone": "timezone", + "session authorization": "session_authorization", } -def _single(name: str, value: Optional[str], dtype: DataType = DataType.TEXT) -> RowBatch: +def _single(name: str, value: str | None, dtype: DataType = DataType.TEXT) -> RowBatch: return RowBatch( columns=[FacadeColumn(name=name, type=dtype)], rows=[{name: value}], ) -def _single_projection(parsed: exp.Expression) -> Optional[exp.Expression]: +def _single_projection(parsed: exp.Expression) -> exp.Expression | None: if not isinstance(parsed, exp.Select): return None exprs = parsed.args.get("expressions") or [] @@ -53,7 +81,7 @@ def _single_projection(parsed: exp.Expression) -> Optional[exp.Expression]: return body -def _show_setting_name(parsed: exp.Expression) -> Optional[str]: +def _show_setting_name(parsed: exp.Expression) -> str | None: if not isinstance(parsed, exp.Command): return None if str(parsed.this).upper() != "SHOW": @@ -65,7 +93,7 @@ def _show_setting_name(parsed: exp.Expression) -> Optional[str]: return name.strip().strip("'\"") -def _anonymous_name(node: exp.Expression) -> Optional[str]: +def _anonymous_name(node: exp.Expression) -> str | None: if isinstance(node, exp.Anonymous): return str(node.this).lower() return None @@ -73,15 +101,28 @@ def _anonymous_name(node: exp.Expression) -> Optional[str]: def match_pg_probe( parsed: exp.Expression, *, datasource: str, version_str: str, -) -> Optional[RowBatch]: + session_settings: dict[str, str] | None = None, +) -> RowBatch | None: """Return a datasource-aware canned ``RowBatch`` for a Postgres probe, - else ``None`` (caller falls back to the shared probe matcher).""" + else ``None`` (caller falls back to the shared probe matcher). + + DEV-1569: ``SHOW`` / ``current_setting`` consult ``session_settings`` + when provided (else fall back to the shared ``SESSION_SETTING_SEED``). + ``set_config`` returns the requested value via the row batch but does + NOT in-place mutate ``session_settings`` — keeping this function pure + so the connection can call it during the Describe phase without + side-effects. The connection applies the mutation on Execute via + ``match_pg_probe_with_mutation``. + """ + settings = session_settings if session_settings is not None else SESSION_SETTING_SEED # SHOW — `server_version` reports the bare "14.0" (matching # ParameterStatus / pg_settings), NOT the full version() string. setting = _show_setting_name(parsed) if setting is not None: - value = _SHOW_DEFAULTS.get(setting.lower(), "") - return _single(setting, value) + key = setting.lower() + key = SHOW_ALIASES.get(key, key) + value = settings.get(key, "") + return _single(key, value) body = _single_projection(parsed) if body is None: @@ -91,30 +132,152 @@ def match_pg_probe( return _single("version", version_str) if isinstance(body, exp.CurrentDatabase) or _anonymous_name(body) == "current_database": return _single("current_database", datasource) + # pgjdbc's PgConnection.getCatalog() issues the niladic `SELECT + # current_catalog`; Metabase's c3p0 pool calls it on every new connection. + if isinstance(body, exp.CurrentCatalog): + return _single("current_catalog", datasource) if isinstance(body, exp.CurrentSchema) or _anonymous_name(body) == "current_schema": return _single("current_schema", "public") + # The facade does not track per-connection login identity; a constant + # satisfies driver probes (the username is ignored at auth anyway). + if isinstance(body, exp.SessionUser): + return _single("session_user", "slayer") + if isinstance(body, exp.CurrentUser): + return _single("current_user", "slayer") name = _anonymous_name(body) if name == "current_setting": - return _single("current_setting", _setting_arg_value(body)) + return _single("current_setting", _setting_value(body, settings)) if name == "set_config": return _single("set_config", _set_config_value(body)) return None -def _first_literal(node: exp.Anonymous, index: int) -> Optional[str]: +def match_pg_probe_with_mutation( + parsed: exp.Expression, *, datasource: str, version_str: str, + session_settings: dict[str, str] | None = None, +) -> ProbeMatcherOutcome | None: + """Mutation-aware variant of :func:`match_pg_probe`. Returns a + :class:`ProbeMatcherOutcome` carrying both the row batch and (for + ``set_config(name, value, ...)`` matches) a :class:`SetSettingOp` + hint the connection applies to its per-connection session-settings + map on Execute. + + Important: this function does NOT in-place mutate ``session_settings``; + the connection applies the mutation only after seeing the + :class:`ProbeResult` in the Execute path (Describe-phase calls into + the translator must remain pure — see DEV-1569 / Codex round 1 + F1+F2 in connection.py). + """ + batch = match_pg_probe( + parsed, datasource=datasource, version_str=version_str, + session_settings=session_settings, + ) + if batch is None: + return None + mutation = _extract_set_config_mutation(parsed) + return ProbeMatcherOutcome(batch=batch, settings_mutation=mutation) + + +def _extract_set_config_mutation(parsed: exp.Expression) -> SetSettingOp | None: + """Inspect a parsed AST root for ``SELECT set_config('', '', + )`` and return a ``SetSettingOp`` carrying the (lowercased + name, raw value) pair; return ``None`` otherwise. + + DEV-1569 / Codex F3: the value may arrive wrapped in an ``exp.Cast`` + (asyncpg / pgjdbc emit ``set_config('app', $1::text, false)`` and the + bound substitution leaves ``'value'::text``); ``_first_literal`` + peers through one level of CAST so the mutation still surfaces. + + DEV-1569 / CodeRabbit thread: ``is_local=true`` is out of scope for + DEV-1569 (we don't model transaction-bound restoration). When the + third argument is explicitly ``true``, return ``None`` so the + connection still emits the row but doesn't persist the value. + """ + body = _single_projection(parsed) + if body is None: + return None + if _anonymous_name(body) != "set_config": + return None + name_lit = _first_literal(body, 0) + value_lit = _first_literal(body, 1) + if name_lit is None or value_lit is None: + return None + if not _set_config_is_session_scope(body): + return None + return SetSettingOp(name=name_lit.lower(), value=value_lit) + + +def _set_config_is_session_scope(node: exp.Anonymous) -> bool: + """Check ``set_config`` 's third argument (``is_local``). + + ``False`` (session scope) is permitted; explicit ``true`` (local scope) + is blocked since DEV-1569 doesn't model transaction-bound restoration. + A missing third argument or an unknown shape is treated as session + scope (per real-Postgres default). + + DEV-1569 / Codex round 2 F1: extended-protocol substitution may + surface the boolean wrapped in an ``exp.Cast`` (``FALSE::boolean`` / + ``CAST(FALSE AS BOOLEAN)``); peer through one Cast level. + """ args = node.args.get("expressions") or [] - if index < len(args) and isinstance(args[index], exp.Literal): # NOSONAR(S6466) — guarded by index < len(args) - return str(args[index].this) + if len(args) < 3: + return True + is_local = args[2] + if isinstance(is_local, exp.Cast): + is_local = is_local.this + if isinstance(is_local, exp.Boolean): + return is_local.this is False + if isinstance(is_local, exp.Literal): + return not _is_truthy_boolean_literal(str(is_local.this)) + # Non-literal / non-boolean third arg: be conservative — skip mutation. + return False + + +def _is_truthy_boolean_literal(value: str) -> bool: + """Match Postgres boolean input rules: accept any unique case-insensitive + prefix of `true` or `yes`, plus the exact strings `on` and `1`. + + Examples that return True: `t`, `tr`, `tru`, `true`, `y`, `ye`, `yes`, + `on`, `1` (case-insensitive). Empty string and any other input return + False. Codex round 5/6. + """ + lowered = value.lower() + if not lowered: + return False + return ( + lowered in ("on", "1") + or "true".startswith(lowered) + or "yes".startswith(lowered) + ) + + +def _first_literal(node: exp.Anonymous, index: int) -> str | None: + """Return the string value of the ``index``-th argument of ``node`` if it + is a string literal (or a CAST around a string literal). Returns + ``None`` otherwise. + + DEV-1569 / Codex F3: drivers emit cast forms like ``'foo'::text`` and + ``cast('foo' AS TEXT)`` for set_config arguments; sqlglot parses both + as ``exp.Cast(this=Literal('foo'), to=DataType(TEXT))``. Peer through + one level of CAST so the underlying literal is reachable. + """ + args = node.args.get("expressions") or [] + if index >= len(args): + return None + arg = args[index] # NOSONAR(S6466) — guarded by the index check on the line above + if isinstance(arg, exp.Cast): + arg = arg.this + if isinstance(arg, exp.Literal): + return str(arg.this) return None -def _setting_arg_value(node: exp.Anonymous) -> str: - """``current_setting('jit')`` → ``'off'``; otherwise empty string.""" +def _setting_value(node: exp.Anonymous, settings: dict[str, str]) -> str: + """``current_setting('')`` → ``settings[name]`` (lowercased lookup); + unknown settings return the empty string.""" setting = (_first_literal(node, 0) or "").lower() - if setting == "jit": - return "off" - return "" + return settings.get(setting, "") def _set_config_value(node: exp.Anonymous) -> str: diff --git a/slayer/pg_facade/protocol.py b/slayer/pg_facade/protocol.py index 68cd091b..309639a1 100644 --- a/slayer/pg_facade/protocol.py +++ b/slayer/pg_facade/protocol.py @@ -16,7 +16,6 @@ from __future__ import annotations import struct -from typing import Dict, List, Optional, Tuple from pydantic import BaseModel @@ -110,7 +109,7 @@ class FieldDescription(BaseModel): column_attr: int = 0 -def encode_row_description(fields: List[FieldDescription]) -> bytes: +def encode_row_description(fields: list[FieldDescription]) -> bytes: payload = struct.pack(">h", len(fields)) for f in fields: payload += _cstr(f.name) @@ -126,7 +125,7 @@ def encode_row_description(fields: List[FieldDescription]) -> bytes: return _msg(b"T", payload) -def encode_data_row(values: List[Optional[bytes]]) -> bytes: +def encode_data_row(values: list[bytes | None]) -> bytes: payload = struct.pack(">h", len(values)) for v in values: if v is None: @@ -164,7 +163,7 @@ def encode_portal_suspended() -> bytes: return _msg(b"s", b"") -def encode_parameter_description(oids: List[int]) -> bytes: +def encode_parameter_description(oids: list[int]) -> bytes: payload = struct.pack(">h", len(oids)) for oid in oids: payload += struct.pack(">i", oid) @@ -195,21 +194,21 @@ def encode_notice_response(*, code: str, message: str, severity: str = "NOTICE") class StartupMessage(BaseModel): protocol_version: int - parameters: Dict[str, str] + parameters: dict[str, str] class ParseMessage(BaseModel): name: str query: str - parameter_oids: List[int] + parameter_oids: list[int] class BindMessage(BaseModel): portal: str statement: str - parameter_format_codes: List[int] - parameter_values: List[Optional[bytes]] - result_format_codes: List[int] + parameter_format_codes: list[int] + parameter_values: list[bytes | None] + result_format_codes: list[int] class DescribeMessage(BaseModel): @@ -275,7 +274,7 @@ def decode_startup(body: bytes) -> StartupMessage: null terminator.""" r = _Reader(buf=body) version = r.int32() - params: Dict[str, str] = {} + params: dict[str, str] = {} while r.pos < len(body): if body[r.pos:r.pos + 1] == b"\x00": break @@ -311,7 +310,7 @@ def decode_bind(body: bytes) -> BindMessage: n_fmt = _nonneg_count(r.int16(), "parameter format code") fmt_codes = [r.int16() for _ in range(n_fmt)] n_params = _nonneg_count(r.int16(), "parameter") - values: List[Optional[bytes]] = [] + values: list[bytes | None] = [] for _ in range(n_params): length = r.int32() if length == -1: @@ -355,13 +354,13 @@ def decode_close(body: bytes) -> CloseMessage: # --- generic message splitting (used by the connection layer + tests) -------- -def split_messages(buf: bytes) -> List[Tuple[str, bytes]]: +def split_messages(buf: bytes) -> list[tuple[str, bytes]]: """Split a byte stream of tagged messages into ``[(type_char, body), …]``. Used by tests to verify encoded server messages and by any client-side helper. The body excludes the type byte and the length prefix. """ - out: List[Tuple[str, bytes]] = [] + out: list[tuple[str, bytes]] = [] pos = 0 while pos < len(buf): type_char = buf[pos:pos + 1].decode("ascii") @@ -372,14 +371,14 @@ def split_messages(buf: bytes) -> List[Tuple[str, bytes]]: return out -def validate_format_codes(codes: List[int]) -> None: +def validate_format_codes(codes: list[int]) -> None: """Reject format codes outside ``{text, binary}`` (protocol violation).""" for c in codes: if c not in (FORMAT_TEXT, FORMAT_BINARY): raise ValueError(f"invalid format code {c!r} (must be 0=text or 1=binary)") -def parse_result_format_codes(codes: List[int], column_count: int) -> List[int]: +def parse_result_format_codes(codes: list[int], column_count: int) -> list[int]: """Resolve Bind result-format codes to one entry per result column. Per the protocol: 0 codes → all text; 1 code → applies to every column; diff --git a/slayer/pg_facade/server.py b/slayer/pg_facade/server.py index d667e378..a1f04cf4 100644 --- a/slayer/pg_facade/server.py +++ b/slayer/pg_facade/server.py @@ -14,10 +14,9 @@ import os import ssl import sys -from typing import Optional -from slayer.pg_facade.auth import validate_bind_address, validate_tls_pair -from slayer.pg_facade.connection import PgConnection +from slayer.pg_facade.auth import Authenticator, validate_bind_address, validate_tls_pair +from slayer.pg_facade.connection import EngineFactory, PgConnection, StorageProvider logger = logging.getLogger(__name__) @@ -28,17 +27,53 @@ async def serve( *, host: str, port: int, - engine, - storage, - token: Optional[str] = None, - tls_ctx: Optional[ssl.SSLContext] = None, + engine=None, + storage=None, + token: str | None = None, + authenticator: Authenticator | None = None, + storage_provider: StorageProvider | None = None, + engine_factory: EngineFactory | None = None, + tls_ctx: ssl.SSLContext | None = None, + catalog_extra_relations=None, + catalog_ttl_seconds: float | None = None, ) -> None: - """Bind and serve forever. Validates the bind/token combination first.""" - validate_bind_address(host=host, token=token) + """Bind and serve forever. Validates the bind/token combination first. + + Supply either a static ``engine`` + ``storage``, or a ``storage_provider`` + that resolves a per-connection (e.g. tenant-scoped) storage from the + authenticated principal. + + ``catalog_extra_relations``: optional iterable of + ``slayer.facade.catalog_sql.CatalogRelation`` that extends or overrides + the default ``pg_catalog`` / ``information_schema`` tables. Embedders + (e.g. Storyline) use this to project real per-tenant data into + ``pg_roles`` / ``pg_database`` / add new tables. Override is by table + name; new tables are appended. + + ``catalog_ttl_seconds``: catalog freshness for each connection. ``None`` + (default) keeps the historical behavior — the catalog is built once at + connect and stays static for the connection's lifetime. A float enables + TTL-gated, on-demand refresh: an idle connection re-checks the storage + fingerprint at most once per window and rebuilds only when it changed, so + long-lived BI sessions pick up model/schema edits without reconnecting. + """ + # A custom authenticator that prompts for a password counts as auth, so the + # non-loopback-requires-a-secret rule is satisfied even without a token. + # When an authenticator is supplied, ``PgConnection`` ignores ``token`` + # entirely — so the bind guard must check the auth mechanism that will + # ACTUALLY be enforced, not a stale token that would never be consulted. + authenticated = authenticator is not None and authenticator.requires_password + effective_token = None if authenticator is not None else token + validate_bind_address(host=host, token=effective_token, authenticated=authenticated) async def handle(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> None: conn = PgConnection( - reader, writer, engine=engine, storage=storage, token=token, tls_ctx=tls_ctx, + reader, writer, engine=engine, storage=storage, + token=token, authenticator=authenticator, + storage_provider=storage_provider, engine_factory=engine_factory, + tls_ctx=tls_ctx, + catalog_extra_relations=catalog_extra_relations, + catalog_ttl_seconds=catalog_ttl_seconds, ) try: await conn.run() @@ -54,7 +89,7 @@ async def handle(reader: asyncio.StreamReader, writer: asyncio.StreamWriter) -> await server.serve_forever() -def _build_tls_context(cert: Optional[str], key: Optional[str]) -> Optional[ssl.SSLContext]: +def _build_tls_context(cert: str | None, key: str | None) -> ssl.SSLContext | None: validate_tls_pair(cert=cert, key=key) if cert is None or key is None: return None @@ -64,12 +99,12 @@ def _build_tls_context(cert: Optional[str], key: Optional[str]) -> Optional[ssl. return ctx -def _resolve_token(token_arg: Optional[str]) -> Optional[str]: +def _resolve_token(token_arg: str | None) -> str | None: """``--token`` wins over the ``$SLAYER_PG_TOKEN`` env var.""" return token_arg or os.environ.get("SLAYER_PG_TOKEN") -def _resolve_host(*, host_arg: Optional[str], demo: bool, token: Optional[str]) -> str: +def _resolve_host(*, host_arg: str | None, demo: bool, token: str | None) -> str: """Explicit --host wins; --demo without a token defaults to loopback so the no-token fallback applies; otherwise bind all interfaces.""" if host_arg is not None: @@ -92,7 +127,7 @@ def run_pg_serve(args, *, resolve_storage, prepare_demo) -> None: prepare_demo(args, storage) engine = SlayerQueryEngine(storage=storage) - token: Optional[str] = _resolve_token(args.token) + token: str | None = _resolve_token(args.token) host = _resolve_host(host_arg=args.host, demo=args.demo, token=token) try: diff --git a/slayer/pg_facade/types.py b/slayer/pg_facade/types.py index 2c496fc8..5caebb76 100644 --- a/slayer/pg_facade/types.py +++ b/slayer/pg_facade/types.py @@ -3,7 +3,8 @@ Three concerns: 1. ``DATATYPE_TO_OID`` / ``datatype_to_oid`` — SLayer ``DataType`` → Postgres - type OID. Only the six built-in OIDs are ever emitted (unknown → text), so + type OID. Only the six built-in OIDs are ever emitted (opaque / unmapped + → text), so asyncpg never has to run its ``pg_type`` introspection path. 2. ``value_to_text`` / ``value_to_binary`` — engine value → wire bytes for a ``DataRow``, in the per-column format the client requested in ``Bind``. @@ -23,7 +24,7 @@ import math import struct from decimal import Decimal -from typing import Any, Dict, Optional +from typing import Any from slayer.core.enums import DataType from slayer.pg_facade.protocol import ( @@ -35,13 +36,15 @@ OID_TIMESTAMP, ) -DATATYPE_TO_OID: Dict[DataType, int] = { +DATATYPE_TO_OID: dict[DataType, int] = { DataType.TEXT: OID_TEXT, DataType.INT: OID_INT8, DataType.DOUBLE: OID_FLOAT8, DataType.BOOLEAN: OID_BOOL, DataType.DATE: OID_DATE, DataType.TIMESTAMP: OID_TIMESTAMP, + # Opaque columns go out as text — see slayer.core.enums.DataType.is_opaque. + DataType.UNKNOWN: OID_TEXT, } # Postgres binary date/timestamp epoch. @@ -49,7 +52,7 @@ _PG_EPOCH_DATETIME = _dt.datetime(2000, 1, 1) -def datatype_to_oid(dt: Optional[DataType]) -> int: +def datatype_to_oid(dt: DataType | None) -> int: """Map a SLayer ``DataType`` to a Postgres OID; unknown / None → text.""" if dt is None: return OID_TEXT @@ -59,10 +62,39 @@ def datatype_to_oid(dt: Optional[DataType]) -> int: # --- text-format output ------------------------------------------------------ -def value_to_text(value: Any) -> Optional[bytes]: # NOSONAR(S3776) — flat per-Python-type dispatch - """Encode an engine value as Postgres text-format bytes (``None`` → SQL NULL).""" +def value_to_text( # NOSONAR(S3776) — flat per-Python-type dispatch + value: Any, oid: int = OID_TEXT, +) -> bytes | None: + """Encode an engine value as Postgres text-format bytes (``None`` → SQL NULL). + + ``oid`` lets the encoder coerce values to match the declared column type. + Notably, DuckDB returns ``DATE``-typed columns as ``datetime.datetime`` + when the value is produced by ``date_trunc(...) :: date``; without + coercion the text-format wire payload becomes ``"2024-06-01 00:00:00"`` + which pgjdbc's ``TimestampUtils.toLocalDate`` mis-parses for a column + declared with OID 1082 (DATE). + """ if value is None: return None + # OID-driven coercion runs first so a datetime in a DATE column is + # narrowed to a date before the per-Python-type dispatch below. + if oid == OID_DATE and isinstance(value, _dt.datetime): + value = value.date() + # DEV-1566: symmetric widening for CAST( AS TIMESTAMP). The + # bare `dt.date` branch below emits `YYYY-MM-DD`, which pgjdbc / + # psycopg2 mis-parse for an OID_TIMESTAMP-declared column. Widen to + # `dt.datetime` so the timestamp formatter emits `YYYY-MM-DD HH:MM:SS`. + if ( + oid == OID_TIMESTAMP + and isinstance(value, _dt.date) + and not isinstance(value, _dt.datetime) + ): + value = _dt.datetime(value.year, value.month, value.day) + # DEV-1566: CAST( AS TEXT) must emit `true`/`false` (Postgres + # text shape), not the BOOL wire shape `t`/`f` the next branch would + # produce. Check OID_TEXT BEFORE the bool branch. + if oid == OID_TEXT and isinstance(value, bool): + return b"true" if value else b"false" if isinstance(value, bool): return b"t" if value else b"f" if isinstance(value, float): @@ -95,7 +127,7 @@ def _format_timestamp(value: _dt.datetime) -> str: # --- binary-format output ---------------------------------------------------- -def value_to_binary(value: Any, oid: int) -> Optional[bytes]: +def value_to_binary(value: Any, oid: int) -> bytes | None: """Encode an engine value as Postgres binary-format bytes for ``oid`` (``None`` → SQL NULL).""" if value is None: diff --git a/slayer/search/__init__.py b/slayer/search/__init__.py index b0366980..36f9441a 100644 --- a/slayer/search/__init__.py +++ b/slayer/search/__init__.py @@ -4,20 +4,18 @@ usage. The ``SearchService`` orchestrator runs up to three retrieval channels — entity-overlap BM25 over memories, tantivy full-text over the unioned corpus, and optional dense embedding similarity gated by the -``embedding_search`` extra — and fuses the memory rankings (and entity -rankings, for channels 2 and 3) via Reciprocal Rank Fusion. +``advanced_search`` extra — and fuses all hits via Reciprocal Rank Fusion +into a single ranked ``results`` list. """ from slayer.search.service import ( - EntityHit, - MemoryHit, + SearchHit, SearchResponse, SearchService, ) __all__ = [ - "EntityHit", - "MemoryHit", + "SearchHit", "SearchResponse", "SearchService", ] diff --git a/slayer/search/cypher_naive.py b/slayer/search/cypher_naive.py new file mode 100644 index 00000000..94f2cb47 --- /dev/null +++ b/slayer/search/cypher_naive.py @@ -0,0 +1,73 @@ +"""Naive Cypher label-filter parser for the no-advanced_search fallback (DEV-1532). + +Supports only: MATCH (var:Label1:Label2:...) RETURN var.id AS id +(case-insensitive, whitespace-tolerant, no WHERE clause, no relationships). + +Used by SearchService.search() when cypher_filter is supplied but LadybugDB +is not installed. Complex Cypher raises SlayerError pointing at the +advanced_search extra. +""" + +from __future__ import annotations + +import re + +from slayer.core.errors import SlayerError + + +_LABEL_TO_KIND: dict[str, str] = { + "memory": "memory", + "datasource": "datasource", + "model": "model", + "modelcolumn": "column", + "column": "column", + "measure": "measure", + "aggregation": "aggregation", +} + +# Structured pattern: one label word optionally followed by (: word)* pairs. +# \s* is only used as a delimiter between fixed tokens (never inside a +# quantified character class that can also match \s), which avoids +# polynomial backtracking on non-matching inputs (Sonar S5852). +_NAIVE_PATTERN = re.compile( + pattern=r"^\s*MATCH\s*\(\s*(?P\w+)\s*:\s*(?P\w+(?:\s*:\s*\w+)*)\s*\)\s*RETURN\s+(?P=var)\.id\s+AS\s+id\s*$", # NOSONAR(S5843) — structural complexity is load-bearing; each token maps to a distinct syntactic Cypher element + flags=re.IGNORECASE, +) + +_AS_ID_RE = re.compile(pattern=r"\bAS\s+id\b", flags=re.IGNORECASE) + + +def parse_naive_label_filter(cypher: str) -> set[str]: + """Parse a simple MATCH (n:Label1:Label2) RETURN n.id AS id expression + and return the set of kind strings to filter search results on. + + Raises SlayerError: + - Missing 'AS id' alias → generic validation message. + - Pattern doesn't match (WHERE, relationship, etc.) → + message mentions advanced_search requirement. + - Unknown label → message says "unknown". + """ + if not _AS_ID_RE.search(cypher): + raise SlayerError( + "cypher_filter must return exactly one column aliased 'id' " + "(e.g. 'RETURN n.id AS id')." + ) + match = _NAIVE_PATTERN.match(cypher) + if not match: + raise SlayerError( + "cypher_filter expression is too complex for the naive fallback; " + "install the advanced_search extra: " + "pip install motley-slayer[advanced_search]" + ) + labels_str = match.group("labels") + labels = [lb.strip() for lb in re.split(r"\s*:\s*", labels_str) if lb.strip()] + kinds: set[str] = set() + for label in labels: + kind = _LABEL_TO_KIND.get(label.lower()) + if kind is None: + raise SlayerError( + f"unknown entity type {label!r} in cypher_filter; " + f"known types: {sorted(_LABEL_TO_KIND)!r}." + ) + kinds.add(kind) + return kinds diff --git a/slayer/search/graph.py b/slayer/search/graph.py new file mode 100644 index 00000000..61880c55 --- /dev/null +++ b/slayer/search/graph.py @@ -0,0 +1,597 @@ +"""Graph-backed Cypher pre-filter for search (DEV-1464). + +Builds an ephemeral in-memory LadybugDB property graph from a StorageBackend +and executes openCypher queries to return a frozenset of canonical IDs that +pre-filter the three search channels. + +Requires the ``advanced_search`` extra (``pip install motley-slayer[advanced_search]``), +which pulls in ``ladybug`` — the active successor to KuzuDB (same codebase, +new name after the original KuzuDB repo was archived post-acquisition). +If LadybugDB is not installed, ``is_available()`` returns ``False`` +and no graph code is reachable. + +Graph schema +------------ +Node tables (one per entity kind): + Memory id STRING (canonical ``memory:`` form), learning STRING + Datasource id STRING, name STRING + Model id STRING, name STRING, description STRING + ModelColumn id STRING, name STRING, data_type STRING, description STRING + Measure id STRING, name STRING, description STRING + Aggregation id STRING, name STRING + +Note: the node table for schema columns is named ``ModelColumn`` (not ``Column``) +because ``Column`` is a reserved keyword in LadybugDB ≥ 0.15. + +Relationship tables: + MENTIONS Memory → {Datasource, Model, ModelColumn, Measure, Aggregation, Memory} + CONTAINS Datasource → Model, Model → {ModelColumn, Measure, Aggregation} + JOINS Model → Model + +All queries must be read-only ``MATCH … RETURN … AS id`` statements. +The ``MATCH (n:A:B)`` multi-label pattern returns nodes from BOTH table A +and table B (union semantics — LadybugDB behaviour). +""" + +from __future__ import annotations + +import asyncio +import os +import re +from functools import lru_cache +from typing import Any + +from pydantic import BaseModel + +from slayer.memories.models import MEMORY_CANONICAL_PREFIX as _MEMORY_PREFIX +from slayer.storage.base import StorageBackend + + +# --------------------------------------------------------------------------- +# Availability +# --------------------------------------------------------------------------- + + +@lru_cache(maxsize=1) +def is_available() -> bool: + """Return True when LadybugDB is importable.""" + try: + import ladybug # noqa: F401 + return True + except ImportError: + return False + + +def _import_graph_module() -> Any: + """Import LadybugDB.""" + try: + import ladybug + return ladybug + except ImportError: + raise ImportError( + "LadybugDB not installed; " + "install with: pip install motley-slayer[advanced_search]" + ) + + +# --------------------------------------------------------------------------- +# Cypher validation +# --------------------------------------------------------------------------- + + +_MUTATION_RE = re.compile( + r"\b(CREATE|MERGE|DELETE|SET|REMOVE|DROP|CALL)\b", re.IGNORECASE +) +_AS_ID_RE = re.compile(r"\bAS\s+id\b", re.IGNORECASE) +# DEV-1464: the validator is fundamentally allowlist-shaped — start with +# MATCH or reject. A denylist over Kuzu's evolving keyword surface (LOAD, +# IMPORT, EXPORT, INSTALL, UNWIND-from-the-front, …) is too fragile to +# defend the search endpoint, since non-mutating but still-dangerous +# clauses like ``LOAD FROM '/etc/passwd' RETURN 1 AS id`` would otherwise +# pass the mutation-keyword check and reach ``conn.execute()``. +_STARTS_WITH_MATCH_RE = re.compile(r"^\s*MATCH\b", re.IGNORECASE) +# Matches single- and double-quoted string literals (with backslash-escape +# support) so we can strip them before scanning for mutation keywords and +# avoid false-positive rejections on property values like 'call me'. +_QUOTED_STRING_RE = re.compile( + r"'(?:[^'\\]|\\.)*'|\"(?:[^\"\\]|\\.)*\"", + re.DOTALL, +) + + +def _validate_cypher(cypher: str) -> None: + """Validate that ``cypher`` is a safe read-only ``MATCH … RETURN … AS id``. + + Raises ``ValueError`` on: + * queries that don't start with ``MATCH`` (read-only allowlist) + * semicolons (multiple statements) + * mutation keywords: CREATE, MERGE, DELETE, SET, REMOVE, DROP, CALL + * missing ``AS id`` alias in the RETURN clause + """ + if not _STARTS_WITH_MATCH_RE.match(cypher): + raise ValueError( + "cypher_filter must start with MATCH (read-only graph " + "traversal); other Cypher clauses are not allowed." + ) + if ";" in cypher: + raise ValueError( + "cypher_filter must be a single statement; " + "semicolons are not allowed." + ) + bare = _QUOTED_STRING_RE.sub("", cypher) + match = _MUTATION_RE.search(bare) + if match: + raise ValueError( + f"cypher_filter must be read-only; " + f"mutation keyword {match.group()!r} is not allowed." + ) + if not _AS_ID_RE.search(cypher): + raise ValueError( + "cypher_filter must return exactly one column aliased 'id' " + "(e.g. 'RETURN n.id AS id')." + ) + + +# --------------------------------------------------------------------------- +# Graph construction helpers +# --------------------------------------------------------------------------- + + +def _create_schema(conn: Any) -> None: + """Create all node and relationship tables.""" + conn.execute( + "CREATE NODE TABLE Memory(" + "id STRING, learning STRING, description STRING, PRIMARY KEY(id))" + ) + conn.execute( + "CREATE NODE TABLE Datasource(" + "id STRING, name STRING, description STRING, PRIMARY KEY(id))" + ) + conn.execute( + "CREATE NODE TABLE Model(" + "id STRING, name STRING, description STRING, PRIMARY KEY(id))" + ) + conn.execute( + "CREATE NODE TABLE ModelColumn(" + "id STRING, name STRING, data_type STRING, description STRING, PRIMARY KEY(id))" + ) + conn.execute( + "CREATE NODE TABLE Measure(" + "id STRING, name STRING, description STRING, PRIMARY KEY(id))" + ) + conn.execute( + "CREATE NODE TABLE Aggregation(" + "id STRING, name STRING, description STRING, PRIMARY KEY(id))" + ) + conn.execute( + "CREATE REL TABLE MENTIONS(" + "FROM Memory TO Datasource, " + "FROM Memory TO Model, " + "FROM Memory TO ModelColumn, " + "FROM Memory TO Measure, " + "FROM Memory TO Aggregation, " + "FROM Memory TO Memory" + ")" + ) + conn.execute( + "CREATE REL TABLE CONTAINS(" + "FROM Datasource TO Model, " + "FROM Model TO ModelColumn, " + "FROM Model TO Measure, " + "FROM Model TO Aggregation" + ")" + ) + conn.execute("CREATE REL TABLE JOINS(FROM Model TO Model)") + + +def _insert_model_child_nodes(conn: Any, canonical_model: str, model: Any) -> None: + """Insert ModelColumn, Measure, and Aggregation nodes for one model.""" + for col in model.columns: + if col.hidden: + continue + conn.execute( + "CREATE (:ModelColumn {" + "id: $id, name: $name, data_type: $dt, description: $descr" + "})", + { + "id": f"{canonical_model}.{col.name}", + "name": col.name, + "dt": col.type.value if col.type is not None else "", + "descr": col.description or "", + }, + ) + for measure in model.measures: + if not measure.name: + continue + conn.execute( + "CREATE (:Measure {id: $id, name: $name, description: $descr})", + { + "id": f"{canonical_model}.{measure.name}", + "name": measure.name, + "descr": measure.description or "", + }, + ) + for agg in model.aggregations: + conn.execute( + "CREATE (:Aggregation {id: $id, name: $name, description: $descr})", + { + "id": f"{canonical_model}.{agg.name}", + "name": agg.name, + "descr": agg.description or "", + }, + ) + + +def _insert_nodes( + conn: Any, + datasource_names: list[str], + visible_models: dict, + memories: list, + datasource_descriptions: dict[str, str | None] | None = None, +) -> None: + """Insert all node rows into the graph. + + DEV-1549: ``datasource_descriptions`` carries + ``{ds_name → DatasourceConfig.description}`` so the graph schema + can expose ``Datasource.description`` for cypher_filter queries, + in parity with the lexical and embedding channels. Likewise + ``Memory.description`` is inserted directly from the memory. + """ + descriptions = datasource_descriptions or {} + for name in datasource_names: + conn.execute( + "CREATE (:Datasource {id: $id, name: $name, description: $descr})", + { + "id": name, + "name": name, + "descr": descriptions.get(name) or "", + }, + ) + + for canonical_model, model in visible_models.items(): + _, model_name = canonical_model.split(".", 1) + conn.execute( + "CREATE (:Model {id: $id, name: $name, description: $descr})", + { + "id": canonical_model, + "name": model_name, + "descr": model.description or "", + }, + ) + _insert_model_child_nodes(conn, canonical_model, model) + + for mem in memories: + conn.execute( + "CREATE (:Memory {id: $id, learning: $learning, description: $descr})", + { + "id": f"{_MEMORY_PREFIX}{mem.id}", + "learning": mem.learning, + "descr": mem.description or "", + }, + ) + + +def _insert_contains_edges( + conn: Any, + datasource_names: list[str], + visible_models: dict, +) -> None: + """Insert CONTAINS edges: Datasource→Model and Model→{ModelColumn/Measure/Agg}.""" + ds_set = set(datasource_names) + for canonical_model, model in visible_models.items(): + ds = canonical_model.split(".", 1)[0] + if ds in ds_set: + conn.execute( + "MATCH (d:Datasource {id: $ds}), (m:Model {id: $model}) " + "CREATE (d)-[:CONTAINS]->(m)", + {"ds": ds, "model": canonical_model}, + ) + for col in model.columns: + if col.hidden: + continue + conn.execute( + "MATCH (m:Model {id: $model}), (c:ModelColumn {id: $col}) " + "CREATE (m)-[:CONTAINS]->(c)", + {"model": canonical_model, "col": f"{canonical_model}.{col.name}"}, + ) + for measure in model.measures: + if not measure.name: + continue + conn.execute( + "MATCH (m:Model {id: $model}), (ms:Measure {id: $ms}) " + "CREATE (m)-[:CONTAINS]->(ms)", + {"model": canonical_model, "ms": f"{canonical_model}.{measure.name}"}, + ) + for agg in model.aggregations: + conn.execute( + "MATCH (m:Model {id: $model}), (a:Aggregation {id: $agg}) " + "CREATE (m)-[:CONTAINS]->(a)", + {"model": canonical_model, "agg": f"{canonical_model}.{agg.name}"}, + ) + + +def _insert_joins_edges(conn: Any, visible_models: dict) -> None: + """Insert JOINS edges: Model→Model (via model.joins). Missing targets silently skipped.""" + for canonical_model, model in visible_models.items(): + ds = canonical_model.split(".", 1)[0] + for join in model.joins: + target_canonical = f"{ds}.{join.target_model}" + if target_canonical not in visible_models: + continue + conn.execute( + "MATCH (src:Model {id: $src}), (tgt:Model {id: $tgt}) " + "CREATE (src)-[:JOINS]->(tgt)", + {"src": canonical_model, "tgt": target_canonical}, + ) + + +def _connect_entity_mention( + conn: Any, + src: str, + entity: str, + ds_set: set[str], + valid_models: set[str], + valid_columns: set[str], + valid_measures: set[str], + valid_aggs: set[str], + valid_memory_canonicals: set[str], +) -> None: + """Create one MENTIONS edge from memory *src* to the matching entity node.""" + if entity in valid_memory_canonicals: + conn.execute( + "MATCH (m1:Memory {id: $src}), (m2:Memory {id: $tgt}) " + "CREATE (m1)-[:MENTIONS]->(m2)", + {"src": src, "tgt": entity}, + ) + elif entity in ds_set: + conn.execute( + "MATCH (m:Memory {id: $src}), (d:Datasource {id: $tgt}) " + "CREATE (m)-[:MENTIONS]->(d)", + {"src": src, "tgt": entity}, + ) + elif entity in valid_models: + conn.execute( + "MATCH (m:Memory {id: $src}), (n:Model {id: $tgt}) " + "CREATE (m)-[:MENTIONS]->(n)", + {"src": src, "tgt": entity}, + ) + elif entity in valid_measures: + conn.execute( + "MATCH (m:Memory {id: $src}), (ms:Measure {id: $tgt}) " + "CREATE (m)-[:MENTIONS]->(ms)", + {"src": src, "tgt": entity}, + ) + elif entity in valid_aggs: + conn.execute( + "MATCH (m:Memory {id: $src}), (a:Aggregation {id: $tgt}) " + "CREATE (m)-[:MENTIONS]->(a)", + {"src": src, "tgt": entity}, + ) + elif entity in valid_columns: + conn.execute( + "MATCH (m:Memory {id: $src}), (c:ModelColumn {id: $tgt}) " + "CREATE (m)-[:MENTIONS]->(c)", + {"src": src, "tgt": entity}, + ) + + +def _build_valid_entity_sets( + visible_models: dict, + memories: list, +) -> tuple[set[str], set[str], set[str], set[str]]: + """Build (valid_columns, valid_measures, valid_aggs, valid_memory_canonicals).""" + valid_columns: set[str] = set() + valid_measures: set[str] = set() + valid_aggs: set[str] = set() + for canonical_model, model in visible_models.items(): + for col in model.columns: + if not col.hidden: + valid_columns.add(f"{canonical_model}.{col.name}") + for measure in model.measures: + if measure.name: + valid_measures.add(f"{canonical_model}.{measure.name}") + for agg in model.aggregations: + valid_aggs.add(f"{canonical_model}.{agg.name}") + valid_memory_canonicals = {f"{_MEMORY_PREFIX}{m.id}" for m in memories} + return valid_columns, valid_measures, valid_aggs, valid_memory_canonicals + + +def _insert_mentions_edges( + conn: Any, + memories: list, + visible_models: dict, + datasource_names: list[str], +) -> None: + """Insert MENTIONS edges: Memory → {Datasource, Model, ModelColumn, Measure, Agg, Memory}.""" + ds_set = set(datasource_names) + valid_models: set[str] = set(visible_models) + valid_columns, valid_measures, valid_aggs, valid_memory_canonicals = ( + _build_valid_entity_sets(visible_models, memories) + ) + + for mem in memories: + src = f"{_MEMORY_PREFIX}{mem.id}" + for entity in mem.entities: + _connect_entity_mention( + conn, src, entity, + ds_set, valid_models, valid_columns, valid_measures, valid_aggs, + valid_memory_canonicals, + ) + + +async def build_graph(storage: StorageBackend) -> tuple[Any, Any]: + """Build an ephemeral in-memory LadybugDB graph from ``storage``. + + Returns ``(db, conn)``. Hidden models and hidden columns are excluded. + Memory canonical IDs are stored in ``memory:`` form. + """ + mod = _import_graph_module() + # No-argument Database() creates an ephemeral in-memory instance; + # no files are written to the working directory. + db = mod.Database() + conn = mod.Connection(db) + _create_schema(conn) + + datasource_names = await storage.list_datasources() + identities = await storage._list_all_model_identities() + + visible_models: dict = {} + for ds, model_name in identities: + model = await storage.get_model(model_name, data_source=ds) + if model is not None and not model.hidden: + visible_models[f"{ds}.{model_name}"] = model + + memories = await storage.list_memories(entities=None) + + # DEV-1549: load datasource descriptions so the graph schema can + # expose `Datasource.description` for cypher_filter queries. + datasource_descriptions: dict[str, str | None] = {} + for ds_name in datasource_names: + cfg = await storage.get_datasource(ds_name) + datasource_descriptions[ds_name] = ( + cfg.description if cfg is not None else None + ) + + _insert_nodes( + conn, + datasource_names, + visible_models, + memories, + datasource_descriptions=datasource_descriptions, + ) + _insert_contains_edges(conn, datasource_names, visible_models) + _insert_joins_edges(conn, visible_models) + _insert_mentions_edges(conn, memories, visible_models, datasource_names) + + return db, conn + + +# --------------------------------------------------------------------------- +# Per-storage cache with double-checked locking +# --------------------------------------------------------------------------- + + +class _GraphCache(BaseModel): + fingerprint: str + db: Any + conn: Any + + model_config = {"arbitrary_types_allowed": True} + + +_cache: dict[str, _GraphCache] = {} +_locks: dict[str, asyncio.Lock] = {} + + +def _storage_key(storage: StorageBackend) -> str: + """Stable path key for cache lookups.""" + from slayer.storage.join_sync import JoinSyncStorage + from slayer.storage.sqlite_storage import SQLiteStorage + from slayer.storage.yaml_storage import YAMLStorage + + if isinstance(storage, JoinSyncStorage): + return _storage_key(storage._inner) + if isinstance(storage, YAMLStorage): + return os.path.abspath(storage.base_dir) + if isinstance(storage, SQLiteStorage): + return os.path.abspath(storage.db_path) + return str(id(storage)) + + +def _get_lock(key: str) -> asyncio.Lock: + """Return the per-key asyncio.Lock, creating it if absent. + + Safe without an outer lock because no ``await`` separates the + membership check from the insertion (asyncio is single-threaded). + """ + if key not in _locks: + _locks[key] = asyncio.Lock() + return _locks[key] + + +def _close_entry(entry: "_GraphCache") -> None: + """Best-effort close of a cached graph entry to release LadybugDB handles.""" + for obj in (entry.conn, entry.db): + try: + obj.close() + except Exception: # noqa: BLE001 + pass + + +def clear_cache() -> None: + """Discard all cached graphs and locks. Primarily used in tests.""" + for entry in _cache.values(): + _close_entry(entry) + _cache.clear() + _locks.clear() + + +async def _get_or_rebuild(storage: StorageBackend) -> tuple[Any, Any]: + """Return cached (db, conn) if the fingerprint matches; else rebuild.""" + key = _storage_key(storage) + + try: + current_fp: str | None = await storage.graph_fingerprint() + except OSError: + current_fp = None + + # Fast path: no lock needed when cache is warm and fingerprint matches. + cached = _cache.get(key) + if cached is not None and current_fp is not None and cached.fingerprint == current_fp: + return cached.db, cached.conn + + lock = _get_lock(key) + async with lock: + # Double-check under the lock so only one rebuild fires. + cached = _cache.get(key) + if cached is not None and current_fp is not None and cached.fingerprint == current_fp: + return cached.db, cached.conn + + old = _cache.get(key) + db, conn = await build_graph(storage) + if old is not None: + _close_entry(old) + _cache[key] = _GraphCache( + fingerprint=current_fp if current_fp is not None else "", + db=db, + conn=conn, + ) + return db, conn + + +async def get_filtered_ids( + cypher: str, + storage: StorageBackend, +) -> frozenset[str]: + """Execute a Cypher query against the storage graph and return the + frozenset of id strings from the result's ``id`` column. + + Raises ``ValueError`` if the query fails validation or if LadybugDB + is not installed. + """ + if not is_available(): + raise ValueError( + "cypher_filter requires LadybugDB; " + "install with: pip install motley-slayer[advanced_search]" + ) + _validate_cypher(cypher) + _db, conn = await _get_or_rebuild(storage) + try: + result = conn.execute(cypher) + col_names: list[str] = result.get_column_names() + if "id" not in col_names: + raise ValueError( + "cypher_filter must return a column named 'id'; " + f"got columns: {col_names!r}." + ) + id_idx = col_names.index("id") + except ValueError: + raise + except Exception as exc: + raise ValueError(f"cypher_filter execution failed: {exc}") from exc + ids: set[str] = set() + while result.has_next(): + row = result.get_next() + if row and row[id_idx] is not None: + ids.add(str(row[id_idx])) + return frozenset(ids) diff --git a/slayer/search/index.py b/slayer/search/index.py index 65fc601c..ddd58485 100644 --- a/slayer/search/index.py +++ b/slayer/search/index.py @@ -19,7 +19,6 @@ from __future__ import annotations -from typing import Dict, List, Optional, Tuple import tantivy from pydantic import BaseModel, ConfigDict @@ -27,12 +26,9 @@ from slayer.core.models import SlayerModel from slayer.memories.models import Memory from slayer.search.render import ( - render_aggregation_text, - render_column_text, - render_datasource_text, - render_measure_text, + collect_model_entity_pairs, + render_datasource_pair, render_memory_text, - render_model_text, ) @@ -51,7 +47,7 @@ class IndexHit(BaseModel): canonical: str text: str score: float - memory_id: Optional[str] = None # populated only when kind == "memory" + memory_id: str | None = None # populated only when kind == "memory" # --------------------------------------------------------------------------- @@ -86,9 +82,10 @@ def _add_doc( def build_in_memory_index( *, - memories: List[Memory], - models: List[SlayerModel], - datasources: List[str], + memories: list[Memory], + models: list[SlayerModel], + datasources: list[str], + datasource_descriptions: dict[str, str | None] | None = None, ) -> tantivy.Index: """Build a fresh in-RAM tantivy index covering the corpus. @@ -96,11 +93,18 @@ def build_in_memory_index( expected to pass datasource names + every model in scope; this function does *not* call into storage. + DEV-1549: ``datasource_descriptions`` mirrors the symmetric kwarg on + :func:`build_in_memory_corpus` so direct callers of this helper can + also surface datasource-description text in the lexical index. + Returns just the tantivy index for callers that don't need the canonical-text lookups. ``build_in_memory_corpus`` returns both. """ corpus = build_in_memory_corpus( - memories=memories, models=models, datasources=datasources, + memories=memories, + models=models, + datasources=datasources, + datasource_descriptions=datasource_descriptions, ) return corpus.index @@ -109,86 +113,82 @@ class Corpus(BaseModel): """The tantivy index plus the parallel ``canonical_id → text`` and ``canonical_id → kind`` maps. The embedding channel (DEV-1386) uses the maps to recover hit text without re-rendering the entity or - round-tripping through the raw ``canonical`` tantivy field.""" + round-tripping through the raw ``canonical`` tantivy field. + + DEV-1549: ``canonical_to_description`` lets the search service + surface the entity's structured description on ``SearchHit`` without + re-loading the entity at hit-construction time. + """ model_config = ConfigDict(arbitrary_types_allowed=True) index: "tantivy.Index" - canonical_to_text: Dict[str, str] - canonical_to_kind: Dict[str, str] - - -def _render_model_subtree_pairs( - model: SlayerModel, -) -> List[Tuple[str, str, str]]: - """Render docs for one model: the model itself + its visible columns + - named measures + custom aggregations. Hidden columns and unnamed - measures are skipped to match the indexer's filter rules.""" - model_canonical = f"{model.data_source}.{model.name}" - pairs: List[Tuple[str, str, str]] = [( - model_canonical, "model", render_model_text(model=model), - )] - for column in model.columns: - if column.hidden: - continue - pairs.append(( - f"{model_canonical}.{column.name}", "column", - render_column_text(model=model, column=column), - )) - for measure in model.measures: - if measure.name is None: - continue - pairs.append(( - f"{model_canonical}.{measure.name}", "measure", - render_measure_text(model=model, measure=measure), - )) - for aggregation in model.aggregations: - pairs.append(( - f"{model_canonical}.{aggregation.name}", "aggregation", - render_aggregation_text(model=model, aggregation=aggregation), - )) - return pairs + canonical_to_text: dict[str, str] + canonical_to_kind: dict[str, str] + canonical_to_description: dict[str, str | None] = {} def _collect_render_pairs( *, - memories: List[Memory], - visible_models: List[SlayerModel], - datasources: List[str], -) -> List[Tuple[str, str, str]]: - """Return ``[(canonical_id, kind, rendered_text), ...]`` for every - doc that goes into the index. Same filter rules as the indexer: - hidden models and hidden columns are skipped.""" - out: List[Tuple[str, str, str]] = [] - models_by_ds: Dict[str, List[SlayerModel]] = {} + memories: list[Memory], + visible_models: list[SlayerModel], + datasources: list[str], + datasource_descriptions: dict[str, str | None] | None = None, +) -> list[tuple[str, str, str, str | None]]: + """Return ``[(canonical_id, kind, rendered_text, description), ...]`` + for every doc that goes into the index. Routes through the unified + dispatch helpers in ``slayer.search.render`` (DEV-1513). Hidden + models and hidden columns are skipped inside the helpers. + + DEV-1549: the fourth tuple element carries the entity's structured + ``description`` field (``None`` when absent) so callers can build a + ``canonical_id → description`` map symmetrical to the existing + canonical-text map. + """ + out: list[tuple[str, str, str, str | None]] = [] + models_by_ds: dict[str, list[SlayerModel]] = {} for m in visible_models: models_by_ds.setdefault(m.data_source, []).append(m) + descriptions = datasource_descriptions or {} for ds in datasources: - out.append(( - ds, "datasource", - render_datasource_text(name=ds, models=models_by_ds.get(ds, [])), - )) + pair = render_datasource_pair( + name=ds, + models=models_by_ds.get(ds, []), + description=descriptions.get(ds), + ) + out.append((pair.canonical_id, pair.kind, pair.text, pair.description)) for model in visible_models: - out.extend(_render_model_subtree_pairs(model)) + for re in collect_model_entity_pairs(model=model): + out.append((re.canonical_id, re.kind, re.text, re.description)) for memory in memories: + # Memories surface ``Memory.description`` directly so the search + # service can flip on compact rendering without re-loading the + # memory. out.append(( f"memory:{memory.id}", "memory", render_memory_text(memory=memory), + memory.description, )) return out def build_in_memory_corpus( *, - memories: List[Memory], - models: List[SlayerModel], - datasources: List[str], + memories: list[Memory], + models: list[SlayerModel], + datasources: list[str], + datasource_descriptions: dict[str, str | None] | None = None, ) -> Corpus: """Build the index AND the parallel canonical lookup maps in one walk. The embedding channel (DEV-1386) reads from the same render pipeline as tantivy, so rendering once here keeps the two channels in sync without paying for two traversals. + + DEV-1549: ``datasource_descriptions`` is an optional + ``{ds_name → description}`` map (``None`` description when the + datasource has none). When omitted, datasource hits get + ``description=None``. """ schema = _build_schema() index = tantivy.Index(schema=schema) @@ -207,10 +207,12 @@ def build_in_memory_corpus( memories=memories, visible_models=visible_models, datasources=datasources, + datasource_descriptions=datasource_descriptions, ) - canonical_to_text: Dict[str, str] = {} - canonical_to_kind: Dict[str, str] = {} - for canonical, kind, text in pairs: + canonical_to_text: dict[str, str] = {} + canonical_to_kind: dict[str, str] = {} + canonical_to_description: dict[str, str | None] = {} + for canonical, kind, text, description in pairs: # Memory docs use ``id="memory:"`` and ``canonical=""`` # to match the DEV-1375 tantivy schema; entity docs use the same # canonical string for both ``id`` and ``canonical`` fields. @@ -227,6 +229,7 @@ def build_in_memory_corpus( ) canonical_to_text[canonical] = text canonical_to_kind[canonical] = kind + canonical_to_description[canonical] = description writer.commit() index.reload() @@ -234,6 +237,7 @@ def build_in_memory_corpus( index=index, canonical_to_text=canonical_to_text, canonical_to_kind=canonical_to_kind, + canonical_to_description=canonical_to_description, ) @@ -246,8 +250,8 @@ def _apply_kind_filter( *, query: "tantivy.Query", schema: "tantivy.Schema", - kind_filter: Optional[str], - exclude_kind: Optional[str], + kind_filter: str | None, + exclude_kind: str | None, ) -> "tantivy.Query": """Wrap ``query`` in a boolean query that ``Must`` includes (or ``MustNot`` excludes) docs whose ``kind`` field exactly equals the @@ -272,10 +276,10 @@ def search_index( index: tantivy.Index, question: str, limit: int = 20, - fields: Optional[List[str]] = None, - kind_filter: Optional[str] = None, - exclude_kind: Optional[str] = None, -) -> List[IndexHit]: + fields: list[str] | None = None, + kind_filter: str | None = None, + exclude_kind: str | None = None, +) -> list[IndexHit]: """Run a tantivy query against ``index``. Args: @@ -318,12 +322,12 @@ def search_index( ) searcher = index.searcher() raw_hits = searcher.search(query, limit).hits - out: List[IndexHit] = [] + out: list[IndexHit] = [] for score, address in raw_hits: doc = searcher.doc(address) kind = str(doc.get_first("kind")) canonical = str(doc.get_first("canonical")) - memory_id: Optional[str] = None + memory_id: str | None = None if kind == "memory": memory_id = canonical or None out.append(IndexHit( diff --git a/slayer/search/render.py b/slayer/search/render.py index ef1cc86c..9be9114a 100644 --- a/slayer/search/render.py +++ b/slayer/search/render.py @@ -26,7 +26,9 @@ from __future__ import annotations -from typing import List +import json + +from pydantic import BaseModel from slayer.core.models import ( Aggregation, @@ -37,7 +39,27 @@ from slayer.memories.models import Memory -def _named_children_csv(items: List[tuple[str, str]]) -> str: +class RenderedEntity(BaseModel): + """One (canonical_id, kind, text) triple produced by the unified + dispatch (DEV-1513). Carries the indexed text + the entity-kind tag + every caller needs (corpus build, embedding refresh, named-entity + surfacing). Single source of truth for "what counts as an indexable + entity" — filter rules (hidden model -> empty list, hidden column + skipped, unnamed measure skipped) live in ``collect_model_entity_pairs`` + and ``render_datasource_pair`` only. + + DEV-1549: ``description`` carries the entity's structured description + field (``None`` when the entity has none). The search service surfaces + it as ``SearchHit.description`` under compact mode. + """ + + canonical_id: str + kind: str + text: str + description: str | None = None + + +def _named_children_csv(items: list[tuple[str, str]]) -> str: """Render ``[("a", "column"), ("b", "column")]`` as ``"a (column), b (column)"``.""" return ", ".join(f"{name} ({kind})" for name, kind in items) @@ -47,12 +69,26 @@ def _named_children_csv(items: List[tuple[str, str]]) -> str: # --------------------------------------------------------------------------- -def render_datasource_text(*, name: str, models: List[SlayerModel]) -> str: - """Datasource doc: name + named-child mentions for each model. +def render_datasource_text( + *, + name: str, + models: list[SlayerModel], + description: str | None = None, +) -> str: + """Datasource doc: name + own description (when set) + named-child + mentions for each model. - No model descriptions — each model has its own indexed doc. + No model descriptions are included here — each model has its own + indexed doc. + + DEV-1549: ``description`` is included so the lexical BM25 / tantivy + channels can match terms that live only in the datasource's + ``DatasourceConfig.description`` field, in parity with the other + entity render helpers. """ - lines: List[str] = [f"Datasource: {name}"] + lines: list[str] = [f"Datasource: {name}"] + if description: + lines.append(f"Description: {description}") visible = [m for m in models if not m.hidden] if visible: lines.append( @@ -63,6 +99,31 @@ def render_datasource_text(*, name: str, models: List[SlayerModel]) -> str: return "\n".join(lines) +def render_datasource_pair( + *, + name: str, + models: list[SlayerModel], + description: str | None = None, +) -> RenderedEntity: + """Unified dispatch (DEV-1513) for the datasource doc. Used by both + the tantivy corpus builder and the embedding refresh path so the + visibility filter is applied in exactly one place. + + DEV-1549: ``description`` is the datasource's free-form description + (DatasourceConfig.description), surfaced as ``SearchHit.description`` + under compact mode AND woven into the indexed text so lexical / + embedding channels can match terms that live only there. + """ + return RenderedEntity( + canonical_id=name, + kind="datasource", + text=render_datasource_text( + name=name, models=models, description=description, + ), + description=description, + ) + + # --------------------------------------------------------------------------- # Model # --------------------------------------------------------------------------- @@ -71,7 +132,7 @@ def render_datasource_text(*, name: str, models: List[SlayerModel]) -> str: def render_model_text(*, model: SlayerModel) -> str: """Model doc: own metadata, non-named children in full, named children by name + kind only.""" - lines: List[str] = [ + lines: list[str] = [ f"Model: {model.data_source}.{model.name}", ] if model.description: @@ -136,7 +197,7 @@ def render_model_text(*, model: SlayerModel) -> str: def render_column_text(*, model: SlayerModel, column: Column) -> str: """Column doc: parent qualifier + per-field metadata + cached sample.""" - lines: List[str] = [ + lines: list[str] = [ f"Column: {model.data_source}.{model.name}.{column.name}", f"Type: {column.type}", ] @@ -152,11 +213,36 @@ def render_column_text(*, model: SlayerModel, column: Column) -> str: lines.append(f"SQL: {column.sql}") if column.filter: lines.append(f"Filter: {column.filter}") - # DEV-1480: skip the line when ``sampled`` is empty (all-NULL profiled - # categorical column). Avoids a bare ``Sample values: `` trailer in the - # embedded doc text, and keeps the content_hash stable for columns whose - # only DEV-1480 change is the new structured ``sampled_values`` field. - if column.sampled: + # DEV-1516: prefer the structured ``sampled_values`` list (full top-50) + # over the 20-truncated ``sampled`` text. ``is None`` gates the fallback + # so an authoritative empty list (``[]``) does not re-surface a stale + # ``sampled`` text; an empty list simply skips the line (avoids a bare + # ``Sample values: `` trailer in the indexed text). + if column.sampled_values is not None: + if column.sampled_values: + # JSON-encode the list to preserve values that contain commas + # (e.g. ``"R$ 1,000–3,000"``) — comma-joining would re-introduce + # the exact ambiguity that the structured ``sampled_values`` field + # was meant to solve. + lines.append( + "Sample values: " + + json.dumps(column.sampled_values, ensure_ascii=False) + ) + # Overflow signal: render true cardinality on a follow-up line + # only when STRICTLY greater than the values we returned. Equal + # means we returned the entire set; emitting a hint would be + # noise. Gated on ``sampled_values is not None`` so the legacy + # ``"... (N distinct)"`` suffix in ``sampled`` text does not get + # duplicated by an extra line. + if ( + column.distinct_count is not None + and column.distinct_count > len(column.sampled_values) + ): + lines.append(f"Distinct count: {column.distinct_count}") + elif column.sampled: + # Fallback for numeric/temporal columns (``sampled`` is a min/max + # range, not a list) and pre-DEV-1480 legacy data where the + # structured field was never populated. lines.append(f"Sample values: {column.sampled}") if column.primary_key: lines.append("Primary key: yes") @@ -170,7 +256,7 @@ def render_column_text(*, model: SlayerModel, column: Column) -> str: def render_measure_text(*, model: SlayerModel, measure: ModelMeasure) -> str: name = measure.name or "" - lines: List[str] = [ + lines: list[str] = [ f"Measure: {model.data_source}.{model.name}.{name}", f"Formula: {measure.formula}", ] @@ -187,7 +273,7 @@ def render_measure_text(*, model: SlayerModel, measure: ModelMeasure) -> str: def render_aggregation_text(*, model: SlayerModel, aggregation: Aggregation) -> str: - lines: List[str] = [ + lines: list[str] = [ f"Aggregation: {model.data_source}.{model.name}.{aggregation.name}", ] if aggregation.formula: @@ -207,22 +293,120 @@ def render_aggregation_text(*, model: SlayerModel, aggregation: Aggregation) -> def render_memory_text(*, memory: Memory) -> str: - """Memory doc for tantivy: learning text + tagged canonical entities - so the memory surfaces both via natural-language search and via - exact-entity search.""" - lines: List[str] = [memory.learning] + """Memory doc for tantivy: learning text + optional description + + tagged canonical entities so the memory surfaces both via + natural-language search and via exact-entity search. + + DEV-1549: ``description`` is included here so the lexical BM25 / + tantivy channels can match terms that live only in + ``Memory.description``. Without this, installs without the optional + embedding extra would lose recall for the new field. + """ + lines: list[str] = [memory.learning] + if memory.description: + lines.append(memory.description) if memory.entities: lines.append("Tagged entities: " + ", ".join(memory.entities)) return "\n".join(lines) +def compact_description_from_learning(learning: str) -> str: + """DEV-1549 compact-mode fallback: take the first non-empty + paragraph (text up to the first blank line) of ``learning`` and + cap at 500 chars (suffix-truncated, no ellipsis). + + No special-case for ``description:`` keyword lines (the user + explicitly rejected that during spec review). + """ + para: list[str] = [] + started = False + for line in learning.splitlines(): + if line.strip(): + para.append(line) + started = True + elif started: + break + return "\n".join(para)[:500] + + def render_memory_text_for_embedding(*, memory: Memory) -> str: - """Memory doc for embeddings: learning text ONLY. + """Memory doc for embeddings: learning text + optional description. + + DEV-1428: entity tags are excluded so the cascade-strip path (which + only rewrites ``entities``) does not change the embedding content + hash. The cascade still skips embedding work for free. - DEV-1428: by excluding the entity tags from the embedded text, the - cascade-strip path (which rewrites the tag list) does not change the - embedding content hash, so the per-memory refresh hash-skips. This - is what lets the cascade live entirely in the storage layer with - zero embedding cost per deleted entity. + DEV-1549 (Codex#5): when ``description`` is set, append it so the + user-supplied summary contributes to semantic recall. Cascade-strip + only touches ``entities``, so the hash skip on tag-only mutations is + preserved. """ + if memory.description: + return f"{memory.learning}\n\n{memory.description}" return memory.learning + + +# --------------------------------------------------------------------------- +# Unified entity-pair dispatch (DEV-1513) +# --------------------------------------------------------------------------- + + +def collect_model_entity_pairs( + *, model: SlayerModel, include_hidden: bool = False +) -> list[RenderedEntity]: + """Walk a model's subtree (model + columns + named measures + + custom aggregations) into the unified ``RenderedEntity`` shape. + + Filter rules (single source of truth): + + * Hidden model -> returns ``[]`` (unless ``include_hidden``). + * Hidden column skipped (unless ``include_hidden``). + * ``ModelMeasure`` whose ``name is None`` skipped (defensive: the + Pydantic validator already rejects unnamed measures, but the skip + keeps the helper aligned with the documented filter set). + + ``include_hidden=True`` (DEV-1588) is the ``inspect`` escape hatch: + it emits the hidden model + hidden columns so a deliberate + point-lookup can render them. The default (``False``) preserves the + search/index corpus behavior — hidden entities never leak. + + Used by the tantivy corpus build, the embedding refresh path, and + the new named-entity surfacing path. The leaf ``render_*_text`` + helpers are still the single source of truth for *what* each kind's + text looks like; this helper is the single source of truth for + *which* entities exist and at which canonical id.""" + if model.hidden and not include_hidden: + return [] + qualifier = f"{model.data_source}.{model.name}" + out: list[RenderedEntity] = [RenderedEntity( + canonical_id=qualifier, + kind="model", + text=render_model_text(model=model), + description=model.description, + )] + for column in model.columns: + if column.hidden and not include_hidden: + continue + out.append(RenderedEntity( + canonical_id=f"{qualifier}.{column.name}", + kind="column", + text=render_column_text(model=model, column=column), + description=column.description, + )) + for measure in model.measures: + if measure.name is None: + continue + out.append(RenderedEntity( + canonical_id=f"{qualifier}.{measure.name}", + kind="measure", + text=render_measure_text(model=model, measure=measure), + description=measure.description, + )) + for aggregation in model.aggregations: + out.append(RenderedEntity( + canonical_id=f"{qualifier}.{aggregation.name}", + kind="aggregation", + text=render_aggregation_text(model=model, aggregation=aggregation), + description=aggregation.description, + )) + return out diff --git a/slayer/search/retriever.py b/slayer/search/retriever.py new file mode 100644 index 00000000..b0799853 --- /dev/null +++ b/slayer/search/retriever.py @@ -0,0 +1,123 @@ +"""Retriever ABC + shared result type for SLayer's pluggable search +facade (DEV-1514). + +A ``Retriever`` is one ranking channel. The orchestrator (see +:class:`slayer.search.service.SearchService`) holds a list of them, +calls each one's :meth:`Retriever.retrieve` in parallel, and fuses the +returned rankings via RRF. Concrete retrievers also own their own +persistence — write-side hooks (``upsert_memory``, +``refresh_model_subtree``, ``refresh_datasource``, plus the three +``delete_*`` futures) default to no-op on the ABC; the embedding +retriever overrides the create/refresh ones to maintain its sidecar +table. The ``delete_*`` hooks default to no-op for every shipping +retriever this PR — :class:`StorageBackend` owns embedding-row cascade +transactionally with the model/datasource/memory row delete; the +hooks live on the ABC so future persistent retrievers (e.g. +persistent tantivy) can override them without an ABC change. + +Returning ``memory_ranking`` + ``entity_ranking`` from a SINGLE +``retrieve`` call (rather than splitting them across two methods) lets +retrievers that share expensive setup across the two partitions — +litellm embed, dim-check, tantivy index handle — do that setup once +per search call. The orchestrator gathers across retrievers but never +within one (DEV-1514, Codex Findings 1 & 2). +""" + +from __future__ import annotations + +from abc import ABC, abstractmethod + +from pydantic import BaseModel, Field + +from slayer.core.models import SlayerModel +from slayer.memories.models import Memory +from slayer.search.index import Corpus + + +class RetrievalResult(BaseModel): + """Combined memory + entity ranking produced by ONE retrieve call. + + * ``memory_ranking`` — memory ids best-first. + * ``text_by_id`` — recovered hit text keyed by memory id, only + populated by retrievers that rendered text for the hit + (tantivy today). Lets :func:`_build_memory_hit` surface the + tantivy hit text in ``MemoryHit.text`` instead of always + falling back to ``Memory.learning``. + * ``entity_ranking`` — canonical entity ids best-first + (``""`` / ``"."`` / ``".."``). + * ``warnings`` — per-call warnings the orchestrator should + surface to the caller (e.g. "embedding channel skipped: + query embedding failed"). + """ + + memory_ranking: list[str] = Field(default_factory=list) + text_by_id: dict[str, str] = Field(default_factory=dict) + entity_ranking: list[str] = Field(default_factory=list) + warnings: list[str] = Field(default_factory=list) + + +class Retriever(ABC): + """One ranking channel. Subclasses set ``name`` and implement + :meth:`retrieve`; they may override any of the write-side hooks + to maintain their own persistence.""" + + name: str = "" + + @abstractmethod + async def retrieve( + self, + *, + query_entities: list[str], + question: str | None, + all_memories: list[Memory], + valid_canonicals: set, + corpus: Corpus | None, + datasource: str | None, + ) -> RetrievalResult: + """Return the channel's memory + entity rankings (plus any + per-call warnings). Empty rankings indicate "channel inactive + for this input"; the orchestrator skips empty channels in the + RRF fusion.""" + + # ------------------------------------------------------------------ + # Create / refresh hooks — default no-op + # ------------------------------------------------------------------ + + async def upsert_memory(self, memory: Memory) -> list[str]: # NOSONAR(S7503) — async signature required by Retriever ABC; subclasses override with truly-async hooks + return [] + + async def refresh_model_subtree(self, model: SlayerModel) -> list[str]: # NOSONAR(S7503) — async signature required by Retriever ABC; subclasses override with truly-async hooks + return [] + + async def refresh_datasource( # NOSONAR(S7503) — async signature required by Retriever ABC; subclasses override with truly-async hooks + self, + *, + name: str, + models: list[SlayerModel], + description: str | None = None, + ) -> list[str]: + return [] + + # ------------------------------------------------------------------ + # Delete hooks — default no-op + # + # All three shipping retrievers no-op these hooks this PR. + # :class:`StorageBackend` owns embedding-row cascade transactionally + # with the model / datasource / memory row delete. The hooks live + # on the ABC so future persistent retrievers (persistent tantivy, + # third-party vector DBs) can override them without an ABC change. + # ------------------------------------------------------------------ + + async def delete_memory(self, memory_id: str) -> None: # NOSONAR(S7503) — async signature required by Retriever ABC; future persistent retrievers will override with truly-async hooks + return None + + async def delete_model( # NOSONAR(S7503) — async signature required by Retriever ABC; future persistent retrievers will override with truly-async hooks + self, *, data_source: str, name: str, + ) -> None: + return None + + async def delete_datasource(self, name: str) -> None: # NOSONAR(S7503) — async signature required by Retriever ABC; future persistent retrievers will override with truly-async hooks + return None + + +__all__ = ["Retriever", "RetrievalResult"] diff --git a/slayer/search/retrievers/__init__.py b/slayer/search/retrievers/__init__.py new file mode 100644 index 00000000..ece14a24 --- /dev/null +++ b/slayer/search/retrievers/__init__.py @@ -0,0 +1,8 @@ +"""Concrete :class:`~slayer.search.retriever.Retriever` implementations +that ship with SLayer (DEV-1514).""" + +from slayer.search.retrievers.bm25 import BM25Retriever +from slayer.search.retrievers.embeddings import EmbeddingRetriever +from slayer.search.retrievers.tantivy import TantivyRetriever + +__all__ = ["BM25Retriever", "EmbeddingRetriever", "TantivyRetriever"] diff --git a/slayer/search/retrievers/bm25.py b/slayer/search/retrievers/bm25.py new file mode 100644 index 00000000..597df6db --- /dev/null +++ b/slayer/search/retrievers/bm25.py @@ -0,0 +1,89 @@ +"""BM25 retriever — entity-overlap BM25 over memory tags (DEV-1514). + +Ports the body of the former ``SearchService._run_channel_1`` into a +standalone :class:`~slayer.search.retriever.Retriever`. Stateless: no +persistence, no write hooks (defaults to ABC no-op). + +DEV-1513: every memory's effective tag list is augmented with +``memory:`` before BM25 ranking, so a user-supplied +``memory:`` ref surfaces the named memory at the top of the BM25 +ranking. Augmentation runs after the stale-tag filter so the self-ref +cannot be stripped even if ``valid_canonicals`` ever drifted. +""" + +from __future__ import annotations + + +from slayer.memories.models import MEMORY_CANONICAL_PREFIX as _MEMORY_PREFIX +from slayer.memories.models import Memory +from slayer.memories.ranker import bm25_rank +from slayer.search.index import Corpus +from slayer.search.retriever import RetrievalResult, Retriever + + +def _filter_memories_entities( + memories: list[Memory], *, valid_canonicals: set, +) -> list[Memory]: + """Return shallow copies of ``memories`` whose ``entities`` lists + are filtered down to ``valid_canonicals`` only (DEV-1428). Used to + feed BM25 a stale-free corpus without writing back to storage.""" + out: list[Memory] = [] + for m in memories: + live = [e for e in m.entities if e in valid_canonicals] + if live == m.entities: + out.append(m) + else: + out.append(m.model_copy(update={"entities": live})) + return out + + +def _augment_with_self_refs(memories: list[Memory]) -> list[Memory]: + """DEV-1513: augment each memory's ``entities`` with + ``memory:`` so a user-supplied ``memory:`` ref surfaces + the named memory at the top of the BM25 ranking. Idempotent.""" + out: list[Memory] = [] + for m in memories: + self_ref = f"{_MEMORY_PREFIX}{m.id}" + if self_ref in m.entities: + out.append(m) + else: + out.append(m.model_copy( + update={"entities": [self_ref, *m.entities]}, + )) + return out + + +class BM25Retriever(Retriever): + """BM25 over memory entity tags. Contributes only to memory + ranking — has nothing to say about entity documents.""" + + name = "bm25" + + async def retrieve( + self, + *, + query_entities: list[str], + question: str | None, + all_memories: list[Memory], + valid_canonicals: set, + corpus: Corpus | None, + datasource: str | None, + ) -> RetrievalResult: + if not query_entities: + return RetrievalResult() + # ``valid_canonicals`` is always supplied (non-Optional in the + # ABC); empty set means "no entities are live — drop every + # stale tag", which makes ``_filter_memories_entities`` strip + # everything and BM25 returns empty. Don't truthy-check it. + filtered = _filter_memories_entities( + all_memories, valid_canonicals=valid_canonicals, + ) + # DEV-1513: self-ref augmentation runs AFTER the stale-tag + # filter so the synthetic ref always survives. + augmented = _augment_with_self_refs(filtered) + ranked = bm25_rank( + memories=augmented, query_entities=query_entities, + ) + return RetrievalResult( + memory_ranking=[mem.id for mem, _ in ranked], + ) diff --git a/slayer/search/retrievers/embeddings.py b/slayer/search/retrievers/embeddings.py new file mode 100644 index 00000000..44000031 --- /dev/null +++ b/slayer/search/retrievers/embeddings.py @@ -0,0 +1,431 @@ +"""Embedding retriever — dense cosine over a sidecar embedding store +(DEV-1514; absorbs the former ``slayer.embeddings.service``). + +Owns the litellm refresh pipeline (write side) and the cosine ranking +(read side). The read side runs ``fetch_corpus`` ONCE, ``embed_question`` +ONCE, and the dim-check ONCE per :meth:`retrieve` call — both the memory +and entity rankings are produced from the same setup (Codex Finding 1). +""" + +from __future__ import annotations + +import hashlib +import logging + +from slayer.core.models import SlayerModel +from slayer.embeddings import client as embedding_client +from slayer.embeddings.client import current_model, embed_batch +from slayer.embeddings.models import Embedding, EntityKind +from slayer.memories.models import MEMORY_CANONICAL_PREFIX as _MEMORY_PREFIX +from slayer.memories.models import Memory +from slayer.memories.resolver import canonical_id_rooted_at +from slayer.search.index import Corpus +from slayer.search.render import ( + collect_model_entity_pairs, + render_datasource_pair, + render_memory_text_for_embedding, +) +from slayer.search.retriever import RetrievalResult, Retriever +from slayer.storage.base import StorageBackend + + +_log = logging.getLogger(__name__) + + +# --------------------------------------------------------------------------- +# Canonical id helpers +# --------------------------------------------------------------------------- + + +def _sha256(text: str) -> str: + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def _memory_canonical_id(memory_id: str) -> str: + return f"{_MEMORY_PREFIX}{memory_id}" + + +def _memory_id_from_canonical(canonical_id: str) -> str | None: + """Parse a memory row's canonical id back into the str memory id. + + Returns ``None`` when the input is not exactly of the shape + ``memory:`` — a corrupted / stale embedding row + carrying ``foo:bar`` must not be mis-mapped to a memory hit + (DEV-1428 review).""" + if not canonical_id.startswith(_MEMORY_PREFIX): + return None + memory_id = canonical_id[len(_MEMORY_PREFIX):] + return memory_id or None + + +# --------------------------------------------------------------------------- +# Read-side helpers +# --------------------------------------------------------------------------- + + +def _filter_embedding_corpus_by_datasource( + rows: list[Embedding], + *, + datasource: str, + eligible_memory_canonicals: set[str], +) -> list[Embedding]: + """DEV-1409: narrow the embedding corpus to rows that survive a + datasource filter. Memory rows must appear in + ``eligible_memory_canonicals`` (already datasource-filtered + upstream); entity rows must be rooted at ``datasource`` per the + dotted-namespace rule.""" + return [ + r for r in rows + if ( + (r.entity_kind == "memory" + and r.canonical_id in eligible_memory_canonicals) + or (r.entity_kind != "memory" + and canonical_id_rooted_at( + canonical_id=r.canonical_id, datasource=datasource, + )) + ) + ] + + +def _rank_embedding_kind( + *, rows, normalised_query, np, normalise_matrix, top_k_cosine, +) -> list[str]: + """Rank one kind of embedding rows by cosine similarity to the + pre-normalised query vector. Returns the rows' ``canonical_id`` + strings in descending similarity order.""" + if not rows: + return [] + matrix = np.array([r.embedding for r in rows], dtype=np.float32) + pairs = top_k_cosine( + query=normalised_query, + matrix=normalise_matrix(matrix), + k=len(rows), + ) + return [rows[idx].canonical_id for idx, _score in pairs] + + +# --------------------------------------------------------------------------- +# Pending refresh unit (write side) +# --------------------------------------------------------------------------- + + +class _PendingRefresh: + """One unit of work — rendered text needing an embedding.""" + + __slots__ = ("canonical_id", "entity_kind", "text", "content_hash") + + canonical_id: str + entity_kind: EntityKind + text: str + content_hash: str + + def __init__( + self, *, + canonical_id: str, + entity_kind: EntityKind, + text: str, + ) -> None: + self.canonical_id = canonical_id + self.entity_kind = entity_kind + self.text = text + self.content_hash = _sha256(text) + + +# --------------------------------------------------------------------------- +# Retriever +# --------------------------------------------------------------------------- + + +class EmbeddingRetriever(Retriever): + """Cosine-similarity retriever over a SQLite-sidecar embedding + store, plus the refresh pipeline that keeps the sidecar in step + with model / memory edits. + + The delete hooks intentionally default to ABC no-op: + :class:`StorageBackend` cascade-deletes embedding rows + transactionally with the model / datasource / memory row delete, + so the retriever has nothing to do on delete this PR.""" + + name = "embeddings" + + def __init__( + self, + *, + storage: StorageBackend, + model_name: str | None = None, + ) -> None: + self._storage = storage + self._model_name = model_name or current_model() + + @property + def model_name(self) -> str: + return self._model_name + + # ------------------------------------------------------------------ + # Read side + # ------------------------------------------------------------------ + + async def fetch_corpus(self) -> list[Embedding]: + """Return every embedding row under the active model name.""" + return await self._storage.list_embeddings( + embedding_model_name=self._model_name, + ) + + async def embed_question(self, question: str) -> list[float] | None: + """Embed a search query string. ``None`` when the channel is + unavailable or the call fails.""" + return await embedding_client.embed_query( + question, model=self._model_name, + ) + + async def retrieve( + self, + *, + query_entities: list[str], + question: str | None, + all_memories: list[Memory], + valid_canonicals: set, + corpus: Corpus | None, + datasource: str | None, + ) -> RetrievalResult: + """Run cosine over the embedding corpus, returning BOTH memory + and entity rankings from a single ``fetch_corpus`` + + ``embed_question`` + dim-check (Codex Finding 1). + + Skipped (with a warning) when: + + * ``question`` is blank, + * the ``advanced_search`` extra is not installed, + * the active model has no embedding rows in storage, + * the query embedding call fails, + * dim mismatch between query vec and corpus. + """ + if corpus is None or not question or not question.strip(): + return RetrievalResult() + if not embedding_client.is_available(): + return RetrievalResult(warnings=[ + "embedding channel skipped: `advanced_search` extra not " + "installed or no API key configured for the active " + "embedding model.", + ]) + + rows = await self.fetch_corpus() + if datasource is not None: + eligible_memory_canonicals = { + f"{_MEMORY_PREFIX}{m.id}" for m in all_memories + } + rows = _filter_embedding_corpus_by_datasource( + rows, + datasource=datasource, + eligible_memory_canonicals=eligible_memory_canonicals, + ) + # Drop sidecar rows that don't correspond to anything in the + # live tantivy corpus (DEV-1414). + live_canonicals = corpus.canonical_to_kind.keys() + rows = [r for r in rows if r.canonical_id in live_canonicals] + if not rows: + return RetrievalResult(warnings=[ + f"embedding channel skipped: no embedding rows for model " + f"{self._model_name!r}. Run `slayer ingest` to populate.", + ]) + + # Inline imports: ``numpy`` and ``slayer.embeddings.ranker`` + # require the optional ``advanced_search`` extra. When the extra + # is not installed, we fall through to a soft warning instead of + # raising at module import time so the rest of slayer keeps + # working without the extra. + try: + import numpy as np + from slayer.embeddings.ranker import ( + normalise, + normalise_matrix, + top_k_cosine, + ) + except ImportError: + return RetrievalResult(warnings=[ + "embedding channel skipped: numpy not installed " + "(reinstall with the `advanced_search` extra).", + ]) + + query_vec = await self.embed_question(question or "") + if query_vec is None: + return RetrievalResult(warnings=[ + "embedding channel skipped: query embedding failed.", + ]) + if len(rows[0].embedding) != len(query_vec): + return RetrievalResult(warnings=[ + f"embedding channel skipped: dim mismatch " + f"(query={len(query_vec)}, corpus={len(rows[0].embedding)}). " + f"Re-run `slayer ingest` to refresh embeddings against " + f"the current model.", + ]) + + memory_rows = [r for r in rows if r.entity_kind == "memory"] + entity_rows = [r for r in rows if r.entity_kind != "memory"] + normalised_query = normalise(query_vec) + ranked_memory_canonicals = _rank_embedding_kind( + rows=memory_rows, + normalised_query=normalised_query, + np=np, + normalise_matrix=normalise_matrix, + top_k_cosine=top_k_cosine, + ) + memory_ranking: list[str] = [] + for canonical in ranked_memory_canonicals: + memory_id = _memory_id_from_canonical(canonical) + if memory_id is not None: + memory_ranking.append(memory_id) + entity_ranking = _rank_embedding_kind( + rows=entity_rows, + normalised_query=normalised_query, + np=np, + normalise_matrix=normalise_matrix, + top_k_cosine=top_k_cosine, + ) + return RetrievalResult( + memory_ranking=memory_ranking, + entity_ranking=entity_ranking, + ) + + # ------------------------------------------------------------------ + # Write side — create / refresh hooks + # ------------------------------------------------------------------ + + async def upsert_memory(self, memory: Memory) -> list[str]: + """Refresh the embedding for a single memory. Returns warning + strings (empty on success or hash-skip). Stays silent on the + write path when the channel is unavailable — that's "feature + not configured", not a runtime failure (the search-side surface + emits one user-visible warning into ``SearchResponse.warnings`` + on the next query).""" + if not embedding_client.is_available(): + return [] + pending = _PendingRefresh( + canonical_id=_memory_canonical_id(memory.id), + entity_kind="memory", + text=render_memory_text_for_embedding(memory=memory), + ) + return await self._apply_pending([pending]) + + async def refresh_datasource( + self, + *, + name: str, + models: list[SlayerModel], + description: str | None = None, + ) -> list[str]: + """Refresh the embedding for one datasource doc. + + Routes through the unified :func:`render_datasource_pair` so the + visibility filter is applied in exactly one place (DEV-1513). + + DEV-1549: ``description`` (DatasourceConfig.description) is + woven into the rendered text so the embedding channel can match + terms that live only in the description, in parity with the + lexical channel. + """ + if not embedding_client.is_available(): + return [] + pair = render_datasource_pair( + name=name, models=models, description=description, + ) + pending = _PendingRefresh( + canonical_id=pair.canonical_id, + entity_kind="datasource", + text=pair.text, + ) + return await self._apply_pending([pending]) + + async def refresh_model_subtree(self, model: SlayerModel) -> list[str]: + """Refresh the model doc + every visible column + named measures + + custom aggregations in a single batch call. + + Routes through the unified :func:`collect_model_entity_pairs` so + the "what counts as an indexable entity" filter rules (hidden + model -> empty; hidden column skipped; unnamed measure skipped) + live in exactly one place (DEV-1513). + """ + if not embedding_client.is_available(): + return [] + pending: list[_PendingRefresh] = [ + _PendingRefresh( + canonical_id=re.canonical_id, + entity_kind=re.kind, # type: ignore[arg-type] + text=re.text, + ) + for re in collect_model_entity_pairs(model=model) + ] + return await self._apply_pending(pending) + + # ------------------------------------------------------------------ + # Internals + # ------------------------------------------------------------------ + + async def _apply_pending( + self, pending: list[_PendingRefresh], + ) -> list[str]: + """Hash-skip, batch-embed, and persist. Returns warning strings. + + DEV-1405: two batched storage round-trips per call — one + ``get_embeddings_for_canonical_ids`` for the hash-skip filter, + one ``save_embeddings`` for the persist step. + """ + if not pending: + return [] + stale, fresh_count = await self._filter_stale(pending) + if not stale: + return [] + texts = [p.text for p in stale] + vectors = await embed_batch(texts, model=self._model_name) + warnings: list[str] = [] + rows: list[Embedding] = [] + for p, vec in zip(stale, vectors): + if vec is None: + warnings.append( + f"embedding refresh failed for {p.canonical_id}; " + f"skipped (search will still find this entity via " + f"tantivy + BM25)." + ) + continue + rows.append(Embedding( + canonical_id=p.canonical_id, + embedding_model_name=self._model_name, + entity_kind=p.entity_kind, + content_hash=p.content_hash, + embedding=vec, + )) + if rows: + try: + await self._storage.save_embeddings(rows) + except Exception as exc: # NOSONAR(S112) — best-effort persistence + canonical_ids = ", ".join(r.canonical_id for r in rows) + warnings.append( + f"embedding batch persist failed for " + f"{len(rows)} row(s) [{canonical_ids}]: {exc}" + ) + _log.debug( + "EmbeddingRetriever: refreshed=%d stale=%d total=%d warnings=%d", + fresh_count, len(stale), len(pending), len(warnings), + ) + return warnings + + async def _filter_stale( + self, pending: list[_PendingRefresh], + ) -> tuple[list[_PendingRefresh], int]: + """Drop pending entries whose stored content_hash already + matches. Returns ``(stale_entries, fresh_skipped_count)``. + DEV-1405: one batched ``get_embeddings_for_canonical_ids`` call + replaces the previous M-iteration point-read loop.""" + existing = await self._storage.get_embeddings_for_canonical_ids( + canonical_ids=[p.canonical_id for p in pending], + embedding_model_name=self._model_name, + ) + stale: list[_PendingRefresh] = [] + fresh = 0 + for p in pending: + match = existing.get(p.canonical_id) + if match is not None and match.content_hash == p.content_hash: + fresh += 1 + continue + stale.append(p) + return stale, fresh diff --git a/slayer/search/retrievers/tantivy.py b/slayer/search/retrievers/tantivy.py new file mode 100644 index 00000000..f4efed4f --- /dev/null +++ b/slayer/search/retrievers/tantivy.py @@ -0,0 +1,94 @@ +"""Tantivy retriever — in-memory full-text channel (DEV-1514). + +Ports the body of the former ``SearchService._run_channel_2`` into a +standalone :class:`~slayer.search.retriever.Retriever`. The two +kind-filtered queries (memory + entity) run SEQUENTIALLY inside one +:meth:`retrieve` call (Codex Finding 2) so neither competes with the +other against the same in-memory ``tantivy.Index``. + +This PR keeps the index in-memory (rebuilt per search call by +:class:`SearchService`). A future PR can override the write hooks to +persist segments on disk and mutate them in step with the embedding +retriever — that change requires no facade modification. +""" + +from __future__ import annotations + + +from slayer.memories.models import Memory +from slayer.search.index import Corpus, IndexHit, search_index +from slayer.search.retriever import RetrievalResult, Retriever + + +def _count_corpus_kinds(corpus: Corpus) -> tuple[int, int]: + """Return ``(memory_count, entity_count)`` for a built corpus. + + Used to pass ``limit = full per-kind corpus size`` to each + kind-filtered tantivy query so neither kind's ranking is truncated + (DEV-1414).""" + memory_count = 0 + entity_count = 0 + for kind in corpus.canonical_to_kind.values(): + if kind == "memory": + memory_count += 1 + else: + entity_count += 1 + return memory_count, entity_count + + +class TantivyRetriever(Retriever): + """Two kind-filtered tantivy queries per retrieve call. Returns + both memory + entity rankings and populates ``text_by_id`` from + the memory hits.""" + + name = "tantivy" + + async def retrieve( + self, + *, + query_entities: list[str], + question: str | None, + all_memories: list[Memory], + valid_canonicals: set, + corpus: Corpus | None, + datasource: str | None, + ) -> RetrievalResult: + if corpus is None or not question or not question.strip(): + return RetrievalResult() + + memory_count, entity_count = _count_corpus_kinds(corpus) + memory_hits: list[IndexHit] = ( + search_index( + index=corpus.index, + question=question, + limit=memory_count, + kind_filter="memory", + ) + if memory_count > 0 + else [] + ) + entity_hits: list[IndexHit] = ( + search_index( + index=corpus.index, + question=question, + limit=entity_count, + exclude_kind="memory", + ) + if entity_count > 0 + else [] + ) + + memory_ranking: list[str] = [] + text_by_id: dict[str, str] = {} + for hit in memory_hits: + if hit.memory_id is None: + continue + memory_ranking.append(hit.memory_id) + text_by_id[hit.memory_id] = hit.text + entity_ranking = [h.id for h in entity_hits] + + return RetrievalResult( + memory_ranking=memory_ranking, + text_by_id=text_by_id, + entity_ranking=entity_ranking, + ) diff --git a/slayer/search/rrf.py b/slayer/search/rrf.py index f4dbe666..b243396a 100644 --- a/slayer/search/rrf.py +++ b/slayer/search/rrf.py @@ -13,14 +13,15 @@ from __future__ import annotations -from typing import Hashable, List, TypeVar +from typing import TypeVar +from collections.abc import Hashable K = TypeVar("K", bound=Hashable) def rrf_fuse( *, - rankings: List[List[K]], + rankings: list[list[K]], k: int = 60, ) -> dict[K, float]: """Fuse multiple ranked lists into one score map. diff --git a/slayer/search/service.py b/slayer/search/service.py index d08161e2..ed22da0e 100644 --- a/slayer/search/service.py +++ b/slayer/search/service.py @@ -1,69 +1,111 @@ -"""SearchService — three-channel + RRF orchestrator (DEV-1375 / DEV-1386). - -* **Channel 1** — entity-overlap BM25 over memories - (``slayer.memories.ranker.bm25_rank``). Skipped when neither - ``entities`` nor ``query`` is supplied. Contributes only to the memory - ranking. -* **Channel 2** — tantivy full-text. Skipped when ``question`` is - empty. Runs as TWO kind-filtered queries per call (DEV-1414): one - with ``kind_filter="memory"`` for the memory ranking, one with - ``exclude_kind="memory"`` for the entity ranking. Each query ranks - the full per-kind subset of the corpus — no over-fetch truncation. -* **Channel 3** — dense embedding similarity (DEV-1386). Skipped when - ``question`` is empty, when the ``embedding_search`` extra is not - installed, when the query embedding call fails, or when there are - no embedding rows for the active model name. The persisted embedding - rows are partitioned by ``entity_kind`` (DEV-1414): memory rows feed - the memory ranking, non-memory rows feed the entity ranking. Each - partition is ranked in full. - -Memory rankings from every active channel are fused via RRF -(``k = 60``). Entity rankings from channels 2 and 3 are fused the same -way. Channel 1 does not contribute to entity ranking (it operates on -memory entity tags, not on entity docs). - -Per-bucket invariance (DEV-1414): because each channel produces a full +"""SearchService — facade orchestrator over a list of +:class:`~slayer.search.retriever.Retriever` instances (DEV-1514) with a +unified flat-results interface (DEV-1532) and an optional graph-backed +Cypher pre-filter (DEV-1464). + +The orchestrator owns: + +* Input validation (``max_results`` >= 1, ``datasource`` known). +* Lenient input-entity resolution (per-token failures → warnings). +* Optional ``cypher_filter`` pre-filter — when set, the result of + the openCypher / naive-fallback query becomes a hard allowlist + applied across every channel (DEV-1464). +* Recency fallback when no channel is active. +* One-shot ``all_memories`` fetch (datasource-filtered, then + cypher_filter-narrowed when applicable). +* One-shot ``valid_canonicals`` set build (datasources + persisted + model identities + memory canonical ids). +* One-shot ``corpus`` build when ``question`` is active. +* Parallel fan-out across retrievers via ``asyncio.gather``. +* Channel-1 named-entity surfacing (DEV-1513): every user-supplied + canonical entity ref is contributed to the entity ranking as itself + (subject to datasource / hidden / missing / cypher_filter checks), + so an explicit ``entities=["."]`` surfaces that entity + at the top of the results even without a fuzzy ``question``. +* RRF fusion (``k=60``) over memory + entity rankings, collapsed into + a single flat ``results: List[SearchHit]`` list capped at + ``max_results`` (DEV-1532). ``kind`` distinguishes memories from + entity hits; ``query`` is populated for query-bearing memory hits. +* Post-fusion cypher_filter / kind_filter narrowing (DEV-1464) — + candidates outside the allowlist (full graph path) or outside the + naive kind list (fallback path) are dropped before the + ``max_results`` cap, so the cap always counts surviving items only. +* Post-fusion column-hit refresh (DEV-1516) — categorical column + hits with stale ``sampled_values`` are re-profiled inline via + :func:`slayer.engine.profiling.ensure_column_sample_fresh` so the + surfaced text reflects live values. Per-model writes serialise + (storage's ``update_column_sampled`` is a model-level + read-modify-write); cross-model writes parallelise via + ``asyncio.gather``. Silently no-op when ``engine`` is None. +* Stale-``Memory.query`` warnings. + +Each registered retriever runs ONCE per search call, returning a +combined :class:`RetrievalResult` with both memory and entity rankings. +The default retriever list is ``[BM25Retriever, TantivyRetriever, +EmbeddingRetriever]``; callers may inject any list via the +``retrievers=`` kwarg. + +Ranking stability (DEV-1414): because each retriever produces a full per-kind ranking — never truncated by a shared candidate-pool budget — -the membership and order of every output bucket (``memories``, -``example_queries``, ``entities``) is a pure function of the corpus, -the question, the datasource filter, and that bucket's own cap. Varying -the other two caps cannot move ids in or out of the returned list nor -reorder it. - -Empty input (no entities, no query, no question) falls back to recency: -newest ``max_memories`` learning-only memories + newest -``max_example_queries`` query-bearing memories, with a warning. +the relative order of any subset of the flat list is stable. Changing +only ``max_results`` never reorders existing entries nor causes an +entry to appear or disappear unless the cap boundary moves past it. + +Write-side (``upsert_memory`` / ``refresh_model_subtree`` / +``refresh_datasource``): fans the call out to every registered +retriever, isolating per-retriever exceptions as prefixed warnings so +the fan-out always reaches the last retriever. Warning aggregation is +deterministic — declared retriever order, not gather completion order. + +This module deliberately does NOT expose ``delete_*`` public methods: +:class:`StorageBackend` owns embedding-row cascade transactionally +with the row delete; adding retriever fan-out would create a second +deletion path on top. The :class:`Retriever` ABC defines the delete +hooks for future use (persistent tantivy will override them). """ from __future__ import annotations -from typing import Dict, List, Optional, Set, Tuple, Union +import asyncio +import logging +from typing import Any from pydantic import BaseModel, Field from slayer.core.errors import AmbiguousModelError, EntityResolutionError from slayer.core.models import SlayerModel from slayer.core.query import SlayerQuery -from slayer.embeddings import client as embedding_client -from slayer.embeddings.models import Embedding +from slayer.engine.profiling import ensure_column_sample_fresh +from slayer.engine.query_engine import SlayerQueryEngine from slayer.memories.models import MEMORY_CANONICAL_PREFIX as _MEMORY_PREFIX from slayer.memories.models import Memory -from slayer.memories.ranker import bm25_rank from slayer.memories.resolver import ( canonical_id_rooted_at, extract_entities_from_query, resolve_entity, ) -from slayer.search.index import ( - Corpus, - IndexHit, - build_in_memory_corpus, - search_index, +from slayer.search import graph as _search_graph +from slayer.search.cypher_naive import parse_naive_label_filter as _parse_naive_cypher +from slayer.search.index import Corpus, build_in_memory_corpus +from slayer.search.render import ( + collect_model_entity_pairs, + compact_description_from_learning, + render_column_text, + render_datasource_pair, +) +from slayer.search.retriever import RetrievalResult, Retriever +from slayer.search.retrievers import ( + BM25Retriever, + EmbeddingRetriever, + TantivyRetriever, ) from slayer.search.rrf import rrf_fuse from slayer.storage.base import StorageBackend +logger = logging.getLogger(__name__) + + _RRF_K = 60 @@ -72,60 +114,88 @@ # --------------------------------------------------------------------------- -class MemoryHit(BaseModel): - """A learning-only memory result (``Memory.query is None``). ``id`` is - the string memory id (suitable for ``forget_memory(id=hit.id)``). - ``score`` is always the Reciprocal-Rank-Fusion score +class SearchHit(BaseModel): + """A unified search result (DEV-1532). ``kind`` is ``"memory"`` for + memories, or the entity kind string (``"datasource"``, ``"model"``, + ``"column"``, ``"measure"``, ``"aggregation"``) for entity hits. + + ``id`` is the raw storage id for memories (suitable for + ``forget_memory(id=hit.id)``) and the canonical entity string for + entity hits. ``score`` is always the Reciprocal-Rank-Fusion score (``Σ 1 / (k + rank)``, ``k=60``); even single-channel searches go through RRF, so the value is comparable across channels but is not - directly the raw BM25 / tantivy / cosine score.""" + directly the raw BM25 / tantivy / cosine score. + + ``matched_entities`` and ``query`` are populated for memory hits + only; entity hits carry empty / ``None`` defaults. + DEV-1549: ``description`` carries a compact preview. For memory + hits in compact mode it is ``Memory.description`` (or a + first-paragraph fallback computed from ``learning``); for entity + hits in any mode it is the entity's structured ``description`` + field. Under compact mode ``text`` is left empty for both kinds. + """ + + kind: str id: str score: float text: str - matched_entities: List[str] = Field(default_factory=list) + description: str | None = None + matched_entities: list[str] = Field(default_factory=list) + query: SlayerQuery | None = None -class ExampleQueryHit(BaseModel): - """A query-bearing memory result (``Memory.query`` is set). Same id / - score / text shape as ``MemoryHit`` but always carries the attached - ``SlayerQuery``. Surfaces in ``SearchResponse.example_queries`` — - bulky reference material, capped independently from learning-only - memories so it cannot crowd them out.""" +# --------------------------------------------------------------------------- +# Lookup result for named-entity surfacing (DEV-1513) +# --------------------------------------------------------------------------- - id: str - score: float + +class LookupFound(BaseModel): + """``_lookup_named_entity`` succeeded; carries ``(kind, text, + description)``. DEV-1549: ``description`` is the entity's + structured description field (``None`` when absent), surfaced as + ``SearchHit.description`` under compact mode.""" + + kind: str text: str - matched_entities: List[str] = Field(default_factory=list) - query: SlayerQuery + description: str | None = None -class EntityHit(BaseModel): - """An entity result. ``id`` is the canonical entity string - (``""``, ``"."``, or ``".."``). - ``score`` is the RRF-fused score across channels 2 and 3 (or the - single-channel raw score when only one channel contributed).""" +class LookupHidden(BaseModel): + """The canonical resolved but is gated by a ``hidden`` flag (on the + model or on the column). ``reason`` is a short human-readable hint + used to compose the caller-facing warning.""" - id: str - kind: str # "datasource" | "model" | "column" | "measure" | "aggregation" - score: float - text: str + reason: str + + +class LookupMissing(BaseModel): + """The canonical resolved at ``_resolve_inputs`` time but the + underlying datasource / model / leaf is no longer present at lookup + time (race between resolve and lookup, or the entity was deleted).""" + + pass + + +LookupResult = LookupFound | LookupHidden | LookupMissing class SearchResponse(BaseModel): - memories: List[MemoryHit] = Field(default_factory=list) - example_queries: List[ExampleQueryHit] = Field(default_factory=list) - entities: List[EntityHit] = Field(default_factory=list) - resolved_input_entities: List[str] = Field(default_factory=list) - warnings: List[str] = Field(default_factory=list) + """Unified search response (DEV-1532). ``results`` is a single flat + list ranked by RRF score; consumers partition by ``kind`` (or by + ``query is None`` for the memory subset) at the call site.""" + + results: list[SearchHit] = Field(default_factory=list) + resolved_input_entities: list[str] = Field(default_factory=list) + warnings: list[str] = Field(default_factory=list) # --------------------------------------------------------------------------- -# Service +# Helpers # --------------------------------------------------------------------------- -def _coerce_query(query: Union[SlayerQuery, dict]) -> SlayerQuery: +def _coerce_query(query: SlayerQuery | dict) -> SlayerQuery: if isinstance(query, SlayerQuery): return query if isinstance(query, dict): @@ -135,9 +205,9 @@ def _coerce_query(query: Union[SlayerQuery, dict]) -> SlayerQuery: ) -def _dedup(items: List[str]) -> List[str]: - seen: Set[str] = set() - out: List[str] = [] +def _dedup(items: list[str]) -> list[str]: + seen: set[str] = set() + out: list[str] = [] for x in items: if x not in seen: seen.add(x) @@ -145,19 +215,38 @@ def _dedup(items: List[str]) -> List[str]: return out +def _filter_memories_by_datasource( + *, memories: list[Memory], datasource: str | None, +) -> list[Memory]: + """DEV-1409: keep memories with at least one entity rooted at + ``datasource``. ``datasource=None`` is a no-op identity filter so + callers can call this unconditionally.""" + if datasource is None: + return memories + return [ + m for m in memories + if any( + canonical_id_rooted_at(canonical_id=e, datasource=datasource) + for e in m.entities + ) + ] + + +def _collect_memory_canonicals(memories: list[Memory]) -> set: + return {f"{_MEMORY_PREFIX}{m.id}" for m in memories} + + def _backfill_memory_by_id( *, memory_by_id: dict, all_memories_by_id: "dict[str, Memory]", mem_ids, ) -> None: - """For each id in ``mem_ids`` not already in ``memory_by_id``, look it - up in ``all_memories_by_id`` and insert it. Mutates ``memory_by_id``. - - Takes a precomputed id→Memory dict (not the raw list) so per-call - backfill stays O(N) instead of O(N²) when every channel returns the - full memory corpus (DEV-1414). - """ + """For each id in ``mem_ids`` not already in ``memory_by_id``, + look it up in ``all_memories_by_id`` and insert it. Mutates + ``memory_by_id``. Takes a precomputed id→Memory dict (not the raw + list) so per-call backfill stays O(N) instead of O(N²) when every + retriever returns the full memory corpus (DEV-1414).""" for mem_id in mem_ids: if mem_id in memory_by_id: continue @@ -168,425 +257,930 @@ def _backfill_memory_by_id( def _build_memory_hit( *, - mem: "Memory", + mem: Memory, memory_id: str, score: float, - index_hits_by_memory_id: dict, - canonical_input_entities: List[str], - valid_canonicals: Optional[set] = None, -) -> Union["MemoryHit", "ExampleQueryHit"]: - """Build the appropriate hit type for ``mem``: ``MemoryHit`` for - learning-only memories (``query is None``), ``ExampleQueryHit`` for - query-bearing ones. ``text`` falls back to ``mem.learning`` when the - memory wasn't reached via tantivy. + text_by_id: dict[str, str], + canonical_input_entities: list[str], + valid_canonicals: set | None = None, + compact: bool = True, +) -> SearchHit: + """Build a SearchHit for a memory (DEV-1532 unified shape). + + ``text`` falls back to ``mem.learning`` when no retriever supplied + a hit text for this memory. DEV-1428: ``matched_entities`` is computed against the LIVE canonical set when ``valid_canonicals`` is supplied, so stale tags - do not surface to the agent.""" + do not surface to the agent. + + DEV-1513: every memory has an implicit ``memory:`` + self-reference; it appears in ``matched_entities`` only when the + user explicitly named that ref (so the surfaced memory honestly + shows the reason it was returned). + + DEV-1549 (compact): + * ``compact=True`` → ``description`` = ``mem.description`` if set, + else the first-paragraph fallback; ``text = ""``. + * ``compact=False`` → ``description`` = ``mem.description`` (or + ``None``; no fallback); ``text`` = full learning rendering. + """ if valid_canonicals is not None: live_entities = [e for e in mem.entities if e in valid_canonicals] else: live_entities = list(mem.entities) + self_ref = f"{_MEMORY_PREFIX}{memory_id}" + if self_ref not in live_entities: + live_entities.append(self_ref) wanted_set = set(canonical_input_entities) matched = sorted(wanted_set & set(live_entities)) if wanted_set else [] - text = ( - index_hits_by_memory_id[memory_id].text - if memory_id in index_hits_by_memory_id - else mem.learning - ) - if mem.query is None: - return MemoryHit( - id=memory_id, score=score, text=text, matched_entities=matched, + if compact: + description = ( + mem.description + if mem.description + else compact_description_from_learning(mem.learning) ) - return ExampleQueryHit( - id=memory_id, score=score, text=text, - matched_entities=matched, query=mem.query, + text = "" + else: + description = mem.description + text = text_by_id.get(memory_id) or mem.learning + return SearchHit( + kind="memory", + id=memory_id, + score=score, + text=text, + description=description, + matched_entities=matched, + query=mem.query, ) -def _filter_memories_entities( - memories: List["Memory"], valid_canonicals: set, -) -> List["Memory"]: - """Return shallow copies of ``memories`` whose ``entities`` lists are - filtered down to ``valid_canonicals`` only. Used to feed BM25 a - stale-free corpus without writing back to storage (DEV-1428).""" - out: List[Memory] = [] - for m in memories: - live = [e for e in m.entities if e in valid_canonicals] - if live == m.entities: - out.append(m) - else: - out.append(m.model_copy(update={"entities": live})) - return out +def _resolve_entity_hit_kind_text( + *, + canonical: str, + corpus: Corpus | None, + named_kind_text: dict[str, tuple[str, str, str | None]] | None, +) -> tuple[str, str, str | None] | None: + """DEV-1513 / DEV-1549: resolve one canonical's + ``(kind, text, description)`` triple for an entity hit. Prefers the + corpus (channels 2/3 already built it); falls back to the channel-1 + ``named_kind_text`` lookup (used on pure-named calls with no + corpus). Returns ``None`` when neither source carries the canonical. + """ + if corpus is not None: + kind = corpus.canonical_to_kind.get(canonical) + text = corpus.canonical_to_text.get(canonical) + if kind is not None and text is not None: + description = corpus.canonical_to_description.get(canonical) + return kind, text, description + if named_kind_text is not None: + triple = named_kind_text.get(canonical) + if triple is not None: + return triple + return None -def _fuse_memory_hits( +def _build_hit_from_fused_key( *, - rankings: List[List[str]], + key: str, + score: float, memory_by_id: dict, - index_hits_by_memory_id: dict, - canonical_input_entities: List[str], - max_memories: int, - max_example_queries: int, - valid_canonicals: Optional[set] = None, -) -> Tuple[List["MemoryHit"], List["ExampleQueryHit"]]: - """RRF-fuse the supplied memory rankings and partition into - learning-only (``MemoryHit``) vs query-bearing (``ExampleQueryHit``) - lists, each capped independently. Empty inner rankings are filtered - out so single-channel results still flow through RRF normalisation.""" - non_empty = [r for r in rankings if r] - fused = rrf_fuse(rankings=non_empty, k=_RRF_K) if non_empty else {} - fused_sorted = sorted(fused.items(), key=lambda kv: kv[1], reverse=True) - - learnings: List[MemoryHit] = [] - examples: List[ExampleQueryHit] = [] - for memory_id, score in fused_sorted: + text_by_id: dict[str, str], + canonical_input_entities: list[str], + corpus: Corpus | None, + named_kind_text: dict[str, tuple[str, str, str | None]] | None, + valid_canonicals: set | None, + candidate_ids: frozenset[str] | None, + kind_filter: set[str] | None, + compact: bool = True, +) -> SearchHit | None: + """Build one SearchHit from a fused (key, score) pair, or return None + to skip. Applies the DEV-1464 cypher_filter (candidate_ids allowlist + for the full graph path; kind_filter for the naive fallback) BEFORE + materialising the hit, so the upstream cap counts surviving items + only. + + DEV-1549: ``compact`` flips memory + entity hit rendering between + description-only and description+full-text shapes. + """ + if key.startswith(_MEMORY_PREFIX): + memory_id = key[len(_MEMORY_PREFIX):] + if candidate_ids is not None and key not in candidate_ids: + return None + if kind_filter is not None and "memory" not in kind_filter: + return None mem = memory_by_id.get(memory_id) if mem is None: - continue - hit = _build_memory_hit( + return None + return _build_memory_hit( mem=mem, memory_id=memory_id, score=score, - index_hits_by_memory_id=index_hits_by_memory_id, + text_by_id=text_by_id, canonical_input_entities=canonical_input_entities, valid_canonicals=valid_canonicals, + compact=compact, ) - if isinstance(hit, MemoryHit) and len(learnings) < max_memories: - learnings.append(hit) - elif isinstance(hit, ExampleQueryHit) and len(examples) < max_example_queries: - examples.append(hit) - if ( - len(learnings) >= max_memories - and len(examples) >= max_example_queries - ): - break - return learnings, examples - - -def _filter_memories_by_datasource( - memories: List["Memory"], datasource: Optional[str], -) -> List["Memory"]: - """DEV-1409: keep memories with at least one entity rooted at - ``datasource``. ``datasource=None`` is a no-op identity filter so - callers can call this unconditionally.""" - if datasource is None: - return memories - return [ - m for m in memories - if any( - canonical_id_rooted_at(canonical_id=e, datasource=datasource) - for e in m.entities - ) - ] + # Entity key. + if candidate_ids is not None and key not in candidate_ids: + return None + resolved = _resolve_entity_hit_kind_text( + canonical=key, + corpus=corpus, + named_kind_text=named_kind_text, + ) + if resolved is None: + return None + kind, text, description = resolved + if kind_filter is not None and kind not in kind_filter: + return None + return SearchHit( + id=key, + kind=kind, + score=score, + text="" if compact else text, + description=description, + ) -def _filter_embedding_corpus_by_datasource( - rows: List["Embedding"], +def _fuse_all_hits( *, - datasource: str, - eligible_memory_canonicals: Set[str], -) -> List["Embedding"]: - """DEV-1409: narrow the embedding corpus to rows that survive a - datasource filter. Memory rows (``entity_kind == 'memory'``) must - appear in the supplied ``eligible_memory_canonicals`` set (already - datasource-filtered upstream); entity rows must be rooted at - ``datasource`` per the dotted-namespace rule.""" - return [ - r for r in rows - if ( - (r.entity_kind == "memory" - and r.canonical_id in eligible_memory_canonicals) - or (r.entity_kind != "memory" - and canonical_id_rooted_at( - canonical_id=r.canonical_id, datasource=datasource, - )) - ) + memory_rankings: list[list[str]], + entity_rankings: list[list[str]], + memory_by_id: dict, + text_by_id: dict[str, str], + canonical_input_entities: list[str], + corpus: Corpus | None, + named_kind_text: dict[str, tuple[str, str, str | None]] | None, + max_results: int, + valid_canonicals: set | None = None, + candidate_ids: frozenset[str] | None = None, + kind_filter: set[str] | None = None, + compact: bool = True, +) -> list[SearchHit]: + """RRF-fuse memory and entity rankings into a single flat list + (DEV-1532). Memory IDs are prefixed with the canonical memory prefix + so the unified pool contains no key collisions. + + DEV-1464: ``candidate_ids`` (full-graph allowlist) and + ``kind_filter`` (naive-fallback kind allowlist) are applied BEFORE + the ``max_results`` cap so the cap always counts surviving items + only — a wrong implementation that filters AFTER capping would + silently drop matching results when an unrelated hit happens to + out-rank them.""" + prefixed_memory_rankings = [ + [f"{_MEMORY_PREFIX}{mid}" for mid in ranking] + for ranking in memory_rankings ] + all_rankings = prefixed_memory_rankings + entity_rankings + non_empty = [r for r in all_rankings if r] + fused = rrf_fuse(rankings=non_empty, k=_RRF_K) if non_empty else {} + fused_sorted = sorted(fused.items(), key=lambda kv: kv[1], reverse=True) + results: list[SearchHit] = [] + for key, score in fused_sorted: + hit = _build_hit_from_fused_key( + key=key, + score=score, + memory_by_id=memory_by_id, + text_by_id=text_by_id, + canonical_input_entities=canonical_input_entities, + corpus=corpus, + named_kind_text=named_kind_text, + valid_canonicals=valid_canonicals, + candidate_ids=candidate_ids, + kind_filter=kind_filter, + compact=compact, + ) + if hit is not None: + results.append(hit) + if len(results) >= max_results: + break + return results -def _count_corpus_kinds(corpus: Corpus) -> Tuple[int, int]: - """Return ``(memory_count, entity_count)`` for a built corpus. Used - by channel 2 to pass ``limit = full per-kind corpus size`` to each - kind-filtered tantivy query so neither kind's ranking is truncated - (DEV-1414).""" - memory_count = 0 - entity_count = 0 - for kind in corpus.canonical_to_kind.values(): - if kind == "memory": - memory_count += 1 - else: - entity_count += 1 - return memory_count, entity_count +def _merge_text_by_id_in_declaration_order( + results: list[RetrievalResult], +) -> dict[str, str]: + """Merge ``text_by_id`` across retriever results. First-non-empty + in retriever declaration order wins per memory id.""" + merged: dict[str, str] = {} + for result in results: + for mem_id, text in result.text_by_id.items(): + if mem_id not in merged and text: + merged[mem_id] = text + return merged -def _collect_memory_canonicals(memories: List["Memory"]) -> set: - """Return ``{"memory:" for m in memories}`` — pulled out so - ``_valid_canonical_set`` stays under the cognitive-complexity gate.""" - return {f"{_MEMORY_PREFIX}{m.id}" for m in memories} +# --------------------------------------------------------------------------- +# DEV-1513 / DEV-1464: named-entity surfacing helpers +# --------------------------------------------------------------------------- -def _memory_id_from_canonical(canonical_id: str) -> Optional[str]: - """Parse a memory row's canonical id back into the str memory id. - Returns ``None`` when the input is not a memory canonical id — i.e. - not exactly of the shape ``memory:``. DEV-1428 review: - a corrupted / stale embedding row carrying ``foo:bar`` would - otherwise be mis-mapped to a memory hit; the prefix gate keeps the - memory channel honest. - """ - if not canonical_id.startswith(_MEMORY_PREFIX): - return None - memory_id = canonical_id[len(_MEMORY_PREFIX):] - return memory_id or None +def _memory_id_off_datasource_warnings( + *, + canonical_input_entities: list[str], + live_memory_ids: set[str], + datasource: str | None, +) -> list[str]: + """DEV-1513: emit one warning per user-supplied ``memory:`` ref + whose memory was dropped by the datasource pre-filter (the memory + has no entities rooted at ``datasource``). Mirrors the entity-side + off-ds drop on the memory side. + + No-op when ``datasource`` is None (nothing was filtered out).""" + if datasource is None: + return [] + out: list[str] = [] + for canonical in canonical_input_entities: + if not canonical.startswith(_MEMORY_PREFIX): + continue + memory_id = canonical[len(_MEMORY_PREFIX):] + if memory_id and memory_id not in live_memory_ids: + out.append( + f"{canonical} is not rooted at datasource " + f"{datasource!r}; dropped." + ) + return out -def _rank_embedding_kind( +def _memory_id_cypher_filter_warnings( *, - rows: List["Embedding"], - normalised_query, - np, - normalise_matrix, - top_k_cosine, -) -> List[str]: - """Rank one kind of embedding rows by cosine similarity to the - pre-normalised query vector. Returns the rows' ``canonical_id`` - strings in descending similarity order. Empty input → empty list. - - Pulls the per-kind matrix build + cosine call out of - ``SearchService._run_channel_3`` so each kind's ranking is a single - line in the caller (DEV-1414 — keeps channel 3 below the - cognitive-complexity gate).""" - if not rows: - return [] - matrix = np.array([r.embedding for r in rows], dtype=np.float32) - pairs = top_k_cosine( - query=normalised_query, - matrix=normalise_matrix(matrix), - k=len(rows), + canonical_input_entities: list[str], + candidate_ids: frozenset[str], +) -> list[str]: + """DEV-1464: emit one warning per user-supplied ``memory:`` ref + that was excluded by the cypher_filter allowlist (the graph query + did not return that memory's canonical id).""" + return [ + f"{c!r} excluded by cypher_filter." + for c in canonical_input_entities + if c.startswith(_MEMORY_PREFIX) and c not in candidate_ids + ] + + +async def _lookup_bare_datasource_canonical( + *, ds: str, storage: StorageBackend, +) -> LookupResult: + """DEV-1513: bare ```` branch of ``_lookup_named_entity``. + Re-verifies the datasource still exists (it may have been deleted + between resolve and lookup) before rendering.""" + known = await storage.list_datasources() + if ds not in known: + return LookupMissing() + identities = await storage._list_all_model_identities() + models: list[SlayerModel] = [] + for ident_ds, name in identities: + if ident_ds != ds: + continue + m = await storage.get_model(name, data_source=ident_ds) + if m is not None: + models.append(m) + cfg = await storage.get_datasource(ds) + ds_description = cfg.description if cfg is not None else None + pair = render_datasource_pair( + name=ds, models=models, description=ds_description, + ) + return LookupFound( + kind=pair.kind, text=pair.text, description=pair.description, ) - return [rows[idx].canonical_id for idx, _score in pairs] -def _fuse_entity_hits( +async def _lookup_model_or_leaf_canonical( *, - rankings: List[List[str]], - corpus: Optional[Corpus], - max_entities: int, -) -> List[EntityHit]: - """RRF-fuse the entity rankings and look text/kind up from the corpus - map. Returns at most ``max_entities`` hits.""" - if corpus is None: - return [] - non_empty = [r for r in rankings if r] - fused = rrf_fuse(rankings=non_empty, k=_RRF_K) if non_empty else {} - fused_sorted = sorted(fused.items(), key=lambda kv: kv[1], reverse=True) - out: List[EntityHit] = [] - for canonical, score in fused_sorted: - if len(out) >= max_entities: - break + canonical: str, + ds: str, + model_name: str, + leaf: str | None, + storage: StorageBackend, +) -> LookupResult: + """DEV-1513: ``.`` and ``..`` branches of + ``_lookup_named_entity``. Returns ``Hidden`` for hidden model / + hidden column, ``Missing`` for "no such entity" (race between resolve + and lookup).""" + model = await storage.get_model(model_name, data_source=ds) + if model is None: + return LookupMissing() + if model.hidden: + return LookupHidden(reason="hidden model") + for re in collect_model_entity_pairs(model=model): + if re.canonical_id == canonical: + return LookupFound( + kind=re.kind, text=re.text, description=re.description, + ) + if leaf is not None: + for column in model.columns: + if column.name == leaf and column.hidden: + return LookupHidden(reason="hidden column") + return LookupMissing() + + +async def _lookup_named_entity( + *, + canonical: str, + storage: StorageBackend, + corpus: Corpus | None, +) -> LookupResult: + """Resolve a canonical id to its ``(kind, text, description)`` triple + for channel-1 named-entity surfacing (DEV-1513 / DEV-1549).""" + if corpus is not None: kind = corpus.canonical_to_kind.get(canonical) text = corpus.canonical_to_text.get(canonical) - if kind is None or text is None: + if kind is not None and text is not None: + return LookupFound( + kind=kind, text=text, + description=corpus.canonical_to_description.get(canonical), + ) + segments = canonical.split(".") + if len(segments) == 1: + return await _lookup_bare_datasource_canonical( + ds=segments[0], storage=storage, + ) + return await _lookup_model_or_leaf_canonical( + canonical=canonical, + ds=segments[0], + model_name=segments[1], + leaf=segments[2] if len(segments) >= 3 else None, + storage=storage, + ) + + +# --------------------------------------------------------------------------- +# DEV-1516 column-hit refresh helpers (adapted to flat SearchHit) +# --------------------------------------------------------------------------- + + +def _group_column_hits( + results: list[SearchHit], +) -> dict[tuple[str, str], list[tuple[int, SearchHit, str]]]: + """DEV-1516 helper: split a fused result list into per-model buckets + for the search-side sample-refresh hook. + + Walks ``results``, keeps only ``kind == "column"`` hits whose + canonical id parses as ``..`` (3 + segments), and groups them by ``(data_source, model_name)`` so the + caller can serialise writes within a model and parallelise across + models. Each member tuple is ``(original_hit_index, hit, + column_name)`` — the index is preserved so caller can splice + refreshed text back into the original list in place.""" + groups: dict[tuple[str, str], list[tuple[int, SearchHit, str]]] = {} + for idx, hit in enumerate(results): + if hit.kind != "column": continue - out.append(EntityHit( - id=canonical, kind=kind, score=score, text=text, - )) - return out + segments = hit.id.split(".") + if len(segments) != 3: + continue + data_source, model_name, column_name = segments + groups.setdefault((data_source, model_name), []).append( + (idx, hit, column_name) + ) + return groups + + +# --------------------------------------------------------------------------- +# Service +# --------------------------------------------------------------------------- class SearchService: - """Orchestrates the three retrieval channels + RRF fusion.""" + """Orchestrates the registered retrievers + RRF fusion.""" - def __init__(self, *, storage: StorageBackend) -> None: + def __init__( + self, + *, + storage: StorageBackend, + engine: SlayerQueryEngine | None = None, + retrievers: list[Retriever] | None = None, + ) -> None: + """DEV-1516: ``engine`` is optional so storage-only test contexts + keep working unchanged. When supplied, the post-fusion column-hit + hook auto-refreshes stale categorical columns via + :func:`ensure_column_sample_fresh` before rendering ``SearchHit.text``. + Without an engine the hook is a silent no-op.""" self._storage = storage + self._engine = engine + self._retrievers: list[Retriever] = ( + list(retrievers) if retrievers is not None + else self._default_retrievers(storage) + ) - async def _validate_datasource_known( - self, datasource: Optional[str], + @staticmethod + def _default_retrievers(storage: StorageBackend) -> list[Retriever]: + return [ + BM25Retriever(), + TantivyRetriever(), + EmbeddingRetriever(storage=storage), + ] + + @property + def retrievers(self) -> list[Retriever]: + return self._retrievers + + async def _refresh_stale_column_hits( + self, + *, + results: list[SearchHit], + compact: bool = True, + ) -> list[SearchHit]: + """DEV-1516 post-fusion column-hit refresh. + + Groups column hits by ``(data_source, model_name)`` and dispatches + each group to :meth:`_refresh_group_worker`. Per-model writes + serialise (storage's ``update_column_sampled`` is a model-level + read-modify-write); cross-model writes parallelise via + ``asyncio.gather``. Returns ``results`` with refreshed text + spliced in for each column hit whose helper call returned a + materially-updated column. + + DEV-1549 (Codex#3): under ``compact=True`` the refresh leaves + ``text=""`` and refreshes ``description`` so the column hit can + never resurrect the full render mid-search. + """ + assert self._engine is not None # caller-guarded + groups = _group_column_hits(results) + if not groups: + return results + refreshed_by_idx: dict[int, SearchHit] = {} + await asyncio.gather(*[ + self._refresh_group_worker( + ds_name=ds, model_name=model_name, + members=members, refreshed_by_idx=refreshed_by_idx, + compact=compact, + ) + for (ds, model_name), members in groups.items() + ]) + if not refreshed_by_idx: + return results + return [ + refreshed_by_idx.get(i, h) for i, h in enumerate(results) + ] + + async def _refresh_group_worker( + self, + *, + ds_name: str, + model_name: str, + members: list[tuple[int, SearchHit, str]], + refreshed_by_idx: dict[int, SearchHit], + compact: bool = True, ) -> None: - """DEV-1409: reject typos in ``datasource`` before any corpus - walk. One ``list_datasources()`` round-trip; both backends back - this with an indexed query so the cost is bounded.""" - if datasource is None: + """Refresh every column hit on one ``(data_source, model_name)`` + group sequentially (per-model serialisation). Loads the model + once, walks members, and writes refreshed hits into the shared + ``refreshed_by_idx`` buffer keyed by original hit index. + + DEV-1549: under ``compact=True`` only refresh + ``SearchHit.description``; leave ``text=""``. + """ + try: + model = await self._storage.get_model( + model_name, data_source=ds_name, + ) + except Exception as exc: # NOSONAR(S112) — best-effort + logger.warning( + "search refresh: failed to load model %s.%s: %s", + ds_name, model_name, exc, + ) return - known = sorted(await self._storage.list_datasources()) - if datasource not in known: - raise ValueError( - f"datasource {datasource!r} not found; known: {known}." + if model is None: + return + for idx, hit, column_name in members: + col = model.get_column(column_name) + if col is None: + continue + refreshed_col = await ensure_column_sample_fresh( + model=model, + column=col, + engine=self._engine, # type: ignore[arg-type] + storage=self._storage, ) + if refreshed_col is col: + # Helper returned the input — cache hit, ineligible, or + # any failure. Leave the hit text as-is. + continue + update: dict[str, Any] = { + "description": refreshed_col.description, + } + if not compact: + update["text"] = render_column_text( + model=model, column=refreshed_col, + ) + refreshed_by_idx[idx] = hit.model_copy(update=update) + + # ------------------------------------------------------------------ + # Read side — search() + # ------------------------------------------------------------------ - async def search( + async def search( # NOSONAR(S3776) — single orchestrator entry point; stages are linear and named self, *, - entities: Optional[List[str]] = None, - query: Optional[Union[SlayerQuery, dict]] = None, - question: Optional[str] = None, - datasource: Optional[str] = None, - max_memories: int = 5, - max_example_queries: int = 2, - max_entities: int = 5, + entities: list[str] | None = None, + query: SlayerQuery | dict | None = None, + question: str | None = None, + datasource: str | None = None, + cypher_filter: str | None = None, + max_results: int = 10, + compact: bool = True, ) -> SearchResponse: - if max_memories < 0: - raise ValueError(f"max_memories must be >= 0; got {max_memories}.") - if max_example_queries < 0: + if max_results < 1: raise ValueError( - f"max_example_queries must be >= 0; got {max_example_queries}." + f"max_results must be >= 1; got {max_results}." ) - if max_entities < 0: - raise ValueError(f"max_entities must be >= 0; got {max_entities}.") await self._validate_datasource_known(datasource) canonical_input_entities, warnings = await self._resolve_inputs( entities=entities, query=query, ) - channel_1_active = (entities is not None and len(entities) > 0) or query is not None + channel_1_active = ( + (entities is not None and len(entities) > 0) or query is not None + ) question_active = bool(question and question.strip()) - # Recency fallback for the all-empty case. + # DEV-1464: optional cypher_filter pre-filter. When the graph + # path runs and returns no ids, short-circuit to an empty result + # with a warning — every channel would otherwise return zero + # surviving hits and we'd burn a corpus build for nothing. + candidate_ids, kind_filter, early = await self._apply_cypher_filter( + cypher_filter=cypher_filter, + canonical_input_entities=canonical_input_entities, + warnings=warnings, + ) + if early is not None: + return early + # Naive kind_filter parity with graph path: warn when a named + # memory: ref would be excluded by the kind filter so the + # caller knows why it doesn't appear in results. + if kind_filter is not None and "memory" not in kind_filter: + for canonical in canonical_input_entities: + if canonical.startswith(_MEMORY_PREFIX): + warnings.append( + f"{canonical} excluded by cypher_filter kind filter " + f"(allowed kinds: {sorted(kind_filter)!r})." + ) + if not channel_1_active and not question_active: return await self._recency_fallback( datasource=datasource, - max_memories=max_memories, - max_example_queries=max_example_queries, + candidate_ids=candidate_ids, + kind_filter=kind_filter, + max_results=max_results, warnings=warnings, + compact=compact, ) - # ``valid_canonicals`` filled in below using the corpus we just - # fetched (after datasource filter) so the lazy GC and recency - # fallback both apply the same predicate. - - # Single memory-corpus fetch shared by all channels. Pre-filtered - # by ``datasource`` so BM25 (channel 1) and the embedding cosine - # (channel 3) consume the narrowed list — IDF / matrix shape - # reflect the filtered subset (DEV-1409). - all_memories: List[Memory] = [] - if channel_1_active or question_active: - all_memories = _filter_memories_by_datasource( - await self._storage.list_memories(entities=None), - datasource, + # Datasource filter runs first so the off-datasource warning + # reflects "memory dropped because of datasource", not + # "memory dropped because of cypher_filter". + datasource_filtered_memories: list[Memory] = ( + _filter_memories_by_datasource( + memories=await self._storage.list_memories(entities=None), + datasource=datasource, ) + ) + # DEV-1513: detect named ``memory:`` refs whose memory was + # filtered out by the datasource pre-filter — emit BEFORE + # cypher_filter narrowing so a memory that IS rooted at the + # datasource but is excluded by cypher_filter doesn't get a + # spurious "not rooted at datasource" warning on top of the + # cypher_filter warning. + warnings = _dedup( + warnings + _memory_id_off_datasource_warnings( + canonical_input_entities=canonical_input_entities, + live_memory_ids={m.id for m in datasource_filtered_memories}, + datasource=datasource, + ) + ) + # DEV-1464: now narrow by the cypher_filter allowlist for the + # retrieval path — BM25 / tantivy / embeddings rank only the + # surviving memories. + if candidate_ids is not None: + all_memories: list[Memory] = [ + m for m in datasource_filtered_memories + if f"{_MEMORY_PREFIX}{m.id}" in candidate_ids + ] + else: + all_memories = datasource_filtered_memories - # DEV-1428: build the live canonical set so stale entity tags - # are excluded from BM25 ranking AND from any surfaced - # ``matched_entities`` list. Built once per call; reused across - # the BM25 path and the recency fallback. valid_canonicals = await self._valid_canonical_set( all_memories=all_memories, datasource=datasource, ) - # Build the in-memory corpus once when question is active — both - # channels 2 and 3 read from it (channel 2 for tantivy search, - # channel 3 to recover hit text by canonical_id). - corpus: Optional[Corpus] = None + corpus: Corpus | None = None if question_active: - all_models, datasources = await self._collect_index_corpus( - datasource=datasource, + all_models, datasources, datasource_descriptions = ( + await self._collect_index_corpus(datasource=datasource) ) corpus = build_in_memory_corpus( memories=all_memories, models=all_models, datasources=datasources, + datasource_descriptions=datasource_descriptions, ) - channel_1_memory_ranking, memory_by_id = self._run_channel_1( - canonical_input_entities=canonical_input_entities, - all_memories=all_memories, - channel_1_active=channel_1_active, - valid_canonicals=valid_canonicals, - ) + # DEV-1464: surface the reason a named memory: ref didn't + # appear in results when cypher_filter excluded it. + if candidate_ids is not None: + warnings = _dedup( + warnings + _memory_id_cypher_filter_warnings( + canonical_input_entities=canonical_input_entities, + candidate_ids=candidate_ids, + ) + ) + # DEV-1513: channel-1 named-entity surfacing. ( - channel_2_memory_ranking, - channel_2_entity_ranking, - index_hits_by_memory_id, - ) = self._run_channel_2( + channel_1_entity_ranking, + named_kind_text, + entity_surfacing_warnings, + ) = await self._build_channel_1_entity_ranking( + canonical_input_entities=canonical_input_entities, + datasource=datasource, corpus=corpus, - question=question, + candidate_ids=candidate_ids, ) - ( - channel_3_memory_ranking, - channel_3_entity_ranking, - channel_3_warnings, - ) = await self._run_channel_3( - question=question, - corpus=corpus, - question_active=question_active, - datasource=datasource, - eligible_memory_canonicals={ - f"{_MEMORY_PREFIX}{m.id}" for m in all_memories - }, + warnings = _dedup(warnings + entity_surfacing_warnings) + + # Fan out to every retriever in parallel. Per-retriever + # exceptions are isolated and converted to prefixed warnings + # in declaration order so a single failure can't crash the + # whole search. + raw_results = await asyncio.gather( + *( + r.retrieve( + query_entities=canonical_input_entities, + question=question, + all_memories=all_memories, + valid_canonicals=valid_canonicals, + corpus=corpus, + datasource=datasource, + ) + for r in self._retrievers + ), + return_exceptions=True, ) - warnings = _dedup(warnings + channel_3_warnings) + results: list[RetrievalResult] = [] + for r, raw in zip(self._retrievers, raw_results): + if isinstance(raw, BaseException): + warnings.append( + f"retriever {r.name!r} retrieve raised: {raw}" + ) + results.append(RetrievalResult()) + else: + results.append(raw) + warnings.extend(raw.warnings) + warnings = _dedup(warnings) - # Backfill memory_by_id from every channel so RRF can resolve - # any memory hit downstream. Build the id→Memory dict once so - # the three backfills stay O(N) overall (DEV-1414). + # Merge text_by_id with first-non-empty-wins precedence. + text_by_id = _merge_text_by_id_in_declaration_order(results) + + # Build memory_by_id from all retrievers' memory rankings. all_memories_by_id = {m.id: m for m in all_memories} - _backfill_memory_by_id( - memory_by_id=memory_by_id, - all_memories_by_id=all_memories_by_id, - mem_ids=channel_1_memory_ranking, - ) - _backfill_memory_by_id( - memory_by_id=memory_by_id, - all_memories_by_id=all_memories_by_id, - mem_ids=index_hits_by_memory_id.keys(), - ) - _backfill_memory_by_id( - memory_by_id=memory_by_id, - all_memories_by_id=all_memories_by_id, - mem_ids=channel_3_memory_ranking, - ) + memory_by_id: dict[str, Memory] = {} + for result in results: + _backfill_memory_by_id( + memory_by_id=memory_by_id, + all_memories_by_id=all_memories_by_id, + mem_ids=result.memory_ranking, + ) - memory_hits, example_query_hits = _fuse_memory_hits( - rankings=[ - channel_1_memory_ranking, - channel_2_memory_ranking, - channel_3_memory_ranking, - ], + all_hits = _fuse_all_hits( + memory_rankings=[r.memory_ranking for r in results], + entity_rankings=( + [channel_1_entity_ranking] + + [r.entity_ranking for r in results] + ), memory_by_id=memory_by_id, - index_hits_by_memory_id=index_hits_by_memory_id, + text_by_id=text_by_id, canonical_input_entities=canonical_input_entities, - max_memories=max_memories, - max_example_queries=max_example_queries, + corpus=corpus, + named_kind_text=named_kind_text, + max_results=max_results, valid_canonicals=valid_canonicals, + candidate_ids=candidate_ids, + kind_filter=kind_filter, + compact=compact, ) - # DEV-1428: stale Memory.query warnings — surface example_queries - # whose attached query references entities that no longer resolve. + + # DEV-1428 + DEV-1513: stale-Memory.query warnings for surfaced + # query-bearing hits AND for explicitly-named ``memory:`` + # refs whose attached query has stale references. + query_bearing_hits = [ + h for h in all_hits + if h.kind == "memory" and h.query is not None + ] warnings = _dedup( warnings + await self._stale_query_warnings( - example_query_hits=example_query_hits, + query_bearing_hits=query_bearing_hits, memory_by_id=memory_by_id, + ) + await self._stale_query_warnings_for_named_memory_refs( + canonical_input_entities=canonical_input_entities, + all_memories=all_memories, + already_warned_ids={h.id for h in query_bearing_hits}, ) ) - entity_hits = _fuse_entity_hits( - rankings=[channel_2_entity_ranking, channel_3_entity_ranking], - corpus=corpus, - max_entities=max_entities, - ) + + # DEV-1516: refresh stale categorical column hits in-place before + # returning. Per-model writes serialise; cross-model writes run + # concurrently. Silently no-op when engine is None. + if self._engine is not None: + all_hits = await self._refresh_stale_column_hits( + results=all_hits, compact=compact, + ) return SearchResponse( - memories=memory_hits, - example_queries=example_query_hits, - entities=entity_hits, + results=all_hits, resolved_input_entities=canonical_input_entities, warnings=warnings, ) - async def _resolve_inputs( + async def _apply_cypher_filter( self, *, - entities: Optional[List[str]], - query: Optional[Union[SlayerQuery, dict]], - ) -> Tuple[List[str], List[str]]: - """Walk ``entities`` + ``query`` into a deduped canonical-entity list - plus a deduped warning list. - - DEV-1428: search is lenient. Per-token resolution failures and - ambiguity errors become warnings (the token is dropped from the - canonical set). Unrelated ``ValueError``s (typing issues) still - raise — those are programmer errors, not data drift. + cypher_filter: str | None, + canonical_input_entities: list[str], + warnings: list[str], + ) -> tuple[ + frozenset[str] | None, + set[str] | None, + SearchResponse | None, + ]: + """DEV-1464: resolve the optional ``cypher_filter`` into + ``(candidate_ids, kind_filter, early)``. + + * ``candidate_ids`` is non-None when the full graph path ran + (advanced_search extra installed). The set is the allowlist + every channel is narrowed against. + * ``kind_filter`` is non-None when the naive fallback ran + (graph extra absent). The set is the entity kinds the result + is filtered down to. + * ``early`` is a short-circuit ``SearchResponse`` when the graph + path returned no ids — we skip the corpus build entirely. """ - canonical: List[str] = [] - warnings: List[str] = [] + if cypher_filter is None: + return None, None, None + if _search_graph.is_available(): + candidate_ids = await _search_graph.get_filtered_ids( + cypher=cypher_filter, storage=self._storage, + ) + if not candidate_ids: + early_warnings = _dedup( + warnings + [ + "cypher_filter returned no matching nodes; " + "search returned no results." + ] + ) + return candidate_ids, None, SearchResponse( + results=[], + resolved_input_entities=canonical_input_entities, + warnings=early_warnings, + ) + return candidate_ids, None, None + return None, _parse_naive_cypher(cypher_filter), None + + async def _build_channel_1_entity_ranking( + self, + *, + canonical_input_entities: list[str], + datasource: str | None, + corpus: Corpus | None, + candidate_ids: frozenset[str] | None = None, + ) -> tuple[list[str], dict[str, tuple[str, str, str | None]], list[str]]: + """DEV-1513: produce channel-1's contribution to the entity + ranking by surfacing each user-named canonical ref as itself. + + Returns ``(entity_ranking, named_kind_text, warnings)``: + + * ``entity_ranking`` — surviving canonicals in user-supplied + order; this is the channel-1 input to the entity-side of + ``_fuse_all_hits``. + * ``named_kind_text`` — ``{canonical: (kind, text)}`` lookup + consumed by ``_fuse_all_hits`` as a fallback when the corpus + doesn't carry the canonical (pure-named call with no corpus, + or hidden-from-corpus refs). + * ``warnings`` — drop reasons per filter (off-datasource, + hidden, missing, cypher_filter exclusion). + + DEV-1464: when ``candidate_ids`` is supplied, entities outside + the allowlist are dropped (with a warning) BEFORE the + rendering / lookup work so we don't waste a storage round-trip. + """ + entity_ranking: list[str] = [] + named_kind_text: dict[str, tuple[str, str, str | None]] = {} + warnings: list[str] = [] + for canonical in canonical_input_entities: + if canonical.startswith(_MEMORY_PREFIX): + # memory: refs participate in the memory ranking only. + continue + if candidate_ids is not None and canonical not in candidate_ids: + warnings.append( + f"entity {canonical!r} excluded by cypher_filter." + ) + continue + if datasource is not None and not canonical_id_rooted_at( + canonical_id=canonical, datasource=datasource, + ): + warnings.append( + f"entity {canonical!r} is not rooted at datasource " + f"{datasource!r}; dropped from entities bucket." + ) + continue + result = await _lookup_named_entity( + canonical=canonical, storage=self._storage, corpus=corpus, + ) + if isinstance(result, LookupHidden): + warnings.append( + f"entity {canonical!r} is on a hidden " + f"{result.reason.removeprefix('hidden ')}; " + f"dropped from entities bucket." + ) + continue + if isinstance(result, LookupMissing): + warnings.append( + f"entity {canonical!r} resolved but is no longer " + f"present in storage; dropped from entities bucket." + ) + continue + entity_ranking.append(canonical) + named_kind_text[canonical] = ( + result.kind, result.text, result.description, + ) + return entity_ranking, named_kind_text, warnings + + # ------------------------------------------------------------------ + # Write side — fan-out to retrievers + # ------------------------------------------------------------------ + + async def upsert_memory(self, memory: Memory) -> list[str]: + return await self._fan_out_with_isolation( + hook_name="upsert_memory", + invoke=lambda r: r.upsert_memory(memory), + ) + + async def refresh_model_subtree( + self, model: SlayerModel, + ) -> list[str]: + return await self._fan_out_with_isolation( + hook_name="refresh_model_subtree", + invoke=lambda r: r.refresh_model_subtree(model), + ) + + async def refresh_datasource( + self, + *, + name: str, + models: list[SlayerModel], + description: str | None = None, + ) -> list[str]: + return await self._fan_out_with_isolation( + hook_name="refresh_datasource", + invoke=lambda r: r.refresh_datasource( + name=name, models=models, description=description, + ), + ) + + async def _fan_out_with_isolation( + self, *, hook_name: str, invoke, + ) -> list[str]: + """Call ``invoke(retriever)`` on every registered retriever in + declaration order, isolating per-retriever exceptions as + prefixed warnings so subsequent retrievers still run. Returns + the deduped warning list.""" + warnings: list[str] = [] + for r in self._retrievers: + try: + warnings.extend(await invoke(r)) + except Exception as exc: # NOSONAR(S112) — best-effort fan-out + warnings.append( + f"retriever {r.name!r} {hook_name} raised: {exc}" + ) + return _dedup(warnings) + + # ------------------------------------------------------------------ + # Internal — input resolution / corpus collection + # ------------------------------------------------------------------ + + async def _validate_datasource_known( + self, datasource: str | None, + ) -> None: + """DEV-1409: reject typos in ``datasource`` before any corpus + walk.""" + if datasource is None: + return + known = sorted(await self._storage.list_datasources()) + if datasource not in known: + raise ValueError( + f"datasource {datasource!r} not found; known: {known}." + ) + + async def _resolve_inputs( + self, + *, + entities: list[str] | None, + query: SlayerQuery | dict | None, + ) -> tuple[list[str], list[str]]: + """Walk ``entities`` + ``query`` into a deduped canonical-entity + list plus a deduped warning list. DEV-1428: lenient — + per-token failures become warnings.""" + canonical: list[str] = [] + warnings: list[str] = [] if entities: for raw in entities: if not isinstance(raw, str): @@ -599,9 +1193,7 @@ async def _resolve_inputs( raw=raw, storage=self._storage, ) except (EntityResolutionError, AmbiguousModelError) as exc: - warnings.append( - f"entity {raw!r} dropped: {exc}" - ) + warnings.append(f"entity {raw!r} dropped: {exc}") continue canonical.extend(result.canonical_forms) warnings.extend(result.warnings) @@ -611,9 +1203,7 @@ async def _resolve_inputs( query=_coerce_query(query), storage=self._storage, ) except (EntityResolutionError, AmbiguousModelError) as exc: - warnings.append( - f"query input dropped: {exc}" - ) + warnings.append(f"query input dropped: {exc}") else: canonical.extend(extraction.canonical_forms) warnings.extend(extraction.warnings) @@ -622,327 +1212,117 @@ async def _resolve_inputs( async def _recency_fallback( self, *, - max_memories: int, - max_example_queries: int, - warnings: List[str], - datasource: Optional[str] = None, + max_results: int, + warnings: list[str], + datasource: str | None = None, + candidate_ids: frozenset[str] | None = None, + kind_filter: set[str] | None = None, + compact: bool = True, ) -> SearchResponse: - """Empty-input branch: partition all memories by recency into the - learning-only bucket (``memories``, capped by ``max_memories``) - and the query-bearing bucket (``example_queries``, capped by - ``max_example_queries``). + """Empty-input branch: return the newest memories (both + learning-only and query-bearing) as a flat list, capped by + ``max_results``. No retriever is invoked on this path. DEV-1409: when ``datasource`` is set, the same memory pre-filter - used by the main search path applies — only memories with at - least one entity rooted at the requested datasource are eligible. - """ + used by the main search path applies. + + DEV-1464: when ``candidate_ids`` is set, only memories whose + canonical id appears in the allowlist survive; when + ``kind_filter`` is set and doesn't include ``"memory"``, the + recency bucket is empty (no entity recency on the fallback + path).""" warnings.append( "no entities, query, or question supplied; returning " "newest memories by recency." ) recency_memories = _filter_memories_by_datasource( - await self._storage.list_memories(entities=None), - datasource, + memories=await self._storage.list_memories(entities=None), + datasource=datasource, + ) + had_candidates_pre_filter = bool(recency_memories) + if candidate_ids is not None: + recency_memories = [ + m for m in recency_memories + if f"{_MEMORY_PREFIX}{m.id}" in candidate_ids + ] + if kind_filter is not None and "memory" not in kind_filter: + recency_memories = [] + # DEV-1464: when cypher_filter (or its naive kind-filter + # fallback) zeroed out an otherwise-populated recency pool, + # surface that explicitly — the generic "returning newest" + # warning would otherwise read as "system is healthy, the + # corpus is just empty," masking that the filter was the cause. + filters_excluded_all = ( + had_candidates_pre_filter + and not recency_memories + and (candidate_ids is not None or kind_filter is not None) ) + if filters_excluded_all: + warnings.append( + "cypher_filter excluded all memory candidates for the " + "empty-input recency fallback; no results." + ) recency_memories.sort(key=lambda m: m.created_at, reverse=True) valid_canonicals = await self._valid_canonical_set( all_memories=recency_memories, datasource=datasource, ) - memory_hits: List[MemoryHit] = [] - example_query_hits: List[ExampleQueryHit] = [] + hits: list[SearchHit] = [] for m in recency_memories: - hit = _build_memory_hit( + if len(hits) >= max_results: + break + hits.append(_build_memory_hit( mem=m, memory_id=m.id, score=0.0, - index_hits_by_memory_id={}, + text_by_id={}, canonical_input_entities=[], valid_canonicals=valid_canonicals, - ) - if isinstance(hit, MemoryHit) and len(memory_hits) < max_memories: - memory_hits.append(hit) - elif ( - isinstance(hit, ExampleQueryHit) - and len(example_query_hits) < max_example_queries - ): - example_query_hits.append(hit) - if ( - len(memory_hits) >= max_memories - and len(example_query_hits) >= max_example_queries - ): - break - # DEV-1428: emit stale-Memory.query warnings on the recency path - # too; otherwise an empty-input search would silently return - # example_queries whose attached queries no longer resolve. + compact=compact, + )) + # DEV-1428: emit stale-Memory.query warnings on the recency path too. memory_by_id = {m.id: m for m in recency_memories} + query_bearing = [h for h in hits if h.query is not None] warnings = _dedup( warnings + await self._stale_query_warnings( - example_query_hits=example_query_hits, + query_bearing_hits=query_bearing, memory_by_id=memory_by_id, ) ) return SearchResponse( - memories=memory_hits, - example_queries=example_query_hits, - entities=[], + results=hits, resolved_input_entities=[], warnings=warnings, ) - def _run_channel_1( - self, - *, - canonical_input_entities: List[str], - all_memories: List[Memory], - channel_1_active: bool, - valid_canonicals: Optional[set] = None, - ) -> Tuple[List[str], dict[str, Memory]]: - """Entity-overlap BM25 channel. Ranks the full memory corpus — - no candidate-pool truncation (DEV-1414). - - DEV-1428: each memory's ``entities`` list is pre-filtered - against ``valid_canonicals`` so stale tags neither contribute - to BM25 scoring nor surface as ``matched_entities``.""" - channel_1_memory_ranking: List[str] = [] - memory_by_id: dict[str, Memory] = {} - if channel_1_active and canonical_input_entities: - filtered_memories = ( - _filter_memories_entities( - all_memories, valid_canonicals, - ) - if valid_canonicals is not None - else all_memories - ) - ranked = bm25_rank( - memories=filtered_memories, - query_entities=canonical_input_entities, - ) - # Use the original memory rows (with stored entity lists) for - # the returned mapping so callers see the un-filtered shape; - # only the ranking input was filtered. - originals_by_id = {m.id: m for m in all_memories} - for memory, _score in ranked: - original = originals_by_id.get(memory.id, memory) - memory_by_id[memory.id] = original - channel_1_memory_ranking.append(memory.id) - return channel_1_memory_ranking, memory_by_id - - def _run_channel_2( - self, - *, - corpus: Optional[Corpus], - question: Optional[str], - ) -> Tuple[List[str], List[str], dict[str, IndexHit]]: - """Tantivy full-text channel. - - DEV-1414: runs as TWO kind-filtered queries — one over memory - docs only, one over entity docs only — so the per-kind ranking - is a pure function of the corpus + question, never affected by - the other kind's cap. The ``limit`` for each call is the size of - the corresponding kind in the corpus, so each query returns the - complete per-kind ranking. - - Returns ``(memory_ranking, entity_ranking_canonicals, - by_memory_id_hits)``. Empty when ``corpus`` or ``question`` is - missing. - """ - if corpus is None or not question or not question.strip(): - return [], [], {} - memory_count, entity_count = _count_corpus_kinds(corpus) - memory_hits = ( - search_index( - index=corpus.index, - question=question, - limit=memory_count, - kind_filter="memory", - ) - if memory_count > 0 - else [] - ) - entity_hits = ( - search_index( - index=corpus.index, - question=question, - limit=entity_count, - exclude_kind="memory", - ) - if entity_count > 0 - else [] - ) - memory_ranking: List[str] = [] - by_memory_id: dict[str, IndexHit] = {} - for hit in memory_hits: - if hit.memory_id is None: - continue - memory_ranking.append(hit.memory_id) - by_memory_id[hit.memory_id] = hit - entity_ranking = [h.id for h in entity_hits] - return memory_ranking, entity_ranking, by_memory_id - - async def _run_channel_3( - self, - *, - question: Optional[str], - corpus: Optional[Corpus], - question_active: bool, - datasource: Optional[str] = None, - eligible_memory_canonicals: Optional[Set[str]] = None, - ) -> Tuple[List[str], List[str], List[str]]: - """Embedding-similarity channel (DEV-1386). Returns - ``(memory_ranking, entity_ranking_canonicals, warnings)``. - - DEV-1414: the corpus is partitioned by ``entity_kind`` and each - kind is ranked in full via two cosine calls. The per-kind - ranking is a pure function of the corpus + question. - - Skipped (with a warning) when: - - * ``question`` is empty, - * the ``embedding_search`` extra is not installed, - * the active model has no embedding rows in storage, - * the query embedding call fails. - - DEV-1409: when ``datasource`` is set, the corpus is pre-filtered - before the matrix build so cosine similarity is computed only - against: - - * entity rows (``entity_kind != 'memory'``) rooted at the - requested datasource (exact match or dotted-path descendant), - * memory rows whose ``canonical_id`` appears in the supplied - ``eligible_memory_canonicals`` set (already datasource-filtered - upstream). - - DEV-1414: rows whose ``canonical_id`` is not in the live tantivy - corpus (stale memory ids, hidden / deleted entities) are dropped - before the matrix build. Otherwise stale rows would consume - cosine rank positions and degrade live docs' RRF scores — - invariant under cap changes (so the per-bucket contract still - holds) but surprising and lossy. The filter keeps the channel's - candidate set aligned with channel 2's tantivy corpus. - """ - if not question_active or corpus is None: - return [], [], [] - if not embedding_client.is_available(): - return [], [], [ - "embedding channel skipped: `embedding_search` extra not " - "installed or no API key configured for the active " - "embedding model.", - ] - - # Local import to break the ``slayer.search`` ↔ ``slayer.embeddings`` - # cycle (the embedding service imports render helpers from - # ``slayer.search.render``). - from slayer.embeddings.service import EmbeddingService - - service = EmbeddingService(storage=self._storage) - rows = await service.fetch_corpus() - if datasource is not None: - rows = _filter_embedding_corpus_by_datasource( - rows, - datasource=datasource, - eligible_memory_canonicals=eligible_memory_canonicals or set(), - ) - # Drop sidecar rows that don't correspond to anything in the - # live tantivy corpus (DEV-1414). Memory rows are keyed - # ``memory:`` in storage and as the corpus's - # ``canonical_to_kind`` key; entity rows share the canonical - # string directly. Both shapes match by single dict lookup. - live_canonicals = corpus.canonical_to_kind.keys() - rows = [r for r in rows if r.canonical_id in live_canonicals] - if not rows: - return [], [], [ - f"embedding channel skipped: no embedding rows for model " - f"{service.model_name!r}. Run `slayer ingest` to populate.", - ] - try: - import numpy as np - from slayer.embeddings.ranker import ( - normalise, - normalise_matrix, - top_k_cosine, - ) - except ImportError: - return [], [], [ - "embedding channel skipped: numpy not installed " - "(reinstall with the `embedding_search` extra).", - ] - query_vec = await service.embed_question(question or "") - if query_vec is None: - return [], [], [ - "embedding channel skipped: query embedding failed.", - ] - # All persisted rows share the active model's dim; sample any - # row to detect a stale-dim corpus before partitioning. - if len(rows[0].embedding) != len(query_vec): - return [], [], [ - f"embedding channel skipped: dim mismatch " - f"(query={len(query_vec)}, corpus={len(rows[0].embedding)}). " - f"Re-run `slayer ingest` to refresh embeddings against " - f"the current model.", - ] - - memory_rows = [r for r in rows if r.entity_kind == "memory"] - entity_rows = [r for r in rows if r.entity_kind != "memory"] - normalised_query = normalise(query_vec) - ranked_memory_canonicals = _rank_embedding_kind( - rows=memory_rows, - normalised_query=normalised_query, - np=np, - normalise_matrix=normalise_matrix, - top_k_cosine=top_k_cosine, - ) - memory_ranking: List[str] = [] - for canonical in ranked_memory_canonicals: - memory_id = _memory_id_from_canonical(canonical) - if memory_id is not None: - memory_ranking.append(memory_id) - entity_ranking = _rank_embedding_kind( - rows=entity_rows, - normalised_query=normalised_query, - np=np, - normalise_matrix=normalise_matrix, - top_k_cosine=top_k_cosine, - ) - return memory_ranking, entity_ranking, [] - async def _valid_canonical_set( self, *, - all_memories: List[Memory], - datasource: Optional[str], + all_memories: list[Memory], + datasource: str | None, ) -> set: - """DEV-1428: live canonical set used to filter stale entity tags - out of memory ``entities`` lists before BM25 ranking and before - ``matched_entities`` is surfaced. - - Walks datasources + every persisted model identity (cheap - per-storage call) + ``memory:`` for every memory in scope. - Datasource-filtered when ``datasource`` is set. - """ canonicals: set = set() canonicals.update( await self._collect_datasource_canonicals(datasource=datasource) ) canonicals.update( - await self._collect_model_subtree_canonicals(datasource=datasource) + await self._collect_model_subtree_canonicals( + datasource=datasource, + ) ) canonicals.update(_collect_memory_canonicals(all_memories)) return canonicals async def _collect_datasource_canonicals( - self, *, datasource: Optional[str], + self, *, datasource: str | None, ) -> set: - """Set of bare datasource canonical ids, narrowed by ``datasource``.""" names = await self._storage.list_datasources() if datasource is not None: names = [d for d in names if d == datasource] return set(names) async def _collect_model_subtree_canonicals( - self, *, datasource: Optional[str], + self, *, datasource: str | None, ) -> set: - """Set of `.[.]` canonical ids across every - persisted model identity, narrowed by ``datasource`` when set.""" out: set = set() identities = await self._storage._list_all_model_identities() for ds, name in identities: @@ -965,15 +1345,14 @@ async def _collect_model_subtree_canonicals( async def _stale_query_warnings( self, *, - example_query_hits: List["ExampleQueryHit"], - memory_by_id: Dict[str, Memory], - ) -> List[str]: - """DEV-1428: emit one warning per example_queries hit whose - attached ``Memory.query`` references entities that no longer - resolve. The query is NOT rewritten — agents who notice the - warning can re-save the memory to clean it.""" - out: List[str] = [] - for hit in example_query_hits: + query_bearing_hits: list[SearchHit], + memory_by_id: dict[str, Memory], + ) -> list[str]: + """Emit one warning per surfaced query-bearing hit whose + attached ``SlayerQuery`` no longer resolves (entities pointing + at deleted/renamed models or columns). DEV-1428.""" + out: list[str] = [] + for hit in query_bearing_hits: mem = memory_by_id.get(hit.id) if mem is None or mem.query is None: continue @@ -988,23 +1367,52 @@ async def _stale_query_warnings( ) return out + async def _stale_query_warnings_for_named_memory_refs( + self, + *, + canonical_input_entities: list[str], + all_memories: list[Memory], + already_warned_ids: set[str], + ) -> list[str]: + """DEV-1513: emit the stale-query warning for any explicitly-named + ``memory:`` ref pointing at a query-bearing memory with + stale refs, regardless of whether the ``max_results`` cap + suppressed the hit. The user explicitly named the memory; they + deserve to know the attached query is broken.""" + memories_by_id = {m.id: m for m in all_memories} + out: list[str] = [] + for canonical in canonical_input_entities: + if not canonical.startswith(_MEMORY_PREFIX): + continue + memory_id = canonical[len(_MEMORY_PREFIX):] + if not memory_id or memory_id in already_warned_ids: + continue + mem = memories_by_id.get(memory_id) + if mem is None or mem.query is None: + continue + try: + await extract_entities_from_query( + query=mem.query, storage=self._storage, + ) + except (EntityResolutionError, AmbiguousModelError) as exc: + out.append( + f"example_query {_MEMORY_PREFIX}{memory_id}: attached " + f"query has stale references ({exc}); re-save to clean." + ) + return out + async def _collect_index_corpus( self, *, - datasource: Optional[str] = None, - ) -> Tuple[List[SlayerModel], List[str]]: - """Walk datasources + models into the in-memory corpus. - - DEV-1409: when ``datasource`` is set, only models in that one - datasource are walked, and only that datasource's doc lands in - the returned list. Validation that ``datasource`` is known - happens upstream in ``SearchService.search`` so this method - stays cheap. - """ + datasource: str | None = None, + ) -> tuple[list[SlayerModel], list[str], dict[str, str | None]]: + """DEV-1549: also returns ``{ds_name → description}`` so the + corpus builder can populate ``canonical_to_description`` for + datasource hits without re-loading the configs.""" datasources = await self._storage.list_datasources() if datasource is not None: datasources = [d for d in datasources if d == datasource] - models: List[SlayerModel] = [] + models: list[SlayerModel] = [] identities = await self._storage._list_all_model_identities() for ds, name in identities: if datasource is not None and ds != datasource: @@ -1012,4 +1420,18 @@ async def _collect_index_corpus( m = await self._storage.get_model(name, data_source=ds) if m is not None: models.append(m) - return models, datasources + descriptions: dict[str, str | None] = {} + for ds_name in datasources: + cfg = await self._storage.get_datasource(ds_name) + descriptions[ds_name] = cfg.description if cfg is not None else None + return models, datasources, descriptions + + +__all__ = [ + "LookupFound", + "LookupHidden", + "LookupMissing", + "SearchHit", + "SearchResponse", + "SearchService", +] diff --git a/slayer/sql/client.py b/slayer/sql/client.py index f0c4dc1e..4d66a302 100644 --- a/slayer/sql/client.py +++ b/slayer/sql/client.py @@ -5,7 +5,8 @@ import functools import logging import time -from typing import Any, Awaitable, Callable, Dict, List, Optional +from typing import Any +from collections.abc import Awaitable, Callable import sqlalchemy as sa import sqlalchemy.engine.url @@ -14,7 +15,14 @@ from sqlalchemy.pool import StaticPool from slayer.core.models import DatasourceConfig -from slayer.sql.sqlite_udfs import register_sqlite_udfs +from slayer.engine import timing +from slayer.sql.dialects.sqlite import SqliteDialect + +# Module-level singleton — its ``register_udfs`` is the SQLAlchemy +# ``connect`` event hook for SQLite engines. The dialect class wraps +# the module-level ``register_sqlite_udfs`` helper in ``dialects/sqlite.py``; +# both engine factories below call into this one instance. +_SQLITE_DIALECT = SqliteDialect() logger = logging.getLogger(__name__) @@ -35,7 +43,7 @@ # value or the path component of `sqlite:///:memory:` connection strings. _MEMORY_DB_NAME = ":memory:" -_sync_engines: Dict[str, sa.Engine] = {} +_sync_engines: dict[str, sa.Engine] = {} def _get_sync_engine(connection_string: str) -> sa.Engine: @@ -56,7 +64,7 @@ def _get_sync_engine(connection_string: str) -> sa.Engine: if engine.dialect.name == "sqlite": @sa_event.listens_for(engine, "connect") def _register_udfs(dbapi_connection, _connection_record): - register_sqlite_udfs(dbapi_connection) + _SQLITE_DIALECT.register_udfs(dbapi_connection) _sync_engines[connection_string] = engine return _sync_engines[connection_string] @@ -79,7 +87,7 @@ def _is_in_memory_sqlite(connection_string: str) -> bool: database = url.database if not database or database == _MEMORY_DB_NAME: return True - query: Dict[str, Any] = dict(url.query) if url.query else {} + query: dict[str, Any] = dict(url.query) if url.query else {} # SQLite honors `mode=memory` and the `file::memory:` URI form ONLY when # the connection is opened with URI handling enabled (`uri=true`). # Without `uri=true`, SQLite treats the database part as a literal @@ -120,13 +128,13 @@ def _create_in_memory_sqlite_engine(connection_string: str) -> sa.Engine: ) @sa_event.listens_for(engine, "connect") def _register_udfs(dbapi_connection, _connection_record): - register_sqlite_udfs(dbapi_connection) + _SQLITE_DIALECT.register_udfs(dbapi_connection) return engine def _resolve_sync_engine( connection_string: str, - override_engine: Optional[sa.Engine] = None, + override_engine: sa.Engine | None = None, ) -> sa.Engine: """Choose the engine for a sync DB call. @@ -153,7 +161,7 @@ def _get_async_engine(connection_string: str): return create_async_engine(connection_string, pool_pre_ping=True) -def _async_connection_string(connection_string: str, db_type: Optional[str]) -> Optional[str]: +def _async_connection_string(connection_string: str, db_type: str | None) -> str | None: """Convert a sync connection string to its async equivalent, or None if no async driver.""" async_scheme = _ASYNC_DRIVERS.get(db_type) if async_scheme is None: @@ -169,13 +177,24 @@ def _async_connection_string(connection_string: str, db_type: Optional[str]) -> # --------------------------------------------------------------------------- -def _map_type_code(type_code, db_type: Optional[str] = None) -> str: +def _map_type_code(type_code, db_type: str | None = None) -> str: """Map a DB-API type_code to a SLayer type category. Handles DuckDB (string type names), SQLite (Python types), asyncpg (Postgres OID integers), and aiomysql (MySQL field-type codes). When ``db_type`` is provided, the correct OID/field-type map is selected. + + DEV-1551: dialect-specific cursor type-code mappings (currently only + Snowflake's snowflake-connector ``FieldType`` integer codes) are + consulted FIRST via ``SqlDialect.map_cursor_type_code``. The base + class returns ``None`` so other dialects fall through to the + Postgres-OID / MySQL-fieldtype / ODBC paths below. """ + if isinstance(type_code, int) and db_type: + from slayer.sql.dialects import dialect_for_ds_type # noqa: PLC0415 + dialect_category = dialect_for_ds_type(db_type).map_cursor_type_code(type_code) + if dialect_category is not None: + return dialect_category if isinstance(type_code, str): # DuckDB returns type name strings like 'INTEGER', 'VARCHAR', etc. tc = type_code.upper() @@ -202,12 +221,22 @@ def _map_type_code(type_code, db_type: Optional[str] = None) -> str: # Select the correct map by database type if db_type and "mysql" in db_type.lower(): return _MYSQL_TYPE_MAP.get(type_code, "string") + if db_type and any(t in db_type.lower() for t in ("mssql", "sqlserver", "tsql")): + return _ODBC_SQL_TYPE_MAP.get(type_code, "string") + # DEV-1551: Snowflake had its first crack at the integer code via + # ``SqlDialect.map_cursor_type_code`` above. If we land here with + # ``db_type='snowflake'`` it means the code wasn't recognised by + # ``_SNOWFLAKE_TYPE_MAP``; default to ``"string"`` rather than + # mis-classifying it through ``_PG_OID_MAP`` (Postgres OID 16 is + # ``boolean`` but undefined on Snowflake). + if db_type and "snowflake" in db_type.lower(): + return "string" return _PG_OID_MAP.get(type_code, "string") return "string" # Postgres OIDs (from pg_type) -_PG_OID_MAP: Dict[int, str] = { +_PG_OID_MAP: dict[int, str] = { 16: "boolean", # bool 20: "number", # int8 (bigint) 21: "number", # int2 (smallint) @@ -229,7 +258,7 @@ def _map_type_code(type_code, db_type: Optional[str] = None) -> str: } # MySQL field-type codes (aiomysql wire protocol) -_MYSQL_TYPE_MAP: Dict[int, str] = { +_MYSQL_TYPE_MAP: dict[int, str] = { 0: "number", # MYSQL_TYPE_DECIMAL 1: "boolean", # MYSQL_TYPE_TINY (TINYINT/BOOL) 2: "number", # MYSQL_TYPE_SHORT @@ -251,8 +280,45 @@ def _map_type_code(type_code, db_type: Optional[str] = None) -> str: 254: "string", # MYSQL_TYPE_STRING } +# ODBC SQL type codes (pyodbc with SQL Server / mssql+pyodbc driver). +# Positive codes are the standard ODBC C-level SQL_* constants; negative codes +# are SQL Server extensions (SQL_SS_*) defined in msodbcsql.h. +_ODBC_SQL_TYPE_MAP: dict[int, str] = { + # Integer / numeric family + 4: "number", # SQL_INTEGER + 5: "number", # SQL_SMALLINT + -6: "number", # SQL_TINYINT + -5: "number", # SQL_BIGINT + 2: "number", # SQL_NUMERIC + 3: "number", # SQL_DECIMAL + 6: "number", # SQL_FLOAT + 7: "number", # SQL_REAL + 8: "number", # SQL_DOUBLE + # String family + 1: "string", # SQL_CHAR + 12: "string", # SQL_VARCHAR + -1: "string", # SQL_LONGVARCHAR + -8: "string", # SQL_WCHAR + -9: "string", # SQL_WVARCHAR + -10: "string", # SQL_WLONGVARCHAR + -152: "string", # SQL_SS_XML + -11: "string", # SQL_GUID (uniqueidentifier) + # Boolean + -7: "boolean", # SQL_BIT + # Binary (rowversion / varbinary — treat as opaque string) + -2: "string", # SQL_BINARY + -3: "string", # SQL_VARBINARY + -4: "string", # SQL_LONGVARBINARY + # Temporal family + 91: "time", # SQL_TYPE_DATE + 92: "time", # SQL_TYPE_TIME + 93: "time", # SQL_TYPE_TIMESTAMP + -154: "time", # SQL_SS_TIMESTAMPOFFSET (datetimeoffset) + -155: "time", # SQL_SS_TIME2 (time with fractional seconds) +} -def _extract_types_from_cursor(result, db_type: Optional[str] = None) -> Dict[str, str]: + +def _extract_types_from_cursor(result, db_type: str | None = None) -> dict[str, str]: """Extract {column_name: type_category} from a SQLAlchemy CursorResult. Uses cursor.description type_code when available (DuckDB, Postgres). @@ -292,6 +358,8 @@ def _extract_types_from_cursor(result, db_type: Optional[str] = None) -> Dict[st # Databases that return all-None cursor.description type codes need a real row _NEEDS_ROW_FOR_TYPES = {"sqlite"} +# T-SQL (SQL Server) does not support LIMIT; use SELECT TOP N instead. +_TSQL_DB_TYPES = frozenset({"mssql", "sqlserver", "tsql"}) # DBs that should call _execute_with_retry_sync inline from async coroutines. # Empty: every dispatch goes through _run_sync_in_thread / _execute_with_retry_threaded # so the event loop is never blocked on DB work or on time.sleep retry backoff. @@ -311,30 +379,84 @@ async def _run_sync_in_thread(func, *args, **kwargs): return await loop.run_in_executor(executor, call) +def _build_type_probe_sql(sql: str, db_type: str | None) -> str: + """Build a row-limiting probe query appropriate for the target dialect.""" + limit = 1 if db_type in _NEEDS_ROW_FOR_TYPES else 0 + if db_type in _TSQL_DB_TYPES: + return f"SELECT TOP {limit} * FROM ({sql}) AS _types" + return f"SELECT * FROM ({sql}) AS _types LIMIT {limit}" + + +def _apply_type_probe_timeout(conn, db_type: str | None, timeout_seconds: int) -> None: + """DEV-1551: apply the dialect's statement-timeout SQL ahead of a + type-probe execution. Snowflake's ``LIMIT 0`` still compiles and + consumes warehouse compute, so an unbounded probe can stall on a + suspended warehouse or a runaway plan compilation. Only fires for + dialects whose ``SqlDialect.statement_timeout_sql`` returns non-None + — base no-op for postgres/mysql/clickhouse (their query-path + timeout SET is handled inline by ``_execute_sql_sync`` and isn't + needed for cursor-metadata probes). + """ + if not db_type: + return + from slayer.sql.dialects import dialect_for_ds_type # noqa: PLC0415 + timeout_sql = dialect_for_ds_type(db_type).statement_timeout_sql(timeout_seconds) + if timeout_sql: + conn.execute(sa.text(timeout_sql)) + + +async def _apply_type_probe_timeout_async(conn, db_type: str | None, timeout_seconds: int) -> None: + """Async sibling of ``_apply_type_probe_timeout``.""" + if not db_type: + return + from slayer.sql.dialects import dialect_for_ds_type # noqa: PLC0415 + timeout_sql = dialect_for_ds_type(db_type).statement_timeout_sql(timeout_seconds) + if timeout_sql: + await conn.execute(sa.text(timeout_sql)) + + +# Default timeout for type probes. Type-probe statements only compile +# (LIMIT 0 / LIMIT 1); 60s is generous for any reasonable query. +_TYPE_PROBE_TIMEOUT_SECONDS = 60 + + def _get_column_types_sync( sql: str, connection_string: str, - db_type: Optional[str], - engine: Optional[sa.Engine] = None, -) -> Dict[str, str]: - """Infer column types. Uses LIMIT 0 for cursor metadata, LIMIT 1 for SQLite.""" + db_type: str | None, + engine: sa.Engine | None = None, +) -> dict[str, str]: + """Infer column types. Uses LIMIT 0 for cursor metadata, LIMIT 1 for SQLite. + T-SQL uses SELECT TOP N instead of LIMIT.""" engine = _resolve_sync_engine(connection_string, override_engine=engine) - limit = 1 if db_type in _NEEDS_ROW_FOR_TYPES else 0 - limit_sql = f"SELECT * FROM ({sql}) AS _types LIMIT {limit}" + limit_sql = _build_type_probe_sql(sql, db_type) with engine.connect() as conn: + _apply_type_probe_timeout(conn, db_type, _TYPE_PROBE_TIMEOUT_SECONDS) result = conn.execute(sa.text(limit_sql)) return _extract_types_from_cursor(result, db_type=db_type) +def get_column_types_sync( + sql: str, *, engine: sa.Engine, db_type: str | None = None +) -> dict[str, str]: + """Public sync column-type inference for a query over an existing engine. + + Stable entry point (e.g. for the OSI importer) around + ``_get_column_types_sync``; returns ``{column_name: type_category}``. + """ + return _get_column_types_sync(sql, connection_string="", db_type=db_type, engine=engine) + + async def _get_column_types_async( sql: str, engine, - db_type: Optional[str], -) -> Dict[str, str]: - """Async version of column type inference. Uses LIMIT 0; LIMIT 1 for SQLite.""" - limit = 1 if db_type in _NEEDS_ROW_FOR_TYPES else 0 - limit_sql = f"SELECT * FROM ({sql}) AS _types LIMIT {limit}" + db_type: str | None, +) -> dict[str, str]: + """Async version of column type inference. Uses LIMIT 0; LIMIT 1 for SQLite. + T-SQL uses SELECT TOP N instead of LIMIT.""" + limit_sql = _build_type_probe_sql(sql, db_type) async with engine.connect() as conn: + await _apply_type_probe_timeout_async(conn, db_type, _TYPE_PROBE_TIMEOUT_SECONDS) result = await conn.execute(sa.text(limit_sql)) return _extract_types_from_cursor(result, db_type=db_type) @@ -353,7 +475,23 @@ class SlayerSQLClient: def __init__(self, datasource: DatasourceConfig): self.datasource = datasource self._async_engine = None - self._sync_engine: Optional[sa.Engine] = None + self._sync_engine: sa.Engine | None = None + + async def aclose(self) -> None: + """Dispose the cached async engine inside the current event loop.""" + engine = self._async_engine + if engine is None: + return + # Null first so a failed dispose can't leave a half-torn engine cached. + self._async_engine = None + try: + await engine.dispose() + except Exception as exc: # pragma: no cover + import logging + logging.getLogger(__name__).warning( + "Async engine dispose failed for datasource %r: %s", + self.datasource.name, exc, + ) def _get_async_engine(self): """Get or create the async engine for this client (cached per instance).""" @@ -366,15 +504,18 @@ def _get_async_engine(self): self._async_engine = _get_async_engine(async_conn_str) return self._async_engine - def _get_sync_engine_for_client(self) -> Optional[sa.Engine]: - """Return a per-client sync engine for in-memory SQLite, else None. + def _get_sync_engine_for_client(self) -> sa.Engine | None: + """Return a per-client sync engine. For ``sqlite:///:memory:`` (and equivalent URI-form variants) every ``SlayerSQLClient`` instance owns its own ``StaticPool`` engine so the single pinned connection is shared across all sync/async paths - on this client — but isolated from other clients. For every other - connection string this returns ``None`` and the helpers fall back - to the module-level engine cache via ``_resolve_sync_engine``. + on this client — but isolated from other clients. + + DEV-1551: every other case delegates to + ``engine_factory.get_engine(self.datasource)`` so dialect runtime + hooks (Snowflake's ``creator=`` bridge and per-connection USE + WAREHOUSE/SCHEMA listener) fire uniformly across consumers. """ if self._sync_engine is not None: return self._sync_engine @@ -382,13 +523,16 @@ def _get_sync_engine_for_client(self) -> Optional[sa.Engine]: if _is_in_memory_sqlite(conn_str): self._sync_engine = _create_in_memory_sqlite_engine(conn_str) return self._sync_engine - return None + # Cached factory engine — dialect hooks attach listeners + creator=. + from slayer.sql import engine_factory # noqa: PLC0415 + self._sync_engine = engine_factory.get_engine(self.datasource) + return self._sync_engine async def execute( self, sql: str, timeout_seconds: int = 120, - ) -> List[Dict[str, Any]]: + ) -> list[dict[str, Any]]: """Execute SQL asynchronously.""" async_engine = self._get_async_engine() db_type = self.datasource.type @@ -415,7 +559,7 @@ async def execute( engine=self._get_sync_engine_for_client(), ) - async def get_column_types(self, sql: str) -> Dict[str, str]: + async def get_column_types(self, sql: str) -> dict[str, str]: """Infer column types by executing SQL with LIMIT 0. Returns {column_name: type_category} where type_category is @@ -444,7 +588,7 @@ def execute_sync( self, sql: str, timeout_seconds: int = 120, - ) -> List[Dict[str, Any]]: + ) -> list[dict[str, Any]]: """Execute SQL synchronously (for CLI, notebooks, tests).""" return _execute_with_retry_sync( sql=sql, @@ -506,11 +650,11 @@ def _is_transient_db_error(exc: BaseException) -> bool: async def _retry_with_backoff( *, sql: str, - do_call: Callable[[], Awaitable[List[Dict[str, Any]]]], + do_call: Callable[[], Awaitable[list[dict[str, Any]]]], max_attempts: int, initial_delay: float, max_delay: float, -) -> List[Dict[str, Any]]: +) -> list[dict[str, Any]]: """Retry an async DB call with exponential backoff on transient errors. `sql` is used only for the warning's excerpt so users can correlate @@ -540,12 +684,12 @@ async def _retry_with_backoff( async def _execute_with_retry_async( sql: str, engine, - db_type: Optional[str], + db_type: str | None, timeout_seconds: int = 120, max_attempts: int = 3, initial_delay: float = 1.0, max_delay: float = 10.0, -) -> List[Dict[str, Any]]: +) -> list[dict[str, Any]]: return await _retry_with_backoff( sql=sql, do_call=lambda: _execute_sql_async( @@ -560,11 +704,14 @@ async def _execute_with_retry_async( async def _execute_sql_async( sql: str, engine, - db_type: Optional[str], + db_type: str | None, timeout_seconds: int = 120, -) -> List[Dict[str, Any]]: +) -> list[dict[str, Any]]: + _t = timing.start() async with engine.connect() as conn: + timing.record("connect", _t) timeout_ms = timeout_seconds * 1000 + _t = timing.start() if db_type in ("mysql", "mariadb"): await conn.execute(sa.text(f"SET max_execution_time = {timeout_ms}")) elif db_type in ("postgres", "postgresql", None): @@ -572,9 +719,22 @@ async def _execute_sql_async( await conn.execute(sa.text(f"SET statement_timeout = {timeout_ms}")) except Exception: pass + else: + # Dialect-specific timeout statement (DEV-1551). The base + # SqlDialect returns None — only dialects with a custom + # statement_timeout_sql (currently SnowflakeDialect) emit a + # SET. + from slayer.sql.dialects import dialect_for_ds_type # noqa: PLC0415 + timeout_sql = dialect_for_ds_type(db_type).statement_timeout_sql(timeout_seconds) + if timeout_sql: + await conn.execute(sa.text(timeout_sql)) + timing.record("set_timeout", _t) + _t = timing.start() result = await conn.execute(sa.text(sql)) columns = list(result.keys()) - return [dict(zip(columns, row)) for row in result.fetchall()] + rows = [dict(zip(columns, row)) for row in result.fetchall()] + timing.record("query", _t) + return rows # --------------------------------------------------------------------------- @@ -585,13 +745,13 @@ async def _execute_sql_async( async def _execute_with_retry_threaded( sql: str, connection_string: str, - db_type: Optional[str], + db_type: str | None, timeout_seconds: int = 120, max_attempts: int = 3, initial_delay: float = 1.0, max_delay: float = 10.0, - engine: Optional[sa.Engine] = None, -) -> List[Dict[str, Any]]: + engine: sa.Engine | None = None, +) -> list[dict[str, Any]]: return await _retry_with_backoff( sql=sql, do_call=lambda: _run_sync_in_thread( @@ -616,13 +776,13 @@ async def _execute_with_retry_threaded( def _execute_with_retry_sync( sql: str, connection_string: str, - db_type: Optional[str], + db_type: str | None, timeout_seconds: int = 120, max_attempts: int = 3, initial_delay: float = 1.0, max_delay: float = 10.0, - engine: Optional[sa.Engine] = None, -) -> List[Dict[str, Any]]: + engine: sa.Engine | None = None, +) -> list[dict[str, Any]]: delay = initial_delay for attempt in range(max_attempts): try: @@ -649,10 +809,10 @@ def _execute_with_retry_sync( def _execute_sql_sync( sql: str, connection_string: str, - db_type: Optional[str], + db_type: str | None, timeout_seconds: int = 120, - engine: Optional[sa.Engine] = None, -) -> List[Dict[str, Any]]: + engine: sa.Engine | None = None, +) -> list[dict[str, Any]]: engine = _resolve_sync_engine(connection_string, override_engine=engine) with engine.connect() as conn: timeout_ms = timeout_seconds * 1000 @@ -665,6 +825,15 @@ def _execute_sql_sync( conn.execute(sa.text(f"SET statement_timeout = {timeout_ms}")) except Exception: pass + else: + # Dialect-specific timeout statement (DEV-1551). The base + # SqlDialect returns None — only dialects with a custom + # statement_timeout_sql (currently SnowflakeDialect) emit a + # SET. + from slayer.sql.dialects import dialect_for_ds_type # noqa: PLC0415 + timeout_sql = dialect_for_ds_type(db_type).statement_timeout_sql(timeout_seconds) + if timeout_sql: + conn.execute(sa.text(timeout_sql)) result = conn.execute(sa.text(sql)) columns = list(result.keys()) return [dict(zip(columns, row)) for row in result.fetchall()] diff --git a/slayer/sql/dialects/__init__.py b/slayer/sql/dialects/__init__.py new file mode 100644 index 00000000..4b5f7af9 --- /dev/null +++ b/slayer/sql/dialects/__init__.py @@ -0,0 +1,114 @@ +"""DEV-1542: SLayer SQL dialect registry. + +Strategy-pattern dispatch: every dialect-specific SQL-generation quirk +lives on a subclass of ``SqlDialect`` (one file per Tier-1 dialect under +this package, Tier-2 dialects together in ``_tier2.py``). The registry +exposes two lookup functions: + +* ``get_dialect(sqlglot_name)`` — strict, raises ``KeyError`` on unknown + (preserves today's ``_build_explain_sql`` ``ValueError`` semantics). +* ``dialect_for_ds_type(ds_type)`` — lenient, falls back to + ``PostgresDialect`` (preserves today's + ``_DIALECT_MAP.get(ds_type or "", "postgres")`` semantics). +""" + +from __future__ import annotations + +from slayer.sql.dialects._tier2 import ( + DatabricksDialect, + OracleDialect, + PrestoDialect, + RedshiftDialect, + SparkDialect, + TrinoDialect, +) +from slayer.sql.dialects.base import SqlDialect +from slayer.sql.dialects.bigquery import BigqueryDialect +from slayer.sql.dialects.clickhouse import ClickhouseDialect +from slayer.sql.dialects.duckdb import DuckdbDialect +from slayer.sql.dialects.mysql import MysqlDialect +from slayer.sql.dialects.postgres import PostgresDialect +from slayer.sql.dialects.snowflake import SnowflakeDialect +from slayer.sql.dialects.sqlite import SqliteDialect +from slayer.sql.dialects.tsql import TsqlDialect + + +__all__ = [ + "SqlDialect", + "SqliteDialect", + "PostgresDialect", + "DuckdbDialect", + "MysqlDialect", + "ClickhouseDialect", + "TsqlDialect", + "SnowflakeDialect", + "BigqueryDialect", + "RedshiftDialect", + "TrinoDialect", + "PrestoDialect", + "DatabricksDialect", + "SparkDialect", + "OracleDialect", + "get_dialect", + "dialect_for_ds_type", +] + + +_ALL_DIALECTS: tuple[SqlDialect, ...] = ( + SqliteDialect(), + PostgresDialect(), + DuckdbDialect(), + MysqlDialect(), + ClickhouseDialect(), + TsqlDialect(), + SnowflakeDialect(), + BigqueryDialect(), + RedshiftDialect(), + TrinoDialect(), + PrestoDialect(), + DatabricksDialect(), + SparkDialect(), + OracleDialect(), +) + + +_BY_SQLGLOT_NAME: dict[str, SqlDialect] = { + d.sqlglot_name: d for d in _ALL_DIALECTS +} + + +_BY_DS_TYPE: dict[str, SqlDialect] = { + alias: d for d in _ALL_DIALECTS for alias in d.ds_type_aliases +} + + +def get_dialect(sqlglot_name: str) -> SqlDialect: + """Strict lookup by sqlglot name. Raises ``KeyError`` on unknown. + + Preserves today's ``_build_explain_sql`` semantics at + ``query_engine.py:132`` — an unrecognised dialect string raises + rather than silently falling back. Internal callers resolve via + ``dialect_for_ds_type`` first, so this branch is unreachable in + normal flow but the strict error is preserved as defence in depth. + """ + return _BY_SQLGLOT_NAME[sqlglot_name] + + +def dialect_for_ds_type(ds_type: str | None) -> SqlDialect: + """Lenient lookup by datasource-config ``type`` string. + + Falls back to ``PostgresDialect`` for ``None``, empty, OR unknown + ds-types. Matches today's + ``_DIALECT_MAP.get(ds_type or "", "postgres")`` semantics exactly. + """ + return _BY_DS_TYPE.get(ds_type or "", _BY_SQLGLOT_NAME["postgres"]) + + +# DEV-1686: quote reserved-word identifiers. Union the curated reserved-word set +# into every dialect generator's RESERVED_KEYWORDS as soon as the registry is +# built, so any ``.sql(dialect=...)`` emission quotes reserved aliases / +# qualifiers / physical names. Imported here (after ``_ALL_DIALECTS`` is defined) +# so the installer runs before any SQL is generated; idempotent. +from slayer.sql.reserved_keywords import install_reserved_keywords # noqa: E402 + +install_reserved_keywords() diff --git a/slayer/sql/dialects/_tier2.py b/slayer/sql/dialects/_tier2.py new file mode 100644 index 00000000..c49b4cb3 --- /dev/null +++ b/slayer/sql/dialects/_tier2.py @@ -0,0 +1,157 @@ +"""DEV-1542: Tier-2 dialect subclasses (no live integration tests). + +Each Tier-2 dialect differs from ``SqlDialect``'s Postgres-shaped defaults +only in scalar config (sqlglot name, EXPLAIN prefix/postfix, log10/log2 +native flags) — no SQL-shape logic. They live together in one file +because they're data-shaped, not logic-shaped. + +Values codify today's behaviour from +``query_engine.py:_EXPLAIN_PREFIX`` / ``_EXPLAIN_POSTFIX`` and +``generator.py:_LOG10_NATIVE_DIALECTS`` / ``_LOG2_NATIVE_DIALECTS``. + +Two dialects were promoted out of this file to their own Tier 1 modules: + +* ``BigqueryDialect`` — see ``slayer/sql/dialects/bigquery.py`` (alias + mangling for joined-column references and per-statement quota tweaks). +* ``SnowflakeDialect`` (DEV-1551) — see ``slayer/sql/dialects/snowflake.py`` + (connection URL builder, ``creator=`` engine bridge, per-connection + session overrides, statement timeout, cursor type-code map). +""" + +from __future__ import annotations + +from collections.abc import Callable + +from sqlglot import exp + +from slayer.sql.dialects.base import SqlDialect + + +class RedshiftDialect(SqlDialect): + sqlglot_name: str = "redshift" + ds_type_aliases: frozenset[str] = frozenset({"redshift"}) + explain_prefix: str | None = "EXPLAIN" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = False + + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """DEV-1708: Redshift (Postgres 8.0.2 fork) has no ``IS NOT DISTINCT + FROM`` — emit the expanded ``a = b OR (a IS NULL AND b IS NULL)``.""" + return self._expanded_null_safe_eq(left, right) + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Redshift: ``APPROXIMATE COUNT(DISTINCT x)`` (keyword prefix).""" + return parse(f"APPROXIMATE COUNT(DISTINCT {col_sql})") + + +class TrinoDialect(SqlDialect): + sqlglot_name: str = "trino" + ds_type_aliases: frozenset[str] = frozenset({"trino"}) + explain_prefix: str | None = "EXPLAIN ANALYZE" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Trino: native ``approx_distinct(x)`` aggregate.""" + return parse(f"approx_distinct({col_sql})") + + +class PrestoDialect(SqlDialect): + sqlglot_name: str = "presto" + # Athena uses the Presto dialect via this alias. + ds_type_aliases: frozenset[str] = frozenset({"presto", "athena"}) + explain_prefix: str | None = "EXPLAIN ANALYZE" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Presto: native ``approx_distinct(x)`` aggregate.""" + return parse(f"approx_distinct({col_sql})") + + +class DatabricksDialect(SqlDialect): + sqlglot_name: str = "databricks" + ds_type_aliases: frozenset[str] = frozenset({"databricks"}) + explain_prefix: str | None = "EXPLAIN EXTENDED" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Databricks: native ``approx_count_distinct(x)`` aggregate.""" + return parse(f"approx_count_distinct({col_sql})") + + +class SparkDialect(SqlDialect): + sqlglot_name: str = "spark" + ds_type_aliases: frozenset[str] = frozenset({"spark"}) + explain_prefix: str | None = "EXPLAIN EXTENDED" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Spark: native ``approx_count_distinct(x)`` aggregate.""" + return parse(f"approx_count_distinct({col_sql})") + + +class OracleDialect(SqlDialect): + sqlglot_name: str = "oracle" + ds_type_aliases: frozenset[str] = frozenset({"oracle"}) + explain_prefix: str | None = "EXPLAIN PLAN FOR" + explain_postfix: str = "" + # Oracle has neither LOG10 nor LOG2 as single-arg functions — keep + # the canonical 2-arg LOG(base, x) form. + log10_native: bool = False + log2_native: bool = False + + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """DEV-1708: Oracle has no ``IS NOT DISTINCT FROM`` — emit the expanded + ``a = b OR (a IS NULL AND b IS NULL)``.""" + return self._expanded_null_safe_eq(left, right) + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Oracle: native ``APPROX_COUNT_DISTINCT(x)`` aggregate. + + Built as an ``exp.Anonymous`` because sqlglot's Oracle dialect + re-emits a parsed ``APPROX_COUNT_DISTINCT`` as ``APPROX_DISTINCT`` + (its Presto-family canonical form), which is not an Oracle function. + """ + return exp.Anonymous(this="APPROX_COUNT_DISTINCT", expressions=[parse(col_sql)]) diff --git a/slayer/sql/dialects/base.py b/slayer/sql/dialects/base.py new file mode 100644 index 00000000..ea253605 --- /dev/null +++ b/slayer/sql/dialects/base.py @@ -0,0 +1,620 @@ +"""DEV-1542: SqlDialect strategy base class. + +Every dialect-specific SQL-generation quirk lives on a subclass of +``SqlDialect``. The base class itself is a fully concrete Postgres-shaped +default — concrete dialects (``SqliteDialect``, ``TsqlDialect``, ...) +override only the methods whose behaviour differs. + +The class is a Pydantic ``BaseModel`` with ``frozen=True`` so registry +singletons can't drift. Method overrides happen via regular subclassing — +fields use class-level defaults (``sqlglot_name: str = "postgres"``). +""" + +from __future__ import annotations + +from typing import TYPE_CHECKING, Any +from collections.abc import Callable + +from pydantic import BaseModel, ConfigDict +from sqlglot import exp + +from slayer.core.enums import TimeGranularity + +if TYPE_CHECKING: + import sqlalchemy as sa + + from slayer.core.models import DatasourceConfig + + +# --------------------------------------------------------------------------- +# Granularity & duration mapping (used by default impls of date_trunc / +# time-offset / interval helpers) +# --------------------------------------------------------------------------- + +_GRANULARITY_TO_DATE_TRUNC = { + TimeGranularity.SECOND: "second", + TimeGranularity.MINUTE: "minute", + TimeGranularity.HOUR: "hour", + TimeGranularity.DAY: "day", + TimeGranularity.WEEK: "week", + TimeGranularity.MONTH: "month", + TimeGranularity.QUARTER: "quarter", + TimeGranularity.YEAR: "year", +} + +_WINDOW_UNIT_SQL = { + "y": "year", + "m": "month", + "w": "week", + "d": "day", + "h": "hour", + "min": "minute", + "s": "second", +} + + +def _granularity_to_unit(granularity: str) -> str: + """Map a granularity string to a SQL INTERVAL unit name. + + Quarter has no INTERVAL unit on most dialects — callers normalise to + ``MONTH`` with the value multiplied by 3 before invoking the default. + Week stays ``WEEK`` (Postgres / MySQL / ClickHouse / BigQuery all + accept it). SQLite + T-SQL override the whole method. + """ + return { + "year": "YEAR", + "month": "MONTH", + "day": "DAY", + "quarter": "MONTH", # caller multiplies by 3 + "week": "WEEK", + # DEV-1572: a one-period shift of a Sunday-week is just one week. + "week_sunday": "WEEK", + "hour": "HOUR", + "minute": "MINUTE", + "second": "SECOND", + }.get(granularity, granularity.upper()) + + +# --------------------------------------------------------------------------- +# Shared variance-decomposition formula (used by MySQL + T-SQL overrides +# of build_covar_2arg). +# --------------------------------------------------------------------------- + + +def _build_covar_decomposition( + *, + col_sql: str, + other_sql: str, + agg: str, + var_fn_samp: str, + var_fn_pop: str, + stddev_fn: str, + parse: Callable[[str], exp.Expression], +) -> exp.Expression: + """Variance-decomposition formula for corr / covar_samp / covar_pop. + + ``cov(x, y) = (Var(x+y) - Var(x) - Var(y)) / 2`` + ``corr(x, y) = cov_samp(x, y) / (Stddev(x) * Stddev(y))`` + + Used by MySQL and T-SQL where the native CORR / COVAR_SAMP / COVAR_POP + functions are absent. Both columns are NULL-guarded against each other + so rows where either leg is NULL are excluded from all variance calls. + + Uses ``exp.Anonymous`` for aggregate calls to bypass sqlglot's MySQL + rewrite that aliases VAR_SAMP → VARIANCE = VAR_POP (silently wrong). + """ + var_fn = var_fn_samp if agg in ("covar_samp", "corr") else var_fn_pop + + x_guarded = parse( + f"CASE WHEN ({other_sql}) IS NOT NULL THEN ({col_sql}) END" + ) + y_guarded = parse( + f"CASE WHEN ({col_sql}) IS NOT NULL THEN ({other_sql}) END" + ) + xy_sum = exp.Add(this=x_guarded, expression=y_guarded) + + var_xy = exp.Anonymous(this=var_fn, expressions=[xy_sum]) + var_x = exp.Anonymous(this=var_fn, expressions=[x_guarded]) + var_y = exp.Anonymous(this=var_fn, expressions=[y_guarded]) + + covar = exp.Div( + this=exp.Paren(this=exp.Sub( + this=exp.Sub(this=var_xy, expression=var_x), + expression=var_y, + )), + expression=exp.Literal.number(2), + ) + + if agg != "corr": + return covar + + std_x = exp.Anonymous(this=stddev_fn, expressions=[x_guarded]) + std_y = exp.Anonymous(this=stddev_fn, expressions=[y_guarded]) + raw_denom = exp.Paren(this=exp.Mul(this=std_x, expression=std_y)) + denom = exp.Anonymous( + this="NULLIF", expressions=[raw_denom, exp.Literal.number(0)] + ) + return exp.Div(this=covar, expression=denom) + + +# --------------------------------------------------------------------------- +# SqlDialect — base class with Postgres-shaped defaults +# --------------------------------------------------------------------------- + + +class SqlDialect(BaseModel): + """Strategy class encapsulating one database's SQL-generation quirks. + + The base class IS the Postgres-shaped default. Concrete dialects + (``SqliteDialect``, ``TsqlDialect``, ...) subclass and override only + what differs. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True, frozen=True) + + sqlglot_name: str = "postgres" + ds_type_aliases: frozenset[str] = frozenset() + explain_prefix: str | None = "EXPLAIN" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + # ------------------------------------------------------------------ + # Null-safe equality (DEV-1708 / Codex F2) + # ------------------------------------------------------------------ + + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """A null-safe equality (``left`` and ``right`` compare equal, and two + NULLs compare equal) for the cross-model grain join-back's ``ON`` clause. + + Base (Postgres-family) uses sqlglot's ``NullSafeEQ`` → ``IS NOT DISTINCT + FROM``, which sqlglot also transpiles correctly for DuckDB / Snowflake / + BigQuery / Trino / Databricks / ClickHouse. MySQL overrides to ``<=>``; + SQLite to bare ``IS``; dialects with no native form (T-SQL / Oracle / + Redshift) to the expanded ``a = b OR (a IS NULL AND b IS NULL)``. + """ + return exp.NullSafeEQ(this=left, expression=right) + + @staticmethod + def _expanded_null_safe_eq( + left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """``left = right OR (left IS NULL AND right IS NULL)`` — the portable + expansion for dialects without a native null-safe equality operator.""" + eq = exp.EQ(this=left.copy(), expression=right.copy()) + both_null = exp.And( + this=exp.Is(this=left.copy(), expression=exp.Null()), + expression=exp.Is(this=right.copy(), expression=exp.Null()), + ) + return exp.paren(exp.Or(this=eq, expression=exp.paren(both_null))) + + # ------------------------------------------------------------------ + # Date-trunc / time arithmetic + # ------------------------------------------------------------------ + + def build_date_trunc( + self, + col_expr: exp.Expression, + granularity: TimeGranularity, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Default: ``DATE_TRUNC('unit', col)`` via sqlglot's ``exp.DateTrunc``. + + Non-bare-column / non-cast operands are wrapped in + ``CAST(... AS TIMESTAMP)`` so Postgres can pick the right + ``date_trunc`` overload — preserving today's + ``generator.py:_build_date_trunc`` behaviour. + """ + if granularity == TimeGranularity.WEEK_SUNDAY: + # DEV-1572: Sunday-anchored week = Monday-week of (col + 1 day), + # shifted back 1 day. This is Metabase's own reference formula and + # reuses each dialect's existing (Monday-based) WEEK truncation, so + # WEEK_SUNDAY's correctness tracks WEEK's per dialect. BigQuery — + # whose native WEEK is Sunday — overrides this to emit + # ``DATE_TRUNC(col, WEEK(SUNDAY))`` directly. + shifted = self.build_time_offset_expr( + col_expr=col_expr, offset=1, granularity="day", + ) + monday = self.build_date_trunc( + col_expr=shifted, granularity=TimeGranularity.WEEK, parse=parse, + ) + return self.build_time_offset_expr( + col_expr=monday, offset=-1, granularity="day", + ) + gran_str = _GRANULARITY_TO_DATE_TRUNC.get(granularity, granularity.value) + if not isinstance(col_expr, (exp.Column, exp.Cast)): + col_expr = exp.Cast(this=col_expr, to=exp.DataType.build("TIMESTAMP")) + return exp.DateTrunc(this=col_expr, unit=exp.Literal.string(gran_str)) + + def build_time_offset_expr( + self, + col_expr: exp.Expression, + offset: int, + granularity: str, + ) -> exp.Expression: + """Default: ``col ± INTERVAL N UNIT`` via ``exp.Add`` / ``exp.Sub``. + + Granularity normalization (preserved across every dialect): + ``quarter`` becomes ``val * 3`` of ``MONTH``. SQLite additionally + normalises ``week`` to ``val * 7`` of ``days`` — that branch lives + on ``SqliteDialect`` since other dialects accept ``WEEK`` natively. + """ + unit = _granularity_to_unit(granularity) + val = offset * 3 if granularity == "quarter" else offset + if val >= 0: + return exp.Add( + this=col_expr, + expression=exp.Interval( + this=exp.Literal.number(val), + unit=exp.Var(this=unit), + ), + ) + return exp.Sub( + this=col_expr, + expression=exp.Interval( + this=exp.Literal.number(-val), + unit=exp.Var(this=unit), + ), + ) + + def duration_interval_exprs( + self, + parts: list[tuple[int, str]], + sign: int = 1, + ) -> list[exp.Expression]: + """Default: one ``exp.Interval`` per (amount, unit) pair. + + The Add-vs-Sub direction is decided by ``add_intervals_expr`` from + its own ``sign`` arg, so the Interval values themselves stay + positive at this layer. sqlglot transpiles each single-unit + interval per dialect (MySQL/ClickHouse/BigQuery all accept + ``INTERVAL N UNIT``). + """ + return [ + exp.Interval( + this=exp.Literal.number(amount), + unit=exp.Var(this=_WINDOW_UNIT_SQL[unit].upper()), + ) + for amount, unit in parts + ] + + def add_intervals_expr( + self, + expr: exp.Expression, + intervals: list[exp.Expression], + sign: int = 1, + ) -> exp.Expression: + """Default: fold ``exp.Add`` (sign>=0) or ``exp.Sub`` (sign<0) over + the interval list.""" + op_cls = exp.Add if sign >= 0 else exp.Sub + result = expr + for iv in intervals: + result = op_cls(this=result, expression=iv) + return result + + # ------------------------------------------------------------------ + # Median / percentile / stat aggregates + # ------------------------------------------------------------------ + + def build_median( + self, + inner: exp.Expression, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Default: ``PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY inner)``.""" + inner_sql = inner.sql(dialect=self.sqlglot_name) + return parse(f"PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY {inner_sql})") + + def build_percentile( + self, + p_str: str, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Default: ``PERCENTILE_CONT(p) WITHIN GROUP (ORDER BY col_sql)``. + + ``p_str`` is the original pre-validated string the user provided — + not a float — so ``0.50`` / ``1`` / scientific notation are + preserved verbatim (Codex finding #3). + """ + return parse( + f"PERCENTILE_CONT({p_str}) WITHIN GROUP (ORDER BY {col_sql})" + ) + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Default: exact ``COUNT(DISTINCT col)`` fallback (DEV-1595). + + Backends with no native approximate-distinct function (Postgres / + SQLite / MySQL) fall back to the exact count, which is *more* + accurate than an approximation — consistent with the "no + approximate SQL" rule. Native-supporting dialects override this to + emit their own approximate-distinct function (DuckDB + ``approx_count_distinct``, ClickHouse ``uniq``, …). + """ + return parse(f"COUNT(DISTINCT {col_sql})") + + def build_stat_agg_1arg( + self, + agg_name: str, + col_expr: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Default: emit canonical Postgres-style name and let sqlglot + transpile per dialect (e.g. var_samp → VARIANCE on SQLite/DuckDB).""" + return parse(f"{agg_name.upper()}({col_expr})") + + def build_covar_2arg( + self, + agg_name: str, + col_sql: str, + other_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Default: native ``CORR(x, y)`` / ``COVAR_SAMP(x, y)`` / + ``COVAR_POP(x, y)``.""" + return parse(f"{agg_name.upper()}({col_sql}, {other_sql})") + + # ------------------------------------------------------------------ + # Log-alias rewrite + # ------------------------------------------------------------------ + + def should_use_native_log(self, base: int) -> bool: + """Whether ``log{N}(x)`` should be emitted as the dialect's native + single-arg function (vs the canonical 2-arg ``LOG(N, x)``). + + Defaults: log10 native = True (every Tier-1+2 dialect except + Oracle), log2 native = True (Postgres-shaped baseline). Concrete + dialects override via the ``log10_native`` / ``log2_native`` + fields. + """ + if base == 10: + return self.log10_native + if base == 2: + return self.log2_native + return False + + # ------------------------------------------------------------------ + # AST rewrite hook + per-connection UDF registration + # ------------------------------------------------------------------ + + def rewrite_parsed_ast(self, tree: exp.Expression) -> exp.Expression: + """Default: identity. SQLite overrides to rewrite JSONExtract to + the function-call form (DEV-1331).""" + return tree + + def rewrite_target_ast(self, tree: exp.Expression) -> exp.Expression: + """Default: identity. Target-keyed AST rewrite (DEV-1576). + + Applied in ``SQLGenerator._parse`` using the generator's **target** + dialect (``self._dialect``), independent of the parse dialect. This is + the place for output-shaping a dialect needs that the input-side + ``rewrite_parsed_ast`` cannot do: formula/measure expressions are + canonically parsed as Postgres regardless of target, so a + ``rewrite_parsed_ast`` override would fire for every backend. + + ``PostgresDialect`` overrides this to wrap the first argument of a + 2-arg ``ROUND`` in a numeric ``CAST`` (Postgres has no + ``round(double precision, integer)`` — only ``round(numeric, int)``). + SQLite / DuckDB round ``DOUBLE`` natively, so they keep the identity. + """ + return tree + + def emit_outer_wrap( + self, + *, + inner_sql: str, + public: list[str], + order: exp.Expression | None, + limit: exp.Expression | None, + offset_arg: exp.Expression | None, + parse: Callable[[str], exp.Expression] | None = None, + ) -> str: + """Emit the DEV-1444 outer-projection wrap around ``inner_sql``. + + Contract: ``inner_sql`` is the inner SELECT with **trailing + pagination already detached** (``SQLGenerator._build_outer_wrap`` + owns the strip). ``order`` / ``limit`` / ``offset_arg`` are the + detached sqlglot AST nodes the caller pulled off the inner; the + hook re-emits them on the outer statement. + + ``parse`` is the generator's ``_parse`` callback when the + generator is the caller (``SQLGenerator._build_outer_wrap``). + T-SQL needs it to preserve SLayer-specific AST rewrites (LOG10/ + LOG2 alias preservation, SQLite JSONExtract function-form) when + the override re-parses ``inner_sql`` to detach the WITH clause. + The base impl ignores it because it embeds ``inner_sql`` verbatim + (no re-parse, no rewrite drift). + + Base impl (Postgres-shaped, used by every dialect except T-SQL):: + + SELECT "alias1", "alias2" + FROM () AS _outer + ORDER BY ... LIMIT N OFFSET M + + Identifier quoting on the public-alias list is driven by sqlglot + via ``self.sqlglot_name`` — backticks on MySQL/BigQuery, brackets + on T-SQL (the override only changes the CTE-hoist shape, not the + quoting), ANSI double quotes on Postgres/SQLite/DuckDB/... + (DEV-1571 Bug 3). + + T-SQL's ``WITH``-must-be-statement-prefix rule means + ``TsqlDialect`` overrides this method to lift the inner top-level + CTEs to the outer statement (DEV-1571 Bug 1). + + ORDER BY may carry inner-CTE qualifiers like ``_base."col"`` from + ``_assemble_combined_sql``; those don't resolve at the outer- + wrapper scope (only ``_outer`` is in scope). The base impl strips + every Column's ``table`` qualifier so the outer scope can resolve + each column by its bare alias name (DEV-1444 behaviour preserved). + """ + del parse # base impl embeds inner_sql verbatim; no re-parse needed. + col_sep = ",\n " + outer_select = col_sep.join( + exp.Identifier(this=a, quoted=True).sql(dialect=self.sqlglot_name) + for a in public + ) + base = ( + f"SELECT\n {outer_select}\n" + f"FROM (\n{inner_sql.rstrip()}\n) AS _outer" + ) + if order is None and limit is None and offset_arg is None: + return base + out = base + if order is not None: + for col in order.find_all(exp.Column): + if col.args.get("table") is not None: + col.set("table", None) + out += "\n" + order.sql(dialect=self.sqlglot_name, pretty=True) + if limit is not None: + out += "\n" + limit.sql(dialect=self.sqlglot_name, pretty=True) + if offset_arg is not None: + out += "\n" + offset_arg.sql(dialect=self.sqlglot_name, pretty=True) + return out + + def rewrite_emitted_sql(self, sql: str) -> str: + """Default: identity. Post-pass string-level rewrite of the final + generator output. + + Symmetric companion to ``rewrite_parsed_ast`` (the input-side + hook): write-side, applied at the end of + ``SQLGenerator.generate()`` AFTER ``_apply_outer_projection_trim``. + + Contract: preserve query semantics. Suitable for alias renames, + identifier mangling/escape, dialect-quoting fixes. Do NOT change + query shape — use the typed ``build_*`` methods on this class for + that. + + Overrides today: ``BigqueryDialect`` mangles dotted aliases that + would otherwise be rejected by BigQuery's output column-name + grammar. + """ + return sql + + def decode_result_keys( + self, + rows: list[dict[str, Any]], + ) -> list[dict[str, Any]]: + """Default: identity. Reverse-pass on result-row keys to undo any + write-side mangling applied by ``rewrite_emitted_sql``. + + Called at the end of ``SlayerQueryEngine.execute()`` so consumers + always see SLayer's universal alias shape (``orders._count``, + ``orders.products.category``) regardless of which dialect a + query ran on. + + Overrides today: ``BigqueryDialect`` decodes the ``___`` mangling + back to dots. + """ + return rows + + def register_udfs(self, dbapi_connection) -> None: + """Default: no-op. SQLite overrides to register Python aggregate + / scalar UDFs on every fresh connection.""" + return None + + # ------------------------------------------------------------------ + # EXPLAIN + # ------------------------------------------------------------------ + + def build_explain_sql(self, sql: str) -> str: + """Wrap ``sql`` in the dialect's EXPLAIN prefix/postfix pair. + + Raises ``ValueError`` when ``explain_prefix`` is ``None`` + (BigQuery — EXPLAIN unsupported). Preserves today's + ``query_engine.py:_build_explain_sql`` semantics. + """ + if self.explain_prefix is None: + raise ValueError( + f"EXPLAIN is not supported for dialect '{self.sqlglot_name}'. " + "Use dry_run=True to inspect the generated SQL instead." + ) + return f"{self.explain_prefix} {sql}{self.explain_postfix}" + + # ------------------------------------------------------------------ + # Engine / connection / runtime hooks + # + # These let a dialect carry its own runtime quirks (connection-string + # form, engine-creation bridge, per-connection session setup, per- + # statement timeout, cursor-type-code mapping) without spilling + # dialect-specific conditionals into ``slayer/sql/engine_factory.py`` + # or ``slayer/sql/client.py``. Defaults are all no-op — concrete + # dialects override what's relevant. + # ------------------------------------------------------------------ + + def build_connection_url( + self, + datasource: "DatasourceConfig", + ) -> str | None: + """Hook: dialect-specific connection-string builder. + + Returning ``None`` (the default) means: defer to + ``DatasourceConfig.get_connection_string()``'s standard branches + (sqlite / duckdb / tsql / generic URL form). SnowflakeDialect + overrides this to emit either the + ``snowflake://?connection_name=`` sentinel or the inline + ``snowflake-sqlalchemy`` URL. + """ + return None + + def build_engine( + self, + datasource: "DatasourceConfig", + *, + connection_string: str, + ) -> "sa.Engine | None": + """Hook: build a dialect-specific SQLAlchemy engine. + + Returning ``None`` (the default) means: ``engine_factory`` falls + back to ``sa.create_engine(connection_string, pool_pre_ping=True)``. + SnowflakeDialect overrides this when the sentinel URL is in play, + wiring the ``creator=`` kwarg to delegate to + ``snowflake.connector.connect(connection_name=...)``. + """ + return None + + def apply_session_overrides( + self, + dbapi_connection: Any, + datasource: "DatasourceConfig", + ) -> None: + """Hook: per-connection session setup (e.g. ``USE WAREHOUSE``). + + Called by ``engine_factory``'s ``connect`` event listener on every + new pooled connection. SnowflakeDialect overrides this to issue + ``USE WAREHOUSE / USE ROLE / USE DATABASE / USE SCHEMA`` from the + DatasourceConfig's typed fields. + """ + return None + + def statement_timeout_sql(self, timeout_seconds: int) -> str | None: + """Hook: SQL to set a per-statement timeout, or ``None`` if the + dialect doesn't expose one or the existing client.py path handles + it via a hardcoded branch (mysql / clickhouse / postgres). + + SnowflakeDialect returns + ``ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = N``. + """ + return None + + def map_cursor_type_code(self, type_code: int) -> str | None: + """Hook: dialect-specific cursor-type-code → SLayer category + (one of ``"number"``, ``"string"``, ``"time"``, ``"boolean"``). + + Returning ``None`` (the default) means: ``client._map_type_code`` + falls back to the Postgres OID map. SnowflakeDialect overrides + this to return the snowflake-connector ``FieldType`` integer + codes' mapping. + """ + return None diff --git a/slayer/sql/dialects/bigquery.py b/slayer/sql/dialects/bigquery.py new file mode 100644 index 00000000..558404cd --- /dev/null +++ b/slayer/sql/dialects/bigquery.py @@ -0,0 +1,188 @@ +"""BigQuery dialect — Tier 1. + +BigQuery is the one dialect today with output-shape logic on top of the +scalar config every other Tier-2 dialect has. It rejects column names +containing ``.`` (output schema names must match ``[A-Za-z_][A-Za-z0-9_]*``), +while SLayer's universal alias convention is dotted +(``orders._count``, ``orders.products.category``). This dialect mangles +``.`` -> ``___`` inside backticked aliases on the write side and decodes +``___`` -> ``.`` on the read side so the mangling is invisible to consumers. + +The ``___`` separator is chosen specifically because ``__`` is already +used by ``_query_as_model`` to flatten cross-model leaves (e.g. +``stores__name``); using a distinct sentinel keeps the two encodings +unambiguous. + +Per DEV-1542's "every dialect quirk lives behind a hook on +``SqlDialect``" rule, this file is BigQuery's home. The plain +``rewrite_emitted_sql`` / ``decode_result_keys`` hooks on the base class +have identity defaults; only ``BigqueryDialect`` (and ``TsqlDialect``, +DEV-1571) override them today. The shared encode/decode bijection lives +in :mod:`slayer.sql.naming` (DEV-1713) and is reused by both dialects — +only the regex anchor (backticks here, brackets in T-SQL) differs. +""" + +from __future__ import annotations + +import json +import re +from typing import TYPE_CHECKING, Any +from collections.abc import Callable + +import sqlalchemy as sa +from sqlglot import exp + +from slayer.core.enums import TimeGranularity +from slayer.sql.naming import decode_alias, encode_alias +from slayer.sql.dialects.base import SqlDialect + +if TYPE_CHECKING: + from slayer.core.models import DatasourceConfig + + +# --------------------------------------------------------------------------- +# Alias mangling — backtick-anchored regex (BigQuery's identifier quote) +# --------------------------------------------------------------------------- + + +# Backtick-quoted dotted alias. The pattern is constrained to identifier +# characters ``\w`` separated by dots so it can't accidentally span +# unrelated SQL between two unrelated backticks. ``re.ASCII`` keeps ``\w`` +# ASCII-only so stray Unicode word-chars in surrounding SQL don't widen +# the match accidentally. +# +# Caveats (documented constraint): +# - Table fully-qualified paths whose project name contains a hyphen +# (e.g. ``\`bigquery-public-data\`.thelook_ecommerce.orders``) are +# safe: the hyphen breaks ``\w``, so the regex doesn't match the +# backticked-project segment, and the inner ``thelook_ecommerce.orders`` +# isn't inside any backticks. +# - A fully backticked dotted path of word-only segments +# (``\`my_dataset.my_table\``) WOULD false-positive mangle. Users +# writing ``Column.sql`` for BigQuery must backtick segments +# individually (``\`my_dataset\`.\`my_table\``) to avoid this. See +# ``tests/dialects/test_bigquery.py::test_rewrite_emitted_sql_false_positive_on_single_backticked_dotted_path`` +# for the characterization pin. +_DOTTED_ALIAS_RE = re.compile(r"`(\w+(?:\.\w+)+)`", re.ASCII) + + +# --------------------------------------------------------------------------- +# BigqueryDialect — Tier 1 (has logic, not just scalar config) +# --------------------------------------------------------------------------- + + +class BigqueryDialect(SqlDialect): + """BigQuery output-alias mangling + scalar config. + + Promoted out of ``_tier2.py`` because it has logic + (``rewrite_emitted_sql`` / ``decode_result_keys`` overrides), not + just scalar config. ``_tier2.py``'s "data-shaped, no SQL-shape logic" + contract stays accurate for the remaining tier-2 dialects. + """ + + sqlglot_name: str = "bigquery" + ds_type_aliases: frozenset[str] = frozenset({"bigquery"}) + # BigQuery has no SQL-level EXPLAIN. + explain_prefix: str | None = None + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """BigQuery: native ``APPROX_COUNT_DISTINCT(x)`` aggregate.""" + return parse(f"APPROX_COUNT_DISTINCT({col_sql})") + + def build_date_trunc( + self, + col_expr: exp.Expression, + granularity: TimeGranularity, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """BigQuery override for WEEK_SUNDAY (DEV-1572). + + BigQuery's native ``DATE_TRUNC(x, WEEK)`` is already Sunday-based, so + the base class's generic +1d/-1d shift (which reuses a Monday-based + WEEK) would double-shift. Emit the native Sunday form + ``DATE_TRUNC(col, WEEK(SUNDAY))`` instead. + + Built as an ``exp.Anonymous`` because sqlglot (30.4.x) drops the + ``(SUNDAY)`` weekday modifier when re-emitting an ``exp.DateTrunc`` — + the anonymous call renders verbatim on the single final emission. + Non-column/non-cast operands are wrapped in ``CAST(... AS TIMESTAMP)`` + to mirror the base class's operand handling. Every other granularity + delegates to the base implementation. + """ + if granularity != TimeGranularity.WEEK_SUNDAY: + return super().build_date_trunc( + col_expr=col_expr, granularity=granularity, parse=parse, + ) + if not isinstance(col_expr, (exp.Column, exp.Cast)): + col_expr = exp.Cast(this=col_expr, to=exp.DataType.build("TIMESTAMP")) + week_sunday = exp.Anonymous(this="WEEK", expressions=[exp.var("SUNDAY")]) + return exp.Anonymous( + this="DATE_TRUNC", expressions=[col_expr, week_sunday], + ) + + def rewrite_emitted_sql(self, sql: str) -> str: + """Replace ``.`` with ``___`` inside backtick-quoted identifiers. + + Applied as a post-pass on the BigQuery dialect's final SQL so + emitted column aliases (``SELECT ... AS \\`orders._count\\``) and + references to those aliases + (``ORDER BY \\`orders._count\\``) comply with BigQuery's column-name + grammar. + """ + return _DOTTED_ALIAS_RE.sub( + lambda m: f"`{encode_alias(m.group(1))}`", sql + ) + + def decode_result_keys( + self, + rows: list[dict[str, Any]], + ) -> list[dict[str, Any]]: + """Reverse the BigQuery alias mangling on result-row keys so + consumers see SLayer's universal dotted alias shape regardless of + whether the query ran against BigQuery or another dialect.""" + return [{decode_alias(k): v for k, v in row.items()} for row in rows] + + def build_engine( + self, + datasource: "DatasourceConfig", + *, + connection_string: str, + ) -> "sa.Engine | None": + """Construct the SQLAlchemy engine with inline service-account JSON + when ``DatasourceConfig.credentials_json`` is set. + + ``sqlalchemy-bigquery`` accepts a ``credentials_info`` kwarg on + ``create_engine`` — a dict matching the service-account key file's + shape. Parse the JSON string into a dict and pass it through; the + BigQuery client builds credentials from it directly, no temp file + needed. When ``credentials_json`` is unset, return ``None`` so + ``engine_factory`` falls back to the default ``create_engine`` and + the BigQuery client picks up Application Default Credentials. + """ + credentials_json = datasource.credentials_json + if not credentials_json: + return None + try: + credentials_info = json.loads(credentials_json) + except json.JSONDecodeError as exc: + raise ValueError( + f"Datasource '{datasource.name}': credentials_json is not valid JSON: {exc}" + ) from exc + if not isinstance(credentials_info, dict): + raise ValueError( + f"Datasource '{datasource.name}': credentials_json must be a JSON object" + ) + return sa.create_engine( + connection_string, + credentials_info=credentials_info, + pool_pre_ping=True, + ) diff --git a/slayer/sql/dialects/clickhouse.py b/slayer/sql/dialects/clickhouse.py new file mode 100644 index 00000000..51265822 --- /dev/null +++ b/slayer/sql/dialects/clickhouse.py @@ -0,0 +1,52 @@ +"""DEV-1542: ClickhouseDialect. + +ClickHouse uses native ``median(x)`` directly and the parametric +``quantile(p)(x)`` form for percentile. CORR / COVAR_SAMP / COVAR_POP +are native. log10 and log2 are native. +""" + +from __future__ import annotations + +from collections.abc import Callable + +from sqlglot import exp + +from slayer.sql.dialects.base import SqlDialect + + +class ClickhouseDialect(SqlDialect): + sqlglot_name: str = "clickhouse" + ds_type_aliases: frozenset[str] = frozenset({"clickhouse"}) + explain_prefix: str | None = "EXPLAIN" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def build_median( + self, + inner: exp.Expression, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """ClickHouse: native ``median(x)`` aggregate.""" + inner_sql = inner.sql(dialect="clickhouse") + return parse(f"median({inner_sql})") + + def build_percentile( + self, + p_str: str, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """ClickHouse: parametric ``quantile(p)(x)`` syntax.""" + return parse(f"quantile({p_str})({col_sql})") + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """ClickHouse: native ``uniq(x)`` approximate-distinct aggregate.""" + return parse(f"uniq({col_sql})") diff --git a/slayer/sql/dialects/duckdb.py b/slayer/sql/dialects/duckdb.py new file mode 100644 index 00000000..c8a9e7da --- /dev/null +++ b/slayer/sql/dialects/duckdb.py @@ -0,0 +1,32 @@ +"""DEV-1542: DuckdbDialect. + +DuckDB shape matches Postgres: native DATE_TRUNC, native PERCENTILE_CONT +(emitted via sqlglot's QUANTILE_CONT translation), native CORR / COVAR / +log10 / log2. +""" + +from __future__ import annotations + +from collections.abc import Callable + +from sqlglot import exp + +from slayer.sql.dialects.base import SqlDialect + + +class DuckdbDialect(SqlDialect): + sqlglot_name: str = "duckdb" + ds_type_aliases: frozenset[str] = frozenset({"duckdb"}) + explain_prefix: str | None = "EXPLAIN ANALYZE" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """DuckDB: native ``approx_count_distinct(x)`` aggregate.""" + return parse(f"approx_count_distinct({col_sql})") diff --git a/slayer/sql/dialects/mysql.py b/slayer/sql/dialects/mysql.py new file mode 100644 index 00000000..c9fd6974 --- /dev/null +++ b/slayer/sql/dialects/mysql.py @@ -0,0 +1,99 @@ +"""DEV-1542: MysqlDialect. + +MySQL has no native ``PERCENTILE_CONT`` (``build_median`` / ``build_percentile`` +raise ``NotImplementedError``) and no native ``CORR`` / ``COVAR_SAMP`` / +``COVAR_POP`` (uses the variance-decomposition formula). + +``var_samp`` / ``var_pop`` need the ``exp.Anonymous`` workaround because +sqlglot's MySQL transpiler rewrites them to ``VARIANCE`` (which on MySQL +is actually ``VAR_POP`` — silently wrong sample variance). +""" + +from __future__ import annotations + +from collections.abc import Callable + +from sqlglot import exp + +from slayer.sql.dialects.base import SqlDialect, _build_covar_decomposition + + +class MysqlDialect(SqlDialect): + sqlglot_name: str = "mysql" + ds_type_aliases: frozenset[str] = frozenset({"mysql", "mariadb"}) + explain_prefix: str | None = "EXPLAIN FORMAT=JSON" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def build_median( + self, + inner: exp.Expression, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + # ``mariadb`` resolves to this same dialect via ``ds_type_aliases``, + # so the error must NOT suggest "use MariaDB" — that would loop the + # user back here. Point them at a datasource with native percentile + # support or client-side computation instead. + raise NotImplementedError( + "Aggregation 'median' is not supported on MySQL: MySQL has no native " + "MEDIAN/PERCENTILE_CONT function and no Python UDF mechanism. " + "Use a datasource with native percentile support (Postgres, DuckDB, " + "ClickHouse, SQLite via UDF) or compute the value client-side." + ) + + def build_percentile( + self, + p_str: str, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + raise NotImplementedError( + "Aggregation 'percentile' is not supported on MySQL: " + "MySQL has no native PERCENTILE_CONT. " + "Use a datasource with native percentile support (Postgres, DuckDB, " + "ClickHouse, SQLite via UDF) or compute the value client-side." + ) + + def build_stat_agg_1arg( + self, + agg_name: str, + col_expr: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """MySQL override for ``var_samp`` / ``var_pop``. + + sqlglot's MySQL transpiler rewrites ``VAR_SAMP`` → ``VARIANCE`` + (which on MySQL is ``VAR_POP``) — silently wrong. Emit the + canonical MySQL names via ``exp.Anonymous`` to bypass sqlglot. + """ + if agg_name in {"var_samp", "var_pop"}: + return exp.Anonymous( + this=agg_name.upper(), + expressions=[parse(col_expr)], + ) + return super().build_stat_agg_1arg(agg_name, col_expr, parse=parse) + + def build_covar_2arg( + self, + agg_name: str, + col_sql: str, + other_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """MySQL has no native CORR / COVAR_* — use the + variance-decomposition formula with MySQL-native VAR_SAMP / + VAR_POP / STDDEV_SAMP names.""" + return _build_covar_decomposition( + col_sql=col_sql, + other_sql=other_sql, + agg=agg_name, + var_fn_samp="VAR_SAMP", + var_fn_pop="VAR_POP", + stddev_fn="STDDEV_SAMP", + parse=parse, + ) diff --git a/slayer/sql/dialects/postgres.py b/slayer/sql/dialects/postgres.py new file mode 100644 index 00000000..65909149 --- /dev/null +++ b/slayer/sql/dialects/postgres.py @@ -0,0 +1,52 @@ +"""DEV-1542: PostgresDialect. + +Postgres is the Postgres-shaped default made explicit. Native DATE_TRUNC, +PERCENTILE_CONT, CORR, COVAR_SAMP, COVAR_POP, native log10/log2. +""" + +from __future__ import annotations + + +from sqlglot import exp + +from slayer.sql.dialects.base import SqlDialect + + +def _cast_round_arg_to_numeric(node: exp.Expression) -> exp.Expression: + """Wrap the first arg of a 2-arg ``ROUND`` in ``CAST(... AS DECIMAL)``. + + Postgres has no ``round(double precision, integer)`` overload — only + ``round(numeric, integer)`` — so a 2-arg round over a DOUBLE expression + fails without an explicit numeric cast. 1-arg round (``round(double)``) + is fine and left alone. Idempotent: skips when the arg is already cast to + a numeric/decimal type. + """ + if not isinstance(node, exp.Round): + return node + decimals = node.args.get("decimals") + if decimals is None: # 1-arg round — no overload problem. + return node + inner = node.this + if isinstance(inner, exp.Cast): + cast_to = inner.to + if isinstance(cast_to, exp.DataType) and cast_to.this in ( + exp.DataType.Type.DECIMAL, + exp.DataType.Type.BIGDECIMAL, + ): + return node # already numeric-cast — idempotent. + node.set("this", exp.cast(inner.copy(), "DECIMAL")) + return node + + +class PostgresDialect(SqlDialect): + sqlglot_name: str = "postgres" + ds_type_aliases: frozenset[str] = frozenset({"postgres", "postgresql"}) + explain_prefix: str | None = "EXPLAIN ANALYZE" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def rewrite_target_ast(self, tree: exp.Expression) -> exp.Expression: + """DEV-1576: numeric-cast the first arg of every 2-arg ROUND so + ``round(double precision, int)`` becomes ``round(numeric, int)``.""" + return tree.transform(_cast_round_arg_to_numeric) diff --git a/slayer/sql/dialects/snowflake.py b/slayer/sql/dialects/snowflake.py new file mode 100644 index 00000000..2fe7a87a --- /dev/null +++ b/slayer/sql/dialects/snowflake.py @@ -0,0 +1,378 @@ +"""DEV-1551: SnowflakeDialect — Tier 1 promotion. + +Promoted from ``_tier2.py`` to its own file because Snowflake now carries +runtime quirks beyond the data-shaped Tier-2 set: + +* Connection-name URL form (``snowflake://?connection_name=``) + + ``sa.create_engine(..., creator=...)`` bridge to delegate to + ``snowflake.connector.connect(connection_name=...)`` for TOML-driven auth. +* Per-connection session overrides (``USE WAREHOUSE / USE ROLE / + USE DATABASE / USE SCHEMA``) from the typed ``DatasourceConfig`` fields. +* Per-statement timeout via ``ALTER SESSION SET + STATEMENT_TIMEOUT_IN_SECONDS``. +* Cursor type-code mapping for ``snowflake-connector-python``'s + ``FieldType`` enum (FIXED/REAL/TEXT/DATE/TIMESTAMP/VARIANT/...). + +SQL generation quirks (DATE_TRUNC, DATEADD, MEDIAN, PERCENTILE_CONT, +STDDEV_*, VAR_*, CORR, COVAR_*) all match the Postgres-shaped base — +sqlglot's snowflake dialect transpiles correctly. The only divergence +from the base is ``log2_native=False`` (Snowflake has no native LOG2). +""" + +from __future__ import annotations + +import re as _re +from typing import TYPE_CHECKING, Any +from collections.abc import Callable +from urllib.parse import quote + +import sqlalchemy as sa +import sqlalchemy.engine.url as _sa_url +from sqlglot import exp + +from slayer.sql.dialects.base import SqlDialect + +if TYPE_CHECKING: + from slayer.core.models import DatasourceConfig + + +# snowflake-connector-python's ``FieldType`` integer codes → SLayer +# category. Codes from ``snowflake.connector.constants.FieldType``; +# kept in sync with the consumer-facing categories used by +# ``slayer.sql.client._map_type_code``. +_SNOWFLAKE_TYPE_MAP: dict[int, str] = { + 0: "number", # FIXED (NUMBER / INT / DECIMAL) + 1: "number", # REAL (FLOAT / DOUBLE) + 2: "string", # TEXT (VARCHAR / STRING) + 3: "time", # DATE + 4: "time", # TIMESTAMP (legacy alias) + 5: "string", # VARIANT (semi-structured JSON / object) + 6: "time", # TIMESTAMP_LTZ + 7: "time", # TIMESTAMP_TZ + 8: "time", # TIMESTAMP_NTZ + 9: "string", # OBJECT + 10: "string", # ARRAY + 11: "string", # BINARY + 12: "time", # TIME + 13: "boolean", # BOOLEAN +} + + +# Sentinel-URL prefix used by ``build_connection_url`` when +# ``DatasourceConfig.connection_name`` is set. ``engine_factory`` +# bridges this URL to ``snowflake.connector.connect(connection_name=...)`` +# via ``sa.create_engine(..., creator=...)`` because snowflake-sqlalchemy +# has no ``connection_name=`` URL knob. +_CONNECTION_NAME_PREFIX = "snowflake://?connection_name=" + + +def _import_snowflake_connector(): + """Lazy import with an actionable install hint.""" + try: + import snowflake.connector # noqa: PLC0415 + return snowflake.connector + except ImportError as exc: + raise ImportError( + "Snowflake support requires the 'snowflake' extra: " + "pip install 'motley-slayer[snowflake]'" + ) from exc + + +def _import_snowflake_sqlalchemy_url(): + """Lazy import for the inline-URL form. Same install hint.""" + try: + from snowflake.sqlalchemy import URL # noqa: PLC0415 + return URL + except ImportError as exc: + raise ImportError( + "Snowflake support requires the 'snowflake' extra: " + "pip install 'motley-slayer[snowflake]'" + ) from exc + + +# Snowflake identifier characters allowed unquoted: letters, digits, +# underscores, dollar signs. Anything else (whitespace, semicolons, +# quotes, parentheses) is rejected up front rather than emitted into a +# ``USE WAREHOUSE/ROLE/DATABASE/SCHEMA`` statement. +_SAFE_SNOWFLAKE_IDENT = _re.compile(r"^[A-Za-z_][A-Za-z0-9_$]*$") + + +def _validate_unquoted_identifier(*, field: str, value: str) -> str: + """Reject Snowflake identifier values that aren't safe to emit unquoted. + + The values that flow into ``USE WAREHOUSE / ROLE / DATABASE / SCHEMA`` + statements come from typed ``DatasourceConfig`` fields. We deliberately + emit them **unquoted** so Snowflake's case-folding rules apply (the + common ``warehouse: compute_wh`` config matches the uppercase storage + ``COMPUTE_WH``); always-quoting would silently break those configs. + To keep that path safe we reject any character that could change the + statement's meaning (whitespace, semicolons, quotes, parens, dots). + """ + if not _SAFE_SNOWFLAKE_IDENT.match(value): + raise ValueError( + f"Invalid Snowflake identifier for DatasourceConfig.{field}: " + f"{value!r}. Only letters, digits, underscores, and '$' are allowed; " + f"the first character must be a letter or underscore. (If you have " + f"a quoted/mixed-case Snowflake object, create the datasource with " + f"the uppercase or canonical form.)" + ) + return value + + +def _is_connection_name_sentinel(connection_string: str) -> bool: + """True iff ``connection_string`` is the + ``snowflake://?connection_name=`` sentinel. + + Cross-source-of-truth: the user can land on this URL either via + ``DatasourceConfig.connection_name`` (typed field) or by typing the + URL into the ``connection_string`` field (e.g. the CLI form + ``slayer datasources create snowflake://?connection_name=default``). + Both paths must route through ``creator=``. + + The recognition is **strict**: the URL must contain exactly one + non-empty query parameter, ``connection_name``. Extra params like + ``warehouse=WH`` are rejected up front because ``build_engine`` only + forwards ``connection_name`` to ``snowflake.connector.connect`` — + silently accepting other params would route the user to the profile + defaults instead of the requested session context. To override + warehouse / role / database / schema, use the typed + ``DatasourceConfig`` fields; those fire via ``apply_session_overrides``. + """ + if not connection_string.startswith("snowflake://"): + return False + try: + url = _sa_url.make_url(connection_string) + except sa.exc.ArgumentError: + return False + name = url.query.get("connection_name") + if not name: + return False + # Reject sentinel URLs with extra query params — they would silently + # be ignored by build_engine's creator= bridge. + extra = {k for k, v in url.query.items() if k != "connection_name" and v != ""} + return not extra + + +def _extract_connection_name(connection_string: str) -> str: + """Parse the ``connection_name=`` value from the sentinel URL. + + ``sa.engine.url.make_url`` already URL-decodes query-string values, + so the value is returned as-is. (Calling ``unquote`` again would + double-decode literal percent-encoded text such as ``%2F`` in + profile names.) + """ + try: + url = _sa_url.make_url(connection_string) + except sa.exc.ArgumentError as exc: + raise ValueError( + f"Could not parse Snowflake sentinel URL: {connection_string!r}" + ) from exc + name = url.query.get("connection_name") + if not name: + raise ValueError( + f"Snowflake URL is missing the 'connection_name' query parameter: " + f"{connection_string!r}" + ) + return name + + +class SnowflakeDialect(SqlDialect): + """Snowflake dialect — Tier 1. + + Inherits Postgres-shaped SQL-generation defaults; sqlglot handles the + DATE_TRUNC / DATEADD / native-aggregate transpilation. Runtime + behavior (connection URL, engine creation, per-connection session + overrides, statement timeout, cursor type map) is encoded on the + class so ``engine_factory`` / ``client`` stay dialect-agnostic. + """ + + sqlglot_name: str = "snowflake" + ds_type_aliases: frozenset[str] = frozenset({"snowflake"}) + explain_prefix: str | None = "EXPLAIN USING JSON" + explain_postfix: str = "" + log10_native: bool = True + # No native LOG2 — falls through to canonical ``LOG(2, x)`` form. + log2_native: bool = False + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """Snowflake: native ``APPROX_COUNT_DISTINCT(x)`` aggregate.""" + return parse(f"APPROX_COUNT_DISTINCT({col_sql})") + + # ------------------------------------------------------------------ + # Connection URL / engine + # ------------------------------------------------------------------ + + def build_connection_url( + self, + datasource: DatasourceConfig, + ) -> str | None: + """Emit the sentinel URL when ``connection_name`` is set, otherwise + build the full snowflake-sqlalchemy URL from inline fields. + + Inline form requires ``host`` (the Snowflake account identifier). + ``warehouse`` and ``role`` populate the URL's query string. + """ + if datasource.connection_name: + return f"{_CONNECTION_NAME_PREFIX}{quote(datasource.connection_name, safe='')}" + if not datasource.host: + raise ValueError( + "Snowflake DatasourceConfig requires either 'connection_name' " + "(profile from ~/.snowflake/connections.toml) or inline credentials. " + "Set 'host' to the Snowflake account identifier (e.g. 'jp13593' or " + "'xy12345.us-east-1'), plus username/password — and optionally " + "database/schema_name/warehouse/role." + ) + URL = _import_snowflake_sqlalchemy_url() + kwargs: dict[str, str] = {"account": datasource.host} + if datasource.username: + kwargs["user"] = datasource.username + if datasource.password: + kwargs["password"] = datasource.password + if datasource.database: + kwargs["database"] = datasource.database + if datasource.schema_name: + kwargs["schema"] = datasource.schema_name + if datasource.warehouse: + kwargs["warehouse"] = datasource.warehouse + if datasource.role: + kwargs["role"] = datasource.role + # ``snowflake.sqlalchemy.URL`` returns a ``URL`` object; cast to + # ``str`` so the ``Optional[str]`` return-type annotation is honest. + return str(URL(**kwargs)) + + def build_engine( + self, + datasource: DatasourceConfig, + *, + connection_string: str, + ) -> sa.Engine | None: + """When the sentinel URL is in play, route through ``creator=`` + so ``snowflake.connector.connect(connection_name=...)`` drives + the auth path. Otherwise return None to let ``engine_factory`` + use the default ``sa.create_engine(connection_string)`` path + (snowflake-sqlalchemy understands the inline URL form natively). + """ + # Defence in depth: if the URL has the snowflake scheme AND a + # ``connection_name=`` query param BUT extra params, the strict + # sentinel check rejects it. Falling through to ``sa.create_engine`` + # would then either (a) trigger a confusing snowflake-sqlalchemy + # parse error or (b) silently connect to the profile defaults. + # Raise an actionable error pointing at the typed DatasourceConfig + # fields instead. + if connection_string.startswith("snowflake://"): + try: + parsed = _sa_url.make_url(connection_string) + except sa.exc.ArgumentError: + parsed = None + if parsed is not None and parsed.query.get("connection_name"): + extras = { + k for k, v in parsed.query.items() + if k != "connection_name" and v + } + if extras: + raise ValueError( + f"Snowflake sentinel URL must contain only the " + f"``connection_name`` query parameter — extra params " + f"{sorted(extras)!r} would be silently dropped by the " + f"snowflake-connector bridge. Set ``warehouse`` / " + f"``role`` / ``database`` / ``schema_name`` on the " + f"typed ``DatasourceConfig`` fields instead — those " + f"fire via ``apply_session_overrides`` on every " + f"pool checkout." + ) + if not _is_connection_name_sentinel(connection_string): + return None + name = _extract_connection_name(connection_string) + + def _create_snowflake_connection(): + sf = _import_snowflake_connector() + return sf.connect(connection_name=name) + + return sa.create_engine( + "snowflake://", + creator=_create_snowflake_connection, + pool_pre_ping=True, + ) + + def apply_session_overrides( + self, + dbapi_connection: Any, + datasource: DatasourceConfig, + ) -> None: + """Issue ``USE WAREHOUSE / USE ROLE / USE DATABASE / USE SCHEMA`` + in order on a fresh DBAPI connection. + + Order matters: + * USE WAREHOUSE first — some accounts require an active + warehouse before USE SCHEMA can resolve. + * USE ROLE second — the role can scope what databases/schemas + are visible. + * USE DATABASE before USE SCHEMA — bare schema names resolve + against the current database. + """ + if not any(( + datasource.warehouse, + datasource.role, + datasource.database, + datasource.schema_name, + )): + return + # Validate every value up front; reject anything that isn't a + # safe-to-emit-unquoted Snowflake identifier (catches embedded + # semicolons, quotes, whitespace). + warehouse = ( + _validate_unquoted_identifier(field="warehouse", value=datasource.warehouse) + if datasource.warehouse else None + ) + role = ( + _validate_unquoted_identifier(field="role", value=datasource.role) + if datasource.role else None + ) + database = ( + _validate_unquoted_identifier(field="database", value=datasource.database) + if datasource.database else None + ) + schema_name = ( + _validate_unquoted_identifier(field="schema_name", value=datasource.schema_name) + if datasource.schema_name else None + ) + cur = dbapi_connection.cursor() + try: + # Order: USE ROLE first — role determines warehouse / database + # privileges. A role granted via ``DatasourceConfig.role`` that + # has access to a warehouse the profile's default role doesn't + # see would otherwise fail at USE WAREHOUSE. + if role: + cur.execute(f"USE ROLE {role}") + if warehouse: + cur.execute(f"USE WAREHOUSE {warehouse}") + if database: + cur.execute(f"USE DATABASE {database}") + if schema_name: + cur.execute(f"USE SCHEMA {schema_name}") + finally: + cur.close() + + # ------------------------------------------------------------------ + # Runtime statement hooks + # ------------------------------------------------------------------ + + def statement_timeout_sql(self, timeout_seconds: int) -> str | None: + """``ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = N``. + + Per-session setting; takes effect for every subsequent statement + on the same connection until the connection is closed or the + setting is reset. + """ + return f"ALTER SESSION SET STATEMENT_TIMEOUT_IN_SECONDS = {timeout_seconds}" + + def map_cursor_type_code(self, type_code: int) -> str | None: + """Map a snowflake-connector ``FieldType`` integer code to a + SLayer category. Returns ``None`` for unknown codes so the caller + can fall through to a default rather than mis-classify.""" + return _SNOWFLAKE_TYPE_MAP.get(type_code) diff --git a/slayer/sql/dialects/sqlite.py b/slayer/sql/dialects/sqlite.py new file mode 100644 index 00000000..ae4957cd --- /dev/null +++ b/slayer/sql/dialects/sqlite.py @@ -0,0 +1,564 @@ +"""DEV-1542: SqliteDialect + the SQLite-specific helpers it depends on. + +This module folds in the content previously in ``slayer/sql/sqlite_dialect.py`` +(the ``rewrite_sqlite_json_extract`` AST rewrite) and +``slayer/sql/sqlite_udfs.py`` (the Python aggregate/scalar UDFs registered +on every fresh SQLite connection). + +The helpers are module-level — ``rewrite_sqlite_json_extract`` and +``register_sqlite_udfs`` and the ``_*Agg`` classes are directly +importable (used by ``tests/test_sqlite_json_extract.py`` and +``tests/test_sqlite_udfs.py``). ``SqliteDialect`` is a thin wrapper that +delegates to them through the ``SqlDialect`` interface. +""" + +from __future__ import annotations + +import math +from collections.abc import Callable + +from sqlglot import exp + +from slayer.core.enums import TimeGranularity +from slayer.sql.dialects.base import SqlDialect + + +# =========================================================================== +# JSON-extract AST rewrite (DEV-1331; was slayer/sql/sqlite_dialect.py) +# =========================================================================== + + +def rewrite_sqlite_json_extract(node: exp.Expression) -> exp.Expression: + """Rewrite every ``exp.JSONExtract`` in the tree rooted at ``node`` to the + function-call form. + + sqlglot's default SQLite generator emits ``exp.JSONExtract`` as + ``col -> '$.path'``. In SQLite the ``->`` operator returns the + JSON-typed form (e.g. ``'"Owned"'`` with literal quotes), whereas + ``json_extract`` and ``->>`` (``exp.JSONExtractScalar``) return the + unquoted scalar. The mismatch silently breaks ``CASE WHEN`` / + equality matches against bare-string literals. + + Returns the (possibly new) root node — callers must use the return + value because ``node`` itself may be a ``JSONExtract`` (e.g. when + parsing a ``Column.sql`` whose entire expression is + ``json_extract(col, path)``), in which case ``Expression.replace`` + is a no-op and a fresh root must be returned. Non-root rewrites + happen in place. + + Loops to a fixed point so nested forms like + ``json_extract(json_extract(j, '$.outer'), '$.inner')`` get + rewritten at every level. + """ + while True: + if isinstance(node, exp.JSONExtract): + node = _to_anonymous(node) + continue + je = node.find(exp.JSONExtract) + if je is None: + return node + je.replace(_to_anonymous(je)) + + +def _to_anonymous(je: exp.JSONExtract) -> exp.Anonymous: + return exp.Anonymous( + this="JSON_EXTRACT", + expressions=[je.this, je.expression], + ) + + +# =========================================================================== +# Python aggregate / scalar UDFs (DEV-1317 / DEV-1337; was sqlite_udfs.py) +# =========================================================================== +# SQLite has a much smaller built-in math/stat catalog than Postgres, +# DuckDB, MySQL, or ClickHouse. To bring SQLite to per-row and +# per-aggregate parity, this section registers Python implementations +# on every new SQLite connection via SQLAlchemy's ``connect`` event. + + +# --------------------------------------------------------------------------- +# Median / percentile (existing — unchanged from sqlite_udfs.py) +# --------------------------------------------------------------------------- + + +class _MedianAgg: + """1-arg median: average of the two middle values for even N.""" + + def __init__(self) -> None: + self._vals: list[float] = [] + + def step(self, value) -> None: + if value is not None: + self._vals.append(value) + + def finalize(self) -> float | None: + if not self._vals: + return None + s = sorted(self._vals) + n = len(s) + mid = n // 2 + if n % 2: + return s[mid] + return (s[mid - 1] + s[mid]) / 2.0 + + +class _PercentileContAgg: + """2-arg PERCENTILE_CONT(value, p): linear interpolation, matches Postgres.""" + + def __init__(self) -> None: + self._vals: list[float] = [] + self._p: float | None = None + + def step(self, value, p) -> None: + if p is not None: + p_float = float(p) + if not 0.0 <= p_float <= 1.0: + raise ValueError(f"percentile p must be in [0, 1], got {p_float}") + self._p = p_float + if value is not None: + self._vals.append(value) + + def finalize(self) -> float | None: + if not self._vals or self._p is None: + return None + s = sorted(self._vals) + n = len(s) + if n == 1: + return s[0] + rank = self._p * (n - 1) + lo = int(rank) + hi = min(lo + 1, n - 1) + return s[lo] + (rank - lo) * (s[hi] - s[lo]) + + +class _PercentileDiscAgg: + """2-arg PERCENTILE_DISC(value, p): smallest value v with cume_dist(v) >= p.""" + + def __init__(self) -> None: + self._vals: list[float] = [] + self._p: float | None = None + + def step(self, value, p) -> None: + if p is not None: + p_float = float(p) + if not 0.0 <= p_float <= 1.0: + raise ValueError(f"percentile p must be in [0, 1], got {p_float}") + self._p = p_float + if value is not None: + self._vals.append(value) + + def finalize(self): + if not self._vals or self._p is None: + return None + s = sorted(self._vals) + n = len(s) + # cume_dist of element at index k (0-based) is (k+1)/n. + # Smallest k with (k+1)/n >= p => k = ceil(p*n) - 1. + k = max(0, math.ceil(self._p * n) - 1) + return s[k] + + +# --------------------------------------------------------------------------- +# Statistical aggregates (DEV-1317): Welford's online algorithm +# --------------------------------------------------------------------------- + + +class _OneVarWelford: + """Shared online-stats state for the four 1-arg stat aggregates. + + Maintains ``(n, mean, M2)`` where ``M2 = sum((x_i - mean)^2)``. + Subclasses pick how to turn it into stddev_samp / stddev_pop / + var_samp / var_pop in ``finalize()``. + + NULL inputs are skipped (don't contribute to ``n``), matching + Postgres semantics for the whole stat-aggregate family. + """ + + def __init__(self) -> None: + self._n: int = 0 + self._mean: float = 0.0 + self._m2: float = 0.0 + + def step(self, value) -> None: + if value is None: + return + self._n += 1 + delta = value - self._mean + self._mean += delta / self._n + self._m2 += delta * (value - self._mean) + + +class _StddevSampAgg(_OneVarWelford): + """Sample standard deviation. NULL when N <= 1.""" + + def finalize(self) -> float | None: + if self._n <= 1: + return None + return math.sqrt(self._m2 / (self._n - 1)) + + +class _StddevPopAgg(_OneVarWelford): + """Population standard deviation. NULL at N=0; 0 at N=1.""" + + def finalize(self) -> float | None: + if self._n == 0: + return None + if self._n == 1: + return 0 + return math.sqrt(self._m2 / self._n) + + +class _VarSampAgg(_OneVarWelford): + """Sample variance. NULL when N <= 1.""" + + def finalize(self) -> float | None: + if self._n <= 1: + return None + return self._m2 / (self._n - 1) + + +class _VarPopAgg(_OneVarWelford): + """Population variance. NULL at N=0; 0 at N=1.""" + + def finalize(self) -> float | None: + if self._n == 0: + return None + if self._n == 1: + return 0 + return self._m2 / self._n + + +class _PairAgg: + """Shared 2-variable Welford state for corr / covar_samp / covar_pop.""" + + def __init__(self) -> None: + self._n: int = 0 + self._mean_x: float = 0.0 + self._mean_y: float = 0.0 + self._m2_x: float = 0.0 + self._m2_y: float = 0.0 + self._c: float = 0.0 + + def step(self, x, y) -> None: + if x is None or y is None: + return + self._n += 1 + dx = x - self._mean_x + self._mean_x += dx / self._n + dy = y - self._mean_y + self._mean_y += dy / self._n + self._m2_x += dx * (x - self._mean_x) + self._m2_y += dy * (y - self._mean_y) + self._c += dx * (y - self._mean_y) + + +class _CorrAgg(_PairAgg): + """Pearson correlation. NULL when fewer than 2 non-null pairs OR + when either side has zero variance (matches Postgres CORR).""" + + def finalize(self) -> float | None: + if self._n < 2: + return None + if self._m2_x == 0 or self._m2_y == 0: + return None + return self._c / math.sqrt(self._m2_x * self._m2_y) + + +class _CovarSampAgg(_PairAgg): + """Sample covariance. NULL when N <= 1.""" + + def finalize(self) -> float | None: + if self._n <= 1: + return None + return self._c / (self._n - 1) + + +class _CovarPopAgg(_PairAgg): + """Population covariance. NULL at N=0; 0 at N=1.""" + + def finalize(self) -> float | None: + if self._n == 0: + return None + if self._n == 1: + return 0 + return self._c / self._n + + +# --------------------------------------------------------------------------- +# Scalar wrappers (DEV-1317) +# --------------------------------------------------------------------------- + + +def _ln(x): + if x is None: + return None + return math.log(x) + + +def _log10(x): + if x is None: + return None + return math.log10(x) + + +def _log2(x): + # DEV-1337: overrides SQLite >=3.35's built-in to give strict + # "errors propagate" semantics matching Postgres. + if x is None: + return None + return math.log2(x) + + +def _log_base_x(b, x): + """``log(B, X)`` returns log_B(X). Base first, value second.""" + if b is None or x is None: + return None + return math.log(x, b) + + +def _exp(x): + if x is None: + return None + return math.exp(x) + + +def _sqrt(x): + if x is None: + return None + return math.sqrt(x) + + +def _pow(x, n): + """``pow(x, n)`` / ``power(x, n)`` — uses ``math.pow`` rather than ``**``. + + ``math.pow`` raises on negative-base-non-integer-exponent (clean + OperationalError at the SQLite boundary) and overflows into IEEE-754 + ``inf`` rather than building an unbounded big-int. + """ + if x is None or n is None: + return None + return math.pow(x, n) + + +def register_sqlite_udfs(dbapi_connection) -> None: + """Register all SLayer SQLite UDFs on a freshly-opened DBAPI connection. + + Wired in via SQLAlchemy's ``connect`` event in ``slayer.sql.client``, + so this is called once per new connection and again on pool refresh. + Idempotent: re-registering a UDF on the same connection replaces the + previous one (sqlite3 default behaviour). + """ + # --- Scalar UDFs ------------------------------------------------------ + dbapi_connection.create_function("ln", 1, _ln) + dbapi_connection.create_function("log10", 1, _log10) + dbapi_connection.create_function("log2", 1, _log2) + # SQLite >=3.35 ships a built-in ``log(B, X)`` that silently returns + # NULL on math-domain inputs. The UDF overrides that with strict + # error-propagating semantics matching Postgres. + dbapi_connection.create_function("log", 2, _log_base_x) + dbapi_connection.create_function("exp", 1, _exp) + dbapi_connection.create_function("sqrt", 1, _sqrt) + dbapi_connection.create_function("pow", 2, _pow) + dbapi_connection.create_function("power", 2, _pow) + + # --- Aggregate UDFs --------------------------------------------------- + dbapi_connection.create_aggregate("median", 1, _MedianAgg) + dbapi_connection.create_aggregate("percentile_cont", 2, _PercentileContAgg) + dbapi_connection.create_aggregate("percentile_disc", 2, _PercentileDiscAgg) + + # Statistical aggregates. Register each under its canonical Postgres- + # style name AND under the name sqlglot rewrites it to on SQLite, so + # generator output that goes through sqlglot still resolves at runtime. + dbapi_connection.create_aggregate("stddev_samp", 1, _StddevSampAgg) + dbapi_connection.create_aggregate("stddev_pop", 1, _StddevPopAgg) + dbapi_connection.create_aggregate("var_samp", 1, _VarSampAgg) + dbapi_connection.create_aggregate("variance", 1, _VarSampAgg) + dbapi_connection.create_aggregate("var_pop", 1, _VarPopAgg) + dbapi_connection.create_aggregate("variance_pop", 1, _VarPopAgg) + dbapi_connection.create_aggregate("corr", 2, _CorrAgg) + dbapi_connection.create_aggregate("covar_samp", 2, _CovarSampAgg) + dbapi_connection.create_aggregate("covar_pop", 2, _CovarPopAgg) + + +# =========================================================================== +# SqliteDialect — overrides for STRFTIME date_trunc, DATETIME-modifier +# time arithmetic, percentile UDF call shape, JSON rewrite, UDF registration. +# =========================================================================== + + +# DATETIME-modifier unit names (SQLite has no INTERVAL syntax). ``week`` +# is folded into ``days`` because SQLite has no week unit either. +_WINDOW_UNIT_SQLITE = { + "y": "years", + "m": "months", + "w": "days", + "d": "days", + "h": "hours", + "min": "minutes", + "s": "seconds", +} + + +class SqliteDialect(SqlDialect): + sqlglot_name: str = "sqlite" + ds_type_aliases: frozenset[str] = frozenset({"sqlite"}) + explain_prefix: str | None = "EXPLAIN QUERY PLAN" + explain_postfix: str = "" + log10_native: bool = True + log2_native: bool = True + + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """DEV-1708: SQLite's ``IS`` is null-safe on every supported version; + ``IS NOT DISTINCT FROM`` (what sqlglot emits for ``NullSafeEQ``) needs + SQLite ≥ 3.39, so anchor on bare ``IS`` instead.""" + return exp.Is(this=left, expression=right) + + def build_date_trunc( + self, + col_expr: exp.Expression, + granularity: TimeGranularity, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """SQLite has no DATE_TRUNC — use STRFTIME (with CASE WHEN for + quarter, weekday-modifier for week).""" + if granularity == TimeGranularity.WEEK_SUNDAY: + # DEV-1572: delegate to the base generic shift, which composes + # SQLite's own day-offset (DATE(col, 'N days')) around SQLite's + # Monday-week truncation — yielding the Sunday-anchored bucket. + return super().build_date_trunc( + col_expr=col_expr, granularity=granularity, parse=parse, + ) + gran_str = granularity.value + fmt_map = { + "year": "%Y-01-01", + "month": "%Y-%m-01", + "day": "%Y-%m-%d", + "hour": "%Y-%m-%d %H:00:00", + "minute": "%Y-%m-%d %H:%M:00", + "second": "%Y-%m-%d %H:%M:%S", + } + if gran_str == "week": + # SQLite weekday 0=Sunday; use date() with weekday modifier + # to back up to the preceding Monday-equivalent start. + return parse( + f"DATE({col_expr.sql(dialect='sqlite')}, 'weekday 0', '-6 days')" + ) + if gran_str == "quarter": + col_sql = col_expr.sql(dialect="sqlite") + return parse( + f"STRFTIME('%Y-', {col_sql}) || CASE " + f"WHEN CAST(STRFTIME('%m', {col_sql}) AS INTEGER) <= 3 THEN '01-01' " + f"WHEN CAST(STRFTIME('%m', {col_sql}) AS INTEGER) <= 6 THEN '04-01' " + f"WHEN CAST(STRFTIME('%m', {col_sql}) AS INTEGER) <= 9 THEN '07-01' " + f"ELSE '10-01' END" + ) + fmt = fmt_map.get(gran_str, "%Y-%m-%d") + return exp.Anonymous( + this="STRFTIME", + expressions=[exp.Literal.string(fmt), col_expr], + ) + + def build_time_offset_expr( + self, + col_expr: exp.Expression, + offset: int, + granularity: str, + ) -> exp.Expression: + """SQLite uses ``DATE(col, 'N units')`` — no INTERVAL syntax. + + Granularity normalization: ``quarter`` → ``val * 3`` of ``months``; + ``week`` → ``val * 7`` of ``days`` (SQLite has no week unit). + """ + sqlite_units = { + "year": "years", "month": "months", "day": "days", + "quarter": "months", "week": "days", "week_sunday": "days", + "hour": "hours", "minute": "minutes", "second": "seconds", + } + sqlite_unit = sqlite_units.get(granularity, granularity.lower() + "s") + val = offset * 3 if granularity == "quarter" else offset + sqlite_val = val * 7 if granularity in ("week", "week_sunday") else val + return exp.Anonymous( + this="DATE", + expressions=[ + col_expr, + exp.Literal.string(f"{sqlite_val} {sqlite_unit}"), + ], + ) + + def duration_interval_exprs( + self, + parts: list[tuple[int, str]], + sign: int = 1, + ) -> list[exp.Expression]: + """SQLite uses DATETIME-modifier string literals with sign baked in. + Week is converted to ``N*7 days`` (no native week unit).""" + prefix = "+" if sign >= 0 else "-" + return [ + exp.Literal.string( + f"{prefix}{(amount * 7 if unit == 'w' else amount)} " + f"{_WINDOW_UNIT_SQLITE[unit]}" + ) + for amount, unit in parts + ] + + def add_intervals_expr( + self, + expr: exp.Expression, + intervals: list[exp.Expression], + sign: int = 1, + ) -> exp.Expression: + """SQLite wraps as ``DATETIME(expr, mod1, mod2, ...)``. + + The sign is already baked into each modifier by + ``duration_interval_exprs`` — the ``sign`` arg is intentionally + ignored here. + """ + return exp.Anonymous(this="DATETIME", expressions=[expr, *intervals]) + + def build_median( + self, + inner: exp.Expression, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """SQLite: parses ``median(inner)`` — registered UDF. + + sqlglot's SQLite generator transpiles ``exp.Median`` to + ``PERCENTILE_CONT(x, 0.5)`` at emission, matching the pair-form + ``percentile_cont`` UDF signature. + """ + inner_sql = inner.sql(dialect="sqlite") + return parse(f"median({inner_sql})") + + def build_percentile( + self, + p_str: str, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """SQLite: ``percentile_cont(value, p)`` — registered UDF. + + ``p_str`` is the original user-supplied string, preserved verbatim + (no float normalization). + """ + return parse(f"percentile_cont({col_sql}, {p_str})") + + def rewrite_parsed_ast(self, tree: exp.Expression) -> exp.Expression: + """SQLite override: rewrites every ``exp.JSONExtract`` to + ``Anonymous(this='JSON_EXTRACT', ...)`` so the emission is the + function-call form (DEV-1331).""" + return rewrite_sqlite_json_extract(tree) + + def register_udfs(self, dbapi_connection) -> None: + """Register the Python aggregate / scalar UDFs on the connection. + + Idempotent — re-registering on the same connection replaces the + previous one (sqlite3 default behaviour). + """ + register_sqlite_udfs(dbapi_connection) + + diff --git a/slayer/sql/dialects/tsql.py b/slayer/sql/dialects/tsql.py new file mode 100644 index 00000000..8cf1a854 --- /dev/null +++ b/slayer/sql/dialects/tsql.py @@ -0,0 +1,374 @@ +"""DEV-1542: TsqlDialect (SQL Server / Microsoft T-SQL). + +T-SQL is the most divergent Tier-1 dialect: + +* ``DATETRUNC(unit, col)`` (SQL Server 2022+) instead of ``DATE_TRUNC`` +* Week uses ``iso_week`` to be ``@@DATEFIRST``-independent (Monday-based) +* ``DATEADD(unit, val, col)`` instead of ``col + INTERVAL N UNIT`` +* ``add_intervals_expr`` chains ``DATEADD`` calls (no INTERVAL) +* ``build_median`` / ``build_percentile`` raise — PERCENTILE_CONT in T-SQL + is a window function only +* Statistical aggregate names: STDEV / STDEVP / VAR / VARP via + ``exp.Anonymous`` (sqlglot's tsql transpiler emits wrong names) +* Variance-decomposition formula for CORR / COVAR_* with the T-SQL names +* EXPLAIN is a session-toggle pair: ``SET SHOWPLAN_ALL ON; ... ; OFF`` +* No native LOG2 +* DEV-1571 Bug 1: T-SQL rejects ``WITH`` inside a derived-table subquery. + ``emit_outer_wrap`` overrides the base to hoist inner top-level CTEs + to the outer statement. +* DEV-1571 Bug 2: T-SQL's ``ORDER BY`` resolver does not treat + ``[a.b]`` as a SELECT alias — it tries to resolve it as a column-name + lookup against the FROM scope. ``rewrite_emitted_sql`` mangles dotted + bracketed aliases to ``[a___b]``; ``decode_result_keys`` reverses on + result rows. Same bijection as ``BigqueryDialect``, different regex + anchor. +""" + +from __future__ import annotations + +import re +from typing import Any +from collections.abc import Callable + +import sqlglot +from sqlglot import exp + +from slayer.core.enums import TimeGranularity +from slayer.sql.naming import decode_alias, encode_alias +from slayer.sql.dialects.base import SqlDialect, _build_covar_decomposition + + +# sqlglot's tsql transpiler emits incorrect names (VAR_SAMP, VARIANCE_POP) +# that do not exist in T-SQL — these are the correct T-SQL canonical names. +_TSQL_STAT_NAMES: dict[str, str] = { + "stddev_samp": "STDEV", + "stddev_pop": "STDEVP", + "var_samp": "VAR", + "var_pop": "VARP", +} + + +# DEV-1571 Bug 2: bracket-quoted dotted alias. Same shape as BigQuery's +# backtick-anchored regex (``\w+(?:\.\w+)+``) with ``re.ASCII`` keeping +# ``\w`` ASCII-only so accented identifiers like ``[café.metric]`` do +# not mangle. +# +# Caveat (documented constraint, identical to BigQuery's): a fully +# bracketed dotted path of word-only segments (e.g. ``[my_schema.my_table]``) +# WOULD false-positive mangle. T-SQL users writing such paths in +# ``Column.sql`` must bracket each segment individually +# (``[my_schema].[my_table]``). T-SQL identifiers with spaces, hyphens, +# or other non-``\w`` characters (``[my table]``) are safe — the +# non-word character breaks the match. +_TSQL_DOTTED_ALIAS_RE = re.compile(r"\[(\w+(?:\.\w+)+)\]", re.ASCII) + + +class TsqlDialect(SqlDialect): + sqlglot_name: str = "tsql" + ds_type_aliases: frozenset[str] = frozenset({"mssql", "sqlserver", "tsql"}) + explain_prefix: str | None = "SET SHOWPLAN_ALL ON;" + explain_postfix: str = "; SET SHOWPLAN_ALL OFF" + log10_native: bool = True + log2_native: bool = False + + def build_null_safe_eq( + self, left: exp.Expression, right: exp.Expression, + ) -> exp.Expression: + """DEV-1708: T-SQL has no ``IS NOT DISTINCT FROM`` / ``<=>`` — emit the + portable expanded ``a = b OR (a IS NULL AND b IS NULL)``.""" + return self._expanded_null_safe_eq(left, right) + + def build_approx_count_distinct( + self, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """T-SQL: native ``APPROX_COUNT_DISTINCT(x)`` (SQL Server 2019+). + + Built as an ``exp.Anonymous`` because sqlglot's T-SQL dialect re-emits + a parsed ``APPROX_COUNT_DISTINCT`` as ``APPROX_DISTINCT`` (its + Presto-family canonical form), which is not a T-SQL function. + """ + return exp.Anonymous(this="APPROX_COUNT_DISTINCT", expressions=[parse(col_sql)]) + + def build_date_trunc( + self, + col_expr: exp.Expression, + granularity: TimeGranularity, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """T-SQL: ``DATETRUNC(unit, col)``. Week uses ``iso_week`` + (Monday-start) to be ``@@DATEFIRST``-independent. ``DATETRUNC`` + requires a temporal type — wrap non-column/cast operands. + + ``DATETRUNC`` requires **SQL Server 2022+**. SLayer's T-SQL + support is documented as 2022+ only (see ``CLAUDE.md`` under + Tier-1 / SQL Server, and ``examples/sqlserver/``). Pre-2022 + SQL Server does not have a single-call truncation function; + an equivalent ``DATEADD(unit, DATEDIFF(unit, 0, col), 0)`` + fallback exists but isn't a current target — track separately + if anyone needs it. + """ + if granularity == TimeGranularity.WEEK_SUNDAY: + # DEV-1572: delegate to the base generic shift, which composes + # T-SQL's DATEADD day-offset around the iso_week (Monday) DATETRUNC. + return super().build_date_trunc( + col_expr=col_expr, granularity=granularity, parse=parse, + ) + gran_str = granularity.value + if not isinstance(col_expr, (exp.Column, exp.Cast)): + col_expr = exp.Cast(this=col_expr, to=exp.DataType.build("TIMESTAMP")) + tsql_gran = "iso_week" if gran_str == "week" else gran_str + return exp.Anonymous( + this="DATETRUNC", + expressions=[exp.Var(this=tsql_gran), col_expr], + ) + + def build_time_offset_expr( + self, + col_expr: exp.Expression, + offset: int, + granularity: str, + ) -> exp.Expression: + """T-SQL: ``DATEADD(unit, val, col)``. INTERVAL is not valid T-SQL syntax. + Quarter normalises to ``val * 3`` of MONTH.""" + unit_map = { + "year": "YEAR", "month": "MONTH", "day": "DAY", + "quarter": "MONTH", "week": "WEEK", + # DEV-1572: a one-period shift of a Sunday-week is one week — same + # normalization the base ``_granularity_to_unit`` applies (without + # it, ``DATEADD(WEEK_SUNDAY, ...)`` is invalid T-SQL). + "week_sunday": "WEEK", + "hour": "HOUR", "minute": "MINUTE", "second": "SECOND", + } + unit = unit_map.get(granularity, granularity.upper()) + val = offset * 3 if granularity == "quarter" else offset + return exp.Anonymous( + this="DATEADD", + expressions=[exp.Var(this=unit), exp.Literal.number(val), col_expr], + ) + + def add_intervals_expr( + self, + expr: exp.Expression, + intervals: list[exp.Expression], + sign: int = 1, + ) -> exp.Expression: + """T-SQL: chain ``DATEADD(unit, ±amount, col)`` calls. + + Each interval in the list is an ``exp.Interval`` from + ``duration_interval_exprs``; extract unit name and amount, negate + when sign < 0. + """ + result = expr + for iv in intervals: + if not isinstance(iv, exp.Interval): + raise TypeError( + f"Expected exp.Interval in T-SQL DATEADD branch, got {type(iv)}" + ) + unit_str = iv.unit.name.upper() + amount = exp.Neg(this=iv.this) if sign < 0 else iv.this + result = exp.Anonymous( + this="DATEADD", + expressions=[exp.Var(this=unit_str), amount, result], + ) + return result + + def build_median( + self, + inner: exp.Expression, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + raise NotImplementedError( + "Aggregation 'median' is not supported on T-SQL (SQL Server): " + "PERCENTILE_CONT in T-SQL is a window function (requires OVER clause) " + "and cannot be used as a GROUP BY aggregate. " + "Use a window subquery or compute the value client-side." + ) + + def build_percentile( + self, + p_str: str, + col_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + raise NotImplementedError( + "Aggregation 'percentile' is not supported on T-SQL (SQL Server): " + "PERCENTILE_CONT requires a window function OVER clause in T-SQL " + "and is not valid as a GROUP BY aggregate. " + "Compute the value client-side or restructure as a window query." + ) + + def build_stat_agg_1arg( + self, + agg_name: str, + col_expr: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """T-SQL: map ``stddev_samp``→``STDEV``, ``stddev_pop``→``STDEVP``, + ``var_samp``→``VAR``, ``var_pop``→``VARP`` via ``exp.Anonymous``.""" + if agg_name in _TSQL_STAT_NAMES: + return exp.Anonymous( + this=_TSQL_STAT_NAMES[agg_name], + expressions=[parse(col_expr)], + ) + return super().build_stat_agg_1arg(agg_name, col_expr, parse=parse) + + def build_covar_2arg( + self, + agg_name: str, + col_sql: str, + other_sql: str, + *, + parse: Callable[[str], exp.Expression], + ) -> exp.Expression: + """T-SQL has no native CORR / COVAR_* — use the + variance-decomposition formula with T-SQL names (VAR / VARP / STDEV).""" + return _build_covar_decomposition( + col_sql=col_sql, + other_sql=other_sql, + agg=agg_name, + var_fn_samp="VAR", + var_fn_pop="VARP", + stddev_fn="STDEV", + parse=parse, + ) + + # ------------------------------------------------------------------ + # DEV-1571 Bug 1: emit_outer_wrap hoists inner top-level CTEs + # ------------------------------------------------------------------ + + def emit_outer_wrap( + self, + *, + inner_sql: str, + public: list[str], + order: exp.Expression | None, + limit: exp.Expression | None, + offset_arg: exp.Expression | None, + parse: Callable[[str], exp.Expression] | None = None, + ) -> str: + """T-SQL: hoist inner top-level CTEs to the outer statement AND + transpose detached pagination to ``TOP`` / ``FETCH NEXT N ROWS + ONLY`` syntax. + + SQL Server allows ``WITH`` only as a statement prefix, not inside + a derived-table subquery. Without this override, SLayer's + DEV-1444 outer-wrap emits ``SELECT ... FROM (WITH ctes SELECT ... + FROM step2) AS _outer ORDER BY ...``, which T-SQL rejects with + ``Incorrect syntax near the keyword 'WITH'``. + + Strategy (single AST path, no fallback to the base impl): + + 1. Parse ``inner_sql`` via the generator's ``_parse`` (when + supplied) so SLayer-specific AST rewrites survive the + round-trip — LOG10/LOG2 alias preservation (DEV-1337) and + SQLite JSONExtract function-form (DEV-1331). + 2. Detach the top-level ``With`` node (if any) from the inner + ``Select`` so the inner main SELECT can be wrapped in the + derived table without re-introducing nested WITH. + 3. Build the outer wrap entirely via sqlglot AST so dialect- + aware rendering transposes the detached ``Limit`` / ``Offset`` + nodes into T-SQL's ``TOP`` / ``FETCH NEXT N ROWS ONLY`` + syntax. A naïve ``limit.sql(dialect="tsql")`` only emits + ``LIMIT N`` because the transposition fires on the wrapping + Select, not on a free-standing Limit node. The CTE-less + branch must take the AST path too, otherwise any T-SQL query + that hits the outer-wrap path without CTEs would still emit + literal ``LIMIT N``. + + When the generator doesn't pass ``parse`` (direct unit-test + invocation), falls back to ``sqlglot.parse_one(dialect="tsql")``. + When the parse itself fails (malformed SQL / sqlglot bug), defers + to the base impl — T-SQL will still reject malformed SQL at the + DB layer, but we don't make it worse. + """ + parse_fn = parse if parse is not None else ( + lambda s: sqlglot.parse_one(s, dialect=self.sqlglot_name) + ) + try: + parsed = parse_fn(inner_sql) + except Exception: + return super().emit_outer_wrap( + inner_sql=inner_sql, + public=public, + order=order, + limit=limit, + offset_arg=offset_arg, + ) + if not isinstance(parsed, exp.Select): + return super().emit_outer_wrap( + inner_sql=inner_sql, + public=public, + order=order, + limit=limit, + offset_arg=offset_arg, + ) + # Detach the With (if present) so the inner main SELECT can be + # wrapped in a derived table. ``with_`` is the sqlglot args key + # (Python-keyword avoidance); other clauses use their natural + # names (``order`` / ``limit`` / ``offset``). + with_node = parsed.args.get("with_") + if with_node is not None: + parsed.set("with_", None) + # Strip inner-CTE qualifiers from detached ORDER BY columns so + # they resolve at the outer-wrapper scope (only ``_outer`` is + # visible). DEV-1444 carry-over. + if order is not None: + for col in order.find_all(exp.Column): + if col.args.get("table") is not None: + col.set("table", None) + derived = exp.Subquery( + this=parsed, + alias=exp.TableAlias(this=exp.to_identifier("_outer")), + ) + outer = exp.Select() + for a in public: + outer = outer.select(exp.Identifier(this=a, quoted=True)) + outer = outer.from_(derived) + if with_node is not None: + outer.set("with_", with_node) + if order is not None: + outer.set("order", order) + if limit is not None: + outer.set("limit", limit) + if offset_arg is not None: + outer.set("offset", offset_arg) + return outer.sql(dialect=self.sqlglot_name, pretty=True) + + # ------------------------------------------------------------------ + # DEV-1571 Bug 2: bracketed dotted-alias mangling + # ------------------------------------------------------------------ + + def rewrite_emitted_sql(self, sql: str) -> str: + """Replace ``.`` with ``___`` inside bracket-quoted identifiers. + + T-SQL's ``ORDER BY`` resolver does not treat ``[a.b]`` as a + SELECT alias — it tries to resolve it as a column-name lookup + against the FROM scope and fails with ``Invalid column name``. + Mangling on emit gives the parser a single dotless identifier + and the alias resolves cleanly. ``decode_result_keys`` reverses + the mangling on result rows so consumers see SLayer's universal + dotted alias shape. + + Uses the same bijection as ``BigqueryDialect`` (shared encode in + ``slayer.sql.naming``); only the regex anchor differs. + """ + return _TSQL_DOTTED_ALIAS_RE.sub( + lambda m: f"[{encode_alias(m.group(1))}]", sql + ) + + def decode_result_keys( + self, + rows: list[dict[str, Any]], + ) -> list[dict[str, Any]]: + """Reverse the T-SQL alias mangling on result-row keys so + consumers see SLayer's universal dotted alias shape regardless + of whether the query ran against T-SQL or another dialect. + """ + return [{decode_alias(k): v for k, v in row.items()} for row in rows] diff --git a/slayer/sql/engine_factory.py b/slayer/sql/engine_factory.py new file mode 100644 index 00000000..5cd46904 --- /dev/null +++ b/slayer/sql/engine_factory.py @@ -0,0 +1,161 @@ +"""Shared SQLAlchemy engine factory (DEV-1551). + +Single source of truth for building ``sa.Engine`` instances from a +``DatasourceConfig``. Every production code path that creates engines — +ingestion, schema_drift, type_refinement, the CLI's datasources-test +command, the MCP server's connectivity probes, and ``SlayerSQLClient`` — +funnels through ``get_engine(datasource)``. + +The factory itself is dialect-agnostic. Each dialect's ``SqlDialect`` +strategy class carries its own runtime hooks +(``build_engine``, ``apply_session_overrides``) under +``slayer/sql/dialects/.py``; this module just calls them and falls +back to a vanilla ``sa.create_engine`` when the dialect declines to +customise. + +Engine caching is keyed on ``DatasourceConfig.get_connection_string()`` +plus a fingerprint of the dialect-relevant runtime fields, so two +datasources that differ only in (e.g.) warehouse get different cached +engines. +""" + +from __future__ import annotations + +import logging + +import sqlalchemy as sa +import sqlalchemy.event as sa_event + +from slayer.core.models import DatasourceConfig +from slayer.sql.dialects import dialect_for_ds_type +from slayer.sql.dialects.base import SqlDialect + +logger = logging.getLogger(__name__) + + +# Engine cache. Key = (connection_string, runtime_fingerprint). +_engine_cache: dict[tuple[str, str], sa.Engine] = {} + + +def _runtime_fingerprint(datasource: DatasourceConfig) -> str: + """Stable fingerprint of dialect-relevant runtime fields for the + cache key. Two datasources differing only in (e.g.) warehouse or + role must NOT share a cached engine — the session-overrides listener + would otherwise apply the wrong USE statements. + + Currently only Snowflake uses any of these fields; for other + dialects the fingerprint collapses to an empty string and the + cache key reduces to the connection_string alone. + """ + if datasource.type != "snowflake": + return "" + parts = ( + ("wh", datasource.warehouse or ""), + ("rl", datasource.role or ""), + ("db", datasource.database or ""), + ("sc", datasource.schema_name or ""), + ) + return "|".join(f"{k}={v}" for k, v in parts) + + +def _attach_session_overrides_listener( + *, + engine: sa.Engine, + datasource: DatasourceConfig, +) -> None: + """Register a ``checkout`` event listener that calls the dialect's + ``apply_session_overrides`` hook every time a connection is taken + from the pool. + + The ``checkout`` event is used (not ``connect``) so the session + state is re-applied on every query — not just on the first physical + connection creation. Without this, anything that mutates Snowflake + session state mid-flight (an inspector probe issuing its own ``USE``, + or a user-issued ``client.execute("USE SCHEMA other")``) would + silently persist on the pooled connection and leak into the next + query. Cost: ~1-4 ``USE`` round-trips per query, dominated by + network latency to Snowflake. Acceptable trade-off for correctness. + + The listener's name is ``_slayer_session_overrides`` so tests can + verify registration without coupling to a private API. + + Skipped when the dialect's hook is the base-class no-op; detection + is by class identity so the no-op default doesn't trigger a + ``checkout`` listener that does nothing. + """ + dialect = dialect_for_ds_type(datasource.type) + base_method = SqlDialect.apply_session_overrides + dialect_method = type(dialect).apply_session_overrides + if dialect_method is base_method: + return + + @sa_event.listens_for(engine, "checkout") + def _slayer_session_overrides(dbapi_connection, _connection_record, _connection_proxy): + dialect.apply_session_overrides( + dbapi_connection=dbapi_connection, + datasource=datasource, + ) + + +def _attach_register_udfs_listener( + *, + engine: sa.Engine, + datasource: DatasourceConfig, +) -> None: + """Register a ``connect`` event listener that calls the dialect's + ``register_udfs`` hook on every new pooled connection. + + Skipped when the dialect's hook is the base-class no-op (every + dialect except SQLite). SQLite needs this to register the median / + percentile_cont / stddev / corr / log10 / log2 / ... UDFs without + which generated SQL like ``STDDEV_SAMP(x)`` fails with + ``sqlite3.OperationalError: no such function``. + """ + dialect = dialect_for_ds_type(datasource.type) + base_method = SqlDialect.register_udfs + dialect_method = type(dialect).register_udfs + if dialect_method is base_method: + return + + @sa_event.listens_for(engine, "connect") + def _slayer_register_udfs(dbapi_connection, _connection_record): + dialect.register_udfs(dbapi_connection) + + +def _build_engine(*, datasource: DatasourceConfig, connection_string: str) -> sa.Engine: + """Construct a new SA engine for the datasource without consulting + the cache. Delegates engine-build to the dialect's ``build_engine`` + hook; falls back to vanilla ``sa.create_engine`` when the dialect + declines (returns ``None``). + """ + dialect = dialect_for_ds_type(datasource.type) + engine = dialect.build_engine(datasource, connection_string=connection_string) + if engine is None: + engine = sa.create_engine(connection_string, pool_pre_ping=True) + _attach_register_udfs_listener(engine=engine, datasource=datasource) + _attach_session_overrides_listener(engine=engine, datasource=datasource) + return engine + + +def get_engine(datasource: DatasourceConfig) -> sa.Engine: + """Return a cached ``sa.Engine`` for the given datasource. Builds one + if the cache misses. + + The cache key includes a fingerprint of dialect runtime fields so + that two datasources differing in (e.g.) warehouse get different + cached engines — otherwise the connect listener would silently + apply the wrong USE statements. + """ + connection_string = datasource.get_connection_string() + cache_key = (connection_string, _runtime_fingerprint(datasource)) + if cache_key not in _engine_cache: + _engine_cache[cache_key] = _build_engine( + datasource=datasource, connection_string=connection_string, + ) + return _engine_cache[cache_key] + + +def reset_cache() -> None: + """Discard every cached engine. Used by tests that need fresh pools; + not called by production code.""" + _engine_cache.clear() diff --git a/slayer/sql/generator.py b/slayer/sql/generator.py index d192e97e..4eb041cd 100644 --- a/slayer/sql/generator.py +++ b/slayer/sql/generator.py @@ -8,22 +8,26 @@ import copy import logging import re -from typing import Any, Dict, List, Optional, Set, Tuple +from typing import AbstractSet, Any, Dict, List, Literal, NamedTuple, Optional, Set, Tuple, Union import sqlglot from sqlglot import exp from slayer.core.enums import ( + BUILTIN_AGGREGATIONS, BUILTIN_AGGREGATION_FORMULAS, BUILTIN_AGGREGATION_REQUIRED_PARAMS, DataType, TimeGranularity, ) -from pydantic import BaseModel, ConfigDict +from pydantic import BaseModel, ConfigDict, field_validator -from slayer.core.errors import AggregationNotAllowedError +from slayer.core.errors import AggregationNotAllowedError, UnresolvableOrderColumnError +from slayer.core.keys import _FrozenKey, _reroot_path_ref, reroot_aggregate_key from slayer.core.models import Aggregation from slayer.core.refs import agg_kwarg_canonical_str +from slayer.core.time_bounds import strip_frame_bounds +from slayer.core.window_duration import parse_window_duration as _parse_window_duration from slayer.engine.column_expansion import ( _is_trivial_base, _walk_path_to_target_sync, @@ -35,10 +39,58 @@ stage_bundle_with_siblings, synthetic_model_from_stage_schema, ) -from slayer.sql.sqlite_dialect import rewrite_sqlite_json_extract +from slayer.sql.dialects import SqlDialect, get_dialect +from slayer.sql.naming import ( + AliasAllocator, + dialect_folds_case, + flat_name, + maybe_quote_ident, + quote_mixed_case_identifiers, + result_key, + result_key_from_alias, +) +from slayer.sql.reserved_keywords import prequote_reserved_identifiers +from slayer.sql.scope import ScopeFrame +from slayer.sql.scope_check import maybe_validate_scopes from slayer.sql.stage_wrapper import build_flat_rename_wrapper +class _OrderColRef(NamedTuple): + """DEV-1645: resolved ORDER BY key. ``is_alias`` distinguishes a projected + output alias (emit whole-quoted, e.g. ``"orders.revenue_sum"``) from a + table.column fallback (emit SPLIT, e.g. ``ranked."time_mark"``), so a sort + on an unprojected/renamed column references the underlying FROM-scope column + instead of a nonexistent composite identifier. + + Ported from ``origin/main`` for the LEGACY enrichment pipeline (this stack is + deleted in DEV-1485 Stage 11); the typed pipeline enforces the same policy + independently in ``_apply_order_limit_from_planned`` + the plan-time order + validation. + """ + text: str # whole resolved string (used for base_cols membership) + is_alias: bool # True => projected output alias; False => table.column fallback + qualifier: str | None # fallback only: FROM-scope alias + column: str | None # fallback only: underlying column short name + + +class ResolvedAggKwarg(BaseModel): + """DEV-1706 — a resolved parametric-aggregation kwarg value (2-kind tag). + + * ``kind="expr"`` — a trusted, scope-resolved sqlglot expression for a + column-ref kwarg (``ColumnKey`` / ``ColumnSqlKey``). Embedded directly; + the crossed join registered at spec-build (the DEV-1527 fix). + * ``kind="str"`` — the legacy canonical-string form (scalars via + ``agg_kwarg_canonical_str``, existing strings), consumed exactly as + before: ``_SAFE_AGG_PARAM_RE`` guard + ``_resolve_sql`` (percentile / + stat) or formula substitution (custom aggregations). + """ + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + kind: Literal["expr", "str"] + value: Union[exp.Expression, str] + + class AggRenderSpec(BaseModel): """DEV-1452 — typed input record for the dialect-aware aggregation helpers (``_build_agg``, ``_build_percentile``, ``_build_stat_agg``, @@ -87,9 +139,31 @@ class AggRenderSpec(BaseModel): """Custom-aggregation definition (formula + params) for aggregations outside the built-in set. ``None`` for built-ins.""" - agg_kwargs: Dict[str, str] = {} - """Query-time aggregation parameter overrides (already stringified via - ``agg_kwarg_canonical_str`` at spec-build time).""" + agg_kwargs: Dict[str, ResolvedAggKwarg] = {} + """Query-time aggregation parameter overrides as typed 2-kind values + (DEV-1706 D-I). Column-ref kwargs arrive as ``kind="expr"`` (scope-resolved + at spec-build); everything else as ``kind="str"``. A bare ``str`` value is + coerced to ``kind="str"`` by ``_coerce_agg_kwargs`` so the legacy + ``EnrichedMeasure`` shim and direct-construction call sites keep working.""" + + @field_validator("agg_kwargs", mode="before") + @classmethod + def _coerce_agg_kwargs(cls, v: Any) -> Any: + """Coerce bare ``str`` kwarg values to ``ResolvedAggKwarg(kind="str")``; + pass ``ResolvedAggKwarg`` through; leave anything else for Pydantic to + reject (``bool`` / ``None`` never reach here from spec-build — they raise + earlier in ``agg_kwarg_canonical_str``).""" + if not isinstance(v, dict): + return v + coerced: Dict[str, Any] = {} + for key, val in v.items(): + if isinstance(val, (ResolvedAggKwarg, dict)): + coerced[key] = val + elif isinstance(val, str): + coerced[key] = ResolvedAggKwarg(kind="str", value=val) + else: + coerced[key] = val # bool / None / other → Pydantic rejects + return coerced filter_sql: Optional[str] = None """Column-filter predicate (``Column.filter``) wired in at aggregation @@ -147,6 +221,16 @@ class FirstLastRenderState(BaseModel): base callers leave this ``None`` and rely on ``aliases_by_slot_id`` threaded through ``_build_where_having_from_planned`` instead.""" + value_alias_by_sql: Dict[str, str] = {} + """DEV-1708 Law 2: RESOLVED value text → ``_val_`` materialisation alias + for an aggregate whose SOURCE crosses a join. Keyed by the resolved + (qualified + ``Column.type`` inner-CAST) value emission — DEV-1709 — so + same-sql-different-type aggregates map to distinct materialisations. The + projected aggregate materialises the crossing value inside the ranked + subquery; a HAVING referencing the same aggregate must bind to the SAME + alias (not re-emit the raw crossing ref, which is out of scope in the + outer SELECT). Empty when every source is local.""" + def _iter_first_last_leaves(key) -> "list": # NOSONAR(S3776) — sequential isinstance dispatch over the closed ValueKey union; each branch is the per-type recursion contract for surfacing first/last AggregateKey leaves. Extracting per-type helpers would scatter the contract. """DEV-1501 (Codex round 3): walk a composite ValueKey for first / @@ -251,7 +335,9 @@ def _wrap_cast_for_type(expr: exp.Expression, dt: Optional[DataType]) -> exp.Exp Skipped when ``dt`` is ``None`` (no declared type) or ``DataType.TEXT`` (cosmetic — SQL TEXT/VARCHAR roundtripping is already a no-op for our purposes and ``CAST(... AS TEXT)`` does not unwrap SQLite's - JSON-quoted-string return values anyway). Skipped when ``expr`` is a + JSON-quoted-string return values anyway). Also skipped when ``dt`` is + opaque (``DataType.UNKNOWN``) — there is no such SQL type, so + ``CAST(x AS UNKNOWN)`` is invalid in every dialect. Skipped when ``expr`` is a plain ``exp.Column`` (possibly qualified ``model.col``) — those are bare column references whose runtime type already matches the declared type by definition; wrapping them in CAST is dead noise and on SQLite @@ -259,7 +345,7 @@ def _wrap_cast_for_type(expr: exp.Expression, dt: Optional[DataType]) -> exp.Exp to a year). Idempotent: if ``expr`` is already a CAST to the same target, return it unchanged. """ - if dt is None or dt == DataType.TEXT: + if dt is None or dt == DataType.TEXT or dt.is_opaque: return expr if isinstance(expr, exp.Column): return expr @@ -334,31 +420,21 @@ def _filter_cast_type(dt: Optional[DataType]) -> Optional[DataType]: # # Name kept as ``_LOCAL_SLICE`` for grep continuity with 7b.8-7b.12 # call sites and tests; the set is no longer local-only. -_BUILTIN_BAREARG_AGGS_LOCAL_SLICE: frozenset[str] = frozenset({ - "sum", "avg", "min", "max", "count", "count_distinct", "median", - "percentile", "weighted_avg", - "corr", "covar_samp", "covar_pop", - "stddev_samp", "stddev_pop", "var_samp", "var_pop", - "first", "last", -}) +# +# DEV-1717: bound to the canonical ``BUILTIN_AGGREGATIONS`` enum rather than a +# hand-maintained duplicate. The two allowlists must stay byte-identical — a +# new built-in aggregation added to the enum is dispatched here automatically, +# so they can never silently desync (a lockstep-edit hazard CodeRabbit flagged +# when ``count_distinct_approx`` had to be added to both). +_BUILTIN_BAREARG_AGGS_LOCAL_SLICE: frozenset[str] = BUILTIN_AGGREGATIONS # DEV-1337: dialects with native single-arg `log10(x)` / `log2(x)`. sqlglot # normalises both into a generic ``Log(this=Literal(base), expression=arg)`` # AST and re-emits as ``LOG(base, x)`` for almost every dialect, which # diverges from the recipe formula text and (on dialects without 2-arg # ``LOG``) can break a previously working call. We rewrite the AST back -# to ``Anonymous(this='log10'|'log2', ...)`` for the dialects below; -# unsupported dialects (oracle; tsql for log2) keep the canonical 2-arg -# form. Mirrored in tests/test_sql_generator.py — keep in sync. -_LOG10_NATIVE_DIALECTS: frozenset[str] = frozenset({ - "sqlite", "postgres", "duckdb", "mysql", "clickhouse", - "snowflake", "bigquery", "redshift", - "trino", "presto", "databricks", "spark", "tsql", -}) -_LOG2_NATIVE_DIALECTS: frozenset[str] = frozenset({ - "sqlite", "postgres", "duckdb", "mysql", "clickhouse", - "bigquery", "trino", "presto", "databricks", "spark", -}) +# to ``Anonymous(this='log10'|'log2', ...)``; the per-dialect native-alias +# decision is delegated to ``SqlDialect.should_use_native_log`` (DEV-1716). # Transforms that use self-join CTEs instead of window functions. # This gives correct results at result-set edges (no NULLs when the DB has the data) @@ -379,6 +455,9 @@ def _filter_cast_type(dt: Optional[DataType]) -> Optional[DataType]: # duplicated across CTE / window emission sites (Sonar S1192). _SQL_WITH = "WITH " _SQL_PARTITION_BY = "PARTITION BY " +# Two-space-indented ``SELECT`` head for hand-assembled CTE bodies (shifted / +# consecutive-periods pairs), extracted so the literal isn't duplicated (S1192). +_SQL_SELECT_HEAD = "SELECT\n " # Matches safe aggregation parameter values: identifiers, qualified names, numeric literals. _SAFE_AGG_PARAM_RE = re.compile( @@ -401,7 +480,43 @@ def _wrap_filter(sql_str: str, filter_sql: Optional[str]) -> str: return sql_str return f"(CASE WHEN {filter_sql} THEN {sql_str} END)" -_WINDOW_DURATION_RE = re.compile(r"(?P\d+)(?Pmin|[ymwdhs])") + +def _first_bare_column_name(key) -> Optional[str]: + """Return the leaf name of the first bare column reference inside a + ROW-phase composite key (DEV-1576 / DEV-1717 error messages). + + Walks ``ArithmeticKey`` operands / ``ScalarCallKey`` args / a + ``TransformKey`` input for a ``ColumnKey`` / ``ColumnSqlKey`` leaf so the + "Bare measure name ''" error names the offending column. Returns + ``None`` when no column ref is found (caller falls back to the alias). + """ + from slayer.core.keys import ( + ArithmeticKey, + ColumnKey, + ColumnSqlKey, + ScalarCallKey, + TransformKey, + ) + + if isinstance(key, ColumnKey): + return key.leaf + if isinstance(key, ColumnSqlKey): + return key.column_name + if isinstance(key, ArithmeticKey): + children = key.operands + elif isinstance(key, ScalarCallKey): + children = key.args + elif isinstance(key, TransformKey): + children = [key.input] + else: + return None + for child in children: + name = _first_bare_column_name(child) + if name is not None: + return name + return None + + _WINDOW_UNIT_SQL = { "y": "year", "m": "month", @@ -475,30 +590,6 @@ def _is_windowed_measure(m: EnrichedMeasure) -> bool: return bool(m.window) -def _parse_window_duration(value: str) -> list[tuple[int, str]]: - """Parse compact durations like 1y2m3w5d6h7min8s.""" - if not value: - raise ValueError("Window duration cannot be empty") - pos = 0 - parts: list[tuple[int, str]] = [] - for match in _WINDOW_DURATION_RE.finditer(value): - if match.start() != pos: - raise ValueError( - f"Invalid window duration '{value}'. Use syntax like '1y2m3w5d6h7min8s'." - ) - amount = int(match.group("num")) - unit = match.group("unit") - if amount <= 0: - raise ValueError(f"Window duration parts must be positive in '{value}'") - parts.append((amount, unit)) - pos = match.end() - if pos != len(value) or not parts: - raise ValueError( - f"Invalid window duration '{value}'. Use syntax like '1y2m3w5d6h7min8s'." - ) - return parts - - def _cte_name_from_alias(prefix: str, alias: str) -> str: """Build a unique CTE name from a measure alias. @@ -506,12 +597,41 @@ def _cte_name_from_alias(prefix: str, alias: str) -> str: with aliases that already contain underscores. E.g.: - ``orders.revenue_sum`` -> ``_fm_orders__revenue_sum`` - ``orders_v2.revenue_sum`` -> ``_fm_orders_v2__revenue_sum`` + + DEV-1713: the ``.`` -> ``__`` flatten delegates to + :func:`slayer.sql.naming.flat_name` (single owner); this adds only the + non-identifier-character sanitisation on top. """ - sanitized = alias.replace(".", "__") + sanitized = flat_name(alias) sanitized = re.sub(r"[^a-zA-Z0-9_]", "_", sanitized) return prefix + sanitized +def _effective_src_filters(*, planned_query, plan) -> list: + """``planned_query.filters_by_phase`` as the windowed ``_src`` scope sees it + (DEV-1732): frame-bound residuals substituted for the host's predicates. + + Returned as ONE list that both ``_resolve_where_filter_joins_via_scope`` and + ``_build_where_having_from_planned`` consume, so join discovery and + rendering are structurally guaranteed to agree. Entries whose filter is + wholly a frame bound need no substitution here — the planner already left + their ids out of ``plan.where_filter_ids``, and the caller's + ``skip_filter_ids`` drops them. + + Returns the original list unchanged when the plan carries no rewrites, so a + query without a split conjunction emits byte-identical SQL. + """ + rewrites = getattr(plan, "src_filter_rewrites", None) + if not rewrites: + return planned_query.filters_by_phase + by_id = {r.filter_id: r.expression for r in rewrites} + return [ + fp if fp.id not in by_id + else fp.model_copy(update={"expression": by_id[fp.id]}) + for fp in planned_query.filters_by_phase + ] + + def _alias_prefixes(model_name: str) -> list: """'a__b__c' → ['a', 'a__b', 'a__b__c']""" parts = model_name.split("__") @@ -589,6 +709,13 @@ def _filter_references_available(f, available_aliases: set) -> bool: ) _TRAILING_LIMIT_RE = re.compile(r"(?is)\s*LIMIT\s+\d+\s*\Z") +# A ``Column.sql`` that is just an unqualified identifier — i.e. the column +# renames a physical column rather than computing an expression. Used to +# reserve star-exported physical names against ``_val_`` collisions +# (DEV-1728). Deliberately rejects dots: ``regions.population`` is a crossing +# reference, not a column of the star-projected relation. +_BARE_IDENT_RE = re.compile(r"[A-Za-z_]\w*") + def _strip_trailing_pagination(sql: str) -> str: """DEV-1444: remove trailing ORDER BY / LIMIT / OFFSET clauses that @@ -638,8 +765,94 @@ def _strip_trailing_pagination(sql: str) -> str: class SQLGenerator: """Generates SQL from an EnrichedQuery.""" - def __init__(self, dialect: str = "postgres"): - self.dialect = dialect + def __init__(self, dialect: "str | SqlDialect" = "postgres"): + if isinstance(dialect, SqlDialect): + self._dialect: SqlDialect = dialect + else: + self._dialect = get_dialect(dialect) + # DEV-1708 (D-E): the generation-wide alias allocator, installed by + # ``generate_from_planned`` for the duration of one render so inline + # forward ``_cm_*`` CTEs and the host base share ``_val_`` naming. + # ``None`` outside a render; direct-call helpers fall back to a local + # allocator. + self._gen_allocator: Optional[AliasAllocator] = None + + @property + def dialect(self) -> str: + """The sqlglot dialect name. Read-only — derived from + ``self._dialect.sqlglot_name``. Mutating it would desync the + strategy object from the string sqlglot consumes (DEV-1716).""" + return self._dialect.sqlglot_name + + def _new_allocator(self) -> AliasAllocator: + """Build an ``AliasAllocator`` carrying this generator's dialect + case-folding policy (DEV-1726): on case-folding dialects the + ``_taken`` comparison folds, so minted CTE / materialisation names can + never collide after the backend folds them. The ONLY construction + site in this module — pinned by test_dev1726_cte_case_folding — so a + new allocation path cannot silently lose dialect awareness.""" + return AliasAllocator(folds_case=dialect_folds_case(self.dialect)) + + @staticmethod + def _reserve_model_column_names(allocator: AliasAllocator, model) -> None: + """Reserve every name a ``.*`` projection of ``model`` can + export, so a minted ``_val_`` (Law-2 materialisation) never shadows a + real column (DEV-1728 / Codex F6). + + Both the SEMANTIC name and — when ``Column.sql`` is a bare identifier — + the PHYSICAL column name are reserved: a star-projection exports the + physical names, and the two differ whenever a column renames its source + (``Column(name="value", sql="_val_0")``). A non-bare ``Column.sql`` is an + expression, not a star-exported column, so it contributes nothing. + + Columns that exist in the database but not on the model are outside what + SLayer can see without reflection; a physical column literally named + ``_val_`` is the only way to hit that residual, which the underscore + prefix makes vanishingly unlikely. + """ + names: List[str] = [] + for c in model.columns: + names.append(c.name) + sql = getattr(c, "sql", None) + if sql and _BARE_IDENT_RE.fullmatch(sql.strip()): + names.append(sql.strip()) + allocator.reserve(*names) + + @staticmethod + def _maybe_quote_ident(ident: Optional[exp.Expression]) -> None: + """Thin delegator to :func:`slayer.sql.naming.maybe_quote_ident` + (DEV-1713 D-b: the mixed-case quoting policy is owned by the naming + module). Kept as a method so existing ``gen._maybe_quote_ident`` call + sites / tests are unchanged.""" + maybe_quote_ident(ident) + + @staticmethod + def _quote_mixed_case_identifiers(node: exp.Expression) -> exp.Expression: + """Thin delegator to + :func:`slayer.sql.naming.quote_mixed_case_identifiers` (DEV-1713 D-b). + Kept as a method so ``tree.transform(gen._quote_mixed_case_identifiers)`` + call sites / tests are unchanged. See the naming module for the policy + (DEV-1645 mixed-case quoting; DEV-1686 reserved-word dependency).""" + return quote_mixed_case_identifiers(node) + + def _to_ident(self, name: str) -> exp.Identifier: + """Build a column/table-name identifier, quoting it when mixed-case + (DEV-1645). Use for real DB column/table names — NOT for aliases or + qualifiers (those stay unquoted via plain ``exp.to_identifier``, and + reserved-word aliases quote at emit via ``RESERVED_KEYWORDS``).""" + ident = exp.to_identifier(name) + self._maybe_quote_ident(ident) + return ident + + def _to_table(self, name: str, alias: Optional[str] = None) -> exp.Expression: + """Build a (possibly schema-qualified) table reference with mixed-case + physical-name parts quoted (DEV-1645). The ``alias`` is SLayer-internal + and stays unquoted (a reserved-word alias still quotes at emit through + ``RESERVED_KEYWORDS`` — DEV-1686).""" + table = exp.to_table(name).transform(self._quote_mixed_case_identifiers) + if alias is not None: + table.set("alias", exp.TableAlias(this=exp.to_identifier(alias))) + return table def _parse(self, sql: str, *, dialect: Optional[str] = None) -> exp.Expression: """Parse ``sql`` via sqlglot, applying SLayer-specific AST rewrites. @@ -660,13 +873,43 @@ def _parse(self, sql: str, *, dialect: Optional[str] = None) -> exp.Expression: site. """ d = dialect or self.dialect + active = self._dialect if d == self.dialect else get_dialect(d) + # DEV-1686: quote any reserved-word qualifier/leaf (``grant.id`` → + # ``"grant".id``) before re-parsing a SLayer-built string, so a bare + # reserved word does not fail at parse time. No-op for ordinary SQL + # (only dot-adjacent reserved words are touched) and idempotent on + # already-quoted identifiers. + sql = prequote_reserved_identifiers(sql, dialect=d) tree = sqlglot.parse_one(sql, dialect=d) - if d == "sqlite": - tree = rewrite_sqlite_json_extract(tree) + # DEV-1716: PARSE-dialect keyed AST rewrite (SQLite rewrites + # JSONExtract to the function-call form — DEV-1331). Default identity. + tree = active.rewrite_parsed_ast(tree) # Log-alias rewrite is multi-dialect; the per-base allowlist check # lives inside ``_rewrite_log_aliases`` so unsupported dialects # (oracle; tsql for log2) keep the canonical 2-arg LOG form. - return tree.transform(self._rewrite_log_aliases) + tree = tree.transform(self._rewrite_log_aliases) + # DEV-1645: quote mixed-case column/table identifiers so case-folding + # dialects reach the right physical object (see the method docstring + # for the DEV-1706 pull-forward rationale). + tree = tree.transform(self._quote_mixed_case_identifiers) + # DEV-1716: TARGET-dialect keyed AST rewrite (Postgres wraps the first + # arg of a 2-arg ROUND in a numeric CAST — DEV-1576). Keyed to the + # generator's target dialect, not the parse dialect. + return self._dialect.rewrite_target_ast(tree) + + def _finalize_scalar_call(self, expr: exp.Expression) -> exp.Expression: + """Apply the target-dialect AST rewrite to a scalar-call expression + (DEV-1576 / DEV-1717). + + Scalar calls (``round``/``abs``/``coalesce``/…) in formulas are + assembled directly as ``exp.func(...)`` AST, never string-parsed, so + the ``rewrite_target_ast`` applied inside ``_parse`` never sees them. + Routing them through the same dialect hook here keeps the 2-arg + Postgres ``ROUND`` numeric-cast (and any future target rewrite) + consistent between parsed and AST-built expressions. Identity for + dialects whose ``rewrite_target_ast`` is a no-op. + """ + return self._dialect.rewrite_target_ast(expr) def _parse_predicate(self, sql: str, *, dialect: Optional[str] = None) -> exp.Expression: """Parse a bare WHERE/HAVING predicate expression (DEV-1378). @@ -686,16 +929,21 @@ def _parse_predicate(self, sql: str, *, dialect: Optional[str] = None) -> exp.Ex possible. """ d = dialect or self.dialect + active = self._dialect if d == self.dialect else get_dialect(d) + # DEV-1686: quote reserved qualifiers/leaves before the re-parse (see + # ``_parse``). No-op for ordinary predicates; idempotent when quoted. + sql = prequote_reserved_identifiers(sql, dialect=d) wrapped = sqlglot.parse_one(f"SELECT 1 WHERE {sql}", dialect=d) where = wrapped.args.get("where") if where is None or where.this is None: # pragma: no cover — defensive raise ValueError( f"Could not extract WHERE predicate from {sql!r} (dialect={d!r})" ) - tree = where.this - if d == "sqlite": - tree = rewrite_sqlite_json_extract(tree) - return tree.transform(self._rewrite_log_aliases) + tree = active.rewrite_parsed_ast(where.this) + tree = tree.transform(self._rewrite_log_aliases) + # DEV-1645: mixed-case identifier quoting (see ``_parse``). + tree = tree.transform(self._quote_mixed_case_identifiers) + return self._dialect.rewrite_target_ast(tree) def generate( self, @@ -755,6 +1003,19 @@ def generate( if render_mode == "outer": sql = self._apply_outer_projection_trim(sql=sql, enriched=enriched) + # DEV-1716: dialect-driven post-pass — BigQuery / T-SQL mangle dotted + # aliases here (identity for every other dialect). Fires for BOTH + # render modes so inner-CTE column names are mangled consistently with + # the outer projection (the outer stage's references to inner columns + # must resolve to the same ``___``-form alias). + sql = self._dialect.rewrite_emitted_sql(sql) + # DEV-1705: scope-closure validation on the final POST-mangle, pre-RLS + # output when SLAYER_VALIDATE_SCOPES is set (no-op otherwise). Post-mangle + # is where BigQuery/T-SQL dotted aliases become unambiguous ``___`` names + # (pre-mangle dotted refs parse as table.column and both false-flag and + # trigger BigQuery's TypeError). Mangling is a scope no-op for other + # dialects. RLS is applied downstream by the engine, so this is pre-RLS. + maybe_validate_scopes(sql, dialect=self.dialect) return sql def _apply_outer_projection_trim( @@ -827,40 +1088,68 @@ def _build_outer_wrap( limit, offset_arg, ) -> str: - """Emit ``SELECT FROM () AS _outer [ORDER/LIMIT/OFFSET]``. - - ``inner_sql`` is used as-is to preserve its formatting (callers - diff against literal ``OVER (...)`` substrings). Trailing - ORDER/LIMIT/OFFSET segments are stripped from ``inner_sql`` and - re-emitted on the outer wrapper. + """Thin delegate to ``self._dialect.emit_outer_wrap`` (DEV-1716). + + Strips trailing ORDER BY / LIMIT / OFFSET from ``inner_sql`` + (text-level) before handing off to the dialect hook, then passes + the detached AST nodes for re-emission on the outer statement. The + hook owns the wrap shape (base derived-table form; ``TsqlDialect`` + hoists inner CTEs) AND the dialect-correct identifier quoting of the + public-alias list (backticks / brackets / ANSI double quotes). """ - outer_select = _SQL_COL_SEP.join(f'"{a}"' for a in public) if order is None and limit is None and offset_arg is None: - return ( - f"SELECT\n {outer_select}\n" - f"FROM (\n{inner_sql.rstrip()}\n) AS _outer" - ) - inner_no_pag = _strip_trailing_pagination(inner_sql) - out = ( - f"SELECT\n {outer_select}\n" - f"FROM (\n{inner_no_pag.rstrip()}\n) AS _outer" - ) - if order is not None: - # DEV-1444 (Codex review on PR #134): the detached ORDER BY - # may carry inner-CTE qualifiers like ``_base."col"`` from - # ``_assemble_combined_sql``; those don't resolve at the - # outer wrapper level (only ``_outer`` is in scope). Strip - # every Column's table qualifier — the outer scope exposes - # each column by its bare alias name. - for col in order.find_all(exp.Column): - if col.args.get("table") is not None: - col.set("table", None) - out += "\n" + order.sql(dialect=self.dialect, pretty=True) - if limit is not None: - out += "\n" + limit.sql(dialect=self.dialect, pretty=True) - if offset_arg is not None: - out += "\n" + offset_arg.sql(dialect=self.dialect, pretty=True) - return out + stripped = inner_sql + else: + stripped = _strip_trailing_pagination(inner_sql) + return self._dialect.emit_outer_wrap( + inner_sql=stripped, + public=public, + order=order, + limit=limit, + offset_arg=offset_arg, + parse=self._parse, + ) + + def _quote_ident(self, name: str) -> str: + """Render ``name`` as ONE dialect-quoted identifier string (DEV-1716). + + Backticks on MySQL/BigQuery, brackets on T-SQL, ANSI double quotes on + Postgres/SQLite/DuckDB. Replaces raw ``f'"{name}"'`` sites in the + string-assembled CTE/projection paths so non-ANSI dialects get correct + quoting in the first place (a terminal string-rewrite can't fix ANSI + quotes — MySQL re-parses them as string literals). The BigQuery / T-SQL + alias-mangling ``rewrite_emitted_sql`` post-pass then fires on the + dotted quoted identifier. Identity round-trip on Postgres/SQLite (still + ``"name"``), so those emissions are unchanged. + """ + return exp.to_identifier(name, quoted=True).sql(dialect=self.dialect) + + def _null_safe_join_pair_sql(self, *, left_sql: str, right_sql: str) -> str: + """Render one dialect-aware null-safe equality (DEV-1708 / Codex F2) for + a grain join-back ``ON`` clause. ``left_sql`` / ``right_sql`` are the + already-quoted qualified column strings (``_base."x"`` / ``_cm."x"``); + they are parsed back to AST so the dialect strategy's + ``build_null_safe_eq`` can wrap them (native ``IS NOT DISTINCT FROM`` / + ``<=>`` / ``IS``, or the expanded ``= … OR (… IS NULL AND … IS NULL)``).""" + left = self._parse(left_sql) + right = self._parse(right_sql) + return self._dialect.build_null_safe_eq(left, right).sql(dialect=self.dialect) + + def _ordered(self, order_col: exp.Expression, *, ascending: bool) -> exp.Ordered: + """Build an ``exp.Ordered`` node, suppressing sqlglot's NULLS-emulation + ``CASE WHEN`` on T-SQL (DEV-1571 Bug 2 / DEV-1716). + + On T-SQL, sqlglot emits ``CASE WHEN IS NULL THEN 1 ELSE 0 END, + `` to emulate NULLS ordering whenever ``nulls_first`` is unset; + the bracketed alias INSIDE the CASE WHEN mis-resolves against the FROM + scope (``Invalid column name``). Pinning ``nulls_first`` to T-SQL's + native default for the direction (FIRST on ASC, LAST on DESC) + suppresses the wrapper. No-op on every other dialect. + """ + kwargs: dict = {"this": order_col, "desc": not ascending} + if self.dialect == "tsql": + kwargs["nulls_first"] = ascending + return exp.Ordered(**kwargs) def _build_combined(self, enriched: EnrichedQuery, base_sql: str) -> list[tuple[str, str]]: @@ -933,7 +1222,11 @@ def _build_combined(self, enriched: EnrichedQuery, alias=exp.to_identifier(cm.source_model_name), ) else: - source_from = exp.to_table(cm.source_sql_table, alias=cm.source_model_name) + # DEV-1686 reserved-word alias + DEV-1645 mixed-case + # physical-name quoting, both via ``_to_table``. + source_from = self._to_table( + cm.source_sql_table, alias=cm.source_model_name, + ) select = select.from_(source_from) # JOIN target model @@ -943,7 +1236,9 @@ def _build_combined(self, enriched: EnrichedQuery, alias=exp.to_identifier(cm.target_model_name), ) else: - target_join = exp.to_table(cm.target_model_sql_table, alias=cm.target_model_name) + target_join = self._to_table( + cm.target_model_sql_table, alias=cm.target_model_name, + ) join_on = exp.and_(*( exp.EQ( this=exp.Column(this=exp.to_identifier(src), table=exp.to_identifier(cm.source_model_name)), @@ -1067,7 +1362,9 @@ def _build_combined(self, enriched: EnrichedQuery, alias=exp.to_identifier(target_alias), ) else: - join_target = exp.to_table(target_table, alias=target_alias) + join_target = self._to_table( + target_table, alias=target_alias, + ) join_on = self._parse(join_cond) select = select.join(join_target, on=join_on, join_type=jtype.upper()) @@ -1092,9 +1389,9 @@ def _build_combined(self, enriched: EnrichedQuery, for m in enriched.measures: if not _has_cross_model_filter(m) and not _is_windowed_measure(m): base_cols.append(m.alias) - final_parts = [f'_base."{a}"' for a in base_cols] + final_parts = [f'_base.{self._quote_ident(a)}' for a in base_cols] for cte_name, alias, _ in measure_cte_refs: - final_parts.append(f'{cte_name}."{alias}"') + final_parts.append(f'{cte_name}.{self._quote_ident(alias)}') from_clause_str = "FROM _base" joined_ctes: set = set() @@ -1108,7 +1405,7 @@ def _build_combined(self, enriched: EnrichedQuery, effective_aliases = cte_join_aliases if cte_join_aliases is not None else join_aliases join_on_parts = [] for a in effective_aliases: - join_on_parts.append(f'_base."{a}" = {cte_name}."{a}"') + join_on_parts.append(f'_base.{self._quote_ident(a)} = {cte_name}.{self._quote_ident(a)}') if join_on_parts: from_clause_str += f"\nLEFT JOIN {cte_name} ON {' AND '.join(join_on_parts)}" else: @@ -1145,12 +1442,14 @@ def _assemble_combined_sql(self, enriched: EnrichedQuery, } for order_item in enriched.order: col = order_item.column - col_name = self._resolve_order_column(col=col, enriched=enriched) + ref = self._resolve_order_column(col=col, enriched=enriched) direction = "ASC" if order_item.direction == "asc" else "DESC" - if col_name in base_cols: - order_parts.append(f'_base."{col_name}" {direction}') + if ref.is_alias and ref.text in base_cols: + order_parts.append(f'_base.{self._quote_ident(ref.text)} {direction}') + elif ref.is_alias: + order_parts.append(f'{self._quote_ident(ref.text)} {direction}') else: - order_parts.append(f'"{col_name}" {direction}') + order_parts.append(f'{self._order_split_sql(ref)} {direction}') sql += "\nORDER BY " + ", ".join(order_parts) if enriched.limit is not None: sql += f"\nLIMIT {enriched.limit}" @@ -1159,16 +1458,29 @@ def _assemble_combined_sql(self, enriched: EnrichedQuery, return sql - @staticmethod - def _apply_pagination_to_sql(enriched: EnrichedQuery, sql: str) -> str: - """Apply ORDER BY, LIMIT, OFFSET to a raw SQL string.""" + def _apply_pagination_to_sql(self, enriched: EnrichedQuery, sql: str) -> str: + """Apply ORDER BY, LIMIT, OFFSET to a raw SQL string. + + This wrapper is only ever applied over the CTE-wrapped computed-column + assembly (its single caller builds ``WITH … SELECT … FROM ``), + so the outer FROM is a CTE — a SPLIT ``.`` reference + would name a table unbound in this scope. An unprojected (non-alias) + sort key is therefore unresolvable here (unlike the base-SELECT applier + ``_apply_order_limit``, where the split IS bound). Reject it rather than + emit invalid SQL — consistent with the typed pipeline's plan-time guard. + """ if enriched.order: order_parts = [] for order_item in enriched.order: col = order_item.column - col_name = SQLGenerator._resolve_order_column(col=col, enriched=enriched) + ref = SQLGenerator._resolve_order_column(col=col, enriched=enriched) direction = "ASC" if order_item.direction == "asc" else "DESC" - order_parts.append(f'"{col_name}" {direction}') + if ref.is_alias: + order_parts.append(f'{self._quote_ident(ref.text)} {direction}') + else: + raise UnresolvableOrderColumnError( + column=ref.column, qualifier=ref.qualifier, + ) sql += "\nORDER BY " + ", ".join(order_parts) if enriched.limit is not None: sql += f"\nLIMIT {enriched.limit}" @@ -1256,70 +1568,26 @@ def _build_time_offset_expr(self, col_expr: exp.Expression, offset: int, Used to shift raw timestamps before DATE_TRUNC in shifted CTEs so that aggregated time buckets align with the base query's buckets. """ - unit_map = {"year": "YEAR", "month": "MONTH", "day": "DAY", - "quarter": "MONTH", "week": "WEEK", "hour": "HOUR", - "minute": "MINUTE", "second": "SECOND"} - unit = unit_map.get(granularity, granularity.upper()) - val = offset * 3 if granularity == "quarter" else offset - - if self.dialect == "sqlite": - sqlite_units = {"YEAR": "years", "MONTH": "months", "DAY": "days", - "WEEK": "days", "HOUR": "hours", "MINUTE": "minutes", - "SECOND": "seconds"} - sqlite_unit = sqlite_units.get(unit, unit.lower() + "s") - sqlite_val = val * 7 if granularity == "week" else val - return exp.Anonymous( - this="DATE", - expressions=[col_expr, exp.Literal.string(f"{sqlite_val} {sqlite_unit}")], - ) - - # Standard SQL: col ± INTERVAL N UNIT (single-unit; sqlglot transpiles - # to the dialect-correct form, e.g. MySQL `INTERVAL N UNIT`, - # ClickHouse same, BigQuery same). - if val >= 0: - return exp.Add(this=col_expr, expression=exp.Interval( - this=exp.Literal.number(val), unit=exp.Var(this=unit), - )) - return exp.Sub(this=col_expr, expression=exp.Interval( - this=exp.Literal.number(-val), unit=exp.Var(this=unit), - )) + return self._dialect.build_time_offset_expr( + col_expr=col_expr, offset=offset, granularity=granularity, + ) def _duration_interval_exprs(self, duration: str, sign: int = 1) -> list[exp.Expression]: """Return per-unit AST nodes that `_add_intervals_expr` will chain. - Non-SQLite: one positive `exp.Interval` per parsed (amount, unit) pair. - The Add-vs-Sub direction is decided by `_add_intervals_expr` from its - own `sign` arg, not baked into the Interval — sqlglot transpiles each - single-unit interval per dialect (MySQL: `INTERVAL N UNIT`; - ClickHouse: same; BigQuery: same), avoiding the broken Postgres-shape - multi-unit literal `INTERVAL '1 year 2 month 3 day'` that fails on - every Tier-1+ non-SQLite/non-Postgres dialect. - - SQLite: one DATETIME-modifier string literal per pair, sign baked in. - Week is converted to `N*7 days` (SQLite has no week unit). + Delegates to the dialect strategy (DEV-1716) — Postgres-shape returns + ``exp.Interval`` nodes; SQLite returns DATETIME-modifier string + literals with sign baked in. """ parts = _parse_window_duration(duration) - if self.dialect == "sqlite": - prefix = "+" if sign >= 0 else "-" - return [ - exp.Literal.string( - f"{prefix}{(amount * 7 if unit == 'w' else amount)} " - f"{_WINDOW_UNIT_SQLITE[unit]}" - ) - for amount, unit in parts - ] - return [ - exp.Interval( - this=exp.Literal.number(amount), - unit=exp.Var(this=_WINDOW_UNIT_SQL[unit].upper()), - ) - for amount, unit in parts - ] + return self._dialect.duration_interval_exprs(parts=parts, sign=sign) def _granularity_interval_expr(self, granularity: TimeGranularity, sign: int = 1) -> list[exp.Expression]: if granularity == TimeGranularity.QUARTER: duration = "3m" - elif granularity == TimeGranularity.WEEK: + elif granularity in (TimeGranularity.WEEK, TimeGranularity.WEEK_SUNDAY): + # DEV-1572: a WEEK_SUNDAY shift spans one calendar week, same as WEEK + # (only the bucket anchor differs — Sunday vs Monday). duration = "1w" else: unit_to_duration = { @@ -1337,20 +1605,13 @@ def _add_intervals_expr(self, expr: exp.Expression, intervals: list[exp.Expressi sign: int = 1) -> exp.Expression: """Compose `expr ± interval [± interval ...]` as AST. - SQLite: wraps as `DATETIME(expr, mod1, mod2, ...)` (sign baked into - each modifier by `_duration_interval_exprs`); the `sign` arg is - ignored on SQLite. - Other dialects: chains `exp.Add` (sign>=0) or `exp.Sub` (sign<0). The - result transpiles per dialect via sqlglot — MySQL renders - `INTERVAL N UNIT` clauses unquoted, ClickHouse same, etc. + Delegates to the dialect strategy (DEV-1716) — defaults to chained + Add/Sub with ``exp.Interval`` nodes; SQLite wraps as ``DATETIME(...)``; + T-SQL chains ``DATEADD(...)`` calls. """ - if self.dialect == "sqlite": - return exp.Anonymous(this="DATETIME", expressions=[expr, *intervals]) - op_cls = exp.Add if sign >= 0 else exp.Sub - result = expr - for iv in intervals: - result = op_cls(this=result, expression=iv) - return result + return self._dialect.add_intervals_expr( + expr=expr, intervals=intervals, sign=sign, + ) def _build_window_source_cols( self, @@ -1474,7 +1735,7 @@ def _build_window_source_select( alias=exp.to_identifier(target_alias), ) else: - join_target = exp.to_table(target_table, alias=target_alias) + join_target = self._to_table(target_table, alias=target_alias) join_on = self._parse(join_cond) select = select.join(join_target, on=join_on, join_type=jtype.upper()) @@ -1577,8 +1838,10 @@ def _generate_base(self, enriched: EnrichedQuery, for dim in enriched.dimensions: col_expr = self._resolve_sql(sql=dim.sql, name=dim.name, model_name=dim.model_name, type=dim.type) if has_first_or_last: - # In ranked subquery, dimensions are already columns — reference directly - col_expr = exp.Column(this=exp.to_identifier(dim.name)) + # In ranked subquery, dimensions are already columns — reference + # directly. DEV-1645: quote a mixed-case name so this outer + # reference matches the ranked subquery's ``model.*`` output. + col_expr = exp.Column(this=self._to_ident(dim.name)) select_columns.append(col_expr.as_(dim.alias)) group_by_columns.append(col_expr) @@ -1636,7 +1899,13 @@ def _generate_base(self, enriched: EnrichedQuery, # isolated measures were skipped (to deduplicate the dimension spine), # or the query is dim-only (auto-dedup distinct dim/time-dim tuples # — applied before LIMIT so a row cap can't drop unique tuples). - dim_only_dedup = bool(group_by_columns) and not enriched.measures + # DEV-1543: distinct_dimension_values=False opts out of the dim-only + # dedup GROUP BY, emitting raw rows instead of distinct tuples. + dim_only_dedup = ( + enriched.distinct_dimension_values + and bool(group_by_columns) + and not enriched.measures + ) needs_group_by = ( has_aggregation or bool(enriched.cross_model_measures) @@ -1692,7 +1961,7 @@ def _generate_base(self, enriched: EnrichedQuery, this=parsed_target, alias=exp.to_identifier(target_alias), ) else: - join_target = exp.to_table(target_table, alias=target_alias) + join_target = self._to_table(target_table, alias=target_alias) join_on = self._parse(join_cond) select = select.join(join_target, on=join_on, join_type=jtype.upper()) @@ -1731,6 +2000,17 @@ def _generate_with_computed(self, enriched: EnrichedQuery, ctes = [("base", base_sql)] available_aliases = set(base_aliases) # Aliases available in the current layer + # DEV-1692: a per-generation collision-safe allocator for the transform + # layer's CTE names. The hoisted transform placeholder name (``t.name``, + # e.g. ``time_shift_inner``) restarts per formula, so two arithmetic- + # wrapped time_shifts would both mint ``shifted__time_shift_inner`` — a + # duplicate WITH name that silently shadows the first. Every existing / + # deterministic CTE name is RESERVED; the ``shifted_`` / ``sjoin_`` + # families are ALLOCATED around them (reserve-not-rename keeps the + # recompute-at-reference CTE families collision-free too — Codex F3). + cte_allocator = self._new_allocator() + cte_allocator.reserve(*(name for name, _ in ctes), *base_aliases) + # All transforms go into a unified layering loop. Each iteration tries # to resolve transforms whose inputs are available. Self-join transforms # (time_shift, change, change_pct) get their own CTE with a LEFT JOIN. @@ -1748,19 +2028,24 @@ def _generate_with_computed(self, enriched: EnrichedQuery, remaining_expressions = [] remaining_transforms = [] - # Collect window transforms and expressions that can go in one layer - layer_parts = [f'"{a}"' for a in sorted(available_aliases)] + # Collect window transforms and expressions that can go in one layer. + # DEV-1716: carried-forward alias refs are dialect-quoted. + layer_parts = [self._quote_ident(a) for a in sorted(available_aliases)] for expr in pending_expressions: if self._deps_available(expr.sql, available_aliases): # DEV-1361: when the source ModelMeasure declared a # result type, wrap the expression in CAST so the outer # SELECT yields the typed value. - expr_sql = expr.sql + # DEV-1716: ``expr.sql`` is ANSI-quoted (enrichment output); + # parse it as Postgres (where ``"..."`` is an identifier, + # NOT a string literal as MySQL would read it) and re-emit + # under the target dialect so identifiers get correct quotes. + parsed_expr = self._parse(expr.sql, dialect="postgres") if expr.type is not None: - wrapped = _wrap_cast_for_type(self._parse(expr_sql), expr.type) - expr_sql = wrapped.sql(dialect=self.dialect) - layer_parts.append(f'{expr_sql} AS "{expr.alias}"') + parsed_expr = _wrap_cast_for_type(parsed_expr, expr.type) + expr_sql = parsed_expr.sql(dialect=self.dialect) + layer_parts.append(f'{expr_sql} AS {self._quote_ident(expr.alias)}') added_this_layer.append(expr.alias) else: remaining_expressions.append(expr) @@ -1783,12 +2068,12 @@ def _generate_with_computed(self, enriched: EnrichedQuery, if t.type is not None: wrapped = _wrap_cast_for_type(self._parse(window_sql), t.type) window_sql = wrapped.sql(dialect=self.dialect) - layer_parts.append(f'{window_sql} AS "{t.alias}"') + layer_parts.append(f'{window_sql} AS {self._quote_ident(t.alias)}') added_this_layer.append(t.alias) # Emit window layer CTE if anything was added if added_this_layer: - layer_name = f"step{layer_num}" + layer_name = cte_allocator.allocate_cte(f"step{layer_num}") layer_select = "SELECT\n " + _SQL_COL_SEP.join(layer_parts) ctes.append((layer_name, f"{layer_select}\nFROM {prev_cte}")) available_aliases.update(added_this_layer) @@ -1799,26 +2084,34 @@ def _generate_with_computed(self, enriched: EnrichedQuery, for t in deferred_self_joins: src_cte = ctes[-1][0] - shift_name = f"shifted_{t.name}" + # DEV-1692: allocate collision-free CTE names (the local vars + # are used at both definition and every reference below, so a + # renamed name stays internally consistent). + shift_name = cte_allocator.allocate_cte(f"shifted_{t.name}") shifted_sql = self._generate_shifted_base( enriched=enriched, transform=t, ) ctes.append((shift_name, shifted_sql)) - # Build the self-join CTE: src LEFT JOIN shifted ON time equality - time_col = f'"{t.time_alias}"' + # Build the self-join CTE: src LEFT JOIN shifted ON time + # equality. DEV-1716: identifier leaves are dialect-quoted so + # MySQL/T-SQL/BigQuery emit correct quotes (not ANSI ``"..."``). + time_col = self._quote_ident(t.time_alias) join_cond = f'{src_cte}.{time_col} = {shift_name}.{time_col}' # Also join on all dimension columns for correct matching for dim in enriched.dimensions: - join_cond += f' AND {src_cte}."{dim.alias}" = {shift_name}."{dim.alias}"' + dim_col = self._quote_ident(dim.alias) + join_cond += f' AND {src_cte}.{dim_col} = {shift_name}.{dim_col}' col_sql = self._build_self_join_column( transform=t.transform, right_table=shift_name, measure_alias=t.measure_alias, ) - join_cols = ", ".join(f'{src_cte}."{a}"' for a in sorted(available_aliases)) - join_layer = f"sjoin_{t.name}" + join_cols = ", ".join( + f'{src_cte}.{self._quote_ident(a)}' for a in sorted(available_aliases) + ) + join_layer = cte_allocator.allocate_cte(f"sjoin_{t.name}") join_sql = ( - f"SELECT {join_cols}, {col_sql} AS \"{t.alias}\"\n" + f"SELECT {join_cols}, {col_sql} AS {self._quote_ident(t.alias)}\n" f"FROM {src_cte}\n" f"LEFT JOIN {shift_name}\n" f" ON {join_cond}" @@ -1839,6 +2132,12 @@ def _generate_with_computed(self, enriched: EnrichedQuery, ) ctes.extend(reset_layer) ctes.extend(value_layer) + # DEV-1692: reserve these deterministic names so a later + # ``shifted_`` / ``sjoin_`` allocation can never collide. + cte_allocator.reserve( + *(name for name, _ in reset_layer), + *(name for name, _ in value_layer), + ) available_aliases.add(t.alias) added_this_layer.append(t.alias) @@ -1856,12 +2155,12 @@ def _generate_with_computed(self, enriched: EnrichedQuery, final_cte = ctes[-1][0] - # Build final SELECT - final_parts = [f'"{a}"' for a in sorted(available_aliases)] + # Build final SELECT (DEV-1716: dialect-quoted projection aliases) + final_parts = [self._quote_ident(a) for a in sorted(available_aliases)] # Add any remaining expressions/transforms that couldn't be layered for expr in pending_expressions: - final_parts.append(f'{expr.sql} AS "{expr.alias}"') + final_parts.append(f'{expr.sql} AS {self._quote_ident(expr.alias)}') for t in pending_transforms: if t.transform in _SELF_JOIN_TRANSFORMS: continue # Should not happen — self-joins are always materialized @@ -1871,7 +2170,7 @@ def _generate_with_computed(self, enriched: EnrichedQuery, if t.type is not None: wrapped = _wrap_cast_for_type(self._parse(window_sql), t.type) window_sql = wrapped.sql(dialect=self.dialect) - final_parts.append(f'{window_sql} AS "{t.alias}"') + final_parts.append(f'{window_sql} AS {self._quote_ident(t.alias)}') outer_select = "SELECT\n " + _SQL_COL_SEP.join(final_parts) @@ -1895,7 +2194,7 @@ def _generate_with_computed(self, enriched: EnrichedQuery, # Wrap qualified names in quotes for alias references for col_name in dict.fromkeys(f.columns): qualified = f"{model}.{col_name}" - qualified_sql = qualified_sql.replace(qualified, f'"{qualified}"') + qualified_sql = qualified_sql.replace(qualified, self._quote_ident(qualified)) conditions.append(qualified_sql) where_clause = _SQL_AND_JOINER.join(conditions) sql = f"SELECT *\nFROM (\n{sql}\n) AS _filtered\nWHERE {where_clause}" @@ -2010,62 +2309,32 @@ def _predicate() -> exp.Expression: return [(reset_cte, reset_sql)], [(value_cte, value_sql)] def _build_date_trunc(self, col_expr: exp.Expression, granularity: TimeGranularity) -> exp.Expression: - """Build a DATE_TRUNC expression, with SQLite STRFTIME fallback. - - When ``col_expr`` is not a bare column reference (e.g., a string - literal or other unknown-typed sub-expression), the result is - wrapped in ``CAST(... AS TIMESTAMP)`` before being passed to - ``DATE_TRUNC``. Postgres has multiple ``date_trunc`` overloads - keyed on the second argument's type; an ``unknown``-typed operand - (the bare literal `'2025-12-01'`) makes the planner fail with - ``function date_trunc(unknown, unknown) is not unique``. The cast - pins one overload. Bare columns are left alone — their live DB - type is already known, and an explicit cast could strip a - ``TIMESTAMPTZ`` to ``TIMESTAMP``. Idempotent: already-cast - expressions pass through unchanged. + """Build a DATE_TRUNC expression. Dispatches to the dialect strategy + (DEV-1716). + + The dialect determines the wire form — DATE_TRUNC for + Postgres/DuckDB/ClickHouse, STRFTIME for SQLite (with CASE WHEN for + quarter and weekday-modifier for week), DATETRUNC for T-SQL, native + Sunday-week for BigQuery. Cast-wrapping of non-column operands and the + WEEK_SUNDAY day-shift are handled inside the dialect (base) impl. """ - gran_str = _GRANULARITY_MAP.get(granularity, granularity.value) - if self.dialect == "sqlite": - # SQLite has no DATE_TRUNC — use STRFTIME - fmt_map = { - "year": "%Y-01-01", - "month": "%Y-%m-01", - "day": "%Y-%m-%d", - "hour": "%Y-%m-%d %H:00:00", - "minute": "%Y-%m-%d %H:%M:00", - "second": "%Y-%m-%d %H:%M:%S", - } - # Week: SQLite weekday 0=Sunday, use date() with weekday modifier - if gran_str == "week": - return self._parse(f"DATE({col_expr.sql(dialect='sqlite')}, 'weekday 0', '-6 days')", dialect="sqlite") - if gran_str == "quarter": - # Quarter start: derive from month - col_sql = col_expr.sql(dialect="sqlite") - return self._parse( - f"STRFTIME('%Y-', {col_sql}) || CASE " - f"WHEN CAST(STRFTIME('%m', {col_sql}) AS INTEGER) <= 3 THEN '01-01' " - f"WHEN CAST(STRFTIME('%m', {col_sql}) AS INTEGER) <= 6 THEN '04-01' " - f"WHEN CAST(STRFTIME('%m', {col_sql}) AS INTEGER) <= 9 THEN '07-01' " - f"ELSE '10-01' END", - dialect="sqlite", - ) - fmt = fmt_map.get(gran_str, "%Y-%m-%d") - return exp.Anonymous( - this="STRFTIME", - expressions=[exp.Literal.string(fmt), col_expr], - ) - if not isinstance(col_expr, (exp.Column, exp.Cast)): - col_expr = exp.Cast(this=col_expr, to=exp.DataType.build("TIMESTAMP")) - return exp.DateTrunc(this=col_expr, unit=exp.Literal.string(gran_str)) + return self._dialect.build_date_trunc( + col_expr=col_expr, granularity=granularity, parse=self._parse, + ) - @staticmethod - def _build_transform_sql(t) -> str: # NOSONAR S3776 — flat dispatch over transform names; per-transform SQL forms read better as one if/elif tree than as named helpers - """Build a window function SQL expression for a transform.""" - measure = f'"{t.measure_alias}"' - time_col = f'"{t.time_alias}"' if t.time_alias else None + def _build_transform_sql(self, t) -> str: # NOSONAR S3776 — flat dispatch over transform names; per-transform SQL forms read better as one if/elif tree than as named helpers + """Build a window function SQL expression for a transform. + + DEV-1716: identifier refs are dialect-quoted (``_quote_ident``) so + MySQL/T-SQL/BigQuery get correct quotes; the subsequent + ``self._parse(window_sql)`` reads them back as identifiers (backticks + on MySQL, brackets on T-SQL) rather than string literals. + """ + measure = self._quote_ident(t.measure_alias) + time_col = self._quote_ident(t.time_alias) if t.time_alias else None partition_cols = getattr(t, "partition_aliases", []) or [] partition_clause = ( - _SQL_PARTITION_BY + ", ".join(f'"{a}"' for a in partition_cols) + _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_cols) if partition_cols else "" ) @@ -2111,11 +2380,14 @@ def _build_transform_sql(t) -> str: # NOSONAR S3776 — flat dispatch over tran else: raise ValueError(f"Unsupported transform: {t.transform}") - @staticmethod - def _build_self_join_column(transform: str, right_table: str, + def _build_self_join_column(self, transform: str, right_table: str, measure_alias: str) -> str: - """Build the SELECT expression for a self-join transform.""" - prev = f'{right_table}."{measure_alias}"' + """Build the SELECT expression for a self-join transform. + + DEV-1716: the column leaf is dialect-quoted so MySQL/T-SQL/BigQuery + get correct quoting (not ANSI ``"..."``). + """ + prev = f'{right_table}.{self._quote_ident(measure_alias)}' if transform == "time_shift": return prev raise ValueError(f"Unknown self-join transform: {transform}") @@ -2125,10 +2397,16 @@ def _apply_order_limit(self, select: exp.Select, enriched: EnrichedQuery) -> exp if enriched.order: for order_item in enriched.order: col = order_item.column - col_name = self._resolve_order_column(col=col, enriched=enriched) - order_col = exp.Column(this=exp.to_identifier(col_name, quoted=True)) + ref = self._resolve_order_column(col=col, enriched=enriched) + if ref.is_alias: + order_col = exp.Column(this=exp.to_identifier(ref.text, quoted=True)) + else: + order_col = exp.Column( + this=self._to_ident(ref.column), + table=exp.to_identifier(ref.qualifier), + ) ascending = order_item.direction == "asc" - select = select.order_by(exp.Ordered(this=order_col, desc=not ascending)) + select = select.order_by(self._ordered(order_col, ascending=ascending)) if enriched.limit is not None: select = select.limit(enriched.limit) @@ -2138,20 +2416,41 @@ def _apply_order_limit(self, select: exp.Select, enriched: EnrichedQuery) -> exp return select + def _order_split_sql(self, ref: _OrderColRef) -> str: + """DEV-1645: emit a non-projected ORDER BY key as a SPLIT + ``qualifier.column`` reference (mixed-case-quoted), not one + composite-quoted token.""" + col = exp.Column(this=self._to_ident(ref.column), table=exp.to_identifier(ref.qualifier)) + return col.sql(dialect=self.dialect) + @staticmethod - def _resolve_order_column(col, enriched: EnrichedQuery) -> str: - """Resolve an order column reference to the correct enriched alias. + def _resolve_order_column(col, enriched: EnrichedQuery) -> _OrderColRef: + """Resolve an order column reference to a discriminated result (DEV-1645). Users refer to columns by their short name (e.g., ``count``, ``revenue_sum``). The enriched query stores fully qualified aliases (e.g., ``orders._count``, ``orders.revenue_sum``). This method - matches the user-provided name against all enriched columns and - returns the matching alias. If no match is found, the name is - qualified with the model name as a fallback. - - For ``*:count`` results, the internal name is ``_count`` but users - refer to it as ``count``. A fallback check for ``_name`` handles - this case. + matches the user-provided name against all enriched columns. + + When it matches a projected alias, the result carries ``is_alias=True`` + and the caller emits it whole-quoted (``"orders.revenue_sum"`` — that IS + the real output column name via ``AS "orders.revenue_sum"``). + + When no projected alias matches (renamed via ``columns:``, or an + inner-stage dim the outer stage dropped), the result carries + ``is_alias=False`` with ``qualifier``/``column`` set, and the caller + emits a SPLIT ``qualifier.column`` reference that resolves against the + FROM-scope table — instead of the old composite ``"."`` + token that Postgres rejects as UndefinedColumn. + + A joined qualifier (anything other than the base model) is rejected with + ``UnresolvableOrderColumnError``: the compiler's outer-wrapping layers + (measure CTEs, pagination, the first/last ranked subquery, projection + trimming) relocate the ORDER BY into a scope where the joined table is + unbound, so emitting a reference there would produce invalid SQL. + + For ``*:count`` results, the internal name is ``_count`` but users refer + to it as ``count``. A fallback check for ``_name`` handles this case. """ user_name = col.name model_prefix = col.model or enriched.model_name @@ -2173,24 +2472,39 @@ def _resolve_order_column(col, enriched: EnrichedQuery) -> str: # Custom field names (e.g., {"formula": "x:count_distinct", "name": "my_name"}) alias_lookup.update(enriched.field_name_aliases) + # A ref qualified with a FOREIGN model (``owners.status`` when the base + # model is ``orders``) must not resolve to a same-named local column via + # the bare-name / ``_name`` lookups — that silently sorts by the wrong + # field. Only unqualified refs, or refs qualified with the base model + # itself, take the bare shortcuts; a foreign qualifier falls through to + # the ``.`` qualified match and then the joined-qualifier + # rejection below. Mirrors the typed pipeline's plan-time guard. + host_local = model_prefix == enriched.model_name + # Direct match on the user-provided name - if user_name in alias_lookup: - return alias_lookup[user_name] + if host_local and user_name in alias_lookup: + return _OrderColRef(alias_lookup[user_name], True, None, None) # Qualified match for cross-model measures: # col.model="customers", col.name="revenue_sum" → "customers.revenue_sum" if col.model: qualified = f"{col.model}.{col.name}" if qualified in alias_lookup: - return alias_lookup[qualified] + return _OrderColRef(alias_lookup[qualified], True, None, None) # Fallback for *:count → _count: user says "count", internal is "_count" prefixed = f"_{user_name}" - if prefixed in alias_lookup: - return alias_lookup[prefixed] - - # Fallback: qualify with model prefix - return f"{model_prefix}.{user_name}" + if host_local and prefixed in alias_lookup: + return _OrderColRef(alias_lookup[prefixed], True, None, None) + + # Fallback: a non-projected order key is only safe to emit as a split + # reference against the BASE-model alias. A joined qualifier (anything + # other than the base model) is rejected — even when a filter pulls the + # join into the base FROM, the outer-wrapping layers relocate the ORDER + # BY into a scope where the joined table is unbound. + if model_prefix != enriched.model_name: + raise UnresolvableOrderColumnError(column=user_name, qualifier=model_prefix) + return _OrderColRef(f"{model_prefix}.{user_name}", False, model_prefix, user_name) # ------------------------------------------------------------------ # FROM / JOIN building @@ -2198,7 +2512,10 @@ def _resolve_order_column(col, enriched: EnrichedQuery) -> str: def _build_from_clause(self, enriched: EnrichedQuery) -> exp.Expression: if enriched.sql_table: - return exp.to_table(enriched.sql_table, alias=enriched.model_name) + # DEV-1686: Identifier-node alias so a reserved-word source relation + # quotes on emit (``AS "order"``). DEV-1645: ``_to_table`` also quotes + # a mixed-case physical table name (``public.MyTable``). + return self._to_table(enriched.sql_table, alias=enriched.model_name) elif enriched.sql: parsed = self._parse(enriched.sql) return exp.Subquery(this=parsed, alias=exp.to_identifier(enriched.model_name)) @@ -2370,9 +2687,9 @@ def _rewrite_log_aliases(self, node: exp.Expression) -> exp.Expression: base_val = float(base.this) except (TypeError, ValueError): return node - if base_val == 10 and self.dialect in _LOG10_NATIVE_DIALECTS: + if base_val == 10 and self._dialect.should_use_native_log(10): return exp.Anonymous(this="log10", expressions=[arg.copy()]) - if base_val == 2 and self.dialect in _LOG2_NATIVE_DIALECTS: + if base_val == 2 and self._dialect.should_use_native_log(2): return exp.Anonymous(this="log2", expressions=[arg.copy()]) return node @@ -2393,11 +2710,14 @@ def _resolve_sql( ``type``. """ if sql is None: - return exp.Column(this=exp.to_identifier(name), table=exp.to_identifier(model_name)) + # DEV-1645: quote the mixed-case column leaf; the model qualifier is + # a SLayer-internal alias and stays unquoted (reserved names quote + # at emit). + return exp.Column(this=self._to_ident(name), table=exp.to_identifier(model_name)) # Bare column name → qualify with model name # Use isidentifier() to distinguish column names from literals (e.g. "1") if sql.isidentifier(): - return exp.Column(this=exp.to_identifier(sql), table=exp.to_identifier(model_name)) + return exp.Column(this=self._to_ident(sql), table=exp.to_identifier(model_name)) return _wrap_cast_for_type(self._parse(sql), type) def _resolve_value_sql(self, spec: AggRenderSpec) -> str: @@ -2414,6 +2734,33 @@ def _resolve_value_sql(self, spec: AggRenderSpec) -> str: type=spec.column_type, ).sql(dialect=self.dialect) + def _agg_param_ast( + self, value: "ResolvedAggKwarg | str", *, model_name: str, + ) -> exp.Expression: + """Resolve a parametric-agg param value to a sqlglot AST. + + DEV-1706 (D-I): a ``ResolvedAggKwarg`` with ``kind="expr"`` is a trusted, + scope-resolved expression embedded directly; ``kind="str"`` (and a plain + model-level default ``str``) resolve through ``_resolve_sql`` so bare + identifiers qualify under ``model_name`` — the pre-DEV-1706 behaviour. + ``_SAFE_AGG_PARAM_RE`` guarding of ``kind="str"`` query values is applied + by the callers before this point. + """ + if isinstance(value, ResolvedAggKwarg): + if value.kind == "expr": + # Return a COPY: the same ResolvedAggKwarg (keyed by AggregateKey) + # is embedded into more than one AST when a C13 slot with two + # declared aliases visits the same key twice in base_render_order. + # sqlglot re-parents a node on attach, so sharing the node would + # corrupt the first tree — mirror ScopeFrame.resolve's .copy() + # discipline (slayer/sql/scope.py). + return value.value.copy() if isinstance(value.value, exp.Expression) \ + else self._parse(value.value) + raw = value.value + else: + raw = value + return self._resolve_sql(sql=raw, name=raw, model_name=model_name) + def _resolve_agg_param( self, spec: AggRenderSpec, @@ -2430,34 +2777,55 @@ def _resolve_agg_param( ``_build_stat_agg`` (``other=``); mirrors ``weighted_avg``'s ``weight=`` flow. """ - raw: Optional[str] = None + value: "ResolvedAggKwarg | str | None" = None if name in spec.agg_kwargs: - raw = spec.agg_kwargs[name] - _validate_agg_param_value(raw, name, agg_name) + value = spec.agg_kwargs[name] + # Guard the untrusted string forms: a ``kind="str"`` wrapper OR a + # bare ``str`` (the legacy ``EnrichedMeasure`` adapter and model-level + # defaults reach here unwrapped). ``kind="expr"`` is a trusted, + # bind-time-resolved expression and is embedded verbatim. + if isinstance(value, ResolvedAggKwarg): + if value.kind == "str": + _validate_agg_param_value(value.value, name, agg_name) + elif isinstance(value, str): + _validate_agg_param_value(value, name, agg_name) elif spec.aggregation_def: for param in spec.aggregation_def.params: if param.name == name: - raw = param.sql + value = param.sql break - if raw is None: + if value is None: raise ValueError( f"Aggregation '{agg_name}' requires parameter '{name}'. " f"Set it in the model's aggregation definition or at query time " f"(e.g., 'measure:{agg_name}({name}=column)')." ) - return self._resolve_sql( - sql=raw, name=raw, model_name=spec.model_name, + return self._agg_param_ast( + value, model_name=spec.model_name, ).sql(dialect=self.dialect) def _build_agg( self, - spec: AggRenderSpec, + spec: "AggRenderSpec | None" = None, rn_suffix_map: Optional[dict[str, str]] = None, default_time_col: Optional[str] = None, filtered_rn_map: Optional[dict[str, str]] = None, filtered_match_map: Optional[dict[str, str]] = None, + *, + measure: "EnrichedMeasure | None" = None, ) -> tuple[exp.Expression, bool]: - """Build an aggregation expression from an AggRenderSpec.""" + """Build an aggregation expression from an ``AggRenderSpec``. + + DEV-1716 compat: callers may pass a legacy ``EnrichedMeasure`` via the + ``measure=`` keyword instead of ``spec`` — it is adapted through + ``_agg_render_spec_from_enriched``. The typed pipeline uses ``spec`` + (DEV-1452 decoupling); the ``measure=`` surface preserves the + main-branch delegation-test interface without reverting that split. + """ + if measure is not None: + spec = _agg_render_spec_from_enriched(measure) + if spec is None: # pragma: no cover — defensive + raise ValueError("_build_agg requires either 'spec' or 'measure'.") agg_name = spec.aggregation if not agg_name: # Not an aggregation — raw expression @@ -2536,6 +2904,18 @@ def _build_agg( # mirrors _build_median. if agg_name in _STAT_AGG_NAMES: return self._build_stat_agg(spec), True + # count_distinct_approx (DEV-1595): dialect-aware approximate- + # distinct — native function (DuckDB/ClickHouse/BigQuery/…) or the + # exact COUNT(DISTINCT) fallback (Postgres/SQLite/MySQL). Built like + # percentile/stat-agg (via _wrap_filter + _resolve_value_sql) so a + # row-level filter wraps as COUNT(DISTINCT (CASE WHEN ... END)). + if agg_name == "count_distinct_approx": + col_expr = _wrap_filter( + self._resolve_value_sql(spec), spec.filter_sql + ) + return self._dialect.build_approx_count_distinct( + col_sql=col_expr, parse=self._parse + ), True return self._build_formula_agg(spec, agg_name), True # --- Resolve inner expression --- @@ -2585,7 +2965,7 @@ def _build_agg( agg_class = agg_class_map[agg_func] return agg_class(this=inner), True - def _build_formula_agg(self, spec: AggRenderSpec, agg_name: str) -> exp.Expression: + def _build_formula_agg(self, spec: AggRenderSpec, agg_name: str) -> exp.Expression: # NOSONAR(S3776) — sequential dispatch over formula source (aggregation_def vs built-in) and per-kind ResolvedAggKwarg substitution (DEV-1527); one cohesive template-substitution contract. """Build SQL for formula-based aggregations (weighted_avg, custom).""" # Get formula: from aggregation_def or built-in formula = None @@ -2606,9 +2986,12 @@ def _build_formula_agg(self, spec: AggRenderSpec, agg_name: str) -> exp.Expressi param_defaults = {p.name: p.sql for p in spec.aggregation_def.params} params = {**param_defaults, **spec.agg_kwargs} - # Validate query-time parameter values to prevent SQL injection + # Validate query-time parameter values to prevent SQL injection. Only the + # untrusted ``kind="str"`` form is guarded; ``kind="expr"`` is a trusted, + # bind-time-resolved expression (DEV-1706 D-I). for pname, pval in spec.agg_kwargs.items(): - _validate_agg_param_value(pval, pname, agg_name) + if isinstance(pval, ResolvedAggKwarg) and pval.kind == "str": + _validate_agg_param_value(pval.value, pname, agg_name) # Validate required params required = BUILTIN_AGGREGATION_REQUIRED_PARAMS.get(agg_name, []) @@ -2631,8 +3014,8 @@ def _build_formula_agg(self, spec: AggRenderSpec, agg_name: str) -> exp.Expressi col_expr = _wrap_filter(self._resolve_value_sql(spec), spec.filter_sql) substituted = formula.replace("{value}", col_expr) for param_name, param_val in params.items(): - param_ast = self._resolve_sql( - sql=param_val, name=param_val, model_name=spec.model_name, + param_ast = self._agg_param_ast( + param_val, model_name=spec.model_name, ) param_expr = param_ast.sql(dialect=self.dialect) if spec.filter_sql and not isinstance(param_ast, exp.Literal): @@ -2642,20 +3025,10 @@ def _build_formula_agg(self, spec: AggRenderSpec, agg_name: str) -> exp.Expressi return self._parse(substituted) def _build_median(self, inner: exp.Expression) -> exp.Expression: - """Build a median aggregation expression (dialect-dependent).""" - inner_sql = inner.sql(dialect=self.dialect) - if self.dialect == "mysql": - raise NotImplementedError( - "Aggregation 'median' is not supported on MySQL: MySQL has no native " - "MEDIAN/PERCENTILE_CONT function and no Python UDF mechanism. " - "Use MariaDB (has MEDIAN()) or compute the value client-side." - ) - if self.dialect in ("sqlite", "clickhouse"): - # SQLite: provided by the median() UDF registered on connect. - # ClickHouse: native median() aggregate. - return self._parse(f"median({inner_sql})") - # Postgres, DuckDB, and most others: PERCENTILE_CONT - return self._parse(f"PERCENTILE_CONT(0.5) WITHIN GROUP (ORDER BY {inner_sql})") + """Build a median aggregation expression. Dispatches to the dialect + (DEV-1716) — MySQL/T-SQL raise NotImplementedError, SQLite/ClickHouse + emit ``median()``, others ``PERCENTILE_CONT(0.5)``.""" + return self._dialect.build_median(inner=inner, parse=self._parse) def _build_percentile(self, spec: AggRenderSpec) -> exp.Expression: """Build a PERCENTILE_CONT(p) aggregation expression (dialect-dependent). @@ -2691,25 +3064,14 @@ def _build_percentile(self, spec: AggRenderSpec) -> exp.Expression: f"Aggregation 'percentile' parameter 'p' must be in [0, 1]; got {p_float}." ) - if self.dialect == "mysql": - raise NotImplementedError( - "Aggregation 'percentile' is not supported on MySQL: MySQL has no native " - "PERCENTILE_CONT function and no Python UDF mechanism. " - "Use MariaDB or compute the value client-side." - ) - + # Pass the **original string** ``p`` (not ``p_float``) to the dialect so + # user literals like ``0.50`` / ``1`` / ``5e-2`` survive verbatim. + # DEV-1716: dialect owns the wire form (MySQL/T-SQL raise, SQLite UDF, + # ClickHouse parametric ``quantile(p)(x)``, others ``PERCENTILE_CONT``). col_expr = _wrap_filter(self._resolve_value_sql(spec), spec.filter_sql) - - if self.dialect == "sqlite": - # Provided by the percentile_cont(value, p) UDF registered on connect. - sql_str = f"percentile_cont({col_expr}, {p})" - elif self.dialect == "clickhouse": - # ClickHouse parametric aggregate syntax. - sql_str = f"quantile({p})({col_expr})" - else: - sql_str = f"PERCENTILE_CONT({p}) WITHIN GROUP (ORDER BY {col_expr})" - - return self._parse(sql_str) + return self._dialect.build_percentile( + p_str=p, col_sql=col_expr, parse=self._parse, + ) def _build_stat_agg(self, spec: AggRenderSpec) -> exp.Expression: """Build SQL for the statistical aggregations added in DEV-1317. @@ -2738,6 +3100,9 @@ def _build_stat_agg(self, spec: AggRenderSpec) -> exp.Expression: # MySQL-not-supported error when both conditions hold — the # missing-param message points at the actual user mistake. Closes # Codex #5 on PR #82. + # Resolve the `other=` kwarg BEFORE any dialect guard so a + # missing-required-param error takes priority over a dialect-specific + # error (the missing-param message points at the actual user mistake). other_expr: Optional[str] = None if agg_name in _TWO_ARG_STAT_AGGS: other_expr = _wrap_filter( @@ -2745,40 +3110,23 @@ def _build_stat_agg(self, spec: AggRenderSpec) -> exp.Expression: spec.filter_sql, ) - if agg_name in _TWO_ARG_STAT_AGGS and self.dialect == "mysql": - raise NotImplementedError( - f"Aggregation '{agg_name}' is not supported on MySQL: MySQL has no " - f"native {agg_name.upper()} function and no Python UDF mechanism. " - f"Use MariaDB or compute the value client-side." - ) - col_expr = _wrap_filter(self._resolve_value_sql(spec), spec.filter_sql) + # DEV-1716: the dialect owns the wire form — native CORR/COVAR on + # Postgres/DuckDB/ClickHouse, variance-decomposition formula on + # MySQL/T-SQL; canonical stddev/var name (sqlglot-transpiled) with the + # MySQL ``exp.Anonymous`` var_samp/var_pop bypass in the dialect class. if agg_name in _TWO_ARG_STAT_AGGS: - sql_str = f"{agg_name.upper()}({col_expr}, {other_expr})" - else: - # stddev_samp, stddev_pop, var_samp, var_pop: emit the - # canonical Postgres-style name and let sqlglot transpile per - # dialect (e.g., var_samp → VARIANCE on SQLite/DuckDB/MySQL, - # var_pop → VARIANCE_POP on SQLite/MySQL). Both spellings - # resolve via the SQLite UDF aliases. - # - # MySQL exception: sqlglot's MySQL dialect rewrites - # ``VAR_POP`` → ``VARIANCE_POP`` (no such function in MySQL — - # only VAR_POP / VARIANCE exist) and ``VAR_SAMP`` → - # ``VARIANCE`` (silently wrong, since MySQL's ``VARIANCE`` - # equals ``VAR_POP`` — sample variance gets aliased to - # population variance). Bypass both by emitting the - # MySQL-native names through ``exp.Anonymous``, which - # sqlglot leaves verbatim. - if self.dialect == "mysql" and agg_name in {"var_samp", "var_pop"}: - return exp.Anonymous( - this=agg_name.upper(), - expressions=[self._parse(col_expr)], - ) - sql_str = f"{agg_name.upper()}({col_expr})" - - return self._parse(sql_str) + assert other_expr is not None # set above when two-arg + return self._dialect.build_covar_2arg( + agg_name=agg_name, + col_sql=col_expr, + other_sql=other_expr, + parse=self._parse, + ) + return self._dialect.build_stat_agg_1arg( + agg_name=agg_name, col_expr=col_expr, parse=self._parse, + ) # ------------------------------------------------------------------ # WHERE / HAVING (filters still use ColumnRef for member resolution) @@ -2886,7 +3234,27 @@ def _build_where_and_having( # so silent parity drift is impossible. # ====================================================================== - def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-model / transform-chain / plain branches plus the conditional outer-trim wrap. Each branch is a coherent compilation strategy; extracting would scatter the shared planned_query / slots_by_id / aliases_by_slot_id state across helpers without simplifying anything. + def generate_from_planned(self, planned_query, *, bundle) -> str: + """Render a typed ``PlannedQuery`` to SQL (public entry). + + DEV-1708 (D-E): installs a fresh generation-wide ``AliasAllocator`` for + the duration of this call and restores the caller's on exit. Inline + forward ``_cm_*`` CTEs and the host base share this one allocator, so + their ``_val_`` materialisation names never collide; a recursive + rerooted sub-generation (``_render_rerooted_cross_model_cte`` → + ``generate_from_planned``) is a self-contained statement and gets its + own allocator, with the parent's restored afterwards. + """ + prev_allocator = getattr(self, "_gen_allocator", None) + self._gen_allocator = self._new_allocator() + try: + return self._generate_from_planned_impl( + planned_query, bundle=bundle, + ) + finally: + self._gen_allocator = prev_allocator + + def _generate_from_planned_impl( # NOSONAR(S3776) — top-level dispatch over cross-model / transform-chain / plain branches plus the conditional outer-trim wrap. Each branch is a coherent compilation strategy; extracting would scatter the shared planned_query / slots_by_id / aliases_by_slot_id state across helpers without simplifying anything. self, planned_query, *, @@ -2894,6 +3262,12 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m ) -> str: """Render a typed ``PlannedQuery`` to SQL. + NOTE (DEV-1716): this is a STAGE renderer — its output feeds + ``generate_planned_stages``' flat-column stage-schema wrapper, so the + dialect ``rewrite_emitted_sql`` alias-mangling post-pass is applied by + the DB-bound terminal (``generate_planned_stages``), NOT here. Mangling + a stage's column names would break the downstream flat-name binding. + Mirrors the local-only branch of ``_generate_base`` but reads from typed PlannedQuery fields (``row_slots`` / ``aggregate_slots`` / ``filters_by_phase`` / ``order`` / ``transform_layers``) @@ -2921,7 +3295,10 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m ) source_relation = planned_query.source_relation - if planned_query.cross_model_aggregate_plans: + if ( + planned_query.cross_model_aggregate_plans + or planned_query.windowed_aggregate_plans + ): return self._render_with_cross_model_plans( planned_query=planned_query, bundle=bundle, ) @@ -3022,7 +3399,13 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m # has_aggregation triggers GROUP BY (dim-only emits GROUP BY # before LIMIT so unique dim tuples can't silently drop past # row N). - dim_only_dedup = bool(group_by_keys) and not has_aggregation + # DEV-1543: distinct_dimension_values=False opts out of the dim-only + # dedup GROUP BY, emitting raw rows instead of distinct tuples. + dim_only_dedup = ( + planned_query.distinct_dimension_values + and bool(group_by_keys) + and not has_aggregation + ) needs_group_by = has_aggregation or dim_only_dedup if needs_group_by and group_by_keys: for gb in group_by_keys.values(): @@ -3067,6 +3450,25 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m # 7b.10 — transform layers present. Build the CTE chain. base_cte_sql = base_select.sql(dialect=self.dialect, pretty=True) ctes: list[tuple[str, str]] = [("base", base_cte_sql)] + # DEV-1692: collision-safe CTE-name allocator for the whole transform + # chain. The hoisted time_shift slot alias (``_time_shift_inner``) + # repeats across arithmetic-wrapped shifts, so two ``shifted_`` / + # ``sjoin_`` pairs would otherwise share a name (duplicate WITH). Every + # CTE name is reserved/allocated through this one allocator so the + # ``step`` / ``shifted_`` / ``sjoin_`` / ``cp_`` families never collide. + cte_allocator = self._new_allocator() + cte_allocator.reserve(*(name for name, _ in ctes)) + # Codex (PR #269): also reserve every already-projected column alias's + # BARE form so a hidden transform alias minted below + # (``_time_shift_inner`` / ``_consecutive_periods_inner``) can never + # shadow a real user column of that name — mirrors the legacy path + # seeding ``base_aliases`` into its allocator. + _alias_prefix = f"{source_relation}." + cte_allocator.reserve(*( + a[len(_alias_prefix):] if a.startswith(_alias_prefix) else a + for aliases in aliases_by_slot_id.values() + for a in aliases + )) # "Pick one" map for transform-input / time-key / partition-key / # order-entry / POST-filter lookups. Initialised from the first # alias of every materialised slot. @@ -3083,11 +3485,13 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m # the same WHERE minus BetweenKey date_range filters). Built # once outside the loop since the source filters don't change # across layers. - shifted_where_parts = self._build_shifted_cte_where_parts( - planned_query=planned_query, - source_relation=source_relation, - source_model=source_model, - bundle=bundle, + shifted_where_parts, shifted_where_join_paths = ( + self._build_shifted_cte_where_parts( + planned_query=planned_query, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + ) ) while pending_layers: ready_window: list = [] @@ -3117,12 +3521,12 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m # --- Window batch (one step CTE per Kahn batch) ---------- if ready_window: step_num += 1 - step_name = f"step{step_num}" + step_name = cte_allocator.allocate_cte(f"step{step_num}") prev_cte = ctes[-1][0] carry_aliases_sorted = sorted( a for aliases in aliases_by_slot_id.values() for a in aliases ) - step_parts = [f'"{a}"' for a in carry_aliases_sorted] + step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] for layer in ready_window: for slot_id in layer.slot_ids: slot = slots_by_id[slot_id] @@ -3144,7 +3548,7 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m self._parse(window_sql), slot.type, ) window_sql = wrapped.sql(dialect=self.dialect) - step_parts.append(f'{window_sql} AS "{full_alias}"') + step_parts.append(f'{window_sql} AS {self._quote_ident(full_alias)}') aliases_by_slot_id.setdefault(slot_id, []).append( full_alias, ) @@ -3164,6 +3568,7 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m self._emit_time_shift_ctes_for_planned( slot=slot, ctes=ctes, + cte_allocator=cte_allocator, slots_by_id=slots_by_id, slot_id_by_key=slot_id_by_key, available_alias_by_slot_id=available_alias_by_slot_id, @@ -3171,6 +3576,7 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m source_model=source_model, source_relation=source_relation, shifted_where_parts=shifted_where_parts, + shifted_where_join_paths=shifted_where_join_paths, planned_query=planned_query, bundle=bundle, ) @@ -3181,6 +3587,7 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m self._emit_consecutive_periods_ctes_for_planned( slot=slot, ctes=ctes, + cte_allocator=cte_allocator, slots_by_id=slots_by_id, slot_id_by_key=slot_id_by_key, available_alias_by_slot_id=available_alias_by_slot_id, @@ -3219,7 +3626,7 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m carry_aliases_sorted = sorted( a for aliases in aliases_by_slot_id.values() for a in aliases ) - step_parts = [f'"{a}"' for a in carry_aliases_sorted] + step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] for cslot in unmaterialised: alias = ( cslot.public_aliases[0] @@ -3238,7 +3645,7 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m self._parse(expr_sql), cslot.type, ) expr_sql = wrapped.sql(dialect=self.dialect) - step_parts.append(f'{expr_sql} AS "{full_alias}"') + step_parts.append(f'{expr_sql} AS {self._quote_ident(full_alias)}') aliases_by_slot_id.setdefault(cslot.id, []).append( full_alias, ) @@ -3260,7 +3667,7 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m ) inner_sql = ( "SELECT\n " - + _SQL_COL_SEP.join(f'"{a}"' for a in inner_sorted) + + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_sorted) + f"\nFROM {final_cte}" ) @@ -3304,7 +3711,7 @@ def generate_from_planned( # NOSONAR(S3776) — top-level dispatch over cross-m public_aliases_user_order.append(alias) outer_sql = ( "SELECT\n " - + _SQL_COL_SEP.join(f'"{a}"' for a in public_aliases_user_order) + + _SQL_COL_SEP.join(self._quote_ident(a) for a in public_aliases_user_order) + f"\nFROM (\n{chain_sql}\n) AS _outer" ) @@ -3452,6 +3859,42 @@ def _walk(key) -> Optional[str]: f"slice.", ) + @staticmethod + def _composite_has_remote_operand( + *, + key, + slots_by_id: Dict[str, Any], + slot_id_by_key: Dict[Any, str], + planned_query, + ) -> bool: + """Whether any operand of ``key`` is materialised OUTSIDE the base CTE. + + DEV-1733: a composite whose operands include a CROSS-MODEL aggregate + (``_cm_`` CTE) or a WINDOWED aggregate (``_wm_`` CTE) cannot render in + ``_base`` — the operand column is not in that scope. Such composites + are owned by the combined SELECT instead, which resolves each operand + to its CTE-qualified column. + """ + from slayer.core.keys import AggregateKey as _AggKey + from slayer.engine.binding import walk_value_keys + + remote_slot_ids = { + p.aggregate_slot_id + for p in planned_query.cross_model_aggregate_plans + } | { + p.aggregate_slot_id + for p in planned_query.windowed_aggregate_plans + } + for node in walk_value_keys(key): + if not isinstance(node, _AggKey): + continue + if getattr(node.source, "path", ()): + return True # cross-model source, even without a plan yet + sid = slot_id_by_key.get(node) + if sid is not None and sid in remote_slot_ids: + return True + return False + @staticmethod def _collect_base_aux_slot_ids( # NOSONAR(S3776) — recursive ValueKey walker (nested ``_collect_from``) over the closed key union plus three top-level passes (transform layers / phase-gated filter deps / order deps). Each pass is one decision; extracting them would scatter the slot-dep contract. *, @@ -3599,6 +4042,26 @@ def _collect_from(key) -> None: if slot is None: continue _collect_from(slot.key) + # DEV-1733: an order-only COMPOSITE (``a:sum / b:sum``, + # ``abs(a:sum)``) needs its OWN materialised column, not just + # its operands — the outer trim wrap orders on a plain quoted + # alias, so the composite has to exist as a column of the inner + # SELECT. ``_collect_from`` deliberately recurses past + # composite nodes (the generator inlines them elsewhere), so + # the slot id is added here explicitly. + # + # Cross-model / windowed composites are EXCLUDED: their + # operands live in ``_cm_`` / ``_wm_`` CTEs, so the composite + # is owned by the combined SELECT and rendering it in ``_base`` + # would reference an out-of-scope column. The cross-model + # renderer routes them via ``outer_composite_slot_ids``. + if isinstance(slot.key, (ArithmeticKey, ScalarCallKey)): + if not SQLGenerator._composite_has_remote_operand( + key=slot.key, slots_by_id=slots_by_id, + slot_id_by_key=slot_id_by_key, + planned_query=planned_query, + ): + out.add(oe.slot_id) return out @@ -3675,6 +4138,185 @@ def _ready(key) -> bool: return False return True + def _resolve_agg_inputs_via_scope( # NOSONAR(S3776) — one cohesive Law-1 discovery pass: three ordered sub-passes (Column.filter → source → kwargs) over the local aggregates via small closures sharing scope/resolved. Extracting them would scatter the ordered-registration contract that keeps the base FROM byte-identical. + self, *, base_render_order, slots_by_id, scope: ScopeFrame, + ) -> "Dict[Any, Dict[str, ResolvedAggKwarg]]": + """Resolve every LOCAL aggregate's join-crossing inputs through the host + ``scope`` (Law 1) — ``scope.resolve`` anchors each ref and registers the + joins it crosses into ``scope.join_paths``, the side effect that + base-pulls the crossed LEFT JOIN. + + Three ordered sub-passes over ``base_render_order`` preserve the pre- + resolver join-registration order (Column.filter → source → kwargs): + + 1. **``Column.filter`` predicates** (DEV-1494; replaces + ``_collect_column_filter_join_paths``). The Mode-A predicate is + dual-scanned via ``_filter_join_paths`` (raw + inline-expanded, so a + placeholder dotted ref that inlines to a constant still pulls its + join) and the paths registered into the scope. + 2. **derived aggregate SOURCES** (``ColumnSqlKey`` whose ``Column.sql`` + crosses a join — DEV-1502; replaces ``_collect_aggregate_source_ + join_paths``). Discovery only; the render spec re-expands the source. + 3. **column-ref KWARGS** (``weight=`` / ``other=`` — DEV-1527). + The resolved expression is returned, keyed by ``AggregateKey`` + (frozen/hashable) → ``{kwarg_name: ResolvedAggKwarg(kind="expr")}``, + for ``_build_agg_render_spec_from_planned`` to embed. Scalar + kwargs are left out (the spec builder canonical-stringifies them). + 3b. **template-fragment KWARGS** (DEV-1709): user-supplied string + kwargs and non-overridden model-default ``AggregationParam.sql`` + fragments are scanned for crossed paths (register-only) — the + fragment text substitutes verbatim into the aggregation template, + so its joins must be in the FROM. + 4. **first/last explicit TIME ARGS** (``amount:last(customers.signup_at)`` + — DEV-1710). Discovery only; the ranked subquery's ORDER BY re-renders + the arg via ``_resolve_explicit_time_col``. Replaces the legacy + ``_collect_joined_paths_for_base`` AGGREGATE arm. A path-bearing + derived (``ColumnSqlKey``) arg — the DEV-1526 residual — is skipped. + + Cross-model aggregates (non-empty ``source.path``) are skipped in every + sub-pass: their inputs are owned by the per-plan ``_cm_*`` CTE + (Stage 4 / DEV-1708). Recurses into composite AGGREGATE keys. + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + ColumnKey, + ColumnSqlKey, + Phase, + ScalarCallKey, + ) + + resolved: "Dict[Any, Dict[str, ResolvedAggKwarg]]" = {} + + def _walk(key, fn) -> None: + if isinstance(key, AggregateKey): + if not getattr(key.source, "path", ()): + fn(key) + elif isinstance(key, ArithmeticKey): + for o in key.operands: + _walk(o, fn) + elif isinstance(key, ScalarCallKey): + for a in key.args: + _walk(a, fn) + + def _for_each_local_agg(fn) -> None: + for sid in base_render_order: + slot = slots_by_id.get(sid) + if slot is not None and slot.phase == Phase.AGGREGATE: + _walk(slot.key, fn) + + def _resolve_column_filter(key) -> None: + cfk = key.column_filter_key + if cfk is None or not cfk.canonical_sql: + return + for p in self._filter_join_paths( + sql=cfk.canonical_sql, source_relation=scope.root_relation, + source_model=scope.root_model, bundle=scope.bundle, + ): + scope.join_paths.add(p) + + def _resolve_source(key) -> None: + if isinstance(key.source, ColumnSqlKey): + scope.resolve(key.source) # register-only; render re-expands + + def _resolve_kwargs(key) -> None: + kw: Dict[str, ResolvedAggKwarg] = {} + for kname, kval in key.kwargs: + if isinstance(kval, (ColumnKey, ColumnSqlKey)): + kw[kname] = ResolvedAggKwarg(kind="expr", value=scope.resolve(kval)) + if kw: + resolved[key] = kw + + def _resolve_fragment_kwargs(key) -> None: + # DEV-1709 (PR #271 Codex review): template-fragment kwargs — + # user-supplied str values and non-overridden model-default + # ``AggregationParam.sql`` fragments — are substituted into the + # aggregation template as qualified SQL text, so their crossed + # joins must register exactly like ``Column.filter`` predicates + # do (the widened Law-3 trigger isolates on them, and the CTE + # sub-render lands here). Scanned with the same + # ``_filter_join_paths`` pipeline; unparseable fragments + # contribute nothing. + fragments = [v for _, v in key.kwargs if isinstance(v, str)] + agg_def = next( + (a for a in (scope.root_model.aggregations or []) + if a.name == key.agg), + None, + ) + if agg_def is not None: + overridden = {name for name, _ in key.kwargs} + fragments.extend( + p.sql for p in (agg_def.params or []) + if p.name not in overridden and p.sql + ) + for frag in fragments: + for p in self._filter_join_paths( + sql=frag, source_relation=scope.root_relation, + source_model=scope.root_model, bundle=scope.bundle, + ): + scope.join_paths.add(p) + + def _resolve_first_last_time_arg(key) -> None: + # DEV-1710 Stage 6 — a first/last explicit ranking-time arg + # (``amount:last(customers.signup_at)``) crosses a join exactly like + # a source / kwarg does; resolving it through the scope registers + # that join (Law 1), so the ranked subquery's ORDER BY ref is in the + # base FROM. Replaces the legacy ``_collect_joined_paths_for_base`` + # AGGREGATE arm. Register-only: the render spec re-resolves via + # ``_resolve_explicit_time_col``. + arg = self._explicit_time_arg_of(key) + if arg is None: + return + # A path-bearing derived (ColumnSqlKey) arg is a hop PAST the target + # (the DEV-1526 residual the render seam raises on) — skip it here; + # anchoring against ``source_relation`` would register a bogus join. + if isinstance(arg, ColumnSqlKey) and arg.path: + return + scope.resolve(arg) + + _for_each_local_agg(_resolve_column_filter) + _for_each_local_agg(_resolve_source) + _for_each_local_agg(_resolve_kwargs) + _for_each_local_agg(_resolve_fragment_kwargs) + _for_each_local_agg(_resolve_first_last_time_arg) + return resolved + + def _resolve_agg_kwargs_for_key( + self, *, key, source_model, source_relation: str, bundle, + ) -> "Optional[Dict[str, ResolvedAggKwarg]]": + """Resolve a single LOCAL aggregate's column-ref kwargs + (``weighted_avg(weight=)`` / ``corr(other=)``) through a fresh + host ``ScopeFrame`` → ``{name: ResolvedAggKwarg(kind="expr")}`` or ``None``. + + The base SELECT uses the batch ``_resolve_agg_inputs_via_scope`` pass over + a shared host scope (which also registers the crossed joins). The HAVING + render path (``_render_value_key_for_filter``) has no such scope, so it + builds a throwaway one here purely to reproduce the SAME anchored kwarg + expression the SELECT emits — the crossed join is already base-pulled + (the HAVING aggregate is also a ``base_render_order`` slot), so the + throwaway scope's own ``join_paths`` are intentionally discarded. + """ + from slayer.core.keys import ColumnKey, ColumnSqlKey + + kwargs = getattr(key, "kwargs", None) + if bundle is None or not kwargs: + return None + allocator = self._new_allocator() + scope = ScopeFrame( + scope_id=allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + resolved = { + kname: ResolvedAggKwarg(kind="expr", value=scope.resolve(kval)) + for kname, kval in kwargs + if isinstance(kval, (ColumnKey, ColumnSqlKey)) + } + return resolved or None + def _build_base_select_for_planned( # NOSONAR(S3776) — join-path collection and derived-dim expansion are extracted to helpers; the residual is the one cohesive per-slot ROW/AGGREGATE projection + GROUP-BY assembly pass. self, *, @@ -3707,67 +4349,80 @@ def _build_base_select_for_planned( # NOSONAR(S3776) — join-path collection a from slayer.core.enums import TimeGranularity from slayer.core.keys import ( AggregateKey, + ArithmeticKey, ColumnKey, ColumnSqlKey, Phase, + ScalarCallKey, TimeTruncKey, ) - # Walk row slots to collect every joined path so the FROM - # clause carries the needed LEFT JOINs in one pass. + # DEV-1706 Stage 2: the host base is a single scope; every join-crossing + # ref registers its path into ``host_scope.join_paths`` as a side effect + # of being resolved through the scope (Law 1 — discovery can never be + # forgotten). The legacy join collectors are gone: their work is now the + # scope passes below. The scope's ordered ``join_paths`` reproduce the + # collectors' first-seen registration order — derived dims → WHERE filters + # → Column.filter → source → kwargs → first/last time args — so the base + # FROM is byte-identical. + # + # Stage 2's host base has no projection boundary, so the allocator mints + # no ``_val_`` names here and a local instance suffices; the generation- + # wide allocator (D-E) arrives with the CTE scopes in Stage 4. + # + # Walk row slots for every joined DIMENSION path first (join-order + # position 1); the scope's paths (derived dims, filters, aggregate + # inputs, and — DEV-1710 Stage 6 — first/last time args) append after it. needed_join_paths = self._collect_joined_paths_for_base( base_render_order=base_render_order, slots_by_id=slots_by_id, - source_model=source_model, - source_relation=source_relation, + ) + # DEV-1708 (D-E): share the generation-wide allocator so host-base and + # per-plan ``_cm_*`` CTE ``_val_`` names are globally unique. + host_allocator = self._gen_allocator or self._new_allocator() + host_scope = ScopeFrame( + scope_id=host_allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, bundle=bundle, + dialect=self._dialect, + allocator=host_allocator, ) # Pre-expand derived (ColumnSqlKey) ROW + TIME dimensions: inline - # sibling/joined derived refs (DEV-1333 / DEV-1410) and pull any joins - # their SQL crosses into the FROM (appended to ``needed_join_paths``). + # sibling/joined derived refs (DEV-1333 / DEV-1410) and register any + # joins their SQL crosses into the scope (position 2). Returns the + # expanded-expr-by-slot-id map the render branch reads. derived_expr_by_sid = self._expand_derived_row_dims( base_render_order=base_render_order, slots_by_id=slots_by_id, source_relation=source_relation, source_model=source_model, - bundle=bundle, needed_join_paths=needed_join_paths, + bundle=bundle, scope=host_scope, ) # WHERE-phase filters referencing joined columns (direct, derived, or - # Mode-A ``__`` paths) pull their joins into the FROM too. Filters - # routed to a cross-model ``_cm_*`` CTE (``skip_filter_ids``) are - # applied there, not on ``_base`` — pulling their join into ``_base`` - # would add an unused (and, for one-to-many joins, cardinality- - # changing) LEFT JOIN. - for p in self._collect_filter_join_paths( - planned_query=planned_query, source_model=source_model, - source_relation=source_relation, bundle=bundle, + # Mode-A ``__`` paths) register their joins into the scope too (position + # 3). Filters routed to a cross-model ``_cm_*`` CTE (``skip_filter_ids``) + # are applied there, not on ``_base`` — registering their join here would + # add an unused (and, for one-to-many joins, cardinality-changing) LEFT + # JOIN. + self._resolve_where_filter_joins_via_scope( + planned_query=planned_query, scope=host_scope, skip_filter_ids=skip_filter_ids, - ): - if p not in needed_join_paths: - needed_join_paths.append(p) - # DEV-1494: a ``Column.filter`` on an aggregated measure becomes a - # CASE-WHEN wrapper; pull any join its predicate crosses (directly, or via - # a derived ref) into the FROM — including filtered aggregates nested in - # composite (arithmetic / scalar-call) AGGREGATE-phase keys. - for p in self._collect_column_filter_join_paths( - base_render_order=base_render_order, slots_by_id=slots_by_id, - source_relation=source_relation, source_model=source_model, - bundle=bundle, - ): - if p not in needed_join_paths: - needed_join_paths.append(p) - # DEV-1502: an AGGREGATE slot whose SOURCE is a derived - # (``ColumnSqlKey``) column whose ``Column.sql`` crosses a join - # (``customers__regions.population``) needs the same discovery the - # dimension path performs (DEV-1484). Symmetric with the filter case - # above; render-time expansion in ``_build_agg_render_spec_from_planned`` - # already qualifies the body, so this pass only closes the join-discovery - # gap. Cross-model aggregate sources are skipped — they're owned by the - # per-plan ``_cm_*`` CTE (the symmetric in-CTE discovery gap is tracked - # separately). - for p in self._collect_aggregate_source_join_paths( - base_render_order=base_render_order, slots_by_id=slots_by_id, - source_relation=source_relation, source_model=source_model, - bundle=bundle, - ): + ) + # Every LOCAL aggregate's join-crossing inputs resolve through the scope + # next: ``Column.filter`` (position 4; DEV-1494), derived aggregate SOURCE + # (position 5; DEV-1502), column-ref KWARGS (position 6; DEV-1527 — + # ``weighted_avg(weight=)`` / ``corr(other=)``, whose resolved + # expression is embedded verbatim (``kind="expr"``) into the render spec, + # replacing the ``agg_kwarg_canonical_str`` round-trip that collapsed a + # derived column to a bare, non-existent name), and first/last explicit + # TIME ARGS (position 7; DEV-1710 — ``amount:last(customers.signup_at)``). + resolved_agg_kwargs = self._resolve_agg_inputs_via_scope( + base_render_order=base_render_order, + slots_by_id=slots_by_id, + scope=host_scope, + ) + # Merge the scope's registered paths (positions 2-7, in first-seen order) + # after the dimension paths (position 1) → byte-identical FROM. + for p in host_scope.join_paths.as_list(): if p not in needed_join_paths: needed_join_paths.append(p) from_clause, base_joins = self._build_from_and_joins( @@ -3871,6 +4526,23 @@ def _record_alias(sid: str, full_alias: str) -> None: select_columns.append(col_expr.copy().as_(full_alias)) group_by_keys.setdefault(sid, col_expr) _record_alias(sid, full_alias) + elif isinstance(key, (ScalarCallKey, ArithmeticKey)): + # DEV-1576 / DEV-1717: a ROW-phase composite here is a + # non-aggregating measure expression (a bare column, or + # arithmetic / scalar-call over bare columns such as + # ``round(amount, 2)`` / ``abs(amount)`` / ``amount + 1``). + # Dimensions are ColumnKey / TimeTruncKey / ColumnSqlKey, + # already handled above; the only way to reach here with a + # composite key is a measure that never aggregates. Raise + # the same actionable "Bare measure name" error the + # enrich_query path raises rather than leaking an internal + # NotImplementedError. + bare = _first_bare_column_name(key) or full_alias + raise ValueError( + f"Bare measure name '{bare}' is not valid. " + f"Use colon syntax (e.g., '{bare}:sum', '{bare}:avg'). " + f"For COUNT(*), use '*:count'." + ) else: raise NotImplementedError( f"DEV-1450 stage 7b.10+: row-phase key type " @@ -3883,13 +4555,19 @@ def _record_alias(sid: str, full_alias: str) -> None: if not isinstance(key, AggregateKey): # AGGREGATE-phase composite (arithmetic / scalar-call of # aggregates, e.g. ``expensenet:avg + benchmarkexp:avg``). - # Render inline; cast the whole composite once. + # Render inline; cast the whole composite once. DEV-1527: + # thread the host scope's resolved column-ref kwargs so a + # crossing derived kwarg inside a composite operand + # (``amount:weighted_avg(weight=) + quantity:sum``) + # embeds its expanded join-anchored expression instead of a + # bare, non-existent name. composite, any_agg = self._render_aggregate_composite_expr( key=key, slot=slot, source_model=source_model, source_relation=source_relation, bundle=bundle, + resolved_agg_kwargs=resolved_agg_kwargs, ) if any_agg: composite = _wrap_cast_for_type(composite, slot.type) @@ -3921,6 +4599,7 @@ def _record_alias(sid: str, full_alias: str) -> None: source_relation=source_relation, full_alias=full_alias, bundle=bundle, + resolved_agg_kwargs=resolved_agg_kwargs.get(key), ) agg_expr, is_agg = self._build_agg(synth) if is_agg: @@ -4042,7 +4721,29 @@ def _resolve_ranking_time_column_from_planned( return f"{source_relation}.{source_model.default_time_dimension}" return None - def _resolve_explicit_time_col( # NOSONAR(S3776) — sequential isinstance dispatch over ColumnKey (bare ref → ``__``-joined path alias) and ColumnSqlKey (derived column → bare-ident-qualify vs complex-emit-verbatim). Extracting the per-shape branches would scatter the time-arg resolution contract; each branch is one decision. + @staticmethod + def _explicit_time_arg_of(key): + """The explicit positional ranking-time arg of a ``first`` / ``last`` + aggregate, or ``None``. + + The SINGLE arg-selection contract shared by the three sites that must + never disagree on WHICH positional arg is the time column (DEV-1710 / + Codex F1): the raise-gate in ``_build_first_last_base_select``, the + join-discovery pass in ``_resolve_agg_inputs_via_scope``, and the render + seam ``_resolve_explicit_time_col``. Returns the FIRST positional arg + iff it is a ``ColumnKey`` / ``ColumnSqlKey``; ``None`` for a + non-first/last agg, empty args, or a first positional arg of any other + type (first/last never takes a leading non-column positional). + """ + from slayer.core.keys import ColumnKey, ColumnSqlKey + + if key.agg not in ("first", "last"): + return None + for a in key.args: + return a if isinstance(a, (ColumnKey, ColumnSqlKey)) else None + return None + + def _resolve_explicit_time_col( self, *, key, @@ -4055,74 +4756,83 @@ def _resolve_explicit_time_col( # NOSONAR(S3776) — sequential isinstance disp ranked subquery. Handles both bare-column refs (``ColumnKey`` — - ``amount:last(created_at)``) and derived-column refs - (``ColumnSqlKey`` — ``amount:last(net_amount_date)`` where - ``net_amount_date`` has a non-trivial ``Column.sql``). For derived - columns the column's ``Column.sql`` is materialised through - ``_expand_derived_column_sql`` (when ``bundle`` is available) so - inner bare refs qualify to ``source_relation`` and joined refs to - their ``__``-path alias — a complex expression like - ``date(created_at)`` can't go ambiguous against a same-named column - on a joined table inside the ranked subquery. Without a ``bundle`` - it falls back to bare-ident qualification / verbatim emit. - - Returns ``None`` for non-first/last aggs and when ``key.args`` is - empty or its first element is neither a ``ColumnKey`` nor a - ``ColumnSqlKey``. Cross-model paths on derived time args - (``ColumnSqlKey`` with non-empty ``path``) raise - ``NotImplementedError`` rather than silently emitting against the - wrong relation alias — that case is tracked alongside bug (c) of - the four-bug Stage B package in DEV-1476. + ``amount:last(created_at)``) and derived-column refs (``ColumnSqlKey`` + — ``amount:last(net_amount_date)`` where ``net_amount_date`` has a + non-trivial ``Column.sql``). DEV-1710 Stage 6: when a ``bundle`` is + available the arg is anchored through a ``ScopeFrame`` (Law 1) — the + same resolver the host base / kwargs passes use — so a bare joined ref + qualifies to its ``__``-path alias, a derived expression's inner bare + refs qualify to ``source_relation`` (never ambiguous against a + same-named joined column), and reserved-word relations are quoted + (DEV-1686). Without a ``bundle`` (the render-spec unit path) it falls + back to bare-ident qualification / verbatim emit. + + Returns ``None`` for non-first/last aggs and when ``key.args`` is empty + or its first element is neither a ``ColumnKey`` nor a ``ColumnSqlKey`` + (see ``_explicit_time_arg_of``). A derived time arg (``ColumnSqlKey``) + whose ``path`` is non-empty AFTER the DEV-1707 cross-model reroot — a + column a hop PAST the target — raises ``NotImplementedError`` rather + than silently emitting against a relation the isolated CTE does not + join; that residual-hop case is tracked as DEV-1526 (Stage 4). The + analogous residual ``ColumnKey`` arg is caught loudly by the + scope-closure validator (``SLAYER_VALIDATE_SCOPES``) instead. """ from slayer.core.keys import ColumnKey, ColumnSqlKey - if key.agg not in ("first", "last"): + arg = self._explicit_time_arg_of(key) + if arg is None: return None - for a in key.args: - if isinstance(a, ColumnKey): - relation = "__".join(a.path) if a.path else source_relation - return f"{relation}.{a.leaf}" - if isinstance(a, ColumnSqlKey): - if a.path: - raise NotImplementedError( - f"Cross-model derived time column " - f"(path={a.path!r}, column={a.column_name!r}) on " - f"first/last positional arg is not yet supported " - f"by the ranked-subquery builder; tracked as " - f"DEV-1476." - ) - col = next( - (c for c in source_model.columns if c.name == a.column_name), - None, + if isinstance(arg, ColumnSqlKey) and arg.path: + raise NotImplementedError( + f"Derived time column with a residual join path " + f"(path={arg.path!r}, column={arg.column_name!r}) on a " + f"first/last positional arg is not yet supported by " + f"the ranked-subquery builder: the isolated CTE does " + f"not pull the residual join. Post-DEV-1707 the " + f"cross-model reroot strips the target prefix, so this " + f"fires only for a time arg a hop PAST the target; " + f"tracked as DEV-1526 (Stage 4)." + ) + # Validate a derived arg's existence up front so the not-found case is a + # clear error rather than the resolver silently anchoring the bare name. + col = None + if isinstance(arg, ColumnSqlKey): + col = next( + (c for c in source_model.columns if c.name == arg.column_name), + None, + ) + if col is None: + raise ValueError( + f"Derived time column {arg.column_name!r} (positional " + f"arg of {key.agg!r}) not found on model " + f"{source_model.name!r}." ) - if col is None: - raise ValueError( - f"Derived time column {a.column_name!r} (positional " - f"arg of {key.agg!r}) not found on model " - f"{source_model.name!r}." - ) - if bundle is not None: - # Qualify inner bare refs against ``source_relation`` (and - # joined refs to their ``__``-path alias) so a complex - # derived time expression can't bind to the wrong table - # inside the ranked subquery's joins — same expansion the - # aggregate-source path uses. - return self._expand_derived_column_sql( - source_model=source_model, - source_relation=source_relation, - column_name=a.column_name, - bundle=bundle, - ) - # No bundle (defensive): bare-ident qualify, else emit verbatim. - col_sql = col.sql if col.sql else col.name - if col_sql.isidentifier(): - return f"{source_relation}.{col_sql}" - return self._parse(col_sql).sql(dialect=self.dialect) - # Unrecognised positional arg type — leave time_column unset and - # let _build_ranked_subquery_from_planned fall back to the - # query's default ranking column. - break - return None + if bundle is not None: + # Law 1 — anchor the arg through a throwaway host-rooted scope. Its + # ``join_paths`` are discarded (discovery is owned by the base + # aggregate-input pass, which registers the same join); this call is + # purely to reproduce the SAME anchored SQL the ORDER BY needs. Same + # throwaway-frame pattern as ``_resolve_agg_kwargs_for_key``. + allocator = self._new_allocator() + scope = ScopeFrame( + scope_id=allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + return scope.resolve(arg).sql(dialect=self.dialect) + # No bundle (defensive; the render-spec unit path): bare ColumnKey + # qualifies to its ``__``-path alias / source relation, a derived + # bare-ident qualifies to the source relation, else emit verbatim. + if isinstance(arg, ColumnKey): + relation = "__".join(arg.path) if arg.path else source_relation + return f"{relation}.{arg.leaf}" + col_sql = col.sql if col.sql else col.name + if col_sql.isidentifier(): + return f"{source_relation}.{col_sql}" + return self._parse(col_sql).sql(dialect=self.dialect) def _build_ranked_subquery_from_planned( # NOSONAR(S3776) — Group 2 already factored the per-spec ROW_NUMBER passes into _build_unfiltered_rn_columns / _build_filtered_rn_columns; what's left is exp.Select / from / joins / where assembly that has to live in one place. self, @@ -4361,13 +5071,11 @@ def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: else: fl_keys = _iter_first_last_leaves(key) for fl in fl_keys: - # An explicit time arg is the first ColumnKey / - # ColumnSqlKey in ``key.args``. - has_explicit = any( - isinstance(a, (ColumnKey, ColumnSqlKey)) - for a in fl.args - ) - if not has_explicit: + # Whether this leaf carries an explicit ranking-time arg is + # the shared ``_explicit_time_arg_of`` contract — the SAME + # selection the render seam uses, so the gate and the render + # can never disagree (DEV-1710 / Codex F1). + if self._explicit_time_arg_of(fl) is None: needs_default = True break if needs_default: @@ -4476,6 +5184,13 @@ def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: # ``base_render_order``; the aggregate value is identical # across visits so synthesise once per sid, keyed by the # first alias. + # + # DEV-1709 (closes the DEV-1527/DEV-1476 first/last kwarg + # deferral): column-ref kwargs are resolved here so the + # spec embeds the expanded, join-anchored expression; a + # CROSSING kwarg expression is then Law-2-materialised in + # the pass below (the outer aggregate body may only + # reference the ranked subquery's projections). if sid not in synth_by_sid: synth_by_sid[sid] = ( self._build_agg_render_spec_from_planned( @@ -4483,6 +5198,10 @@ def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: source_relation=source_relation, full_alias=full_alias, bundle=bundle, + resolved_agg_kwargs=self._resolve_agg_kwargs_for_key( + key=slot.key, source_model=source_model, + source_relation=source_relation, bundle=bundle, + ), ) ) @@ -4498,6 +5217,7 @@ def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: # inlined inside the composite render). Keyed by the AggregateKey # itself so two composites sharing the same operand dedupe. composite_synth_by_key: Dict[Any, "EnrichedMeasure"] = {} + composite_resolved_kwargs: Dict[Any, Dict[str, ResolvedAggKwarg]] = {} for sid in base_render_order: slot = slots_by_id[sid] if slot.phase != Phase.AGGREGATE: @@ -4524,6 +5244,107 @@ def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: bundle=bundle, ) ) + leaf_kw = self._resolve_agg_kwargs_for_key( + key=agg_leaf, source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + if leaf_kw: + composite_resolved_kwargs[agg_leaf] = leaf_kw + + # DEV-1709 / DEV-1531 — Law 2 in the ranked subquery. The subquery + # re-exports only ``source_relation.*`` + rank / ``_td`` / ``_dim`` + # columns, so ANY crossing expression the OUTER SELECT consumes — + # an aggregate SOURCE sql or a column-ref KWARG value — must be + # materialised as a ``_val_`` projection inside the subquery and + # the outer consumer rewritten to the bare alias. Applies to + # first/last AND regular aggregates alike (DEV-1702-B1). After the + # widened Law-3 trigger, crossing inputs reach this path only + # inside a host-rooted isolation CTE's sub-render (where inline + # joins are legal but the ranked-scope boundary still applies) or + # under ``disable_host_rooted_isolation``. + # + # The materialised projection is the RESOLVED value — qualified, + # with the ``Column.type`` inner CAST for non-bare expressions + # (``_resolve_value_sql``'s rule) — so the outer aggregate consumes + # exactly the value the inline path would have aggregated + # (``SUM(CAST(x * 2 AS INT))`` semantics preserved). Dedupe is by + # that resolved text: same sql + different type differ by the CAST + # and never collapse onto one ``_val``; bare refs are type-agnostic + # (no CAST) and sharing IS correct. + allocator = self._gen_allocator or self._new_allocator() + self._reserve_model_column_names(allocator, source_model) + value_alias_by_sql: Dict[str, str] = {} + + def _materialize_if_crossing( + spec: "AggRenderSpec", + ) -> Optional[str]: + if not spec.sql: + return None + value_expr = self._resolve_sql( + sql=spec.sql, name=spec.name, + model_name=spec.model_name, type=spec.column_type, + ) + resolved_key = value_expr.sql(dialect=self.dialect) + if resolved_key in value_alias_by_sql: + return value_alias_by_sql[resolved_key] + if not self._joined_paths_in_sql( + sql_expr=value_expr, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ): + return None + val_alias = allocator.allocate_val() + extra_projections.append((val_alias, value_expr)) + value_alias_by_sql[resolved_key] = val_alias + return val_alias + + def _materialize_spec_kwargs( + kw: "Optional[Dict[str, ResolvedAggKwarg]]", + ) -> "Optional[Dict[str, ResolvedAggKwarg]]": + """Rewrite crossing ``kind="expr"`` kwarg values to their + materialised ``_val`` aliases; local values pass through.""" + if not kw: + return kw + new_kw: Dict[str, ResolvedAggKwarg] = {} + for name, rk in kw.items(): + if rk.kind == "expr" and self._joined_paths_in_sql( + sql_expr=rk.value, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ): + kw_sql = rk.value.sql(dialect=self.dialect) + val_alias = value_alias_by_sql.get(kw_sql) + if val_alias is None: + val_alias = allocator.allocate_val() + extra_projections.append((val_alias, rk.value)) + value_alias_by_sql[kw_sql] = val_alias + new_kw[name] = ResolvedAggKwarg( + kind="expr", + value=exp.column(val_alias, table=source_relation), + ) + else: + new_kw[name] = rk + return new_kw + + outer_synth_by_sid: Dict[str, "EnrichedMeasure"] = {} + for sid, spec in synth_by_sid.items(): + updates: Dict[str, Any] = {} + src_alias = _materialize_if_crossing(spec) + if src_alias is not None: + updates["sql"] = src_alias + new_kw = _materialize_spec_kwargs(spec.agg_kwargs) + if new_kw is not spec.agg_kwargs: + updates["agg_kwargs"] = new_kw + outer_synth_by_sid[sid] = ( + spec.model_copy(update=updates) if updates else spec + ) + for leaf_key, spec in composite_synth_by_key.items(): + # Composite leaves re-synthesise inside the pass-2 composite + # render; registering their crossing sql here lets that render + # swap in the alias via ``value_alias_by_sql``. + _materialize_if_crossing(spec) + leaf_kw = composite_resolved_kwargs.get(leaf_key) + new_leaf_kw = _materialize_spec_kwargs(leaf_kw) + if new_leaf_kw is not leaf_kw and new_leaf_kw is not None: + composite_resolved_kwargs[leaf_key] = new_leaf_kw # WHERE goes inside the ranked subquery (raw-row filtering before # ranking). HAVING is recomputed and applied by the caller. @@ -4591,8 +5412,12 @@ def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: aliases_by_slot_id.setdefault(sid, []).append(full_alias) elif slot.phase == Phase.AGGREGATE: if sid in synth_by_sid: + # DEV-1709: the OUTER aggregate consumes the Law-2 + # rewritten spec (crossing source / kwarg expressions + # swapped for their ``_val`` aliases); the ranked + # subquery consumed the originals. agg_expr, is_agg = self._build_agg( - synth_by_sid[sid], + outer_synth_by_sid[sid], rn_suffix_map=rn_suffix_map, default_time_col=default_time_col_sql, filtered_rn_map=filtered_rn_map, @@ -4614,6 +5439,12 @@ def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: agg_key: spec.alias for agg_key, spec in composite_synth_by_key.items() } + # DEV-1709 (closes the DEV-1527/DEV-1476 first/last kwarg + # deferral): thread the per-leaf resolved kwargs (crossing + # values already Law-2-rewritten to their ``_val`` aliases) + # and the source-value alias map so composite leaves bind + # to the ranked subquery's projections instead of leaking + # crossing refs into the outer scope. agg_expr, is_agg = self._render_aggregate_composite_expr( key=slot.key, slot=slot, source_model=source_model, source_relation=source_relation, @@ -4623,6 +5454,8 @@ def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: filtered_rn_map=filtered_rn_map, filtered_match_map=filtered_match_map, composite_alias_by_key=composite_alias_by_key, + resolved_agg_kwargs=composite_resolved_kwargs, + value_alias_by_sql=value_alias_by_sql, ) if is_agg: agg_expr = _wrap_cast_for_type(agg_expr, slot.type) @@ -4643,6 +5476,9 @@ def _build_first_last_base_select( # NOSONAR(S3776) — single conceptual unit: default_time_col_sql=default_time_col_sql, filtered_rn_map=dict(filtered_rn_map), filtered_match_map=dict(filtered_match_map), + # DEV-1709: HAVING re-synths of a crossing-source aggregate bind + # to the materialised alias instead of the raw crossing ref. + value_alias_by_sql=dict(value_alias_by_sql), ) return ( base_select, aliases_by_slot_id, has_aggregation, @@ -4662,6 +5498,8 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc filtered_rn_map: Optional[Dict[str, str]] = None, filtered_match_map: Optional[Dict[str, str]] = None, composite_alias_by_key: Optional[Dict[Any, str]] = None, + resolved_agg_kwargs: "Optional[Dict[Any, Dict[str, ResolvedAggKwarg]]]" = None, + value_alias_by_sql: Optional[Dict[str, str]] = None, ) -> "tuple[exp.Expression, bool]": """Render an AGGREGATE-phase composite key (``ArithmeticKey`` / ``ScalarCallKey`` of aggregates, e.g. ``expensenet:avg + @@ -4673,6 +5511,14 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc contains_aggregate)``. Cross-model operand aggregates (non-empty ``source.path``) are not yet handled here — they need CTE routing. + DEV-1527 (composite local half): ``resolved_agg_kwargs`` is the host + scope's per-``AggregateKey`` column-ref kwarg map (``weight=`` / + ``other=``); each operand leaf looks up its own entry so a + crossing derived kwarg embeds its expanded, join-anchored expression + instead of collapsing to a bare (non-existent) name. Threaded only from + the non-first/last base path; the first/last ranked-subquery caller passes + ``None`` (that path's derived-kwarg support is deferred — see DEV-1476). + DEV-1501 (Codex round 3): when the host base is built via the first/last ranked-subquery path, the caller threads the rn maps here so a composite expression like ``last(amount, created_at) + @@ -4713,7 +5559,21 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc slot=slot, key=key, source_model=source_model, source_relation=source_relation, full_alias=op_alias, bundle=bundle, - ) + resolved_agg_kwargs=(resolved_agg_kwargs or {}).get(key), + ) + # DEV-1709 Law 2: inside the first/last ranked path, a crossing + # composite-leaf SOURCE was materialised as a ``_val_`` + # projection in the ranked subquery — rebind the re-synthesised + # leaf to that alias (keyed by the RESOLVED value text, so + # same-sql-different-type leaves bind to their own casts) so + # the outer composite never references the crossing expression + # out of scope. + if value_alias_by_sql and synth.sql is not None: + resolved_key = self._resolve_value_sql(synth) + if resolved_key in value_alias_by_sql: + synth = synth.model_copy( + update={"sql": value_alias_by_sql[resolved_key]}, + ) agg_expr, is_agg = self._build_agg( synth, rn_suffix_map=rn_suffix_map, @@ -4735,6 +5595,8 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc filtered_rn_map=filtered_rn_map, filtered_match_map=filtered_match_map, composite_alias_by_key=composite_alias_by_key, + resolved_agg_kwargs=resolved_agg_kwargs, + value_alias_by_sql=value_alias_by_sql, ) operands.append(e) any_agg = any_agg or a @@ -4743,7 +5605,16 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc args = [] any_agg = False for a in key.args: - if isinstance(a, (AggregateKey, ArithmeticKey, ScalarCallKey, LiteralKey)): + # DEV-1733: dispatch on the KEY BASE, not a hand-listed subset. + # The trailing ``else`` below stringifies whatever it does not + # recognise, so a row-column argument + # (``coalesce(revenue:sum, quantity)``) used to render as the + # SQL string literal ``'path=() leaf=''quantity'''`` — valid + # SQL, silently wrong results. Routing every ValueKey through + # the recursive renderer makes an unsupported operand hit the + # same terminal NotImplementedError the arithmetic path raises, + # and leaves the ``else`` for genuine Python literals only. + if isinstance(a, _FrozenKey): e, ag = self._render_aggregate_composite_expr( key=a, slot=slot, source_model=source_model, source_relation=source_relation, @@ -4753,6 +5624,8 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc filtered_rn_map=filtered_rn_map, filtered_match_map=filtered_match_map, composite_alias_by_key=composite_alias_by_key, + resolved_agg_kwargs=resolved_agg_kwargs, + value_alias_by_sql=value_alias_by_sql, ) args.append(e) any_agg = any_agg or ag @@ -4766,7 +5639,9 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc args.append(exp.Literal.string(str(a))) if key.name == "like": return exp.Like(this=args[0], expression=args[1]), any_agg - return exp.func(key.name.upper(), *args), any_agg + return self._finalize_scalar_call( + exp.func(key.name.upper(), *args) + ), any_agg if isinstance(key, LiteralKey): v = key.value if v is None: @@ -4781,6 +5656,197 @@ def _render_aggregate_composite_expr( # NOSONAR(S3776) — sequential isinstanc f"{type(key).__name__} not supported." ) + def _render_window_measure_cte_from_planned( # NOSONAR(S3776) — one cohesive host-rooted range-join CTE build: ``_src`` projection (dims / other-time-dims / raw-window-time / value) with Law-1 join discovery, WHERE inheritance minus date_range, and the ``_base LEFT JOIN _src`` interval range join. Splitting scatters the shared scope / grain-alias / join-eq state. + self, + *, + plan, + agg_slot, + source_model, + source_relation: str, + bundle, + planned_query, + slots_by_id: Dict[str, Any], + aliases_by_slot_id: Dict[str, List[str]], + full_agg_alias: str, + ) -> Tuple[str, List[str]]: + """Render one ``_wm_`` duration-windowed-measure CTE (DEV-1714 Stage 10). + + The CTE is host-rooted: ``FROM _base LEFT JOIN (<_src>) AS _src`` where + ``_src`` self-selects the host rows (dims → ``_w_dim_``, other time + dims date-trunc'd → ``_w_td_``, the raw window time column → + ``_w_time``, the value → ``_w_value``), and the join predicate pairs the + grain equalities with the trailing ``INTERVAL`` range + (``_src._w_time >= bucket_end - window`` / ``< bucket_end``). The result + is grouped at the host grain and LEFT-JOINed back to ``_base`` by the + caller. Returns ``(cte_sql, grain_aliases)``. + """ + from slayer.core.keys import AggregateKey + + key = agg_slot.key + assert isinstance(key, AggregateKey) + + allocator = self._gen_allocator or self._new_allocator() + src_scope = ScopeFrame( + scope_id=allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + + def _base_col(alias: str) -> exp.Column: + return exp.Column( + this=exp.to_identifier(alias, quoted=True), + table=exp.to_identifier("_base"), + ) + + def _src_col(name: str) -> exp.Column: + return exp.Column( + this=exp.to_identifier(name), table=exp.to_identifier("_src"), + ) + + def _alias_of(sid: str) -> str: + al = aliases_by_slot_id.get(sid) or [] + return al[0] if al else sid + + src_cols: List[exp.Expression] = [] + join_eqs: List[exp.Expression] = [] + grain_aliases: List[str] = [] + + # Query dimensions → ``_w_dim_`` (Law-1 resolve registers crossed + # joins into ``src_scope``). + for idx, sid in enumerate(plan.dimension_slot_ids): + dslot = slots_by_id.get(sid) + base_alias = _alias_of(sid) + expr = src_scope.resolve(dslot.key) + src_cols.append(expr.as_(f"_w_dim_{idx}")) + join_eqs.append(exp.EQ( + this=_src_col(f"_w_dim_{idx}"), expression=_base_col(base_alias), + )) + grain_aliases.append(base_alias) + + # Non-window time dimensions → ``_w_td_`` (date-trunc'd), equality- + # joined so the trailing window does not fan out across their values. + for idx, sid in enumerate(plan.other_time_dimension_slot_ids): + tslot = slots_by_id.get(sid) + base_alias = _alias_of(sid) + # Codex#1: register any join the time column crosses into the scope + # (a joined time dimension would otherwise reference an unbound alias + # in _src); the expression itself comes from the is_root/derived-aware + # helper below. + src_scope.resolve(tslot.key.column) + raw = self._raw_time_col_expr_for_planned( + time_column=tslot.key.column, source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + trunc = self._build_date_trunc( + col_expr=raw, granularity=TimeGranularity(tslot.key.granularity), + ) + src_cols.append(trunc.as_(f"_w_td_{idx}")) + join_eqs.append(exp.EQ( + this=_src_col(f"_w_td_{idx}"), expression=_base_col(base_alias), + )) + grain_aliases.append(base_alias) + + # The window time dimension's RAW column → ``_w_time`` (the range axis). + wtd_slot = slots_by_id.get(plan.window_time_dimension_slot_id) + wtd_alias = _alias_of(plan.window_time_dimension_slot_id) + # Codex#1: register the window time column's crossed join (if any) too. + src_scope.resolve(wtd_slot.key.column) + raw_time = self._raw_time_col_expr_for_planned( + time_column=wtd_slot.key.column, source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + src_cols.append(raw_time.copy().as_("_w_time")) + grain_aliases.append(wtd_alias) + + # The measure value → ``_w_value`` (CASE-wrapped by ``Column.filter``). + val_expr = src_scope.resolve(key.source) + if key.column_filter_key is not None: + pred_sql = src_scope.resolve_predicate_sql( + key.column_filter_key.canonical_sql, + ) + val_expr = exp.Case( + ifs=[exp.If(this=self._parse_predicate(pred_sql), true=val_expr)], + ) + src_cols.append(val_expr.as_("_w_value")) + + # WHERE-phase row filters (model + user) inherited into ``_src``, minus + # their frame bounds (``plan.where_filter_ids`` / + # ``plan.src_filter_rewrites``, DEV-1714 + DEV-1732). ONE effective list + # feeds both join discovery (Law 1) and rendering, so the two can never + # disagree about what this CTE contains. + all_filter_ids = {fp.id for fp in planned_query.filters_by_phase} + skip_for_src = all_filter_ids - set(plan.where_filter_ids) + src_filters = _effective_src_filters(planned_query=planned_query, plan=plan) + self._resolve_where_filter_joins_via_scope( + planned_query=planned_query, scope=src_scope, + skip_filter_ids=skip_for_src, filters_override=src_filters, + ) + src_where, _src_having = self._build_where_having_from_planned( + planned_query=planned_query, source_relation=source_relation, + source_model=source_model, bundle=bundle, + skip_filter_ids=skip_for_src, filters_override=src_filters, + ) + + # ``_src`` FROM + joins from the scope's discovered paths. + from_expr, src_joins = self._build_from_and_joins( + source_model=source_model, source_relation=source_relation, + joined_paths=src_scope.join_paths.as_list(), bundle=bundle, + ) + src_select = exp.Select().select(*src_cols).from_(from_expr) + for join_expr, on_expr, join_type in src_joins: + src_select = src_select.join( + join_expr, on=on_expr, join_type=join_type, + ) + if src_where is not None: + src_select = src_select.where(src_where) + src_subq = exp.Subquery( + this=src_select, alias=exp.TableAlias(this=exp.to_identifier("_src")), + ) + + # Trailing-window range predicate: ``_src._w_time`` in + # ``[bucket_end - window, bucket_end)`` where ``bucket_end`` is the + # host bucket's exclusive upper edge (grain + 1 grain). + frame_time = _base_col(wtd_alias) + bucket_end = self._add_intervals_expr( + frame_time, + self._granularity_interval_expr( + TimeGranularity(plan.window_granularity), sign=1, + ), + sign=1, + ) + lower_bound = self._add_intervals_expr( + bucket_end, + self._dialect.duration_interval_exprs( + parts=[tuple(p) for p in plan.window_parts], sign=-1, + ), + sign=-1, + ) + src_w_time = _src_col("_w_time") + on_range = exp.and_( + *join_eqs, + exp.GTE(this=src_w_time, expression=lower_bound), + exp.LT(this=src_w_time.copy(), expression=bucket_end.copy()), + ) + + agg_cls = exp.Sum if plan.agg == "sum" else exp.Avg + agg_expr = _wrap_cast_for_type( + agg_cls(this=_src_col("_w_value")), agg_slot.type, + ) + + outer = exp.Select() + for ga in grain_aliases: + outer = outer.select(_base_col(ga)) + outer = outer.select(agg_expr.as_(exp.to_identifier(full_agg_alias, quoted=True))) + outer = outer.from_(exp.Table(this=exp.to_identifier("_base"))) + outer = outer.join(src_subq, on=on_range, join_type="LEFT") + for ga in grain_aliases: + outer = outer.group_by(_base_col(ga)) + + return outer.sql(dialect=self.dialect, pretty=True), grain_aliases + def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host ``_base`` CTE + per-plan ``_cm_*`` CTEs + combined SELECT + transform-chain step CTEs + outer ORDER BY/LIMIT wrap. Each block is a coherent compilation stage sharing planned_query / slots_by_id / cma_slot_ids / seen_base_ids state; extracting per-stage helpers would scatter the cross-cutting state. self, *, @@ -4836,6 +5902,12 @@ def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host cma_slot_ids = { p.aggregate_slot_id for p in planned_query.cross_model_aggregate_plans } + # DEV-1714 Stage 10 — windowed aggregate slots render via their own + # host-rooted ``_wm_`` range-join CTEs (below); like cross-model slots + # they are excluded from ``_base`` and joined back on the shared grain. + windowed_slot_ids = { + p.aggregate_slot_id for p in planned_query.windowed_aggregate_plans + } # DEV-1503 — outer combined-SELECT WHERE wrapper. Identify # AGGREGATE-phase host filters whose value-key references any @@ -4872,9 +5944,9 @@ def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host # DEV-1503 (Codex round 2 #1) — composite projection slots whose # value-key tree walks an ISOLATED cross-model aggregate must NOT # render in ``_base``. Inline rendering pulls the filter-target - # joins back into the host CTE (``_collect_column_filter_join_paths``) - # and computes the formula against the host rowset — silently - # corrupting both aggregates when two filter-target INNER joins + # joins back into the host CTE (the host scope's Column.filter + # resolve pass) and computes the formula against the host rowset — + # silently corrupting both aggregates when two filter-target INNER joins # intersect to different rows. Route them to the outer combined # SELECT where the joined-back ``_cm_*`` columns resolve. # @@ -4911,13 +5983,21 @@ def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host for k in walk_value_keys(slot.key): if isinstance(k, AggregateKey): s = slot_by_key.get(k) - if s is not None and s.id in cma_slot_ids: + # DEV-1733: a WINDOWED operand routes the composite outward + # for the same reason a cross-model one does — the value + # lives in a ``_wm_`` CTE joined back to ``_base``, so + # rendering the composite inside ``_base`` would silently + # substitute a PLAIN aggregate for the rolling one. + if s is not None and ( + s.id in cma_slot_ids or s.id in windowed_slot_ids + ): outer_composite_slot_ids.add(slot.id) break base_projection = [ sid for sid in planned_query.projection if sid not in cma_slot_ids and sid not in outer_composite_slot_ids + and sid not in windowed_slot_ids ] # Hidden ORDER-BY-only LOCAL slots (``ORDER BY revenue:sum`` with @@ -4933,8 +6013,13 @@ def _render_with_cross_model_plans( # NOSONAR(S3776) — orchestration of host if ( sid in cma_slot_ids or sid in outer_composite_slot_ids + or sid in windowed_slot_ids or sid in seen_base_ids ): + # DEV-1714: a windowed slot lives in its ``_wm_`` CTE, never + # ``_base`` — materialising it here as an order-only local slot + # would emit a dead plain aggregate in ``_base``. It resolves in + # the combined ORDER BY via its bare projected alias instead. continue slot = slots_by_id.get(sid) if slot is None: @@ -5002,7 +6087,16 @@ def _add_local_aux_slots( dep = slot_by_key.get(k) if dep is None: continue - if dep.id in cma_slot_ids or dep.id in seen_base_ids: + # DEV-1733: a WINDOWED operand is owned by its ``_wm_`` CTE, + # exactly like a cross-model one is owned by ``_cm_``. + # Promoting it into ``_base`` would emit a dead PLAIN + # aggregate under the windowed slot's alias, which the outer + # composite would then read instead of the rolling value. + if ( + dep.id in cma_slot_ids + or dep.id in windowed_slot_ids + or dep.id in seen_base_ids + ): continue base_render_order.append(dep.id) seen_base_ids.add(dep.id) @@ -5062,23 +6156,31 @@ def _add_local_aux_slots( # ``_base`` is empty and the combined query returns 0 # rows (correct host-filter semantics). # - # Round 6 (Codex): walk the non-routed filters' join paths - # via ``_collect_filter_join_paths`` and pull them in via + # Round 6 (Codex): register the non-routed filters' join paths + # into a host ScopeFrame (Law 1 — same single resolver as the + # main host base, D-J) and pull them in via # ``_build_from_and_joins`` — a filter like # ``claim.claim_number = '...'`` references a joined alias # that must be in scope; without the join, the WHERE # references an undefined alias. - placeholder_join_paths = self._collect_filter_join_paths( - planned_query=planned_query, - source_model=source_model, - source_relation=source_relation, + placeholder_allocator = self._gen_allocator or self._new_allocator() + placeholder_scope = ScopeFrame( + scope_id=placeholder_allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, bundle=bundle, + dialect=self._dialect, + allocator=placeholder_allocator, + ) + self._resolve_where_filter_joins_via_scope( + planned_query=planned_query, + scope=placeholder_scope, skip_filter_ids=routed_ids, ) placeholder_from, placeholder_joins = self._build_from_and_joins( source_model=source_model, source_relation=source_relation, - joined_paths=placeholder_join_paths, + joined_paths=placeholder_scope.join_paths.as_list(), bundle=bundle, ) base_select = exp.Select().select( @@ -5162,7 +6264,11 @@ def _add_local_aux_slots( # dangle joined-column aliases on the outer SELECT scope. if base_where is not None and not _base_where_consumed: base_select = base_select.where(base_where) - base_dim_only_dedup = bool(base_group_by) and not base_has_agg + base_dim_only_dedup = ( + planned_query.distinct_dimension_values + and bool(base_group_by) + and not base_has_agg + ) if (base_has_agg or base_dim_only_dedup) and base_group_by: for gb in base_group_by.values(): base_select = base_select.group_by(gb) @@ -5238,6 +6344,46 @@ def _add_local_aux_slots( joinback_pairs_for_plan[plan.aggregate_slot_id] = joinback_pairs agg_col_alias_for_plan[plan.aggregate_slot_id] = agg_col_alias + # DEV-1714 Stage 10 — per-plan ``_wm_`` windowed range-join CTEs. Each + # is host-rooted (``FROM _base LEFT JOIN _src``), grouped at the query + # grain, and joined back to ``_base`` on that grain (host alias == cte + # column alias, since the CTE projects the grain under the same alias). + wm_ctes: List[Tuple[str, str]] = [] + wm_cte_name_for_plan: Dict[str, str] = {} + wm_agg_col_for_plan: Dict[str, str] = {} + wm_joinback_pairs_for_plan: Dict[str, List[Tuple[str, str]]] = {} + # Codex round 4: mint ``_wm_`` CTE names through the DEV-1726 collision- + # aware allocator so two measures whose aliases lossy-sanitise to the + # same name (``rev-a`` / ``rev_a``), or case-only variants on a + # case-folding dialect, get distinct auto-numbered names instead of + # tripping the CTE-name-collision belt. + wm_allocator = self._gen_allocator or self._new_allocator() + for plan in planned_query.windowed_aggregate_plans: + agg_slot = slots_by_id.get(plan.aggregate_slot_id) + if agg_slot is None or not isinstance(agg_slot.key, AggregateKey): + raise RuntimeError( + f"WindowedAggregatePlan {plan.aggregate_slot_id!r} references " + f"a missing or non-aggregate slot.", + ) + full_agg_alias = self._full_alias_for_slot( + slot=agg_slot, source_relation=source_relation, alias_index={}, + ) + cte_name = wm_allocator.allocate_cte( + _cte_name_from_alias("_wm_", full_agg_alias), + ) + cte_sql, grain_aliases = self._render_window_measure_cte_from_planned( + plan=plan, agg_slot=agg_slot, source_model=source_model, + source_relation=source_relation, bundle=bundle, + planned_query=planned_query, slots_by_id=slots_by_id, + aliases_by_slot_id=aliases_by_slot_id, full_agg_alias=full_agg_alias, + ) + wm_ctes.append((cte_name, cte_sql)) + wm_cte_name_for_plan[plan.aggregate_slot_id] = cte_name + wm_agg_col_for_plan[plan.aggregate_slot_id] = full_agg_alias + wm_joinback_pairs_for_plan[plan.aggregate_slot_id] = [ + (a, a) for a in grain_aliases + ] + # Codex MED fold-in: surface dropped-filter warnings from each # plan via Python ``warnings`` so callers using # ``warnings.catch_warnings()`` see what was dropped. The @@ -5274,7 +6420,7 @@ def _add_local_aux_slots( for sid in host_combined_ids: aliases = aliases_by_slot_id.get(sid, []) for full_alias in aliases: - combined_parts.append(f'_base."{full_alias}"') + combined_parts.append(f'_base.{self._quote_ident(full_alias)}') if aliases: combined_aliases_by_slot_id[sid] = list(aliases) # DEV-1503 (Codex round 2 #1) — composite slots routed to the outer @@ -5294,6 +6440,17 @@ def _add_local_aux_slots( outer_composite_cm_map[plan.aggregate_slot_id] = ( cte_name, agg_col_alias, ) + # DEV-1733: windowed operands resolve the same way — the renderer + # substitutes ``.""`` for any slot id in this map, and a + # ``_wm_`` CTE is joined into the combined FROM exactly as a + # ``_cm_`` one is. Without these entries the operand would fall + # through to the ``_base.`` fallback and read a plain + # aggregate (or dangle). + for plan in planned_query.windowed_aggregate_plans: + outer_composite_cm_map[plan.aggregate_slot_id] = ( + wm_cte_name_for_plan[plan.aggregate_slot_id], + wm_agg_col_for_plan[plan.aggregate_slot_id], + ) def _render_outer_composite(cslot) -> str: rendered = self._render_filter_for_outer_wrapper( @@ -5331,7 +6488,7 @@ def _render_outer_composite(cslot) -> str: outer_emission_count[sid] = idx + 1 full_alias = f"{source_relation}.{public_alias}" combined_parts.append( - f'{_render_outer_composite(cslot)} AS "{full_alias}"', + f'{_render_outer_composite(cslot)} AS {self._quote_ident(full_alias)}', ) combined_aliases_by_slot_id.setdefault(sid, []).append( full_alias, @@ -5372,22 +6529,73 @@ def _render_outer_composite(cslot) -> str: canonical_alias = canonical_alias_for_plan[plan.aggregate_slot_id] agg_col_alias = agg_col_alias_for_plan[plan.aggregate_slot_id] cte_name = _cte_name_from_alias("_cm_", canonical_alias) - public_aliases = self._public_aliases_for_cross_model_agg( - slot=agg_slot, - source_relation=source_relation, - canonical_alias=canonical_alias, + # DEV-1495 bug 2 / DEV-1712: an order-by-only (hidden) cross-model + # aggregate never surfaces in the combined projection — its CTE is + # still joined below, and the ORDER BY references it CTE-qualified + # (``hidden_cte_order_refs``). Trimming it keeps the outer SELECT to + # the user-declared columns (Law 2 projection boundary). Only when + # there is NO transform chain: a hidden CMA feeding a transform + # layer (``cumsum(customers.revenue:sum)``) must stay projected so + # the step CTE can consume it — the transform outer wrap does the + # public-vs-hidden trim in that path. + trim_hidden = plan.hidden and not planned_query.transform_layers + public_aliases = ( + [] + if trim_hidden + else self._public_aliases_for_cross_model_agg( + slot=agg_slot, + source_relation=source_relation, + canonical_alias=canonical_alias, + ) ) for pub in public_aliases: if pub == agg_col_alias: - combined_parts.append(f'{cte_name}."{agg_col_alias}"') + combined_parts.append(f'{cte_name}.{self._quote_ident(agg_col_alias)}') else: combined_parts.append( - f'{cte_name}."{agg_col_alias}" AS "{pub}"', + f'{cte_name}.{self._quote_ident(agg_col_alias)} AS {self._quote_ident(pub)}', ) combined_aliases_by_slot_id[plan.aggregate_slot_id] = list( public_aliases, ) + # DEV-1714 Stage 10 — windowed side: project each ``_wm_`` CTE's + # aggregate column. Codex#2: one occurrence per declared user alias (C13 + # lets the same windowed key be selected under multiple names — the CTE + # holds one aggregate column, remapped ``AS`` each public alias). The + # column name already IS the primary dotted result key, so that occurrence + # needs no remap. Like the cross-model ``_cm_`` columns above, windowed + # columns are grouped after the ``_base`` projection rather than woven + # into ``planned_query.projection`` order — deterministic (measure + # declaration order) and harmless because results are keyed by name, not + # position. + for plan in planned_query.windowed_aggregate_plans: + agg_slot = slots_by_id[plan.aggregate_slot_id] + cte_name = wm_cte_name_for_plan[plan.aggregate_slot_id] + agg_col = wm_agg_col_for_plan[plan.aggregate_slot_id] + # DEV-1733: an order-only (hidden) windowed aggregate never surfaces + # in the combined projection — its ``_wm_`` CTE is still joined + # below and the ORDER BY references it CTE-qualified + # (``hidden_wm_order_ref``). Same trim predicate the hidden + # cross-model aggregate uses: with a transform chain on top the + # column must stay projected so the step CTE can consume it, and + # the transform outer wrap does the public-vs-hidden trim there. + if plan.hidden and not planned_query.transform_layers: + combined_aliases_by_slot_id[plan.aggregate_slot_id] = [] + continue + public_names = list(agg_slot.public_aliases) or ( + [agg_slot.public_name] if agg_slot.public_name else [] + ) + full_aliases = [f"{source_relation}.{p}" for p in public_names] or [agg_col] + for full in full_aliases: + if full == agg_col: + combined_parts.append(f'{cte_name}.{self._quote_ident(agg_col)}') + else: + combined_parts.append( + f'{cte_name}.{self._quote_ident(agg_col)} AS {self._quote_ident(full)}', + ) + combined_aliases_by_slot_id[plan.aggregate_slot_id] = list(full_aliases) + from_clause_str = "FROM _base" joined_cte_names: set = set() for plan in planned_query.cross_model_aggregate_plans: @@ -5399,9 +6607,39 @@ def _render_outer_composite(cslot) -> str: joinback_pairs = joinback_pairs_for_plan.get( plan.aggregate_slot_id, [], ) + if joinback_pairs: + # DEV-1708 / Codex F2: the grain join-back uses a dialect-aware + # NULL-SAFE equality so NULL dimension values and nullable + # truncated time grains join back instead of dropping their + # aggregate (a plain ``=`` yields NULL for NULL = NULL). + join_parts = [ + self._null_safe_join_pair_sql( + left_sql=f'_base.{self._quote_ident(host)}', + right_sql=f'{cte_name}.{self._quote_ident(cte_col)}', + ) + for host, cte_col in joinback_pairs + ] + from_clause_str += ( + f"\nLEFT JOIN {cte_name} ON " + _SQL_AND_JOINER.join(join_parts) + ) + else: + from_clause_str += f"\nCROSS JOIN {cte_name}" + + # DEV-1714 Stage 10 — LEFT JOIN each ``_wm_`` CTE back to ``_base`` on + # the shared grain (null-safe, so NULL-dim / nullable-grain groups keep + # a row; the windowed value for a NULL-dim group is NULL — the plain + # ``=`` inside the CTE never matches NULL, a documented consequence). + for plan in planned_query.windowed_aggregate_plans: + cte_name = wm_cte_name_for_plan[plan.aggregate_slot_id] + joinback_pairs = wm_joinback_pairs_for_plan.get( + plan.aggregate_slot_id, [], + ) if joinback_pairs: join_parts = [ - f'_base."{host}" = {cte_name}."{cte_col}"' + self._null_safe_join_pair_sql( + left_sql=f'_base.{self._quote_ident(host)}', + right_sql=f'{cte_name}.{self._quote_ident(cte_col)}', + ) for host, cte_col in joinback_pairs ] from_clause_str += ( @@ -5453,11 +6691,52 @@ def _render_outer_composite(cslot) -> str: "\nWHERE " + _SQL_AND_JOINER.join(outer_where_parts) ) + # DEV-1714 Stage 10 — POST-phase filters referencing a windowed measure + # render as an outer WHERE on the combined SELECT (never HAVING on the + # plain base aggregate), resolving each windowed slot to its ``_wm_`` + # CTE's joined-back aggregate column. + if planned_query.windowed_aggregate_plans: + wm_slot_to_cte: Dict[str, Tuple[str, str]] = { + p.aggregate_slot_id: ( + wm_cte_name_for_plan[p.aggregate_slot_id], + wm_agg_col_for_plan[p.aggregate_slot_id], + ) + for p in planned_query.windowed_aggregate_plans + } + wm_post_parts: List[str] = [] + for fp in planned_query.filters_by_phase: + if fp.phase != Phase.POST or fp.expression is None: + continue + rendered = self._render_filter_for_outer_wrapper( + key=fp.expression.value_key, + slot_by_key=slot_by_key, + cross_model_agg_slot_to_cm=wm_slot_to_cte, + aliases_by_slot_id=aliases_by_slot_id, + ) + if isinstance(rendered, (exp.And, exp.Or)): + rendered = exp.Paren(this=rendered) + wm_post_parts.append(rendered.sql(dialect=self.dialect)) + if wm_post_parts: + connector = "\nAND " if outer_where_filters else "\nWHERE " + combined_select_sql += connector + _SQL_AND_JOINER.join(wm_post_parts) + # DEV-1450 stage 7b.15e (C2): a transform layer over a cross-model # aggregate (``cumsum(customers.avg_score:avg)``) runs on TOP of the # combined cross-model result — the combined SELECT becomes the base # CTE and the window step CTEs / outer wrap are layered above it. if planned_query.transform_layers: + if wm_ctes: + # Unreachable today — guard G4 rejects a windowed measure that + # coexists with a transform — but the combined SELECT already + # projects/joins the _wm_ CTEs, which this prelude omits. Fail + # loudly so lifting G4 (DEV-1504) can't silently emit a statement + # referencing undefined _wm_ CTEs. + raise NotImplementedError( + "DEV-1714 Stage 10: a windowed measure combined with a " + "transform layer is not supported (guarded at plan time by " + "G4); the cross-model transform chain does not carry `_wm_` " + "CTEs.", + ) return self._render_cross_model_transform_chain( prelude_ctes=[("_base", base_cte_sql)] + cm_ctes, combined_select_sql=combined_select_sql, @@ -5467,7 +6746,7 @@ def _render_outer_composite(cslot) -> str: source_relation=source_relation, ) - all_ctes = [("_base", base_cte_sql)] + cm_ctes + [("_combined", combined_select_sql)] + all_ctes = [("_base", base_cte_sql)] + cm_ctes + wm_ctes + [("_combined", combined_select_sql)] # Stitch the WITH chain together. Inner CTEs first; the final # ``_combined`` is the outermost FROM target. @@ -5478,14 +6757,44 @@ def _render_outer_composite(cslot) -> str: # level. ORDER BY columns must be qualified — ``_base`` columns # use ``_base."..."``, cross-model columns use the bare alias # (only present on one side). + # DEV-1712 / DEV-1495 bug 2: hidden (order-only) cross-model aggregates + # are trimmed from the projection above, so their ORDER BY term must be + # CTE-qualified (``_cm_*.""``) rather than the bare + # combined-SELECT alias. + hidden_cte_order_refs: Dict[str, str] = {} + for plan in planned_query.cross_model_aggregate_plans: + # Only CMAs actually trimmed from the projection (hidden + no + # transform chain) need the CTE-qualified ORDER BY reference. + if not (plan.hidden and not planned_query.transform_layers): + continue + _canon = canonical_alias_for_plan[plan.aggregate_slot_id] + _agg_col = agg_col_alias_for_plan[plan.aggregate_slot_id] + _cte = _cte_name_from_alias("_cm_", _canon) + hidden_cte_order_refs[plan.aggregate_slot_id] = ( + f'{_cte}.{self._quote_ident(_agg_col)}' + ) + # DEV-1733: same treatment for a hidden (order-only) WINDOWED aggregate + # trimmed from the combined projection above — reference its ``_wm_`` + # CTE column rather than a bare alias the SELECT no longer emits. + for plan in planned_query.windowed_aggregate_plans: + if not (plan.hidden and not planned_query.transform_layers): + continue + hidden_cte_order_refs[plan.aggregate_slot_id] = ( + f'{wm_cte_name_for_plan[plan.aggregate_slot_id]}.' + f'{self._quote_ident(wm_agg_col_for_plan[plan.aggregate_slot_id])}' + ) order_sql = self._build_combined_order_by_sql( planned_query=planned_query, slots_by_id=slots_by_id, cma_slot_ids=cma_slot_ids, cm_alias_for_plan=canonical_alias_for_plan, - bare_order_slot_ids=set(order_only_local_ids), + # DEV-1714: windowed slots are referenced bare in the combined ORDER + # BY — they surface as a projected combined-SELECT column (from their + # ``_wm_`` CTE), so a ``_base.`` qualifier would dangle. + bare_order_slot_ids=set(order_only_local_ids) | windowed_slot_ids, outer_composite_aliases=outer_composite_order_alias_by_sid, outer_composite_expressions=outer_composite_order_expressions, + hidden_cte_order_refs=hidden_cte_order_refs, ) if order_sql: sql += "\n" + order_sql @@ -5575,7 +6884,7 @@ def _render_cross_model_transform_chain( carry_aliases_sorted = sorted( a for aliases in aliases_by_slot_id.values() for a in aliases ) - step_parts = [f'"{a}"' for a in carry_aliases_sorted] + step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] for layer in ready: for slot_id in layer.slot_ids: slot = slots_by_id[slot_id] @@ -5596,7 +6905,7 @@ def _render_cross_model_transform_chain( window_sql = _wrap_cast_for_type( self._parse(window_sql), slot.type, ).sql(dialect=self.dialect) - step_parts.append(f'{window_sql} AS "{full_alias}"') + step_parts.append(f'{window_sql} AS {self._quote_ident(full_alias)}') aliases_by_slot_id.setdefault(slot_id, []).append(full_alias) available_alias_by_slot_id.setdefault(slot_id, full_alias) step_sql = ( @@ -5629,7 +6938,7 @@ def _render_cross_model_transform_chain( carry_aliases_sorted = sorted( a for aliases in aliases_by_slot_id.values() for a in aliases ) - step_parts = [f'"{a}"' for a in carry_aliases_sorted] + step_parts = [self._quote_ident(a) for a in carry_aliases_sorted] for cslot in unmaterialised: alias = ( cslot.public_aliases[0] @@ -5647,7 +6956,7 @@ def _render_cross_model_transform_chain( expr_sql = _wrap_cast_for_type( self._parse(expr_sql), cslot.type, ).sql(dialect=self.dialect) - step_parts.append(f'{expr_sql} AS "{full_alias}"') + step_parts.append(f'{expr_sql} AS {self._quote_ident(full_alias)}') aliases_by_slot_id.setdefault(cslot.id, []).append(full_alias) available_alias_by_slot_id.setdefault(cslot.id, full_alias) step_sql = ( @@ -5663,7 +6972,7 @@ def _render_cross_model_transform_chain( ) inner_sql = ( "SELECT\n " - + _SQL_COL_SEP.join(f'"{a}"' for a in inner_sorted) + + _SQL_COL_SEP.join(self._quote_ident(a) for a in inner_sorted) + f"\nFROM {final_cte}" ) cte_clause = ( @@ -5700,7 +7009,7 @@ def _render_cross_model_transform_chain( public_aliases_user_order.append(alias) outer_sql = ( "SELECT\n " - + _SQL_COL_SEP.join(f'"{a}"' for a in public_aliases_user_order) + + _SQL_COL_SEP.join(self._quote_ident(a) for a in public_aliases_user_order) + f"\nFROM (\n{chain_sql}\n) AS _outer" ) @@ -5902,7 +7211,6 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share """ from slayer.core.enums import TimeGranularity from slayer.core.keys import ( - AggregateKey, ColumnKey, ColumnSqlKey, Phase, @@ -5935,8 +7243,31 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share # column on the left side). Intersect with the host's actual # projection ids so only projected slots flow into the CTE. cte_select_columns: List[exp.Expression] = [] + # DEV-1728: two GROUP-BY lists — ``cte_group_by`` is the OUTER GROUP BY + # (an alias ref ``_val_`` for a first/last-materialised crossing + # grain, the raw expression otherwise); ``cte_partition_exprs`` is the + # ranked-subquery PARTITION BY, always the RAW expression (valid inside + # the subquery where the crossed join is bound). They are identical for + # every non-first/last query and every non-crossing grain. cte_group_by: List[exp.Expression] = [] + cte_partition_exprs: List[exp.Expression] = [] shared_grain_aliases: List[str] = [] + # DEV-1701: join paths crossed by a shared-grain derived TIME dimension's + # expanded ``Column.sql``. Collected during the loop (which runs before + # the CTE scope's join set is assembled) and merged into it below. + shared_grain_join_paths: List[Tuple[str, ...]] = [] + # DEV-1728: first/last grain materialisations (``_val_`` projections + # to inject INTO the ranked subquery for crossing derived grains). The + # generation-wide allocator is hoisted here so grain + source ``_val``s + # share one monotonic sequence. Reserve the target's physical column + # names (Codex F6): the ranked subquery re-exports ``target.*``, so a + # minted ``_val_`` must never collide with a real target column of + # that name — mirrors the host-path reservation in + # ``_build_first_last_base_select``. + cte_allocator = self._gen_allocator or self._new_allocator() + self._reserve_model_column_names(cte_allocator, target_model) + is_first_or_last = agg_slot.key.agg in ("first", "last") + grain_extra_projections: List[Tuple[str, exp.Expression]] = [] for sid in plan.shared_grain_slots: if sid not in base_projection_ids: continue @@ -5949,6 +7280,12 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share path = key.path elif isinstance(key, TimeTruncKey): path = key.column.path + elif isinstance(key, ColumnSqlKey): + # DEV-1708 / DEV-1728: a plain derived (non-time) dim carries its + # own path; a path-bearing one renders here like any grain, a + # host-local ``path == ()`` one falls through to the CROSS-JOIN + # broadcast below (unchanged). + path = key.path if not path: # Local-only host dim — broadcast via CROSS JOIN. continue @@ -5977,13 +7314,43 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share grain_column = key.column if isinstance(key, TimeTruncKey) else key if isinstance(grain_column, _ColumnSqlKey): - col_expr = self._parse(self._expand_derived_column_sql( + # DEV-1728: a derived (ColumnSqlKey) grain — plain dimension OR + # time dimension — expands its Column.sql rooted at the target and + # renders here. (The DEV-1708 raise for a PLAIN derived grain is + # gone: DEV-1713 fixed the naming half, so the host's dotted alias + # and the CTE join-back now agree.) + expanded_grain_sql = self._expand_derived_column_sql( source_model=target_model, source_relation=target_relation, column_name=grain_column.column_name, bundle=bundle, - )) + ) + col_expr = self._parse(expanded_grain_sql) leaf = grain_column.column_name + if not isinstance(key, TimeTruncKey): + # DEV-1728: a PLAIN derived grain is CAST to its declared type + # to match the host base's ``_wrap_cast_for_type`` (a bare + # column ref / TEXT is skipped there and here identically), so + # the join-back compares identically-typed values. A + # TimeTrunc-wrapped grain keeps ``_build_date_trunc``'s own + # temporal shape (no extra cast — parity with the host base). + grain_col = next( + (c for c in target_model.columns + if c.name == grain_column.column_name), + None, + ) + col_expr = _wrap_cast_for_type( + col_expr, grain_col.type if grain_col else None, + ) + # DEV-1701: register every further join the derived grain's + # expanded sql crosses (rooted at the target relation), so the + # CTE's FROM pulls it. Merged into the CTE join set below. + for _p in self._joined_paths_in_sql( + sql_expr=col_expr, source_relation=target_relation, + source_model=target_model, bundle=bundle, + ): + if _p not in shared_grain_join_paths: + shared_grain_join_paths.append(_p) else: leaf = grain_column.leaf col_expr = exp.Column( @@ -6001,8 +7368,32 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share # ``_build_base_select_for_planned`` already aliases that # way for joined ROW slots. host_alias = planned_query.source_relation + "." + ".".join(path) + f".{leaf}" - cte_select_columns.append(col_expr.copy().as_(host_alias)) - cte_group_by.append(col_expr) + # DEV-1728 Law 2: for a first/last aggregate the CTE's FROM is a + # ROW_NUMBER-ranked subquery that re-exports only ``target.*`` + rank + # columns. A grain whose expression CROSSES a join references a table + # bound ONLY inside that subquery, so the outer SELECT / GROUP BY + # cannot name it — materialise it as a ``_val_`` projection inside + # the subquery, group the outer SELECT on the alias, and keep the RAW + # expression for PARTITION BY (evaluated where the join is bound). A + # target-local grain (no crossing) needs no materialisation — it is + # re-exported by ``target.*``. + grain_crosses = is_first_or_last and bool( + self._joined_paths_in_sql( + sql_expr=col_expr, source_relation=target_relation, + source_model=target_model, bundle=bundle, + ) + ) + if grain_crosses: + val_alias = cte_allocator.allocate_val() + grain_extra_projections.append((val_alias, col_expr.copy())) + alias_ref = exp.column(val_alias) + cte_select_columns.append(alias_ref.copy().as_(host_alias)) + cte_group_by.append(alias_ref.copy()) + cte_partition_exprs.append(col_expr.copy()) + else: + cte_select_columns.append(col_expr.copy().as_(host_alias)) + cte_group_by.append(col_expr.copy()) + cte_partition_exprs.append(col_expr.copy()) shared_grain_aliases.append(host_alias) # Aggregate column: synthesise an EnrichedMeasure ROOTED at the @@ -6017,57 +7408,73 @@ def _render_cross_model_cte( # NOSONAR(S3776) — single conceptual unit: share # here with both source and ``other`` rooted at ``("customers",)``. # Stripping the prefix in lockstep means the synth helper's # path-validation invariant (``kwarg.path == source.path``) holds. - # Re-root the aggregate SOURCE and any column-valued kwargs to the - # target's local scope (path=()). Covers a derived (ColumnSqlKey) - # source like ``customers.net:sum`` — Codex: otherwise the - # host-rooted derived key renders against the wrong alias inside the - # CTE. StarKey ignores path (COUNT(*)), so leave it as-is. - _src = agg_slot.key.source - cross_model_path = getattr(_src, "path", ()) - if isinstance(_src, ColumnKey): - local_source_key = ColumnKey(path=(), leaf=_src.leaf) - elif isinstance(_src, ColumnSqlKey): - local_source_key = ColumnSqlKey( - path=(), model=_src.model, column_name=_src.column_name, - ) - else: - local_source_key = _src - - def _reroot_kwarg(kval): - if isinstance(kval, ColumnKey) and kval.path == cross_model_path: - return ColumnKey(path=(), leaf=kval.leaf) - if isinstance(kval, ColumnSqlKey) and kval.path == cross_model_path: - return ColumnSqlKey( - path=(), model=kval.model, column_name=kval.column_name, - ) - return kval - - local_kwargs = tuple( - (k, _reroot_kwarg(v)) for k, v in agg_slot.key.kwargs - ) - # DEV-1476 bug (c): symmetric reroot of positional args. An explicit - # time arg ``customers.amount:last(customers.signup_at)`` arrives - # here with ``key.args=(ColumnKey(path=("customers",), - # leaf="signup_at"),)``. Without rerooting, ``_resolve_explicit_ - # time_col`` qualifies the time column under the wrong alias inside - # the target-rooted CTE. ``_reroot_kwarg`` already does the right - # thing for ``ColumnKey`` / ``ColumnSqlKey``; reuse it. - local_args = tuple( - _reroot_kwarg(a) if isinstance(a, (ColumnKey, ColumnSqlKey)) else a - for a in agg_slot.key.args - ) - local_agg_key = AggregateKey( - source=local_source_key, - agg=agg_slot.key.agg, - args=local_args, - kwargs=local_kwargs, - column_filter_key=agg_slot.key.column_filter_key, + # Re-root the aggregate SOURCE and ALL embedded refs (positional args + # AND column-valued kwargs) from the host's coordinate system into the + # target's local scope in one symmetric pass (DEV-1707). Covers a + # derived (ColumnSqlKey) source like ``customers.net:sum`` — otherwise + # the host-rooted derived key renders against the wrong alias inside + # the CTE — and the DEV-1476(c) explicit time arg + # ``customers.amount:last(customers.signup_at)``, whose positional arg + # must strip the host prefix in lockstep with the source so + # ``_resolve_explicit_time_col`` qualifies the time column under the + # target relation. ``column_filter_key`` rides through unchanged + # (owner-anchored, invariant under reroot). + cross_model_path = getattr(agg_slot.key.source, "path", ()) + local_agg_key = reroot_aggregate_key( + agg_slot.key, target_path=cross_model_path, ) # The local_agg_key was built from the target's own column. # column_filter_key (if set) carries the canonical filter SQL # from the target's Column.filter — the synth helper qualifies # bare refs against target_model. local_slot = agg_slot.model_copy(update={"key": local_agg_key}) + + # DEV-1708 Law 1: every expression rendered into this CTE enters through + # a single ScopeFrame rooted at the target relation. ``resolve`` anchors + # each ref and REGISTERS the joins it crosses into ``cte_scope.join_paths`` + # as a side effect — the CTE's FROM is built from that set below, so a + # crossed join can never be forgotten (replaces the ad-hoc + # ``_add_cte_join_paths`` closure + per-carrier collectors). The scope + # shares the generation-wide allocator (``cte_allocator``, hoisted above + # the grain loop) so ``_val_`` materialisation names (Law 2) are + # unique across the host base, the grain projections, and every CTE. + cte_scope = ScopeFrame( + scope_id=cte_allocator.next_scope_id(target_relation), + root_model=target_model, + root_relation=target_relation, + bundle=bundle, + dialect=self._dialect, + allocator=cte_allocator, + ) + # DEV-1701: merge the shared-grain derived-TIME-dim crossed joins + # collected in the loop above. + for _p in shared_grain_join_paths: + cte_scope.join_paths.add(_p) + # DEV-1526: register the rerooted aggregate SOURCE's crossed joins (a + # derived ``ColumnSqlKey`` source like ``customers_v2.deep_pop:sum`` whose + # ``Column.sql`` = ``regions.population`` must pull the customers_v2 → + # regions join into the CTE). Registration only — the render spec + # re-expands the source itself. + if isinstance(local_agg_key.source, ColumnSqlKey): + cte_scope.resolve(local_agg_key.source) + # DEV-1476(c) / Codex F1: register every positional ARG's crossed joins + # (the explicit first/last time arg may itself be a derived column whose + # sql crosses a further join — its ranking ORDER BY needs that join). + for _arg in local_agg_key.args: + if isinstance(_arg, (ColumnKey, ColumnSqlKey)): + cte_scope.resolve(_arg) + # DEV-1527 (cross-model remainder): resolve each column-ref KWARG through + # the scope — anchors the expanded expression AND registers its join — + # then embed it as a trusted ``kind="expr"`` into the render spec so a + # derived kwarg emits its expanded sql (``regions.weight``) instead of a + # bare, non-existent ``customers_v2.deep_weight``. + cte_resolved_kwargs: "Dict[str, ResolvedAggKwarg]" = {} + for _kname, _kval in local_agg_key.kwargs: + if isinstance(_kval, (ColumnKey, ColumnSqlKey)): + cte_resolved_kwargs[_kname] = ResolvedAggKwarg( + kind="expr", value=cte_scope.resolve(_kval), + ) + synth = self._build_agg_render_spec_from_planned( slot=local_slot, key=local_agg_key, @@ -6075,6 +7482,7 @@ def _reroot_kwarg(kval): source_relation=target_relation, full_alias=full_agg_alias, bundle=bundle, + resolved_agg_kwargs=cte_resolved_kwargs or None, ) # DEV-1476 bug (c): for first/last aggregates the FROM must be a @@ -6090,8 +7498,10 @@ def _reroot_kwarg(kval): # relation). If even that is unset, raise the standard # "first/last requires a ranking time column" error rather than # silently emitting an agg_expr that references a non-existent - # ``_first_rn`` / ``_last_rn`` column. - is_first_or_last = local_agg_key.agg in ("first", "last") + # ``_first_rn`` / ``_last_rn`` column. (``is_first_or_last`` is computed + # once above the grain loop from ``agg_slot.key.agg`` — reroot preserves + # the aggregation name — so the grain materialisation and this branch + # agree.) time_col_sql: Optional[str] = synth.time_column if is_first_or_last and time_col_sql is None: if target_model.default_time_dimension: @@ -6114,24 +7524,25 @@ def _reroot_kwarg(kval): # subquery — otherwise rows excluded by a filter could still # win ``_last_rn = 1`` and yield NULL aggregates. # DEV-1494: join paths the CTE's own filters cross — the target measure's - # ``Column.filter`` and the target-model filters. Each ``_cm_*`` CTE is an - # isolated per-(target, grain) computation, so adding these joins to ITS - # FROM affects only this measure (not siblings) — it resolves the filter's - # refs without the cross-measure cardinality concern DEV-1503 owns. - cte_join_paths: List[Tuple[str, ...]] = [] - - def _add_cte_join_paths(sql_text: Optional[str]) -> None: + # ``Column.filter`` and the target-model filters — registered into the + # CTE scope (Law 1). Each ``_cm_*`` CTE is an isolated per-(target, grain) + # computation, so adding these joins to ITS FROM affects only this + # measure (not siblings) — it resolves the filter's refs without the + # cross-measure cardinality concern DEV-1503 owns. Free-SQL predicates + # keep the quote-tolerant dual-scan of ``_filter_join_paths`` (raw + + # inline-expanded — the DEV-1494/dedup contract) while writing into the + # single ``cte_scope.join_paths`` set. + def _register_filter_join_paths(sql_text: Optional[str]) -> None: if not sql_text: return for p in self._filter_join_paths( sql=sql_text, source_relation=target_relation, source_model=target_model, bundle=bundle, ): - if p not in cte_join_paths: - cte_join_paths.append(p) + cte_scope.join_paths.add(p) if local_agg_key.column_filter_key is not None: - _add_cte_join_paths(local_agg_key.column_filter_key.canonical_sql) + _register_filter_join_paths(local_agg_key.column_filter_key.canonical_sql) where_parts: List[exp.Expression] = [] for filter_text in plan.target_model_filters: @@ -6139,8 +7550,8 @@ def _add_cte_join_paths(sql_text: Optional[str]) -> None: # non-trivial derived column on the target (bare OR a dotted ref to a # derived column on a joined model) is inline-expanded; base-only # filters keep the AST bare-ref qualification. The crossed join is - # pulled into this CTE's FROM via ``cte_join_paths``. - _add_cte_join_paths(filter_text) + # pulled into this CTE's FROM via ``cte_scope.join_paths``. + _register_filter_join_paths(filter_text) qualified = self._render_mode_a_predicate( sql=filter_text, source_model=target_model, @@ -6161,6 +7572,22 @@ def _add_cte_join_paths(sql_text: Optional[str]) -> None: f"Target model filter on {target_model_name!r} could " f"not be parsed: {filter_text!r}", ) + # DEV-1708 / Codex F4: pre-pass — walk the FULL ValueKey tree of every + # routed WHERE and HAVING filter (nested arithmetic / boolean / IN + # operands, aggregate leaves' source + args + kwargs + column_filter, + # derived ColumnSqlKey refs) and register the joins they cross into the + # CTE scope BEFORE the FROM is built. HAVING is rendered later (it needs + # the ranked-subquery rn maps), so its joins would otherwise register + # too late to reach the FROM. + self._register_routed_filter_joins( + planned_query=planned_query, + filter_ids=list(plan.where_filter_ids) + list(plan.having_filter_ids), + target_relation=target_relation, + target_model=target_model, + bundle=bundle, + scope=cte_scope, + target_path=target_path, + ) cte_where = self._collect_routed_filters( planned_query=planned_query, filter_ids=plan.where_filter_ids, @@ -6186,6 +7613,7 @@ def _add_cte_join_paths(sql_text: Optional[str]) -> None: # over the filtered row set; otherwise a filtered-out row could win # ``_last_rn = 1`` and the ``MAX(CASE WHEN _last_rn = 1 ...)`` # aggregate would return NULL. + cte_join_paths = cte_scope.join_paths.as_list() if cte_join_paths: target_from, cte_base_joins = self._build_from_and_joins( source_model=target_model, source_relation=target_relation, @@ -6197,14 +7625,57 @@ def _add_cte_join_paths(sql_text: Optional[str]) -> None: ) cte_base_joins = [] ranked_from: Optional[exp.Expression] = None + cte_value_alias_by_sql: Dict[str, str] = {} if is_first_or_last: assert time_col_sql is not None # narrowed by the guard above + # DEV-1708 Law 2 (DEV-1702 B2, forward variant): the ranked subquery + # re-exports only ``target.*`` + rank columns. If the first/last + # SOURCE value crosses a join, the crossing ref must be materialised + # as a ``_val_`` projection INSIDE the subquery and the outer + # aggregate rewritten to reference the alias — otherwise the outer + # ``MAX(CASE WHEN _last_rn = 1 THEN END)`` references a + # table bound only inside the subquery (out of scope). The FILTER + # refs are consumed inside the subquery (the ``_last_rn_fN`` / + # ``_match_fN`` rank columns) and need no outer alias. A LOCAL source + # value is already covered by ``target.*`` — no materialisation. + outer_synth = synth + # DEV-1728: seed with the crossing-grain ``_val_`` projections + # collected in the grain loop, so a first/last aggregate grouped by a + # crossing derived grain materialises that grain INSIDE the subquery + # too (the outer SELECT / GROUP BY reference the alias). + extra_projections: List[Tuple[str, exp.Expression]] = list( + grain_extra_projections, + ) + if synth.sql: + # DEV-1709: materialise the RESOLVED value (qualified + + # ``Column.type`` inner CAST for non-bare expressions) and + # key the alias map by that resolved text — mirrors the + # host-path materialisation in + # ``_build_first_last_base_select`` so typed non-bare + # sources keep ``MAX(CASE ... THEN CAST(x AS t) END)`` + # semantics inside the CTE too. + value_expr = self._resolve_sql( + sql=synth.sql, name=synth.name, + model_name=synth.model_name, type=synth.column_type, + ) + if self._joined_paths_in_sql( + sql_expr=value_expr, source_relation=target_relation, + source_model=target_model, bundle=bundle, + ): + val_alias = cte_allocator.allocate_val() + extra_projections.append((val_alias, value_expr)) + outer_synth = synth.model_copy(update={"sql": val_alias}) + # A HAVING on this same aggregate must reference the alias, + # not the raw crossing ref (out of scope in the outer SELECT). + cte_value_alias_by_sql[ + value_expr.sql(dialect=self.dialect) + ] = val_alias ranked_from, rn_suffix_map, filtered_rn_map, filtered_match_map = ( self._build_ranked_subquery_from_planned( source_relation=target_relation, default_time_col_sql=time_col_sql, - partition_exprs=list(cte_group_by), - extra_projections=[], + partition_exprs=list(cte_partition_exprs), + extra_projections=extra_projections, synth_specs=[synth], from_clause=target_from, base_joins=cte_base_joins, @@ -6212,7 +7683,7 @@ def _add_cte_join_paths(sql_text: Optional[str]) -> None: ) ) agg_expr, is_agg = self._build_agg( - synth, + outer_synth, rn_suffix_map=rn_suffix_map, default_time_col=time_col_sql, filtered_rn_map=filtered_rn_map, @@ -6261,6 +7732,7 @@ def _add_cte_join_paths(sql_text: Optional[str]) -> None: filtered_rn_map=dict(filtered_rn_map), filtered_match_map=dict(filtered_match_map), agg_synth_alias=full_agg_alias, + value_alias_by_sql=dict(cte_value_alias_by_sql), ) cte_having = self._collect_routed_filters( planned_query=planned_query, @@ -6276,6 +7748,104 @@ def _add_cte_join_paths(sql_text: Optional[str]) -> None: cte_sql = cte_select.sql(dialect=self.dialect, pretty=True) return cte_sql, shared_grain_aliases + def _register_routed_filter_joins( # NOSONAR(S3776) — a cohesive recursive ValueKey tree-walk dispatcher (the heavy AggregateKey arm is already extracted to _register_agg_key_joins); the remaining branches are the closed-union dispatch contract, mirroring the sibling walkers _value_key_join_paths / _collect_base_aux_slot_ids in this file. + self, + *, + planned_query, + filter_ids: List[str], + target_relation: str, + target_model, + bundle, + scope: ScopeFrame, + target_path: Tuple[str, ...], + ) -> None: + """DEV-1708 / Codex F4 — register the joins crossed by every routed + WHERE/HAVING filter into ``scope.join_paths``, walking the FULL + ``ValueKey`` tree so nested arithmetic/boolean/IN operands and aggregate + leaves (source + positional args + column-ref kwargs + ``column_filter``) + all contribute BEFORE the CTE FROM is assembled. + + Registration only — the render passes (``_collect_routed_filters`` for + WHERE, and the HAVING render below) emit the SQL themselves. The scope's + ``resolve`` anchors each typed leaf at the target relation and records + the path it crosses; free-SQL ``column_filter`` predicates keep the + quote-tolerant dual-scan of ``_filter_join_paths``. + """ + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + BetweenKey, + ColumnKey, + ColumnSqlKey, + InKey, + ScalarCallKey, + ) + + if not filter_ids: + return + wanted = set(filter_ids) + + def _walk(vk) -> None: + if isinstance(vk, (ColumnKey, ColumnSqlKey)): + # Reroot a path-qualified leaf into the target's local scope by + # stripping the CTE's ``target_path`` prefix (NOT the ref's own + # path — a ref one hop past the target keeps its residual so the + # deeper join still registers), then resolve. + local = ( + _reroot_path_ref(vk, target_path=target_path) + if vk.path else vk + ) + scope.resolve(local) + elif isinstance(vk, AggregateKey): + self._register_agg_key_joins( + agg_key=vk, scope=scope, target_relation=target_relation, + target_model=target_model, bundle=bundle, + ) + elif isinstance(vk, ArithmeticKey): + for op in vk.operands: + _walk(op) + elif isinstance(vk, ScalarCallKey): + for a in vk.args: + _walk(a) + elif isinstance(vk, BetweenKey): + _walk(vk.column) + _walk(vk.low) + _walk(vk.high) + elif isinstance(vk, InKey): + _walk(vk.column) + + for fp in planned_query.filters_by_phase: + if fp.id in wanted and fp.expression is not None: + _walk(fp.expression.value_key) + + def _register_agg_key_joins( + self, *, agg_key, scope: ScopeFrame, target_relation: str, + target_model, bundle, + ) -> None: + """Register the joins an aggregate leaf crosses (source + positional + args + column-ref kwargs + ``column_filter``) into ``scope.join_paths`` + — the ``AggregateKey`` arm of ``_register_routed_filter_joins``'s tree + walk, extracted so the walker stays a thin dispatcher (DEV-1708).""" + from slayer.core.keys import ColumnKey, ColumnSqlKey + + cross_model_path = getattr(agg_key.source, "path", ()) + local_agg = reroot_aggregate_key(agg_key, target_path=cross_model_path) + if isinstance(local_agg.source, ColumnSqlKey): + scope.resolve(local_agg.source) + for a in local_agg.args: + if isinstance(a, (ColumnKey, ColumnSqlKey)): + scope.resolve(a) + for _k, v in local_agg.kwargs: + if isinstance(v, (ColumnKey, ColumnSqlKey)): + scope.resolve(v) + cfk = local_agg.column_filter_key + if cfk is not None and cfk.canonical_sql: + for p in self._filter_join_paths( + sql=cfk.canonical_sql, source_relation=target_relation, + source_model=target_model, bundle=bundle, + ): + scope.join_paths.add(p) + def _collect_routed_filters( self, *, @@ -6342,10 +7912,12 @@ def _render_filter_value_key_in_target_scope( # NOSONAR(S3776) — sequential i from slayer.core.keys import ( AggregateKey, ArithmeticKey, + BetweenKey, ColumnKey, ColumnSqlKey, InKey, LiteralKey, + ScalarCallKey, ) if isinstance(value_key, ColumnSqlKey): @@ -6393,55 +7965,19 @@ def _render_filter_value_key_in_target_scope( # NOSONAR(S3776) — sequential i if isinstance(value_key, AggregateKey): # HAVING-route: render the aggregate against the target. # Reuse the synthesise helper with target_model as scope. - from slayer.core.keys import ( - AggregateKey as _AggKey, ColumnSqlKey as _ColSqlKey, - ) - # DEV-1501 (Codex round 9): mirror the projection-path - # rerooting (lines ~5685-5730) here. The routed AggregateKey - # carries args/kwargs still rooted at the cross-model path + # DEV-1501 (Codex round 9): the routed AggregateKey carries + # source / args / kwargs still rooted at the cross-model path # (``customers.regions.amount:last(customers.regions.opened_at)`` # arrives with ``args=(ColumnKey(path=("customers","regions"), - # leaf="opened_at"),)``). Inside the target CTE scope those - # refs must qualify under the local relation, not the - # host-rooted ``__``-path alias — same fix the projection - # path applies via ``_reroot_kwarg``. Without this, the - # ranked subquery's ``ORDER BY`` qualifies the time column - # under a non-existent alias inside the CTE. + # leaf="opened_at"),)``). Inside the target CTE scope every ref + # must qualify under the local relation, not the host-rooted + # ``__``-path alias — the SAME symmetric reroot the projection + # path applies (DEV-1707). Without it, the ranked subquery's + # ``ORDER BY`` qualifies the time column under a non-existent + # alias inside the CTE. cross_model_path = getattr(value_key.source, "path", ()) - - def _reroot_having(kval): - if isinstance(kval, ColumnKey) and kval.path == cross_model_path: - return ColumnKey(path=(), leaf=kval.leaf) - if isinstance(kval, _ColSqlKey) and kval.path == cross_model_path: - return _ColSqlKey( - path=(), model=kval.model, - column_name=kval.column_name, - ) - return kval - - if isinstance(value_key.source, ColumnKey): - local_source = ColumnKey(path=(), leaf=value_key.source.leaf) - elif isinstance(value_key.source, _ColSqlKey): - local_source = _ColSqlKey( - path=(), model=value_key.source.model, - column_name=value_key.source.column_name, - ) - else: - local_source = value_key.source - local_args = tuple( - _reroot_having(a) - if isinstance(a, (ColumnKey, _ColSqlKey)) else a - for a in value_key.args - ) - local_kwargs = tuple( - (k, _reroot_having(v)) for k, v in value_key.kwargs - ) - local_agg = _AggKey( - source=local_source, - agg=value_key.agg, - args=local_args, - kwargs=local_kwargs, - column_filter_key=value_key.column_filter_key, + local_agg = reroot_aggregate_key( + value_key, target_path=cross_model_path, ) from slayer.engine.planned import ValueSlot as _Slot tmp_slot = _Slot( @@ -6470,6 +8006,20 @@ def _reroot_having(kval): full_alias=having_full_alias, bundle=bundle, ) + # DEV-1708 Law 2: if the projected first/last materialised its + # crossing SOURCE value as a ``_val_`` column inside the ranked + # subquery, the HAVING aggregate must bind to that SAME alias — the + # outer ``MAX(CASE WHEN _last_rn = 1 THEN END)`` + # would otherwise reference a table bound only inside the subquery. + # DEV-1709: the alias map is keyed by the RESOLVED value text + # (qualified + typed inner CAST) so same-sql-different-type + # aggregates bind to their own materialisations. + if first_last_state is not None and synth.sql: + resolved_key = self._resolve_value_sql(synth) + if resolved_key in first_last_state.value_alias_by_sql: + synth = synth.model_copy(update={ + "sql": first_last_state.value_alias_by_sql[resolved_key], + }) # Thread the cross-model CTE's rn maps so the HAVING # aggregate uses the same ``_first_rn`` / ``_last_rn{suffix}`` # / ``_last_rn_fN`` column the CTE SELECT projects. @@ -6508,6 +8058,48 @@ def _reroot_having(kval): for op_key in value_key.operands ] return self._build_arith_or_cmp_ast(op=op, operands=rendered_operands) + if isinstance(value_key, ScalarCallKey): + # DEV-1708 (Codex): a routed filter wrapping a target ref in a scalar + # call (``abs(customers.deep_pop) > 5``) — render each arg in the + # target scope (a derived arg expands + pulls its join) and rebuild + # the call, mirroring the local filter-render path's ScalarCallKey + # branch (``like`` → ``exp.Like``; else ``func(NAME, *args)`` through + # the dialect rewrite). Without this the key falls through to the + # scalar fallback and emits its repr as a bogus string literal. + rendered_args = [ + self._render_filter_value_key_in_target_scope( + value_key=a, + target_relation=target_relation, + target_model=target_model, + planned_query=planned_query, + bundle=bundle, + first_last_state=first_last_state, + ) + for a in value_key.args + ] + if value_key.name == "like": + return exp.Like(this=rendered_args[0], expression=rendered_args[1]) + return self._finalize_scalar_call( + exp.func(value_key.name.upper(), *rendered_args), + ) + if isinstance(value_key, BetweenKey): + # DEV-1708: a routed ``date_range``-derived BETWEEN over a target + # (possibly crossing) column — render each operand in the target + # scope, mirroring the local filter path's ``exp.Between``. + def _render(k): + return self._render_filter_value_key_in_target_scope( + value_key=k, + target_relation=target_relation, + target_model=target_model, + planned_query=planned_query, + bundle=bundle, + first_last_state=first_last_state, + ) + return exp.Between( + this=_render(value_key.column), + low=_render(value_key.low), + high=_render(value_key.high), + ) if isinstance(value_key, InKey): # DEV-1475: cross-model IN filter — render the LHS column # rooted at the CTE's target relation (so a bare ``name`` on @@ -6614,6 +8206,7 @@ def _build_combined_order_by_sql( bare_order_slot_ids: Optional[Set[str]] = None, outer_composite_aliases: Optional[Dict[str, str]] = None, outer_composite_expressions: Optional[Dict[str, str]] = None, + hidden_cte_order_refs: Optional[Dict[str, str]] = None, ) -> Optional[str]: """Build the ORDER BY clause for the combined SELECT. @@ -6641,6 +8234,7 @@ def _build_combined_order_by_sql( bare_ids = bare_order_slot_ids or set() outer_aliases = outer_composite_aliases or {} outer_expressions = outer_composite_expressions or {} + hidden_cte_refs = hidden_cte_order_refs or {} parts: List[str] = [] for entry in planned_query.order: slot = slots_by_id.get(entry.slot_id) @@ -6655,6 +8249,7 @@ def _build_combined_order_by_sql( bare_ids=bare_ids, outer_aliases=outer_aliases, outer_expressions=outer_expressions, + hidden_cte_refs=hidden_cte_refs, ) if term is not None: parts.append(term) @@ -6673,6 +8268,7 @@ def _resolve_combined_order_term( bare_ids: Set[str], outer_aliases: Dict[str, str], outer_expressions: Optional[Dict[str, str]] = None, + hidden_cte_refs: Optional[Dict[str, str]] = None, ) -> Optional[str]: """Resolve one ``OrderEntry`` to its ``"alias" `` term. @@ -6687,13 +8283,23 @@ def _resolve_combined_order_term( rendered). """ direction = "ASC" if entry.direction == "asc" else "DESC" + # DEV-1712 / DEV-1733: a HIDDEN (order-only) aggregate that lives in its + # own CTE — cross-model (``_cm_``) or windowed (``_wm_``) — is trimmed + # from the combined projection, so the bare alias no longer names a + # projected column. Reference the CTE-qualified column instead. Checked + # BEFORE the ``cma_slot_ids`` gate because a windowed slot is not a + # cross-model slot and would otherwise fall through to the bare-alias + # branch below and dangle. + hidden_ref = (hidden_cte_refs or {}).get(entry.slot_id) + if hidden_ref is not None: + return f'{hidden_ref} {direction}' if entry.slot_id in cma_slot_ids: alias = cm_alias_for_plan.get(entry.slot_id) if alias is None: return None - return f'"{alias}" {direction}' + return f'{self._quote_ident(alias)} {direction}' if entry.slot_id in outer_aliases: - return f'"{outer_aliases[entry.slot_id]}" {direction}' + return f'{self._quote_ident(outer_aliases[entry.slot_id])} {direction}' if outer_expressions and entry.slot_id in outer_expressions: return f'{outer_expressions[entry.slot_id]} {direction}' full_alias = self._full_alias_for_slot( @@ -6702,8 +8308,8 @@ def _resolve_combined_order_term( alias_index={}, ) if entry.slot_id in bare_ids: - return f'"{full_alias}" {direction}' - return f'_base."{full_alias}" {direction}' + return f'{self._quote_ident(full_alias)} {direction}' + return f'_base.{self._quote_ident(full_alias)} {direction}' def _full_alias_for_slot( self, @@ -6719,16 +8325,24 @@ def _full_alias_for_slot( ``_pick_alias_for_planned_slot`` for C13 multi-alias slots) or the planner's canonical ``declared_name``. - DEV-1450 stage 7b.12: joined ROW slots (``ColumnKey.path != ()`` - / ``TimeTruncKey.column.path != ()``) emit the FULL dotted + DEV-1450 stage 7b.12: joined ROW slots emit the FULL dotted result-key form (``orders.customers.region_id``), preserving the result-key contract (P10). The planner's flat ``declared_name`` is the DEV-1449 / C4 downstream-stage binding name and remains untouched on the slot for stage-2 references; only the public SQL alias differs. + + DEV-1713 (D3 / DEV-1495 bug 1): the ROW branch covers all three + row key shapes — ``ColumnKey``, ``ColumnSqlKey`` (a joined DERIVED + column, which previously fell through to the flat ``declared_name`` + and surfaced as ``orders.customers__revenue``), and ``TimeTruncKey`` + over either. All route through :func:`slayer.sql.naming.result_key`, + the single owner of the dotted form; response_meta mirrors this + via the same builder so the two producers cannot drift. """ from slayer.core.keys import ( ColumnKey, + ColumnSqlKey, Phase, TimeTruncKey, column_leaf, @@ -6741,51 +8355,46 @@ def _full_alias_for_slot( leaf: Optional[str] = None if isinstance(key, ColumnKey): path, leaf = key.path, key.leaf + elif isinstance(key, ColumnSqlKey): + # DEV-1713: a joined derived column's leaf is its column_name. + path, leaf = key.path, key.column_name elif isinstance(key, TimeTruncKey): # DEV-1450 #4a: a derived TD's leaf is its column_name, so the # public result-key shape matches the base-column TD. path, leaf = column_path(key.column), column_leaf(key.column) if path and leaf is not None: - return f"{source_relation}." + ".".join(path) + f".{leaf}" - # Local + AGGREGATE / POST slots: existing alias selection. + return result_key( + source_relation=source_relation, path=path, leaf=leaf, + ) + # Local + AGGREGATE / POST slots: existing alias selection. The alias + # may embed hop dots (a cross-model measure alias such as + # ``customers.revenue_sum``), so use the canonical-alias builder. if slot.public_aliases: alias = self._pick_alias_for_planned_slot( slot=slot, alias_index=alias_index, ) else: alias = slot.declared_name - return f"{source_relation}.{alias}" + return result_key_from_alias(source_relation=source_relation, alias=alias) - def _collect_joined_paths_for_base( # NOSONAR(S3776) — sequential per-slot dispatch over ROW (ColumnKey / TimeTruncKey path) vs AGGREGATE (top-level AggregateKey first/last + composite first/last leaves; ColumnKey args qualify directly, ColumnSqlKey args expand-and-scan through the derived sql). Each branch is the per-slot join-discovery contract; extracting per-shape helpers would scatter the contract. + def _collect_joined_paths_for_base( self, *, base_render_order: List[str], slots_by_id: Dict[str, Any], - source_model=None, - source_relation: Optional[str] = None, - bundle=None, ) -> List[Tuple[str, ...]]: - """Walk ROW slots in render order to collect unique joined paths. - - Only paths needed for projection / GROUP BY surface here. Cross- - model aggregate slots are NEVER walked — their joins live in - ``CrossModelAggregatePlan.join_chain`` and are rendered inside - the per-plan ``_cm_*`` CTE. - - Local ``first`` / ``last`` AGGREGATE slots additionally contribute - any joined path named by an explicit ranking-time arg - (``amount:last(stores.opened_at)``) — the ranked subquery's - ``ORDER BY`` references that column, so the join must be in scope. - Derived (``ColumnSqlKey``) time args (``amount:last(net_amount_date)`` - where ``net_amount_date.sql`` references ``customers.signed_up_at``) - are expanded through ``_expand_derived_column_sql`` and then scanned - with ``_joined_paths_in_sql`` so their crossed joins also land in the - FROM — requires ``source_model`` / ``source_relation`` / ``bundle`` - (the existing ROW-derived expand path uses the same triple). + """Walk ROW slots in render order to collect unique joined DIMENSION + paths needed for projection / GROUP BY. + + Cross-model aggregate slots are NEVER walked — their joins live in + ``CrossModelAggregatePlan.join_chain`` and render inside the per-plan + ``_cm_*`` CTE. Local ``first`` / ``last`` explicit-time-arg joins are no + longer collected here either: DEV-1710 Stage 6 moved that discovery into + ``_resolve_agg_inputs_via_scope`` (sub-pass 4), where anchoring the arg + through the host ``ScopeFrame`` registers its crossed join as a Law-1 + side effect (bare, derived, and multi-hop args alike). """ - from slayer.core.keys import ( - AggregateKey, ColumnKey, ColumnSqlKey, Phase, TimeTruncKey, - ) + from slayer.core.keys import ColumnKey, Phase, TimeTruncKey seen: set = set() ordered: List[Tuple[str, ...]] = [] @@ -6806,57 +8415,6 @@ def _add(path: Tuple[str, ...]) -> None: _add(key.path) elif isinstance(key, TimeTruncKey): _add(key.column.path) - elif slot.phase == Phase.AGGREGATE: - # DEV-1501 (Codex round 5): walk top-level AggregateKey - # slots AND first/last leaves inside composite slots - # (ArithmeticKey / ScalarCallKey). A composite operand - # ``amount:last(stores.opened_at) + 1`` orders the ranked - # subquery by ``stores.opened_at`` and requires the - # ``stores`` join to be in scope. - fl_keys: list = [] - if ( - isinstance(key, AggregateKey) - and key.agg in ("first", "last") - and not getattr(key.source, "path", ()) - ): - fl_keys = [key] - elif not isinstance(key, AggregateKey): - fl_keys = _iter_first_last_leaves(key) - for fl in fl_keys: - for a in fl.args: - if isinstance(a, ColumnKey): - _add(a.path) - elif ( - # DEV-1501 (Codex round 8): a derived time arg - # (``amount:last(net_amount_date)``) is expanded - # against the source relation inside the ranked - # subquery's ``ORDER BY``; any joined ref the - # expansion introduces (``customers.signed_up_at``) - # must pull its join into ``_base``. Without - # ``bundle`` (defensive entry point), skip — the - # ranked subquery falls back to verbatim emit and - # the missing join would surface as broken SQL at - # runtime, but no upstream caller hits this path - # without a bundle. - isinstance(a, ColumnSqlKey) - and not a.path - and source_model is not None - and source_relation is not None - and bundle is not None - ): - expanded = self._expand_derived_column_sql( - source_model=source_model, - source_relation=source_relation, - column_name=a.column_name, - bundle=bundle, - ) - for p in self._joined_paths_in_sql( - sql_expr=self._parse(expanded), - source_relation=source_relation, - source_model=source_model, - bundle=bundle, - ): - _add(p) return ordered def _build_from_and_joins( @@ -6914,13 +8472,18 @@ def _build_from_and_joins( if next_alias not in emitted_aliases: join_on_parts = [] for src_col, tgt_col in join_def.join_pairs: + # DEV-1645: the join keys are physical DB columns — + # quote them when mixed-case (``merchantId``) via + # ``_to_ident`` so a case-folding backend resolves them; + # the table qualifiers are SLayer-internal aliases + # (reserved names quote at emit via RESERVED_KEYWORDS). join_on_parts.append(exp.EQ( this=exp.Column( - this=exp.to_identifier(src_col), + this=self._to_ident(src_col), table=exp.to_identifier(current_alias), ), expression=exp.Column( - this=exp.to_identifier(tgt_col), + this=self._to_ident(tgt_col), table=exp.to_identifier(next_alias), ), )) @@ -6933,7 +8496,9 @@ def _build_from_and_joins( alias=exp.to_identifier(next_alias), ) else: - join_expr = exp.to_table(target_table, alias=next_alias) + # DEV-1686 reserved-word alias + DEV-1645 mixed-case + # physical-name quoting: ``FROM "Order" AS "order"``. + join_expr = self._to_table(target_table, alias=next_alias) on_expr = ( exp.and_(*join_on_parts) if len(join_on_parts) > 1 @@ -7063,7 +8628,7 @@ def _render_window_transform_sql( f"op={key.op!r}, input_key={key.input!r}.", ) input_alias = available_alias_by_slot_id[input_sid] - measure = f'"{input_alias}"' + measure = self._quote_ident(input_alias) # Resolve time-key alias (None for rank-family without time). time_alias: Optional[str] = None @@ -7075,7 +8640,7 @@ def _render_window_transform_sql( f"slot id={slot.id!r}, op={key.op!r}, " f"time_key={key.time_key!r}.", ) - time_alias = f'"{available_alias_by_slot_id[tk_sid]}"' + time_alias = self._quote_ident(available_alias_by_slot_id[tk_sid]) # Resolve partition aliases. Explicit partition_keys take # precedence; otherwise auto-partition by query dimension slots @@ -7115,7 +8680,7 @@ def _render_window_transform_sql( partition_aliases.append(alias) partition_clause = ( - _SQL_PARTITION_BY + ", ".join(f'"{a}"' for a in partition_aliases) + _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_aliases) if partition_aliases else "" ) @@ -7309,7 +8874,7 @@ def recurse(k) -> exp.Expression: ] if key.name == "like": return exp.Like(this=args[0], expression=args[1]) - return exp.func(key.name.upper(), *args) + return self._finalize_scalar_call(exp.func(key.name.upper(), *args)) if isinstance(key, BetweenKey): return exp.Between( @@ -7442,7 +9007,7 @@ def _apply_order_limit_to_planned_sql_string( direction = ( "ASC" if order_entry.direction == "asc" else "DESC" ) - order_parts.append(f'"{alias}" {direction}') + order_parts.append(f'{self._quote_ident(alias)} {direction}') if order_parts: sql += "\nORDER BY " + ", ".join(order_parts) if planned_query.limit is not None: @@ -7465,74 +9030,131 @@ def _build_shifted_cte_where_parts( source_relation: str, source_model, bundle, - ) -> List[str]: + ) -> Tuple[List[str], List[Tuple[str, ...]]]: """Build the WHERE clauses for the shifted CTE that re-aggregates - the source relation. - - 7b.3c invariant: ``BetweenKey`` filters (those derived from - ``TimeDimension.date_range``) MUST be omitted from the shifted - inner CTE so the earliest visible bucket can still carry a - non-null shifted value. Other ROW-phase filters - (e.g. ``status = 'active'``) are propagated unchanged so the - shifted aggregation runs over the same row population. + the source relation, plus the join paths those clauses cross. + + 7b.3c invariant, generalised by DEV-1732: a FRAME BOUND must be omitted + from the shifted inner CTE so the earliest visible bucket can still + carry a non-null shifted value. That covers the ``BetweenKey`` a + ``date_range`` produces AND the explicit relational spelling of the same + intent (``created_at >= '2024-01-01'``), which used to be propagated — + so the two spellings gave different numbers. A filter that is only + PARTLY a frame bound propagates as its residual population predicate. + + Other ROW-phase filters (e.g. ``status = 'active'``) are propagated + unchanged so the shifted aggregation runs over the same row population. AGGREGATE / POST phase filters never apply to the shifted CTE (they're outer-projection concerns). + + DEV-1711: a ROW filter referencing a JOINED column + (``stores.name = 'North'``) is now supported — the shifted CTE is a + real ``ScopeFrame`` whose FROM pulls the join, so the guard that used + to raise on joined refs is gone. The returned ``crossed_paths`` list is + registered into the caller's shifted scope so the LEFT JOIN the filter + needs is emitted. Filters over the same join set the base already + applies keep population parity between ``_base`` and the shifted CTE. """ - from slayer.core.keys import BetweenKey, Phase - - def _guard_no_joined_refs(rendered_part: exp.Expression, *, fid) -> None: - # The shifted CTE re-aggregates the bare source (no joins), so a - # ROW filter referencing a joined column cannot be applied here. - # This combination (time_shift + joined-column filter) is deferred - # — raise loudly rather than emit SQL that references an unjoined - # alias. - for c in rendered_part.find_all(exp.Column): - tbl = c.args.get("table") - if tbl is not None and tbl.name not in ( - source_relation, source_model.name, - ): - raise NotImplementedError( - f"DEV-1450: time_shift combined with a ROW filter on " - f"a joined column ({tbl.name}.{c.name}) is not yet " - f"supported (the shifted CTE carries no joins). " - f"filter id={fid!r}." - ) + from slayer.core.keys import Phase out: List[str] = [] + crossed_paths: List[Tuple[str, ...]] = [] + # DEV-1732: the frame-bound column set is computed once by the planner + # and carried on the plan, so this path and the windowed ``_src`` path + # cannot drift apart. + time_cols = frozenset(planned_query.frame_bound_columns) for fp in planned_query.filters_by_phase: if fp.phase != Phase.ROW: continue - if fp.expression is not None: - if isinstance(fp.expression.value_key, BetweenKey): - # date_range filter — omit from inner shifted CTE. - continue - rendered = self._render_value_key_for_filter( - key=fp.expression.value_key, - source_relation=source_relation, - source_model=source_model, - bundle=bundle, - ) - if isinstance(rendered, (exp.And, exp.Or)): - rendered = exp.Paren(this=rendered) - _guard_no_joined_refs(rendered, fid=fp.id) - out.append(rendered.sql(dialect=self.dialect)) - elif fp.text is not None: - qualified = self._render_model_filter_sql( - sql=fp.text, - columns=fp.text_columns, - source_model=source_model, - source_relation=source_relation, - bundle=bundle, - ) - _guard_no_joined_refs(self._parse(qualified), fid=fp.id) - out.append(qualified) - return out + rendered = self._shifted_where_part( + fp=fp, source_relation=source_relation, + source_model=source_model, bundle=bundle, + time_columns=time_cols, + ) + if rendered is None: + continue + part, paths = rendered + out.append(part) + for p in paths: + if p not in crossed_paths: + crossed_paths.append(p) + return out, crossed_paths + + def _shifted_where_part( + self, *, fp, source_relation: str, source_model, bundle, + time_columns: "AbstractSet[Any]", + ) -> "Optional[Tuple[str, List[Tuple[str, ...]]]]": + """Render one ROW-phase filter for the shifted CTE, returning its SQL + plus the join paths it crosses — or ``None`` to omit it entirely. + + A filter that is wholly a FRAME BOUND on one of ``time_columns`` is + omitted; one that is partly a frame bound renders as its residual + population predicate (DEV-1732). This subsumes the old + ``isinstance(..., BetweenKey)`` special case: a ``date_range``'s + ``BetweenKey`` column is always a query time dimension's raw column, so + ``strip_frame_bounds`` returns ``None`` for it — same behaviour, one + rule. + + Mode-A ``text`` filters are exempt from the analysis and always + propagate (a model filter defines which rows EXIST, not the frame). + + ``time_columns`` is REQUIRED, deliberately (Codex): ``strip_frame_bounds`` + returns its input unchanged for an empty set, so a default would let a + future caller silently start rendering every ``date_range`` into the + shifted CTE — the exact 7b.3c regression this method exists to prevent. + + The join paths are collected per carrier kind (CodeRabbit): a TYPED + filter is scanned STRUCTURALLY on its already-rendered AST via + ``_joined_paths_in_sql`` — the expression is fully qualified/expanded, + so its crossed joins are visible directly and there is no text + round-trip that could silently swallow a parse failure. A Mode-A + ``text`` filter has only its string form, so it keeps the + ``_filter_join_paths`` dual raw + inline-expanded scan (the DEV-1494 + contract that surfaces a derived ref's expansion joins). + + Note the scan runs on the RESIDUAL, so the shifted CTE's join set + follows what it actually renders. + """ + if fp.expression is not None: + residual = strip_frame_bounds( + key=fp.expression.value_key, time_columns=time_columns, + ) + if residual is None: + return None # wholly a frame bound — omit from the shifted CTE. + rendered = self._render_value_key_for_filter( + key=residual, + source_relation=source_relation, + source_model=source_model, + bundle=bundle, + ) + if isinstance(rendered, (exp.And, exp.Or)): + rendered = exp.Paren(this=rendered) + paths = self._joined_paths_in_sql( + sql_expr=rendered, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ) + return rendered.sql(dialect=self.dialect), paths + if fp.text is not None: + qualified = self._render_model_filter_sql( + sql=fp.text, + columns=fp.text_columns, + source_model=source_model, + source_relation=source_relation, + bundle=bundle, + ) + paths = self._filter_join_paths( + sql=qualified, source_relation=source_relation, + source_model=source_model, bundle=bundle, + ) + return qualified, paths + return None - def _emit_time_shift_ctes_for_planned( + def _emit_time_shift_ctes_for_planned( # NOSONAR(S3776) — single conceptual unit for one time_shift slot: partition/time resolution through the shifted ScopeFrame + shifted-CTE body assembly + collision-safe CTE naming (cte_allocator) + sjoin grain join-back, all sharing tightly-coupled per-slot state (time_alias / input_alias / partition_specs / shifted_cte_name / carry aliases). Splitting forces that cross-cutting state through many-argument helpers without simplifying anything — same shape as the sibling _render_cross_model_cte's suppression. self, *, slot, ctes: list, + cte_allocator: AliasAllocator, slots_by_id: Dict[str, Any], slot_id_by_key: Dict[Any, str], available_alias_by_slot_id: Dict[str, str], @@ -7540,6 +9162,7 @@ def _emit_time_shift_ctes_for_planned( source_model, source_relation: str, shifted_where_parts: List[str], + shifted_where_join_paths: List[Tuple[str, ...]], planned_query, bundle, ) -> None: @@ -7560,6 +9183,19 @@ def _emit_time_shift_ctes_for_planned( * **partition_keys**: DEV-1450 C6 — explicit ``partition_by`` on ``change`` / ``time_shift`` threads through as additional equality keys in the LEFT JOIN (not just query dimensions). + + DEV-1711 (Stage 7): the shifted CTE is a ``ScopeFrame`` (Laws 1 & 2). + Every partition key and the shift-axis time expression enters through + ``scope.resolve`` — anchoring the ref AND registering the join it + crosses in one call — so the shifted CTE's FROM (built from + ``scope.join_paths``) pulls exactly the LEFT JOINs the shifted + projection references. This makes CROSS-MODEL partitions (``stores. + name``), DERIVED dim partitions (local ``upper(status)`` or joined + ``stores.tier``), SECONDARY time-dimension partitions, and joined-column + ROW filters all work, and removes the joinless-CTE guards. The sjoin + grain join-back (time axis + every partition) is dialect-aware + null-safe (Codex F2) so NULL dim / NULL time-bucket groups keep their + shifted value instead of silently dropping. """ from slayer.core.enums import TimeGranularity from slayer.core.keys import ( @@ -7639,20 +9275,60 @@ def _emit_time_shift_ctes_for_planned( ) input_alias = available_alias_by_slot_id[input_sid] + # DEV-1711 (Law 1): the shifted CTE is a ScopeFrame. Every partition + # key and the shift-axis time expression enters through ``resolve``, + # which anchors the ref AND registers the join it crosses. The FROM + # (built below from ``shifted_scope.join_paths``) then pulls exactly + # those LEFT JOINs — a cross-model / derived / secondary-time partition + # can never reference an unjoined table. The scope shares the + # generation-wide allocator so any ``_val_`` names stay unique + # across the base and every CTE. + shifted_allocator = self._gen_allocator or self._new_allocator() + shifted_scope = ScopeFrame( + scope_id=shifted_allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=shifted_allocator, + ) + # 3. partition_keys (DEV-1450 C6) + auto-include query dimensions. # # Legacy auto-joins on EVERY query dimension regardless of # partition_by (``_generate_with_computed:1559``). Without this, # ``time_shift(amount:sum, periods=-1)`` with ``status`` in # ``dimensions`` would broadcast the prior-period total across - # every status value. The typed pipeline mirrors this: explicit - # ``partition_keys`` may add MORE columns (DEV-1450 C6), but - # query dimensions are always included. + # every status value. The typed pipeline mirrors this AND extends it + # (DEV-1711): the sjoin grain is EVERY projected dimension — joined + # ``ColumnKey``, derived ``ColumnSqlKey``, and SECONDARY ``TimeTruncKey`` + # (a second time dim, distinct from the shift axis) — plus any explicit + # ``partition_keys`` (C6). The shift axis itself is the time-join + # column, excluded by slot id. from slayer.core.keys import Phase as _Phase partition_specs: list[tuple[str, str, exp.Expression]] = [] - # entries: (slot_id, full_alias, raw_column_expr_for_group_by) + # entries: (slot_id, base_alias, resolved_expr_for_select_and_group_by) seen_partition_sids: set = set() + def _resolve_partition_expr(pk_obj) -> exp.Expression: + # A SECONDARY time dimension renders as DATE_TRUNC over its resolved + # (possibly joined / derived) raw column; a plain / derived column + # renders as its resolved expression. ``resolve`` registers the + # crossed join in both cases (Law 1). + if isinstance(pk_obj, TimeTruncKey): + raw = shifted_scope.resolve(pk_obj.column) + return self._build_date_trunc( + col_expr=raw, + granularity=TimeGranularity(pk_obj.granularity), + ) + if isinstance(pk_obj, (ColumnKey, ColumnSqlKey)): + return shifted_scope.resolve(pk_obj) + raise NotImplementedError( + f"time_shift partition on {type(pk_obj).__name__} is not " + f"supported (only column / derived-column / time-dimension " + f"partitions render in the shifted CTE). slot id={slot.id!r}.", + ) + def _add_partition(pk_obj, *, where: str) -> None: pk_sid = slot_id_by_key.get(pk_obj) if pk_sid is None or pk_sid not in available_alias_by_slot_id: @@ -7660,44 +9336,47 @@ def _add_partition(pk_obj, *, where: str) -> None: f"time_shift {where} not materialised: " f"slot id={slot.id!r}, key={pk_obj!r}.", ) - if pk_sid in seen_partition_sids: + # The shift axis is the time-join column, never a partition pair. + if pk_sid == time_sid or pk_sid in seen_partition_sids: return pk_alias = available_alias_by_slot_id[pk_sid] - if isinstance(pk_obj, ColumnKey): - if pk_obj.path != (): - raise NotImplementedError( - f"DEV-1450 stage 7b.12: cross-model partition " - f"(path={pk_obj.path!r}) deferred to the " - f"cross-model slice (slot id={slot.id!r}).", - ) - col_expr = self._dim_column_expr_from_planned( - source_model=source_model, - source_relation=source_relation, - leaf=pk_obj.leaf, - ) - else: - raise NotImplementedError( - f"DEV-1450 stage 7b.11: partition on " - f"{type(pk_obj).__name__} not supported (only " - f"ColumnKey leaves render in the shifted CTE).", - ) - partition_specs.append((pk_sid, pk_alias, col_expr)) + partition_specs.append((pk_sid, pk_alias, _resolve_partition_expr(pk_obj))) seen_partition_sids.add(pk_sid) - # Auto-include query-dimension ColumnKey row slots (NOT TimeTruncKey; - # the time-key is already the time-join axis). + # Auto-include EVERY projected row dimension (column, derived column, or + # secondary time dimension); the shift axis is skipped by slot id above. for sid in planned_query.projection: dim_slot = slots_by_id.get(sid) if dim_slot is None or dim_slot.phase != _Phase.ROW: continue - if not isinstance(dim_slot.key, ColumnKey): + if not isinstance(dim_slot.key, (ColumnKey, ColumnSqlKey, TimeTruncKey)): continue _add_partition(dim_slot.key, where="query dimension") - # Explicit partition_keys (DEV-1450 C6) may add more. + # Explicit partition_keys (DEV-1450 C6) may add more (deduped by slot id + # against the auto-included dims — see the DEV-1711 dedup test). for pk in sorted(key.partition_keys, key=lambda k: repr(k)): _add_partition(pk, where="partition_key") + # DEV-1711 defensive completeness: a LOCAL aggregate whose source / + # column-filter / kwargs cross a join is isolated upstream (Stage 5) and + # would have raised 7b.15e before reaching a time_shift CTE, so these + # registrations are provably no-ops today — but routing them through the + # scope keeps Law 1 total (no render path skips join discovery). + if isinstance(inner_key, AggregateKey): + if isinstance(inner_key.source, ColumnSqlKey): + shifted_scope.resolve(inner_key.source) + for _kname, _kval in inner_key.kwargs: + if isinstance(_kval, (ColumnKey, ColumnSqlKey)): + shifted_scope.resolve(_kval) + if inner_key.column_filter_key is not None: + for _p in self._filter_join_paths( + sql=inner_key.column_filter_key.canonical_sql, + source_relation=source_relation, + source_model=source_model, bundle=bundle, + ): + shifted_scope.join_paths.add(_p) + # Build the shifted time-column expression. Calendar offset is # ``-periods`` units in the SHIFT granularity (periods=-1 -> +1 unit). # The shift granularity is the explicit 3rd arg @@ -7712,15 +9391,12 @@ def _add_partition(pk_obj, *, where: str) -> None: str(shift_gran_raw) if shift_gran_raw is not None else time_key.granularity ) - # DEV-1450 #4a: a derived (ColumnSqlKey) time column yields its - # EXPANDED expression here; the calendar offset and DATE_TRUNC apply - # over that expression exactly as over a bare column. - raw_time_col_expr = self._raw_time_col_expr_for_planned( - time_column=time_key.column, - source_model=source_model, - source_relation=source_relation, - bundle=bundle, - ) + # DEV-1450 #4a / DEV-1711: the shift-axis raw time expression resolves + # through the SAME scope (Law 1) — a derived (ColumnSqlKey) time column + # yields its EXPANDED expression, and a JOINED time axis (``stores. + # opened_at``) registers its join so the shifted FROM binds it. The + # calendar offset and DATE_TRUNC then apply over that expression. + raw_time_col_expr = shifted_scope.resolve(time_key.column) shifted_raw_expr = self._build_time_offset_expr( col_expr=raw_time_col_expr, offset=-periods, @@ -7738,14 +9414,14 @@ def _add_partition(pk_obj, *, where: str) -> None: # Projected: time-trunc shifted under the base time alias. shifted_trunc_sql = shifted_trunc_expr.sql(dialect=self.dialect) shifted_select_parts.append( - f'{shifted_trunc_sql} AS "{time_alias}"', + f'{shifted_trunc_sql} AS {self._quote_ident(time_alias)}', ) shifted_group_by.append(shifted_trunc_sql) # partition_keys: SELECT + GROUP BY under their base aliases. for _, pk_alias, pk_expr in partition_specs: pk_sql = pk_expr.sql(dialect=self.dialect) - shifted_select_parts.append(f'{pk_sql} AS "{pk_alias}"') + shifted_select_parts.append(f'{pk_sql} AS {self._quote_ident(pk_alias)}') shifted_group_by.append(pk_sql) # Aggregate: re-emit the AggregateKey using the same synth / @@ -7772,29 +9448,49 @@ def _add_partition(pk_obj, *, where: str) -> None: agg_expr, _ = self._build_agg(synth) agg_expr = _wrap_cast_for_type(agg_expr, inner_slot.type) shifted_select_parts.append( - f'{agg_expr.sql(dialect=self.dialect)} AS "{input_alias}"', + f'{agg_expr.sql(dialect=self.dialect)} AS {self._quote_ident(input_alias)}', ) else: - # Row-level column input (not aggregated). Pass-through. - col_expr = self._dim_column_expr_from_planned( + # Row-level column input (not aggregated). Resolve through the scope + # so a joined / derived input registers its join and anchors + # correctly (Law 1), same as every other ref in this CTE. + col_expr = shifted_scope.resolve(inner_key) + shifted_select_parts.append( + f'{col_expr.sql(dialect=self.dialect)} AS {self._quote_ident(input_alias)}', + ) + shifted_group_by.append(col_expr.sql(dialect=self.dialect)) + + # DEV-1711: register the join paths the shifted WHERE filters cross + # (computed once by ``_build_shifted_cte_where_parts``) so a joined-column + # ROW filter (``stores.name = 'North'``) pulls its LEFT JOIN into this + # CTE. Then build the FROM from the scope's full registered set — a + # crossed join can never be forgotten because discovery is a side effect + # of resolving each ref above. + for _p in shifted_where_join_paths: + shifted_scope.join_paths.add(_p) + shifted_join_paths = shifted_scope.join_paths.as_list() + if shifted_join_paths: + from_clause, shifted_joins = self._build_from_and_joins( source_model=source_model, source_relation=source_relation, - leaf=inner_key.leaf, + joined_paths=shifted_join_paths, + bundle=bundle, ) - shifted_select_parts.append( - f'{col_expr.sql(dialect=self.dialect)} AS "{input_alias}"', + else: + from_clause = self._build_from_clause_from_planned( + source_model=source_model, source_relation=source_relation, ) - shifted_group_by.append(col_expr.sql(dialect=self.dialect)) + shifted_joins = [] - from_clause = self._build_from_clause_from_planned( - source_model=source_model, source_relation=source_relation, - ) - from_sql = from_clause.sql(dialect=self.dialect) + from_parts = [f"FROM {from_clause.sql(dialect=self.dialect)}"] + for join_expr, on_expr, join_type in shifted_joins: + from_parts.append( + f"{join_type} JOIN {join_expr.sql(dialect=self.dialect)} " + f"ON {on_expr.sql(dialect=self.dialect)}" + ) - shifted_sql_parts = [ - "SELECT\n " + ",\n ".join(shifted_select_parts), - f"FROM {from_sql}", - ] + shifted_sql_parts = [_SQL_SELECT_HEAD + ",\n ".join(shifted_select_parts)] + shifted_sql_parts.extend(from_parts) if shifted_where_parts: shifted_sql_parts.append( "WHERE " + _SQL_AND_JOINER.join(shifted_where_parts), @@ -7810,10 +9506,20 @@ def _add_partition(pk_obj, *, where: str) -> None: # slot with multiple ``public_aliases``; the sjoin CTE projects # the shifted measure under EACH alias so the outer SELECT # carries both. - slot_aliases: List[str] = list(slot.public_aliases) or [slot.declared_name] + # DEV-1692: a HIDDEN inner time_shift slot's declared_name + # (``_time_shift_inner``) is NOT unique across sibling shifts with + # different offsets — two would project + resolve downstream under the + # same column, silently collapsing ``growth_2m`` onto ``growth_1m``'s + # shift. Allocate a unique internal alias for the hidden case; USER + # aliases (public_aliases, already unique) are left untouched. + if slot.public_aliases: + slot_aliases: List[str] = list(slot.public_aliases) + else: + slot_aliases = [cte_allocator.allocate_cte(slot.declared_name)] cte_name_alias = slot_aliases[0] - shifted_cte_name = f"shifted_{cte_name_alias}" - sjoin_cte_name = f"sjoin_{cte_name_alias}" + # DEV-1692: allocate collision-free CTE names too. + shifted_cte_name = cte_allocator.allocate_cte(f"shifted_{cte_name_alias}") + sjoin_cte_name = cte_allocator.allocate_cte(f"sjoin_{cte_name_alias}") ctes.append((shifted_cte_name, shifted_sql)) @@ -7826,24 +9532,44 @@ def _add_partition(pk_obj, *, where: str) -> None: a for aliases in aliases_by_slot_id.values() for a in aliases ) sjoin_select_parts = [ - f'{prev_cte}."{a}"' for a in carry_aliases_sorted + f'{prev_cte}.{self._quote_ident(a)}' for a in carry_aliases_sorted ] slot_full_aliases: List[str] = [] for slot_alias in slot_aliases: full_slot_alias = f"{source_relation}.{slot_alias}" slot_full_aliases.append(full_slot_alias) sjoin_select_parts.append( - f'{shifted_cte_name}."{input_alias}" AS "{full_slot_alias}"', + f'{shifted_cte_name}.{self._quote_ident(input_alias)} AS {self._quote_ident(full_slot_alias)}', ) - # JOIN conditions: time equality + every partition equality. - join_conds = [ - f'{prev_cte}."{time_alias}" = {shifted_cte_name}."{time_alias}"', - ] + # JOIN conditions: time equality + every partition equality, all + # dialect-aware NULL-SAFE (DEV-1711 / Codex F2). The sjoin is a grain + # join-back — a NULL dimension value (e.g. a LEFT-joined ``stores.name`` + # with no matching store) or a NULL time bucket must match its own group + # instead of silently dropping to a NULL shifted value under plain ``=``. + # + # The predicate is built from AST nodes DIRECTLY — not via + # ``_null_safe_join_pair_sql``'s string round-trip — because a dotted + # public alias (``orders.created_at``) re-parses on BigQuery/T-SQL as a + # multi-part reference and the DEV-1713 alias mangling then corrupts it + # (``base.`orders.created_at``` → ``base___orders`.`created_at```). The + # alias as a single ``quoted=True`` identifier matches the SELECT parts' + # ``_quote_ident`` output byte-for-byte on every dialect and survives the + # post-generation mangling intact. + def _grain_eq(a: str) -> str: + left = exp.Column( + this=exp.to_identifier(a, quoted=True), + table=exp.to_identifier(prev_cte), + ) + right = exp.Column( + this=exp.to_identifier(a, quoted=True), + table=exp.to_identifier(shifted_cte_name), + ) + return self._dialect.build_null_safe_eq(left, right).sql(dialect=self.dialect) + + join_conds = [_grain_eq(time_alias)] for _, pk_alias, _ in partition_specs: - join_conds.append( - f'{prev_cte}."{pk_alias}" = {shifted_cte_name}."{pk_alias}"', - ) + join_conds.append(_grain_eq(pk_alias)) sjoin_sql = ( "SELECT " + ", ".join(sjoin_select_parts) @@ -7861,11 +9587,12 @@ def _add_partition(pk_obj, *, where: str) -> None: # ``available_alias_by_slot_id`` is "pick one" — first alias wins. available_alias_by_slot_id.setdefault(slot.id, slot_full_aliases[0]) - def _emit_consecutive_periods_ctes_for_planned( + def _emit_consecutive_periods_ctes_for_planned( # NOSONAR(S3776) — one cohesive per-slot consecutive_periods emission: predicate-shape decision, unique hidden alias plus collision-safe reset and value CTE names, the reset-group window layer, then the count-within-group window layer. Each block shares the slot registry and alias maps and cte_allocator; extracting helpers would scatter that contract without simplifying it. self, *, slot, ctes: list, + cte_allocator: AliasAllocator, slots_by_id: Dict[str, Any], slot_id_by_key: Dict[Any, str], available_alias_by_slot_id: Dict[str, str], @@ -7935,7 +9662,7 @@ def _emit_consecutive_periods_ctes_for_planned( ) input_alias = available_alias_by_slot_id[input_sid] predicate_sql = ( - f'"{input_alias}" IS NOT NULL AND "{input_alias}" <> 0' + f'{self._quote_ident(input_alias)} IS NOT NULL AND {self._quote_ident(input_alias)} <> 0' ) predicate_is_boolean = False elif isinstance(inner_key, ArithmeticKey): @@ -7979,11 +9706,16 @@ def _emit_consecutive_periods_ctes_for_planned( if alias is not None: partition_aliases.append(alias) - slot_alias = ( - slot.public_aliases[0] - if slot.public_aliases - else slot.declared_name - ) + # DEV-1692: a HIDDEN inner consecutive_periods slot's declared_name + # (``_consecutive_periods_inner``) is NOT unique across sibling slots — + # two would collide on ``full_slot_alias`` / ``cp_reset_alias`` and + # collapse downstream, the same failure mode fixed for time_shift. + # Allocate a unique internal alias for the hidden case; USER aliases + # (already unique) are left untouched. + if slot.public_aliases: + slot_alias = slot.public_aliases[0] + else: + slot_alias = cte_allocator.allocate_cte(slot.declared_name) full_slot_alias = f"{source_relation}.{slot_alias}" cp_reset_alias = f"_cp_reset_{full_slot_alias}" @@ -7992,24 +9724,24 @@ def _emit_consecutive_periods_ctes_for_planned( carry_aliases_sorted = sorted( a for aliases in aliases_by_slot_id.values() for a in aliases ) - carry_select = ",\n ".join(f'"{a}"' for a in carry_aliases_sorted) + carry_select = ",\n ".join(self._quote_ident(a) for a in carry_aliases_sorted) partition_clause = ( - _SQL_PARTITION_BY + ", ".join(f'"{a}"' for a in partition_aliases) + _SQL_PARTITION_BY + ", ".join(self._quote_ident(a) for a in partition_aliases) if partition_aliases else "" ) over_reset = " ".join(p for p in ( partition_clause, - f'ORDER BY "{time_alias}"', + f'ORDER BY {self._quote_ident(time_alias)}', "ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW", ) if p) reset_window_sql = ( f'SUM(CASE WHEN {pred_in_case} THEN 0 ELSE 1 END) ' - f'OVER ({over_reset}) AS "{cp_reset_alias}"' + f'OVER ({over_reset}) AS {self._quote_ident(cp_reset_alias)}' ) - cp_reset_cte_name = f"cp_reset_{slot_alias}" + cp_reset_cte_name = cte_allocator.allocate_cte(f"cp_reset_{slot_alias}") cp_reset_sql = ( - "SELECT\n " + carry_select + _SQL_SELECT_HEAD + carry_select + ",\n " + reset_window_sql + f"\nFROM {prev_cte}" ) @@ -8020,11 +9752,11 @@ def _emit_consecutive_periods_ctes_for_planned( # counted within its own reset group. value_partition_aliases = partition_aliases + [cp_reset_alias] value_partition_clause = _SQL_PARTITION_BY + ", ".join( - f'"{a}"' for a in value_partition_aliases + self._quote_ident(a) for a in value_partition_aliases ) over_value = " ".join(( value_partition_clause, - f'ORDER BY "{time_alias}"', + f'ORDER BY {self._quote_ident(time_alias)}', "ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW", )) # Outer CASE WHEN guarantees rows where the predicate is false @@ -8036,11 +9768,11 @@ def _emit_consecutive_periods_ctes_for_planned( value_outer_case = ( f'CASE WHEN {pred_in_case} ' f'THEN {value_inner_window_sql} ELSE 0 END ' - f'AS "{full_slot_alias}"' + f'AS {self._quote_ident(full_slot_alias)}' ) - cp_value_cte_name = f"cp_value_{slot_alias}" + cp_value_cte_name = cte_allocator.allocate_cte(f"cp_value_{slot_alias}") cp_value_sql = ( - "SELECT\n " + carry_select + _SQL_SELECT_HEAD + carry_select + ",\n " + value_outer_case + f"\nFROM {cp_reset_cte_name}" ) @@ -8310,142 +10042,22 @@ def _scan(text: Optional[str]) -> None: _scan(rendered) return ordered - def _collect_column_filter_join_paths( # NOSONAR(S3776) — one cohesive recursive walk of AGGREGATE composite keys (mirrors _render_aggregate_composite_expr) collecting Column.filter join paths. - self, *, base_render_order, slots_by_id, source_relation: str, - source_model, bundle, - ) -> List[Tuple[str, ...]]: - """Join paths needed by aggregation-time ``Column.filter`` CASE-WHEN - wrappers on LOCAL aggregate slots (DEV-1494). - - Recurses into AGGREGATE-phase composite keys (``ArithmeticKey`` / - ``ScalarCallKey``) so a filtered aggregate nested inside e.g. - ``a:sum + b:sum`` — rendered by ``_render_aggregate_composite_expr`` — - discovers its crossed join exactly like a top-level aggregate. Cross-model - aggregate sources (non-empty ``source.path``) are excluded: their filter - joins belong in the per-plan ``_cm_*`` CTE (DEV-1503). - """ - from slayer.core.keys import ( - AggregateKey, - ArithmeticKey, - Phase, - ScalarCallKey, - ) - - paths: List[Tuple[str, ...]] = [] - - def _visit(key) -> None: - if isinstance(key, AggregateKey): - if getattr(key.source, "path", ()): - return - cfk = key.column_filter_key - if cfk is None or not cfk.canonical_sql: - return - for p in self._filter_join_paths( - sql=cfk.canonical_sql, source_relation=source_relation, - source_model=source_model, bundle=bundle, - ): - if p not in paths: - paths.append(p) - elif isinstance(key, ArithmeticKey): - for operand in key.operands: - _visit(operand) - elif isinstance(key, ScalarCallKey): - for arg in key.args: - _visit(arg) - - for sid in base_render_order: - slot = slots_by_id.get(sid) - if slot is not None and slot.phase == Phase.AGGREGATE: - _visit(slot.key) - return paths - - def _collect_aggregate_source_join_paths( # NOSONAR(S3776) — one cohesive recursive walk of AGGREGATE composite keys (mirrors _collect_column_filter_join_paths) collecting derived-source Column.sql join paths. + def _expand_derived_row_dims( # NOSONAR(S3776) — one cohesive per-slot pass expanding derived ROW/TIME dimensions and registering the joins they cross. self, *, base_render_order, slots_by_id, source_relation: str, - source_model, bundle, - ) -> List[Tuple[str, ...]]: - """Join paths needed by LOCAL aggregate slots whose ``source`` is a - derived (``ColumnSqlKey``) column whose ``Column.sql`` crosses a - join (DEV-1502). - - Symmetric to the dimension treatment (DEV-1484) and the - ``Column.filter`` treatment (DEV-1494): the aggregate body already - renders the path-aliased ref via ``_expand_derived_column_sql`` at - spec-build time; this pass closes the loop by pulling the implied - ``LEFT JOIN``s into the host base FROM. - - Recurses into AGGREGATE-phase composite keys (``ArithmeticKey`` / - ``ScalarCallKey``) so a path-aliased source nested inside e.g. - ``a:sum + b:sum`` or ``coalesce(a:sum, 0)`` discovers its crossed - join. Cross-model aggregate sources (non-empty ``source.path``) - are skipped: their joins live in the per-plan ``_cm_*`` CTE; the - symmetric in-CTE discovery gap is tracked separately. - """ - from slayer.core.keys import ( - AggregateKey, - ArithmeticKey, - ColumnSqlKey, - Phase, - ScalarCallKey, - ) - - paths: List[Tuple[str, ...]] = [] - - def _visit(key) -> None: - if isinstance(key, AggregateKey): - source = key.source - if not isinstance(source, ColumnSqlKey): - return - if source.path: - return - col = next( - (c for c in source_model.columns if c.name == source.column_name), - None, - ) - if col is None or col.sql is None: - return - expanded = self._expand_derived_column_sql( - source_model=source_model, - source_relation=source_relation, - column_name=source.column_name, - bundle=bundle, - ) - for p in self._joined_paths_in_sql( - sql_expr=self._parse(expanded), - source_relation=source_relation, - source_model=source_model, - bundle=bundle, - ): - if p not in paths: - paths.append(p) - elif isinstance(key, ArithmeticKey): - for operand in key.operands: - _visit(operand) - elif isinstance(key, ScalarCallKey): - for arg in key.args: - _visit(arg) - - for sid in base_render_order: - slot = slots_by_id.get(sid) - if slot is not None and slot.phase == Phase.AGGREGATE: - _visit(slot.key) - return paths - - def _expand_derived_row_dims( # NOSONAR(S3776) — one cohesive per-slot pass expanding derived ROW/TIME dimensions and collecting the joins they cross. - self, *, base_render_order, slots_by_id, source_relation: str, - source_model, bundle, needed_join_paths: List[Tuple[str, ...]], + source_model, bundle, scope: ScopeFrame, ) -> Dict[str, exp.Expression]: """Pre-expand derived (``ColumnSqlKey``) ROW dimensions and derived TIME dimensions for the base SELECT: inline sibling/joined derived refs - (DEV-1333 / DEV-1410), append any joins their SQL crosses to - ``needed_join_paths`` (in place), and return the expanded-expr-by-slot-id - map the render branch reads from. Extracted from - ``_build_base_select_for_planned``. + (DEV-1333 / DEV-1410), register any joins their SQL crosses into + ``scope.join_paths`` (Law 1 — the join-discovery side effect), and return + the expanded-expr-by-slot-id map the render branch reads from. Extracted + from ``_build_base_select_for_planned``. """ from slayer.core.keys import ColumnSqlKey, Phase, TimeTruncKey def _add(path: Tuple[str, ...]) -> None: - if path and path not in needed_join_paths: - needed_join_paths.append(path) + if path: + scope.join_paths.add(path) derived_expr_by_sid: Dict[str, exp.Expression] = {} for sid in base_render_order: @@ -8461,6 +10073,13 @@ def _add(path: Tuple[str, ...]) -> None: time_column=key.column, source_model=source_model, source_relation=source_relation, bundle=bundle, ) + # DEV-1701: register the join to the derived TD's OWNING model + # (``key.column.path``) plus every further join its expanded sql + # crosses — parity with the plain joined-derived-dimension branch + # below. ``is_root=False`` (in ``_raw_time_col_expr_for_planned``) + # already anchored the inner refs at the host-path alias, so the + # scan and the render agree. + _add(key.column.path) for p in self._joined_paths_in_sql( sql_expr=raw, source_relation=source_relation, source_model=source_model, bundle=bundle, @@ -8540,7 +10159,9 @@ def _build_from_clause_from_planned( source_relation: str, ) -> exp.Expression: if source_model.sql_table: - return exp.to_table(source_model.sql_table, alias=source_relation) + # DEV-1686 reserved-word alias + DEV-1645 mixed-case physical-name + # quoting via ``_to_table``. + return self._to_table(source_model.sql_table, alias=source_relation) if source_model.sql: return exp.Subquery( this=self._parse(source_model.sql), @@ -8599,11 +10220,21 @@ def _raw_time_col_expr_for_planned( f"{time_column.path[-1]!r} which is not in the resolved " f"source bundle.", ) + # DEV-1701: a JOINED derived TIME dimension whose ``Column.sql`` + # crosses a FURTHER join must anchor its inner refs at the + # host-path alias (``customers_v2__regions``), not the bare + # direct-join alias (``regions``) — otherwise the host base + # SELECT references a table its FROM never joins. ``is_root= + # False`` carries the full ``__`` prefix, exactly as the plain + # joined-derived-dimension branch in ``_expand_derived_row_dims`` + # does. The ``continue``-less callers (base render, ranked + # subquery, default-time-col) all render in the host frame. expanded_sql = self._expand_derived_column_sql( source_model=joined_model, source_relation="__".join(time_column.path), column_name=time_column.column_name, bundle=bundle, + is_root=False, ) else: expanded_sql = self._expand_derived_column_sql( @@ -8685,12 +10316,17 @@ def _joined_paths_in_sql( bundle=bundle, ) - def _collect_filter_join_paths( - self, *, planned_query, source_model, source_relation: str, bundle, + def _resolve_where_filter_joins_via_scope( + self, *, planned_query, scope: ScopeFrame, skip_filter_ids: Optional[Set[str]] = None, - ) -> List[Tuple[str, ...]]: - """Collect the join paths a query's WHERE-phase filters reference so - the FROM pulls them in. + filters_override: "Optional[List[Any]]" = None, + ) -> None: + """Register into ``scope.join_paths`` the joins every WHERE-phase filter + references (Law 1 — discovery is a side effect of resolving the filter + through the scope). A 1:1 replacement for the former + ``_collect_filter_join_paths`` (wrap-and-reuse, D-G): it delegates to the + same ``_value_key_join_paths`` / ``_filter_join_paths`` sub-scanners in + the same ``filters_by_phase`` order, so the base FROM stays byte-identical. Covers three shapes: * typed joined column ref (``customers.regions.name == 'US'``) — @@ -8700,27 +10336,31 @@ def _collect_filter_join_paths( ``ColumnSqlKey``, expanded then scanned; * Mode-A ``SlayerModel.filters`` text with a ``__`` join path (``customers__regions.name = 'EU'``) — parsed and scanned. - """ - from slayer.core.keys import Phase - seen: set = set() - ordered: List[Tuple[str, ...]] = [] + Filters routed to a per-plan ``_cm_*`` CTE (``skip_filter_ids``) are + applied there, not on the host base, so their joins are not registered + here. - def _merge(paths: List[Tuple[str, ...]]) -> None: - for p in paths: - if p not in seen: - seen.add(p) - ordered.append(p) + ``filters_override`` (DEV-1732) replaces the filter list being scanned — + the windowed ``_src`` scope passes the SAME rewritten list it renders, so + discovery and rendering can never disagree about what the CTE contains. + """ + from slayer.core.keys import Phase skip = skip_filter_ids or set() - for fp in planned_query.filters_by_phase: + filters = ( + planned_query.filters_by_phase + if filters_override is None else filters_override + ) + for fp in filters: if fp.phase != Phase.ROW or fp.id in skip: continue if fp.expression is not None: - _merge(self._value_key_join_paths( - key=fp.expression.value_key, source_model=source_model, - source_relation=source_relation, bundle=bundle, - )) + for p in self._value_key_join_paths( + key=fp.expression.value_key, source_model=scope.root_model, + source_relation=scope.root_relation, bundle=scope.bundle, + ): + scope.join_paths.add(p) elif fp.text is not None: # DEV-1450 #4b / DEV-1494: discover joins from BOTH the # un-inlined text (a placeholder dotted ref like @@ -8728,11 +10368,11 @@ def _merge(paths: List[Tuple[str, ...]]) -> None: # to a constant) AND the inline-expanded text (a bare/dotted # DERIVED ref like ``is_eu`` surfaces the join its expansion # crosses). See ``_filter_join_paths``. - _merge(self._filter_join_paths( - sql=fp.text, source_relation=source_relation, - source_model=source_model, bundle=bundle, - )) - return ordered + for p in self._filter_join_paths( + sql=fp.text, source_relation=scope.root_relation, + source_model=scope.root_model, bundle=scope.bundle, + ): + scope.join_paths.add(p) def _value_key_join_paths( # NOSONAR(S3776) — one cohesive recursive ValueKey-tree walk; complexity is the per-key-type dispatch. self, *, key, source_model, source_relation: str, bundle, @@ -8741,9 +10381,10 @@ def _value_key_join_paths( # NOSONAR(S3776) — one cohesive recursive ValueKey DEV-1475): a direct ``ColumnKey.path``; a derived ``ColumnSqlKey`` (local or joined — expanded then scanned for the joins its ``sql`` crosses); and recursively through ``ArithmeticKey`` / ``ScalarCallKey`` / - ``BetweenKey`` / ``InKey`` operands. Extracted from - ``_collect_filter_join_paths``; ``_joined_paths_in_sql`` already emits - path prefixes, and ``ColumnKey.path`` prefixes are expanded here. + ``BetweenKey`` / ``InKey`` operands. Sub-scanner shared by + ``_resolve_where_filter_joins_via_scope``; ``_joined_paths_in_sql`` + already emits path prefixes, and ``ColumnKey.path`` prefixes are + expanded here. """ from slayer.core.keys import ( ArithmeticKey, @@ -8850,20 +10491,27 @@ def _validate_aggregate_kwarg_paths( source, src_leaf: str, ) -> None: - """Reject kwarg column refs whose join path disagrees with the - aggregate source path. - - A kwarg path that doesn't match the aggregate source path would - silently bind the kwarg to a different model (host vs joined - target) than the aggregate value column — meaningless SQL - semantically. Caller-side cross-model rerooting strips the - matching prefix from source AND kwargs before reaching this - point; any residual mismatch surfaces here. Both bare-column + """Reject CROSS-MODEL aggregates' kwarg column refs whose join path + disagrees with the aggregate source path. + + For a target-rooted aggregate, a kwarg path that doesn't match the + source path after reroot prefix-stripping would silently bind the + kwarg to a different model than the aggregate value column — + meaningless SQL semantically; any residual mismatch surfaces here. + + DEV-1709: LOCAL aggregates (``source.path == ()``) are exempt — a + structurally-crossing kwarg (``weighted_avg(weight=customers.w)``) + is now a supported crossing INPUT: the widened Law-3 trigger + isolates the aggregate host-rooted, and inside that CTE's + sub-render the kwarg resolves through the host scope (join + registration + path-aliased emission). Both bare-column (``ColumnKey``) and derived-column (``ColumnSqlKey``) kwarg refs go through this gate (CodeRabbit fold-in on PR #144). """ from slayer.core.keys import ColumnKey, ColumnSqlKey + if not source.path: + return for kname, kval in key.kwargs: if isinstance(kval, (ColumnKey, ColumnSqlKey)) and kval.path != source.path: raise AggregationNotAllowedError( @@ -8887,6 +10535,7 @@ def _build_agg_render_spec_from_planned( # NOSONAR(S3776) — sequential isinst source_relation: str, full_alias: str, bundle=None, + resolved_agg_kwargs: "Optional[Dict[str, ResolvedAggKwarg]]" = None, ) -> AggRenderSpec: """Build an ``AggRenderSpec`` from a planned aggregate slot so ``_build_agg`` / ``_resolve_sql`` / ``_wrap_cast_for_type`` emit @@ -8989,15 +10638,18 @@ def _build_agg_render_spec_from_planned( # NOSONAR(S3776) — sequential isinst ) else: sql_text = col.sql if col.sql else col.name - # DEV-1450 stage 7b.13: stringify kwargs through the shared - # helper. ``AggRenderSpec.agg_kwargs`` is ``Dict[str, str]`` - # and downstream ``_validate_agg_param_value`` rejects - # anything not matching ``_SAFE_AGG_PARAM_RE``; the helper - # emits identifiers / dotted identifiers / numeric literals - # that satisfy the regex, and rejects bool / None / unknown - # types at the boundary. + # DEV-1527: a column-ref kwarg (``weight=`` / ``other=``) + # that the pre-FROM scope pass resolved is embedded as a trusted + # ``kind="expr"`` expression (its join already base-pulled); every + # other kwarg (scalar / string / a column-ref on a path this call + # has no scope for — e.g. the cross-model CTE build) canonical- + # stringifies as before and coerces to ``kind="str"`` via the + # ``AggRenderSpec`` before-validator (guarded downstream by + # ``_validate_agg_param_value`` / ``_SAFE_AGG_PARAM_RE``). + resolved_kw = resolved_agg_kwargs or {} agg_kwargs_str = { - k: agg_kwarg_canonical_str(v) for k, v in key.kwargs + k: (resolved_kw[k] if k in resolved_kw else agg_kwarg_canonical_str(v)) + for k, v in key.kwargs } # DEV-1450 stage 7b.12: propagate ``AggregateKey.column_filter_key`` # into ``AggRenderSpec.filter_sql`` so ``_build_agg`` wraps the @@ -9034,7 +10686,7 @@ def _build_agg_render_spec_from_planned( # NOSONAR(S3776) — sequential isinst f"AggregateKey source {type(source).__name__} not supported.", ) - def _build_where_having_from_planned( + def _build_where_having_from_planned( # NOSONAR(S3776) — one cohesive pass over filters_by_phase routing each entry to WHERE / HAVING / POST by phase, with the per-carrier (typed vs Mode-A text) rendering and the HAVING grouped-column guard inline. The complexity is pre-existing; DEV-1732 added only the `filters_override` list selection. Splitting the phase routing from the rendering would thread slot_by_key / first_last_state / where_parts / having_parts through helpers without simplifying anything. self, *, planned_query, @@ -9044,7 +10696,10 @@ def _build_where_having_from_planned( skip_filter_ids: Optional[Set[str]] = None, first_last_state: Optional[FirstLastRenderState] = None, aliases_by_slot_id: Optional[Dict[str, List[str]]] = None, + filters_override: "Optional[List[Any]]" = None, ): + """``filters_override`` (DEV-1732) replaces ``filters_by_phase`` as the + list being rendered — see ``_effective_src_filters``.""" from slayer.core.keys import Phase skip = skip_filter_ids or set() @@ -9060,7 +10715,11 @@ def _build_where_having_from_planned( } where_parts: list[str] = [] having_parts: list[str] = [] - for fp in planned_query.filters_by_phase: + filters = ( + planned_query.filters_by_phase + if filters_override is None else filters_override + ) + for fp in filters: if fp.id in skip: # DEV-1450 stage 7b.12: filters routed into a per-plan # cross-model CTE (where_filter_ids / having_filter_ids) @@ -9136,7 +10795,7 @@ def _build_where_having_from_planned( # legacy `_build_where_and_having` at generator.py:2566. # DEV-1450 #4b: a reference to a non-trivial derived column # is inline-expanded (and pulls its crossed joins into the - # FROM via _collect_filter_join_paths). + # FROM via _resolve_where_filter_joins_via_scope). target_parts.append(self._render_model_filter_sql( sql=fp.text, columns=fp.text_columns, @@ -9255,7 +10914,7 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di Supports ``ColumnKey`` (local AND joined ``path != ()`` — emitted as ``<__path_alias>.``; the join is pulled into the FROM by - ``_collect_filter_join_paths``), ``ColumnSqlKey`` (derived column — + ``_resolve_where_filter_joins_via_scope``), ``ColumnSqlKey`` (derived column — expanded inline, sibling/joined refs resolved), ``LiteralKey``, ``ArithmeticKey``, ``ScalarCallKey``, ``BetweenKey``, and a LOCAL ``AggregateKey`` (for HAVING — rendered as the bare aggregate @@ -9305,6 +10964,18 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di and aliases_by_slot_id.get(slot.id) ): having_full_alias = aliases_by_slot_id[slot.id][0] + # DEV-1527: resolve this local aggregate's column-ref kwargs + # (``weighted_avg(weight=)`` / ``corr(other=)``) through a + # host scope so a derived/crossing kwarg renders its expanded, join- + # anchored expression HERE too — matching the base SELECT — instead of + # collapsing to a bare, non-existent name. The crossed join is already + # base-pulled by ``_resolve_agg_inputs_via_scope`` (this HAVING + # aggregate is also a ``base_render_order`` slot), so the throwaway + # scope is used only to reproduce the same anchored expression. + having_kwargs = self._resolve_agg_kwargs_for_key( + key=key, source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) synth = self._build_agg_render_spec_from_planned( slot=slot, key=key, @@ -9312,6 +10983,7 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di source_relation=source_relation, full_alias=having_full_alias, bundle=bundle, + resolved_agg_kwargs=having_kwargs, ) # DEV-1501: thread the rn suffix maps from the base SELECT # so a HAVING reference to a hidden first/last aggregate @@ -9346,7 +11018,7 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di # Joined column ref (``customers.regions.name``) — emit the # ``__``-canonical path alias (``customers__regions.name``). # The join is pulled into the FROM by - # ``_collect_filter_join_paths``. + # ``_resolve_where_filter_joins_via_scope``. return exp.Column( this=exp.to_identifier(key.leaf), table=exp.to_identifier("__".join(key.path)), @@ -9372,7 +11044,7 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di # Expand the column's ``sql`` rooted at the JOINED model, # qualifying bare refs to the ``__``-canonical path alias; the # join itself is pulled into the FROM by - # ``_collect_filter_join_paths`` (which adds ``key.path``). + # ``_resolve_where_filter_joins_via_scope`` (which adds ``key.path``). joined_model = bundle.get_referenced_model(key.path[-1]) if joined_model is None: raise ValueError( @@ -9397,7 +11069,7 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di ) # Derived column (``Column.sql`` set) — expand inline, resolving # sibling / joined derived refs and pulling crossed joins into the - # FROM (via ``_collect_filter_join_paths``). + # FROM (via ``_resolve_where_filter_joins_via_scope``). expanded_sql = self._expand_derived_column_sql( source_model=source_model, source_relation=source_relation, @@ -9447,6 +11119,15 @@ def _render_value_key_for_filter( # NOSONAR(S3776) — sequential isinstance di )) if key.name == "like": return exp.Like(this=args[0], expression=args[1]) + # DEV-1576: a 2-arg ROUND needs the Postgres numeric cast, so it + # must be a TYPED node (exp.Round) routed through the target-dialect + # rewrite. Only ROUND is retyped: the string-hygiene functions + # (substr / concat / lower / ...) must emit literally as written + # (DEV-1484), which exp.func would break by transpiling them per + # dialect — so they stay as Anonymous passthrough. + typed = exp.func(key.name.upper(), *args) + if isinstance(typed, exp.Round): + return self._finalize_scalar_call(typed) return exp.Anonymous(this=key.name.upper(), expressions=args) if isinstance(key, BetweenKey): col_expr = self._render_value_key_for_filter( @@ -9625,6 +11306,15 @@ def _slot_alias_column(slot) -> Optional[exp.Expression]: )) if key.name == "like": return exp.Like(this=args[0], expression=args[1]) + # DEV-1576: a 2-arg ROUND needs the Postgres numeric cast, so it + # must be a TYPED node (exp.Round) routed through the target-dialect + # rewrite. Only ROUND is retyped: the string-hygiene functions + # (substr / concat / lower / ...) must emit literally as written + # (DEV-1484), which exp.func would break by transpiling them per + # dialect — so they stay as Anonymous passthrough. + typed = exp.func(key.name.upper(), *args) + if isinstance(typed, exp.Round): + return self._finalize_scalar_call(typed) return exp.Anonymous(this=key.name.upper(), expressions=args) if isinstance(key, BetweenKey): col_expr = self._render_filter_for_outer_wrapper( @@ -9730,23 +11420,33 @@ def _scalar_to_sqlglot(v) -> exp.Expression: ) @staticmethod - def _build_arithmetic_for_filter( + def _paren_if_binary(node: exp.Expression) -> exp.Expression: + """DEV-1539: wrap a multi-term operand in ``(...)`` when it is a + ``Binary`` (arithmetic ``a + b``, or an ``AND``/``OR`` connector) so a + surrounding comparator's precedence is explicit by inspection, not only + by SQL operator-precedence rules — ``(a + b) > 7``, not ``a + b > 7``. + Bare columns, literals, function calls, and already-enclosed forms + (``CAST(...)`` / ``Paren``) are not ``Binary`` and pass through.""" + return exp.Paren(this=node) if isinstance(node, exp.Binary) else node + + @staticmethod + def _build_arithmetic_for_filter( # NOSONAR(S3776) — sequential per-operator dispatch (==/!= → EQ/NEQ, comparison, arithmetic) with DEV-1539 precedence paren-wrapping; each branch is the per-op emission contract. *, op: str, operands: list, ) -> exp.Expression: # DSL ``==``/``!=`` map to sqlglot EQ/NEQ; sqlglot then emits the - # dialect-correct SQL operator (postgres ``=``/``!=``). - if op in ("==", "="): - return exp.EQ(this=operands[0], expression=operands[1]) - if op in ("!=", "<>"): - return exp.NEQ(this=operands[0], expression=operands[1]) - if op == "<": - return exp.LT(this=operands[0], expression=operands[1]) - if op == "<=": - return exp.LTE(this=operands[0], expression=operands[1]) - if op == ">": - return exp.GT(this=operands[0], expression=operands[1]) - if op == ">=": - return exp.GTE(this=operands[0], expression=operands[1]) + # dialect-correct SQL operator (postgres ``=``/``!=``). DEV-1539: a + # multi-term comparison operand is parenthesised so its precedence is + # explicit (``(a + b) > 7`` / ``x = (a OR b)``). + _cmp = { + "==": exp.EQ, "=": exp.EQ, "!=": exp.NEQ, "<>": exp.NEQ, + "<": exp.LT, "<=": exp.LTE, ">": exp.GT, ">=": exp.GTE, + } + cmp_cls = _cmp.get(op) + if cmp_cls is not None: + return cmp_cls( + this=SQLGenerator._paren_if_binary(operands[0]), + expression=SQLGenerator._paren_if_binary(operands[1]), + ) if op == "+": # Unary plus is a no-op; legacy never emits it explicitly. if len(operands) == 1: @@ -9908,7 +11608,24 @@ def _apply_order_limit_from_planned( # NOSONAR(S3776) — per-order-entry slot- targets out of ``base_render_order``, preserving today's ``NotImplementedError``). """ - from slayer.core.keys import AggregateKey + from slayer.core.keys import ( + AggregateKey, + ArithmeticKey, + ColumnKey, + ColumnSqlKey, + ScalarCallKey, + TimeTruncKey, + TransformKey, + ) + + # DEV-1733: the EXACT set of hidden key kinds that resolve to a + # materialised alias. Deliberately enumerated rather than "any hidden + # slot that happens to carry an alias" — a hidden ROW slot with an + # alias must still hit the split-emission / invariant branches below, + # never be ordered on as a bare column that is not in the GROUP BY. + _MATERIALISED_ORDER_KINDS = ( + AggregateKey, ArithmeticKey, ScalarCallKey, TransformKey, + ) for order_entry in planned_query.order: slot = slots_by_id.get(order_entry.slot_id) @@ -9925,42 +11642,110 @@ def _apply_order_limit_from_planned( # NOSONAR(S3776) — per-order-entry slot- if aliases_by_slot_id is not None else [] ) - if aliases and isinstance(slot.key, AggregateKey): + if aliases and isinstance(slot.key, _MATERIALISED_ORDER_KINDS): full_alias = aliases[0] order_col = exp.Column( this=exp.to_identifier(full_alias, quoted=True), ) ascending = order_entry.direction == "asc" select = select.order_by( - exp.Ordered(this=order_col, desc=not ascending), + self._ordered(order_col, ascending=ascending), ) continue - # Hidden ROW / transform / cross-model / composite ORDER - # targets aren't materialised in the local-only SELECT — - # preserved as NotImplementedError. DEV-1501's - # ``aggregates_only=True`` keeps hidden ROW targets out - # of ``base_render_order``; hidden composite-aggregate - # ORDER BY is rejected at the ``OrderItem`` input - # validation layer. + # DEV-1712 (Law 2, split emission): a hidden LOCAL ROW column + # ordered in an UNGROUPED query. The plan-time order validation + # (``plan_query``) guarantees the only hidden ROW slot that + # reaches here is a local (empty-path) column in a query with no + # GROUP BY — grouped row columns and joined columns are rejected + # up front, aggregates take the branch above. Emit a SPLIT + # ``.`` reference (mixed-case-aware) against + # the base FROM scope, identical to how the column would render + # if it were a projected dimension. + key = slot.key + row_key = key.column if isinstance(key, TimeTruncKey) else key + # A bare LOCAL column — split-emit the qualified column ref. + if ( + source_model is not None + and isinstance(row_key, ColumnKey) + and not row_key.path + ): + order_col = self._joined_or_local_dim_expr( + path=(), leaf=row_key.leaf, source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + ascending = order_entry.direction == "asc" + select = select.order_by( + self._ordered(order_col, ascending=ascending), + ) + continue + # A LOCAL DERIVED column (``ColumnSqlKey``, path empty): resolve + # its ``Column.sql`` through a throwaway host scope. That both + # anchors the expansion AND surfaces whether the SQL crosses a + # join. A hidden order-only derived column is NOT projected, so + # its join was never pulled into the base FROM — ordering on it + # would reference an unbound table. Reject that (project it), + # rather than emit invalid SQL; a non-crossing derived column + # (e.g. a bare mixed-case identifier) orders on its expression. + if ( + source_model is not None + and bundle is not None + and isinstance(row_key, ColumnSqlKey) + and not row_key.path + ): + # Detect join crossing via a throwaway scope (register-only); + # the resolved expr is discarded — its expansion lacks the + # DEV-1645 mixed-case quoting the planned-dim helper applies. + allocator = self._new_allocator() + scope = ScopeFrame( + scope_id=allocator.next_scope_id(source_relation), + root_model=source_model, + root_relation=source_relation, + bundle=bundle, + dialect=self._dialect, + allocator=allocator, + ) + scope.resolve(row_key) + if scope.join_paths: + # The derived column IS local (``orders.cust_region``); + # it merely depends on an unpulled join. Report its own + # qualified name, not a fabricated ``customers.cust_region``. + raise UnresolvableOrderColumnError( + column=row_key.column_name, qualifier=source_relation, + ) + # Non-crossing local derived column — emit through the + # planned-dim helper so the expansion is quoted identically + # to a projected dimension (mixed-case-safe). + order_col = self._joined_or_local_dim_expr( + path=(), leaf=row_key.column_name, + source_model=source_model, + source_relation=source_relation, bundle=bundle, + ) + ascending = order_entry.direction == "asc" + select = select.order_by( + self._ordered(order_col, ascending=ascending), + ) + continue + # Defensive: any other hidden shape should have been rejected at + # plan time (transform / composite / joined / grouped-row). raise NotImplementedError( - f"DEV-1450 stage 7b.10+: ORDER BY references a " - f"hidden slot (id={slot.id!r}, key=" - f"{type(slot.key).__name__}) not materialised in " - f"the local-only SELECT. Deferred to a later slice." - ) - if slot.public_aliases: - alias = slot.public_aliases[0] - elif slot.public_name: - alias = slot.public_name - else: - alias = slot.declared_name - full_alias = f"{source_relation}.{alias}" + f"ORDER BY references a hidden slot (id={slot.id!r}, key=" + f"{type(slot.key).__name__}) that was not resolved at plan " + f"time — this is an internal invariant violation." + ) + # DEV-1713: resolve to the SAME full alias the projection emits — + # a joined ROW dimension projects under the DOTTED result key + # (``orders.customers.regions.name``), so the ORDER BY must match + # it, not the flat ``declared_name`` (``customers__regions__name``), + # which would name a column the SELECT never projects. + full_alias = self._full_alias_for_slot( + slot=slot, source_relation=source_relation, alias_index={}, + ) order_col = exp.Column( this=exp.to_identifier(full_alias, quoted=True), ) ascending = order_entry.direction == "asc" select = select.order_by( - exp.Ordered(this=order_col, desc=not ascending), + self._ordered(order_col, ascending=ascending), ) if planned_query.limit is not None: @@ -10057,9 +11842,15 @@ def generate_planned_stages( if not planned_queries: raise ValueError("generate_planned_stages requires at least one stage") if len(planned_queries) == 1: - return generate_from_planned( + # DEV-1716: single-stage DB-bound terminal — apply the dialect alias + # mangling post-pass (BigQuery / T-SQL; identity otherwise). + sql = generate_from_planned( planned_queries[0], bundle=bundle, dialect=dialect, ) + sql = get_dialect(dialect).rewrite_emitted_sql(sql) + # DEV-1705: validate the final POST-mangle, pre-RLS statement (env-gated). + maybe_validate_scopes(sql, dialect=dialect) + return sql schema_by_name = { p.stage_schema.relation_name: p.stage_schema @@ -10110,7 +11901,17 @@ def generate_planned_stages( for cte in existing_ctes: root_ast = root_ast.with_(cte.args["alias"], as_=cte.this, dialect=dialect) - return root_ast.sql(dialect=dialect, pretty=True) + # DEV-1716: terminal emit of the multi-stage root — apply the dialect + # rewrite_emitted_sql post-pass (BigQuery / T-SQL alias mangling; identity + # otherwise). The re-parse/with_ grafting above can surface dotted aliases + # the per-stage emits already mangled, so mangle once more here + # (idempotent) to catch the root's own projection. + sql = root_ast.sql(dialect=dialect, pretty=True) + sql = get_dialect(dialect).rewrite_emitted_sql(sql) + # DEV-1705: validate the final POST-mangle, pre-RLS multi-stage root + # (env-gated). One validation per final terminal (single- vs multi-stage). + maybe_validate_scopes(sql, dialect=dialect) + return sql def _stage_rename_wrapper(*, planned, stage_sql, dialect): diff --git a/slayer/sql/naming.py b/slayer/sql/naming.py new file mode 100644 index 00000000..6b579790 --- /dev/null +++ b/slayer/sql/naming.py @@ -0,0 +1,347 @@ +"""DEV-1706 Stage 2 — minimal, collision-safe alias allocator. + +A single ``AliasAllocator`` is created per top-level ``generate_from_planned`` +call and threaded to every ``ScopeFrame`` built during that call. It mints: + +* ``_val_`` materialisation aliases (Law 2 — projection-boundary columns), +* CTE names, + +seeded from every name already in scope (bundle relations, ``__``-path join +aliases, public projection aliases, model names) so a minted name can never +collide with a user column, a path alias, or a reserved public alias. It also +hands out generation-local ``ScopeFrame`` ids. + +The allocator is the *minimal* collision primitive (subsumes DEV-1692's +collision check). DEV-1713 Stage 9 grew this module into the single owner of +every alias / result-key decision: + +* :func:`result_key` / :func:`result_key_from_alias` — the DOTTED user-facing + FINAL-stage keys (``orders.customers.regions.name``); +* :func:`flat_name` — the ``__``-joined INNER-stage downstream schema names + (``customers__regions__name``, the StageSchema bind contract); +* :func:`encode_alias` / :func:`decode_alias` — the BigQuery / T-SQL dotted + alias mangling bijection (DEV-1571), relocated here from the dialect package; +* :func:`quote_mixed_case_identifiers` / :func:`maybe_quote_ident` — the + DEV-1645 mixed-case identifier-quoting policy, relocated here from the + generator; +* :func:`assert_unique_cte_names` — the DEV-1692 per-``WITH``-scope CTE + name-collision belt. +""" + +from __future__ import annotations + +from typing import Optional, Tuple + +import sqlglot +from pydantic import BaseModel, ConfigDict, PrivateAttr +from sqlglot import exp + +# --------------------------------------------------------------------------- +# Dialect case-folding policy (DEV-1726). +# +# SLayer-minted names (CTE families, ``_val_`` materialisation aliases) are +# emitted unquoted, so on case-folding backends two names differing only in +# case fold to the same identifier — two user measure aliases ``Foo``/``foo`` +# both driving time_shift CTEs would produce a duplicate ``WITH`` name. The +# policy of WHICH sqlglot dialects fold lives HERE (naming policy, per the +# Stage-9 ownership decision) because this module must stay an import leaf — +# dialect modules import from it. +# +# Membership notes (confirmed against vendor docs, sqlglot's +# NORMALIZATION_STRATEGY, and — for SQLite/DuckDB — empirically): +# * BigQuery FOLDS: GoogleSQL's case-sensitivity table marks "aliases within +# a query" (which CTE names are) case-insensitive; only real table/dataset +# names are case-sensitive. This corrects the DEV-1726 issue text. +# * SQLite and DuckDB reject case-differing CTE names even when QUOTED. +# * MySQL / T-SQL fold DELIBERATELY despite platform/collation dependence: +# folding is rename-only-safe (every reference uses the allocated name), +# while not folding leaves the collision live on the majority configs +# (Windows/macOS MySQL, default-collation SQL Server). +# * ClickHouse identifiers are case-sensitive — exact comparison. +# * Unknown dialect strings compare exact (previous behavior, fail-safe). +# +# The fold KEY is ``str.lower()`` — parity with sqlglot's +# ``normalize_identifier``; ``str.casefold()`` would over-equate (``ß``→``ss``). +# --------------------------------------------------------------------------- + +CASE_FOLDING_SQLGLOT_DIALECTS: frozenset[str] = frozenset({ + "postgres", "redshift", "snowflake", "oracle", "mysql", "tsql", + "sqlite", "duckdb", "trino", "presto", "databricks", "spark", "bigquery", +}) + +# Explicit, so "does not fold" is a decision, not an omission: every registry +# dialect must appear in exactly one of the two sets (pinned by +# tests/test_dev1726_cte_case_folding.py against the dialect registry). +KNOWN_CASE_SENSITIVE_SQLGLOT_DIALECTS: frozenset[str] = frozenset({"clickhouse"}) + + +def dialect_folds_case(dialect: str) -> bool: + """True iff ``dialect`` case-folds unquoted identifiers (CTE names in + particular). Input is normalized via ``strip().lower()``; an unknown + dialect string returns False (exact comparison — fail-safe).""" + return dialect.strip().lower() in CASE_FOLDING_SQLGLOT_DIALECTS + + +class AliasAllocator(BaseModel): + """Per-generation collision-safe name allocator (mutable). + + With ``folds_case=True`` (case-folding dialects — DEV-1726, set via + :func:`dialect_folds_case` by ``SQLGenerator._new_allocator``), every + ``_taken`` comparison folds with ``str.lower()`` while names are still + returned in the caller's original case — so ``shifted_Foo`` blocks + ``shifted_foo`` and the second mint walks to ``shifted_foo_2``. + """ + + model_config = ConfigDict(arbitrary_types_allowed=True) + + # Fold every _taken comparison with str.lower() (DEV-1726). Comparison + # only: allocated names keep the caller's original case. + folds_case: bool = False + + # External names the allocator must avoid (user columns, join aliases, + # public projection aliases, model names). Stored FOLDED when folds_case. + # NOSONAR lines below: Pydantic v2 PrivateAttr idiom — the annotation is the + # attribute's runtime type after model init; the ``PrivateAttr(...)`` sentinel + # is replaced by Pydantic. S5890 can't model this and is a false positive. + _reserved: set[str] = PrivateAttr(default_factory=set) # NOSONAR(S5890) + # Names already handed out by this allocator. + _used: set[str] = PrivateAttr(default_factory=set) # NOSONAR(S5890) + # Monotonic ``_val_`` cursor (never reset per scope, so sibling scopes + # in one generation cannot mint the same ``_val_0``). + _val_seq: int = PrivateAttr(default=0) # NOSONAR(S5890) + # Monotonic scope-id cursor. + _scope_seq: int = PrivateAttr(default=0) # NOSONAR(S5890) + + def _fold(self, name: str) -> str: + """The comparison key: ``str.lower()`` when folding, else identity.""" + return name.lower() if self.folds_case else name + + def reserve(self, *names: str) -> None: + """Mark ``names`` as taken so they are never allocated.""" + self._reserved.update(self._fold(n) for n in names) + + def _taken(self, name: str) -> bool: + key = self._fold(name) + return key in self._reserved or key in self._used + + def allocate(self, preferred: str) -> str: + """Return ``preferred`` if free, else ``preferred_2``, ``preferred_3``, …""" + candidate = preferred + suffix = 2 + while self._taken(candidate): + candidate = f"{preferred}_{suffix}" + suffix += 1 + self._used.add(self._fold(candidate)) + return candidate + + def allocate_val(self) -> str: + """Return the next free ``_val_`` materialisation alias.""" + while True: + candidate = f"_val_{self._val_seq}" + self._val_seq += 1 + if not self._taken(candidate): + self._used.add(self._fold(candidate)) + return candidate + + def allocate_cte(self, preferred: str) -> str: + """Return a collision-safe CTE name (same walk as :meth:`allocate`).""" + return self.allocate(preferred) + + def next_scope_id(self, root_relation: str) -> str: + """Return a generation-local ``ScopeFrame`` id, ``#``. + + Ephemeral — used only for in-generation materialisation dedup; it is + never emitted into SQL, result keys, or persisted state (D-F / Codex L1). + """ + scope_id = f"{root_relation}#{self._scope_seq}" + self._scope_seq += 1 + return scope_id + + +# --------------------------------------------------------------------------- +# Result-key / flat-name builders (DEV-1713 Stage 9). +# +# A query renders as either a FINAL stage (its columns are the user-facing +# result keys — DOTTED, ``orders.customers.regions.name``) or an INNER stage +# of a multi-stage DAG (its columns are downstream bind names — ``__``-joined, +# ``customers__regions__name``). These two builders are the single owners of +# those two forms; the planner's is-final flag picks between them so the two +# can never mix (D3 / DEV-1495 bug 1). +# --------------------------------------------------------------------------- + + +def result_key(*, source_relation: str, path: Tuple[str, ...] = (), leaf: str) -> str: + """Build the DOTTED final-stage result key from STRUCTURED parts. + + ``source_relation`` then each ``path`` hop then ``leaf``, dot-joined: + ``result_key(source_relation="orders", path=("customers",), leaf="revenue")`` + → ``"orders.customers.revenue"``. + + ``leaf`` must not contain a dot — hop information belongs in ``path`` so + ownership is unambiguous. For an already-canonical relative alias that + legitimately embeds hop dots (a cross-model measure alias such as + ``customers.revenue_sum``), use :func:`result_key_from_alias` instead. + """ + if "." in leaf: + raise ValueError( + f"result_key leaf must not contain '.': {leaf!r}. Pass hops via " + f"`path`, or use result_key_from_alias for a canonical dotted alias." + ) + return ".".join((source_relation, *path, leaf)) + + +def result_key_from_alias(*, source_relation: str, alias: str) -> str: + """Build a final-stage result key from an already-canonical relative + ``alias`` that may embed hop dots (e.g. a cross-model measure alias + ``customers.revenue_sum`` → ``orders.customers.revenue_sum``).""" + return f"{source_relation}.{alias}" + + +def flat_name(dotted: str, *, strip_relation: Optional[str] = None) -> str: + """Flatten a dotted name to its ``__``-joined INNER-stage bind name. + + When ``strip_relation`` is given, the exact ``f"{strip_relation}."`` + prefix is removed first (a dot-boundary match, so ``strip_relation='orders'`` + strips ``orders.`` but never the char prefix of a sibling ``orders_archive``). + Remaining dots become ``__``: + ``flat_name("orders.customers.revenue", strip_relation="orders")`` → + ``"customers__revenue"``. + """ + remainder = dotted + if strip_relation is not None: + prefix = f"{strip_relation}." + if remainder.startswith(prefix): + remainder = remainder[len(prefix):] + return remainder.replace(".", "__") + + +# --------------------------------------------------------------------------- +# BigQuery / T-SQL dotted-alias mangling bijection (DEV-1571). +# +# Relocated from ``slayer/sql/dialects/_alias_mangle.py`` (DEV-1713 D-a) so the +# naming module owns the result-key <-> wire-identifier bijection. Used by +# ``BigqueryDialect`` (backtick-anchored regex) and ``TsqlDialect`` (bracket- +# anchored regex): both need IDENTICAL encode/decode logic — BigQuery rejects +# dotted output-column names; T-SQL's ORDER BY parser does not resolve bracketed +# dotted identifiers as SELECT aliases. The fix is the same: mangle ``.`` to +# ``___`` on emit, decode on result-row keys. +# +# The bijection's only domain constraint is that ``decode_alias`` inverts +# ``encode_alias`` ONLY on the latter's image. A key like ``my___metric`` (no +# dot in the original) is OUTSIDE the image — decoding it would corrupt the +# value to ``my.metric``. This never bites because SLayer projection aliases are +# always model-qualified (``.``), so they always contain a dot +# and always pass through ``encode_alias``. +# --------------------------------------------------------------------------- + +_ALIAS_SEP = "___" + + +def encode_alias(alias: str) -> str: + """Forward encode: escape any pre-existing ``___`` to ``______``, then + map ``.`` to ``___``. Inverse is :func:`decode_alias`.""" + return alias.replace(_ALIAS_SEP, _ALIAS_SEP * 2).replace(".", _ALIAS_SEP) + + +def decode_alias(key: str) -> str: + """Reverse of :func:`encode_alias`. Walks ``key`` left-to-right, consuming + the escape-doubled ``______`` BEFORE the plain ``___`` so the two encodings + stay unambiguous. Inverse of ``encode_alias`` only on its image (see the + module-level bijection note).""" + out: list[str] = [] + i = 0 + n = len(key) + esc = _ALIAS_SEP * 2 + while i < n: + if key.startswith(esc, i): + out.append(_ALIAS_SEP) + i += len(esc) + elif key.startswith(_ALIAS_SEP, i): + out.append(".") + i += len(_ALIAS_SEP) + else: + out.append(key[i]) + i += 1 + return "".join(out) + + +# --------------------------------------------------------------------------- +# Mixed-case identifier quoting (DEV-1645). +# +# Relocated from ``SQLGenerator`` (DEV-1713 D-b) so the naming module owns the +# identifier-quoting policy. Case-folding dialects (Postgres/Redshift fold to +# lower; Snowflake/Oracle to upper) reach the wrong physical object unless a +# mixed-case identifier is quoted. The generator keeps thin delegators. +# --------------------------------------------------------------------------- + + +def maybe_quote_ident(ident: Optional[exp.Expression]) -> None: + """Set ``quoted=True`` in place on ``ident`` when it is an unquoted + ``Identifier`` containing an uppercase letter. No-op otherwise (None, + already-quoted, all-lowercase, non-Identifier).""" + if ( + isinstance(ident, exp.Identifier) + and not ident.quoted + and any(c.isupper() for c in ident.this) + ): + ident.set("quoted", True) + + +def quote_mixed_case_identifiers(node: exp.Expression) -> exp.Expression: + """Quote mixed-case DB identifiers so case-folding dialects reach the right + physical object. Context-aware: quotes only the column-name leaf of a + ``Column`` and the physical-table name parts of a ``Table`` — never table + aliases or the qualifier side of a column reference (SLayer-internal + aliases that fold consistently within a query). Idempotent; intended as a + ``.transform(...)`` callback.""" + if isinstance(node, exp.Column): + maybe_quote_ident(node.this) + elif isinstance(node, exp.Table): + maybe_quote_ident(node.this) + maybe_quote_ident(node.args.get("db")) + maybe_quote_ident(node.args.get("catalog")) + return node + + +# --------------------------------------------------------------------------- +# CTE name-collision belt (DEV-1692). +# --------------------------------------------------------------------------- + + +def assert_unique_cte_names(sql: str, *, dialect: str = "postgres") -> None: + """Assert every CTE name is unique WITHIN each ``WITH`` scope. + + CTE names must be unique inside a single ``WITH`` clause, but the same name + may legally recur in a separate nested ``WITH`` scope (an inner subquery); + each ``exp.With`` is validated independently. Raises ``ValueError`` on a + same-scope duplicate — the loud failure the DEV-1692 de-collision guards + against (a duplicate ``shifted_*`` CTE otherwise silently shadows). + + On case-folding dialects (:func:`dialect_folds_case`) names are compared + case-folded, REGARDLESS of identifier quoting (DEV-1726). That is + deliberately over-strict for quoted names on Postgres/Snowflake/Oracle: + this belt validates SLayer's own allocator-sanitized output — which never + quotes CTE names — so a fold-collision here always signals an + allocator-bypass bug, never a legitimately-distinct quoted pair. It is + not a general-purpose validator of arbitrary SQL. + """ + fold = dialect_folds_case(dialect) + parsed = sqlglot.parse_one(sql, dialect=dialect) + for with_node in parsed.find_all(exp.With): + names = [cte.alias_or_name for cte in with_node.expressions] + seen: dict[str, str] = {} + for name in names: + key = name.lower() if fold else name + if key in seen: + first = seen[key] + fold_note = ( + f" ({first!r} and {name!r} case-fold to {key!r} on " + f"{dialect})" + if first != name + else "" + ) + raise ValueError( + f"Duplicate CTE name {name!r} within one WITH scope" + f"{fold_note}: {names}" + ) + seen[key] = name diff --git a/slayer/sql/reserved_keywords.py b/slayer/sql/reserved_keywords.py new file mode 100644 index 00000000..4b2a2736 --- /dev/null +++ b/slayer/sql/reserved_keywords.py @@ -0,0 +1,140 @@ +"""DEV-1686: quote SQL reserved words used as identifiers. + +sqlglot's per-dialect ``Generator.RESERVED_KEYWORDS`` is empty for Postgres, +T-SQL, SQLite, ClickHouse, Snowflake, Databricks, Spark, and Oracle, so a +SLayer model / column / alias named after a reserved word (``grant``, ``order``, +``user``, ``select``, ...) is emitted BARE and produces syntactically-invalid +SQL (``syntax error at or near "grant"``). Table *names* are quoted; table +*aliases* and *qualifiers* were not. + +This module is the single source of truth for the reserved-word set, consumed by +two mechanisms: + +1. :func:`install_reserved_keywords` unions the set into every dialect generator + SLayer targets, so sqlglot's ``identifier_sql`` quotes reserved-word + identifiers built as AST (base FROM alias + qualifiers, cross-model CTEs, + physical names) at emit time. +2. :func:`prequote_reserved_identifiers` token-quotes reserved qualifiers/leaves + in a SLayer-generated string *before* it is re-parsed (``join_cond``, + ``measure.filter_sql``, qualified WHERE, the first/last ranked subquery, and + the pre-generator ``Column.sql`` parses in ``column_expansion``). Emit-time + quoting cannot help there because a bare reserved word fails at *parse* time. + +NOTE: :func:`install_reserved_keywords` mutates sqlglot ``Generator`` classes +process-globally. This is deliberate and idempotent; the only observable effect +on unrelated in-process sqlglot use is strictly-more-correct quoting. +""" + +from __future__ import annotations + +import sqlglot +from sqlglot import exp +from sqlglot.tokens import TokenType + +# ANSI SQL:2016 + Postgres reserved words (lowercase). Curated to the +# "cannot be a bare identifier" set. Type-ish NON-reserved words that show up as +# real column names (date, time, timestamp, name, value, count, sum, id, text, +# number, ...) are intentionally EXCLUDED so we never quote a common column. +SLAYER_RESERVED_KEYWORDS: frozenset[str] = frozenset({ + "all", "alter", "analyse", "analyze", "and", "any", "array", "as", "asc", + "asymmetric", "authorization", "between", "binary", "both", "case", "cast", + "check", "collate", "collation", "column", "concurrently", "constraint", + "create", "cross", "current_catalog", "current_date", "current_role", + "current_time", "current_timestamp", "current_user", "default", + "deferrable", "desc", "distinct", "do", "drop", "else", "end", "except", + "false", "fetch", "for", "foreign", "freeze", "from", "full", "glob", + "grant", "group", "having", "ilike", "in", "initially", "inner", "insert", + "intersect", "into", "is", "isnull", "join", "lateral", "leading", "left", + "like", "limit", "localtime", "localtimestamp", "natural", "not", "notnull", + "null", "offset", "on", "only", "or", "order", "out", "outer", "overlaps", + "partitioned_by", "placing", "primary", "qualify", "references", "regexp", + "returning", "revoke", "right", "rlike", "rollback", "select", + "session_user", "similar", "some", "symmetric", "table", "tablesample", + "then", "to", "trailing", "true", "uncache", "union", "unique", "user", + "using", "variadic", "verbose", "when", "where", "window", "with", "xor", +}) + + +def install_reserved_keywords() -> None: + """Idempotently union :data:`SLAYER_RESERVED_KEYWORDS` into the + ``RESERVED_KEYWORDS`` of every generator SLayer targets. + + Assigns a FRESH set per generator class so we never mutate sqlglot's shared + base empty-set singleton (Postgres / T-SQL / SQLite / ... all inherit the + same ``Generator.RESERVED_KEYWORDS`` object). Each dialect keeps its own + native reserved words (union, not replace). + """ + from sqlglot.dialects.dialect import Dialect + + from slayer.sql.dialects import _ALL_DIALECTS + + for d in _ALL_DIALECTS: + gen_cls = Dialect.get_or_raise(d.sqlglot_name).generator_class + gen_cls.RESERVED_KEYWORDS = set(gen_cls.RESERVED_KEYWORDS) | SLAYER_RESERVED_KEYWORDS + + +def _reserved_dot_edit( + sql: str, toks: list, i: int, dialect: str +) -> tuple[int, int, str] | None: + """Return the ``(start, end, quoted)`` edit for token ``i`` when it is a + reserved word in QUALIFIER (``word.``) or LEAF (``.word``) position whose + offsets map cleanly back to ``sql``; otherwise ``None``. Factored out of + :func:`prequote_reserved_identifiers` to keep that function's cognitive + complexity within the analysis budget.""" + tok = toks[i] + if tok.text.lower() not in SLAYER_RESERVED_KEYWORDS: + return None + # Defensive: offsets must map back to the original text (Token.end is + # inclusive). Skip anything that doesn't round-trip cleanly. + if sql[tok.start:tok.end + 1] != tok.text: + return None + prev_tok = toks[i - 1] if i else None + next_tok = toks[i + 1] if i + 1 < len(toks) else None + adj_dot = ( + (prev_tok is not None and prev_tok.token_type == TokenType.DOT) + or (next_tok is not None and next_tok.token_type == TokenType.DOT) + ) + if not adj_dot: + return None + quoted = exp.Identifier(this=tok.text, quoted=True).sql(dialect=dialect) + return (tok.start, tok.end, quoted) + + +def prequote_reserved_identifiers(sql: str, *, dialect: str) -> str: + """Quote reserved-word identifiers sitting in QUALIFIER (``word.``) or LEAF + (``.word``) position so a generated string embedding a bare reserved + qualifier/leaf parses. + + Token-based (via ``sqlglot.tokenize``) so it is literal/comment/quoted-ident + safe: a reserved word inside ``'...'`` / ``E'...'`` / ``$$...$$`` / ``--`` / + ``/* */`` / an already-quoted identifier is a distinct token type and is + never rewritten. Quotes for the PARSE ``dialect`` (some callers parse as + postgres while the target is mysql/tsql/bigquery), and the resulting quoted + identifier re-emits with the target dialect's quote char downstream. + + Does NOT mutate stored metadata — callers pass a copy of the SQL string, so + metadata scans (e.g. ``_window_referenced_aliases`` over + ``measure.filter_sql``) keep seeing the original unquoted text. + + Known limitation: only reserved words in QUALIFIER/LEAF (dot-adjacent) + position are quoted. A physical column whose bare name is a *statement-initial* + keyword (``grant``/``select``/``insert``/``create``/``drop``/...) referenced + UNQUALIFIED inside a compound expression (``Column.sql = "grant + 1"``) is not + quoted here — sqlglot parses such a bare word as a statement even in + expression context, and blindly quoting non-dot-adjacent reserved words would + corrupt genuine keywords (``CASE``/``WHEN``/``AND``/...). The trivial form + (``sql="grant"``) and the qualified form (``sql="t.grant + 1"`` → dot-adjacent) + both work; only bare statement-keyword columns in expressions do not. + """ + try: + toks = sqlglot.tokenize(sql, dialect=dialect) + except Exception: # unsupported lexer construct — leave unchanged + return sql + edits = [ + edit + for i in range(len(toks)) + if (edit := _reserved_dot_edit(sql, toks, i, dialect)) is not None + ] + for start, end, replacement in sorted(edits, reverse=True): + sql = sql[:start] + replacement + sql[end + 1:] + return sql diff --git a/slayer/sql/scope.py b/slayer/sql/scope.py new file mode 100644 index 00000000..c9e93db8 --- /dev/null +++ b/slayer/sql/scope.py @@ -0,0 +1,223 @@ +"""DEV-1706 Stage 2 — ``ScopeFrame`` + the single resolver (Laws 1 & 2). + +A query renders as a tree of SELECT scopes, each rooted at one relation. Every +expression enters a scope through :meth:`ScopeFrame.resolve`, which: + +* **Law 1 (anchored rendering):** expands derived refs (reserved-word + identifiers prequoted — DEV-1686; multi-term derived expansions parenthesised + — DEV-1539), anchors every reference at the scope root or a ``__``-path join + alias, and REGISTERS each crossed join path into ``join_paths`` in the same + call. Discovery is a side effect of rendering — it can never be forgotten. +* **Law 2 (projection boundaries):** when a ``consumer`` scope is named, the + value is materialised as a ``_val_`` projection in THIS (producing) scope + and a bare alias is returned for the consumer. Materialisations dedup by a + scope-safe key (producing-scope id + anchored AST + dialect — Codex F6). + +Stage 2 migrates the host base SELECT, which is a single scope with no +projection boundary, so the materialise branch is exercised only by direct unit +tests here; Stage 4 is its first generated-SQL consumer. The resolver reuses the +existing engine-layer expansion/scan helpers (D-G wrap-and-reuse). +""" + +from __future__ import annotations + +from typing import List, Optional, Tuple, Union + +import sqlglot +from pydantic import BaseModel, ConfigDict, Field +from sqlglot import exp + +from slayer.core.keys import ColumnKey, ColumnSqlKey +from slayer.core.models import SlayerModel +from slayer.engine.column_expansion import ( + collect_root_scope_joined_paths, + expand_derived_refs_sync, +) +from slayer.engine.source_bundle import ResolvedSourceBundle +from slayer.sql.dialects.base import SqlDialect +from slayer.sql.naming import AliasAllocator +from slayer.sql.reserved_keywords import ( + install_reserved_keywords, + prequote_reserved_identifiers, +) + +# The resolver relies on sqlglot's reserved-word quoting on emit (DEV-1686). +install_reserved_keywords() + +# A ref that can enter a scope. Stage 2 exercises structural column refs, derived +# columns, and free Mode-A / predicate text; later stages widen this union. +Ref = Union[ColumnKey, ColumnSqlKey, str] + + +class _OrderedPathSet: + """Insertion-ordered, de-duplicated set of ``__``-join-path tuples. + + Backed by a dict so membership is O(1) and iteration/`as_list` preserve + first-seen order — the join emission order ``_build_from_and_joins`` reads. + """ + + def __init__(self) -> None: + self._d: "dict[Tuple[str, ...], None]" = {} + + def add(self, path: Tuple[str, ...]) -> None: + self._d.setdefault(path, None) + + def __contains__(self, path: object) -> bool: + return path in self._d + + def __iter__(self): + return iter(self._d) + + def __len__(self) -> int: + return len(self._d) + + def as_list(self) -> List[Tuple[str, ...]]: + return list(self._d) + + +class Materialization(BaseModel): + """A Law-2 ``_val_`` projection produced in a scope for a consumer.""" + + model_config = ConfigDict(frozen=True, arbitrary_types_allowed=True) + + alias: str + expr: exp.Expression # anchored template, projected in the producing scope + # (producing_scope_id, ast.sql(dialect=sqlglot_name), sqlglot_name) — Codex F6/M3. + dedup_key: Tuple[str, str, str] + + +class ScopeFrame(BaseModel): + """One SELECT scope rooted at ``root_relation`` (Laws 1 & 2).""" + + model_config = ConfigDict(arbitrary_types_allowed=True) + + scope_id: str # generation-local, ephemeral, never emitted (D-F / Codex L1) + root_model: SlayerModel + root_relation: str + bundle: ResolvedSourceBundle + dialect: SqlDialect + allocator: AliasAllocator + join_paths: _OrderedPathSet = Field(default_factory=_OrderedPathSet) + materializations: List[Materialization] = Field(default_factory=list) + + # ---- Law 1 ------------------------------------------------------------- + def resolve(self, ref: Ref, *, consumer: "ScopeFrame | None" = None) -> exp.Expression: + """Anchor ``ref`` in this scope, register the joins it crosses, and — + when a ``consumer`` scope is named — materialise it and return the bare + alias for the consumer. + """ + template = self._anchor(ref) + for path in collect_root_scope_joined_paths( + parsed=template, + source_model=self.root_model, + source_relation=self.root_relation, + bundle=self.bundle, + ): + self.join_paths.add(path) + + if consumer is not None and not self.may_inline(self.join_paths.as_list()): + alias = self._materialize(template) + return exp.column(alias) + # Return a copy so a caller attaching this into its tree can never + # corrupt a value the scope (or another caller) also holds (D-L / M1). + return template.copy() + + def resolve_predicate_sql(self, ref: Ref) -> Optional[str]: + """Resolve a predicate ref to a SQL string for WHERE/HAVING builders.""" + expr = self.resolve(ref) + return None if expr is None else expr.sql(dialect=self.dialect.sqlglot_name) + + def _anchor(self, ref: Ref) -> exp.Expression: + if isinstance(ref, ColumnKey): + alias = self.root_relation if not ref.path else "__".join(ref.path) + return exp.Column( + this=exp.to_identifier(ref.leaf), + table=exp.to_identifier(alias), + ) + if isinstance(ref, ColumnSqlKey): + model = self._model_for(ref.model) + col = next( + (c for c in model.columns if c.name == ref.column_name), None, + ) + if col is None: + raw_sql = ref.column_name + elif col.sql: + raw_sql = col.sql + else: + raw_sql = col.name + # DEV-1711: a derived column ON a JOINED model (``path`` non-empty, + # e.g. ``stores.tier`` where ``tier`` lives on the joined ``stores``) + # must anchor at the ``__``-path alias with ``is_root=False`` so a + # bare inner ref (``name``) qualifies to ``stores.name`` — and a + # further-joined inner ref (``regions.population``) to the full + # ``stores__regions`` path (the DEV-1701 shape). A local derived + # column (empty path) keeps anchoring at the scope root. + if ref.path: + alias_path = "__".join(ref.path) + is_root = False + else: + alias_path = self.root_relation + is_root = True + expanded = expand_derived_refs_sync( + sql=raw_sql, + model=model, + alias_path=alias_path, + resolve_model=self.bundle.get_referenced_model, + dialect=self.dialect.sqlglot_name, + is_root=is_root, + ) + return self._parse(expanded or raw_sql) + if isinstance(ref, str): + prequoted = prequote_reserved_identifiers( + ref, dialect=self.dialect.sqlglot_name, + ) + expanded = expand_derived_refs_sync( + sql=prequoted, + model=self.root_model, + alias_path=self.root_relation, + resolve_model=self.bundle.get_referenced_model, + dialect=self.dialect.sqlglot_name, + is_root=True, + ) + return self._parse(expanded or prequoted) + raise NotImplementedError( + f"ScopeFrame.resolve does not yet handle ref type {type(ref).__name__}", + ) + + def _model_for(self, name: str) -> SlayerModel: + if name == self.root_model.name: + return self.root_model + return self.bundle.get_referenced_model(name) or self.root_model + + def _parse(self, sql: str) -> exp.Expression: + return sqlglot.parse_one(sql, dialect=self.dialect.sqlglot_name) + + # ---- Law 2 ------------------------------------------------------------- + def may_inline(self, crossed_paths: List[Tuple[str, ...]]) -> bool: # NOSONAR(S1172) — crossed_paths is the documented v1 API seam; the Stage-N inlining optimisation reads it, hardcoded False until then. + """Whether a crossing value may be inlined back into the consumer scope + instead of materialised. Hardcoded ``False`` in v1 (the seam Stage-N+ + optimisation grows into).""" + return False + + def _materialize(self, template: exp.Expression) -> str: + key = ( + self.scope_id, + template.sql(dialect=self.dialect.sqlglot_name), + self.dialect.sqlglot_name, + ) + for m in self.materializations: + if m.dedup_key == key: + return m.alias + alias = self.allocator.allocate_val() + self.materializations.append( + Materialization(alias=alias, expr=template, dedup_key=key), + ) + return alias + + def apply_materializations(self, select: exp.Select) -> exp.Select: + """Project each materialisation as ``