Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
e0c1007
WIP(DEV-1688): recovered uncommitted work from 2026-07-31 disk-corrup…
Jul 31, 2026
c22b080
Merge origin/main into egor/dev-1688-cleaner-handling-of-join-arity
ZmeiGorynych Aug 4, 2026
4f23040
DEV-1688: fix composite-PK false positive, re-home docs after merge
ZmeiGorynych Aug 4, 2026
fbd0918
DEV-1688: address Codex + Sonar review findings
ZmeiGorynych Aug 4, 2026
b4a3da5
DEV-1688: reject unique-index key-sets with expression members
ZmeiGorynych Aug 4, 2026
669201a
DEV-1688: reject partial unique indexes; skip cross-schema FKs
ZmeiGorynych Aug 4, 2026
3bfd489
DEV-1688: make _safe_get_pk_constraint always return a mapping
ZmeiGorynych Aug 4, 2026
f48fb61
DEV-1688: close two gaps in the cross-schema / partial-index guards
ZmeiGorynych Aug 4, 2026
091a5fb
DEV-1688: apply the session policy to cardinality profiling scans
ZmeiGorynych Aug 4, 2026
cd38bdb
DEV-1688: an empty key population is no evidence (new no_evidence ver…
ZmeiGorynych Aug 4, 2026
f031b65
DEV-1688: address round-6 CodeRabbit review (2 threads + 3 nitpicks)
ZmeiGorynych Aug 4, 2026
055b86b
DEV-1688: split composite assertion (Sonar python:S9073)
ZmeiGorynych Aug 4, 2026
de80274
DEV-1688: cover populated-source / empty-target (CodeRabbit nitpick)
ZmeiGorynych Aug 4, 2026
50602ca
Merge origin/main into egor/dev-1688-cleaner-handling-of-join-arity
ZmeiGorynych Aug 5, 2026
26cbfcb
DEV-1688: fold cardinality detection into validate-models; purge issu…
ZmeiGorynych Aug 16, 2026
21911d2
DEV-1688: address round-7 review (Codex x2, Sonar x6, one self-found)
ZmeiGorynych Aug 16, 2026
3c3ad15
DEV-1688: address round-8 review (Codex x1, Sonar x1)
ZmeiGorynych Aug 16, 2026
3ac094f
DEV-1688: keyword-call the last PR-added inspector.get_pk_constraint
ZmeiGorynych Aug 16, 2026
d996aed
Merge origin/main into egor/dev-1688-cleaner-handling-of-join-arity
ZmeiGorynych Aug 16, 2026
10d38c7
fix: FK joins must name the model, not the live object (DEV-1688 / DE…
ZmeiGorynych Aug 16, 2026
0f3e02f
fix: repair legacy join targets on re-ingest; don't conflate [] joins…
ZmeiGorynych Aug 16, 2026
fb5206c
fix: require a full signature match before repairing a legacy join ta…
ZmeiGorynych Aug 16, 2026
08ad6d0
test: make the legacy-target test actually guard the signature-match fix
ZmeiGorynych Aug 16, 2026
20aabf8
docs: log the join-target naming decision in DECISIONS.md
ZmeiGorynych Aug 16, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .claude/skills/slayer-models.md
Original file line number Diff line number Diff line change
Expand Up @@ -57,9 +57,10 @@ Models can declare LEFT JOIN relationships to other models:
joins:
- target_model: customers
join_pairs: [["customer_id", "id"]]
cardinality: many_to_one # optional; source→target arity
```

Enables cross-model measures (`customers.score:avg`), multi-hop dimensions (`customers.regions.name`), and transforms on joined measures (`cumsum(customers.score:avg)`). Auto-ingestion creates one direct join per FK on the source table. Multi-hop paths (e.g. `orders → customers → regions`) are resolved at query time by walking each intermediate model's own joins. Diamond joins (same table via different paths) are supported — each path gets a unique `__`-delimited alias (e.g., `customers__regions` vs `warehouses__regions`).
Enables cross-model measures (`customers.score:avg`), multi-hop dimensions (`customers.regions.name`), and transforms on joined measures (`cumsum(customers.score:avg)`). Auto-ingestion creates one direct join per FK on the source table (composite FKs stay a single join with multiple `join_pairs`). `cardinality` (`one_to_one` / `one_to_many` / `many_to_one` / `many_to_many`, omit when undetermined) is descriptive metadata, orthogonal to the always-LEFT join type; auto-ingestion fills it structurally, and `slayer validate-models --cardinality [--persist-cardinality]` infers it from the data. See [models.md#join-cardinality](../../docs/concepts/models.md#join-cardinality). Multi-hop paths (e.g. `orders → customers → regions`) are resolved at query time by walking each intermediate model's own joins. Diamond joins (same table via different paths) are supported — each path gets a unique `__`-delimited alias (e.g., `customers__regions` vs `warehouses__regions`).

**Derived-on-derived chaining.** A `Column.sql` may reference another *derived* column — local same-model or via the join graph (single-dot `B.col` or `__`-delimited `B__C.col` path). Same-model refs can be **bare** (`A.ratio = "bar / foo_normalized"`) or **qualified** (`A.ratio = "A.bar / A.foo_normalized"`) — both inline identically. The engine recursively inlines those references at query time, so you can write `A.ratio = "A.bar / B.foo_normalized"` even when `B.foo_normalized.sql = "foo_raw / 100.0"`. No need to inline derivations at every consumer site. Refs inside a nested scope (sub-query, `UNION` branch, CTE, `VALUES`) are left alone — they belong to the inner rowset. Cycles raise `ColumnCycleError` (a subclass of `ValueError`) at `save_model` time, so a cyclic model never reaches a query.

Expand Down Expand Up @@ -148,7 +149,7 @@ models = ingest_datasource(datasource=ds, schema="public")
```

Generates:
- One `Column` per non-joined database column (with `type` inferred). PK columns get `primary_key=True`. A column literally named `count` is renamed to `count_col` to avoid clashing with `*:count`.
- One `Column` per non-joined database column (with `type` inferred). PK columns get `primary_key=True`; single-column `UNIQUE` constraints set `unique=True`. A column literally named `count` is renamed to `count_col` to avoid clashing with `*:count`.
- `*:count` is always available without an explicit definition; aggregation is picked per query via colon syntax (e.g., `amount:sum`).
- **Dynamic joins**: detects FK relationships and emits explicit join metadata (LEFT JOINs built at query time).
- FK columns are excluded from joinable models; ID-like columns (`*_id`, `*_key`) are usable as group-by columns only via the `primary_key` flag.
Expand Down
2 changes: 2 additions & 0 deletions DECISIONS.md

Large diffs are not rendered by default.

7 changes: 5 additions & 2 deletions docs/concepts/ingestion.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,14 +42,17 @@ Tables with no FK references use their plain table name with no joins.
SLayer introspects each table's column types and generates a model:

- **One `Column`** per non-joined column on the source table — name, `type` inferred from the database (`string` / `number` / `boolean` / `time` / `date`), `primary_key=True` for PKs. Whether each column is used as a group-by dimension or as an aggregation source is decided per query.
- **`unique=True`** for columns that alone form a `UNIQUE` constraint or unique index. PK columns are not stamped redundantly — `primary_key` already implies uniqueness. Composite uniqueness is evaluated per key-set during join-cardinality inference rather than being flattened onto individual columns.
- **A column literally named `count`** is renamed to `count_col` to avoid clashing with the always-available `*:count`.
- **No auto-generated `measures`** — `SlayerModel.measures` is the named-formula library and stays empty after ingestion. You can add named formulas later via the API/MCP if you want bare-name shortcuts (`{"formula": "aov"}`).
- **`*:count`** is always available without any model definition.
- The `allowed_aggregations` whitelist is left at the default for the column's data type. PK columns are restricted to `count`/`count_distinct` automatically.

FK columns from referenced tables are excluded from the source model to avoid redundancy — they're reachable via the join graph as `customers.id` etc.

All models use `sql_table` (the source table) plus `joins` (direct FK joins only, storing source/target column pairs). Multi-hop JOINs are resolved dynamically at query time by walking the join graph.
All models use `sql_table` (the source table) plus `joins` (direct FK joins only, storing source/target column pairs). Multi-hop JOINs are resolved dynamically at query time by walking the join graph. A **composite** foreign key becomes a single join carrying all of its column pairs, not one join per column.

Each FK join also gets a structural [`cardinality`](models.md#join-cardinality) guess from the key constraints alone (no data is read): `many_to_one` by default, upgrading to `one_to_one` when the source key is itself unique. A side counts as unique only when some PK/unique key-set is a subset of the join key — if `(a)` is unique then `(a, b)` is too, but a constraint on `(a, b)` does not make `(a)` unique. When the target key cannot be *verified* unique from its constraints, cardinality is left unset rather than guessed. To infer it from the data instead, run `slayer validate-models --cardinality`.

### SQLite affinity probing

Expand Down Expand Up @@ -273,7 +276,7 @@ Ingest-on-startup: N/M datasources ingested (K failed: name1, name2)
`slayer ingest` (and the equivalent MCP / REST entry points) is idempotent by default — re-runs are safe. For each in-scope live table:

- **No persisted model with that name** → ingest from scratch via the path above.
- **Existing `sql_table`-mode model** → append new columns and joins from the live schema. Existing columns and joins are **never** mutated — `description`, `label`, `format`, `meta`, and `allowed_aggregations` are preserved verbatim.
- **Existing `sql_table`-mode model** → append new columns and joins from the live schema. Existing user metadata is **never** overwritten — `description`, `label`, `format`, `meta`, and `allowed_aggregations` are preserved verbatim. The only in-place updates are strictly additive gap-fills: a join's `cardinality` is set only when it is currently unset (a value you chose is never replaced), and a column's `unique` is only ever turned on, never off. Filling either one is enough to trigger a save, so a re-ingest that adds no columns or joins still persists newly-discovered constraint metadata.
- **Existing `sql`-mode or query-backed model with the matching name** → skipped silently; those are user-authored.

With the default YAML storage, two live tables whose quoted names differ only by letter case (`"Orders"` vs `orders`) cannot both be persisted — model names collide as filenames on macOS / Windows, so the save is rejected (`IdCollisionError`). The first table wins; the second surfaces as a per-model entry in `IdempotentIngestResult.errors` (or a per-model message on the CLI / MCP paths) without aborting the rest of the ingest. SQLite storage persists both.
Expand Down
25 changes: 25 additions & 0 deletions docs/concepts/models.md
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ A column is the unit of structure on the model. The same column entry can serve
| `sql` | string | No | (bare column name) | SQL expression — defaults to the column's name |
| `type` | string | No | `string` | `string`, `number`, `boolean`, `time`, `date` |
| `primary_key` | bool | No | `false` | Restricts aggregation to `count` / `count_distinct` |
| `unique` | bool | No | `false` | Single-column uniqueness (non-PK). `primary_key` implies unique. Auto-set from `UNIQUE` constraints / unique indexes; used to infer one-to-one joins |
| `hidden` | bool | No | `false` | Hide from listings |
| `format` | dict | No | — | `NumberFormat` used by response metadata |
| `allowed_aggregations` | list[str] | No | — | Whitelist (must be a subset of the type-default eligibility set, or a custom aggregation defined on this model) |
Expand Down Expand Up @@ -301,6 +302,30 @@ joins:

Joins enable **cross-model measures** — querying a measure from a joined model alongside the main model's data. See [Cross-Model Measures](queries.md#cross-model-measures). During [auto-ingestion](ingestion.md), joins are generated automatically from foreign-key relationships; multi-hop paths are resolved at query time by walking each intermediate model's own joins.

### Join cardinality

A join optionally records its **arity**, read source→target:

```yaml
joins:
- target_model: customers
join_pairs: [["customer_id", "id"]]
cardinality: many_to_one # many orders → one customer
```

`cardinality` is one of `one_to_one`, `one_to_many`, `many_to_one`, `many_to_many` (omit it when undetermined). It is **descriptive metadata, orthogonal to the join type** — joins stay LEFT regardless — and is representational today (query results are unaffected).

Auto-ingestion fills it structurally from key constraints: an FK join defaults to `many_to_one`, upgrading to `one_to_one` when the source key is itself unique. To infer it from the actual data instead, run:

```bash
slayer validate-models --datasource mydb --cardinality # report only
slayer validate-models --datasource mydb --cardinality --persist-cardinality # write it back
```

Detection full-scans each side of the join and reports the observed arity, a `verdict` (whether it confirms, refines, or hard-contradicts the stored value), and any column declared `unique` that the data shows has duplicates. It is a strong guess, not a guarantee — a duplicate disproves uniqueness with certainty, but the absence of duplicates only suggests it.

A side with no non-null key rows reports `no_evidence` and detects nothing: an empty scan would trivially look unique, and that is not weak evidence — it is none. Re-run once the table has data. A join whose scan fails outright reports `scan_failed` and does not stop the rest of the report. Full verdict table: [CLI reference](../reference/cli.md#slayer-validate-models).

### Path-based table aliases

Joined tables use `__`-delimited path aliases in generated SQL so **diamond joins** stay unambiguous — when the same table is reachable via multiple paths. For example, if `orders` joins both `customers` and `warehouses`, each referencing `regions`:
Expand Down
16 changes: 11 additions & 5 deletions docs/concepts/schema-drift.md
Original file line number Diff line number Diff line change
Expand Up @@ -103,11 +103,17 @@ be no-ops once the model is gone).
* **REST.** `POST /validate-models` — read-only. Query-time failures
attributed to drift surface as **HTTP 422** with body
`{error: "schema_drift", models, to_delete, original}`.
* **CLI.** `slayer validate-models [--datasource X] [--force-clean]
[--yes]`. Without `--force-clean`, prints the diff and exits 0.
`--force-clean` prompts (or skips with `--yes`), applies via
`apply_drift_deletes`, and exits non-zero on per-entry errors or
non-empty residual drift.
* **CLI.** `slayer validate-models [--datasource X] [--model M]
[--format text|json] [--force-clean] [--yes]`. Without `--force-clean`,
prints the diff and exits 0. `--force-clean` prompts (or skips with
`--yes`), applies via `apply_drift_deletes`, and exits non-zero on
per-entry errors or non-empty residual drift. `--model` scopes both the
report and the applied deletes. Unlike the engine method, an unscoped CLI
run validates each datasource explicitly and exits 1 if any of them fails,
rather than dropping it from the result. `--cardinality` adds an opt-in
[join-arity profiling](models.md#join-cardinality) pass, which runs after
any `--force-clean` apply. See the
[CLI reference](../reference/cli.md#slayer-validate-models).

`--force-clean` is intentionally CLI-only — destructive auto-application
must be opt-in at the human-typed layer.
Expand Down
3 changes: 3 additions & 0 deletions docs/dbt/dbt_import.md
Original file line number Diff line number Diff line change
Expand Up @@ -48,8 +48,11 @@ The converter builds an entity registry by scanning all models, then resolves fo
joins:
- target_model: customers
join_pairs: [["customer_id", "id"]]
cardinality: many_to_one
```

Each generated join carries a [`cardinality`](../concepts/models.md#join-cardinality) read source→target: a foreign→primary entity reference is `many_to_one`, while a **peer** join (two models sharing the same primary/unique entity) is `one_to_one`. The reverse INNER edge the converter mirrors onto the target model carries the inverted arity.

### Measures — Column + ModelMeasure Split

dbt bakes aggregation into each measure (`agg: sum`). SLayer separates them — a row-level expression lives on a `Column`, and the aggregation is named on a `ModelMeasure` formula.
Expand Down
24 changes: 24 additions & 0 deletions docs/interfaces/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,30 @@ slayer ingest --datasource my_postgres --exclude migrations,django_session
| `--exclude` | No | Comma-separated tables to exclude |
| `--storage` | No | Storage path |

### `slayer validate-models`

Diff persisted models against the live database schemas (read-only), and optionally profile each join's arity from the data. See [Schema Drift](../concepts/schema-drift.md) and [Join cardinality](../concepts/models.md#join-cardinality).

```bash
slayer validate-models
slayer validate-models --datasource jaffle_shop
slayer validate-models --model orders --format json
slayer validate-models --cardinality --persist-cardinality
slayer validate-models --force-clean --yes
```

| Flag | Default | Description |
|------|---------|-------------|
| `--datasource X` | all | Limit to one datasource. |
| `--model M` | all | Limit the whole report — and `--force-clean` — to one model. |
| `--cardinality` | off | Also profile join arity (full table scans). |
| `--persist-cardinality` | off | Write the detected `cardinality` back onto each join. Implies `--cardinality`. |
| `--format` | `text` | `text` or `json`. Not combinable with `--force-clean`. |
| `--force-clean` | off | Prompt to apply each delete. Destructive. |
| `-y` / `--yes` | off | Skip the `--force-clean` prompt. |

With a cardinality flag the output gains two labelled sections, and each join reports a `verdict` — `fills_none`, `confirms`, `refines`, `contradicts_hard`, `skipped_unsupported`, `no_evidence`, or `scan_failed`. Full table: [CLI reference](../reference/cli.md#slayer-validate-models).

### `slayer import-dbt`

Import dbt Semantic Layer definitions into SLayer.
Expand Down
2 changes: 1 addition & 1 deletion docs/osi/osi_import.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ Spec versions `1.0`, `0.1.0`, `0.1.1`, and `0.2.0.dev0` are all accepted (they a
| field `expression` (derived, e.g. `UPPER(x)`) | a derived `Column` with `sql` set |
| field `dimension.is_time` | column typed temporal; sets `default_time_dimension` |
| dataset `primary_key` | `Column.primary_key = true` |
| `relationships[]` (`from` → `to`) | a LEFT `ModelJoin` on the `from` model |
| `relationships[]` (`from` → `to`) | a LEFT `ModelJoin` on the `from` model, [`cardinality`](../concepts/models.md#join-cardinality) `many_to_one` (OSI relationships are direction-implied: `from` = many, `to` = one) |
| `metrics[]` (raw SQL aggregation) | a `ModelMeasure` formula |
| `ai_context` (instructions + synonyms) | entity `description` + `meta["osi_ai_context"]` |
| `unique_keys` / `custom_extensions` | model/column `meta` |
Expand Down
45 changes: 45 additions & 0 deletions docs/reference/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -201,6 +201,51 @@ leave an internal out of the store entirely.
An empty result prints the available schemas so a mistyped `--schema` is
obvious rather than silent.


### `slayer validate-models`

Diff persisted models against the live database schemas (read-only) and, optionally, profile each join's arity from the data. See [Schema Drift](../concepts/schema-drift.md) and [Join cardinality](../concepts/models.md#join-cardinality).

```bash
slayer validate-models # every datasource
slayer validate-models --datasource jaffle_shop
slayer validate-models --model orders --format json
slayer validate-models --cardinality # + profile join arity
slayer validate-models --cardinality --persist-cardinality
slayer validate-models --datasource jaffle_shop --force-clean --yes
```

| Flag | Default | Description |
|------|---------|-------------|
| `--datasource X` | all | Limit to one datasource. Unknown names fail fast. |
| `--model M` | all | Limit the whole report — and `--force-clean` — to one model. Resolves across every datasource that has a model of that name. |
| `--cardinality` | off | Also profile each join's arity from the data. Full-scans both sides of every join, so it is opt-in. |
| `--persist-cardinality` | off | Write the detected `cardinality` back onto each matching join (identified by target model + key pairs). Implies `--cardinality`. |
| `--format` | `text` | `text`, or `json` for one `{"drift": [...], "cardinality": {...}}` document. Cannot be combined with `--force-clean`. |
| `--force-clean` | off | After printing the diff, prompt to apply each delete. Destructive; opt-in only. |
| `-y` / `--yes` | off | With `--force-clean`, skip the confirmation prompt. |

Without a cardinality flag the output is the drift diff alone. With one, the report gains two labelled sections — `Schema drift` and `Join cardinality` — and any `--force-clean` apply happens *between* them, so profiling reads the repaired models.

Exit code is `0` for any report about the data, including `contradicts_hard` — the command did its job and the answer is unwelcome. It is `1` when the command could not do its job: an unknown datasource or model, a datasource that failed validation (an unscoped run validates each one explicitly rather than silently skipping failures), a profiling failure, any `scan_failed` finding, or residual drift after `--force-clean`. Diagnostics go to stderr, so stdout stays parseable under `--format json`.

#### Cardinality verdicts

| Verdict | Meaning |
|---------|---------|
| `fills_none` | No cardinality was stored; the detected value fills the gap. |
| `confirms` | Detected value matches what was stored. |
| `refines` | Differs from the stored value, but the data does not disprove it — "no duplicates observed" only *suggests* uniqueness. |
| `contradicts_hard` | The data **disproves** the stored value: a side it claimed unique has duplicates. |
| `skipped_unsupported` | Not profilable — a non-`sql_table` model (sql-mode / query-backed) or an expression-valued join key. |
| `no_evidence` | Profiled fine, but one side had no non-null key rows. An empty scan says nothing about arity, so nothing is detected or written. Worth re-running once data lands — unlike `skipped_unsupported`, which never becomes profilable. |
| `scan_failed` | The scan itself raised; the message is in `note`. Contained per join, so one unreadable table costs one finding rather than the whole report — but the command still exits 1, since that join was not profiled. |

Columns declared `unique` (or `primary_key`) that the data shows have duplicates are reported under `unique_contradictions`; detection never mutates `Column.unique`.

Detection is a strong guess, not a guarantee: a duplicate disproves uniqueness with certainty, but its absence only suggests uniqueness.


### `slayer import-dbt`

Import dbt Semantic Layer definitions into SLayer.
Expand Down
Loading
Loading