Skip to content

Add ODCS type: library quality metric support #1485

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Christopher-Lawford wants to merge 4 commits into
base: main
Choose a base branch
from
Open

Add ODCS type: library quality metric support #1485

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
c6a5c1f
style: apply black formatting to datacontract generator and tests
Christopher-Lawford Aug 25, 2026
52bda65
Fix ODCS type: library metric review findings
Christopher-Lawford Sep 7, 2026
3f9c700
Merge branch 'main' into worktree-odcs-library-metrics-wayfinder
mwojtyczka Sep 9, 2026
fcb1ee6
Squashed commit of the following:
Christopher-Lawford Sep 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
255 changes: 253 additions & 2 deletions docs/dqx/docs/guide/data_contract_quality_rules_generation.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -535,6 +535,249 @@ If you don't have LLM dependencies installed or want to skip text processing:
</TabItem>
</Tabs>

## Metric Rule Generation

@vb-dbrks vb-dbrks Sep 7, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Request on framing for this section, rather than the mechanics.

I would like the docs to position library metrics as the option for simple contracts, not as the recommended way to express quality in an ODCS contract. Concretely:

  • say plainly that library metrics are intended for simple contracts
  • point at type: custom with engine: dqx as the recommended path for anything beyond the five metrics
  • drop the framing that suggests this is the primary way to express quality
  • where our semantics are a judgement call the spec does not settle (duplicateValues, the percent denominator, the invalidValues combination), say so explicitly and say what we chose

Reasoning is in the review summary.


<FeatureTags>
<AvailableSinceVersion version="0.17.0" heading={false} />
</FeatureTags>

ODCS quality entries with `type: library` (or, per the spec, no `type` at all as long as `metric` is set) reference a named, standard data quality metric instead of a natural-language description or a hand-written DQX check. DQX recognizes five of these metrics — `rowCount`, `nullValues`, `missingValues`, `invalidValues`, and `duplicateValues` — and maps each one onto the DQX check(s) that best express it.

<Admonition type="note" title="A fallback for simple contracts, not the recommended path">
Library metrics exist for **portability**: the same five-metric vocabulary is understood by any ODCS-compliant tool, including [datacontract-cli](https://github.com/datacontract/datacontract-cli). They are not a capability DQX otherwise lacks — every one of the five is already expressible directly with `is_aggr_*` checks, `is_unique`, and `column: "*"` — and they cover a narrow slice of what DQX can check (no outlier detection, freshness, foreign keys, schema validation, or dataset comparison). For anything beyond simple row-count/null/duplicate/allowlist checks on a contract that must stay portable across tools, prefer [explicit rules](#explicit-rule-generation) with `type: custom` and `engine: dqx`, which give you the full DQX check surface. Treat `type: library` as the option for the simple, common cases, not the default way to express quality in a contract.

This five-metric surface is intentionally closed: DQX does not plan to grow this mapping into a general ODCS interpreter for arbitrary quality vocabularies.
</Admonition>

Every `type: library` entry is processed whenever `generate_rules_from_contract` runs, unless disabled with `process_library_rules=False` (default `True`), matching the `generate_predefined_rules`/`process_text_rules` opt-out pattern used elsewhere. An entry that DQX cannot map is skipped with a logged warning instead of failing the whole contract (see [Malformed and Unrecognized Entries](#malformed-and-unrecognized-entries)).

<Admonition type="note" title="Where the ODCS spec is silent, we match the reference implementation">
Several of the mapping decisions below are not settled by the ODCS spec text. Where that's the case, DQX matches [datacontract-cli](https://github.com/datacontract/datacontract-cli)'s reference mapping (onto [Soda Core](https://github.com/sodadata/soda-core) checks) rather than inventing its own reading, so a contract produces the same verdict regardless of which tool evaluates it. Each judgment call is called out explicitly where it applies: `duplicateValues`'s counting unit, `mustBeBetween`/`mustNotBeBetween`'s bound inclusivity, and `invalidValues`'s combined-condition handling of `validValues` + `pattern`.
</Admonition>

### Defining Metric Rules in ODCS v3.x Contracts

A library quality entry sets `type: library` and a `metric` name in the `quality` section of a property (for `nullValues`, `missingValues`, `invalidValues`, and the single-column form of `duplicateValues`) or a schema (for `rowCount` and the composite-key form of `duplicateValues`):

```yaml
quality:
- type: library
metric: nullValues
mustBe: 0
dimension: completeness # optional; defaults per metric, see below
severity: critical # optional; recorded verbatim, DQX defines no vocabulary for it
```

Every metric shares the same **threshold fields** (one of which must be set) and an optional **`unit`** — see [Threshold Fields and the SQL Fallback](#threshold-fields-and-the-sql-fallback) and [Rows vs. Percent](#rows-vs-percent) below. `dimension` and `severity` are recorded in `user_metadata` for every metric rule; `dimension` falls back to a per-metric ODCS default when the contract omits it, and `severity` is only present when the contract sets it.

### Supported Metrics

| Metric | Level | Default `dimension` | What it validates |
|--------|-------|----------------------|--------------------|
| `rowCount` | Schema | `completeness` | The total row count of the dataset. |
| `nullValues` | Property | `completeness` | The count or percentage of `NULL` values in a column. |
| `missingValues` | Property | `completeness` | The count or percentage of `NULL`s plus contract-supplied sentinel values (e.g. `""`, `"N/A"`) in a column. |
| `invalidValues` | Property | `conformity` | The count or percentage of values in a column that fail an allowlist and/or a regex pattern. |
| `duplicateValues` | Property (single column) or Schema (composite key) | `uniqueness` | The count or percentage of rows that duplicate another row on one or more key columns. |

`nullValues`, `missingValues`, and `invalidValues` are property-level only; a schema-level entry for one of these is skipped with a warning. `rowCount` is schema-level only. `duplicateValues` supports both: a property-level entry with no arguments checks that single column, while a schema-level entry reads a composite key from `arguments.properties`.

### Threshold Fields and the SQL Fallback

Every library metric is evaluated against one of eight ODCS threshold fields, tried in this order — the first one present on the entry is used: `mustBe`, `mustNotBe`, `mustBeGreaterOrEqualTo`, `mustBeLessOrEqualTo`, `mustBeGreaterThan`, `mustBeLessThan`, `mustBeBetween`, `mustNotBeBetween`.

- **`mustBe: 0`** is special-cased for `nullValues`, `missingValues`, `invalidValues`, and `duplicateValues`: it maps onto a cheap **row-level** check (`is_not_null`, `is_not_in_list`, `is_in_list`/`regex_match`, or `is_unique`) that pinpoints the offending rows, rather than a dataset-level count.
- **`mustBe`, `mustNotBe`, `mustBeGreaterOrEqualTo`, `mustBeLessOrEqualTo`** (including `mustBe` with a non-zero value) map onto exact-fit **dataset-level aggregate checks** — `is_aggr_equal`, `is_aggr_not_equal`, `is_aggr_not_less_than`, `is_aggr_not_greater_than` — over the metric's count or percentage, for `rowCount`, `nullValues`, `missingValues`, and `invalidValues`.
- **`mustBeGreaterThan`, `mustBeLessThan`, `mustBeBetween`, `mustNotBeBetween`** have no strict-inequality/range equivalent among DQX's aggregate checks, so they fall back to a dataset-level [`sql_query`](/docs/reference/quality_checks#using-sql-query) check with `condition_column: "condition"` (`true` means a violation). For `mustBeBetween`/`mustNotBeBetween`, **both bounds are inclusive** — a value exactly equal to either bound counts as being "between" them. The ODCS spec text doesn't settle this either way; DQX matches datacontract-cli's reference mapping (SodaCL's plain `between`, which is inclusive on both ends unless a bound is written with a round bracket) rather than picking its own reading.
- **`duplicateValues` is the one exception**: every non-`mustBe: 0` threshold (not just the four strict-inequality/range ones above) falls back to `sql_query`. The duplicate count/percentage is computed via a `GROUP BY ... HAVING` subquery rather than a plain aggregate expression, since Spark rejects a window function (the `PARTITION BY` used to detect duplicates) nested inside an aggregate function (`SUM`/`AVG`) — see [duplicateValues](#duplicatevalues) below for what it counts.

<Admonition type="note" title="No recognized threshold field">
An entry with none of the eight fields set is skipped with a warning naming all eight, so a contract author can spot a typo (e.g. `mustbe` instead of `mustBe`) without reading DQX source.
</Admonition>

### Rows vs. Percent

The optional `unit` field controls whether a non-`mustBe: 0` threshold is evaluated as a raw row count (`unit: rows`, the default) or a percentage of all rows (`unit: percent`). This applies to `nullValues`, `missingValues`, `invalidValues`, and `duplicateValues`; `rowCount` is inherently a count and does not branch on `unit` (whatever value is set is still recorded in `user_metadata`, but does not change the generated check). For `nullValues`, `missingValues`, `invalidValues`, and `duplicateValues`, an unrecognized `unit` value (anything other than `rows` or `percent`) causes the entry to be skipped with a warning.

### rowCount

<Tabs>
<TabItem value="YAML" label="Contract (YAML)" default>
```yaml
schema:
- name: orders
physicalType: table
quality:
- type: library
metric: rowCount
mustBeGreaterOrEqualTo: 1 # → is_aggr_not_less_than(column="*", aggr_type="count")
```
</TabItem>
</Tabs>

A `mustBeGreaterThan`, `mustBeLessThan`, `mustBeBetween`, or `mustNotBeBetween` threshold on `rowCount` falls back to a `sql_query` check counting `COUNT(*)` over the whole dataset.

### nullValues

<Tabs>
<TabItem value="YAML" label="Contract (YAML)" default>
```yaml
properties:
- name: customer_email
logicalType: string
physicalType: string
quality:
- type: library
metric: nullValues
mustBe: 0 # → row-level is_not_null(column="customer_email")

- name: middle_name
logicalType: string
physicalType: string
quality:
- type: library
metric: nullValues
mustBeLessOrEqualTo: 5
unit: percent # → is_aggr_not_greater_than over the null-percentage indicator
```
</TabItem>
</Tabs>

### missingValues

`missingValues` combines real `NULL`s with a contract-supplied sentinel list, so it needs `arguments.missingValues` — a list that may include a literal `null` alongside sentinel strings:

<Tabs>
<TabItem value="YAML" label="Contract (YAML)" default>
```yaml
properties:
- name: country_code
logicalType: string
physicalType: string
quality:
- type: library
metric: missingValues
mustBe: 0
arguments:
missingValues: [null, "", "N/A", "UNKNOWN"]
# → row-level is_not_null (for the null entry)
# + row-level is_not_in_list(forbidden=["", "N/A", "UNKNOWN"])
```
</TabItem>
</Tabs>

`NULL` is always counted as missing, whether or not a literal `null` appears in `arguments.missingValues` — listing it is redundant, not required. With `mustBe: 0`, a row-level `is_not_null` rule is always generated, plus a row-level `is_not_in_list` rule when non-null sentinels are present; both rules share the same `user_metadata` and are OR'd together via DQX's own per-row error/warning union, since `is_not_in_list`'s `forbidden` list can never itself match a real SQL `NULL`. Any other threshold routes through a dataset-level count/percentage of rows matching `column IS NULL OR column IN (...)` — the same unconditional-`NULL` condition, so the metric means the same thing at every threshold.

### invalidValues

`invalidValues` accepts an allowlist (`arguments.validValues`), a regex (`arguments.pattern`), or both — a row fails if it matches neither. When both are present, DQX combines them into a single "invalid" condition (`NOT (in the allowlist) OR NOT (matches the pattern)`) rather than treating them as two independent thresholds, so `mustBeLessOrEqualTo: 5` means at most 5 rows failing *either* criterion, not up to 5 failing each independently. This combination is a DQX judgment call — datacontract-cli's reference mapping ignores `arguments.pattern` for `invalidValues` entirely and only honors `validValues`, so `pattern` support here is a DQX-only extension beyond what the reference implementation covers, even though the spec documents the field:

<Tabs>
<TabItem value="YAML" label="Contract (YAML)" default>
```yaml
properties:
- name: order_status
logicalType: string
physicalType: string
quality:
- type: library
metric: invalidValues
mustBe: 0
arguments:
validValues: ["pending", "confirmed", "shipped", "delivered", "cancelled"]
# → row-level is_in_list(allowed=[...], case_sensitive=true)

- name: postal_code
logicalType: string
physicalType: string
quality:
- type: library
metric: invalidValues
mustBeLessOrEqualTo: 0.5
unit: percent
arguments:
pattern: '^[0-9]{5}$'
# → is_aggr_not_greater_than over the invalid-percentage indicator
```
</TabItem>
</Tabs>

<Admonition type="warning" title="Pattern safety check">
`arguments.pattern` is checked at generation time for patterns that could cause catastrophic regex backtracking (ReDoS). A pattern that fails this check is treated as absent — its own warning is logged, and the entry still proceeds using `validValues` alone if present. Keep patterns simple and avoid nested or alternation-heavy quantifiers.
</Admonition>

### duplicateValues

The single-column, argument-less form is a property-level entry; the composite-key form is a schema-level entry with `arguments.properties`.

<Admonition type="note" title="Counts distinct recurring values, not rows in a duplicated group">
For a non-`mustBe: 0` threshold, `duplicateValues` counts the number of *distinct values (or key combinations) that recur* — for a column holding `[A, A, A, B, B, C]` that's 2 (`A` and `B` each recur), not 5 (the rows sitting in those two groups). `unit: percent` divides that count by the total row count. This matches datacontract-cli's reference mapping onto Soda's `duplicate_count`; the ODCS spec text ("Counts duplicate values in a column") doesn't settle it either way, so DQX follows the reference rather than counting affected rows. `mustBe: 0` is unaffected by this choice, since both readings agree at zero.
</Admonition>

<Tabs>
<TabItem value="YAML" label="Single column (property-level)" default>
```yaml
properties:
- name: order_id
logicalType: string
physicalType: string
quality:
- type: library
metric: duplicateValues
mustBe: 0 # → row-level is_unique(columns=["order_id"])
```
</TabItem>
<TabItem value="YAML" label="Composite key (schema-level)">
```yaml
schema:
- name: order_lines
physicalType: table
quality:
- type: library
metric: duplicateValues
mustBeLessOrEqualTo: 1
unit: percent
arguments:
properties: [order_id, line_number]
# → sql_query over the percentage of distinct recurring (order_id, line_number)
# combinations, grouped by (order_id, line_number)
```
</TabItem>
</Tabs>

### Malformed and Unrecognized Entries

DQX **warns and skips** an individual `type: library` entry — logging the reason and continuing to process the rest of the contract — rather than failing generation, whenever it encounters:

- A missing or unrecognized `metric` name (the warning names all five supported metrics).
- No recognized threshold field set (none of the eight `mustBe...`/`mustNotBe...` fields).
- An unrecognized `unit` (anything other than `rows` or `percent`) — see [Rows vs. Percent](#rows-vs-percent).
- A schema-level entry for a property-only metric (`nullValues`, `missingValues`, `invalidValues`).
- Malformed `arguments` — not a dict, a missing or wrong-typed key (e.g. `arguments.missingValues` not a list), or an empty list where a non-empty one is required.
- For `invalidValues`, an entry with neither a usable `arguments.validValues` list nor a usable `arguments.pattern` string (after the pattern safety check, described above).

This per-entry warn-and-skip behavior is independent of `process_library_rules`: even with `process_library_rules=True` (the default), an individual malformed entry is skipped rather than failing the whole contract. Set `process_library_rules=False` to skip *all* `type: library` entries instead — e.g. if a contract's library entries need to be ignored altogether.

<Admonition type="note" title="No dedup against predefined rules">
DQX does not deduplicate library metric rules against predefined rules generated from schema constraints. A property marked `required: true` (which generates a predefined `is_not_null` rule) that also carries a `nullValues`/`mustBe: 0` entry produces two separate `is_not_null` rules under different names. Both are harmless (they agree on every row), but if you want only one, disable one of the two sources — e.g. `generate_predefined_rules=False` or omit the redundant `nullValues` entry.
</Admonition>

### Disabling Library Metric Processing

If a contract's `type: library` entries misbehave, or you'd rather express quality entirely through [explicit rules](#explicit-rule-generation):

<Tabs>
<TabItem value="Python" label="Python" default>
```python
# Skip type: library processing, generate only predefined/explicit/text rules
rules = generator.generate_rules_from_contract(
contract_file="contract.yaml",
process_library_rules=False # Skip library metric mapping
)
```
</TabItem>
</Tabs>

## Complete Usage Example

Here's a complete example showing contract-based rule generation with all three rule types:
Expand Down Expand Up @@ -692,9 +935,15 @@ All generated rules include rich metadata that traces them back to the source co
| `contract_version` | Contract version | `2.1.0` |
| `odcs_version` | ODCS API version (standard version) | `v3.0.2` |
| `schema` | Schema/table name (ODCS v3.x) | `sensor_readings` |
| `field` | Property/column name (if property-level rule) | `sensor_id` |
| `rule_type` | How rule was generated | `predefined`, `explicit`, or `text_llm` |
| `field` | Property/column name (single-field property-level rule) | `sensor_id` |
| `fields` | Composite key columns (schema-level `duplicateValues` metric rule only) | `["order_id", "line_number"]` |
| `rule_type` | How rule was generated | `predefined`, `explicit`, `text_llm`, or `metric` (see [Metric Rule Generation](#metric-rule-generation)) |
| `text_expectation` | Original text (for text-based rules only) | Natural language description |
| `metric` | ODCS library metric name (`rule_type: "metric"` only) | `rowCount`, `nullValues`, `missingValues`, `invalidValues`, or `duplicateValues` |
| `threshold_field` | ODCS threshold field that produced the check (`rule_type: "metric"` only) | `mustBe`, `mustBeGreaterOrEqualTo`, `mustBeBetween`, etc. |
| `unit` | Threshold unit for the metric (`rule_type: "metric"` only) | `rows` (default) or `percent` |
| `dimension` | Data quality dimension — the contract's own `dimension` if set, else the metric's ODCS default | `completeness`, `uniqueness`, or `conformity` |
| `severity` | Contract's own `severity` value, recorded verbatim for audit only (never used to derive `criticality`); present only when the contract sets it | Any contract-defined string |

This metadata enables:
- **Traceability**: Link quality check results back to contract specifications
Expand Down Expand Up @@ -957,4 +1206,6 @@ pip install 'databricks-labs-dqx[datacontract]'
- Dataset-level quality checks must be defined as explicit DQX rules (custom quality checks).
- Generated rules quality depends on the completeness and correctness of the contract definition.
- Complex business logic (text based rules) may still require manual rule definition or refinement.
- `type: library` covers five metrics only (see [Metric Rule Generation](#metric-rule-generation)) and is not intended to grow into a general ODCS interpreter; use `type: custom` with `engine: dqx` for anything beyond simple row-count/null/duplicate/allowlist checks.
- `type: library` entries are matched on the current `metric` field only; the deprecated ODCS 3.0.x `rule` key (superseded by `metric` in 3.1+) is not recognized, so a contract still authored against 3.0.x semantics needs its `rule` entries migrated to `metric` before DQX will map them.

Loading