Add opt-in semantic-aware profiling to DQProfiler

[IvannKurchenko]

Changes

Introduce opt-in semantic-aware profiling in DQProfiler. A lightweight classification stage runs between metric collection and check generation, so each column receives one consistent family of checks instead of contradictory overlaps (e.g. vehicle_type no longer gets both is_in([...]) and min_max(...)).

The feature is fully backward compatible: when a DQProfiler is constructed without a semantic_registry, no detection runs and the generated profiles are byte-identical to today.

New public surface (all in src/databricks/labs/dqx/profiler/semantic.py):

• DQSemanticType — Pydantic v2 model naming a column's semantic meaning (e.g. enum, key, measurement, text) plus optional properties.
• DQSemanticTypeDetector — named callable (DQProfileContext) -> DQSemanticType | None.
• DQProfileContext — frozen context passed to detectors and contextual builders. Carries df, column_name, column_type, metrics, options, and semantic_type. It exposes with_metrics(...) so the profiler loop can refresh the frozen context after earlier builders write back resolved metrics (e.g. min/max).
• SemanticRegistry — immutable, name-unique, ordered chain of detectors. Composition surface:
  • construction: SemanticRegistry.default(), SemanticRegistry.of(*detectors), SemanticRegistry(detectors=(...))
  • composition: prepend(detector), append(detector), insert(name, detector) (after the named entry), replace(name, detector) (position-preserving swap), remove(name)
  • all methods return a new registry; uniqueness is enforced via a model_validator.
• default_semantic_detectors() and four built-in detectors: DEFAULT_ENUM_DETECTOR, DEFAULT_KEY_DETECTOR, DEFAULT_MEASUREMENT_DETECTOR, DEFAULT_TEXT_DETECTOR plus threshold constants (ENUM_MAX_CARDINALITY_RATIO, KEY_MIN_DENSITY_RATIO, KEY_MIN_LENGTH_STABILITY_RATIO).

Profiler wiring (profiler/profiler.py, profiler/profile_builder.py, profiler/profile.py):

• DQProfiler.__init__ gains a keyword-only semantic_registry: SemanticRegistry | None = None. Presence of the argument opts the run into semantic detection — there is no per-call override.
• DQProfile gains an optional semantic_type: str | None = None field so generated profiles record why a check was emitted (survives YAML/JSON round-trip; defaults to None).
• DQProfileBuilder supports two mutually-exclusive callback shapes: the legacy 5-argument builder (unchanged) and the new contextual_builder(ctx: DQProfileContext). @register_profile_builder gains a kind=\"context\" opt-in for the new shape; existing legacy registrations keep working without changes.
• The four library-native builders (null_or_empty, is_in, min_max, has_no_outliers) are migrated to the contextual form. min_max and has_no_outliers now skip emission unless ctx.semantic_type is None or \"measurement\"; is_in reuses the enum detector's already-collected distinct values so no second Spark .distinct().collect() runs.
• The _detect_enum gate and the legacy is_in gate both compute distinct_count / count_non_null and honour the profiler-wide distinct_ratio option, so semantic classification and legacy emission agree on low-repetition columns even in null-heavy datasets. ShortType is a first-class enum candidate (added to _supports_distinct).

Linked issues

Resolves #1343

Tests

• added unit tests — tests/unit/profiler/test_semantic.py covers each detector (positive + negative), SemanticRegistry immutability and uniqueness invariants (constructor, of, prepend, append, insert, replace, remove), chain semantics (first-match-wins), the tightened enum cardinality guard, numeric-density and string-length-stability key guards including the empty-string edge case, and DQProfileContext.with_metrics snapshot semantics. tests/unit/test_profile_builder.py covers kind=\"context\" vs legacy registration, mutually-exclusive callback validation, and per-builder positive/negative behaviour (including ShortType for is_in) through the new contextual path.
• added integration tests — tests/integration/test_profile_semantic.py covers the default-registry classification of the grounded design columns (vehicle_type, cargo_weight, deal_value, user_id, order_id, user_name, work_description), the no-registry byte-identical default, custom-chain composition via prepend / SemanticRegistry.of(...), the ShortType enum → is_in regression guard, the enum-value reuse optimisation (single .distinct() invocation per enum column), and a public-API behavioural test that a contextual builder ordered after min_max observes the resolved min/max via ctx.metrics.
• added end-to-end tests
• added performance tests

Documentation and Demos

• added/updated demos
• added/updated docs — new "Semantic-aware profiling" section in docs/dqx/docs/reference/profiler.mdx tagged <FeatureLifecycleStage stage=\"beta\"> / <AvailableSinceVersion version=\"0.17.0\">. Documents the opt-in model, the four built-in detectors and their applicability rules and thresholds, the immutable SemanticRegistry composition patterns (prepend, append, insert, replace, remove, of), the @register_profile_builder(kind=\"context\") opt-in, and includes a copy-paste example specialised (UUID) detector.
• added/updated agent skills

[mwojtyczka]

Automated code review (5 findings). Comments posted inline below.

[mwojtyczka]

Follow-up: ergonomics of the SemanticRegistry customization surface.

[IvannKurchenko]

@mwojtyczka May I ask for another review round, please? Your previous comments has been addressed. Thank you

[mwojtyczka]

Code review findings — semantic-aware profiling (#1491)

Verified against head a19ae55. Ranked by severity.

1. is_in distinct-ratio denominator changed total_count → count_non_null — breaks the "byte-identical" no-registry guarantee

src/databricks/labs/dqx/profiler/profile_builder.py — make_is_in_profile (~L150)

- distinct_ratio = (1.0 * distinct_count) / total_count
+ distinct_ratio = (1.0 * distinct_count) / count_non_null

This runs on the default (no semantic_registry) path — when semantic_type is None, control falls through to this calc. For a null-heavy column (1000 rows, 200 nulls, 40 distinct): old ratio 40/1000 = 0.04 < 0.05 → is_in emitted; new ratio 40/800 = 0.05 → not emitted. This contradicts the PR's stated guarantee ("byte-identical to the pre-feature behaviour", and the make_is_in_profile docstring "applicability follows today's byte-identical rules") and the test asserting "No registry → profiler output is identical to the pre-feature behaviour" (which only passes because its fixtures contain no nulls).

This change came from aligning the is_in builder with _detect_enum (a good goal), but because the builder is shared with the legacy path, it shifted legacy output. Either accept the change and reword/drop the "byte-identical" guarantee, or align the two paths without altering the no-registry denominator.

2. ShortType added to _supports_distinct — also breaks "byte-identical"

src/databricks/labs/dqx/profiler/profile_builder.py — _supports_distinct (~L332)

- return isinstance(column_type, (T.IntegerType, T.LongType) + TEXT_TYPES)
+ return isinstance(column_type, (T.IntegerType, T.LongType, T.ShortType) + TEXT_TYPES)

_supports_distinct is used by the legacy is_in builder too, so a low-cardinality ShortType column now emits an is_in rule where it previously produced none — a behavior change for users who never opt into the feature. Same trade-off as #1 (this addressed the earlier "ShortType loses all checks" review comment, but via the shared helper). Worth an explicit decision on whether the no-registry path is allowed to change.

3. profile_table no longer emits the nested profile telemetry event (low)

src/databricks/labs/dqx/profiler/profiler.py — profile_table (~L164)

profile_table was rerouted from self.profile(...) (decorated @telemetry_logger("profiler","profile")) to the undecorated self._profile_dataframe(...). Previously table profiling fired both profile_table and profile; now only profile_table. This is plausibly intentional de-duplication (the old path double-counted), but it's an observable, undocumented change — dashboards/usage counts keying on the profile event will see table-profiling volume drop. Please confirm it's intentional and note it in release notes.

4. Stat parse failure mislabels distribution as "constant" instead of "unknown" (low)

src/databricks/labs/dqx/profiler/semantic.py — _detect_measurement (~L329)

except (TypeError, ValueError):
    span = 0.0; stddev_f = 0.0; mean_f = 0.0
if span <= 0:
    distribution = "constant"

When min/max/mean/stddev fail to cast to float, the zero-fallback drives span <= 0 → distribution="constant" even for a highly variable column. Since the field is a best-effort guess documented to be "unknown" when stats are unavailable, the except branch should yield "unknown", not "constant" — otherwise MeasurementProperties.distribution is misleading.

5. Redundant DQProfileContext metrics snapshot (minor cleanup)

src/databricks/labs/dqx/profiler/profiler.py — _build_profiles_for_column (~L524)

builder_ctx is constructed with metrics=metrics, but the loop's first contextual builder immediately does builder_ctx = builder_ctx.with_metrics(metrics) (metrics unchanged at that point), discarding the first snapshot. Plain (non-contextual) builders never read builder_ctx. Net: one wasted context/dict copy per column. Minor — skip the first refresh when metrics are unchanged, or construct once and refresh only after the min_max write-back.