Fix escaping during check execution by ghanse · Pull Request #1486 · databrickslabs/dqx

ghanse · 2026-08-25T19:18:19Z

Changes

This PR adds escaping for column names during check execution.

Note: Auto-generated check names (e.g. when checks are not named by the user) are normalized to replace reserved characters with _.

Linked issues

Resolves #1481

Tests

Documentation and Demos

added/updated demos
added/updated docs
added/updated agent skills

codecov · 2026-08-25T19:27:07Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.80%. Comparing base (8b6539c) to head (0ffe3ac).

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1486      +/-   ##
==========================================
- Coverage   92.88%   92.80%   -0.09%     
==========================================
  Files         142      142              
  Lines       13864    13875      +11     
  Branches      151      151              
==========================================
- Hits        12878    12877       -1     
- Misses        917      929      +12     
  Partials       69       69

Flag	Coverage Δ
anomaly	`51.50% <84.00%> (+0.06%)`	⬆️
anomaly-serverless	`51.51% <84.00%> (+0.06%)`	⬆️
integration	`47.18% <68.00%> (-0.12%)`	⬇️
integration-serverless	`49.00% <68.00%> (-0.12%)`	⬇️
mcp	`80.34% <ø> (ø)`
unit	`66.34% <100.00%> (+0.04%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2026-08-25T19:41:01Z

✅ 1029/1029 passed, 63 skipped, 5h31m56s total

_{Running from acceptance #5899}

github-actions · 2026-08-25T19:48:11Z

✅ 1/1 passed, 26m5s total

_{Running from mcp #648}

github-actions · 2026-08-25T20:09:57Z

✅ 195/195 passed, 1 skipped, 6h51m1s total

_{Running from anomaly #2013}

Copilot

Pull request overview

This PR addresses #1481 by ensuring column names that require SQL identifier escaping (spaces, non-ASCII, etc.) are safely handled during check execution (not just validation), preventing INVALID_IDENTIFIER failures when building Spark expressions.

Changes:

Add utils.normalize_column_expr() and use it in check_funcs._get_column_expr() so string column inputs are safely passed to F.expr.
Adjust get_normalized_column_and_expr() so display/normalized names come from the original user input string (not the escaped Spark expression).
Extend unit + integration coverage, including contract-rule generation for escaped columns.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`src/databricks/labs/dqx/utils.py`	Introduces `normalize_column_expr()` and supporting regexes to decide when to back-quote identifiers vs pass through SQL expressions.
`src/databricks/labs/dqx/check_funcs.py`	Uses `normalize_column_expr()` when converting string inputs via `F.expr`, and tweaks name normalization/display behavior.
`src/databricks/labs/dqx/datacontract/contract_rules_generator.py`	Uses `normalize_column_expr()` when embedding column names into generated `sql_expression` rules.
`tests/unit/test_utils.py`	Adds parametric unit tests for `normalize_column_expr()` behavior (identifiers, nested paths, expressions).
`tests/unit/test_row_checks.py`	Adds unit test ensuring normalized/display names derive from the raw input column string.
`tests/unit/test_datacontract_generator.py`	Adds unit test verifying generated SQL expressions back-quote a column requiring escaping.
`tests/integration/test_apply_checks.py`	Adds integration tests proving row-level checks run end-to-end on columns requiring escaping (metadata + class-based APIs).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

mwojtyczka · 2026-09-01T09:18:53Z

    col_expr = _get_column_expr(column)
-    column_str = get_column_name_or_alias(col_expr)
-    col_str_norm = get_column_name_or_alias(col_expr, normalize=True)
+    if isinstance(column, str):


shouldn't this logic be implemented inside get_column_name_or_alias?
We already handle the case there:

if isinstance(column, str): col_str = column

we just need to extend it with an option to normalize:

if isinstance(column, str): if normalize: col_str = normalize_col_str(column) else: col_str = column

Then we don't need to change check_funcs.py

column_str = get_column_name_or_alias(col_expr) col_str_norm = get_column_name_or_alias(col_expr, normalize=True)

This is still not resolved. The current implementation branches on isinstance(column, str) inside get_normalized_column_and_expr and calls unquote_column_name + normalize_col_str directly, rather than adding a normalize option to get_column_name_or_alias as suggested — so check_funcs.py was still changed. Leaving open to confirm whether we want to centralize this in get_column_name_or_alias instead.

mwojtyczka · 2026-09-09T15:14:03Z

+        return column
+
+    return ".".join(
+        segment if VALID_UNQUOTED_IDENTIFIER_PATTERN.match(segment) else quote_column_name(segment)


Finding: a column whose name equals a zero-arg SQL function resolves to the function, not the column (silent wrong result)

A bare identifier that matches VALID_UNQUOTED_IDENTIFIER_PATTERN is returned unchanged, so a real column literally named current_date, current_timestamp, or current_user is passed to F.expr unquoted and evaluates the SQL function instead of the column.

Failure scenario: a DataFrame has a column named current_date. normalize_column_expr("current_date") returns "current_date" (valid identifier, left alone), so F.expr("current_date") yields today's date rather than the column's value. is_not_null etc. then check the wrong value with no error surfaced. Only closable by always back-quoting a name-classified segment, or schema-aware resolution. Lower priority (needs an unusually-named column), flagging for awareness.

This is also DBSQL's convention. Users must escape zero-arg identifiers in any DBSQL query if they use them as column names. Should we change this or align with DBSQL?

sounds good, I think it's better to stay aligned with DBSQL

mwojtyczka · 2026-09-09T15:14:03Z

+        The column name without surrounding back-quotes.
+    """
+    if len(name) >= 2 and name.startswith("`") and name.endswith("`"):
+        return name[1:-1].replace("``", "`")


Finding: dotted path of separately back-quoted segments is mangled (cosmetic)

unquote_column_name only strips the outer back-quotes, so a path whose segments are individually quoted comes back malformed.

Failure scenario: column="Odd Name.field" → unquote_column_name(...) returns "Odd Name.field", from which normalize_col_str derives a garbled check name and the _errors/_warnings message shows a corrupted column name. Resolution itself is fine (normalize_column_expr leaves the string unchanged and F.expr resolves it), so this is display-only — but user-visible. Low severity.

mwojtyczka · 2026-09-09T15:14:03Z

+
+def normalize_column_expr(column: str) -> str:
+    """
+    Prepares a column reference string for use with ``F.expr``, back-quoting names that require SQL


Nit: docstring uses double-backtick object references, against the repo convention

AGENTS.md (Docstrings) says: No backticks around object names — use italics instead (e.g., arg1, column). Backticks cause rendering issues in API docs. This docstring uses F.expr (and quote_column_name / unquote_column_name use F.expr and :func: cross-refs) — should be italics, e.g. F.expr. Note _get_column_expr in check_funcs.py already does this correctly (normalize_column_expr).

mwojtyczka

Previous comments all addressed except 1, added a few new comments

ghanse · 2026-09-14T16:11:06Z

@mwojtyczka ready for another review. test_apply_checks.py is quite large and failing pylint rules now, should we decompose this into a few separate test modules?

Fix escaping during check execution

95ffb4b

ghanse requested a review from a team as a code owner August 25, 2026 19:18

ghanse requested review from nehamilak-db and removed request for a team August 25, 2026 19:18

ghanse temporarily deployed to tool August 25, 2026 19:18 — with GitHub Actions Inactive

ghanse had a problem deploying to tool August 25, 2026 19:18 — with GitHub Actions Failure

ghanse temporarily deployed to tool August 25, 2026 19:18 — with GitHub Actions Inactive

ghanse had a problem deploying to tool August 25, 2026 19:18 — with GitHub Actions Failure

ghanse temporarily deployed to tool August 25, 2026 19:19 — with GitHub Actions Inactive

Fix escaping during check execution

0b987ab

ghanse temporarily deployed to tool August 25, 2026 21:38 — with GitHub Actions Inactive

ghanse temporarily deployed to tool August 26, 2026 12:59 — with GitHub Actions Inactive

ghanse requested a lite review from Copilot August 29, 2026 22:22

Copilot started reviewing on behalf of ghanse August 29, 2026 22:22 View session

Copilot AI reviewed Aug 29, 2026

View reviewed changes

Comment thread src/databricks/labs/dqx/utils.py Outdated

mwojtyczka reviewed Sep 1, 2026

View reviewed changes

Comment thread src/databricks/labs/dqx/utils.py Outdated

mwojtyczka reviewed Sep 1, 2026

View reviewed changes

Comment thread src/databricks/labs/dqx/utils.py

mwojtyczka temporarily deployed to tool September 9, 2026 14:57 — with GitHub Actions Inactive

mwojtyczka temporarily deployed to tool September 9, 2026 14:58 — with GitHub Actions Inactive

mwojtyczka reviewed Sep 9, 2026

View reviewed changes

mwojtyczka requested changes Sep 9, 2026

View reviewed changes

Merge branch 'main' into fix-escaping

eda7515

ghanse deployed to tool September 10, 2026 21:31 — with GitHub Actions Active

ghanse deployed to tool September 10, 2026 21:32 — with GitHub Actions Active

Move normalization into shared helper

c96ae40

ghanse deployed to tool September 11, 2026 02:42 — with GitHub Actions Active

Merge branch 'main' into fix-escaping

0ffe3ac

ghanse deployed to tool September 14, 2026 13:36 — with GitHub Actions Active

ghanse deployed to tool September 14, 2026 13:38 — with GitHub Actions Active

ghanse requested a review from mwojtyczka September 14, 2026 16:11

Uh oh!

Conversation

ghanse commented Aug 25, 2026

Changes

Linked issues

Tests

Documentation and Demos

Uh oh!

codecov Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

mwojtyczka Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mwojtyczka Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

mwojtyczka Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

ghanse Sep 10, 2026

Choose a reason for hiding this comment

Uh oh!

mwojtyczka Sep 11, 2026

Choose a reason for hiding this comment

Uh oh!

mwojtyczka Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

mwojtyczka Sep 9, 2026

Choose a reason for hiding this comment

Uh oh!

mwojtyczka left a comment

Choose a reason for hiding this comment

Uh oh!

ghanse commented Sep 14, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

codecov Bot commented Aug 25, 2026 •

edited

Loading

github-actions Bot commented Aug 25, 2026 •

edited

Loading

github-actions Bot commented Aug 25, 2026 •

edited

Loading

github-actions Bot commented Aug 25, 2026 •

edited

Loading

mwojtyczka Sep 1, 2026 •

edited

Loading