Skip to content

Add row-level geo distance check and fix documentation examples - #1510

Open
simplegaurav wants to merge 3 commits into
databrickslabs:mainfrom
simplegaurav:feature/maritime-geofencing
Open

Add row-level geo distance check and fix documentation examples#1510
simplegaurav wants to merge 3 commits into
databrickslabs:mainfrom
simplegaurav:feature/maritime-geofencing

Conversation

@simplegaurav

Copy link
Copy Markdown

Changes

Adds is_geo_within_distance, a row-level geospatial check that flags values farther than a
maximum geodesic distance from a reference geography.

Distance is measured in meters along the WGS 84 ellipsoid via st_distance on GEOGRAPHY
values, so the check is meaningful for global data where planar GEOMETRY distances are not.
A row is reported when the shortest distance to the reference is strictly greater than distance.

- criticality: error
  check:
    function: is_geo_within_distance
    arguments:
      column: location
      reference_geometry: "POINT(4.90 52.37)"
      distance: 1000
      convert_column: true
      convert_reference_geometry: true

Behaviour

  • reference_geometry accepts a literal WKT/WKB/EWKT/EWKB string or bytes value, or a Column
    expression. A plain string is always a literal, never a column name.
  • distance accepts a non-negative number, a Column, or a string SQL expression, so the radius
    can vary per row.
  • Null column values are skipped. Rows where distance evaluates to null are also skipped, since
    the comparison is unknown rather than violated.
  • Unparseable column values and an unparseable reference are reported with separate messages,
    so the error names the value that actually has to be fixed.
  • Numeric distance literals are validated up front: negative, NaN, infinite and boolean values
    raise InvalidParameterError. (bool is a subclass of int, so it is rejected explicitly
    rather than silently becoming a 0/1 metre radius.)

Consistency with existing geo checks

  • convert_column / convert_reference_geometry default to False, matching every existing
    is_geo_* relationship check, so the two are not surprising to switch between.
  • The alias is <column>_is_not_within_distance_from_reference_geometry, following the
    _reference_geometry suffix family. Including the reference in the alias also avoids a
    collision when the same column is checked against two different radii.
  • How the offending value is rendered follows the input contract: the raw value when the input is
    converted from WKT/WKB (where st_astext would be NULL on exactly the unparseable values the
    message is about), and st_astext when the column is already a native GEOGRAPHY.

Second commit — pre-existing documentation bug

The programmatic examples for seven is_geo_* checks (is_geo_contains, is_geo_covers ×2,
is_geo_intersects ×2, is_geo_touches, is_geo_within) construct rules with DQDatasetRule,
but all of these are registered with @register_rule("row"). Copying the snippets as written
fails:

InvalidCheckError: Function 'is_geo_within' is not a dataset-level rule. Use DQRowRule instead.

Fixed to DQRowRule plus the missing import. are_polygons_mutually_disjoint is a genuine
dataset-level rule and is left unchanged. These examples are not covered by
test_apply_checks_all_geo_checks_using_classes, which is why it went unnoticed.

Tests

  • manually tested
  • added unit tests
  • added integration tests
  • added end-to-end tests
  • added performance tests

Unit (tests/unit/test_geo_check_funcs.py) — literal and Column references, WKB reference,
Column and SQL-expression distances, zero distance, parametrized rejection of negative / NaN /
±inf / boolean distances, and alias assertions for both the converted and native-GEOGRAPHY paths.
Registered in EXPECTED_PARAMETER_ORDER in tests/unit/test_check_func_signatures.py.

Integration (tests/integration/test_row_checks_geo.py) — inside radius, on the reference at
zero radius, outside radius, per-row radius via a column, null radius skipped, and the invalid
column and invalid reference paths asserted separately. Added to
tests/resources/all_row_geo_checks.yaml and to test_apply_checks_all_geo_checks_using_classes.

Performance (tests/perf/test_apply_checks.py) — test_benchmark_is_geo_within_distance,
alongside the existing is_geo_covers benchmarks.

Documentation and Demos

  • added/updated demos
  • added/updated docs
  • added/updated agent skills

docs/dqx/docs/reference/quality_checks.mdx — added the row-level checks table entry, a YAML
usage example, and a programmatic DQRowRule example, each placed at the end of the geo group.
Also contains the DQDatasetRuleDQRowRule fix described above.

@simplegaurav
simplegaurav requested a review from a team as a code owner September 5, 2026 22:00
@simplegaurav
simplegaurav requested review from nehamilak-db and removed request for a team September 5, 2026 22:00
@CLAassistant

CLAassistant commented Sep 5, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@github-actions

github-actions Bot commented Sep 5, 2026

Copy link
Copy Markdown
Contributor

All commits in PR should be signed ('git commit -S ...'). See https://docs.github.com/en/authentication/managing-commit-signature-verification/signing-commits

simplegaurav and others added 2 commits September 6, 2026 03:45
Adds a row-level geo check that flags values farther than a maximum
geodesic distance from a reference geography. Distance is measured in
meters along the WGS 84 ellipsoid via `st_distance` on GEOGRAPHY values,
so the check is meaningful for global data where planar GEOMETRY
distances are not.

The reference accepts a literal WKT/WKB value or a Column expression, and
the maximum distance accepts a number, a Column, or a SQL expression so
the radius can vary per row. Null column values and null distances are
skipped; unparseable column and reference values are reported separately
so the message names the value that has to be fixed.

Numeric distance literals are validated up front: negative, NaN, infinite
and boolean values are rejected with InvalidParameterError.

The convert_column / convert_reference_geometry flags default to False,
matching the existing is_geo_* relationship checks, and the rendering of
the offending value follows that contract - the raw value when the input
is converted from WKT/WKB, st_astext when the column is already a native
GEOGRAPHY.

Covered by unit tests, integration tests, the all-row-geo metadata
fixture, the programmatic class-based integration test, a performance
benchmark, and the quality checks reference documentation.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The programmatic examples for is_geo_contains, is_geo_covers,
is_geo_intersects, is_geo_touches and is_geo_within construct the rules
with DQDatasetRule, but all of these checks are registered with
@register_rule("row"). Copying the snippets as written fails:

    InvalidCheckError: Function 'is_geo_within' is not a dataset-level
    rule. Use DQRowRule instead.

Switch the seven affected examples to DQRowRule and add the missing
import. are_polygons_mutually_disjoint is a genuine dataset-level rule
and is left unchanged.

These examples are not exercised by
test_apply_checks_all_geo_checks_using_classes, which is why the error
went unnoticed; backfilling that coverage needs a workspace to pick
reference geometries that pass, and is left as a follow-up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@ghanse ghanse left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very good contribution. Left a few minor comments. Need to add 1 test case.


Both the target column and the reference geometry are always handled as `GEOGRAPHY`.
When conversion is requested (*convert_column* or *convert_reference_geometry* set to True),
*try_to_geography* is applied to parse the value from any supported format (WKT, WKB, EWKT, EWKB).

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should include GeoJSON in the supported formats as well.

Comment thread src/databricks/labs/dqx/geo/check_funcs.py

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we're missing a test case for convert_column=False

@ghanse ghanse added the under-review This PR is currently being reviewed by one of DQX maintainers. label Sep 6, 2026
Include GeoJSON in the documented input formats. The docstring listed the
try_to_geometry formats (WKT, WKB, EWKT, EWKB) copied from the sibling
relationship checks, but this check parses with try_to_geography, which
also accepts GeoJSON. Updated the docstring, the reference_geometry
argument description and the reference documentation table.

Add integration coverage for convert_column=False. Both new tests build a
native GEOGRAPHY column with try_to_geography and then leave the convert
flags at their defaults, covering the pass and violation paths and
exercising the st_astext rendering branch used when the column is already
a GEOGRAPHY value.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@simplegaurav

Copy link
Copy Markdown
Author

Thanks for the review — all three addressed in 3ce2ddd6.

GeoJSON in supported formats

Good catch. The docstring listed the try_to_geometry formats (WKT, WKB, EWKT, EWKB) that I'd
copied from the sibling relationship checks, but this check parses with try_to_geography, which
also accepts GeoJSON. Updated in three places: the docstring, the reference_geometry argument
description, and the row-level checks table in quality_checks.mdx.

Missing convert_column=False test

Added two integration tests covering the native GEOGRAPHY path — one pass case, one violation:

def test_is_geo_within_distance_native_geography_violation(skip_if_runtime_not_geo_compatible, spark):
    """A native GEOGRAPHY value outside the radius is flagged, with the value rendered via st_astext."""
    point = "POINT(5.05 52.37)"
    test_df = spark.createDataFrame([[point]], _GEO_SCHEMA).select(
        F.call_function("try_to_geography", F.col("geom")).alias("geom")
    )
    condition = is_geo_within_distance("geom", F.call_function("try_to_geography", F.lit(_POINT_INSIDE)), 1000)
    ...

They build the column with try_to_geography in a select and then leave both convert flags at
their defaults, so the check receives a genuinely GEOGRAPHY-typed column. The violation case also
exercises the st_astext rendering branch, which is only reachable on that path — with
convert_column=True the message renders the raw input instead, since st_astext returns NULL for
exactly the unparseable values the invalid-geography message is about.

I assert on the condition column alone rather than also selecting geom, since a GEOGRAPHY-typed
column can't be compared against a string schema.

One thing I noticed while writing those

No test in the repo currently builds a native GEOGRAPHY column. Every is_geo_* check defaults to
convert_column=False, but all existing tests pass True, so the default path is untested across
the geo module rather than just here. Might be worth a companion issue to #1513 — happy to open one
if useful.

Thanks for opening #1513.

@simplegaurav
simplegaurav requested a review from ghanse September 6, 2026 18:05

@ghanse ghanse left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few minor suggestions.

Comment on lines +1420 to +1436
text_value_col = col_expr.cast("string") if convert_column else F.call_function("st_astext", col_geog)

invalid_column_message = F.concat_ws(
"",
F.lit("value `"),
text_value_col,
F.lit(f"` in column `{col_expr_str}` is not a valid geography"),
)
invalid_reference_message = F.lit(f"reference geometry for column `{col_expr_str}` is not a valid geography")
too_far_message = F.concat_ws(
"",
F.lit("value `"),
text_value_col,
F.lit(f"` in column `{col_expr_str}` is farther than "),
distance_expr.cast("string"),
F.lit(" meters from the reference geometry"),
)

@ghanse ghanse Sep 8, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This may not leave the most useful messages if the user passes WKB/EWKB with convert_column=True. For other checks, we surround the parsed geometry with st_astext:

Suggested change
text_value_col = col_expr.cast("string") if convert_column else F.call_function("st_astext", col_geog)
invalid_column_message = F.concat_ws(
"",
F.lit("value `"),
text_value_col,
F.lit(f"` in column `{col_expr_str}` is not a valid geography"),
)
invalid_reference_message = F.lit(f"reference geometry for column `{col_expr_str}` is not a valid geography")
too_far_message = F.concat_ws(
"",
F.lit("value `"),
text_value_col,
F.lit(f"` in column `{col_expr_str}` is farther than "),
distance_expr.cast("string"),
F.lit(" meters from the reference geometry"),
)
text_value_col = col_expr.cast("string") if convert_column else F.call_function("st_astext", col_geog)
invalid_column_message = F.concat_ws(
"",
F.lit("value `"),
text_value_col,
F.lit(f"` in column `{col_expr_str}` is not a valid geography"),
)
invalid_reference_message = F.lit(f"reference geometry for column `{col_expr_str}` is not a valid geography")
too_far_value_col = F.call_function("st_astext", col_geog)
too_far_message = F.concat_ws(
"",
F.lit("value `"),
too_far_value_col,
F.lit(f"` in column `{col_expr_str}` is farther than "),
distance_expr.cast("string"),
F.lit(" meters from the reference geometry"),
)

"""
# `bool` is a subclass of `int`, so it would otherwise slip through as a 0/1 metre radius.
if isinstance(distance, bool) or (
isinstance(distance, (int, float)) and not (math.isfinite(distance) and distance >= 0)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

math.isfinite may overflow on very large input distances. We might want to catch this and raise InvalidParameterError?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

under-review This PR is currently being reviewed by one of DQX maintainers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants