Skip to content

[FEATURE]: Unify parsing semantics for geospatial checks #1513

Description

@ghanse

Is there an existing issue for this?

  • I have searched the existing issues

Problem statement

Spark 4.2 introduced native geometry and geography types for DataFrames. DQX offers a module for validating properties of geospatial data. However, this module does not:

  • Consistently accept and handle geometry and geography columns as check function inputs
  • Consistently generate metadata when checks are skipped due to invalid WKB/WKT/GeoJSON parsing; These currently appear in results as NULL when the input values are invalid

Proposed Solution

  • Update the geospatial check functions to allow geometry and geography columns
  • Create metadata messages when checks are skipped because the input WKB/WKT/GeoJSON values are invalid

Additional Context

These changes were introduced for is_geo_within_distance in #1510 . This issue expands that work to other geospatial check functions.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

DQX CoreFeature/Bug Related to DQX Core functionalityenhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions