Is there an existing issue for this?
Problem statement
Spark 4.2 introduced native geometry and geography types for DataFrames. DQX offers a module for validating properties of geospatial data. However, this module does not:
- Consistently accept and handle
geometry and geography columns as check function inputs
- Consistently generate metadata when checks are skipped due to invalid WKB/WKT/GeoJSON parsing; These currently appear in results as
NULL when the input values are invalid
Proposed Solution
- Update the geospatial check functions to allow
geometry and geography columns
- Create metadata messages when checks are skipped because the input WKB/WKT/GeoJSON values are invalid
Additional Context
These changes were introduced for is_geo_within_distance in #1510 . This issue expands that work to other geospatial check functions.
Is there an existing issue for this?
Problem statement
Spark 4.2 introduced native geometry and geography types for DataFrames. DQX offers a module for validating properties of geospatial data. However, this module does not:
geometryandgeographycolumns as check function inputsNULLwhen the input values are invalidProposed Solution
geometryandgeographycolumnsAdditional Context
These changes were introduced for
is_geo_within_distancein #1510 . This issue expands that work to other geospatial check functions.