TASK 12: Add Parquet mapping for GEOGRAPHY type#14
Conversation
davidchaava
changed the title
da986fc to
309e7e5
Compare
gkalashyan-akv
changed the title
309e7e5 to
251e3bd
Compare
251e3bd to
fe3bf24
Compare
|
EnrichedRowData that the wrapper the filesystem source uses to merge partition columns never got getGeography, so we will throw UnsupportedOperationException on any partitioned table. |
|
Parquet writer uses native geographyType() annotation, but readers never look at it: files with a different CRS, GEOMETRY (planar) annotation, or plain unannotated BINARY all silently load as spherical GEOGRAPHY, and spec-valid 3D WKB (type 1001) is rejected as malformed. |
| /** Utilities for converting GEOGRAPHY values to and from portable OGC encodings. */ | ||
| @Internal | ||
| final class GeographyConversionUtils { | ||
|
|
There was a problem hiding this comment.
GeographyConversionUtils shares static WKBReader/WKBWriter/WKTReader/WKTWriter across all task threads; JTS readers/writers hold per-call mutable state, so concurrent ST_* calls can produce corrupted geometries or spurious parse errors.
|
|
||
| this.fileSchema = footer.getFileMetaData().getSchema(); | ||
| ParquetReadOptions parquetReadOptions = | ||
| ParquetReadOptions.builder() |
There was a problem hiding this comment.
builds bare ParquetReadOptions instead of HadoopReadOptions.builder(conf), dropping encryption, checksum, and filter settings for all Parquet reads, and leaks the file stream when schema checks throw.
|
|
||
| <dependency> | ||
| <groupId>org.locationtech.jts</groupId> | ||
| <artifactId>jts-core</artifactId> |
There was a problem hiding this comment.
Let use shading with jts-core. It is bundled into the flink-table-runtime fat jar without relocation, inviting classpath clashes with user geospatial jars.
| BuiltInSqlFunction.newBuilder() | ||
| .name(BuiltInFunctionDefinitions.ST_ASTEXT.getName()) | ||
| .returnType(VARCHAR_FORCE_NULLABLE) | ||
| .operandTypeChecker(OperandTypes.family(SqlTypeFamily.ANY)) |
There was a problem hiding this comment.
the ST_* operator-table entries accept SqlTypeFamily.ANY, so ST_ASTEXT(42) passes the validator and dies later in codegen.
3 participants
Adds Flink-Parquet support for the GEOGRAPHY logical type.
The mapping writes Flink GEOGRAPHY as Parquet BINARY with the native
GEOGRAPHY logical annotation and stores the raw ISO WKB payload without
Flink serializer framing. The vectorized reader maps Parquet GEOGRAPHY
back to Flink GEOGRAPHY and preserves WKB bytes.
This also updates the Parquet reader path to use Flink's ParquetInputFile
abstraction where needed for parquet-java 1.16+ compatibility, and adds a
focused round-trip test that verifies the Parquet annotation and WKB
readback.