Skip to content

parquet-adapter - #570

Open
juliacher wants to merge 47 commits into
masterfrom
parquet-adapter
Open

parquet-adapter#570
juliacher wants to merge 47 commits into
masterfrom
parquet-adapter

Conversation

@juliacher

@juliacher juliacher commented Jul 11, 2026

Copy link
Copy Markdown

Summary

Adds the Parquet adapter implementation, including relational and document source support, schema discovery, partition handling, query pushdown optimizations, workflow integration, tests, and final project documentation.

Fixes: N/A

Changes

  • Added Parquet source integration as a Polypheny plugin for relational and document models.
  • Added schema discovery for single files, multi-file folders, dataset roots, and Hive-style partitioned layouts.
  • Added flat and normalized relational schema modes for nested Parquet structures.
  • Added adapter-level planning and execution paths for scans, filters, joins, and aggregates.
  • Added metadata-backed statistics and file pruning based on partition values and Parquet footer statistics.
  • Added workflow extract/load support for Parquet files.
  • Cleaned and aligned project documentation with the current implementation.

Features

  • Relational Parquet source adapter.
  • Document Parquet source adapter.
  • Multi-file Parquet table discovery.
  • Hive-style partition column discovery.
  • Flat nested scalar field exposure.
  • Normalized nested table generation with synthetic structural columns.
  • Projection and filter pushdown.
  • Native Parquet predicate pushdown where supported.
  • Adapter-level parent/child structural joins.
  • Metadata and data aggregate pushdown.
  • Metadata-backed table and column statistics.
  • Parquet workflow import and export activities.

Bug Fixes

N/A


Tests

  • Added unit coverage for schema discovery, partition handling, filter translation, file pruning, primitive reads, document conversion, aggregate execution, statistics, nested join planning, IO utilities, and workflow Parquet support
  • Added integration-style coverage for importing and querying Parquet-backed relational sources.

ToDo

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant