feat: writing/reading to parquet by arnavdham · Pull Request #1002 · AIDASoft/podio

arnavdham · 2026-08-14T08:20:17Z

This PR extends the existing Arrow backend in Podio by adding the capability to write and read datamodels directly to and from Apache Parquet files. This significantly improves data interoperability with Python ecosystems and column-oriented analysis workflows.

Key Features & Additions:

ArrowWriter & ArrowReader: Implemented robust C++ classes to handle serialization of the Arrow backend's Frame objects to Parquet (saving one .parquet file per category) alongside a metadata.json for configuration and format tracking.
podioArrow Target: Expanded the existing core Arrow processing logic. Populated the ArrowTypeRegistry and ArrowConverterRegistry with primitive type converters to support translating built-in types for file I/O operations.
Build System Integration: Fixed the existing PODIO_ADD_ARROW macro in podioMacros.cmake to ensure downstream datamodel targets correctly link against podio::podioArrow (resolving undefined references).
Comprehensive Testing: Added a full suite of tests in tests/arrow_io/ mirroring the existing I/O tests. This includes Python/C++ interoperability tests, multithreaded read/write testing, and interface roundtripping for the Parquet files.

BEGINRELEASENOTES

Add Parquet file reading and writing capabilities to the Arrow backend (ArrowWriter and ArrowReader).
Populate ArrowTypeRegistry and ArrowConverterRegistry with primitive type converters for file I/O.
Fix missing linking dependencies in podioMacros.cmake for downstream datamodel targets using the Arrow backend.

ENDRELEASENOTES

tmadlener

I haven't looked into any of the details so far. I have a conceptual question; Currently this would create directories with files. Would it make sense to grab libarchive and "tar those up" on the fly so that in the end we deal with one "file" again?

arnavdham · 2026-08-19T09:46:42Z

I haven't looked into any of the details so far. I have a conceptual question; Currently this would create directories with files. Would it make sense to grab libarchive and "tar those up" on the fly so that in the end we deal with one "file" again?

I am not sure why this would help. From a user perspective, a directory with multiple files would be easier to use directly in python environments.

tmadlener

I see your point, fair enough. It would in any case be fairly straight forward to fix that later.

I have had a first look and in general there seem to be mainly minor things. Just to confirm: The tests are essentially copies from the other I/O backends just using the Arrow{Writer,Reader} instead, right?

Final question, is Arrow{Reader,Writer} the best name, or should it also contain Parquet somehow?

arnavdham · 2026-08-19T15:42:14Z

I see your point, fair enough. It would in any case be fairly straight forward to fix that later.

I have had a first look and in general there seem to be mainly minor things. Just to confirm: The tests are essentially copies from the other I/O backends just using the Arrow{Writer,Reader} instead, right?

Final question, is Arrow{Reader,Writer} the best name, or should it also contain Parquet somehow?

Yes tests are copies from other backends. Have no strong opinion on the name though cause my initial thought was ParquetWriter/Reader itself but that didn't have Arrow so I wasn't a fan.

arnavdham · 2026-08-21T06:33:03Z

Have addressed all the comments.

jmcarcell · 2026-08-24T09:13:52Z

I don't know if this is elsewhere but there are a few things that would be good or necessary to have:

Documentation (docs folder)
podio-dump working with parquet files

arnavdham · 2026-08-24T12:22:09Z

I don't know if this is elsewhere but there are a few things that would be good or necessary to have:

Documentation (docs folder)

podio-dump working with parquet files

I think podio-dump already works with parquet files will check this regardless.
For documentation will adding a section in doc/reading_writing.md for Arrow would be enough?

arnavdham · 2026-08-24T14:18:34Z

I don't know if this is elsewhere but there are a few things that would be good or necessary to have:

Documentation (docs folder)

podio-dump working with parquet files

I think podio-dump already works with parquet files will check this regardless. For documentation will adding a section in doc/reading_writing.md for Arrow would be enough?

jug_dev> arnavdham@ARNAVDesktop:~/gsoc/podio$ LD_PRELOAD=$PWD/build/tests/libTestDataModelPodioArrow.so ./install/bin/podio-dump example_frame_zstd.arrow
input file: example_frame_zstd.arrow
            (written with podio version: 1.7.99)

datamodel model definitions stored in this file:
 - datamodel (1.7.99)

Frame categories in this file:
Name    Entries  
------  -------  
events  10       
################################### events: 0 ####################################
Collections:
Name (s = subset collection)  ValueType                                   Size  ID          
----------------------------  ------------------------------------------  ----  --------    
arrays                        ExampleWithArray                            1     d5f00a11    
clusters                      ExampleCluster                              3     05393275    
emptyCollection               ExampleCluster                              0     35ddd2cf    
emptySubsetColl (s)           ExampleHit                                  0     d76d338d    
fixedWidthInts                ExampleWithFixedWidthIntegers               3     62b8b748    
hitRefs (s)                   ExampleHit                                  2     96140608    
hits                          ExampleHit                                  2     fb2c5a48    
info                          EventInfo                                   1     66667806    
links                         podio::Link<ExampleHit,ExampleCluster>      2     1e2dcdb5    
links_with_interfaces         podio::Link<ExampleCluster,TypeWithEnergy>  3     2c119831    
mcParticleRefs (s)            ExampleMC                                   10    b7f1d19b    
mcparticles                   ExampleMC                                   10    7e15ccd5    
moreMCs                       ExampleMC                                   10    ae0f2334    
OneRelation                   ExampleWithOneRelation                      2     a1c1a57f    
refs                          ExampleReferencingType                      2     3443cc91    
refs2                         ExampleReferencingType                      1     15966606    
userDoubles                   double                                      100   3a47f784    
userInts                      uint64_t                                    1     ceb7bde4    
WithNamespaceMember           ex42::ExampleWithNamespace                  25    6b99e124    
WithNamespaceRelation         ex42::ExampleWithARelation                  5     3d3e8081    
WithNamespaceRelationCopy     ex42::ExampleWithARelation                  5     d57c3b9e    
WithVectorMember              ExampleWithVectorMember                     2     2ce6aaf8    

Parameters:
Name             Type         Elements  
---------------  -----------  --------  
anInt            int          1         
SomeValue        int          1         
SomeVectorData   int          4         
UserEventWeight  float        1         
SomeVectorData   double       2         
SomeValue        std::string  1         
SomeVectorData   std::string  3         
UserEventName    std::string  1

Podio dump does work.

veprbl

Sorry, meant to post this earlier.

veprbl · 2026-08-21T13:06:04Z

+  });
+
+  registry.registerReader(typeName,
+                          [](const std::shared_ptr<arrow::Array>& array, int64_t rowIndex, bool isSubset,


Newline and reformat?

This is clang formattted. The rules come up as errors in pre-commit so nothing much I can do about it.

I suspect if you add a newline after ( it won't undo it.

You could try that, but otherwise clang-format decides here, I would say.

I tried and clang format just un-did my changes so I don't think this would be possible.

veprbl · 2026-08-24T16:58:31Z

 #else
    throw std::runtime_error("SIO writer not available. Please recompile with SIO support.");
+#endif
+  } else if (endsWith(filename, ".arrow") || lower(type) == "arrow") {


Suggested change

} else if (endsWith(filename, ".arrow") || lower(type) == "arrow") {

} else if (endsWith(filename, ".podio_parquet") || lower(type) == "parquet") {

Does it make sense to be explicit about the flavor of arrow format?
Also I would not simply say "arrow" without denoting that this is a podio layout.

#1002 (comment) I just changed this to arrow. Also btw, this is the directory name in Arrow's case so arrow should suffice. The files inside are category_name.parquet and metadata.json.

Yes, this is the suggestion to the current state.

Have changed this.

arnavdham · 2026-08-25T02:59:47Z

@tmadlener @jmcarcell @veprbl Can I get a list of things to be resolved/added so that this could be moved to closure? I am a bit confused with so many comments still open.

tmadlener · 2026-08-25T10:05:17Z

I have resolved a few of the conversations that you addressed. From my point of view there is essentially nothing left to do here apart from some minor things where @veprbl and @jmcarcell have originally commented. (I have added a suggestion from my side to those, but will let them have the final say).

arnavdham · 2026-08-25T12:00:46Z

Have addressed everything now.

tmadlener · 2026-08-26T08:20:17Z

Looks like everyone is happy now. I will merge this later today, unless there are last minute comments. I have created an issue to keep track of the potential optimisation (#1009).

arnavdham marked this pull request as draft August 14, 2026 08:20

arnavdham marked this pull request as ready for review August 14, 2026 08:41

arnavdham marked this pull request as draft August 16, 2026 16:16

tmadlener reviewed Aug 17, 2026

View reviewed changes

arnavdham mentioned this pull request Aug 19, 2026

Implement Parquet file I/O for the Arrow backend #1006

Closed

arnavdham marked this pull request as ready for review August 19, 2026 09:42

tmadlener reviewed Aug 19, 2026

View reviewed changes

tmadlener reviewed Aug 21, 2026

View reviewed changes

Comment thread .github/workflows/pre-commit.yml

tmadlener reviewed Aug 24, 2026

View reviewed changes

Comment thread src/ArrowReader.cc