-
Notifications
You must be signed in to change notification settings - Fork 664
Pnm mesh datapipes #1512
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Pnm mesh datapipes #1512
Changes from 3 commits
Commits
Show all changes
58 commits
Select commit
Hold shift + click to select a range
2b1f535
Adding mesh datapipe temporarily
coreyjadams 132d72a
first, very early, draft of mesh-based datapipe
coreyjadams 8987172
Update datapipes for physicsnemo mesh: now can read and transform sur…
coreyjadams aeb8c06
Add a functionally working - but not yet cxonverging - xaero recipe
coreyjadams 6459144
Add functional physicsnemo mesh pipeline
coreyjadams 5821b15
Update README
coreyjadams 0da2cd1
Snapshotting training recipe
coreyjadams f5cb9bc
Merge branch 'NVIDIA:main' into pnm_mesh_datapipes
coreyjadams 42740e8
Remove Ahmed dataset with physicsnemo mesh, it's not needed.
coreyjadams beb2da0
Simplify and consolidate, or remove statistics work for this example.
coreyjadams 36d3365
Remove inspect_data, and ensure readme is updated
coreyjadams f7e77be
Remove AGENT.md
coreyjadams 6977270
Training recipe is basically complete
coreyjadams 5db9027
Merge branch 'main' into pnm_mesh_datapipes
coreyjadams bb9e995
Merge branch 'main' into pnm_mesh_datapipes
coreyjadams 0c5d58e
Merge branch 'main' into pnm_mesh_datapipes
coreyjadams b669aa7
Update physicsnemo/datapipes/readers/mesh.py
coreyjadams e8ef574
Update physicsnemo/datapipes/transforms/mesh/transforms.py
coreyjadams 5493428
Update physicsnemo/datapipes/readers/mesh.py
coreyjadams 70e806b
Addressing some PR comments.
coreyjadams 0c21bb1
make sure rotations are flexible, add docstrings
coreyjadams 1277d3d
Update distribution handling per PR review.
coreyjadams c8370d2
Merge branch 'NVIDIA:main' into pnm_mesh_datapipes
coreyjadams 492892a
Merge branch 'NVIDIA:main' into pnm_mesh_datapipes
coreyjadams 93b10d1
This commit enablers the datapipes to be reproducible
coreyjadams 4a13b3e
Add testing for the datapipes
coreyjadams d7d2e01
Add missing rng updates
coreyjadams 1fea170
Update training python codes
coreyjadams 53069e2
Add geotransolver configs with obscured paths
coreyjadams aa8cced
Remove shift suv configs for now
coreyjadams c5b8e6d
Merge branch 'main' into pnm_mesh_datapipes
coreyjadams 7b79c58
Adding configs for flare and geotransolver with a flare backend.
coreyjadams d77b582
Update README.md
coreyjadams 6189fe5
Add datapipes.md. Describes some of the computational design of the d…
coreyjadams 957d6c3
Update datapipes to use non-blocking stream syncs when streams are ac…
coreyjadams 019e46b
Fix cuda memory access errors arising from incorrect fake tensor ops.…
coreyjadams bed7113
Merge branch 'main' into pnm_mesh_datapipes
coreyjadams c0463a7
Merge branch 'main' into pnm_mesh_datapipes
coreyjadams bc57d80
Merge branch 'main' into pnm_mesh_datapipes
coreyjadams 1176c5f
Adding datapipe tests for new functionality for mesh datapipe loader
coreyjadams 698989e
Trying to close down github comments
coreyjadams b1fdccb
Use apply instead of _map_meshes
coreyjadams 13f38b6
Make sure poisson subsampling has no duplicates if replacement=false
coreyjadams f9281c2
merging
coreyjadams 763a6b1
Update comments to .pmsh or .pdmsh instead of .pt for physicsnemo dat…
coreyjadams 73559db
Migrate generators ad distributions in transforms.base too
coreyjadams 61227c5
Add tests for distribution and generator migration
coreyjadams 32c611a
Update rmse to relative mse naming, and make sure huber is default in…
coreyjadams bcbaf6e
Fix DDP bug with drop last, and consolidate data paths to unified loc…
coreyjadams 3bbba4a
Use match-case for loud and explicit failure, and default to uniform,…
coreyjadams 43c59ed
Move re-dimensionalization to utils until inference script is added i…
coreyjadams 964d858
Use a simpler parametrization to create random rotation matrices
coreyjadams 4c0f447
undo uv.lock changes
coreyjadams 8484718
Revert uv.lock to main
coreyjadams 13c39fd
Merge branch 'main' into pnm_mesh_datapipes
coreyjadams 00f7c8b
Add ruff corrections
coreyjadams b8a3d79
Update change log and fix loss function.
coreyjadams 1a5e33d
Merge branch 'main' into pnm_mesh_datapipes
coreyjadams File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,291 @@ | ||
| We're working on adding support for physicsnemo Mesh objects to datapipes. this needs to be supported in a couple ways. | ||
|
|
||
| First, we'll support reading mesh objects directly, write off disk, and passing | ||
| them instead of tensordicts (since they are, really, a class of tensordicts). | ||
| I want to be careful here that we compose transforms properly ... only transforms | ||
| meant to consume a mesh should work with meshes. | ||
|
|
||
| So, we will need a special organization layer of datapipes to handle mesh | ||
| specific objects in the datapipes, I think. And then we will need some transforms. | ||
| I think we will need both generic transforms, as well as a series of augmentations, | ||
| which are on-the-fly style transforms that are randomizations. These should be | ||
| separate somehow from decimations. | ||
|
|
||
| I'm going to be working with several key datasets as "benchmark" datasets and | ||
| I'll be updating this file as I go with the schema. So continue to revisit | ||
| this file for information. | ||
|
|
||
|
|
||
| Here is a view of the DrivaerML file structure: | ||
|
|
||
| /lustre/fsw/portfolios/coreai/projects/coreai_modulus_cae/datasets/drivaer_aws/drivaer_data_pnm_mesh/run_1/ | ||
| ├── boundary_1.vtp.pt | ||
| │ ├── meta.json | ||
| │ └── _tensordict | ||
| │ ├── _cache | ||
| │ │ ├── cell | ||
| │ │ │ └── meta.json | ||
| │ │ ├── meta.json | ||
| │ │ └── point | ||
| │ │ └── meta.json | ||
| │ ├── cell_data | ||
| │ │ ├── CpMeanTrim.memmap | ||
| │ │ ├── meta.json | ||
| │ │ ├── pMeanTrim.memmap | ||
| │ │ ├── pPrime2MeanTrim.memmap | ||
| │ │ └── wallShearStressMeanTrim.memmap | ||
| │ ├── cells.memmap | ||
| │ ├── global_data | ||
| │ │ ├── meta.json | ||
| │ │ └── TimeValue.memmap | ||
| │ ├── meta.json | ||
| │ ├── point_data | ||
| │ │ └── meta.json | ||
| │ └── points.memmap | ||
| ├── drivaer_1_single_solid.stl.pt | ||
| │ ├── meta.json | ||
| │ └── _tensordict | ||
| │ ├── _cache | ||
| │ │ ├── cell | ||
| │ │ │ └── meta.json | ||
| │ │ ├── meta.json | ||
| │ │ └── point | ||
| │ │ └── meta.json | ||
| │ ├── cell_data | ||
| │ │ └── meta.json | ||
| │ ├── cells.memmap | ||
| │ ├── global_data | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ ├── point_data | ||
| │ │ └── meta.json | ||
| │ └── points.memmap | ||
| ├── drivaer_1.stl.pt | ||
| │ ├── meta.json | ||
| │ └── _tensordict | ||
| │ ├── _cache | ||
| │ │ ├── cell | ||
| │ │ │ └── meta.json | ||
| │ │ ├── meta.json | ||
| │ │ └── point | ||
| │ │ └── meta.json | ||
| │ ├── cell_data | ||
| │ │ └── meta.json | ||
| │ ├── cells.memmap | ||
| │ ├── global_data | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ ├── point_data | ||
| │ │ └── meta.json | ||
| │ └── points.memmap | ||
| └── volume_1.vtu.pt | ||
| ├── meta.json | ||
| └── _tensordict | ||
| ├── _cache | ||
| │ ├── cell | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ └── point | ||
| │ └── meta.json | ||
| ├── cell_data | ||
| │ └── meta.json | ||
| ├── global_data | ||
| │ ├── meta.json | ||
| │ └── TimeValue.memmap | ||
| ├── meta.json | ||
| ├── point_data | ||
| │ ├── CpMeanTrim.memmap | ||
| │ ├── CptMeanTrim.memmap | ||
| │ ├── magUMeanNormTrim.memmap | ||
| │ ├── meta.json | ||
| │ ├── microDragMeanTrim.memmap | ||
| │ ├── nutMeanTrim.memmap | ||
| │ ├── pMeanTrim.memmap | ||
| │ ├── pPrime2MeanTrim.memmap | ||
| │ ├── turbulenceProperties:RMeanTrim.memmap | ||
| │ ├── UMeanTrim.memmap | ||
| │ └── UPrime2MeanTrim.memmap | ||
| └── points.memmap | ||
|
|
||
| 32 directories, 55 files | ||
|
|
||
|
|
||
| Here is a view of a SHIFT-SUV "Fastback" file: | ||
|
|
||
| ⬢ [podman] ❯ tree $GROUP_LUSTRE/datasets/shift_suv_pnm_mesh/run_00002_fastback/ | ||
| /lustre/fsw/portfolios/coreai/projects/coreai_modulus_cae/datasets/shift_suv_pnm_mesh/run_00002_fastback/ | ||
| ├── merged_surfaces_filled.stl.pt | ||
| │ ├── meta.json | ||
| │ └── _tensordict | ||
| │ ├── _cache | ||
| │ │ ├── cell | ||
| │ │ │ └── meta.json | ||
| │ │ ├── meta.json | ||
| │ │ └── point | ||
| │ │ └── meta.json | ||
| │ ├── cell_data | ||
| │ │ └── meta.json | ||
| │ ├── cells.memmap | ||
| │ ├── global_data | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ ├── point_data | ||
| │ │ └── meta.json | ||
| │ └── points.memmap | ||
| ├── merged_surfaces.stl.pt | ||
| │ ├── meta.json | ||
| │ └── _tensordict | ||
| │ ├── _cache | ||
| │ │ ├── cell | ||
| │ │ │ └── meta.json | ||
| │ │ ├── meta.json | ||
| │ │ └── point | ||
| │ │ └── meta.json | ||
| │ ├── cell_data | ||
| │ │ └── meta.json | ||
| │ ├── cells.memmap | ||
| │ ├── global_data | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ ├── point_data | ||
| │ │ └── meta.json | ||
| │ └── points.memmap | ||
| ├── merged_surfaces.vtp.pt | ||
| │ ├── meta.json | ||
| │ └── _tensordict | ||
| │ ├── _cache | ||
| │ │ ├── cell | ||
| │ │ │ └── meta.json | ||
| │ │ ├── meta.json | ||
| │ │ └── point | ||
| │ │ └── meta.json | ||
| │ ├── cell_data | ||
| │ │ ├── meta.json | ||
| │ │ ├── Normals.memmap | ||
| │ │ ├── pressure_average.memmap | ||
| │ │ ├── pressure.memmap | ||
| │ │ ├── vtkOriginalCellIds.memmap | ||
| │ │ ├── wall_shear_stress_average.memmap | ||
| │ │ └── wall_shear_stress.memmap | ||
| │ ├── cells.memmap | ||
| │ ├── global_data | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ ├── point_data | ||
| │ │ ├── meta.json | ||
| │ │ ├── Normals.memmap | ||
| │ │ └── vtkOriginalPointIds.memmap | ||
| │ └── points.memmap | ||
| └── merged_volumes.vtu.pt | ||
| ├── meta.json | ||
| └── _tensordict | ||
| ├── _cache | ||
| │ ├── cell | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ └── point | ||
| │ └── meta.json | ||
| ├── cell_data | ||
| │ └── meta.json | ||
| ├── global_data | ||
| │ └── meta.json | ||
| ├── meta.json | ||
| ├── point_data | ||
| │ ├── meta.json | ||
| │ ├── pressure_average.memmap | ||
| │ ├── pressure.memmap | ||
| │ ├── velocity_average.memmap | ||
| │ └── velocity.memmap | ||
| └── points.memmap | ||
|
|
||
| 32 directories, 51 files | ||
|
|
||
| While here is a SHIFT-SUV "Estate" file: | ||
|
|
||
| ⬢ [podman] ❯ tree $GROUP_LUSTRE/datasets/shift_suv_pnm_mesh/run_00004_estate/ | ||
| /lustre/fsw/portfolios/coreai/projects/coreai_modulus_cae/datasets/shift_suv_pnm_mesh/run_00004_estate/ | ||
| ├── merged_surfaces_filled.stl.pt | ||
| │ ├── meta.json | ||
| │ └── _tensordict | ||
| │ ├── _cache | ||
| │ │ ├── cell | ||
| │ │ │ └── meta.json | ||
| │ │ ├── meta.json | ||
| │ │ └── point | ||
| │ │ └── meta.json | ||
| │ ├── cell_data | ||
| │ │ └── meta.json | ||
| │ ├── cells.memmap | ||
| │ ├── global_data | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ ├── point_data | ||
| │ │ └── meta.json | ||
| │ └── points.memmap | ||
| ├── merged_surfaces.stl.pt | ||
| │ ├── meta.json | ||
| │ └── _tensordict | ||
| │ ├── _cache | ||
| │ │ ├── cell | ||
| │ │ │ └── meta.json | ||
| │ │ ├── meta.json | ||
| │ │ └── point | ||
| │ │ └── meta.json | ||
| │ ├── cell_data | ||
| │ │ └── meta.json | ||
| │ ├── cells.memmap | ||
| │ ├── global_data | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ ├── point_data | ||
| │ │ └── meta.json | ||
| │ └── points.memmap | ||
| ├── merged_surfaces.vtp.pt | ||
| │ ├── meta.json | ||
| │ └── _tensordict | ||
| │ ├── _cache | ||
| │ │ ├── cell | ||
| │ │ │ └── meta.json | ||
| │ │ ├── meta.json | ||
| │ │ └── point | ||
| │ │ └── meta.json | ||
| │ ├── cell_data | ||
| │ │ ├── meta.json | ||
| │ │ ├── Normals.memmap | ||
| │ │ ├── pressure_average.memmap | ||
| │ │ ├── pressure.memmap | ||
| │ │ ├── vtkOriginalCellIds.memmap | ||
| │ │ ├── wall_shear_stress_average.memmap | ||
| │ │ └── wall_shear_stress.memmap | ||
| │ ├── cells.memmap | ||
| │ ├── global_data | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ ├── point_data | ||
| │ │ ├── meta.json | ||
| │ │ ├── Normals.memmap | ||
| │ │ └── vtkOriginalPointIds.memmap | ||
| │ └── points.memmap | ||
| └── merged_volumes.vtu.pt | ||
| ├── meta.json | ||
| └── _tensordict | ||
| ├── _cache | ||
| │ ├── cell | ||
| │ │ └── meta.json | ||
| │ ├── meta.json | ||
| │ └── point | ||
| │ └── meta.json | ||
| ├── cell_data | ||
| │ └── meta.json | ||
| ├── global_data | ||
| │ └── meta.json | ||
| ├── meta.json | ||
| ├── point_data | ||
| │ ├── meta.json | ||
| │ ├── pressure_average.memmap | ||
| │ ├── pressure.memmap | ||
| │ ├── velocity_average.memmap | ||
| │ └── velocity.memmap | ||
| └── points.memmap | ||
|
|
||
| 32 directories, 51 files | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.