prgenv-gnu with ROCm 7: vanilla MPICH#299
Conversation
…with llvm-amdgpu to avoid rebuilding python packages
…linked to rocsparse
Issue is:
ImportError: /user-environment/env/default/lib/libhipsparse.so.4: undefined symbol: rocsparse_destroy_spmv_descr
which is strange because:
ldd /user-environment/env/default/lib/libhipsparse.so.4
linux-vdso.so.1 (0x00007ffddc1c1000)
/user-environment/env/default/lib/librocsparse.so (0x00007f6af9a27000)
nm /user-environment/linux-zen3/rocsparse-7.1.0-qfflrfxfu2vaqu52hitvwyukdwq767rf/lib/librocsparse.so | grep rocsparse_destroy_spmv_descr
0000000002f33fc0 T rocsparse_destroy_spmv_descr
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
|
Note that this PR is required for reframe tests to pick the correct mpi type for slurm: eth-cscs/cscs-reframe-tests#531. |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
1 similar comment
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
|
eth-cscs/cscs-reframe-tests#537 is at the very least required for the failing rocHPL tests. However, it seems like they still fail after that PR as well. |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
2 similar comments
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
| - NCCL_NET_PLUGIN: "ofi" | ||
| - HSA_NO_SCRATCH_RECLAIM: "1" |
There was a problem hiding this comment.
Unsure if these two belong here, but adding them for now for testing.
The new NCCL plugin isn't found anymore by NCCL_NET="AWS Libfabric" (nor NCCL_NET="Libfabric"). RCCL only looks for libnccl-net.so, not librccl-net.so which does exist in the uenv. NCCL_NET_PLUGIN=ofi or NCCL_NET_PLUGIN=librccl-net.so both work with the new uenv. NCCL_NET_PLUGIN=ofi only works with the new uenv but noot the old 6.3.3 uenv. I'll have to see how to best deal with this in the reframe tests.
HSA_NO_SCRATCH_RECLAIM=1 is required with the new uenv because RCCL warns about NCCL WARN HSA_NO_SCRATCH_RECLAIM=1 must be set to avoid RCCL perf hit, rocm ver:70126014... except that if I don't set it nccl-tests don't run at all so it's a bit more than a perf hit.
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
|
At least the RCCLTestsUENV reframe tests now work correctly with the latest environment variables (though I would probably move them to cscs-reframe-tests). sendrecv performance is worse than the reference on mi200 but that can be updated. We don't care much about mi200, particularly not for RCCL (and performance was always worse on mi200 vs mi300). I'm still waiting to see if the patchelf'd cray-mpich works similarly well on #273. I've applied the same aws-ofi-nccl changes and environment variables there. Build here: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/551234120955960/1440398897047560/-/pipelines/2376126538. |
3 participants
No description provided.