prgenv-gnu with ROCm 7#273
Conversation
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12 |
|
|
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12 |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12 |
|
ROCm is a menace: https://gitlab.com/cscs-ci/ci-testing/webhook-ci/mirrors/551234120955960/1440398897047560/-/jobs/12201564988#L3724. hipblaslt seems to be picking up amdclang++ from the system... needs further investigation. |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12 |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12 |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12 |
|
Hi, |
I'll try it out, hopefully no bigger issues (though note that I have issues with other packages before this is usable unfortunately). |
|
spack/spack-packages#2287 to add rocm 7.1.0 is also currently open. It may help, it may make things worse... The PR description does mention a change to hipblaslt, which may change something. |
Yes, it looks like an older version of ROCm is installed on the system and it's choosing the incorrect version of amdclang++. I'll see if I can reproduce the issue and put in a fix. |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12 |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:25.12 |
iomaganaris
dismissed
their stale review
We should fix the issue with the CI before merging this PR
| uenv: | ||
| add_compilers: true | ||
| prefix_paths: | ||
| LD_LIBRARY_PATH: [lib, lib64] |
There was a problem hiding this comment.
| LD_LIBRARY_PATH: [lib, lib64] | |
| LD_LIBRARY_PATH: [lib, lib64] | |
| env_vars: | |
| set: | |
| - NCCL_NET_PLUGIN: "ofi" | |
| - HSA_NO_SCRATCH_RECLAIM: "1" |
see https://github.com/eth-cscs/alps-uenv/pull/299/files#r2912985599.
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
|
This build also seems to be working a lot better (similar to the vanilla mpich build in #299 (comment)) after removing the LD_PRELOAD, adding eth-cscs/alps-cluster-config#79, and updating aws-ofi-nccl, but not perfectly. RCCL tests have slightly worse performance on mi200 but that's ok and we can update the references in cscs-reframe-tests. HPL is still failing and I have no idea what's going on there. |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
|
I reverted commenting out |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
No worries :P I commented it out again (but feel free to reenable it again if you need a new build). I updated the recipe to use the new cray-mpich 9.1 which should officially be meant for rocm 7 (thanks @simonpintarelli for updating it!). |
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
1 similar comment
|
cscs-ci run alps;system=beverin;uarch=mi200;uenv=prgenv-gnu:7.1.0 |
6 participants
No description provided.