prgenv-gnu with ROCm 7: vanilla MPICH #299
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| gcc: | ||
| version: "13" | ||
| llvm-amdgpu: | ||
| version: "7.1.0" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,9 @@ | ||
| name: prgenv-gnu | ||
| version: 2 | ||
| spack: | ||
| commit: releases/v1.0 | ||
| repo: https://github.com/spack/spack.git | ||
| packages: | ||
| repo: https://github.com/msimberg/spack-packages.git | ||
| commit: aws-ofi-nccl-with-rocm # develop on 2025-12-19 + https://github.com/spack/spack-packages/compare/develop...msimberg:spack-packages:aws-ofi-nccl-with-rocm | ||
| description: GNU Compiler toolchain with cray-mpich, Python, CMake and other development tools for gfx90a, gfx942 (MI300A, MI250X) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,85 @@ | ||
| gcc-env: | ||
| compiler: [gcc, llvm-amdgpu] | ||
| network: | ||
| mpi: mpich+rocm | ||
| unify: when_possible | ||
| duplicates: | ||
| strategy: full | ||
| specs: | ||
| - boost@1.88.0 +chrono +filesystem +iostreams +mpi +python +regex +serialization +shared +system +timer | ||
| - cmake | ||
| - fftw | ||
| - fmt | ||
| - gsl | ||
| - hdf5+cxx+hl+fortran | ||
| - netlib-scalapack | ||
| - lua | ||
| - libtree | ||
| - lz4 | ||
| - meson | ||
| - ninja | ||
| - openblas threads=openmp | ||
| - osu-micro-benchmarks ~xccl | ||
| - python@3 | ||
| - zlib-ng | ||
| # add GPU-specific packages here, for easier comparison with mc version | ||
| - hip@7.1.0 ^mesa@23.3.6 | ||
| - hdf5+mpi | ||
| - hipcub | ||
| - llvm-amdgpu | ||
| # rocm packages | ||
| - rocblas ^py-msgpack %cxx=gcc | ||
| - rocfft | ||
| - rocm-cmake %cxx=gcc | ||
| - rocm-openmp-extras | ||
| - rocm-core %cxx=gcc | ||
| - rocminfo | ||
| - rocprim | ||
| - rocprofiler-dev | ||
| - rocprofiler-sdk | ||
| - rocprof-trace-decoder | ||
| - rocprofiler-systems +internal-dyninst+internal-boost+trace-decoder | ||
| - rocprofiler-compute | ||
| - rocrand | ||
| - rocsolver | ||
| - rocsparse | ||
| - rocthrust | ||
| - roctracer-dev-api %cxx=gcc | ||
| - roctracer-dev | ||
| - rocm-gdb | ||
| - rocm-debug-agent | ||
| - hsa-rocr-dev | ||
| - comgr | ||
| # hip packages | ||
| - hipblas | ||
| - hipblaslt | ||
| - hiprand | ||
| - hipsparse~shared-libs | ||
| - hipfft | ||
| - hipblas-common | ||
| # - miopen-hip # TODO(iomaganaris): enable for julia AMD gpu build | ||
| # networking | ||
| - rccl | ||
| # commits pinned for reproducibility, not bugs; update when updating recipe | ||
| - aws-ofi-nccl@1.18.0 | ||
| - rccl-tests@git.6405c76e6826663bbb67bd40aeee8c70aa5d3094=develop %cxx=llvm-amdgpu | ||
| variants: | ||
| - +mpi | ||
| - +rocm | ||
| - amdgpu_target=gfx942,gfx90a | ||
| - amdgpu_target_sram_ecc=gfx942,gfx90a | ||
| - ^python@3.12.12 | ||
| views: | ||
| default: | ||
| link: roots | ||
| exclude: ["llvm"] | ||
| uenv: | ||
| add_compilers: true | ||
| prefix_paths: | ||
| LD_LIBRARY_PATH: [lib, lib64] | ||
| env_vars: | ||
| set: | ||
| - FI_PROVIDER: "cxi" | ||
| - MPIR_CVAR_CH4_OFI_ENABLE_HMEM: "1" | ||
| - NCCL_NET_PLUGIN: "ofi" | ||
|
msimberg marked this conversation as resolved.
|
||
| - HSA_NO_SCRATCH_RECLAIM: "1" | ||
|
Comment on lines
+84
to
+85
Collaborator
There was a problem hiding this comment. Unsure if these two belong here, but adding them for now for testing. The new NCCL plugin isn't found anymore by
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| default: | ||
| features: | ||
| - rocm | ||
| - mpi | ||
| - rccl | ||
| - rccl-tests | ||
| - openmp | ||
| - prgenv | ||
| - serial | ||
| - mpich | ||
| - osu-micro-benchmarks | ||
| cc: mpicc | ||
| cxx: mpic++ | ||
| ftn: mpifort | ||
| views: | ||
| - default |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| modules: | ||
| # Paths to check when creating modules for all module sets | ||
| prefix_inspections: | ||
| bin: | ||
| - PATH | ||
| lib: | ||
| - LD_LIBRARY_PATH | ||
| lib64: | ||
| - LD_LIBRARY_PATH | ||
|
|
||
| default: | ||
| arch_folder: false | ||
| # Where to install modules | ||
| roots: | ||
| tcl: /user-environment/modules | ||
| tcl: | ||
| all: | ||
| autoload: none | ||
| hash_length: 0 | ||
| exclude_implicits: true | ||
| exclude: ['%gcc@7.5.0', 'gcc %gcc@7.5.0'] | ||
| projections: | ||
| all: '{name}/{version}' |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| #!/bin/bash | ||
|
|
||
| set -e | ||
|
|
||
| # remove offending environment variables | ||
| jq '.views["default"].env.values.scalar["HIPCC_LINK_FLAGS_APPEND"]|="" | .views["default"].env.values.scalar["HIPCC_COMPILE_FLAGS_APPEND"]|="" ' /user-environment/meta/env.json > /tmp/env.json | ||
| # copy file back to destination | ||
| cp /tmp/env.json /user-environment/meta/env.json |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| diff --git a/library/CMakeLists.txt b/library/CMakeLists.txt | ||
| index 865dcd9..0ca1815 100644 | ||
| --- a/library/CMakeLists.txt | ||
| +++ b/library/CMakeLists.txt | ||
| @@ -72,6 +72,7 @@ include(src/CMakeLists.txt) | ||
| # Create hipBLASLt library | ||
| add_library(hipblaslt ${hipblaslt_source} ${hipblaslt_headers_public}) | ||
| add_library(roc::hipblaslt ALIAS hipblaslt) | ||
| +target_include_directories( hipblaslt PRIVATE ${MSGPACK_DIR}/include ) | ||
|
|
||
| # Target compile definitions | ||
| if(NOT BUILD_CUDA) | ||
| @@ -154,7 +155,7 @@ if( LEGACY_HIPBLAS_DIRECT ) | ||
| ) | ||
| else() | ||
| target_include_directories(hipblaslt | ||
| - PUBLIC $<BUILD_INTERFACE:${HIPBLAS-COMMON_INCLUDE_DIRS}> | ||
| + PUBLIC ${HIPBLAS-COMMON_INCLUDE_DIRS} | ||
| ) | ||
| endif() | ||
|
|
||
| diff --git a/tensilelite/Tensile/Common.py b/tensilelite/Tensile/Common.py | ||
| index 6b5cdc8..9f28e46 100644 | ||
| --- a/tensilelite/Tensile/Common.py | ||
| +++ b/tensilelite/Tensile/Common.py | ||
| @@ -277,6 +277,8 @@ globalParameters["LazyLibraryLoading"] = False # Load library and code object fi | ||
| globalParameters["UseUserArgs"] = False | ||
|
|
||
| globalParameters["RotatingBufferSize"] = 0 # Size in MB | ||
| +globalParameters["LLVMPath"] = os.environ.get("LLVM_PATH") | ||
| +globalParameters["ROCmSMIPath"] = os.environ.get("ROCM_SMI_PATH") | ||
| globalParameters["RotatingMode"] = 0 # Default is 0, allocated in order A0B0C0D0..ANBNCNDN. 1 is in order A0 pad B0 pad .... AN pad BN pad. | ||
| # Mode 0 requires memcpy everytime when the problem changes to reset the data, but mode 1 doesn't. | ||
|
|
||
| @@ -1576,7 +1578,7 @@ def assignGlobalParameters( config ): | ||
| if os.name == "nt": | ||
| globalParameters["ROCmAgentEnumeratorPath"] = locateExe(globalParameters["ROCmBinPath"], "hipinfo.exe") | ||
| else: | ||
| - globalParameters["ROCmAgentEnumeratorPath"] = locateExe(globalParameters["ROCmBinPath"], "rocm_agent_enumerator") | ||
| + globalParameters["ROCmAgentEnumeratorPath"] = os.environ.get("ROCM_AGENT_ENUMERATOR_PATH") | ||
|
|
||
| if "CxxCompiler" in config: | ||
| globalParameters["CxxCompiler"] = config["CxxCompiler"] | ||
| @@ -1602,8 +1604,7 @@ def assignGlobalParameters( config ): | ||
| compiler = "clang++" if globalParameters["CxxCompiler"] == "hipcc" else "amdclang++" | ||
| globalParameters["AssemblerPath"] = locateExe(os.path.join(globalParameters["ROCmPath"], bin_path), compiler) | ||
|
|
||
| - globalParameters["ROCmSMIPath"] = locateExe(globalParameters["ROCmBinPath"], "rocm-smi") | ||
| - globalParameters["ROCmLdPath"] = locateExe(os.path.join(globalParameters["ROCmPath"], "llvm/bin"), "ld.lld") | ||
| + globalParameters["ROCmLdPath"] = locateExe(os.path.join(globalParameters["LLVMPath"], "bin"), "ld.lld") | ||
|
|
||
| globalParameters["ExtractKernelPath"] = locateExe(os.path.join(globalParameters["ROCmPath"], "hip/bin"), "extractkernel") | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,88 @@ | ||
| From 085d965e11cda1830cf325e0d12db3faf61a94d0 Mon Sep 17 00:00:00 2001 | ||
| From: sreenivasa murthy kolam <sreenivasamurthy.kolam@amd.com> | ||
| Date: Thu, 23 May 2024 05:49:34 +0000 | ||
| Subject: [PATCH] Add hipblas include dir in CMakeLists.txt and Modify the LLVM | ||
| Path in the Tensile code for Spack | ||
|
|
||
| --- | ||
| clients/gtest/CMakeLists.txt | 1 + | ||
| library/CMakeLists.txt | 2 ++ | ||
| tensilelite/Tensile/Common.py | 7 ++++--- | ||
| tensilelite/Tensile/Ops/gen_assembly.sh | 2 +- | ||
| 4 files changed, 8 insertions(+), 4 deletions(-) | ||
|
|
||
| diff --git a/clients/gtest/CMakeLists.txt b/clients/gtest/CMakeLists.txt | ||
| index 825bdca..f817e12 100644 | ||
| --- a/clients/gtest/CMakeLists.txt | ||
| +++ b/clients/gtest/CMakeLists.txt | ||
| @@ -53,6 +53,7 @@ target_include_directories( hipblaslt-test | ||
| $<BUILD_INTERFACE:${BLAS_INCLUDE_DIR}> | ||
| $<BUILD_INTERFACE:${BLIS_INCLUDE_DIR}> # may be blank if not used | ||
| $<BUILD_INTERFACE:${GTEST_INCLUDE_DIRS}> | ||
| + $<BUILD_INTERFACE:${HIPBLAS_INCLUDE_DIRS}> | ||
| ) | ||
| message("BLIS_INCLUDE_DIR=" ${BLIS_INCLUDE_DIR}) | ||
| target_link_libraries( hipblaslt-test PRIVATE ${BLAS_LIBRARY} ${GTEST_BOTH_LIBRARIES} roc::hipblaslt ) | ||
| diff --git a/library/CMakeLists.txt b/library/CMakeLists.txt | ||
| index 3252da0..1b8d628 100644 | ||
| --- a/library/CMakeLists.txt | ||
| +++ b/library/CMakeLists.txt | ||
| @@ -72,6 +72,8 @@ include(src/CMakeLists.txt) | ||
| # Create hipBLASLt library | ||
| add_library(hipblaslt ${hipblaslt_source} ${hipblaslt_headers_public}) | ||
| add_library(roc::hipblaslt ALIAS hipblaslt) | ||
| +target_include_directories( hipblaslt PRIVATE ${HIPBLAS_INCLUDE_DIRS} ) | ||
| +target_include_directories( hipblaslt PRIVATE ${MSGPACK_DIR}/include ) | ||
|
|
||
| # Target compile definitions | ||
| if(NOT BUILD_CUDA) | ||
| diff --git a/tensilelite/Tensile/Common.py b/tensilelite/Tensile/Common.py | ||
| index 8ee6373..52d6a97 100644 | ||
| --- a/tensilelite/Tensile/Common.py | ||
| +++ b/tensilelite/Tensile/Common.py | ||
| @@ -273,6 +273,7 @@ globalParameters["LazyLibraryLoading"] = False # Load library and code object fi | ||
| globalParameters["UseUserArgs"] = False | ||
|
|
||
| globalParameters["RotatingBufferSize"] = 0 # Size in MB | ||
| +globalParameters["LLVMPath"] = os.environ.get("LLVM_PATH") | ||
|
|
||
| # Save a copy - since pytest doesn't re-run this initialization code and YAML files can override global settings - odd things can happen | ||
| defaultGlobalParameters = deepcopy(globalParameters) | ||
| @@ -1488,10 +1489,10 @@ def assignGlobalParameters( config ): | ||
| if os.name == "nt": | ||
| globalParameters["AssemblerPath"] = locateExe(globalParameters["ROCmBinPath"], "clang++.exe") | ||
| else: | ||
| - globalParameters["AssemblerPath"] = locateExe(os.path.join(globalParameters["ROCmPath"], "llvm/bin"), "clang++") | ||
| + globalParameters["AssemblerPath"] = locateExe(os.path.join(globalParameters["LLVMPath"], "bin"), "clang++") | ||
|
|
||
| globalParameters["ROCmSMIPath"] = locateExe(globalParameters["ROCmBinPath"], "rocm-smi") | ||
| - globalParameters["ROCmLdPath"] = locateExe(os.path.join(globalParameters["ROCmPath"], "llvm/bin"), "ld.lld") | ||
| + globalParameters["ROCmLdPath"] = locateExe(os.path.join(globalParameters["LLVMPath"], "bin"), "ld.lld") | ||
|
|
||
| globalParameters["ExtractKernelPath"] = locateExe(os.path.join(globalParameters["ROCmPath"], "hip/bin"), "extractkernel") | ||
|
|
||
| @@ -1501,7 +1502,7 @@ def assignGlobalParameters( config ): | ||
| if os.name == "nt": | ||
| globalParameters["ClangOffloadBundlerPath"] = locateExe(globalParameters["ROCmBinPath"], "clang-offload-bundler.exe") | ||
| else: | ||
| - globalParameters["ClangOffloadBundlerPath"] = locateExe(os.path.join(globalParameters["ROCmPath"], "llvm/bin"), "clang-offload-bundler") | ||
| + globalParameters["ClangOffloadBundlerPath"] = locateExe(os.path.join(globalParameters["LLVMPath"], "bin"), "clang-offload-bundler") | ||
|
|
||
| if "ROCmAgentEnumeratorPath" in config: | ||
| globalParameters["ROCmAgentEnumeratorPath"] = config["ROCmAgentEnumeratorPath"] | ||
| diff --git a/tensilelite/Tensile/Ops/gen_assembly.sh b/tensilelite/Tensile/Ops/gen_assembly.sh | ||
| index 230c446..9f7dc6d 100644 | ||
| --- a/tensilelite/Tensile/Ops/gen_assembly.sh | ||
| +++ b/tensilelite/Tensile/Ops/gen_assembly.sh | ||
| @@ -32,7 +32,7 @@ if ! [ -z ${ROCM_PATH+x} ]; then | ||
| rocm_path=${ROCM_PATH} | ||
| fi | ||
|
|
||
| -toolchain=${rocm_path}/llvm/bin/clang++ | ||
| +toolchain=${LLVM_PATH}/bin/clang++ | ||
|
|
||
| . ${venv}/bin/activate | ||
|
|
||
| -- | ||
| 2.39.3 | ||
|
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| diff --git a/tensilelite/Tensile/Common/GlobalParameters.py b/tensilelite/Tensile/Common/GlobalParameters.py | ||
| index 4cbf586..e2602b9 100644 | ||
| --- a/tensilelite/Tensile/Common/GlobalParameters.py | ||
| +++ b/tensilelite/Tensile/Common/GlobalParameters.py | ||
| @@ -552,7 +552,7 @@ def assignGlobalParameters(config, isaInfoMap: Dict[IsaVersion, IsaInfo]): | ||
|
|
||
| globalParameters["ROCmBinPath"] = os.path.join(globalParameters["ROCmPath"], "bin") | ||
| try: | ||
| - globalParameters["ROCmSMIPath"] = locateExe(globalParameters["ROCmBinPath"], "rocm-smi") | ||
| + globalParameters["ROCmSMIPath"] = os.environ.get("ROCM_SMI_PATH") | ||
| except OSError: | ||
| if os.name == "nt": | ||
| # rocm-smi is not presently supported on Windows so do not require it. | ||
| @@ -560,7 +560,7 @@ def assignGlobalParameters(config, isaInfoMap: Dict[IsaVersion, IsaInfo]): | ||
| else: | ||
| raise | ||
| globalParameters["ROCmLdPath"] = locateExe( | ||
| - os.path.join(globalParameters["ROCmPath"], "lib/llvm/bin"), | ||
| + os.environ.get("ROCM_LD_PATH"), | ||
| "ld.lld" if os.name != "nt" else "ld.lld.exe" | ||
| ) | ||
|
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.