From 95f78440f12c7c0bb2e415ac2bff692c594f8a0c Mon Sep 17 00:00:00 2001 From: Peter Scheibel Date: Sat, 26 Aug 2023 15:46:25 -0700 Subject: [PATCH] External ROCm: add example configuration (#39602) * add an example of an external rocm configuration * include more info * generalize section to all GPU support --- lib/spack/docs/gpu_configuration.rst | 113 +++++++++++++++++++++++++++ lib/spack/docs/index.rst | 1 + 2 files changed, 114 insertions(+) create mode 100644 lib/spack/docs/gpu_configuration.rst diff --git a/lib/spack/docs/gpu_configuration.rst b/lib/spack/docs/gpu_configuration.rst new file mode 100644 index 0000000000..1f807bb3b8 --- /dev/null +++ b/lib/spack/docs/gpu_configuration.rst @@ -0,0 +1,113 @@ +.. Copyright 2013-2023 Lawrence Livermore National Security, LLC and other + Spack Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: (Apache-2.0 OR MIT) + +========================== +Using External GPU Support +========================== + +Many packages come with a ``+cuda`` or ``+rocm`` variant. With no added +configuration Spack will download and install the needed components. +It may be preferable to use existing system support: the following sections +help with using a system installation of GPU libraries. + +----------------------------------- +Using an External ROCm Installation +----------------------------------- + +Spack breaks down ROCm into many separate component packages. The following +is an example ``packages.yaml`` that organizes a consistent set of ROCm +components for use by dependent packages: + +.. code-block:: yaml + + packages: + all: + compiler: [rocmcc@=5.3.0] + variants: amdgpu_target=gfx90a + hip: + buildable: false + externals: + - spec: hip@5.3.0 + prefix: /opt/rocm-5.3.0/hip + hsa-rocr-dev: + buildable: false + externals: + - spec: hsa-rocr-dev@5.3.0 + prefix: /opt/rocm-5.3.0/ + llvm-amdgpu: + buildable: false + externals: + - spec: llvm-amdgpu@5.3.0 + prefix: /opt/rocm-5.3.0/llvm/ + comgr: + buildable: false + externals: + - spec: comgr@5.3.0 + prefix: /opt/rocm-5.3.0/ + hipsparse: + buildable: false + externals: + - spec: hipsparse@5.3.0 + prefix: /opt/rocm-5.3.0/ + hipblas: + buildable: false + externals: + - spec: hipblas@5.3.0 + prefix: /opt/rocm-5.3.0/ + rocblas: + buildable: false + externals: + - spec: rocblas@5.3.0 + prefix: /opt/rocm-5.3.0/ + rocprim: + buildable: false + externals: + - spec: rocprim@5.3.0 + prefix: /opt/rocm-5.3.0/rocprim/ + +This is in combination with the following compiler definition: + +.. code-block:: yaml + + compilers: + - compiler: + spec: rocmcc@=5.3.0 + paths: + cc: /opt/rocm-5.3.0/bin/amdclang + cxx: /opt/rocm-5.3.0/bin/amdclang++ + f77: null + fc: /opt/rocm-5.3.0/bin/amdflang + operating_system: rhel8 + target: x86_64 + +This includes the following considerations: + +- Each of the listed externals specifies ``buildable: false`` to force Spack + to use only the externals we defined. +- ``spack external find`` can automatically locate some of the ``hip``/``rocm`` + packages, but not all of them, and furthermore not in a manner that + guarantees a complementary set if multiple ROCm installations are available. +- The ``prefix`` is the same for several components, but note that others + require listing one of the subdirectories as a prefix. + +----------------------------------- +Using an External CUDA Installation +----------------------------------- + +CUDA is split into fewer components and is simpler to specify: + +.. code-block:: yaml + + packages: + all: + variants: + - cuda_arch=70 + cuda: + buildable: false + externals: + - spec: cuda@11.0.2 + prefix: /opt/cuda/cuda-11.0.2/ + +where ``/opt/cuda/cuda-11.0.2/lib/`` contains ``libcudart.so``. diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index 6a80aa1a24..be4a42045d 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -77,6 +77,7 @@ or refer to the full manual below. extensions pipelines signing + gpu_configuration .. toctree:: :maxdepth: 2