hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

tutorial: move tutorial to standalone site (#13450)

Browse source 
* docs: add a spack environment for building the docs

* docs: remove tutorial and link to spack-tutorial.readthedocs.io

The tutorial now has its own standalone website, versioned by instances
of the tutorial.  Link to that instead of versioning it directly with Spack.
This commit is contained in:
Todd Gamblin 2019-10-25 21:49:27 -07:00 committed by GitHub
parent 4af448724f
commit 847703c13d
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
31 changed files with 28 additions and 7565 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
lib/spack/docs/.gitignore vendored
View file

@ -3,3 +3,5 @@ command_index.rst
spack*.rst
llnl*.rst
_build
.spack-env
spack.lock

4
lib/spack/docs/build_systems/intelpackage.rst
View file

@ -802,7 +802,7 @@ by one of the following means:
Configure the order of compilers in the appropriate ``packages.yaml`` file,
under either an ``all:`` or client-package-specific entry, in a
``compiler:`` list. Consult the Spack documentation for
:ref:`Configuring Package Preferences <configs-tutorial-package-prefs>`
`Configuring Package Preferences <https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html#configuring-package-preferences>`_
and
:ref:`Concretization Preferences <concretization-preferences>`.
@ -851,7 +851,7 @@ client packages, edit the ``packages.yaml`` file. Customize, either in the
the virtual packages and whose values are the Spack specs that satisfy the
virtual package, in order of decreasing preference. To learn more about the
``providers:`` settings, see the Spack tutorial for
:ref:`Configuring Package Preferences <configs-tutorial-package-prefs>`
`Configuring Package Preferences <https://spack-tutorial.readthedocs.io/en/latest/tutorial_configuration.html#configuring-package-preferences>`_
and the section
:ref:`Concretization Preferences <concretization-preferences>`.

2
lib/spack/docs/conf.py
View file

@ -159,7 +159,7 @@ def setup(sphinx):
# List of patterns, relative to source directory, that match files and
# directories to ignore when looking for source files.
exclude_patterns = ['_build', '_spack_root']
exclude_patterns = ['_build', '_spack_root', '.spack-env']
# The reST default role (used for this markup: `text`) to use for all documents.
#default_role = None

2
lib/spack/docs/index.rst
View file

@ -55,7 +55,7 @@ or refer to the full manual below.
getting_started
basic_usage
workflows
tutorial
Tutorial: Spack 101 <https://spack-tutorial.readthedocs.io>
known_issues
.. toctree::

19
lib/spack/docs/spack.yaml Normal file
View file

@ -0,0 +1,19 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
#
# These are requirements for building the documentation. You can run
# these commands in this directory to install Sphinx and its plugins,
# then build the docs:
#
# spack install
# spack env activate .
# make
#
spack:
specs:
- py-sphinx
- py-sphinxcontrib-programoutput
- py-sphinx-rtd-theme

75
lib/spack/docs/tutorial.rst
View file

@ -1,75 +0,0 @@
.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _spack-101:
=============================
Tutorial: Spack 101
=============================
This is a full-day introduction to Spack with lectures and live demos.
It was last presented at the `Practice and Experience in Advanced
Research Computing Conference (PEARC19)
<https://www.pearc19.pearc.org/>`_ on July 31, 2019.
You can use these materials to teach a course on Spack at your own site,
or you can just skip ahead and read the live demo scripts to see how
Spack is used in practice.
.. _sc16-slides:
.. rubric:: Slides
.. figure:: tutorial/sc16-tutorial-slide-preview.png
:target: https://spack.io/slides/spack-pearc19-tutorial-slides.pdf
:height: 72px
:align: left
:alt: Slide Preview
`Download Slides <https://spack.io/slides/spack-pearc19-tutorial-slides.pdf>`_.
**Full citation:** Levi Baber, Gregory Becker, Adam J. Stewart, and Todd
Gamblin. Managing HPC Software Complexity with Spack. Tutorial presented
at the Practice and Experience in Advanced Research Computing Conference
(PEARC19). July 31, 2019. Chicago, IL, USA.
.. _sc16-live-demos:
.. rubric:: Live Demos
We provide scripts that take you step-by-step through basic Spack tasks.
They correspond to sections in the slides above. You can use one of the
following methods to run through the scripts:
1. We provide the `spack/tutorial
<https://hub.docker.com/r/spack/tutorial>`_ container image on
Docker Hub that you can use to do the tutorial on your local
machine. You can invoke ``docker run -it spack/tutorial`` to start
using the container.
2. When we host the tutorial, we also provision VM instances in `AWS
<https://aws.amazon.com/>`_, so that users who are unfamiliar with
Docker can simply log into a VPM to do the demo exercises.
You should now be ready to run through our demo scripts:
1. :ref:`basics-tutorial`
2. :ref:`configs-tutorial`
3. :ref:`packaging-tutorial`
4. :ref:`environments-tutorial`
5. :ref:`modules-tutorial`
6. :ref:`build-systems-tutorial`
7. :ref:`advanced-packaging-tutorial`
Full contents:
.. toctree::
tutorial_basics
tutorial_configuration
tutorial_packaging
tutorial_environments
tutorial_modules
tutorial_buildsystems
tutorial_advanced_packaging

39
lib/spack/docs/tutorial/examples/0.package.py
View file

@ -1,39 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
#
# This is a template package file for Spack. We've put "FIXME"
# next to all the things you'll want to change. Once you've handled
# them, you can save this file and test your package like this:
#
# spack install mpileaks
#
# You can edit this file again by typing:
#
# spack edit mpileaks
#
# See the Spack documentation for more information on packaging.
# If you submit this package back to Spack as a pull request,
# please first remove this boilerplate and all FIXME comments.
#
from spack import *
class Mpileaks(Package):
"""FIXME: Put a proper description of your package here."""
# FIXME: Add a proper url for your package's homepage here.
homepage = "http://www.example.com"
url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
# FIXME: Add dependencies if required.
# depends_on('foo')
def install(self, spec, prefix):
# FIXME: Unknown build system
make()
make('install')

24
lib/spack/docs/tutorial/examples/1.package.py
View file

@ -1,24 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Mpileaks(Package):
"""Tool to detect and report MPI objects like MPI_Requests and
MPI_Datatypes."""
homepage = "https://github.com/LLNL/mpileaks"
url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
# FIXME: Add dependencies if required.
# depends_on('foo')
def install(self, spec, prefix):
# FIXME: Unknown build system
make()
make('install')

25
lib/spack/docs/tutorial/examples/2.package.py
View file

@ -1,25 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Mpileaks(Package):
"""Tool to detect and report MPI objects like MPI_Requests and
MPI_Datatypes."""
homepage = "https://github.com/LLNL/mpileaks"
url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
depends_on('mpi')
depends_on('adept-utils')
depends_on('callpath')
def install(self, spec, prefix):
# FIXME: Unknown build system
make()
make('install')

25
lib/spack/docs/tutorial/examples/3.package.py
View file

@ -1,25 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Mpileaks(Package):
"""Tool to detect and report MPI objects like MPI_Requests and
MPI_Datatypes."""
homepage = "https://github.com/LLNL/mpileaks"
url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
depends_on('mpi')
depends_on('adept-utils')
depends_on('callpath')
def install(self, spec, prefix):
configure()
make()
make('install')

27
lib/spack/docs/tutorial/examples/4.package.py
View file

@ -1,27 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Mpileaks(Package):
"""Tool to detect and report MPI objects like MPI_Requests and
MPI_Datatypes."""
homepage = "https://github.com/LLNL/mpileaks"
url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
depends_on('mpi')
depends_on('adept-utils')
depends_on('callpath')
def install(self, spec, prefix):
configure('--prefix={0}'.format(prefix),
'--with-adept-utils={0}'.format(spec['adept-utils'].prefix),
'--with-callpath={0}'.format(spec['callpath'].prefix))
make()
make('install')

42
lib/spack/docs/tutorial/examples/5.package.py
View file

@ -1,42 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Mpileaks(Package):
"""Tool to detect and report MPI objects like MPI_Requests and
MPI_Datatypes."""
homepage = "https://github.com/LLNL/mpileaks"
url = "https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
version('1.0', sha256='2e34cc4505556d1c1f085758e26f2f8eea0972db9382f051b2dcfb1d7d9e1825')
variant('stackstart', values=int, default=0,
description='Specify the number of stack frames to truncate')
depends_on('mpi')
depends_on('adept-utils')
depends_on('callpath')
def install(self, spec, prefix):
stackstart = int(spec.variants['stackstart'].value)
args = [
'--prefix={0}'.format(prefix),
'--with-adept-utils={0}'.format(spec['adept-utils'].prefix),
'--with-callpath={0}'.format(spec['callpath'].prefix),
]
if stackstart:
args.extend([
'--with-stack-start-c={0}'.format(stackstart),
'--with-stack-start-fortran={0}'.format(stackstart)
])
configure(*args)
make()
make('install')

27
lib/spack/docs/tutorial/examples/Autotools/0.package.py
View file

@ -1,27 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Mpileaks(AutotoolsPackage):
"""Tool to detect and report leaked MPI objects like MPI_Requests and
MPI_Datatypes."""
homepage = "https://github.com/hpc/mpileaks"
url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
version('1.0', '8838c574b39202a57d7c2d68692718aa')
depends_on("mpi")
depends_on("adept-utils")
depends_on("callpath")
def install(self, spec, prefix):
configure("--prefix=" + prefix,
"--with-adept-utils=" + spec['adept-utils'].prefix,
"--with-callpath=" + spec['callpath'].prefix)
make()
make("install")

32
lib/spack/docs/tutorial/examples/Autotools/1.package.py
View file

@ -1,32 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Mpileaks(AutotoolsPackage):
"""Tool to detect and report leaked MPI objects like MPI_Requests and
MPI_Datatypes."""
homepage = "https://github.com/hpc/mpileaks"
url = "https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz"
version('1.0', '8838c574b39202a57d7c2d68692718aa')
variant("stackstart", values=int, default=0,
description="Specify the number of stack frames to truncate")
depends_on("mpi")
depends_on("adept-utils")
depends_on("callpath")
def configure_args(self):
stackstart = int(self.spec.variants['stackstart'].value)
args = ["--with-adept-utils=" + spec['adept-utils'].prefix,
"--with-callpath=" + spec['callpath'].prefix]
if stackstart:
args.extend(['--with-stack-start-c=%s' % stackstart,
'--with-stack-start-fortran=%s' % stackstart])
return args

41
lib/spack/docs/tutorial/examples/Cmake/0.package.py
View file

@ -1,41 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
#
# This is a template package file for Spack. We've put "FIXME"
# next to all the things you'll want to change. Once you've handled
# them, you can save this file and test your package like this:
#
# spack install callpath
#
# You can edit this file again by typing:
#
# spack edit callpath
#
# See the Spack documentation for more information on packaging.
# If you submit this package back to Spack as a pull request,
# please first remove this boilerplate and all FIXME comments.
#
from spack import *
class Callpath(CMakePackage):
"""FIXME: Put a proper description of your package here."""
# FIXME: Add a proper url for your package's homepage here.
homepage = "http://www.example.com"
url = "https://github.com/llnl/callpath/archive/v1.0.1.tar.gz"
version('1.0.3', 'c89089b3f1c1ba47b09b8508a574294a')
# FIXME: Add dependencies if required.
# depends_on('foo')
def cmake_args(self):
# FIXME: Add arguments other than
# FIXME: CMAKE_INSTALL_PREFIX and CMAKE_BUILD_TYPE
# FIXME: If not needed delete this function
args = []
return args

23
lib/spack/docs/tutorial/examples/Cmake/1.package.py
View file

@ -1,23 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Callpath(CMakePackage):
"""Library for representing callpaths consistently in
distributed-memory performance tools."""
homepage = "https://github.com/llnl/callpath"
url = "https://github.com/llnl/callpath/archive/v1.0.3.tar.gz"
version('1.0.3', 'c89089b3f1c1ba47b09b8508a574294a')
depends_on("elf", type="link")
depends_on("libdwarf")
depends_on("dyninst")
depends_on("adept-utils")
depends_on("mpi")
depends_on("cmake@2.8:", type="build")

33
lib/spack/docs/tutorial/examples/Cmake/2.package.py
View file

@ -1,33 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Callpath(CMakePackage):
"""Library for representing callpaths consistently in
distributed-memory performance tools."""
homepage = "https://github.com/llnl/callpath"
url = "https://github.com/llnl/callpath/archive/v1.0.3.tar.gz"
version('1.0.3', 'c89089b3f1c1ba47b09b8508a574294a')
depends_on("elf", type="link")
depends_on("libdwarf")
depends_on("dyninst")
depends_on("adept-utils")
depends_on("mpi")
depends_on("cmake@2.8:", type="build")
def cmake_args(self):
args = ["-DCALLPATH_WALKER=dyninst"]
if self.spec.satisfies("^dyninst@9.3.0:"):
std.flag = self.compiler.cxx_flag
args.append("-DCMAKE_CXX_FLAGS='{0}' -fpermissive'".format(
std_flag))
return args

26
lib/spack/docs/tutorial/examples/Makefile/0.package.py
View file

@ -1,26 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Bowtie(MakefilePackage):
"""FIXME: Put a proper description of your package here."""
# FIXME: Add a proper url for your package's homepage here.
homepage = "http://www.example.com"
url = "https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip"
version('1.2.1.1', 'ec06265730c5f587cd58bcfef6697ddf')
# FIXME: Add dependencies if required.
# depends_on('foo')
def edit(self, spec, prefix):
# FIXME: Edit the Makefile if necessary
# FIXME: If not needed delete this function
# makefile = FileFilter('Makefile')
# makefile.filter('CC = .*', 'CC = cc')
return

27
lib/spack/docs/tutorial/examples/Makefile/1.package.py
View file

@ -1,27 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Bowtie(MakefilePackage):
"""Bowtie is an ultrafast, memory efficient short read aligner
for short DNA sequences (reads) from next-gen sequencers."""
homepage = "https://sourceforge.net/projects/bowtie-bio/"
url = "https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip"
version('1.2.1.1', 'ec06265730c5f587cd58bcfef6697ddf')
variant("tbb", default=False, description="Use Intel thread building block")
depends_on("tbb", when="+tbb")
def edit(self, spec, prefix):
# FIXME: Edit the Makefile if necessary
# FIXME: If not needed delete this function
# makefile = FileFilter('Makefile')
# makefile.filter('CC = .*', 'CC = cc')
return

25
lib/spack/docs/tutorial/examples/Makefile/2.package.py
View file

@ -1,25 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Bowtie(MakefilePackage):
"""Bowtie is an ultrafast, memory efficient short read aligner
for short DNA sequences (reads) from next-gen sequencers."""
homepage = "https://sourceforge.net/projects/bowtie-bio/"
url = "https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip"
version('1.2.1.1', 'ec06265730c5f587cd58bcfef6697ddf')
variant("tbb", default=False, description="Use Intel thread building block")
depends_on("tbb", when="+tbb")
def edit(self, spec, prefix):
makefile = FileFilter("Makefile")
makefile.filter('CC= .*', 'CC = ' + env['CC'])
makefile.filter('CXX = .*', 'CXX = ' + env['CXX'])

36
lib/spack/docs/tutorial/examples/Makefile/3.package.py
View file

@ -1,36 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Bowtie(MakefilePackage):
"""Bowtie is an ultrafast, memory efficient short read aligner
for short DNA sequences (reads) from next-gen sequencers."""
homepage = "https://sourceforge.net/projects/bowtie-bio/"
url = "https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip"
version('1.2.1.1', 'ec06265730c5f587cd58bcfef6697ddf')
variant("tbb", default=False, description="Use Intel thread building block")
depends_on("tbb", when="+tbb")
def edit(self, spec, prefix):
makefile = FileFilter("Makefile")
makefile.filter('CC= .*', 'CC = ' + env['CC'])
makefile.filter('CXX = .*', 'CXX = ' + env['CXX'])
@property
def build_targets(self):
if "+tbb" in spec:
return []
else:
return ["NO_TBB=1"]
@property
def install_targets(self):
return ['prefix={0}'.format(self.prefix), 'install']

41
lib/spack/docs/tutorial/examples/PyPackage/0.package.py
View file

@ -1,41 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
#
# This is a template package file for Spack. We've put "FIXME"
# next to all the things you'll want to change. Once you've handled
# them, you can save this file and test your package like this:
#
# spack install py-pandas
#
# You can edit this file again by typing:
#
# spack edit py-pandas
#
# See the Spack documentation for more information on packaging.
# If you submit this package back to Spack as a pull request,
# please first remove this boilerplate and all FIXME comments.
#
from spack import *
class PyPandas(PythonPackage):
"""FIXME: Put a proper description of your package here."""
# FIXME: Add a proper url for your package's homepage here.
homepage = "http://www.example.com"
url = "https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz"
version('0.19.0', 'bc9bb7188e510b5d44fbdd249698a2c3')
# FIXME: Add dependencies if required.
# depends_on('py-setuptools', type='build')
# depends_on('py-foo', type=('build', 'run'))
def build_args(self, spec, prefix):
# FIXME: Add arguments other than --prefix
# FIXME: If not needed delete this function
args = []
return args

32
lib/spack/docs/tutorial/examples/PyPackage/1.package.py
View file

@ -1,32 +0,0 @@
# Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class PyPandas(PythonPackage):
"""pandas is a Python package providing fast, flexible, and expressive
data structures designed to make working with relational or
labeled data both easy and intuitive. It aims to be the
fundamental high-level building block for doing practical, real
world data analysis in Python. Additionally, it has the broader
goal of becoming the most powerful and flexible open source data
analysis / manipulation tool available in any language.
"""
homepage = "http://pandas.pydata.org/"
url = "https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz"
version('0.19.0', 'bc9bb7188e510b5d44fbdd249698a2c3')
version('0.18.0', 'f143762cd7a59815e348adf4308d2cf6')
version('0.16.1', 'fac4f25748f9610a3e00e765474bdea8')
version('0.16.0', 'bfe311f05dc0c351f8955fbd1e296e73')
depends_on('py-dateutil', type=('build', 'run'))
depends_on('py-numpy', type=('build', 'run'))
depends_on('py-setuptools', type='build')
depends_on('py-cython', type='build')
depends_on('py-pytz', type=('build', 'run'))
depends_on('py-numexpr', type=('build', 'run'))
depends_on('py-bottleneck', type=('build', 'run'))

BIN
lib/spack/docs/tutorial/sc16-tutorial-slide-preview.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 70 KiB

515
lib/spack/docs/tutorial_advanced_packaging.rst
View file

@ -1,515 +0,0 @@
.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _advanced-packaging-tutorial:
============================
Advanced Topics in Packaging
============================
Spack tries to automatically configure packages with information from
dependencies such that all you need to do is to list the dependencies
(i.e., with the ``depends_on`` directive) and the build system (for example
by deriving from :code:`CmakePackage`).
However, there are many special cases. Often you need to retrieve details
about dependencies to set package-specific configuration options, or to
define package-specific environment variables used by the package's build
system. This tutorial covers how to retrieve build information from
dependencies, and how you can automatically provide important information to
dependents in your package.
----------------------
Setup for the tutorial
----------------------
.. note::
We do not recommend doing this section of the tutorial in a
production Spack instance.
The tutorial uses custom package definitions with missing sections that
will be filled in during the tutorial. These package definitions are stored
in a separate package repository, which can be enabled with:
.. code-block:: console
$ spack repo add --scope=site var/spack/repos/tutorial
This section of the tutorial may also require a newer version of
gcc. If you have not already installed gcc@7.2.0 and added it to your
configuration, you can do so with:
.. code-block:: console
$ spack install gcc@7.2.0 %gcc@5.4.0
$ spack compiler add --scope=site `spack location -i gcc@7.2.0 %gcc@5.4.0`
If you are using the tutorial docker image, all dependency packages
will have been installed. Otherwise, to install these packages you can use
the following commands:
.. code-block:: console
$ spack install openblas
$ spack install netlib-lapack
$ spack install mpich
Now, you are ready to set your preferred ``EDITOR`` and continue with
the rest of the tutorial.
.. note::
Several of these packages depend on an MPI implementation. You can use
OpenMPI if you install it from scratch, but this is slow (>10 min.).
A binary cache of MPICH may be provided, in which case you can force
the package to use it and install quickly. All tutorial examples with
packages that depend on MPICH include the spec syntax for building with it
.. _adv_pkg_tutorial_start:
---------------------------------------
Modifying a package's build environment
---------------------------------------
Spack sets up several environment variables like ``PATH`` by default to aid in
building a package, but many packages make use of environment variables which
convey specific information about their dependencies (e.g., ``MPICC``).
This section covers how to update your Spack packages so that package-specific
environment variables are defined at build-time.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Set environment variables in dependent packages at build-time
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Dependencies can set environment variables that are required when their
dependents build. For example, when a package depends on a python extension
like py-numpy, Spack's ``python`` package will add it to ``PYTHONPATH``
so it is available at build time; this is required because the default setup
that spack does is not sufficient for python to import modules.
Any package can override the
:py:func:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>`
method to setup the build environment for a dependent.
This method takes as an argument a :py:class:`EnvironmentModifications <spack.util.environment.EnvironmentModifications>`
object which includes convenience methods to update the environment. For
example, an MPI implementation can set ``MPICC`` for packages that depend on it:
.. code-block:: python
def setup_dependent_build_environment(self, env, dependent_spec):
env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
In this case packages that depend on ``mpi`` will have ``MPICC`` defined in
their environment when they build. This section is focused on setting up the
build-time environment but it's worth noting that a similar method called
:py:func:`setup_dependent_run_environment <spack.package.PackageBase.setup_dependent_run_environment>`
can be used to code modifications that will be included in Spack's automatically-generated
module files.
We can practice by editing the ``mpich`` package to set the ``MPICC``
environment variable in the build-time environment of dependent packages.
.. code-block:: console
root@advanced-packaging-tutorial:/# spack edit mpich
Once you're finished, the method should look like this:
.. code-block:: python
def setup_dependent_build_environment(self, env, dependent_spec):
env.set('MPICC', join_path(self.prefix.bin, 'mpicc'))
env.set('MPICXX', join_path(self.prefix.bin, 'mpic++'))
env.set('MPIF77', join_path(self.prefix.bin, 'mpif77'))
env.set('MPIF90', join_path(self.prefix.bin, 'mpif90'))
env.set('MPICH_CC', spack_cc)
env.set('MPICH_CXX', spack_cxx)
env.set('MPICH_F77', spack_f77)
env.set('MPICH_F90', spack_fc)
env.set('MPICH_FC', spack_fc)
At this point we can, for instance, install ``netlib-scalapack`` with
``mpich``:
.. code-block:: console
root@advanced-packaging-tutorial:/# spack install netlib-scalapack ^mpich
...
==> Created stage in /usr/local/var/spack/stage/netlib-scalapack-2.0.2-km7tsbgoyyywonyejkjoojskhc5knz3z
==> No patches needed for netlib-scalapack
==> Building netlib-scalapack [CMakePackage]
==> Executing phase: 'cmake'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed netlib-scalapack
Fetch: 0.01s. Build: 3m 59.86s. Total: 3m 59.87s.
[+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-scalapack-2.0.2-km7tsbgoyyywonyejkjoojskhc5knz3z
and double check the environment logs to verify that every variable was
set to the correct value.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Set environment variables in your own package
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Packages can override the
:py:func:`setup_build_environment <spack.package.PackageBase.setup_build_environment>`
or the
:py:func:`setup_run_environment <spack.package.PackageBase.setup_run_environment>`
methods to modify their own build-time or run-time environment respectively.
An example of a package that overrides both methods is ``qt``:
.. code-block:: python
def setup_build_environment(self, env):
env.set('MAKEFLAGS', '-j{0}'.format(make_jobs))
def setup_run_environment(self, env):
env.set('QTDIR', self.prefix)
When ``qt`` builds, ``MAKEFLAGS`` will be defined in the environment. Likewise, when a
module file is created for ``qt`` it will contain commands to define ``QTDIR`` appropriately.
To contrast with ``qt``'s
:py:func:`setup_dependent_build_environment <spack.package.PackageBase.setup_dependent_build_environment>`
function:
.. code-block:: python
def setup_dependent_build_environment(self, env, dependent_spec):
env.set('QTDIR', self.prefix)
Let's see how it works by completing the ``elpa`` package:
.. code-block:: console
root@advanced-packaging-tutorial:/# spack edit elpa
In the end your method should look like:
.. code-block:: python
def setup_build_environment(self, env):
spec = self.spec
env.set('CC', spec['mpi'].mpicc)
env.set('FC', spec['mpi'].mpifc)
env.set('CXX', spec['mpi'].mpicxx)
env.set('SCALAPACK_LDFLAGS', spec['scalapack'].libs.joined())
env.append_flags('LDFLAGS', spec['lapack'].libs.search_flags)
env.append_flags('LIBS', spec['lapack'].libs.link_flags)
At this point it's possible to proceed with the installation of ``elpa ^mpich``
------------------------------
Retrieving library information
------------------------------
Although Spack attempts to help packages locate their dependency libraries
automatically (e.g. by setting ``PKG_CONFIG_PATH`` and ``CMAKE_PREFIX_PATH``),
a package may have unique configuration options that are required to locate
libraries. When a package needs information about dependency libraries, the
general approach in Spack is to query the dependencies for the locations of
their libraries and set configuration options accordingly. By default most
Spack packages know how to automatically locate their libraries. This section
covers how to retrieve library information from dependencies and how to locate
libraries when the default logic doesn't work.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Accessing dependency libraries
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you need to access the libraries of a dependency, you can do so
via the ``libs`` property of the spec, for example in the ``arpack-ng``
package:
.. code-block:: python
def install(self, spec, prefix):
lapack_libs = spec['lapack'].libs.joined(';')
blas_libs = spec['blas'].libs.joined(';')
cmake(*[
'-DLAPACK_LIBRARIES={0}'.format(lapack_libs),
'-DBLAS_LIBRARIES={0}'.format(blas_libs)
], '.')
Note that ``arpack-ng`` is querying virtual dependencies, which Spack
automatically resolves to the installed implementation (e.g. ``openblas``
for ``blas``).
We've started work on a package for ``armadillo``. You should open it,
read through the comment that starts with ``# TUTORIAL:`` and complete
the ``cmake_args`` section:
.. code-block:: console
root@advanced-packaging-tutorial:/# spack edit armadillo
If you followed the instructions in the package, when you are finished your
``cmake_args`` method should look like:
.. code-block:: python
def cmake_args(self):
spec = self.spec
return [
# ARPACK support
'-DARPACK_LIBRARY={0}'.format(spec['arpack-ng'].libs.joined(";")),
# BLAS support
'-DBLAS_LIBRARY={0}'.format(spec['blas'].libs.joined(";")),
# LAPACK support
'-DLAPACK_LIBRARY={0}'.format(spec['lapack'].libs.joined(";")),
# SuperLU support
'-DSuperLU_INCLUDE_DIR={0}'.format(spec['superlu'].prefix.include),
'-DSuperLU_LIBRARY={0}'.format(spec['superlu'].libs.joined(";")),
# HDF5 support
'-DDETECT_HDF5={0}'.format('ON' if '+hdf5' in spec else 'OFF')
]
As you can see, getting the list of libraries that your dependencies provide
is as easy as accessing the their ``libs`` attribute. Furthermore, the interface
remains the same whether you are querying regular or virtual dependencies.
At this point you can complete the installation of ``armadillo`` using ``openblas``
as a LAPACK provider (``armadillo ^openblas ^mpich``):
.. code-block:: console
root@advanced-packaging-tutorial:/# spack install armadillo ^openblas ^mpich
==> pkg-config is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkg-config-0.29.2-ae2hwm7q57byfbxtymts55xppqwk7ecj
...
==> superlu is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/superlu-5.2.1-q2mbtw2wo4kpzis2e2n227ip2fquxrno
==> Installing armadillo
==> Using cached archive: /usr/local/var/spack/cache/armadillo/armadillo-8.100.1.tar.xz
==> Staging archive: /usr/local/var/spack/stage/armadillo-8.100.1-n2eojtazxbku6g4l5izucwwgnpwz77r4/armadillo-8.100.1.tar.xz
==> Created stage in /usr/local/var/spack/stage/armadillo-8.100.1-n2eojtazxbku6g4l5izucwwgnpwz77r4
==> Applied patch undef_linux.patch
==> Building armadillo [CMakePackage]
==> Executing phase: 'cmake'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed armadillo
Fetch: 0.01s. Build: 3.96s. Total: 3.98s.
[+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/armadillo-8.100.1-n2eojtazxbku6g4l5izucwwgnpwz77r4
Hopefully the installation went fine and the code we added expanded to the right list
of semicolon separated libraries (you are encouraged to open ``armadillo``'s
build logs to double check).
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Providing libraries to dependents
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Spack provides a default implementation for ``libs`` which often works
out of the box. A user can write a package definition without having to
implement a ``libs`` property and dependents can retrieve its libraries
as shown in the above section. However, the default implementation assumes that
libraries follow the naming scheme ``lib<package name>.so`` (or e.g.
``lib<package name>.a`` for static libraries). Packages which don't
follow this naming scheme must implement this function themselves, e.g.
``opencv``:
.. code-block:: python
@property
def libs(self):
shared = "+shared" in self.spec
return find_libraries(
"libopencv_*", root=self.prefix, shared=shared, recurse=True
)
This issue is common for packages which implement an interface (i.e.
virtual package providers in Spack). If we try to build another version of
``armadillo`` tied to ``netlib-lapack`` (``armadillo ^netlib-lapack ^mpich``)
we'll notice that this time the installation won't complete:
.. code-block:: console
root@advanced-packaging-tutorial:/# spack install armadillo ^netlib-lapack ^mpich
==> pkg-config is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/pkg-config-0.29.2-ae2hwm7q57byfbxtymts55xppqwk7ecj
...
==> openmpi is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f
==> Installing arpack-ng
==> Using cached archive: /usr/local/var/spack/cache/arpack-ng/arpack-ng-3.5.0.tar.gz
==> Already staged arpack-ng-3.5.0-bloz7cqirpdxj33pg7uj32zs5likz2un in /usr/local/var/spack/stage/arpack-ng-3.5.0-bloz7cqirpdxj33pg7uj32zs5likz2un
==> No patches needed for arpack-ng
==> Building arpack-ng [Package]
==> Executing phase: 'install'
==> Error: RuntimeError: Unable to recursively locate netlib-lapack libraries in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-lapack-3.6.1-jjfe23wgt7nkjnp2adeklhseg3ftpx6z
RuntimeError: RuntimeError: Unable to recursively locate netlib-lapack libraries in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netlib-lapack-3.6.1-jjfe23wgt7nkjnp2adeklhseg3ftpx6z
/usr/local/var/spack/repos/builtin/packages/arpack-ng/package.py:105, in install:
5 options.append('-DCMAKE_INSTALL_NAME_DIR:PATH=%s/lib' % prefix)
6
7 # Make sure we use Spack's blas/lapack:
>> 8 lapack_libs = spec['lapack'].libs.joined(';')
9 blas_libs = spec['blas'].libs.joined(';')
10
11 options.extend([
See build log for details:
/usr/local/var/spack/stage/arpack-ng-3.5.0-bloz7cqirpdxj33pg7uj32zs5likz2un/arpack-ng-3.5.0/spack-build-out.txt
Unlike ``openblas`` which provides a library named ``libopenblas.so``,
``netlib-lapack`` provides ``liblapack.so``, so it needs to implement
customized library search logic. Let's edit it:
.. code-block:: console
root@advanced-packaging-tutorial:/# spack edit netlib-lapack
and follow the instructions in the ``# TUTORIAL:`` comment as before.
What we need to implement is:
.. code-block:: python
@property
def lapack_libs(self):
shared = True if '+shared' in self.spec else False
return find_libraries(
'liblapack', root=self.prefix, shared=shared, recursive=True
)
i.e., a property that returns the correct list of libraries for the LAPACK interface.
We use the name ``lapack_libs`` rather than ``libs`` because
``netlib-lapack`` can also provide ``blas``, and when it does it is provided
as a separate library file. Using this name ensures that when
dependents ask for ``lapack`` libraries, ``netlib-lapack`` will retrieve only
the libraries associated with the ``lapack`` interface. Now we can finally
install ``armadillo ^netlib-lapack ^mpich``:
.. code-block:: console
root@advanced-packaging-tutorial:/# spack install armadillo ^netlib-lapack ^mpich
...
==> Building armadillo [CMakePackage]
==> Executing phase: 'cmake'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed armadillo
Fetch: 0.01s. Build: 3.75s. Total: 3.76s.
[+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/armadillo-8.100.1-sxmpu5an4dshnhickh6ykchyfda7jpyn
Since each implementation of a virtual package is responsible for locating the
libraries associated with the interfaces it provides, dependents do not need
to include special-case logic for different implementations and for example
need only ask for :code:`spec['blas'].libs`.
----------------------
Other Packaging Topics
----------------------
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Attach attributes to other packages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Build tools usually also provide a set of executables that can be used
when another package is being installed. Spack gives you the opportunity
to monkey-patch dependent modules and attach attributes to them. This
helps make the packager experience as similar as possible to what would
have been the manual installation of the same package.
An example here is the ``automake`` package, which overrides
:py:func:`setup_dependent_package <spack.package.PackageBase.setup_dependent_package>`:
.. code-block:: python
def setup_dependent_package(self, module, dependent_spec):
# Automake is very likely to be a build dependency,
# so we add the tools it provides to the dependent module
executables = ['aclocal', 'automake']
for name in executables:
setattr(module, name, self._make_executable(name))
so that every other package that depends on it can use directly ``aclocal``
and ``automake`` with the usual function call syntax of :py:class:`Executable <spack.util.executable.Executable>`:
.. code-block:: python
aclocal('--force')
^^^^^^^^^^^^^^^^^^^^^^^
Extra query parameters
^^^^^^^^^^^^^^^^^^^^^^^
An advanced feature of the Spec's build-interface protocol is the support
for extra parameters after the subscript key. In fact, any of the keys used in the query
can be followed by a comma-separated list of extra parameters which can be
inspected by the package receiving the request to fine-tune a response.
Let's look at an example and try to install ``netcdf ^mpich``:
.. code-block:: console
root@advanced-packaging-tutorial:/# spack install netcdf ^mpich
==> libsigsegv is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz
==> m4 is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-r5envx3kqctwwflhd4qax4ahqtt6x43a
...
==> Error: AttributeError: 'list' object has no attribute 'search_flags'
AttributeError: AttributeError: 'list' object has no attribute 'search_flags'
/usr/local/var/spack/repos/builtin/packages/netcdf/package.py:207, in configure_args:
50 # used instead.
51 hdf5_hl = self.spec['hdf5:hl']
52 CPPFLAGS.append(hdf5_hl.headers.cpp_flags)
>> 53 LDFLAGS.append(hdf5_hl.libs.search_flags)
54
55 if '+parallel-netcdf' in self.spec:
56 config_args.append('--enable-pnetcdf')
See build log for details:
/usr/local/var/spack/stage/netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj/netcdf-4.4.1.1/spack-build-out.txt
We can see from the error that ``netcdf`` needs to know how to link the *high-level interface*
of ``hdf5``, and thus passes the extra parameter ``hl`` after the request to retrieve it.
Clearly the implementation in the ``hdf5`` package is not complete, and we need to fix it:
.. code-block:: console
root@advanced-packaging-tutorial:/# spack edit hdf5
If you followed the instructions correctly, the code added to the
``lib`` property should be similar to:
.. code-block:: python
:emphasize-lines: 1
query_parameters = self.spec.last_query.extra_parameters
key = tuple(sorted(query_parameters))
libraries = query2libraries[key]
shared = '+shared' in self.spec
return find_libraries(
libraries, root=self.prefix, shared=shared, recurse=True
)
where we highlighted the line retrieving the extra parameters. Now we can successfully
complete the installation of ``netcdf ^mpich``:
.. code-block:: console
root@advanced-packaging-tutorial:/# spack install netcdf ^mpich
==> libsigsegv is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libsigsegv-2.11-fypapcprssrj3nstp6njprskeyynsgaz
==> m4 is already installed in /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/m4-1.4.18-r5envx3kqctwwflhd4qax4ahqtt6x43a
...
==> Installing netcdf
==> Using cached archive: /usr/local/var/spack/cache/netcdf/netcdf-4.4.1.1.tar.gz
==> Already staged netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj in /usr/local/var/spack/stage/netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj
==> Already patched netcdf
==> Building netcdf [AutotoolsPackage]
==> Executing phase: 'autoreconf'
==> Executing phase: 'configure'
==> Executing phase: 'build'
==> Executing phase: 'install'
==> Successfully installed netcdf
Fetch: 0.01s. Build: 24.61s. Total: 24.62s.
[+] /usr/local/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/netcdf-4.4.1.1-gk2xxhbqijnrdwicawawcll4t3c7dvoj

1736
lib/spack/docs/tutorial_basics.rst
View file

File diff suppressed because it is too large Load diff

807
lib/spack/docs/tutorial_buildsystems.rst
View file

@ -1,807 +0,0 @@
.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _build-systems-tutorial:
==============================
Spack Package Build Systems
==============================
You may begin to notice after writing a couple of package template files a
pattern emerge for some packages. For example, you may find yourself writing
an :code:`install()` method that invokes: :code:`configure`, :code:`cmake`,
:code:`make`, :code:`make install`. You may also find yourself writing
:code:`"prefix=" + prefix` as an argument to :code:`configure` or :code:`cmake`.
Rather than having you repeat these lines for all packages, Spack has
classes that can take care of these patterns. In addition,
these package files allow for finer grained control of these build systems.
In this section, we will describe each build system and give examples on
how these can be manipulated to install a package.
-----------------------
Package Class Hierarchy
-----------------------
.. graphviz::
digraph G {
node [
shape = "record"
]
edge [
arrowhead = "empty"
]
PackageBase -> Package [dir=back]
PackageBase -> MakefilePackage [dir=back]
PackageBase -> AutotoolsPackage [dir=back]
PackageBase -> CMakePackage [dir=back]
PackageBase -> PythonPackage [dir=back]
}
The above diagram gives a high level view of the class hierarchy and how each
package relates. Each subclass inherits from the :code:`PackageBaseClass`
super class. The bulk of the work is done in this super class which includes
fetching, extracting to a staging directory and installing. Each subclass
then adds additional build-system-specific functionality. In the following
sections, we will go over examples of how to utilize each subclass and to see
how powerful these abstractions are when packaging.
-----------------
Package
-----------------
We've already seen examples of a :code:`Package` class in our walkthrough for writing
package files, so we won't be spending much time with them here. Briefly,
the Package class allows for abitrary control over the build process, whereas
subclasses rely on certain patterns (e.g. :code:`configure` :code:`make`
:code:`make install`) to be useful. :code:`Package` classes are particularly useful
for packages that have a non-conventional way of being built since the packager
can utilize some of Spack's helper functions to customize the building and
installing of a package.
-------------------
Autotools
-------------------
As we have seen earlier, packages using :code:`Autotools` use :code:`configure`,
:code:`make` and :code:`make install` commands to execute the build and
install process. In our :code:`Package` class, your typical build incantation will
consist of the following:
.. code-block:: python
def install(self, spec, prefix):
configure("--prefix=" + prefix)
make()
make("install")
You'll see that this looks similar to what we wrote in our packaging tutorial.
The :code:`Autotools` subclass aims to simplify writing package files and provides
convenience methods to manipulate each of the different phases for a :code:`Autotools`
build system.
:code:`Autotools` packages consist of four phases:
1. :code:`autoreconf()`
2. :code:`configure()`
3. :code:`build()`
4. :code:`install()`
Each of these phases have sensible defaults. Let's take a quick look at some
the internals of the :code:`Autotools` class:
.. code-block:: console
$ spack edit --build-system autotools
This will open the :code:`AutotoolsPackage` file in your text editor.
.. note::
The examples showing code for these classes is abridged to avoid having
long examples. We only show what is relevant to the packager.
.. literalinclude:: _spack_root/lib/spack/spack/build_systems/autotools.py
:language: python
:emphasize-lines: 33,36,54
:lines: 30-76,240-248
:linenos:
Important to note are the highlighted lines. These properties allow the
packager to set what build targets and install targets they want for their
package. If, for example, we wanted to add as our build target :code:`foo`
then we can append to our :code:`build_targets` property:
.. code-block:: python
build_targets = ["foo"]
Which is similiar to invoking make in our Package
.. code-block:: python
make("foo")
This is useful if we have packages that ignore environment variables and need
a command-line argument.
Another thing to take note of is in the :code:`configure()` method.
Here we see that the :code:`prefix` argument is already included since it is a
common pattern amongst packages using :code:`Autotools`. We then only have to
override :code:`configure_args()`, which will then return it's output to
to :code:`configure()`. Then, :code:`configure()` will append the common
arguments
Packagers also have the option to run :code:`autoreconf` in case a package
needs to update the build system and generate a new :code:`configure`. Though,
for the most part this will be unnecessary.
Let's look at the :code:`mpileaks` package.py file that we worked on earlier:
.. code-block:: console
$ spack edit mpileaks
Notice that mpileaks is a :code:`Package` class but uses the :code:`Autotools`
build system. Although this package is acceptable let's make this into an
:code:`AutotoolsPackage` class and simplify it further.
.. literalinclude:: tutorial/examples/Autotools/0.package.py
:language: python
:emphasize-lines: 9
:linenos:
We first inherit from the :code:`AutotoolsPackage` class.
Although we could keep the :code:`install()` method, most of it can be handled
by the :code:`AutotoolsPackage` base class. In fact, the only thing that needs
to be overridden is :code:`configure_args()`.
.. literalinclude:: tutorial/examples/Autotools/1.package.py
:language: python
:emphasize-lines: 25,26,27,28,29,30,31,32
:linenos:
Since Spack takes care of setting the prefix for us we can exclude that as
an argument to :code:`configure`. Our packages look simpler, and the packager
does not need to worry about whether they have properly included :code:`configure`
and :code:`make`.
This version of the :code:`mpileaks` package installs the same as the previous,
but the :code:`AutotoolsPackage` class lets us do it with a cleaner looking
package file.
-----------------
Makefile
-----------------
Packages that utilize :code:`Make` or a :code:`Makefile` usually require you
to edit a :code:`Makefile` to set up platform and compiler specific variables.
These packages are handled by the :code:`Makefile` subclass which provides
convenience methods to help write these types of packages.
A :code:`MakefilePackage` class has three phases that can be overridden. These include:
1. :code:`edit()`
2. :code:`build()`
3. :code:`install()`
Packagers then have the ability to control how a :code:`Makefile` is edited, and
what targets to include for the build phase or install phase.
Let's also take a look inside the :code:`MakefilePackage` class:
.. code-block:: console
$ spack edit --build-system makefile
Take note of the following:
.. literalinclude:: _spack_root/lib/spack/spack/build_systems/makefile.py
:language: python
:lines: 14,43-61,70-88
:emphasize-lines: 21,27,34
:linenos:
Similar to :code:`Autotools`, :code:`MakefilePackage` class has properties
that can be set by the packager. We can also override the different
methods highlighted.
Let's try to recreate the Bowtie_ package:
.. _Bowtie: http://bowtie-bio.sourceforge.net/index.shtml
.. code-block:: console
$ spack create -f https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip
==> This looks like a URL for bowtie
==> Found 1 version of bowtie:
1.2.1.1 https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip
==> How many would you like to checksum? (default is 1, q to abort) 1
==> Downloading...
==> Fetching https://downloads.sourceforge.net/project/bowtie-bio/bowtie/1.2.1.1/bowtie-1.2.1.1-src.zip
######################################################################## 100.0%
==> Checksummed 1 version of bowtie
==> This package looks like it uses the makefile build system
==> Created template for bowtie package
==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/bowtie/package.py
Once the fetching is completed, Spack will open up your text editor in the
usual fashion and create a template of a :code:`MakefilePackage` package.py.
.. literalinclude:: tutorial/examples/Makefile/0.package.py
:language: python
:linenos:
Spack was successfully able to detect that :code:`Bowtie` uses :code:`Make`.
Let's add in the rest of our details for our package:
.. literalinclude:: tutorial/examples/Makefile/1.package.py
:language: python
:emphasize-lines: 10,11,13,14,18,20
:linenos:
As we mentioned earlier, most packages using a :code:`Makefile` have hard-coded
variables that must be edited. These variables are fine if you happen to not
care about setup or types of compilers used but Spack is designed to work with
any compiler. The :code:`MakefilePackage` subclass makes it easy to edit
these :code:`Makefiles` by having an :code:`edit()` method that
can be overridden.
Let's take a look at the default :code:`Makefile` that :code:`Bowtie` provides.
If we look inside, we see that :code:`CC` and :code:`CXX` point to our GNU
compiler:
.. code-block:: console
$ spack stage bowtie
.. note::
As usual make sure you have shell support activated with spack:
:code:`source /path/to/spack_root/spack/share/spack/setup-env.sh`
.. code-block:: console
$ spack cd -s bowtie
$ cd bowtie-1.2
$ vim Makefile
.. code-block:: make
CPP = g++ -w
CXX = $(CPP)
CC = gcc
LIBS = $(LDFLAGS) -lz
HEADERS = $(wildcard *.h)
To fix this, we need to use the :code:`edit()` method to write our custom
:code:`Makefile`.
.. literalinclude:: tutorial/examples/Makefile/2.package.py
:language: python
:emphasize-lines: 23,24,25
:linenos:
Here we use a :code:`FileFilter` object to edit our :code:`Makefile`. It takes
in a regular expression and then replaces :code:`CC` and :code:`CXX` to whatever
Spack sets :code:`CC` and :code:`CXX` environment variables to. This allows us to
build :code:`Bowtie` with whatever compiler we specify through Spack's
:code:`spec` syntax.
Let's change the build and install phases of our package:
.. literalinclude:: tutorial/examples/Makefile/3.package.py
:language: python
:emphasize-lines: 28,29,30,31,32,35,36
:linenos:
Here demonstrate another strategy that we can use to manipulate our package
We can provide command-line arguments to :code:`make()`. Since :code:`Bowtie`
can use :code:`tbb` we can either add :code:`NO_TBB=1` as a argument to prevent
:code:`tbb` support or we can just invoke :code:`make` with no arguments.
:code:`Bowtie` requires our :code:`install_target` to provide a path to
the install directory. We can do this by providing :code:`prefix=` as a command
line argument to :code:`make()`.
Let's look at a couple of other examples and go through them:
.. code-block:: console
$ spack edit esmf
Some packages allow environment variables to be set and will honor them.
Packages that use :code:`?=` for assignment in their :code:`Makefile`
can be set using environment variables. In our :code:`esmf` example we
set two environment variables in our :code:`edit()` method:
.. code-block:: python
def edit(self, spec, prefix):
for var in os.environ:
if var.startswith('ESMF_'):
os.environ.pop(var)
# More code ...
if self.compiler.name == 'gcc':
os.environ['ESMF_COMPILER'] = 'gfortran'
elif self.compiler.name == 'intel':
os.environ['ESMF_COMPILER'] = 'intel'
elif self.compiler.name == 'clang':
os.environ['ESMF_COMPILER'] = 'gfortranclang'
elif self.compiler.name == 'nag':
os.environ['ESMF_COMPILER'] = 'nag'
elif self.compiler.name == 'pgi':
os.environ['ESMF_COMPILER'] = 'pgi'
else:
msg = "The compiler you are building with, "
msg += "'{0}', is not supported by ESMF."
raise InstallError(msg.format(self.compiler.name))
As you may have noticed, we didn't really write anything to the :code:`Makefile`
but rather we set environment variables that will override variables set in
the :code:`Makefile`.
Some packages include a configuration file that sets certain compiler variables,
platform specific variables, and the location of dependencies or libraries.
If the file is simple and only requires a couple of changes, we can overwrite
those entries with our :code:`FileFilter` object. If the configuration involves
complex changes, we can write a new configuration file from scratch.
Let's look at an example of this in the :code:`elk` package:
.. code-block:: console
$ spack edit elk
.. code-block:: python
def edit(self, spec, prefix):
# Dictionary of configuration options
config = {
'MAKE': 'make',
'AR': 'ar'
}
# Compiler-specific flags
flags = ''
if self.compiler.name == 'intel':
flags = '-O3 -ip -unroll -no-prec-div'
elif self.compiler.name == 'gcc':
flags = '-O3 -ffast-math -funroll-loops'
elif self.compiler.name == 'pgi':
flags = '-O3 -lpthread'
elif self.compiler.name == 'g95':
flags = '-O3 -fno-second-underscore'
elif self.compiler.name == 'nag':
flags = '-O4 -kind=byte -dusty -dcfuns'
elif self.compiler.name == 'xl':
flags = '-O3'
config['F90_OPTS'] = flags
config['F77_OPTS'] = flags
# BLAS/LAPACK support
# Note: BLAS/LAPACK must be compiled with OpenMP support
# if the +openmp variant is chosen
blas = 'blas.a'
lapack = 'lapack.a'
if '+blas' in spec:
blas = spec['blas'].libs.joined()
if '+lapack' in spec:
lapack = spec['lapack'].libs.joined()
# lapack must come before blas
config['LIB_LPK'] = ' '.join([lapack, blas])
# FFT support
if '+fft' in spec:
config['LIB_FFT'] = join_path(spec['fftw'].prefix.lib,
'libfftw3.so')
config['SRC_FFT'] = 'zfftifc_fftw.f90'
else:
config['LIB_FFT'] = 'fftlib.a'
config['SRC_FFT'] = 'zfftifc.f90'
# MPI support
if '+mpi' in spec:
config['F90'] = spec['mpi'].mpifc
config['F77'] = spec['mpi'].mpif77
else:
config['F90'] = spack_fc
config['F77'] = spack_f77
config['SRC_MPI'] = 'mpi_stub.f90'
# OpenMP support
if '+openmp' in spec:
config['F90_OPTS'] += ' ' + self.compiler.openmp_flag
config['F77_OPTS'] += ' ' + self.compiler.openmp_flag
else:
config['SRC_OMP'] = 'omp_stub.f90'
# Libxc support
if '+libxc' in spec:
config['LIB_libxc'] = ' '.join([
join_path(spec['libxc'].prefix.lib, 'libxcf90.so'),
join_path(spec['libxc'].prefix.lib, 'libxc.so')
])
config['SRC_libxc'] = ' '.join([
'libxc_funcs.f90',
'libxc.f90',
'libxcifc.f90'
])
else:
config['SRC_libxc'] = 'libxcifc_stub.f90'
# Write configuration options to include file
with open('make.inc', 'w') as inc:
for key in config:
inc.write('{0} = {1}\n'.format(key, config[key]))
:code:`config` is just a dictionary that we can add key-value pairs to. By the
end of the :code:`edit()` method we write the contents of our dictionary to
:code:`make.inc`.
---------------
CMake
---------------
CMake_ is another common build system that has been gaining popularity. It works
in a similar manner to :code:`Autotools` but with differences in variable names,
the number of configuration options available, and the handling of shared libraries.
Typical build incantations look like this:
.. _CMake: https://cmake.org
.. code-block:: python
def install(self, spec, prefix):
cmake("-DCMAKE_INSTALL_PREFIX:PATH=/path/to/install_dir ..")
make()
make("install")
As you can see from the example above, it's very similar to invoking
:code:`configure` and :code:`make` in an :code:`Autotools` build system. However,
the variable names and options differ. Most options in CMake are prefixed
with a :code:`'-D'` flag to indicate a configuration setting.
In the :code:`CMakePackage` class we can override the following phases:
1. :code:`cmake()`
2. :code:`build()`
3. :code:`install()`
The :code:`CMakePackage` class also provides sensible defaults so we only need to
override :code:`cmake_args()`.
Let's look at these defaults in the :code:`CMakePackage` class in the :code:`_std_args()` method:
.. code-block:: console
$ spack edit --build-system cmake
.. literalinclude:: _spack_root/lib/spack/spack/build_systems/cmake.py
:language: python
:lines: 102-147
:emphasize-lines: 10,18,24,36,37,38,44
:linenos:
Some :code:`CMake` packages use different generators. Spack is able to support
Unix-Makefile_ generators as well as Ninja_ generators.
.. _Unix-Makefile: https://cmake.org/cmake/help/v3.4/generator/Unix%20Makefiles.html
.. _Ninja: https://cmake.org/cmake/help/v3.4/generator/Ninja.html
If no generator is specified Spack will default to :code:`Unix Makefiles`.
Next we setup the build type. In :code:`CMake` you can specify the build type
that you want. Options include:
1. :code:`empty`
2. :code:`Debug`
3. :code:`Release`
4. :code:`RelWithDebInfo`
5. :code:`MinSizeRel`
With these options you can specify whether you want your executable to have
the debug version only, release version or the release with debug information.
Release executables tend to be more optimized than Debug. In Spack, we set
the default as RelWithDebInfo unless otherwise specified through a variant.
Spack then automatically sets up the :code:`-DCMAKE_INSTALL_PREFIX` path,
appends the build type (:code:`RelWithDebInfo` default), and then specifies a verbose
:code:`Makefile`.
Next we add the :code:`rpaths` to :code:`-DCMAKE_INSTALL_RPATH:STRING`.
Finally we add to :code:`-DCMAKE_PREFIX_PATH:STRING` the locations of all our
dependencies so that :code:`CMake` can find them.
In the end our :code:`cmake` line will look like this (example is :code:`xrootd`):
.. code-block:: console
$ cmake $HOME/spack/var/spack/stage/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/xrootd-4.6.0 -G Unix Makefiles -DCMAKE_INSTALL_PREFIX:PATH=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk -DCMAKE_BUILD_TYPE:STRING=RelWithDebInfo -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON -DCMAKE_FIND_FRAMEWORK:STRING=LAST -DCMAKE_INSTALL_RPATH_USE_LINK_PATH:BOOL=FALSE -DCMAKE_INSTALL_RPATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib:$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/xrootd-4.6.0-4ydm74kbrp4xmcgda5upn33co5pwddyk/lib64 -DCMAKE_PREFIX_PATH:STRING=$HOME/spack/opt/spack/darwin-sierra-x86_64/clang-9.0.0-apple/cmake-3.9.4-hally3vnbzydiwl3skxcxcbzsscaasx5
We can see now how :code:`CMake` takes care of a lot of the boilerplate code
that would have to be otherwise typed in.
Let's try to recreate callpath_:
.. _callpath: https://github.com/LLNL/callpath.git
.. code-block:: console
$ spack create -f https://github.com/llnl/callpath/archive/v1.0.3.tar.gz
==> This looks like a URL for callpath
==> Found 4 versions of callpath:
1.0.3 https://github.com/LLNL/callpath/archive/v1.0.3.tar.gz
1.0.2 https://github.com/LLNL/callpath/archive/v1.0.2.tar.gz
1.0.1 https://github.com/LLNL/callpath/archive/v1.0.1.tar.gz
1.0 https://github.com/LLNL/callpath/archive/v1.0.tar.gz
==> How many would you like to checksum? (default is 1, q to abort) 1
==> Downloading...
==> Fetching https://github.com/LLNL/callpath/archive/v1.0.3.tar.gz
######################################################################## 100.0%
==> Checksummed 1 version of callpath
==> This package looks like it uses the cmake build system
==> Created template for callpath package
==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/callpath/package.py
which then produces the following template:
.. literalinclude:: tutorial/examples/Cmake/0.package.py
:language: python
:linenos:
Again we fill in the details:
.. literalinclude:: tutorial/examples/Cmake/1.package.py
:language: python
:linenos:
:emphasize-lines: 9,13,14,18,19,20,21,22,23
As mentioned earlier, Spack will use sensible defaults to prevent repeated code
and to make writing :code:`CMake` package files simpler.
In callpath, we want to add options to :code:`CALLPATH_WALKER` as well as add
compiler flags. We add the following options like so:
.. literalinclude:: tutorial/examples/Cmake/2.package.py
:language: python
:linenos:
:emphasize-lines: 26,30,31
Now we can control our build options using :code:`cmake_args()`. If defaults are
sufficient enough for the package, we can leave this method out.
:code:`CMakePackage` classes allow for control of other features in the
build system. For example, you can specify the path to the "out of source"
build directory and also point to the root of the :code:`CMakeLists.txt` file if it
is placed in a non-standard location.
A good example of a package that has its :code:`CMakeLists.txt` file located at a
different location is found in :code:`spades`.
.. code-block:: console
$ spack edit spades
.. code-block:: python
root_cmakelists_dir = "src"
Here :code:`root_cmakelists_dir` will tell Spack where to find the location
of :code:`CMakeLists.txt`. In this example, it is located a directory level below in
the :code:`src` directory.
Some :code:`CMake` packages also require the :code:`install` phase to be
overridden. For example, let's take a look at :code:`sniffles`.
.. code-block:: console
$ spack edit sniffles
In the :code:`install()` method, we have to manually install our targets
so we override the :code:`install()` method to do it for us:
.. code-block:: python
# the build process doesn't actually install anything, do it by hand
def install(self, spec, prefix):
mkdir(prefix.bin)
src = "bin/sniffles-core-{0}".format(spec.version.dotted)
binaries = ['sniffles', 'sniffles-debug']
for b in binaries:
install(join_path(src, b), join_path(prefix.bin, b))
--------------
PythonPackage
--------------
Python extensions and modules are built differently from source than most
applications. Python uses a :code:`setup.py` script to install Python modules.
The script consists of a call to :code:`setup()` which provides the information
required to build a module to Distutils. If you're familiar with pip or
easy_install, setup.py does the same thing.
These modules are usually installed using the following line:
.. code-block:: console
$ python setup.py install
There are also a list of commands and phases that you can call. To see the full
list you can run:
.. code-block:: console
$ python setup.py --help-commands
Standard commands:
build build everything needed to install
build_py "build" pure Python modules (copy to build directory)
build_ext build C/C++ extensions (compile/link to build directory)
build_clib build C/C++ libraries used by Python extensions
build_scripts "build" scripts (copy and fixup #! line)
clean (no description available)
install install everything from build directory
install_lib install all Python modules (extensions and pure Python)
install_headers install C/C++ header files
install_scripts install scripts (Python or otherwise)
install_data install data files
sdist create a source distribution (tarball, zip file, etc.)
register register the distribution with the Python package index
bdist create a built (binary) distribution
bdist_dumb create a "dumb" built distribution
bdist_rpm create an RPM distribution
bdist_wininst create an executable installer for MS Windows
upload upload binary package to PyPI
check perform some checks on the package
We can write package files for Python packages using the :code:`Package` class,
but the class brings with it a lot of methods that are useless for Python packages.
Instead, Spack has a :code:`PythonPackage` subclass that allows packagers
of Python modules to be able to invoke :code:`setup.py` and use :code:`Distutils`,
which is much more familiar to a typical python user.
To see the defaults that Spack has for each a methods, we will take a look
at the :code:`PythonPackage` class:
.. code-block:: console
$ spack edit --build-system python
We see the following:
.. literalinclude:: _spack_root/lib/spack/spack/build_systems/python.py
:language: python
:lines: 19,146-357
:linenos:
Each of these methods have sensible defaults or they can be overridden.
We will write a package file for Pandas_:
.. _pandas: https://pandas.pydata.org
.. code-block:: console
$ spack create -f https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz
==> This looks like a URL for pandas
==> Warning: Spack was unable to fetch url list due to a certificate verification problem. You can try running spack -k, which will not check SSL certificates. Use this at your own risk.
==> Found 1 version of pandas:
0.19.0 https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz
==> How many would you like to checksum? (default is 1, q to abort) 1
==> Downloading...
==> Fetching https://pypi.io/packages/source/p/pandas/pandas-0.19.0.tar.gz
######################################################################## 100.0%
==> Checksummed 1 version of pandas
==> This package looks like it uses the python build system
==> Changing package name from pandas to py-pandas
==> Created template for py-pandas package
==> Created package file: /Users/mamelara/spack/var/spack/repos/builtin/packages/py-pandas/package.py
And we are left with the following template:
.. literalinclude:: tutorial/examples/PyPackage/0.package.py
:language: python
:linenos:
As you can see this is not any different than any package template that we have
written. We have the choice of providing build options or using the sensible
defaults
Luckily for us, there is no need to provide build args.
Next we need to find the dependencies of a package. Dependencies are usually
listed in :code:`setup.py`. You can find the dependencies by searching for
:code:`install_requires` keyword in that file. Here it is for :code:`Pandas`:
.. code-block:: python
# ... code
if sys.version_info[0] >= 3:
setuptools_kwargs = {
'zip_safe': False,
'install_requires': ['python-dateutil >= 2',
'pytz >= 2011k',
'numpy >= %s' % min_numpy_ver],
'setup_requires': ['numpy >= %s' % min_numpy_ver],
}
if not _have_setuptools:
sys.exit("need setuptools/distribute for Py3k"
"\n$ pip install distribute")
# ... more code
You can find a more comprehensive list at the Pandas documentation_.
.. _documentation: https://pandas.pydata.org/pandas-docs/stable/install.html
By reading the documentation and :code:`setup.py` we found that :code:`Pandas`
depends on :code:`python-dateutil`, :code:`pytz`, and :code:`numpy`, :code:`numexpr`,
and finally :code:`bottleneck`.
Here is the completed :code:`Pandas` script:
.. literalinclude:: tutorial/examples/PyPackage/1.package.py
:language: python
:linenos:
It is quite important to declare all the dependencies of a Python package.
Spack can "activate" Python packages to prevent the user from having to
load each dependency module explictly. If a dependency is missed, Spack will
be unable to properly activate the package and it will cause an issue. To
learn more about extensions go to :ref:`cmd-spack-extensions`.
From this example, you can see that building Python modules is made easy
through the :code:`PythonPackage` class.
-------------------
Other Build Systems
-------------------
Although we won't get in depth with any of the other build systems that Spack
supports, it is worth mentioning that Spack does provide subclasses
for the following build systems:
1. :code:`IntelPackage`
2. :code:`SconsPackage`
3. :code:`WafPackage`
4. :code:`RPackage`
5. :code:`PerlPackage`
6. :code:`QMakePackage`
Each of these classes have their own abstractions to help assist in writing
package files. For whatever doesn't fit nicely into the other build-systems,
you can use the :code:`Package` class.
Hopefully by now you can see how we aim to make packaging simple and
robust through these classes. If you want to learn more about these build
systems, check out :ref:`installation_procedure` in the Packaging Guide.

951
lib/spack/docs/tutorial_configuration.rst
View file

@ -1,951 +0,0 @@
.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _configs-tutorial:
======================
Configuration Tutorial
======================
This tutorial will guide you through various configuration options
that allow you to customize Spack's behavior with respect to
software installation. We will first cover the configuration file
hierarchy. Then, we will cover configuration options for compilers,
focusing on how they can be used to extend Spack's compiler auto-detection.
Next, we will cover the packages configuration file, focusing on
how it can be used to override default build options as well as
specify external package installations to use. Finally, we will
briefly touch on the config configuration file, which manages more
high-level Spack configuration options.
For all of these features, we will demonstrate how we build up a full
configuration file. For some, we will then demonstrate how the
configuration affects the install command, and for others we will use
the ``spack spec`` command to demonstrate how the configuration
changes have affected Spack's concretization algorithm. The provided
output is all from a server running Ubuntu version 16.04.
.. _configs-tutorial-scopes:
--------------------
Configuration Scopes
--------------------
Depending on your use case, you may want to provide configuration
settings common to everyone on your team, or you may want to set
default behaviors specific to a single user account. Spack provides
six configuration *scopes* to handle this customization. These scopes,
in order of decreasing priority, are:
============ ===================================================
Scope Directory
============ ===================================================
Command-line N/A
Custom Custom directory, specified with ``--config-scope``
User ``~/.spack/``
Site ``$SPACK_ROOT/etc/spack/``
System ``/etc/spack/``
Defaults ``$SPACK_ROOT/etc/spack/defaults/``
============ ===================================================
Spack's default configuration settings reside in
``$SPACK_ROOT/etc/spack/defaults``. These are useful for reference,
but should never be directly edited. To override these settings,
create new configuration files in any of the higher-priority
configuration scopes.
A particular cluster may have multiple Spack installations associated
with different projects. To provide settings common to all Spack
installations, put your configuration files in ``/etc/spack``.
To provide settings specific to a particular Spack installation,
you can use the ``$SPACK_ROOT/etc/spack`` directory.
For settings specific to a particular user, you will want to add
configuration files to the ``~/.spack`` directory. When Spack first
checked for compilers on your system, you may have noticed that it
placed your compiler configuration in this directory.
Configuration settings can also be placed in a custom location,
which is then specified on the command line via ``--config-scope``.
An example use case is managing two sets of configurations, one for
development and another for production preferences.
Settings specified on the command line have precedence over all
other configuration scopes.
You can also use ``spack config blame <config>`` for displaying
the effective configuration. Spack will show from which scopes
the configuration has been assembled.
^^^^^^^^^^^^^^^^^^^^^^^^
Platform-specific Scopes
^^^^^^^^^^^^^^^^^^^^^^^^
Some facilities manage multiple platforms from a single shared
file system. In order to handle this, each of the configuration
scopes listed above has two *sub-scopes*: platform-specific and
platform-independent. For example, compiler settings can be stored
in ``compilers.yaml`` configuration files in the following locations:
#. ``~/.spack/<platform>/compilers.yaml``
#. ``~/.spack/compilers.yaml``
#. ``$SPACK_ROOT/etc/spack/<platform>/compilers.yaml``
#. ``$SPACK_ROOT/etc/spack/compilers.yaml``
#. ``/etc/spack/<platform>/compilers.yaml``
#. ``/etc/spack/compilers.yaml``
#. ``$SPACK_ROOT/etc/defaults/<platform>/compilers.yaml``
#. ``$SPACK_ROOT/etc/defaults/compilers.yaml``
These files are listed in decreasing order of precedence, so files in
``~/.spack/<platform>`` will override settings in ``~/.spack``.
-----------
YAML Format
-----------
Spack configurations are YAML dictionaries. Every configuration file
begins with a top-level dictionary that tells Spack which
configuration set it modifies. When Spack checks its configuration,
the configuration scopes are updated as dictionaries in increasing
order of precedence, allowing higher precedence files to override
lower. YAML dictionaries use a colon ":" to specify key-value
pairs. Spack extends YAML syntax slightly to allow a double-colon
"::" to specify a key-value pair. When a double-colon is used to
specify a key-value pair, instead of adding that section, Spack
replaces what was in that section with the new value. For example,
consider a user's compilers configuration file as follows:
.. code-block:: yaml
compilers::
- compiler:
environment: {}
extra_rpaths: []
flags: {}
modules: []
operating_system: ubuntu16.04
paths:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
spec: gcc@5.4.0
target: x86_64
This ensures that no other compilers are used, as the user configuration
scope is the last scope searched and the ``compilers::`` line replaces
all previous configuration files information. If the same
configuration file had a single colon instead of the double colon, it
would add the GCC version 5.4.0 compiler to whatever other compilers
were listed in other configuration files.
.. _configs-tutorial-compilers:
----------------------
Compiler Configuration
----------------------
For most tasks, we can use Spack with the compilers auto-detected the
first time Spack runs on a system. As discussed in the basic
installation tutorial, we can also tell Spack where compilers are
located using the ``spack compiler add`` command. However, in some
circumstances we want even more fine-grained control over the
compilers available. This section will teach you how to exercise that
control using the compilers configuration file.
We will start by opening the compilers configuration file:
.. code-block:: console
$ spack config edit compilers
.. code-block:: yaml
compilers:
- compiler:
environment: {}
extra_rpaths: []
flags: {}
modules: []
operating_system: ubuntu16.04
paths:
cc: /usr/bin/clang-3.7
cxx: /usr/bin/clang++-3.7
f77: null
fc: null
spec: clang@3.7.1-2ubuntu2
target: x86_64
- compiler:
environment: {}
extra_rpaths: []
flags: {}
modules: []
operating_system: ubuntu16.04
paths:
cc: /usr/bin/clang
cxx: /usr/bin/clang++
f77: null
fc: null
spec: clang@3.8.0-2ubuntu4
target: x86_64
- compiler:
environment: {}
extra_rpaths: []
flags: {}
modules: []
operating_system: ubuntu16.04
paths:
cc: /usr/bin/gcc-4.7
cxx: /usr/bin/g++-4.7
f77: /usr/bin/gfortran-4.7
fc: /usr/bin/gfortran-4.7
spec: gcc@4.7
target: x86_64
- compiler:
environment: {}
extra_rpaths: []
flags: {}
modules: []
operating_system: ubuntu16.04
paths:
cc: /usr/bin/gcc
cxx: /usr/bin/g++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
spec: gcc@5.4.0
target: x86_64
This specifies two versions of the GCC compiler and two versions of the
Clang compiler with no Flang compiler. Now suppose we have a code that
we want to compile with the Clang compiler for C/C++ code, but with
gfortran for Fortran components. We can do this by adding another entry
to the ``compilers.yaml`` file.
.. code-block:: yaml
- compiler:
environment: {}
extra_rpaths: []
flags: {}
modules: []
operating_system: ubuntu16.04
paths:
cc: /usr/bin/clang
cxx: /usr/bin/clang++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
spec: clang@3.8.0-gfortran
target: x86_64
Let's talk about the sections of this compiler entry that we've changed.
The biggest change we've made is to the ``paths`` section. This lists
the paths to the compilers to use for each language/specification.
In this case, we point to the Clang compiler for C/C++ and the gfortran
compiler for both specifications of Fortran. We've also changed the
``spec`` entry for this compiler. The ``spec`` entry is effectively the
name of the compiler for Spack. It consists of a name and a version
number, separated by the ``@`` sigil. The name must be one of the supported
compiler names in Spack (gcc, intel, pgi, xl, xl_r, clang, nag, cce, arm).
The version number can be an arbitrary string of alphanumeric characters,
as well as ``-``, ``.``, and ``_``. The ``target`` and ``operating_system``
sections we leave unchanged. These sections specify when Spack can use
different compilers, and are primarily useful for configuration files that
will be used across multiple systems.
We can verify that our new compiler works by invoking it now:
.. code-block:: console
$ spack install --no-cache zlib %clang@3.8.0-gfortran
...
This new compiler also works on Fortran codes:
.. code-block:: console
$ spack install --no-cache cfitsio~bzip2 %clang@3.8.0-gfortran
...
^^^^^^^^^^^^^^
Compiler Flags
^^^^^^^^^^^^^^
Some compilers may require specific compiler flags to work properly in
a particular computing environment. Spack provides configuration
options for setting compiler flags every time a specific compiler is
invoked. These flags become part of the package spec and therefore of
the build provenance. As on the command line, the flags are set
through the implicit build variables ``cflags``, ``cxxflags``, ``cppflags``,
``fflags``, ``ldflags``, and ``ldlibs``.
Let's open our compilers configuration file again and add a compiler flag:
.. code-block:: yaml
- compiler:
environment: {}
extra_rpaths: []
flags:
cppflags: -g
modules: []
operating_system: ubuntu16.04
paths:
cc: /usr/bin/clang
cxx: /usr/bin/clang++
f77: /usr/bin/gfortran
fc: /usr/bin/gfortran
spec: clang@3.8.0-gfortran
target: x86_64
We can test this out using the ``spack spec`` command to show how the
spec is concretized:
.. code-block:: console
$ spack spec cfitsio %clang@3.8.0-gfortran
Input spec
--------------------------------
cfitsio%clang@3.8.0-gfortran
Normalized
--------------------------------
cfitsio%clang@3.8.0-gfortran
Concretized
--------------------------------
cfitsio@3.410%clang@3.8.0-gfortran cppflags="-g" +bzip2+shared arch=linux-ubuntu16.04-x86_64
^bzip2@1.0.6%clang@3.8.0-gfortran cppflags="-g" +shared arch=linux-ubuntu16.04-x86_64
We can see that ``cppflags="-g"`` has been added to every node in the DAG.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Advanced Compiler Configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
There are three fields of the compiler configuration entry that we
have not yet talked about.
The ``modules`` field of the compiler is used primarily on Cray systems,
but can be useful on any system that has compilers that are only
useful when a particular module is loaded. Any modules in the
``modules`` field of the compiler configuration will be loaded as part
of the build environment for packages using that compiler.
The ``extra_rpaths`` field of the compiler configuration is used for
compilers that do not rpath all of their dependencies by
default. Since compilers are often installed externally to Spack,
Spack is unable to manage compiler dependencies and enforce
rpath usage. This can lead to packages not finding link dependencies
imposed by the compiler properly. For compilers that impose link
dependencies on the resulting executables that are not rpath'ed into
the executable automatically, the ``extra_rpaths`` field of the compiler
configuration tells Spack which dependencies to rpath into every
executable created by that compiler. The executables will then be able
to find the link dependencies imposed by the compiler. As an example,
this field can be set by:
.. code-block:: yaml
- compiler:
...
extra_rpaths:
- /apps/intel/ComposerXE2017/compilers_and_libraries_2017.5.239/linux/compiler/lib/intel64_lin
...
The ``environment`` field of the compiler configuration is used for
compilers that require environment variables to be set during build
time. For example, if your Intel compiler suite requires the
``INTEL_LICENSE_FILE`` environment variable to point to the proper
license server, you can set this in ``compilers.yaml`` as follows:
.. code-block:: yaml
- compiler:
environment:
set:
INTEL_LICENSE_FILE: 1713@license4
...
In addition to ``set``, ``environment`` also supports ``unset``,
``prepend-path``, and ``append-path``.
.. _configs-tutorial-package-prefs:
-------------------------------
Configuring Package Preferences
-------------------------------
Package preferences in Spack are managed through the ``packages.yaml``
configuration file. First, we will look at the default
``packages.yaml`` file.
.. code-block:: console
$ spack config --scope defaults edit packages
.. literalinclude:: _spack_root/etc/spack/defaults/packages.yaml
:language: yaml
This sets the default preferences for compilers and for providers of
virtual packages. To illustrate how this works, suppose we want to
change the preferences to prefer the Clang compiler and to prefer
MPICH over OpenMPI. Currently, we prefer GCC and OpenMPI.
.. code-block:: console
$ spack spec hdf5
Input spec
--------------------------------
hdf5
Concretized
--------------------------------
hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
^openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64
^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64
^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64
^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64
^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
Now we will open the packages configuration file and update our
preferences.
.. code-block:: console
$ spack config edit packages
.. code-block:: yaml
packages:
all:
compiler: [clang, gcc, intel, pgi, xl, nag]
providers:
mpi: [mpich, openmpi]
Because of the configuration scoping we discussed earlier, this
overrides the default settings just for these two items.
.. code-block:: console
$ spack spec hdf5
Input spec
--------------------------------
hdf5
Concretized
--------------------------------
hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
^mpich@3.2.1%clang@3.8.0-2ubuntu4 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64
^findutils@4.6.0%clang@3.8.0-2ubuntu4 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64
^autoconf@2.69%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^m4@1.4.18%clang@3.8.0-2ubuntu4 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
^libsigsegv@2.11%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^perl@5.26.2%clang@3.8.0-2ubuntu4+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
^gdbm@1.14.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^readline@7.0%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^ncurses@6.1%clang@3.8.0-2ubuntu4~symlinks~termlib arch=linux-ubuntu16.04-x86_64
^pkgconf@1.4.2%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^automake@1.16.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^libtool@2.4.6%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^texinfo@6.5%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^zlib@1.2.11%clang@3.8.0-2ubuntu4+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
^^^^^^^^^^^^^^^^^^^
Variant Preferences
^^^^^^^^^^^^^^^^^^^
The packages configuration file can also set variant preferences for
package variants. For example, let's change our preferences to build all
packages without shared libraries. We will accomplish this by turning
off the ``shared`` variant on all packages that have one.
.. code-block:: yaml
packages:
all:
compiler: [clang, gcc, intel, pgi, xl, nag]
providers:
mpi: [mpich, openmpi]
variants: ~shared
We can check the effect of this command with ``spack spec hdf5`` again.
.. code-block:: console
$ spack spec hdf5
Input spec
--------------------------------
hdf5
Concretized
--------------------------------
hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl+mpi+pic~shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
^mpich@3.2.1%clang@3.8.0-2ubuntu4 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64
^findutils@4.6.0%clang@3.8.0-2ubuntu4 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64
^autoconf@2.69%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^m4@1.4.18%clang@3.8.0-2ubuntu4 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
^libsigsegv@2.11%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^perl@5.26.2%clang@3.8.0-2ubuntu4+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac ~shared+threads arch=linux-ubuntu16.04-x86_64
^gdbm@1.14.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^readline@7.0%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^ncurses@6.1%clang@3.8.0-2ubuntu4~symlinks~termlib arch=linux-ubuntu16.04-x86_64
^pkgconf@1.4.2%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^automake@1.16.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^libtool@2.4.6%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^texinfo@6.5%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^zlib@1.2.11%clang@3.8.0-2ubuntu4+optimize+pic~shared arch=linux-ubuntu16.04-x86_64
So far we have only made global changes to the package preferences. As
we've seen throughout this tutorial, HDF5 builds with MPI enabled by
default in Spack. If we were working on a project that would routinely
need serial HDF5, that might get annoying quickly, having to type
``hdf5~mpi`` all the time. Instead, we'll update our preferences for
HDF5.
.. code-block:: yaml
packages:
all:
compiler: [clang, gcc, intel, pgi, xl, nag]
providers:
mpi: [mpich, openmpi]
variants: ~shared
hdf5:
variants: ~mpi
Now hdf5 will concretize without an MPI dependency by default.
.. code-block:: console
$ spack spec hdf5
Input spec
--------------------------------
hdf5
Concretized
--------------------------------
hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl~mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
^zlib@1.2.11%clang@3.8.0-2ubuntu4+optimize+pic~shared arch=linux-ubuntu16.04-x86_64
In general, every attribute that we can set for all packages we can
set separately for an individual package.
^^^^^^^^^^^^^^^^^
External Packages
^^^^^^^^^^^^^^^^^
The packages configuration file also controls when Spack will build
against an externally installed package. On these systems we have a
pre-installed zlib.
.. code-block:: yaml
packages:
all:
compiler: [clang, gcc, intel, pgi, xl, nag]
providers:
mpi: [mpich, openmpi]
variants: ~shared
hdf5:
variants: ~mpi
zlib:
paths:
zlib@1.2.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64: /usr
Here, we've told Spack that zlib 1.2.8 is installed on our system.
We've also told it the installation prefix where zlib can be found.
We don't know exactly which variants it was built with, but that's
okay.
.. code-block:: console
$ spack spec hdf5
Input spec
--------------------------------
hdf5
Concretized
--------------------------------
hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran~hl~mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
^zlib@1.2.8%gcc@5.4.0+optimize+pic~shared arch=linux-ubuntu16.04-x86_64
You'll notice that Spack is now using the external zlib installation,
but the compiler used to build zlib is now overriding our compiler
preference of clang. If we explicitly specify Clang:
.. code-block:: console
$ spack spec hdf5 %clang
Input spec
--------------------------------
hdf5%clang
Concretized
--------------------------------
hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl~mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
^zlib@1.2.11%clang@3.8.0-2ubuntu4+optimize+pic~shared arch=linux-ubuntu16.04-x86_64
Spack concretizes to both HDF5 and zlib being built with Clang.
This has a side-effect of rebuilding zlib. If we want to force
Spack to use the system zlib, we have two choices. We can either
specify it on the command line, or we can tell Spack that it's
not allowed to build its own zlib. We'll go with the latter.
.. code-block:: yaml
packages:
all:
compiler: [clang, gcc, intel, pgi, xl, nag]
providers:
mpi: [mpich, openmpi]
variants: ~shared
hdf5:
variants: ~mpi
zlib:
paths:
zlib@1.2.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64: /usr
buildable: False
Now Spack will be forced to choose the external zlib.
.. code-block:: console
$ spack spec hdf5 %clang
Input spec
--------------------------------
hdf5%clang
Concretized
--------------------------------
hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl~mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
^zlib@1.2.8%gcc@5.4.0+optimize+pic~shared arch=linux-ubuntu16.04-x86_64
This gets slightly more complicated with virtual dependencies. Suppose
we don't want to build our own MPI, but we now want a parallel version
of HDF5? Well, fortunately we have MPICH installed on these systems.
.. code-block:: yaml
packages:
all:
compiler: [clang, gcc, intel, pgi, xl, nag]
providers:
mpi: [mpich, openmpi]
variants: ~shared
hdf5:
variants: ~mpi
zlib:
paths:
zlib@1.2.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64: /usr
buildable: False
mpich:
paths:
mpich@3.2%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64: /usr
buildable: False
If we concretize ``hdf5+mpi`` with this configuration file, we will just
build with an alternate MPI implementation.
.. code-block:: console
$ spack spec hdf5+mpi %clang
Input spec
--------------------------------
hdf5%clang+mpi
Concretized
--------------------------------
hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
^openmpi@3.1.3%clang@3.8.0-2ubuntu4~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64
^hwloc@1.11.9%clang@3.8.0-2ubuntu4~cairo~cuda+libxml2+pci~shared arch=linux-ubuntu16.04-x86_64
^libpciaccess@0.13.5%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^libtool@2.4.6%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^m4@1.4.18%clang@3.8.0-2ubuntu4 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
^libsigsegv@2.11%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^pkgconf@1.4.2%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^util-macros@1.19.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^libxml2@2.9.8%clang@3.8.0-2ubuntu4~python arch=linux-ubuntu16.04-x86_64
^xz@5.2.4%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^zlib@1.2.8%gcc@5.4.0+optimize+pic~shared arch=linux-ubuntu16.04-x86_64
^numactl@2.0.11%clang@3.8.0-2ubuntu4 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64
^autoconf@2.69%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^perl@5.26.2%clang@3.8.0-2ubuntu4+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac ~shared+threads arch=linux-ubuntu16.04-x86_64
^gdbm@1.14.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^readline@7.0%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
^ncurses@6.1%clang@3.8.0-2ubuntu4~symlinks~termlib arch=linux-ubuntu16.04-x86_64
^automake@1.16.1%clang@3.8.0-2ubuntu4 arch=linux-ubuntu16.04-x86_64
We have only expressed a preference for MPICH over other MPI
implementations, and Spack will happily build with one we haven't
forbid it from building. We could resolve this by requesting
``hdf5+mpi%clang^mpich`` explicitly, or we can configure Spack not to
use any other MPI implementation. Since we're focused on
configurations here and the former can get tedious, we'll need to
modify our ``packages.yaml`` file again.
While we're at it, we can configure HDF5 to build with MPI by default
again.
.. code-block:: yaml
packages:
all:
compiler: [clang, gcc, intel, pgi, xl, nag]
providers:
mpi: [mpich, openmpi]
variants: ~shared
zlib:
paths:
zlib@1.2.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64: /usr
buildable: False
mpich:
paths:
mpich@3.2%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64: /usr
buildable: False
openmpi:
buildable: False
mvapich2:
buildable: False
intel-mpi:
buildable: False
intel-parallel-studio:
buildable: False
spectrum-mpi:
buildable: False
mpilander:
buildable: False
charm:
buildable: False
charmpp:
buildable: False
Now that we have configured Spack not to build any of the possible
providers for MPI, we can try again.
.. code-block:: console
$ spack spec hdf5 %clang
Input spec
--------------------------------
hdf5%clang
Concretized
--------------------------------
hdf5@1.10.4%clang@3.8.0-2ubuntu4~cxx~debug~fortran~hl+mpi+pic~shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
^mpich@3.2%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64
^zlib@1.2.8%gcc@5.4.0+optimize+pic~shared arch=linux-ubuntu16.04-x86_64
By configuring most of our package preferences in ``packages.yaml``,
we can cut down on the amount of work we need to do when specifying
a spec on the command line. In addition to compiler and variant
preferences, we can specify version preferences as well. Except for
selecting providers via `^`, anything that you can specify on the
command line can be specified in ``packages.yaml`` with the exact
same spec syntax.
^^^^^^^^^^^^^^^^^^^^^^^^
Installation Permissions
^^^^^^^^^^^^^^^^^^^^^^^^
The ``packages.yaml`` file also controls the default permissions
to use when installing a package. You'll notice that by default,
the installation prefix will be world readable but only user writable.
Let's say we need to install ``converge``, a licensed software package.
Since a specific research group, ``fluid_dynamics``, pays for this
license, we want to ensure that only members of this group can access
the software. We can do this like so:
.. code-block:: yaml
packages:
converge:
permissions:
read: group
group: fluid_dynamics
Now, only members of the ``fluid_dynamics`` group can use any
``converge`` installations.
.. warning::
Make sure to delete or move the ``packages.yaml`` you have been
editing up to this point. Otherwise, it will change the hashes
of your packages, leading to differences in the output of later
tutorial sections.
-----------------
High-level Config
-----------------
In addition to compiler and package settings, Spack allows customization
of several high-level settings. These settings are stored in the generic
``config.yaml`` configuration file. You can see the default settings by
running:
.. code-block:: console
$ spack config --scope defaults edit config
.. literalinclude:: _spack_root/etc/spack/defaults/config.yaml
:language: yaml
As you can see, many of the directories Spack uses can be customized.
For example, you can tell Spack to install packages to a prefix
outside of the ``$SPACK_ROOT`` hierarchy. Module files can be
written to a central location if you are using multiple Spack
instances. If you have a fast scratch file system, you can run builds
from this file system with the following ``config.yaml``:
.. code-block:: yaml
config:
build_stage:
- /scratch/$user/spack-stage
.. note::
It is important to distinguish the build stage directory from other
directories in your scratch space to ensure ``spack clean`` does not
inadvertently remove unrelated files. Spack prepends ``spack-stage-``
to temporary staging directory names to reduce this risk. Using a
combination of ``spack`` and or ``stage`` in each specified path, as
shown in the default settings and documented examples, will add
another layer of protection. See :ref:`config-yaml` for details.
On systems with compilers that absolutely *require* environment variables
like ``LD_LIBRARY_PATH``, it is possible to prevent Spack from cleaning
the build environment with the ``dirty`` setting:
.. code-block:: yaml
config:
dirty: true
However, this is strongly discouraged, as it can pull unwanted libraries
into the build.
One last setting that may be of interest to many users is the ability
to customize the parallelism of Spack builds. By default, Spack
installs all packages in parallel with the number of jobs equal to the
number of cores on the node (up to a maximum of 16). For example, on a
node with 16 cores, this will look like:
.. code-block:: console
$ spack install --no-cache --verbose --overwrite zlib
==> Installing zlib
==> Using cached archive: /home/user/spack/var/spack/cache/zlib/zlib-1.2.11.tar.gz
==> Staging archive: /home/user/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb/zlib-1.2.11.tar.gz
==> Created stage in /home/user/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb
==> No patches needed for zlib
==> Building zlib [Package]
==> Executing phase: 'install'
==> './configure' '--prefix=/home/user/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb'
...
==> 'make' '-j16'
...
==> 'make' '-j16' 'install'
...
==> Successfully installed zlib
Fetch: 0.00s. Build: 1.03s. Total: 1.03s.
[+] /home/user/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb
As you can see, we are building with all 16 cores on the node. If you are
on a shared login node, this can slow down the system for other users. If
you have a strict ulimit or restriction on the number of available licenses,
you may not be able to build at all with this many cores. On nodes with 64+
cores, you may not see a significant speedup of the build anyway. To limit
the number of cores our build uses, set ``build_jobs`` like so:
.. code-block:: yaml
config:
build_jobs: 2
If we uninstall and reinstall zlib, we see that it now uses only 2 cores:
.. code-block:: console
$ spack install --no-cache --verbose --overwrite zlib
==> Installing zlib
==> Using cached archive: /home/user/spack/var/spack/cache/zlib/zlib-1.2.11.tar.gz
==> Staging archive: /home/user/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb/zlib-1.2.11.tar.gz
==> Created stage in /home/user/spack/var/spack/stage/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb
==> No patches needed for zlib
==> Building zlib [Package]
==> Executing phase: 'install'
==> './configure' '--prefix=/home/user/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb'
...
==> 'make' '-j2'
...
==> 'make' '-j2' 'install'
...
==> Successfully installed zlib
Fetch: 0.00s. Build: 1.03s. Total: 1.03s.
[+] /home/user/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/zlib-1.2.11-5nus6knzumx4ik2yl44jxtgtsl7d54xb
Obviously, if you want to build everything in serial for whatever reason,
you would set ``build_jobs`` to 1.
--------
Examples
--------
For examples of how other sites configure Spack, see
https://github.com/spack/spack-configs. If you use Spack at your site
and want to share your config files, feel free to submit a pull request!

815
lib/spack/docs/tutorial_environments.rst
View file

@ -1,815 +0,0 @@
.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _environments-tutorial:
=====================
Environments Tutorial
=====================
We've shown you how to install and remove packages with Spack. You can
use :ref:`cmd-spack-install` to install packages,
:ref:`cmd-spack-uninstall` to remove them, and :ref:`cmd-spack-find` to
look at and query what is installed. We've also shown you how to
customize Spack's installation with configuration files like
:ref:`packages.yaml <build-settings>`.
If you build a lot of software, or if you work on multiple projects,
managing everything in one place can be overwhelming. The default ``spack
find`` output may contain many packages, but you may want to *just* focus
on packages for a particular project. Moreover, you may want to include
special configuration with your package groups, e.g., to build all the
packages in the same group the same way.
Spack **environments** provide a way to handle these problems.
-------------------
Environment basics
-------------------
Let's look at the output of ``spack find`` at this point in the tutorial.
.. code-block:: console
$ spack find
==> 70 installed packages
-- linux-ubuntu16.04-x86_64 / clang@3.8.0-2ubuntu4 --------------
tcl@8.6.8 zlib@1.2.8 zlib@1.2.11
-- linux-ubuntu16.04-x86_64 / gcc@4.7 ---------------------------
zlib@1.2.11
-- linux-ubuntu16.04-x86_64 / gcc@5.4.0 -------------------------
adept-utils@1.0.1 hdf5@1.10.4 mpc@1.1.0 perl@5.26.2
autoconf@2.69 hdf5@1.10.4 mpfr@3.1.6 pkgconf@1.4.2
automake@1.16.1 hdf5@1.10.4 mpich@3.2.1 readline@7.0
boost@1.68.0 hwloc@1.11.9 mpileaks@1.0 suite-sparse@5.3.0
bzip2@1.0.6 hypre@2.15.1 mumps@5.1.1 tar@1.30
callpath@1.0.4 hypre@2.15.1 mumps@5.1.1 tcl@8.6.8
cmake@3.12.3 isl@0.18 ncurses@6.1 tcl@8.6.8
diffutils@3.6 libdwarf@20180129 netcdf@4.6.1 texinfo@6.5
dyninst@9.3.2 libiberty@2.31.1 netcdf@4.6.1 trilinos@12.12.1
elfutils@0.173 libpciaccess@0.13.5 netlib-scalapack@2.0.2 trilinos@12.12.1
findutils@4.6.0 libsigsegv@2.11 netlib-scalapack@2.0.2 util-macros@1.19.1
gcc@7.2.0 libtool@2.4.6 numactl@2.0.11 xz@5.2.4
gdbm@1.14.1 libxml2@2.9.8 openblas@0.3.3 zlib@1.2.8
gettext@0.19.8.1 m4@1.4.18 openmpi@3.1.3 zlib@1.2.8
glm@0.9.7.1 matio@1.5.9 openssl@1.0.2o zlib@1.2.11
gmp@6.1.2 matio@1.5.9 parmetis@4.0.3
hdf5@1.10.4 metis@5.1.0 parmetis@4.0.3
This is a complete, but cluttered view. There are packages built with
both ``openmpi`` and ``mpich``, as well as multiple variants of other
packages, like ``zlib``. The query mechanism we learned about in ``spack
find`` can help, but it would be nice if we could start from a clean
slate without losing what we've already done.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Creating and activating environments
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The ``spack env`` command can help. Let's create a new environment:
.. code-block:: console
$ spack env create myproject
==> Created environment 'myproject' in ~/spack/var/spack/environments/myproject
An environment is a virtualized ``spack`` instance that you can use for a
specific purpose. You can see the environments we've created so far like this:
.. code-block:: console
$ spack env list
==> 1 environments
myproject
And you can **activate** an environment with ``spack env activate``:
.. code-block:: console
$ spack env activate myproject
Once you enter an environment, ``spack find`` shows only what is in the
current environment. That's nothing, so far:
.. code-block:: console
$ spack find
==> In environment myproject
==> No root specs
==> 0 installed packages
The ``spack find`` output is still *slightly* different. It tells you
that you're in the ``myproject`` environment, so that you don't panic
when you see that there is nothing installed. It also says that there
are *no root specs*. We'll get back to what that means later.
If you *only* want to check what environment you are in, you can use
``spack env status``:
.. code-block:: console
$ spack env status
==> In environment myproject
And, if you want to leave this environment and go back to normal Spack,
you can use ``spack env deactivate``. We like to use the
``despacktivate`` alias (which Spack sets up automatically) for short:
.. code-block:: console
$ despacktivate # short alias for `spack env deactivate`
$ spack env status
==> No active environment
$ spack find
netcdf@4.6.1 readline@7.0 zlib@1.2.11
diffutils@3.6 hdf5@1.10.4 m4@1.4.18 netcdf@4.6.1 suite-sparse@5.3.0
dyninst@10.0.0 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 tar@1.30
elfutils@0.173 hypre@2.15.1 matio@1.5.9 netlib-scalapack@2.0.2 tcl@8.6.8
findutils@4.6.0 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 tcl@8.6.8
gcc@7.2.0 intel-tbb@2019 mpc@1.1.0 openblas@0.3.3 texinfo@6.5~
^^^^^^^^^^^^^^^^^^^
Installing packages
^^^^^^^^^^^^^^^^^^^
Ok, now that we understand how creation and activation work, let's go
back to ``myproject`` and *install* a few packages:
.. code-block:: console
$ spack env activate myproject
$ spack install tcl
==> tcl is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8-qhwyccywhx2i6s7ob2gvjrjtj3rnfuqt
$ spack install trilinos
==> trilinos is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-rlsruavxqvwk2tgxzxboclbo6ykjf54r
$ spack find
==> In environment myproject
==> Root specs
tcl trilinos
==> 22 installed packages
-- linux-ubuntu16.04-x86_64 / gcc@5.4.0 -------------------------
boost@1.68.0 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 xz@5.2.4
bzip2@1.0.6 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0 zlib@1.2.11
glm@0.9.7.1 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 tcl@8.6.8
hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 trilinos@12.12.1
We've installed ``tcl`` and ``trilinos`` in our environment, along with
all of their dependencies. We call ``tcl`` and ``trilinos`` the
**roots** because we asked for them explicitly. The other 20 packages
listed under "installed packages" are present because they were needed as
dependencies. So, these are the roots of the packages' dependency graph.
The "<package> is already installed" messages above are generated because
we already installed these packages in previous steps of the tutorial,
and we don't have to rebuild them to put them in an environment.
Now let's create *another* project. We'll call this one ``myproject2``:
.. code-block:: console
$ spack env create myproject2
==> Created environment 'myproject2' in ~/spack/var/spack/environments/myproject2
$ spack env activate myproject2
$ spack install hdf5
==> hdf5 is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-ozyvmhzdew66byarohm4p36ep7wtcuiw
$ spack install trilinos
==> trilinos is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-rlsruavxqvwk2tgxzxboclbo6ykjf54r
$ spack find
==> In environment myproject2
==> Root specs
hdf5 trilinos
==> 22 installed packages
-- linux-ubuntu16.04-x86_64 / gcc@5.4.0 -------------------------
boost@1.68.0 hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 xz@5.2.4
bzip2@1.0.6 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 zlib@1.2.11
glm@0.9.7.1 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0
hdf5@1.10.4 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 trilinos@12.12.1
Now we have two environments: one with ``tcl`` and ``trilinos``, and
another with ``hdf5`` and ``trilinos``.
We can uninstall trilinos from ``myproject2`` as you would expect:
.. code-block:: console
$ spack uninstall trilinos
==> The following packages will be uninstalled:
-- linux-ubuntu16.04-x86_64 / gcc@5.4.0 -------------------------
rlsruav trilinos@12.12.1%gcc~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2
==> Do you want to proceed? [y/N] y
$ spack find
==> In environment myproject2
==> Root specs
hdf5
==> 8 installed packages
-- linux-ubuntu16.04-x86_64 / gcc@5.4.0 -------------------------
hdf5@1.10.4 libpciaccess@0.13.5 numactl@2.0.11 xz@5.2.4
hwloc@1.11.9 libxml2@2.9.8 openmpi@3.1.3 zlib@1.2.11
Now there is only one root spec, ``hdf5``, which requires fewer
additional dependencies.
However, we still needed ``trilinos`` for the ``myproject`` environment!
What happened to it? Let's switch back and see.
.. code-block:: console
$ despacktivate
$ spack env activate myproject
$ spack find
==> In environment myproject
==> Root specs
tcl trilinos
==> 22 installed packages
-- linux-ubuntu16.04-x86_64 / gcc@5.4.0 -------------------------
boost@1.68.0 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 xz@5.2.4
bzip2@1.0.6 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0 zlib@1.2.11
glm@0.9.7.1 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 tcl@8.6.8
hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 trilinos@12.12.1
Spack is smart enough to realize that ``trilinos`` is still present in
the other environment. Trilinos won't *actually* be uninstalled unless
it is no longer needed by any environments or packages. If it is still
needed, it is only removed from the environment.
-------------------------------
Dealing with many specs at once
-------------------------------
In the above examples, we just used ``install`` and ``uninstall``. There
are other ways to deal with groups of packages, as well.
^^^^^^^^^^^^^
Adding specs
^^^^^^^^^^^^^
Let's go back to our first ``myproject`` environment and *add* a few specs instead of installing them:
.. code-block:: console
$ spack add hdf5
==> Adding hdf5 to environment myproject
$ spack add gmp
==> Adding mumps to environment myproject
$ spack find
==> In environment myproject
==> Root specs
gmp hdf5 tcl trilinos
==> 22 installed packages
-- linux-ubuntu16.04-x86_64 / gcc@5.4.0 -------------------------
boost@1.68.0 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 xz@5.2.4
bzip2@1.0.6 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0 zlib@1.2.11
glm@0.9.7.1 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 tcl@8.6.8
hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 trilinos@12.12.1
Let's take a close look at what happened. The two packages we added,
``hdf5`` and ``gmp``, are present, but they're not installed in the
environment yet. ``spack add`` just adds *roots* to the environment, but
it does not automatically install them.
We can install *all* the as-yet uninstalled packages in an environment by
simply running ``spack install`` with no arguments:
.. code-block:: console
$ spack install
==> Concretizing hdf5
[+] ozyvmhz hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
[+] 3njc4q5 ^openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64
[+] 43tkw5m ^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64
[+] 5urc6tc ^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
[+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] milz7fm ^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] wpexsph ^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64
[+] teneqii ^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
[+] ft463od ^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64
[+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
[+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
[+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
==> Concretizing gmp
[+] qc4qcfz gmp@6.1.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
[+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
[+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
[+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
==> Installing environment myproject
==> tcl is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/tcl-8.6.8-qhwyccywhx2i6s7ob2gvjrjtj3rnfuqt
==> trilinos is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-rlsruavxqvwk2tgxzxboclbo6ykjf54r
==> hdf5 is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/hdf5-1.10.4-ozyvmhzdew66byarohm4p36ep7wtcuiw
==> gmp is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/gmp-6.1.2-qc4qcfz4monpllc3nqupdo7vwinf73sw
Spack will concretize the new roots, and install everything you added to
the environment. Now we can see the installed roots in the output of
``spack find``:
.. code-block:: console
$ spack find
==> In environment myproject
==> Root specs
gmp hdf5 tcl trilinos
==> 24 installed packages
-- linux-ubuntu16.04-x86_64 / gcc@5.4.0 -------------------------
boost@1.68.0 hdf5@1.10.4 libpciaccess@0.13.5 mumps@5.1.1 openblas@0.3.3 tcl@8.6.8
bzip2@1.0.6 hdf5@1.10.4 libxml2@2.9.8 netcdf@4.6.1 openmpi@3.1.3 trilinos@12.12.1
glm@0.9.7.1 hwloc@1.11.9 matio@1.5.9 netlib-scalapack@2.0.2 parmetis@4.0.3 xz@5.2.4
gmp@6.1.2 hypre@2.15.1 metis@5.1.0 numactl@2.0.11 suite-sparse@5.3.0 zlib@1.2.11
We can build whole environments this way, by adding specs and installing
all at once, or we can install them with the usual ``install`` and
``uninstall`` portions. The advantage to doing them all at once is that
we don't have to write a script outside of Spack to automate this, and we
can kick off a large build of many packages easily.
^^^^^^^^^^^^^^^^^^^^^
Configuration
^^^^^^^^^^^^^^^^^^^^^
So far, ``myproject`` does not have any special configuration associated
with it. The specs concretize using Spack's defaults:
.. code-block:: console
$ spack spec hypre
Input spec
--------------------------------
hypre
Concretized
--------------------------------
hypre@2.15.1%gcc@5.4.0~debug~int64+internal-superlu+mpi+shared arch=linux-ubuntu16.04-x86_64
^openblas@0.3.3%gcc@5.4.0 cpu_target= ~ilp64 patches=47cfa7a952ac7b2e4632c73ae199d69fb54490627b66a62c681e21019c4ddc9d,714aea33692304a50bd0ccde42590c176c82ded4a8ac7f06e573dc8071929c33 +pic+shared threads=none ~virtual_machine arch=linux-ubuntu16.04-x86_64
^openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64
^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64
^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64
^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64
^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
You may want to add extra configuration to your environment. You can see
how your environment is configured using ``spack config get``:
.. code-block:: console
$ spack config get
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs: [tcl, trilinos, hdf5, gmp]
It turns out that this is a special configuration format where Spack
stores the state for the environment. Currently, the file is just a
``spack:`` header and a list of ``specs``. These are the roots.
You can edit this file to add your own custom configuration. Spack
provides a shortcut to do that:
.. code-block:: console
spack config edit
You should now see the same file, and edit it to look like this:
.. code-block:: yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
packages:
all:
providers:
mpi: [mpich]
# add package specs to the `specs` list
specs: [tcl, trilinos, hdf5, gmp]
Now if we run ``spack spec`` again in the environment, specs will concretize with ``mpich`` as the MPI implementation:
.. code-block:: console
$ spack spec hypre
Input spec
--------------------------------
hypre
Concretized
--------------------------------
hypre@2.15.1%gcc@5.4.0~debug~int64+internal-superlu+mpi+shared arch=linux-ubuntu16.04-x86_64
^mpich@3.2.1%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64
^findutils@4.6.0%gcc@5.4.0 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64
^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^texinfo@6.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
^openblas@0.3.3%gcc@5.4.0 cpu_target= ~ilp64 patches=47cfa7a952ac7b2e4632c73ae199d69fb54490627b66a62c681e21019c4ddc9d,714aea33692304a50bd0ccde42590c176c82ded4a8ac7f06e573dc8071929c33 +pic+shared threads=none ~virtual_machine arch=linux-ubuntu16.04-x86_64
In addition to the ``specs`` section, an environment's configuration can
contain any of the configuration options from Spack's various config
sections. You can add custom repositories, a custom install location,
custom compilers, or custom external packages, in addition to the ``package``
preferences we show here.
But now we have a problem. We already installed part of this environment
with openmpi, but now we want to install it with ``mpich``.
You can run ``spack concretize`` inside of an environment to concretize
all of its specs. We can run it here:
.. code-block:: console
$ spack concretize -f
==> Concretizing tcl
[+] qhwyccy tcl@8.6.8%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
==> Concretizing trilinos
[+] kqc52mo trilinos@12.12.1%gcc@5.4.0~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 arch=linux-ubuntu16.04-x86_64
[+] zbgfxap ^boost@1.68.0%gcc@5.4.0+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199 +program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=linux-ubuntu16.04-x86_64
[+] ufczdvs ^bzip2@1.0.6%gcc@5.4.0+shared arch=linux-ubuntu16.04-x86_64
[+] 2rhuivg ^diffutils@3.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
[+] otafqzh ^cmake@3.12.3%gcc@5.4.0~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-ubuntu16.04-x86_64
[+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
[+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] b4y3w3b ^openssl@1.0.2o%gcc@5.4.0+systemcerts arch=linux-ubuntu16.04-x86_64
[+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
[+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] jnw622j ^glm@0.9.7.1%gcc@5.4.0 build_type=RelWithDebInfo arch=linux-ubuntu16.04-x86_64
[+] xxd7syh ^hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
[+] p3f7p2r ^mpich@3.2.1%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64
[+] d4iajxs ^findutils@4.6.0%gcc@5.4.0 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64
[+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
[+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] zs7a2pc ^texinfo@6.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] obewuoz ^hypre@2.15.1%gcc@5.4.0~debug~int64~internal-superlu+mpi+shared arch=linux-ubuntu16.04-x86_64
[+] cyeg2yi ^openblas@0.3.3%gcc@5.4.0 cpu_target= ~ilp64 patches=47cfa7a952ac7b2e4632c73ae199d69fb54490627b66a62c681e21019c4ddc9d,714aea33692304a50bd0ccde42590c176c82ded4a8ac7f06e573dc8071929c33 +pic+shared threads=none ~virtual_machine arch=linux-ubuntu16.04-x86_64
[+] gvyqldh ^matio@1.5.9%gcc@5.4.0+hdf5+shared+zlib arch=linux-ubuntu16.04-x86_64
[+] 3wnvp4j ^metis@5.1.0%gcc@5.4.0 build_type=Release ~gdb~int64 patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1 ~real64+shared arch=linux-ubuntu16.04-x86_64
[+] cumcj5a ^mumps@5.1.1%gcc@5.4.0+complex+double+float~int64~metis+mpi~parmetis~ptscotch~scotch+shared arch=linux-ubuntu16.04-x86_64
[+] p7iln2p ^netlib-scalapack@2.0.2%gcc@5.4.0 build_type=RelWithDebInfo ~pic+shared arch=linux-ubuntu16.04-x86_64
[+] wmmx5sg ^netcdf@4.6.1%gcc@5.4.0~dap~hdf4 maxdims=1024 maxvars=8192 +mpi~parallel-netcdf+shared arch=linux-ubuntu16.04-x86_64
[+] jehtata ^parmetis@4.0.3%gcc@5.4.0 build_type=RelWithDebInfo ~gdb patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d +shared arch=linux-ubuntu16.04-x86_64
[+] zaau4ki ^suite-sparse@5.3.0%gcc@5.4.0~cuda~openmp+pic~tbb arch=linux-ubuntu16.04-x86_64
==> Concretizing hdf5
- zjgyn3w hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran~hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
[+] p3f7p2r ^mpich@3.2.1%gcc@5.4.0 device=ch3 +hydra netmod=tcp +pmi+romio~verbs arch=linux-ubuntu16.04-x86_64
[+] d4iajxs ^findutils@4.6.0%gcc@5.4.0 patches=84b916c0bf8c51b7e7b28417692f0ad3e7030d1f3c248ba77c42ede5c1c5d11e,bd9e4e5cc280f9753ae14956c4e4aa17fe7a210f55dd6c84aa60b12d106d47a2 arch=linux-ubuntu16.04-x86_64
[+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
[+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
[+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
[+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] zs7a2pc ^texinfo@6.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
==> Concretizing gmp
[+] qc4qcfz gmp@6.1.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
[+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
[+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
[+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
Now, all the specs in the environment are concrete and ready to be
installed with ``mpich`` as the MPI implementation.
Normally, we could just run ``spack config edit``, edit the environment
configuration, ``spack add`` some specs, and ``spack install``.
But, when we already have installed packages in the environment, we have
to force everything in the environment to be re-concretized using ``spack
concretize -f``. *Then* we can re-run ``spack install``.
-----------------------------------
``spack.yaml`` and ``spack.lock``
-----------------------------------
So far we've shown you how to interact with environments from the command
line, but they also have a file-based interface that can be used by
developers and admins to manage workflows for projects.
In this section we'll dive a little deeper to see how environments are
implemented, and how you could use this in your day-to-day development.
^^^^^^^^^^^^^^
``spack.yaml``
^^^^^^^^^^^^^^
Earlier, we changed an environment's configuration using ``spack config
edit``. We were actually editing a special file called ``spack.yaml``.
Let's take a look.
We can get directly to the current environment's location using ``spack cd``:
.. code-block:: console
$ spack cd -e myproject
$ pwd
~/spack/var/spack/environments/myproject
$ ls
spack.lock spack.yaml
We notice two things here. First, the environment is just a directory
inside of ``var/spack/environments`` within the Spack installation.
Second, it contains two important files: ``spack.yaml`` and
``spack.lock``.
``spack.yaml`` is the configuration file for environments that we've
already seen, but it does not *have* to live inside Spack. If you create
an environment using ``spack env create``, it is *managed* by
Spack in the ``var/spack/environments`` directory, and you can refer to
it by name.
You can actually put a ``spack.yaml`` file *anywhere*, and you can use it
to bundle an environment, or a list of dependencies to install, with your
project. Let's make a simple project:
.. code-block:: console
$ cd
$ mkdir code
$ cd code
$ spack env create -d .
==> Created environment in ~/code
Here, we made a new directory called *code*, and we used the ``-d``
option to create an environment in it.
What really happened?
.. code-block:: console
$ ls
spack.yaml
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs: []
Spack just created a ``spack.yaml`` file in the code directory, with an
empty list of root specs. Now we have a Spack environment, *in a
directory*, that we can use to manage dependencies. Suppose your project
depends on ``boost``, ``trilinos``, and ``openmpi``. You can add these
to your spec list:
.. code-block:: yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs:
- boost
- trilinos
- openmpi
And now *anyone* who uses the *code* repository can use this format to
install the project's dependencies. They need only clone the repository,
``cd`` into it, and type ``spack install``:
.. code-block:: console
$ spack install
==> Concretizing boost
[+] zbgfxap boost@1.68.0%gcc@5.4.0+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199 +program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=linux-ubuntu16.04-x86_64
[+] ufczdvs ^bzip2@1.0.6%gcc@5.4.0+shared arch=linux-ubuntu16.04-x86_64
[+] 2rhuivg ^diffutils@3.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
==> Concretizing trilinos
[+] rlsruav trilinos@12.12.1%gcc@5.4.0~alloptpkgs+amesos+amesos2+anasazi+aztec+belos+boost build_type=RelWithDebInfo ~cgns~complex~dtk+epetra+epetraext+exodus+explicit_template_instantiation~float+fortran~fortrilinos+gtest+hdf5+hypre+ifpack+ifpack2~intrepid~intrepid2~isorropia+kokkos+metis~minitensor+ml+muelu+mumps~nox~openmp~phalanx~piro~pnetcdf~python~rol~rythmos+sacado~shards+shared~stk+suite-sparse~superlu~superlu-dist~teko~tempus+teuchos+tpetra~x11~xsdkflags~zlib+zoltan+zoltan2 arch=linux-ubuntu16.04-x86_64
[+] zbgfxap ^boost@1.68.0%gcc@5.4.0+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=2ab6c72d03dec6a4ae20220a9dfd5c8c572c5294252155b85c6874d97c323199 +program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=linux-ubuntu16.04-x86_64
[+] ufczdvs ^bzip2@1.0.6%gcc@5.4.0+shared arch=linux-ubuntu16.04-x86_64
[+] 2rhuivg ^diffutils@3.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
[+] otafqzh ^cmake@3.12.3%gcc@5.4.0~doc+ncurses+openssl+ownlibs patches=dd3a40d4d92f6b2158b87d6fb354c277947c776424aa03f6dc8096cf3135f5d0 ~qt arch=linux-ubuntu16.04-x86_64
[+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
[+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] b4y3w3b ^openssl@1.0.2o%gcc@5.4.0+systemcerts arch=linux-ubuntu16.04-x86_64
[+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
[+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] jnw622j ^glm@0.9.7.1%gcc@5.4.0 build_type=RelWithDebInfo arch=linux-ubuntu16.04-x86_64
[+] oqwnui7 ^hdf5@1.10.4%gcc@5.4.0~cxx~debug~fortran+hl+mpi+pic+shared~szip~threadsafe arch=linux-ubuntu16.04-x86_64
[+] 3njc4q5 ^openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64
[+] 43tkw5m ^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64
[+] 5urc6tc ^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
[+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] milz7fm ^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] wpexsph ^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64
[+] teneqii ^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] ft463od ^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64
[+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] fshksdp ^hypre@2.15.1%gcc@5.4.0~debug~int64~internal-superlu+mpi+shared arch=linux-ubuntu16.04-x86_64
[+] cyeg2yi ^openblas@0.3.3%gcc@5.4.0 cpu_target= ~ilp64 patches=47cfa7a952ac7b2e4632c73ae199d69fb54490627b66a62c681e21019c4ddc9d,714aea33692304a50bd0ccde42590c176c82ded4a8ac7f06e573dc8071929c33 +pic+shared threads=none ~virtual_machine arch=linux-ubuntu16.04-x86_64
[+] lmzdgss ^matio@1.5.9%gcc@5.4.0+hdf5+shared+zlib arch=linux-ubuntu16.04-x86_64
[+] 3wnvp4j ^metis@5.1.0%gcc@5.4.0 build_type=Release ~gdb~int64 patches=4991da938c1d3a1d3dea78e49bbebecba00273f98df2a656e38b83d55b281da1 ~real64+shared arch=linux-ubuntu16.04-x86_64
[+] acsg2dz ^mumps@5.1.1%gcc@5.4.0+complex+double+float~int64~metis+mpi~parmetis~ptscotch~scotch+shared arch=linux-ubuntu16.04-x86_64
[+] wotpfwf ^netlib-scalapack@2.0.2%gcc@5.4.0 build_type=RelWithDebInfo ~pic+shared arch=linux-ubuntu16.04-x86_64
[+] mhm4izp ^netcdf@4.6.1%gcc@5.4.0~dap~hdf4 maxdims=1024 maxvars=8192 +mpi~parallel-netcdf+shared arch=linux-ubuntu16.04-x86_64
[+] uv6h3sq ^parmetis@4.0.3%gcc@5.4.0 build_type=RelWithDebInfo ~gdb patches=4f892531eb0a807eb1b82e683a416d3e35154a455274cf9b162fb02054d11a5b,50ed2081bc939269689789942067c58b3e522c269269a430d5d34c00edbc5870,704b84f7c7444d4372cb59cca6e1209df4ef3b033bc4ee3cf50f369bce972a9d +shared arch=linux-ubuntu16.04-x86_64
[+] zaau4ki ^suite-sparse@5.3.0%gcc@5.4.0~cuda~openmp+pic~tbb arch=linux-ubuntu16.04-x86_64
==> Concretizing openmpi
[+] 3njc4q5 openmpi@3.1.3%gcc@5.4.0~cuda+cxx_exceptions fabrics= ~java~legacylaunchers~memchecker~pmi schedulers= ~sqlite3~thread_multiple+vt arch=linux-ubuntu16.04-x86_64
[+] 43tkw5m ^hwloc@1.11.9%gcc@5.4.0~cairo~cuda+libxml2+pci+shared arch=linux-ubuntu16.04-x86_64
[+] 5urc6tc ^libpciaccess@0.13.5%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] o2pfwjf ^libtool@2.4.6%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] suf5jtc ^m4@1.4.18%gcc@5.4.0 patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,c0a408fbffb7255fcc75e26bd8edab116fc81d216bfd18b473668b7739a4158e,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-ubuntu16.04-x86_64
[+] fypapcp ^libsigsegv@2.11%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] fovrh7a ^pkgconf@1.4.2%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] milz7fm ^util-macros@1.19.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] wpexsph ^libxml2@2.9.8%gcc@5.4.0~python arch=linux-ubuntu16.04-x86_64
[+] teneqii ^xz@5.2.4%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 5nus6kn ^zlib@1.2.11%gcc@5.4.0+optimize+pic+shared arch=linux-ubuntu16.04-x86_64
[+] ft463od ^numactl@2.0.11%gcc@5.4.0 patches=592f30f7f5f757dfc239ad0ffd39a9a048487ad803c26b419e0f96b8cda08c1a arch=linux-ubuntu16.04-x86_64
[+] 3sx2gxe ^autoconf@2.69%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] ic2kyoa ^perl@5.26.2%gcc@5.4.0+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=linux-ubuntu16.04-x86_64
[+] q4fpyuo ^gdbm@1.14.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] nxhwrg7 ^readline@7.0%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
[+] 3o765ou ^ncurses@6.1%gcc@5.4.0~symlinks~termlib arch=linux-ubuntu16.04-x86_64
[+] rymw7im ^automake@1.16.1%gcc@5.4.0 arch=linux-ubuntu16.04-x86_64
==> Installing environment ~/code
==> boost is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/boost-1.68.0-zbgfxapchxa4awxdwpleubfuznblxzvt
==> trilinos is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/trilinos-12.12.1-rlsruavxqvwk2tgxzxboclbo6ykjf54r
==> openmpi is already installed in ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx
Spack concretizes the specs in the ``spack.yaml`` file and installs them.
What happened here? If you ``cd`` into a directory that has a
``spack.yaml`` file in it, Spack considers this directory's environment
to be activated. The directory does not have to live within Spack; it
can be anywhere.
So, from ``~/code``, we can actually manipulate ``spack.yaml`` using
``spack add`` and ``spack remove`` (just like managed environments):
.. code-block:: console
$ spack add hdf5@5.5.1
==> Adding hdf5 to environment ~/code
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs:
- boost
- trilinos
- openmpi
- hdf5@5.5.1
$ spack remove hdf5
==> Removing hdf5 from environment ~/code
$ cat spack.yaml
# This is a Spack Environment file.
#
# It describes a set of packages to be installed, along with
# configuration settings.
spack:
# add package specs to the `specs` list
specs:
- boost
- trilinos
- openmpi
^^^^^^^^^^^^^^
``spack.lock``
^^^^^^^^^^^^^^
Okay, we've covered managed environments, environments in directories, and
the last thing we'll cover is ``spack.lock``. You may remember that when
we ran ``spack install``, Spack concretized all the specs in the
``spack.yaml`` file and installed them.
Whenever we concretize Specs in an environment, all concrete specs in the
environment are written out to a ``spack.lock`` file *alongside*
``spack.yaml``. The ``spack.lock`` file is not really human-readable
like the ``spack.yaml`` file. It is a ``json`` format that contains all
the information that we need to *reproduce* the build of an
environment:
.. code-block:: console
$ head spack.lock
{
"concrete_specs": {
"teneqii2xv5u6zl5r6qi3pwurc6pmypz": {
"xz": {
"version": "5.2.4",
"arch": {
"platform": "linux",
"platform_os": "ubuntu16.04",
"target": "x86_64"
},
...
``spack.yaml`` and ``spack.lock`` correspond to two fundamental concepts
in Spack, but for environments:
* ``spack.yaml`` is the set of *abstract* specs and configuration that
you want to install.
* ``spack.lock`` is the set of all fully *concretized* specs generated
from concretizing ``spack.yaml``
Using either of these, you can recreate an environment that someone else
built. ``spack env create`` takes an extra optional argument, which can
be either a ``spack.yaml`` or a ``spack.lock`` file:
.. code-block:: console
$ spack env create my-project spack.yaml
$ spack env create my-project spack.lock
Both of these create a new environment called ``my-project``, but which
one you choose to use depends on your needs:
#. copying the yaml file allows someone else to build your *requirements*,
potentially a different way.
#. copying the lock file allows someone else to rebuild your
*installation* exactly as you built it.
The first use case can *re-concretize* the same specs on new platforms in
order to build, but it will preserve the abstract requirements. The
second use case (currently) requires you to be on the same machine, but
it retains all decisions made during concretization and is faithful to a
prior install.

1575
lib/spack/docs/tutorial_modules.rst
View file

File diff suppressed because it is too large Load diff

559
lib/spack/docs/tutorial_packaging.rst
View file

@ -1,559 +0,0 @@
.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _packaging-tutorial:
=========================
Package Creation Tutorial
=========================
This tutorial will walk you through the steps behind building a simple
package installation script. We'll focus on writing a package for
mpileaks, an MPI debugging tool. By creating a package file we're
essentially giving Spack a recipe for how to build a particular piece of
software. We're describing some of the software's dependencies, where to
find the package, what commands and options are used to build the package
from source, and more. Once we've specified a package's recipe, we can
ask Spack to build that package in many different ways.
This tutorial assumes you have a basic familiarity with some of the Spack
commands, and that you have a working version of Spack installed. If
not, we suggest looking at Spack's :ref:`getting_started` guide. This
tutorial also assumes you have at least a beginner's-level familiarity
with Python.
Also note that this document is a tutorial. It can help you get started
with packaging, but is not intended to be complete. See Spack's
:ref:`packaging-guide` for more complete documentation on this topic.
---------------
Getting Started
---------------
A few things before we get started:
- We'll refer to the Spack installation location via the environment
variable ``SPACK_ROOT``. You should point ``SPACK_ROOT`` at wherever
you have Spack installed.
- Add ``$SPACK_ROOT/bin`` to your ``PATH`` before you start.
- Make sure your ``EDITOR`` environment variable is set to your
preferred text editor.
- We'll be writing Python code as part of this tutorial. You can find
successive versions of the Python code in
``$SPACK_ROOT/lib/spack/docs/tutorial/examples``.
-------------------------
Creating the Package File
-------------------------
We will use a separate package repository for the tutorial. Package
repositories allow you to separate sets of packages that take
precedence over one another. We will use the tutorial repo that ships
with Spack to avoid breaking the builtin Spack packages.
.. code-block:: console
$ spack repo add $SPACK_ROOT/var/spack/repos/tutorial/
==> Added repo with namespace 'tutorial'.
Spack comes with a handy command to create a new package: ``spack create``.
This command is given the location of a package's source code, downloads
the code, and sets up some basic packaging infrastructure for you. The
mpileaks source code can be found on GitHub, and here's what happens when
we run ``spack create`` on it:
.. code-block:: console
$ spack create -t generic https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz
==> This looks like a URL for mpileaks
==> Found 1 version of mpileaks:
1.0 https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz
==> How many would you like to checksum? (default is 1, q to abort) 1
==> Downloading...
==> Fetching https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz
############################################################################# 100.0%
==> Checksummed 1 version of mpileaks
==> Using specified package template: 'generic'
==> Created template for mpileaks package
==> Created package file: ~/spack/var/spack/repos/tutorial/packages/mpileaks/package.py
Spack should spawn a text editor with this file:
.. literalinclude:: tutorial/examples/0.package.py
:language: python
Spack has created this file in
``$SPACK_ROOT/var/spack/repos/tutorial/packages/mpileaks/package.py``. Take a
moment to look over the file. There's a few placeholders that Spack has
created, which we'll fill in as part of this tutorial:
- We'll document some information about this package in the comments.
- We'll fill in the dependency list for this package.
- We'll fill in some of the configuration arguments needed to build this
package.
For the moment, exit your editor and let's see what happens when we try
to build this package:
.. code-block:: console
$ spack install mpileaks
==> Installing mpileaks
==> Searching for binary cache of mpileaks
==> Warning: No Spack mirrors are currently configured
==> No binary for mpileaks found: installing from source
==> Fetching https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz
############################################################################# 100.0%
==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-sv75n3u5ev6mljwcezisz3slooozbbxu/mpileaks-1.0.tar.gz
==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-sv75n3u5ev6mljwcezisz3slooozbbxu
==> No patches needed for mpileaks
==> Building mpileaks [Package]
==> Executing phase: 'install'
==> Error: ProcessError: Command exited with status 2:
'make' '-j16'
1 error found in build log:
1 ==> Executing phase: 'install'
2 ==> 'make' '-j16'
>> 3 make: *** No targets specified and no makefile found. Stop.
See build log for details:
~/spack/var/spack/stage/mpileaks-1.0-sv75n3u5ev6mljwcezisz3slooozbbxu/spack-build-out.txt
This obviously didn't work; we need to fill in the package-specific
information. Specifically, Spack didn't try to build any of mpileaks'
dependencies, nor did it use the proper configure arguments. Let's start
fixing things.
---------------------
Package Documentation
---------------------
We can bring the ``package.py`` file back into our ``EDITOR`` with the
``spack edit`` command:
.. code-block:: console
$ spack edit mpileaks
Let's remove some of the ``FIXME`` comments, add links to the mpileaks
homepage, and document what mpileaks does. I'm also going to cut out the
Copyright clause at this point to keep this tutorial document shorter,
but you shouldn't do that normally. The results of these changes can be
found in ``$SPACK_ROOT/lib/spack/docs/tutorial/examples/1.package.py``
and are displayed below. Make these changes to your ``package.py``:
.. literalinclude:: tutorial/examples/1.package.py
:lines: 6-
:language: python
We've filled in the comment that describes what this package does and
added a link to its website. That won't help us build yet, but it will
allow Spack to provide some documentation on this package to other users:
.. code-block:: console
$ spack info mpileaks
Package: mpileaks
Description:
Tool to detect and report MPI objects like MPI_Requests and
MPI_Datatypes.
Homepage: https://github.com/LLNL/mpileaks
Tags:
None
Preferred version:
1.0 https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz
Safe versions:
1.0 https://github.com/LLNL/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz
Variants:
None
Installation Phases:
install
Build Dependencies:
None
Link Dependencies:
None
Run Dependencies:
None
Virtual Packages:
None
As we fill in more information about this package the ``spack info`` command
will become more informative. Now let's start making this package build.
------------
Dependencies
------------
The mpileaks package depends on three other packages: ``mpi``,
``adept-utils``, and ``callpath``. Let's add those via the
``depends_on`` command in our ``package.py`` (this version is in
``$SPACK_ROOT/lib/spack/docs/tutorial/examples/2.package.py``):
.. literalinclude:: tutorial/examples/2.package.py
:lines: 6-
:language: python
Now when we go to build mpileaks, Spack will fetch and build these
dependencies before building mpileaks. Note that the mpi dependency is a
different kind of beast than the adept-utils and callpath dependencies;
there is no mpi package available in Spack. Instead mpi is a *virtual
dependency*. Spack may satisfy that dependency by installing packages
such as ``openmpi`` or ``mvapich2``. See the :ref:`packaging-guide` for more
information on virtual dependencies.
Now when we try to install this package, a lot more happens:
.. code-block:: console
$ spack install mpileaks
...
==> Successfully installed libdwarf from binary cache
[+] ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/libdwarf-20180129-p4jeflorwlnkoq2vpuyocwrbcht2ayak
==> Installing callpath
==> Searching for binary cache of callpath
==> Installing callpath from binary cache
==> Fetching file:///mirror/build_cache/linux-ubuntu16.04-x86_64/gcc-5.4.0/callpath-1.0.4/linux-ubuntu16.04-x86_64-gcc-5.4.0-callpath-1.0.4-empvyxdkc4j4pwg7gznwhbiumruey66x.spack
######################################################################## 100.0%
==> Successfully installed callpath from binary cache
[+] ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/callpath-1.0.4-empvyxdkc4j4pwg7gznwhbiumruey66x
==> Installing mpileaks
==> Searching for binary cache of mpileaks
==> No binary for mpileaks found: installing from source
==> Using cached archive: ~/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz
==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0.tar.gz
==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb
==> No patches needed for mpileaks
==> Building mpileaks [Package]
==> Executing phase: 'install'
==> Error: ProcessError: Command exited with status 2:
'make' '-j16'
1 error found in build log:
1 ==> Executing phase: 'install'
2 ==> 'make' '-j16'
>> 3 make: *** No targets specified and no makefile found. Stop.
See build log for details:
~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0/spack-build-out.txt
Note that this command may take a while to run and produce more output if
you don't have an MPI already installed or configured in Spack.
Now Spack has identified and made sure all of our dependencies have been
built. It found the ``openmpi`` package that will satisfy our ``mpi``
dependency, and the ``callpath`` and ``adept-utils`` package to satisfy our
concrete dependencies.
------------------------
Debugging Package Builds
------------------------
Our ``mpileaks`` package is still not building. It may be obvious to
many of you that we never ran the configure script. Let's add a
call to ``configure()`` to the top of the install routine. The resulting
``package.py`` is in ``$SPACK_ROOT/lib/spack/docs/tutorial/examples/3.package.py``:
.. literalinclude:: tutorial/examples/3.package.py
:lines: 6-
:language: python
If we re-run we still get errors:
.. code-block:: console
$ spack install mpileaks
...
==> Installing mpileaks
==> Searching for binary cache of mpileaks
==> Finding buildcaches in /mirror/build_cache
==> No binary for mpileaks found: installing from source
==> Using cached archive: ~/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz
==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0.tar.gz
==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb
==> No patches needed for mpileaks
==> Building mpileaks [Package]
==> Executing phase: 'install'
==> Error: ProcessError: Command exited with status 1:
'./configure'
1 error found in build log:
25 checking for ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1.3-3
njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc... ~/spack/opt/spack/linux-ubuntu16.04-
x86_64/gcc-5.4.0/openmpi-3.1.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc
26 Checking whether ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1
.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc responds to '-showme:compile'... no
27 Checking whether ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1
.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc responds to '-showme'... no
28 Checking whether ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1
.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc responds to '-compile-info'... no
29 Checking whether ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.1
.3-3njc4q5pqdpptq6jvqjrezkffwokv2sx/bin/mpicc responds to '-show'... no
30 ./configure: line 4809: Echo: command not found
>> 31 configure: error: unable to locate adept-utils installation
See build log for details:
~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0/spack-build-out.txt
Again, the problem may be obvious. But let's pretend we're not
all experienced Autotools developers and use this opportunity to spend some
time debugging. We have a few options that can tell us about
what's going wrong:
As per the error message, Spack has given us a ``spack-build-out.txt`` debug
log:
.. code-block:: console
==> Executing phase: 'install'
==> './configure'
checking metadata... no
checking installation directory variables... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... /home/spack1/spack/lib/spack/env/gcc/gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /home/spack1/spack/lib/spack/env/gcc/gcc accepts -g... yes
checking for /home/spack1/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of /home/spack1/spack/lib/spack/env/gcc/gcc... gcc3
checking whether /home/spack1/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes
checking whether we are using the GNU C++ compiler... yes
checking whether /home/spack1/spack/lib/spack/env/gcc/g++ accepts -g... yes
checking dependency style of /home/spack1/spack/lib/spack/env/gcc/g++... gcc3
checking for /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc... /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc
Checking whether /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc responds to '-showme:compile'... yes
configure: error: unable to locate adept-utils installation
This gives us the output from the build, and mpileaks isn't
finding its ``adept-utils`` package. Spack has
automatically added the include and library directories of
``adept-utils`` to the compiler's search path, but some packages like
mpileaks can sometimes be picky and still want things spelled out on
their command line. But let's continue to pretend we're not experienced
developers, and explore some other debugging paths:
We can also enter the build area and try to manually run the build:
.. code-block:: console
$ spack build-env mpileaks bash
$ spack cd mpileaks
The ``spack build-env`` command spawned a new shell that contains the same
environment that Spack used to build the mpileaks package (you can
substitute bash for your favorite shell). The ``spack cd`` command
changed our working dirctory to the last attempted build for mpileaks.
From here we can manually re-run the build:
.. code-block:: console
$ ./configure
checking metadata... no
checking installation directory variables... yes
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... /home/spack1/spack/lib/spack/env/gcc/gcc
checking for C compiler default output file name... a.out
checking whether the C compiler works... yes
checking whether we are cross compiling... no
checking for suffix of executables...
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /home/spack1/spack/lib/spack/env/gcc/gcc accepts -g... yes
checking for /home/spack1/spack/lib/spack/env/gcc/gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of /home/spack1/spack/lib/spack/env/gcc/gcc... gcc3
checking whether /home/spack1/spack/lib/spack/env/gcc/gcc and cc understand -c and -o together... yes
checking whether we are using the GNU C++ compiler... yes
checking whether /home/spack1/spack/lib/spack/env/gcc/g++ accepts -g... yes
checking dependency style of /home/spack1/spack/lib/spack/env/gcc/g++... gcc3
checking for /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc... /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc
Checking whether /home/spack1/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/openmpi-3.0.0-yo5qkfvumpmgmvlbalqcadu46j5bd52f/bin/mpicc responds to '-showme:compile'... yes
configure: error: unable to locate adept-utils installation
We're seeing the same error, but now we're in a shell where we can run
the command ourselves and debug as needed. We could, for example, run
``./configure --help`` to see what options we can use to specify
dependencies.
We can use the ``exit`` command to leave the shell spawned by ``spack
build-env``.
------------------------------
Specifying Configure Arguments
------------------------------
Let's add the configure arguments to the mpileaks' ``package.py``. This
version can be found in
``$SPACK_ROOT/lib/spack/docs/tutorial/examples/4.package.py``:
.. literalinclude:: tutorial/examples/4.package.py
:lines: 6-
:language: python
This is all we need for a working mpileaks package! If we install now we'll see:
.. code-block:: console
$ spack install mpileaks
...
==> Installing mpileaks
==> Searching for binary cache of mpileaks
==> Finding buildcaches in /mirror/build_cache
==> No binary for mpileaks found: installing from source
==> Using cached archive: ~/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz
==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb/mpileaks-1.0.tar.gz
==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb
==> No patches needed for mpileaks
==> Building mpileaks [Package]
==> Executing phase: 'install'
==> Successfully installed mpileaks
Fetch: 0.00s. Build: 9.41s. Total: 9.41s.
[+] ~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpileaks-1.0-csoikctsalli4cdkkdk377gprkc472rb
There are some special circumstances in this package that are worth highlighting.
Normally, Spack would have automatically detected that mpileaks was an
Autotools-based package when we ran ``spack create`` and made it an ``AutoToolsPackage``
class (except we added the ``-t generic`` option to skip this). Instead of
a full install routine we would have just written:
.. code-block:: python
def configure_args(self):
return [
'--with-adept-utils={0}'.format(self.spec['adept-utils'].prefix),
'--with-callpath={0}'.format(self.spec['callpath'].prefix)
]
Similarly, if this had been a CMake-based package we
would have been filling in a ``cmake_args`` function instead of
``configure_args``. There are similar default package types for
many build environments that will be discussed later in the tutorial.
--------
Variants
--------
We have a successful mpileaks build, but let's take some time to improve
it. ``mpileaks`` has a build-time option to truncate parts of the stack
that it walks. Let's add a variant to allow users to set this when they
build mpileaks with Spack.
To do this, we'll add a variant to our package, as per the following (see
``$SPACK_ROOT/lib/spack/docs/tutorial/examples/5.package.py``):
.. literalinclude:: tutorial/examples/5.package.py
:lines: 6-
:language: python
We've added the variant ``stackstart``, and given it a default value of
``0``. If we install now we can see the stackstart variant added to the
configure line (output truncated for length):
.. code-block:: console
$ spack install --verbose mpileaks stackstart=4
...
==> Installing mpileaks
==> Searching for binary cache of mpileaks
==> Finding buildcaches in /mirror/build_cache
==> No binary for mpileaks found: installing from source
==> Using cached archive: ~/spack/var/spack/cache/mpileaks/mpileaks-1.0.tar.gz
==> Staging archive: ~/spack/var/spack/stage/mpileaks-1.0-meufjojkxve3l7rci2mbud3faidgplto/mpileaks-1.0.tar.gz
==> Created stage in ~/spack/var/spack/stage/mpileaks-1.0-meufjojkxve3l7rci2mbud3faidgplto
==> No patches needed for mpileaks
==> Building mpileaks [Package]
==> Executing phase: 'install'
==> './configure' '--prefix=~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/mpileaks-1.0-meufjojkxve3l7rci2mbud3faidgplto' '--with-adept-utils=~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/adept-utils-1.0.1-7tippnvo5g76wpijk7x5kwfpr3iqiaen' '--with-callpath=~/spack/opt/spack/linux-ubuntu16.04-x86_64/gcc-5.4.0/callpath-1.0.4-empvyxdkc4j4pwg7gznwhbiumruey66x' '--with-stack-start-c=4' '--with-stack-start-fortran=4'
---------------
The Spec Object
---------------
This tutorial has glossed over a few important features, which weren't
too relevant for mpileaks but may be useful for other packages. There
were several places we reference the ``self.spec`` object. This is a
powerful class for querying information about what we're building. For
example, you could use the spec to query information about how a
package's dependencies were built, or what compiler was being used, or
what version of a package is being installed. Full documentation can be
found in the :ref:`packaging-guide`, but here's some quick snippets with
common queries:
- Am I building ``mpileaks`` version ``1.1`` or greater?
.. code-block:: python
if self.spec.satisfies('@1.1:'):
# Do things needed for 1.1+
- Is ``openmpi`` the MPI I'm building with?
.. code-block:: python
if self.spec['mpi'].name == 'openmpi':
# Do openmpi things
- Am I building with ``gcc`` version less than ``5.0.0``:
.. code-block:: python
if self.spec.satisfies('%gcc@:5.0.0'):
# Add arguments specific to gcc's earlier than 5.0.0
- Am I building with the ``debug`` variant:
.. code-block:: python
if self.spec.satisfies('+debug'):
# Add -g option to configure flags
- Is my ``dyninst`` dependency greater than version ``8.0``?
.. code-block:: python
if self.spec['dyninst'].satisfies('@8.0:'):
# Use newest dyninst options
More examples can be found in the thousands of packages already added to
Spack in ``$SPACK_ROOT/var/spack/repos/builtin/packages``.
Good Luck!
To ensure that future sections of the tutorial run properly, please
uninstall mpileaks and remove the tutorial repo from your
configuration.
.. code-block:: console
$ spack uninstall -ay mpileaks
$ spack repo remove tutorial
$ rm -rf $SPACK_ROOT/var/spack/repos/tutorial/packages/mpileaks