hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

docs: initial documentation for environments

Browse source
This commit is contained in:
Gregory Becker 2019-03-28 19:23:16 -07:00 committed by Todd Gamblin
parent d450a2fce2
commit 12c8996294
2 changed files with 636 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

635
lib/spack/docs/environments.rst Normal file
View file

@ -0,0 +1,635 @@
.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
Spack Project Developers. See the top-level COPYRIGHT file for details.
SPDX-License-Identifier: (Apache-2.0 OR MIT)
.. _environments:
============
Environments
============
An environment is used to group together a set of specs for the
purpose of building, rebuilding and deploying in a coherent fashion.
Environments provide a number of advantages over the a la carte
approach of building and loading individual Spack modules:
#. Environments separate the steps of (a) choosing what to
install, (b) concretizing, and (c) installing. This allows
Environments to remain stable and repeatable, even if Spack packages
are upgraded: specs are only re-concretized when the user
explicitly asks for it. It is even be possible to reliably
transport environments between different computers running
different versions of Spack!
#. Environments allow several specs to be built at once; a more robust
solution than ad-hoc scripts making multiple calls to ``spack
install``.
#. An Environment that is built as a whole can be loaded as a whole
into the user environment. An Environment can be build to maintain
a filesystem view of its packages, and the environment can load
that view into the user environment at activation time. Spack can
also generate a script to load all modules related to an
environment.
Other packaging systems also provide environments that are similar in
some ways to Spack environments; for example, `Conda environments
<https://conda.io/docs/user-guide/tasks/manage-environments.html>`_ or
`Python Virtual Environments
<https://docs.python.org/3/tutorial/venv.html>`_. Spack environments
provide some distinctive features:
#. A spec installed "in" an environment is no different from the same
spec installed anywhere else in Spack. Environments are assembled
simply by collecting together a set of specs.
#. Spack Environments may contain more than one spec of the same
package.
Spack uses a "manifest and lock" model similar to `Bundler gemfiles
<https://bundler.io/man/gemfile.5.html>`_ and other package
managers. The user input file is named ``spack.yaml`` and the lock
file is named ``spack.lock``
------------------
Using Environments
------------------
Here we follow a typical use case of creating, concretizing,
installing and loading an environment.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Creating a named Environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
An environment is created by:
.. code-block:: console
$ spack env create myenv
Spack then creates the directory ``var/spack/environments/myenv``.
.. note::
All named environments are stored in the ``var/spack/environments`` folder.
In the ``var/spack/environments/myenv`` directory, Spack creates the
file ``spack.yaml`` and the hidden directory ``.spack-env``.
Spack stores metadata in the ``.spack-env`` directory. User
interaction will occur through the ``spack.yaml`` file and the Spack
commands that affect it. When the environment is concretized, Spack
will create a file ``spack.lock`` with the concrete information for
the environment.
In addition to being the default location for the view associated with
an Environment, the ``.spack-env`` directory also contains:
* ``repo/``: A repo consisting of the Spack packages used in this
environment. This allows the environment to build the same, in
theory, even on different verions of Spack with different
packages!
* ``logs/``: A directory containing the build logs for the packages
in this Environment.
Spack Environments can also be created from either a ``spack.yaml``
manifest or a ``spack.lock`` lockfile. To create an Environment from a
``spack.yaml`` manifest:
.. code-block:: console
$ spack env create myenv spack.yaml
To create an Environment from a ``spack.lock`` lockfile:
.. code-block:: console
$ spack env create myenv spack.lock
Either of these commands can also take a full path to the
initialization file.
A Spack Environment created from a ``spack.yaml`` manifest is
guaranteed to have the same root specs as the original Environment,
but may concretize differently. A Spack Environment created from a
``spack.lock`` lockfile is guaranteed to have the same concrete specs
as the original Environment. Either may obviously then differ as the
user modifies it.
^^^^^^^^^^^^^^^^^^^^^^^^^
Activating an Environment
^^^^^^^^^^^^^^^^^^^^^^^^^
To activate an environment, use the following command:
.. code-block:: console
$ spack env activate myenv
By default, the ``spack env activate`` will load the view associated
with the Environment into the user environment. The ``-v,
--with-view`` argument ensures this behavior, and the ``-V,
--without-vew`` argument activates the environment without changing
the user environment variables.
The ``-p`` option to the ``spack env activate`` command modifies the
user's prompt to begin with the environment name in brackets.
.. code-block:: console
$ spack env activate -p myenv
[myenv] $ ...
To deactivate an environment, use the command:
.. code-block:: console
$ spack env deactivate
or the shortcut alias
.. code-block:: console
$ despacktivate
If the environment was activated with its view, deactivating the
environment will remove the view from the user environment.
^^^^^^^^^^^^^^^^^^^^^^
Anonymous Environments
^^^^^^^^^^^^^^^^^^^^^^
Any directory can be treated as an environment if it contains a file
``spack.yaml``. To load an anonymous environment, use:
.. code-block:: console
$ spack env activate -d /path/to/directory
Spack commands that are environment sensitive will also act on the
environment any time the current working directory contains a
``spack.yaml`` file. Changing working directory to a directory
containing a ``spack.yaml`` file is equivalent to the command:
.. code-block:: console
$ spack env activate -d /path/to/dir --without-view
Anonymous specs can be created in place using the command:
.. code-block:: console
$ spack env create -d .
In this case Spack simply creates a spack.yaml file in the requested
directory.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Environment Sensitive Commands
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Spack commands are environment sensitive. For example, the ``find``
command shows only the specs in the active Environment if an
Environment has been activated. Similarly, the ``install`` and
``uninstall`` commands act on the active environment.
.. code-block:: console
$ spack find
==> 0 installed packages
$ spack install zlib@1.2.11
==> Installing zlib
==> Searching for binary cache of zlib
==> Warning: No Spack mirrors are currently configured
==> No binary for zlib found: installing from source
==> Fetching http://zlib.net/fossils/zlib-1.2.11.tar.gz
######################################################################## 100.0%
==> Staging archive: /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur/zlib-1.2.11.tar.gz
==> Created stage in /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
==> No patches needed for zlib
==> Building zlib [Package]
==> Executing phase: 'install'
==> Successfully installed zlib
Fetch: 0.36s. Build: 11.58s. Total: 11.93s.
[+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
$ spack env activate myenv
$ spack find
==> In environment myenv
==> No root specs
==> 0 installed packages
$ spack install zlib@1.2.8
==> Installing zlib
==> Searching for binary cache of zlib
==> Warning: No Spack mirrors are currently configured
==> No binary for zlib found: installing from source
==> Fetching http://zlib.net/fossils/zlib-1.2.8.tar.gz
######################################################################## 100.0%
==> Staging archive: /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7/zlib-1.2.8.tar.gz
==> Created stage in /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
==> No patches needed for zlib
==> Building zlib [Package]
==> Executing phase: 'install'
==> Successfully installed zlib
Fetch: 0.26s. Build: 2.08s. Total: 2.35s.
[+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
$ spack find
==> In environment myenv
==> Root specs
zlib@1.2.8
==> 1 installed package
-- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
zlib@1.2.8
$ despacktivate
$ spack find
==> 2 installed packages
-- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
zlib@1.2.8 zlib@1.2.11
Note that when we installed the abstract spec ``zlib@1.2.8``, it was
presented as a root of the Environment. All explicitly installed
packages will be listed as roots of the Environment.
All of the Spack commands that act on the list of installed specs are
Environment-sensitive in this way, including ``install``,
``uninstall``, ``activate``, ``deactivate``, ``find``, ``extensions``,
and more. In the :ref:`environment-configuration` section we will discuss
Environment-sensitive commands further.
^^^^^^^^^^^^^^^^^^^^^
Adding Abstract Specs
^^^^^^^^^^^^^^^^^^^^^
An abstract spec is the user-specified spec before Spack has applied
any defaults or dependency information.
Users can add abstract specs to an Environment using the ``spack add``
command. The most important component of an Environment is a list of
abstract specs.
Adding a spec adds to the manifest (the ``spack.yaml`` file) and to
the roots of the Environment, but does not affect the concrete specs
in the lockfile, nor does it install the spec.
The ``spack add`` command is environment aware. It adds to the
currently active environment. All environment aware commands can also
be called using the ``spack -E`` flag to specify the environment.
.. code-block:: console
$ spack activate myenv
$ spack add mpileaks
or
.. code-block:: console
$ spack -E myenv add python
^^^^^^^^^^^^
Concretizing
^^^^^^^^^^^^
Once some user specs have been added to an environment, they can be
concretized. The following command will concretize all user specs
that have been added and not yet concretized:
.. code-block:: console
[myenv]$ spack concretize
This command will re-concretize all specs:
.. code-block:: console
[myenv]$ spack concretize -f
When the ``-f`` flag is not used to reconcretize all specs, Spack
guarantees that already concretized specs are unchanged in the
environment.
The ``concretize`` command does not install any packages. For packages
that have already been installed outside of the environment, the
process of adding the spec and concretizing is identical to installing
the spec assuming it concretizes to the exact spec that was installed
ouside of the environment.
The ``spack find`` command can show concretized specs separately from
installed specs using the ``-c`` (``--concretized``) flag.
.. code-block:: console
[myenv]$ spack add zlib
[myenv]$ spack concretize
[myenv]$ spack find -c
==> In environment myenv
==> Root specs
zlib
==> Concretized roots
-- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
zlib@1.2.11
==> 0 installed packages
^^^^^^^^^^^^^^^^^^^^^^^^^
Installing an Environment
^^^^^^^^^^^^^^^^^^^^^^^^^
In addition to installing individual specs into an Environment, one
can install the entire Environment at once using the command
.. code-block:: console
[myenv]$ spack install
If the Environment has been concretized, Spack will install the
concretized specs. Otherwise, ``spack install`` will first concretize
the Environment and then install the concretized specs.
As it installs, ``spack install`` creates symbolic links in the
``logs/`` directory in the Environment, allowing for easy inspection
of build logs related to that environment. The ``spack install``
command also stores a Spack repo containing the ``package.py`` file
used at install time for each package in the ``repos/`` directory in
the Environment.
^^^^^^^
Loading
^^^^^^^
Once an environment has been installed, the following creates a load script for it:
.. code-block:: console
$ spack env myenv loads -r
This creates a file called ``loads`` in the environment directory.
Sourcing that file in Bash will make the environment available to the
user; and can be included in ``.bashrc`` files, etc. The ``loads``
file may also be copied out of the environment, renamed, etc.
----------
spack.yaml
----------
Spack environments can be customized at finer granularity by editing
the ``spack.yaml`` manifest file directly.
.. _environment-configuration:
^^^^^^^^^^^^^^^^^^^^^^^^
Configuring Environments
^^^^^^^^^^^^^^^^^^^^^^^^
A variety of Spack behaviors are changed through Spack configuration
files, covered in more detail in the :ref:`configuration`
section.
Spack Environments provide an additional level of configuration scope
between the custom scope and the user scope discussed in the
configuration documentation.
There are two ways to include configuration information in a Spack Environment:
#. Inline in the ``spack.yaml`` file
#. Included in the ``spack.yaml`` file from another file.
"""""""""""""""""""""
Inline configurations
"""""""""""""""""""""
Inline Environment-scope configuration is done using the same yaml
format as standard Spack configuration scopes, covered in the
:ref:`configuration` section. Each section is contained under a
top-level yaml object with it's name. For example, a ``spack.yaml``
manifest file containing some package preference configuration (as in
a ``packages.yaml`` file) could contain:
.. code-block:: yaml
spack:
...
packages:
all:
compilers: [intel]
...
This configuration sets the default compiler for all packages to
``intel``.
"""""""""""""""""""""""
Included configurations
"""""""""""""""""""""""
Spack environments allow an ``include`` heading in their yaml
schema. This heading pulls in external configuration files and applies
them to the Environment.
.. code-block:: yaml
spack:
include:
- relative/path/to/config.yaml
- /absolute/path/to/packages.yaml
Environments can include files either with a relative or absolute
paths. Inline configurations take precedence over included
configurations, so you don't have to change shared configuration files
to make small changes to an individual Environment. Included configs
listed later will have higher precedence, as the included configs are
applied in order.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Manually Editing the Specs List
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The list of abstract/root specs in the Environment is maintained in
the ``spack.yaml`` manifest under the heading ``specs``.
.. code-block:: yaml
spack:
specs:
- ncview
- netcdf
- nco
- py-sphinx
Appending to this list in the yaml is identical to using the ``spack
add`` command from the command line. However, there is more power
available from the yaml file.
"""""""""""""
Spec Matrices
"""""""""""""
Entries in the ``specs`` list can be individual abstract specs or a
spec matrix.
A spec matrix is a yaml object containing multiple lists of specs, and
evaluates to the cross-product of those specs. Spec matrices also
contain an ``excludes`` directive, which eliminates certain
combinations from the evaluated result.
The following two Environment manifests are identical:
.. code-block:: yaml
spack:
specs:
- zlib %gcc@7.1.0
- zlib %gcc@4.9.3
- libelf %gcc@7.1.0
- libelf %gcc@4.9.3
- libdwarf %gcc@7.1.0
- cmake
spack:
specs:
- matrix:
- [zlib, libelf, libdwarf]
- ['%gcc@7.1.0', '%gcc@4.9.3']
exclude:
- libdwarf%gcc@4.9.3
- cmake
Spec matrices can be used to install swaths of software across various
toolchains.
The concretization logic for spec matrices differs slightly from the
rest of Spack. If a variant or dependency constraint from a matrix is
invalid, Spack will reject the constraint and try again without
it. For example, the following two Environment manifests will produce
the same specs:
.. code-block:: yaml
spack:
specs:
- matrix:
- [zlib, libelf, hdf5+mpi]
- [^mvapich2@2.2, ^openmpi@3.1.0]
spack:
specs:
- zlib
- libelf
- hdf5+mpi ^mvapich2@2.2
- hdf5+mpi ^openmpi@3.1.0
This allows one to create toolchains out of combinations of
constraints and apply them somewhat indiscriminantly to packages,
without regard for the applicability of the constraint.
""""""""""""""""""""
Spec List References
""""""""""""""""""""
The last type of possible entry in the specs list is a reference.
The Spack Environment manifest yaml schema contains an additional
heading ``definitions``. Under definitions is an array of yaml
objects. Each object has one or two fields. The one required field is
a name, and the optional field is a ``when`` clause.
The named field is a spec list. The spec list uses the same syntax as
the ``specs`` entry. Each entry in the spec list can be a spec, a spec
matrix, or a reference to an earlier named list. References are
specified using the ``$`` sigil, and are "splatted" into place
(i.e. the elements of the referent are at the same level as the
elements listed separately). As an example, the following two manifest
files are identical.
.. code-block:: yaml
spack:
definitions:
- first: [libelf, libdwarf]
- compilers: ['%gcc', '^intel']
- second:
- $first
- matrix:
- [zlib]
- [$compilers]
specs:
- $second
- cmake
spack:
specs:
- libelf
- libdwarf
- zlib%gcc
- zlib%intel
- cmake
.. note::
Named a spec list in the definitions section may only refer
to a named list defined above itself. Order matters.
In short files like the example, it may be easier to simply list the
included specs. However for more complicated examples involving many
packages across many toolchains, separately factored lists make
Environments substantially more manageable.
Additionally, the ``-l`` option to the ``spack add`` command allows
one to add to named lists in the definitions section of the manifest
file directly from the command line.
The ``when`` directive can be used to conditionally add specs to a
named list. The ``when`` directive takes a string of python code
referring to a restricted set of variables, and evaluates to a
boolean. The specs listed are appended to the named list if the
``when`` string evaluates to ``True``. In the following snippet, the
named list ``compilers`` is ``['%gcc', '%clang', '%intel']`` on
``x86_64`` systems and ``['%gcc', '%clang']`` on all other systems.
.. code-block:: yaml
spack:
definitions:
- compilers: ['%gcc', '%clang']
- when: target == 'x86_64'
compilers: ['%intel']
.. note::
Any definitions with the same named list with true ``when``
clauses (or absent ``when`` clauses) will be appended together
The valid variables for a ``when`` clause are:
#. ``platform``. The platform string of the default Spack
architecture on the system.
#. ``os``. The os string of the default Spack architecture on
the system.
#. ``target``. The target string of the default Spack
architecture on the system.
#. ``architecture`` or ``arch``. The full string of the
default Spack architecture on the system.
#. ``re``. The standard regex module in python.
#. ``env``. The user environment (usually ``os.environ`` in python).
#. ``hostname``. The hostname of the system (if ``hostname`` is an
executable in the user's PATH).
^^^^^^^^^^^^^^^^^^^^^^^^^
Environment-managed Views
^^^^^^^^^^^^^^^^^^^^^^^^^
Code for Spack Environments is in code review at the
moment. Documentation will be updated when the code is finalized.

1
lib/spack/docs/index.rst
View file

@ -65,6 +65,7 @@ or refer to the full manual below.
configuration
config_yaml
build_settings
environments
mirrors
module_file_support
repositories