diff --git a/lib/spack/docs/environments.rst b/lib/spack/docs/environments.rst new file mode 100644 index 0000000000..2f619bf5d8 --- /dev/null +++ b/lib/spack/docs/environments.rst @@ -0,0 +1,635 @@ +.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other + Spack Project Developers. See the top-level COPYRIGHT file for details. + + SPDX-License-Identifier: (Apache-2.0 OR MIT) + +.. _environments: + +============ +Environments +============ + +An environment is used to group together a set of specs for the +purpose of building, rebuilding and deploying in a coherent fashion. +Environments provide a number of advantages over the a la carte +approach of building and loading individual Spack modules: + +#. Environments separate the steps of (a) choosing what to + install, (b) concretizing, and (c) installing. This allows + Environments to remain stable and repeatable, even if Spack packages + are upgraded: specs are only re-concretized when the user + explicitly asks for it. It is even be possible to reliably + transport environments between different computers running + different versions of Spack! +#. Environments allow several specs to be built at once; a more robust + solution than ad-hoc scripts making multiple calls to ``spack + install``. +#. An Environment that is built as a whole can be loaded as a whole + into the user environment. An Environment can be build to maintain + a filesystem view of its packages, and the environment can load + that view into the user environment at activation time. Spack can + also generate a script to load all modules related to an + environment. + +Other packaging systems also provide environments that are similar in +some ways to Spack environments; for example, `Conda environments +`_ or +`Python Virtual Environments +`_. Spack environments +provide some distinctive features: + +#. A spec installed "in" an environment is no different from the same + spec installed anywhere else in Spack. Environments are assembled + simply by collecting together a set of specs. +#. Spack Environments may contain more than one spec of the same + package. + +Spack uses a "manifest and lock" model similar to `Bundler gemfiles +`_ and other package +managers. The user input file is named ``spack.yaml`` and the lock +file is named ``spack.lock`` + +------------------ +Using Environments +------------------ + +Here we follow a typical use case of creating, concretizing, +installing and loading an environment. + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Creating a named Environment +^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +An environment is created by: + +.. code-block:: console + + $ spack env create myenv + +Spack then creates the directory ``var/spack/environments/myenv``. + +.. note:: + + All named environments are stored in the ``var/spack/environments`` folder. + +In the ``var/spack/environments/myenv`` directory, Spack creates the +file ``spack.yaml`` and the hidden directory ``.spack-env``. + +Spack stores metadata in the ``.spack-env`` directory. User +interaction will occur through the ``spack.yaml`` file and the Spack +commands that affect it. When the environment is concretized, Spack +will create a file ``spack.lock`` with the concrete information for +the environment. + +In addition to being the default location for the view associated with +an Environment, the ``.spack-env`` directory also contains: + + * ``repo/``: A repo consisting of the Spack packages used in this + environment. This allows the environment to build the same, in + theory, even on different verions of Spack with different + packages! + * ``logs/``: A directory containing the build logs for the packages + in this Environment. + +Spack Environments can also be created from either a ``spack.yaml`` +manifest or a ``spack.lock`` lockfile. To create an Environment from a +``spack.yaml`` manifest: + +.. code-block:: console + + $ spack env create myenv spack.yaml + +To create an Environment from a ``spack.lock`` lockfile: + +.. code-block:: console + + $ spack env create myenv spack.lock + +Either of these commands can also take a full path to the +initialization file. + +A Spack Environment created from a ``spack.yaml`` manifest is +guaranteed to have the same root specs as the original Environment, +but may concretize differently. A Spack Environment created from a +``spack.lock`` lockfile is guaranteed to have the same concrete specs +as the original Environment. Either may obviously then differ as the +user modifies it. + +^^^^^^^^^^^^^^^^^^^^^^^^^ +Activating an Environment +^^^^^^^^^^^^^^^^^^^^^^^^^ + +To activate an environment, use the following command: + +.. code-block:: console + + $ spack env activate myenv + +By default, the ``spack env activate`` will load the view associated +with the Environment into the user environment. The ``-v, +--with-view`` argument ensures this behavior, and the ``-V, +--without-vew`` argument activates the environment without changing +the user environment variables. + +The ``-p`` option to the ``spack env activate`` command modifies the +user's prompt to begin with the environment name in brackets. + +.. code-block:: console + + $ spack env activate -p myenv + [myenv] $ ... + +To deactivate an environment, use the command: + +.. code-block:: console + + $ spack env deactivate + +or the shortcut alias + +.. code-block:: console + + $ despacktivate + +If the environment was activated with its view, deactivating the +environment will remove the view from the user environment. + +^^^^^^^^^^^^^^^^^^^^^^ +Anonymous Environments +^^^^^^^^^^^^^^^^^^^^^^ + +Any directory can be treated as an environment if it contains a file +``spack.yaml``. To load an anonymous environment, use: + +.. code-block:: console + + $ spack env activate -d /path/to/directory + +Spack commands that are environment sensitive will also act on the +environment any time the current working directory contains a +``spack.yaml`` file. Changing working directory to a directory +containing a ``spack.yaml`` file is equivalent to the command: + +.. code-block:: console + + $ spack env activate -d /path/to/dir --without-view + +Anonymous specs can be created in place using the command: + +.. code-block:: console + + $ spack env create -d . + +In this case Spack simply creates a spack.yaml file in the requested +directory. + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Environment Sensitive Commands +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Spack commands are environment sensitive. For example, the ``find`` +command shows only the specs in the active Environment if an +Environment has been activated. Similarly, the ``install`` and +``uninstall`` commands act on the active environment. + +.. code-block:: console + + $ spack find + ==> 0 installed packages + + $ spack install zlib@1.2.11 + ==> Installing zlib + ==> Searching for binary cache of zlib + ==> Warning: No Spack mirrors are currently configured + ==> No binary for zlib found: installing from source + ==> Fetching http://zlib.net/fossils/zlib-1.2.11.tar.gz + ######################################################################## 100.0% + ==> Staging archive: /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur/zlib-1.2.11.tar.gz + ==> Created stage in /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur + ==> No patches needed for zlib + ==> Building zlib [Package] + ==> Executing phase: 'install' + ==> Successfully installed zlib + Fetch: 0.36s. Build: 11.58s. Total: 11.93s. + [+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur + + $ spack env activate myenv + + $ spack find + ==> In environment myenv + ==> No root specs + + ==> 0 installed packages + + $ spack install zlib@1.2.8 + ==> Installing zlib + ==> Searching for binary cache of zlib + ==> Warning: No Spack mirrors are currently configured + ==> No binary for zlib found: installing from source + ==> Fetching http://zlib.net/fossils/zlib-1.2.8.tar.gz + ######################################################################## 100.0% + ==> Staging archive: /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7/zlib-1.2.8.tar.gz + ==> Created stage in /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7 + ==> No patches needed for zlib + ==> Building zlib [Package] + ==> Executing phase: 'install' + ==> Successfully installed zlib + Fetch: 0.26s. Build: 2.08s. Total: 2.35s. + [+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7 + + $ spack find + ==> In environment myenv + ==> Root specs + zlib@1.2.8 + + ==> 1 installed package + -- linux-rhel7-x86_64 / gcc@4.9.3 ------------------------------- + zlib@1.2.8 + + $ despacktivate + $ spack find + ==> 2 installed packages + -- linux-rhel7-x86_64 / gcc@4.9.3 ------------------------------- + zlib@1.2.8 zlib@1.2.11 + +Note that when we installed the abstract spec ``zlib@1.2.8``, it was +presented as a root of the Environment. All explicitly installed +packages will be listed as roots of the Environment. + +All of the Spack commands that act on the list of installed specs are +Environment-sensitive in this way, including ``install``, +``uninstall``, ``activate``, ``deactivate``, ``find``, ``extensions``, +and more. In the :ref:`environment-configuration` section we will discuss +Environment-sensitive commands further. + +^^^^^^^^^^^^^^^^^^^^^ +Adding Abstract Specs +^^^^^^^^^^^^^^^^^^^^^ + +An abstract spec is the user-specified spec before Spack has applied +any defaults or dependency information. + +Users can add abstract specs to an Environment using the ``spack add`` +command. The most important component of an Environment is a list of +abstract specs. + +Adding a spec adds to the manifest (the ``spack.yaml`` file) and to +the roots of the Environment, but does not affect the concrete specs +in the lockfile, nor does it install the spec. + +The ``spack add`` command is environment aware. It adds to the +currently active environment. All environment aware commands can also +be called using the ``spack -E`` flag to specify the environment. + +.. code-block:: console + + $ spack activate myenv + $ spack add mpileaks + +or + +.. code-block:: console + + $ spack -E myenv add python + +^^^^^^^^^^^^ +Concretizing +^^^^^^^^^^^^ + +Once some user specs have been added to an environment, they can be +concretized. The following command will concretize all user specs +that have been added and not yet concretized: + +.. code-block:: console + + [myenv]$ spack concretize + +This command will re-concretize all specs: + +.. code-block:: console + + [myenv]$ spack concretize -f + +When the ``-f`` flag is not used to reconcretize all specs, Spack +guarantees that already concretized specs are unchanged in the +environment. + +The ``concretize`` command does not install any packages. For packages +that have already been installed outside of the environment, the +process of adding the spec and concretizing is identical to installing +the spec assuming it concretizes to the exact spec that was installed +ouside of the environment. + +The ``spack find`` command can show concretized specs separately from +installed specs using the ``-c`` (``--concretized``) flag. + +.. code-block:: console + + [myenv]$ spack add zlib + [myenv]$ spack concretize + [myenv]$ spack find -c + ==> In environment myenv + ==> Root specs + zlib + + ==> Concretized roots + -- linux-rhel7-x86_64 / gcc@4.9.3 ------------------------------- + zlib@1.2.11 + + ==> 0 installed packages + +^^^^^^^^^^^^^^^^^^^^^^^^^ +Installing an Environment +^^^^^^^^^^^^^^^^^^^^^^^^^ + +In addition to installing individual specs into an Environment, one +can install the entire Environment at once using the command + +.. code-block:: console + + [myenv]$ spack install + +If the Environment has been concretized, Spack will install the +concretized specs. Otherwise, ``spack install`` will first concretize +the Environment and then install the concretized specs. + +As it installs, ``spack install`` creates symbolic links in the +``logs/`` directory in the Environment, allowing for easy inspection +of build logs related to that environment. The ``spack install`` +command also stores a Spack repo containing the ``package.py`` file +used at install time for each package in the ``repos/`` directory in +the Environment. + +^^^^^^^ +Loading +^^^^^^^ + +Once an environment has been installed, the following creates a load script for it: + +.. code-block:: console + + $ spack env myenv loads -r + +This creates a file called ``loads`` in the environment directory. +Sourcing that file in Bash will make the environment available to the +user; and can be included in ``.bashrc`` files, etc. The ``loads`` +file may also be copied out of the environment, renamed, etc. + +---------- +spack.yaml +---------- + +Spack environments can be customized at finer granularity by editing +the ``spack.yaml`` manifest file directly. + +.. _environment-configuration: + +^^^^^^^^^^^^^^^^^^^^^^^^ +Configuring Environments +^^^^^^^^^^^^^^^^^^^^^^^^ + +A variety of Spack behaviors are changed through Spack configuration +files, covered in more detail in the :ref:`configuration` +section. + +Spack Environments provide an additional level of configuration scope +between the custom scope and the user scope discussed in the +configuration documentation. + +There are two ways to include configuration information in a Spack Environment: + +#. Inline in the ``spack.yaml`` file + +#. Included in the ``spack.yaml`` file from another file. + +""""""""""""""""""""" +Inline configurations +""""""""""""""""""""" + +Inline Environment-scope configuration is done using the same yaml +format as standard Spack configuration scopes, covered in the +:ref:`configuration` section. Each section is contained under a +top-level yaml object with it's name. For example, a ``spack.yaml`` +manifest file containing some package preference configuration (as in +a ``packages.yaml`` file) could contain: + +.. code-block:: yaml + + spack: + ... + packages: + all: + compilers: [intel] + ... + +This configuration sets the default compiler for all packages to +``intel``. + +""""""""""""""""""""""" +Included configurations +""""""""""""""""""""""" + +Spack environments allow an ``include`` heading in their yaml +schema. This heading pulls in external configuration files and applies +them to the Environment. + +.. code-block:: yaml + + spack: + include: + - relative/path/to/config.yaml + - /absolute/path/to/packages.yaml + +Environments can include files either with a relative or absolute +paths. Inline configurations take precedence over included +configurations, so you don't have to change shared configuration files +to make small changes to an individual Environment. Included configs +listed later will have higher precedence, as the included configs are +applied in order. + +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Manually Editing the Specs List +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The list of abstract/root specs in the Environment is maintained in +the ``spack.yaml`` manifest under the heading ``specs``. + +.. code-block:: yaml + + spack: + specs: + - ncview + - netcdf + - nco + - py-sphinx + +Appending to this list in the yaml is identical to using the ``spack +add`` command from the command line. However, there is more power +available from the yaml file. + +""""""""""""" +Spec Matrices +""""""""""""" + +Entries in the ``specs`` list can be individual abstract specs or a +spec matrix. + +A spec matrix is a yaml object containing multiple lists of specs, and +evaluates to the cross-product of those specs. Spec matrices also +contain an ``excludes`` directive, which eliminates certain +combinations from the evaluated result. + +The following two Environment manifests are identical: + +.. code-block:: yaml + + spack: + specs: + - zlib %gcc@7.1.0 + - zlib %gcc@4.9.3 + - libelf %gcc@7.1.0 + - libelf %gcc@4.9.3 + - libdwarf %gcc@7.1.0 + - cmake + + spack: + specs: + - matrix: + - [zlib, libelf, libdwarf] + - ['%gcc@7.1.0', '%gcc@4.9.3'] + exclude: + - libdwarf%gcc@4.9.3 + - cmake + +Spec matrices can be used to install swaths of software across various +toolchains. + +The concretization logic for spec matrices differs slightly from the +rest of Spack. If a variant or dependency constraint from a matrix is +invalid, Spack will reject the constraint and try again without +it. For example, the following two Environment manifests will produce +the same specs: + +.. code-block:: yaml + + spack: + specs: + - matrix: + - [zlib, libelf, hdf5+mpi] + - [^mvapich2@2.2, ^openmpi@3.1.0] + + spack: + specs: + - zlib + - libelf + - hdf5+mpi ^mvapich2@2.2 + - hdf5+mpi ^openmpi@3.1.0 + +This allows one to create toolchains out of combinations of +constraints and apply them somewhat indiscriminantly to packages, +without regard for the applicability of the constraint. + +"""""""""""""""""""" +Spec List References +"""""""""""""""""""" + +The last type of possible entry in the specs list is a reference. + +The Spack Environment manifest yaml schema contains an additional +heading ``definitions``. Under definitions is an array of yaml +objects. Each object has one or two fields. The one required field is +a name, and the optional field is a ``when`` clause. + +The named field is a spec list. The spec list uses the same syntax as +the ``specs`` entry. Each entry in the spec list can be a spec, a spec +matrix, or a reference to an earlier named list. References are +specified using the ``$`` sigil, and are "splatted" into place +(i.e. the elements of the referent are at the same level as the +elements listed separately). As an example, the following two manifest +files are identical. + +.. code-block:: yaml + + spack: + definitions: + - first: [libelf, libdwarf] + - compilers: ['%gcc', '^intel'] + - second: + - $first + - matrix: + - [zlib] + - [$compilers] + specs: + - $second + - cmake + + spack: + specs: + - libelf + - libdwarf + - zlib%gcc + - zlib%intel + - cmake + +.. note:: + + Named a spec list in the definitions section may only refer + to a named list defined above itself. Order matters. + +In short files like the example, it may be easier to simply list the +included specs. However for more complicated examples involving many +packages across many toolchains, separately factored lists make +Environments substantially more manageable. + +Additionally, the ``-l`` option to the ``spack add`` command allows +one to add to named lists in the definitions section of the manifest +file directly from the command line. + +The ``when`` directive can be used to conditionally add specs to a +named list. The ``when`` directive takes a string of python code +referring to a restricted set of variables, and evaluates to a +boolean. The specs listed are appended to the named list if the +``when`` string evaluates to ``True``. In the following snippet, the +named list ``compilers`` is ``['%gcc', '%clang', '%intel']`` on +``x86_64`` systems and ``['%gcc', '%clang']`` on all other systems. + +.. code-block:: yaml + + spack: + definitions: + - compilers: ['%gcc', '%clang'] + - when: target == 'x86_64' + compilers: ['%intel'] + +.. note:: + + Any definitions with the same named list with true ``when`` + clauses (or absent ``when`` clauses) will be appended together + +The valid variables for a ``when`` clause are: + +#. ``platform``. The platform string of the default Spack + architecture on the system. + +#. ``os``. The os string of the default Spack architecture on + the system. + +#. ``target``. The target string of the default Spack + architecture on the system. + +#. ``architecture`` or ``arch``. The full string of the + default Spack architecture on the system. + +#. ``re``. The standard regex module in python. + +#. ``env``. The user environment (usually ``os.environ`` in python). + +#. ``hostname``. The hostname of the system (if ``hostname`` is an + executable in the user's PATH). + +^^^^^^^^^^^^^^^^^^^^^^^^^ +Environment-managed Views +^^^^^^^^^^^^^^^^^^^^^^^^^ + +Code for Spack Environments is in code review at the +moment. Documentation will be updated when the code is finalized. diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index 5ad55b6320..5c93285cbb 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -65,6 +65,7 @@ or refer to the full manual below. configuration config_yaml build_settings + environments mirrors module_file_support repositories