docs: initial documentation for environments

lib/spack/docs/environments.rst

@@ -0,0 +1,635 @@
+.. Copyright 2013-2019 Lawrence Livermore National Security, LLC and other
+   Spack Project Developers. See the top-level COPYRIGHT file for details.
+
+   SPDX-License-Identifier: (Apache-2.0 OR MIT)
+
+.. _environments:
+
+============
+Environments
+============
+
+An environment is used to group together a set of specs for the
+purpose of building, rebuilding and deploying in a coherent fashion.
+Environments provide a number of advantages over the a la carte
+approach of building and loading individual Spack modules:
+
+#. Environments separate the steps of (a) choosing what to
+   install, (b) concretizing, and (c) installing.  This allows
+   Environments to remain stable and repeatable, even if Spack packages
+   are upgraded: specs are only re-concretized when the user
+   explicitly asks for it.  It is even be possible to reliably
+   transport environments between different computers running
+   different versions of Spack!
+#. Environments allow several specs to be built at once; a more robust
+   solution than ad-hoc scripts making multiple calls to ``spack
+   install``.
+#. An Environment that is built as a whole can be loaded as a whole
+   into the user environment. An Environment can be build to maintain
+   a filesystem view of its packages, and the environment can load
+   that view into the user environment at activation time. Spack can
+   also generate a script to load all modules related to an
+   environment.
+
+Other packaging systems also provide environments that are similar in
+some ways to Spack environments; for example, `Conda environments
+<https://conda.io/docs/user-guide/tasks/manage-environments.html>`_ or
+`Python Virtual Environments
+<https://docs.python.org/3/tutorial/venv.html>`_.  Spack environments
+provide some distinctive features:
+
+#. A spec installed "in" an environment is no different from the same
+   spec installed anywhere else in Spack.  Environments are assembled
+   simply by collecting together a set of specs.
+#. Spack Environments may contain more than one spec of the same
+   package.
+
+Spack uses a "manifest and lock" model similar to `Bundler gemfiles
+<https://bundler.io/man/gemfile.5.html>`_ and other package
+managers. The user input file is named ``spack.yaml`` and the lock
+file is named ``spack.lock``
+
+------------------
+Using Environments
+------------------
+
+Here we follow a typical use case of creating, concretizing,
+installing and loading an environment.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Creating a named Environment
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+An environment is created by:
+
+.. code-block:: console
+
+   $ spack env create myenv
+
+Spack then creates the directory ``var/spack/environments/myenv``.
+
+.. note::
+
+   All named environments are stored in the ``var/spack/environments`` folder.
+
+In the ``var/spack/environments/myenv`` directory, Spack creates the
+file ``spack.yaml`` and the hidden directory ``.spack-env``.
+
+Spack stores metadata in the ``.spack-env`` directory. User
+interaction will occur through the ``spack.yaml`` file and the Spack
+commands that affect it. When the environment is concretized, Spack
+will create a file ``spack.lock`` with the concrete information for
+the environment.
+
+In addition to being the default location for the view associated with
+an Environment, the ``.spack-env`` directory also contains:
+
+  * ``repo/``: A repo consisting of the Spack packages used in this
+    environment.  This allows the environment to build the same, in
+    theory, even on different verions of Spack with different
+    packages!
+  * ``logs/``: A directory containing the build logs for the packages
+    in this Environment.
+
+Spack Environments can also be created from either a ``spack.yaml``
+manifest or a ``spack.lock`` lockfile. To create an Environment from a
+``spack.yaml`` manifest:
+
+.. code-block:: console
+
+   $ spack env create myenv spack.yaml
+
+To create an Environment from a ``spack.lock`` lockfile:
+
+.. code-block:: console
+
+   $ spack env create myenv spack.lock
+
+Either of these commands can also take a full path to the
+initialization file.
+
+A Spack Environment created from a ``spack.yaml`` manifest is
+guaranteed to have the same root specs as the original Environment,
+but may concretize differently. A Spack Environment created from a
+``spack.lock`` lockfile is guaranteed to have the same concrete specs
+as the original Environment. Either may obviously then differ as the
+user modifies it.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Activating an Environment
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+To activate an environment, use the following command:
+
+.. code-block:: console
+
+   $ spack env activate myenv
+
+By default, the ``spack env activate`` will load the view associated
+with the Environment into the user environment. The ``-v,
+--with-view`` argument ensures this behavior, and the ``-V,
+--without-vew`` argument activates the environment without changing
+the user environment variables.
+
+The ``-p`` option to the ``spack env activate`` command modifies the
+user's prompt to begin with the environment name in brackets.
+
+.. code-block:: console
+
+   $ spack env activate -p myenv
+   [myenv] $ ...
+
+To deactivate an environment, use the command:
+
+.. code-block:: console
+
+   $ spack env deactivate
+
+or the shortcut alias
+
+.. code-block:: console
+
+   $ despacktivate
+
+If the environment was activated with its view, deactivating the
+environment will remove the view from the user environment.
+
+^^^^^^^^^^^^^^^^^^^^^^
+Anonymous Environments
+^^^^^^^^^^^^^^^^^^^^^^
+
+Any directory can be treated as an environment if it contains a file
+``spack.yaml``. To load an anonymous environment, use:
+
+.. code-block:: console
+
+   $ spack env activate -d /path/to/directory
+
+Spack commands that are environment sensitive will also act on the
+environment any time the current working directory contains a
+``spack.yaml`` file. Changing working directory to a directory
+containing a ``spack.yaml`` file is equivalent to the command:
+
+.. code-block:: console
+
+   $ spack env activate -d /path/to/dir --without-view
+
+Anonymous specs can be created in place using the command:
+
+.. code-block:: console
+
+   $ spack env create -d .
+
+In this case Spack simply creates a spack.yaml file in the requested
+directory.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Environment Sensitive Commands
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Spack commands are environment sensitive. For example, the ``find``
+command shows only the specs in the active Environment if an
+Environment has been activated. Similarly, the ``install`` and
+``uninstall`` commands act on the active environment.
+
+.. code-block:: console
+
+  $ spack find
+  ==> 0 installed packages
+
+  $ spack install zlib@1.2.11
+  ==> Installing zlib
+  ==> Searching for binary cache of zlib
+  ==> Warning: No Spack mirrors are currently configured
+  ==> No binary for zlib found: installing from source
+  ==> Fetching http://zlib.net/fossils/zlib-1.2.11.tar.gz
+  ######################################################################## 100.0%
+  ==> Staging archive: /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur/zlib-1.2.11.tar.gz
+  ==> Created stage in /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
+ ==> No patches needed for zlib
+  ==> Building zlib [Package]
+  ==> Executing phase: 'install'
+  ==> Successfully installed zlib
+    Fetch: 0.36s.  Build: 11.58s.  Total: 11.93s.
+  [+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
+
+  $ spack env activate myenv
+
+  $ spack find
+  ==> In environment myenv
+  ==> No root specs
+
+  ==> 0 installed packages
+
+  $ spack install zlib@1.2.8
+  ==> Installing zlib
+  ==> Searching for binary cache of zlib
+  ==> Warning: No Spack mirrors are currently configured
+  ==> No binary for zlib found: installing from source
+  ==> Fetching http://zlib.net/fossils/zlib-1.2.8.tar.gz
+  ######################################################################## 100.0%
+  ==> Staging archive: /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7/zlib-1.2.8.tar.gz
+  ==> Created stage in /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
+  ==> No patches needed for zlib
+  ==> Building zlib [Package]
+  ==> Executing phase: 'install'
+  ==> Successfully installed zlib
+    Fetch: 0.26s.  Build: 2.08s.  Total: 2.35s.
+  [+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
+
+  $ spack find
+  ==> In environment myenv
+  ==> Root specs
+  zlib@1.2.8
+
+  ==> 1 installed package
+  -- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
+  zlib@1.2.8
+
+  $ despacktivate
+  $ spack find
+  ==> 2 installed packages
+  -- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
+  zlib@1.2.8  zlib@1.2.11
+
+Note that when we installed the abstract spec ``zlib@1.2.8``, it was
+presented as a root of the Environment. All explicitly installed
+packages will be listed as roots of the Environment.
+
+All of the Spack commands that act on the list of installed specs are
+Environment-sensitive in this way, including ``install``,
+``uninstall``, ``activate``, ``deactivate``, ``find``, ``extensions``,
+and more. In the :ref:`environment-configuration` section we will discuss
+Environment-sensitive commands further.
+
+^^^^^^^^^^^^^^^^^^^^^
+Adding Abstract Specs
+^^^^^^^^^^^^^^^^^^^^^
+
+An abstract spec is the user-specified spec before Spack has applied
+any defaults or dependency information.
+
+Users can add abstract specs to an Environment using the ``spack add``
+command. The most important component of an Environment is a list of
+abstract specs.
+
+Adding a spec adds to the manifest (the ``spack.yaml`` file) and to
+the roots of the Environment, but does not affect the concrete specs
+in the lockfile, nor does it install the spec.
+
+The ``spack add`` command is environment aware. It adds to the
+currently active environment. All environment aware commands can also
+be called using the ``spack -E`` flag to specify the environment.
+
+.. code-block:: console
+
+   $ spack activate myenv
+   $ spack add mpileaks
+
+or
+
+.. code-block:: console
+
+   $ spack -E myenv add python
+
+^^^^^^^^^^^^
+Concretizing
+^^^^^^^^^^^^
+
+Once some user specs have been added to an environment, they can be
+concretized.  The following command will concretize all user specs
+that have been added and not yet concretized:
+
+.. code-block:: console
+
+   [myenv]$ spack concretize
+
+This command will re-concretize all specs:
+
+.. code-block:: console
+
+   [myenv]$ spack concretize -f
+
+When the ``-f`` flag is not used to reconcretize all specs, Spack
+guarantees that already concretized specs are unchanged in the
+environment.
+
+The ``concretize`` command does not install any packages. For packages
+that have already been installed outside of the environment, the
+process of adding the spec and concretizing is identical to installing
+the spec assuming it concretizes to the exact spec that was installed
+ouside of the environment.
+
+The ``spack find`` command can show concretized specs separately from
+installed specs using the ``-c`` (``--concretized``) flag.
+
+.. code-block:: console
+
+  [myenv]$ spack add zlib
+  [myenv]$ spack concretize
+  [myenv]$ spack find -c
+  ==> In environment myenv
+  ==> Root specs
+  zlib
+
+  ==> Concretized roots
+  -- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
+  zlib@1.2.11
+
+  ==> 0 installed packages
+
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Installing an Environment
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+In addition to installing individual specs into an Environment, one
+can install the entire Environment at once using the command
+
+.. code-block:: console
+
+   [myenv]$ spack install
+
+If the Environment has been concretized, Spack will install the
+concretized specs. Otherwise, ``spack install`` will first concretize
+the Environment and then install the concretized specs.
+
+As it installs, ``spack install`` creates symbolic links in the
+``logs/`` directory in the Environment, allowing for easy inspection
+of build logs related to that environment. The ``spack install``
+command also stores a Spack repo containing the ``package.py`` file
+used at install time for each package in the ``repos/`` directory in
+the Environment.
+
+^^^^^^^
+Loading
+^^^^^^^
+
+Once an environment has been installed, the following creates a load script for it:
+
+.. code-block:: console
+
+   $ spack env myenv loads -r
+
+This creates a file called ``loads`` in the environment directory.
+Sourcing that file in Bash will make the environment available to the
+user; and can be included in ``.bashrc`` files, etc.  The ``loads``
+file may also be copied out of the environment, renamed, etc.
+
+----------
+spack.yaml
+----------
+
+Spack environments can be customized at finer granularity by editing
+the ``spack.yaml`` manifest file directly.
+
+.. _environment-configuration:
+
+^^^^^^^^^^^^^^^^^^^^^^^^
+Configuring Environments
+^^^^^^^^^^^^^^^^^^^^^^^^
+
+A variety of Spack behaviors are changed through Spack configuration
+files, covered in more detail in the :ref:`configuration`
+section.
+
+Spack Environments provide an additional level of configuration scope
+between the custom scope and the user scope discussed in the
+configuration documentation.
+
+There are two ways to include configuration information in a Spack Environment:
+
+#. Inline in the ``spack.yaml`` file
+
+#. Included in the ``spack.yaml`` file from another file.
+
+"""""""""""""""""""""
+Inline configurations
+"""""""""""""""""""""
+
+Inline Environment-scope configuration is done using the same yaml
+format as standard Spack configuration scopes, covered in the
+:ref:`configuration` section. Each section is contained under a
+top-level yaml object with it's name. For example, a ``spack.yaml``
+manifest file containing some package preference configuration (as in
+a ``packages.yaml`` file) could contain:
+
+.. code-block:: yaml
+
+   spack:
+     ...
+     packages:
+       all:
+         compilers: [intel]
+     ...
+
+This configuration sets the default compiler for all packages to
+``intel``.
+
+"""""""""""""""""""""""
+Included configurations
+"""""""""""""""""""""""
+
+Spack environments allow an ``include`` heading in their yaml
+schema. This heading pulls in external configuration files and applies
+them to the Environment.
+
+.. code-block:: yaml
+
+   spack:
+     include:
+     - relative/path/to/config.yaml
+     - /absolute/path/to/packages.yaml
+
+Environments can include files either with a relative or absolute
+paths. Inline configurations take precedence over included
+configurations, so you don't have to change shared configuration files
+to make small changes to an individual Environment. Included configs
+listed later will have higher precedence, as the included configs are
+applied in order.
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Manually Editing the Specs List
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The list of abstract/root specs in the Environment is maintained in
+the ``spack.yaml`` manifest under the heading ``specs``.
+
+.. code-block:: yaml
+
+   spack:
+       specs:
+         - ncview
+         - netcdf
+         - nco
+         - py-sphinx
+
+Appending to this list in the yaml is identical to using the ``spack
+add`` command from the command line. However, there is more power
+available from the yaml file.
+
+"""""""""""""
+Spec Matrices
+"""""""""""""
+
+Entries in the ``specs`` list can be individual abstract specs or a
+spec matrix.
+
+A spec matrix is a yaml object containing multiple lists of specs, and
+evaluates to the cross-product of those specs. Spec matrices also
+contain an ``excludes`` directive, which eliminates certain
+combinations from the evaluated result.
+
+The following two Environment manifests are identical:
+
+.. code-block:: yaml
+
+   spack:
+     specs:
+       - zlib %gcc@7.1.0
+       - zlib %gcc@4.9.3
+       - libelf %gcc@7.1.0
+       - libelf %gcc@4.9.3
+       - libdwarf %gcc@7.1.0
+       - cmake
+
+   spack:
+     specs:
+       - matrix:
+           - [zlib, libelf, libdwarf]
+           - ['%gcc@7.1.0', '%gcc@4.9.3']
+         exclude:
+           - libdwarf%gcc@4.9.3
+       - cmake
+
+Spec matrices can be used to install swaths of software across various
+toolchains.
+
+The concretization logic for spec matrices differs slightly from the
+rest of Spack. If a variant or dependency constraint from a matrix is
+invalid, Spack will reject the constraint and try again without
+it. For example, the following two Environment manifests will produce
+the same specs:
+
+.. code-block:: yaml
+
+   spack:
+     specs:
+       - matrix:
+           - [zlib, libelf, hdf5+mpi]
+           - [^mvapich2@2.2, ^openmpi@3.1.0]
+
+   spack:
+     specs:
+       - zlib
+       - libelf
+       - hdf5+mpi ^mvapich2@2.2
+       - hdf5+mpi ^openmpi@3.1.0
+
+This allows one to create toolchains out of combinations of
+constraints and apply them somewhat indiscriminantly to packages,
+without regard for the applicability of the constraint.
+
+""""""""""""""""""""
+Spec List References
+""""""""""""""""""""
+
+The last type of possible entry in the specs list is a reference.
+
+The Spack Environment manifest yaml schema contains an additional
+heading ``definitions``. Under definitions is an array of yaml
+objects. Each object has one or two fields. The one required field is
+a name, and the optional field is a ``when`` clause.
+
+The named field is a spec list. The spec list uses the same syntax as
+the ``specs`` entry. Each entry in the spec list can be a spec, a spec
+matrix, or a reference to an earlier named list. References are
+specified using the ``$`` sigil, and are "splatted" into place
+(i.e. the elements of the referent are at the same level as the
+elements listed separately). As an example, the following two manifest
+files are identical.
+
+.. code-block:: yaml
+
+   spack:
+     definitions:
+       - first: [libelf, libdwarf]
+       - compilers: ['%gcc', '^intel']
+       - second:
+           - $first
+           - matrix:
+               - [zlib]
+               - [$compilers]
+     specs:
+       - $second
+       - cmake
+
+   spack:
+     specs:
+       - libelf
+       - libdwarf
+       - zlib%gcc
+       - zlib%intel
+       - cmake
+
+.. note::
+
+   Named a spec list in the definitions section may only refer
+   to a named list defined above itself. Order matters.
+
+In short files like the example, it may be easier to simply list the
+included specs. However for more complicated examples involving many
+packages across many toolchains, separately factored lists make
+Environments substantially more manageable.
+
+Additionally, the ``-l`` option to the ``spack add`` command allows
+one to add to named lists in the definitions section of the manifest
+file directly from the command line.
+
+The ``when`` directive can be used to conditionally add specs to a
+named list. The ``when`` directive takes a string of python code
+referring to a restricted set of variables, and evaluates to a
+boolean. The specs listed are appended to the named list if the
+``when`` string evaluates to ``True``. In the following snippet, the
+named list ``compilers`` is ``['%gcc', '%clang', '%intel']`` on
+``x86_64`` systems and ``['%gcc', '%clang']`` on all other systems.
+
+.. code-block:: yaml
+
+   spack:
+     definitions:
+       - compilers: ['%gcc', '%clang']
+       - when: target == 'x86_64'
+         compilers: ['%intel']
+
+.. note::
+
+   Any definitions with the same named list with true ``when``
+   clauses (or absent ``when`` clauses) will be appended together
+
+The valid variables for a ``when`` clause are:
+
+#. ``platform``. The platform string of the default Spack
+   architecture on the system.
+
+#. ``os``. The os string of the default Spack architecture on
+   the system.
+
+#. ``target``. The target string of the default Spack
+   architecture on the system.
+
+#. ``architecture`` or ``arch``. The full string of the
+   default Spack architecture on the system.
+
+#. ``re``. The standard regex module in python.
+
+#. ``env``. The user environment (usually ``os.environ`` in python).
+
+#. ``hostname``. The hostname of the system (if ``hostname`` is an
+   executable in the user's PATH).
+
+^^^^^^^^^^^^^^^^^^^^^^^^^
+Environment-managed Views
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Code for Spack Environments is in code review at the
+moment. Documentation will be updated when the code is finalized.

lib/spack/docs/index.rst

@@ -65,6 +65,7 @@ or refer to the full manual below.
    configuration
    config_yaml
    build_settings
+   environments
    mirrors
    module_file_support
    repositories