hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Documentation Improvements for SC16 (#1676)

Browse source 
* Transferred pending changes from efischer/develop

* 1. Rewrite of "Getting Started": everything you need to set up Spack, even on old/ornery systems.  This is not a reference manual section; items covered here are covered more systematically elsewhere in the manual.  Some sections were moved here from elsewhere.

2. Beginning to write three methods of application developer support.  Two methods were moved from elsewhere.

* Edits...

* Moved sections in preparation for additional text to be added from old efischer/docs branch.

* Moved 2 more sections.

* Avoid accid

* Applied proofreading edits from @adamjstewart

* Fixed non-standard section characters.

* Moved section on profiling to the developer's guide.

* Still working on Spack workflows...

* Finished draft of packaging_guide.rst

* Renamed sample projects.

* Updates to docstrings

* Added documentation to resolve #638 (content taken from #846)

* Added section on resolving inconsistent run dependencies.  Addresses #645

* Showed how to build Python extensions only compatible with certain versions of Python.

* Added examples of getting the right behavior from depends_on().  See #1035

* Added section on Intel compilers and their GCC masquerading feature.  Addresses #638, #1687.

* Fixed formatting

* Added fixes to filesystem views.  Added a caveats section to ``spack setup``.

* Updated section on Intel compiler configuration because compiler flags currently do not work (see #1687)

* Defined trusted downloads, and updated text based on them. (See #1696)

* Added workflow to deal with buggy upstream software.  See #1683

* Added proper separation between Spack Docs vs. Reference Manual

* Renamed spack_workflows to workflows.  Resolves a conflict with the .gitignore file.

* Removed repeated section.

* Created new "Vendor Specific Compiler Configuration" section and organized existing Intel section into it.  Added new PGI and NAG sections; but they need to be expanded  / rewritten based on the existing text plus research through Spack issues on GitHub.

* Fixed text on `spack load --dependencies` to conform to reality.  See #1662

* Added patching as option for upstream bugfixes.

* Added section on using licensed compilers.

* Added section on non-downloadable tarballs.

* Wrote sections on NAG and PGI.  Arranged compilers in alphabetical order.

* Fix indent.

* Fixed typos.

* Clarified dependency types.

* Applied edits from Adam J. Stewart.  Spellchecked workflows and getting_started.

* Removed spurious header

* Fixed Sphinx errors

* Fixed erroneous symbol in docstring.

* Fix many typos and formatting problems.

* Spacing changes

* Added section on fixing Git problems.  See #1779

* Fixed signature of install() method.

* Addressed system packages in greater detail.  See #1794 #1795

* Fixed typos

* Fixed quotes

* Duplicate section on Spack profiling removed from configuration.rst.  It had earlier been moved to developer_guide.rst, where it fits better.

* Minor edits

- Tweak supported platform language.
- Various small changes to the new getting started guide.

* Fixed bug with quotes.
This commit is contained in:
Elizabeth Fischer 2016-10-05 16:00:27 -04:00 committed by Todd Gamblin
parent abc9412f23
commit 015e29efe1
11 changed files with 2766 additions and 874 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
.gitignore vendored
View file

@ -17,3 +17,5 @@
/TAGS /TAGS
/htmlcov /htmlcov
.coverage .coverage
#*
.#*

641
lib/spack/docs/basic_usage.rst
View file

@ -230,6 +230,54 @@ but you risk breaking other installed packages. In general, it is safer to
remove dependent packages *before* removing their dependencies or use the remove dependent packages *before* removing their dependencies or use the
``--dependents`` option. ``--dependents`` option.
.. _nondownloadable:
^^^^^^^^^^^^^^^^^^^^^^^^^
Non-Downloadable Tarballs
^^^^^^^^^^^^^^^^^^^^^^^^^
The tarballs for some packages cannot be automatically downloaded by
Spack. This could be for a number of reasons:
#. The author requires users to manually accept a license agreement
before downloading (``jdk`` and ``galahad``).
#. The software is proprietary and cannot be downloaded on the open
Internet.
To install these packages, one must create a mirror and manually add
the tarballs in question to it (see :ref:`mirrors`):
#. Create a directory for the mirror. You can create this directory
anywhere you like, it does not have to be inside ``~/.spack``:
.. code-block:: console
$ mkdir ~/.spack/manual_mirror
#. Register the mirror with Spack by creating ``~/.spack/mirrors.yaml``:
.. code-block:: yaml
mirrors:
manual: file:///home/me/.spack/manual_mirror
#. Put your tarballs in it. Tarballs should be named
``<package>/<package>-<version>.tar.gz``. For example:
.. code-block:: console
$ ls -l manual_mirror/galahad
-rw-------. 1 me me 11657206 Jun 21 19:25 galahad-2.60003.tar.gz
#. Install as usual:
.. code-block:: console
$ spack install galahad
------------------------- -------------------------
Seeing installed packages Seeing installed packages
------------------------- -------------------------
@ -382,175 +430,6 @@ with the 'debug' compile-time option enabled.
The full spec syntax is discussed in detail in :ref:`sec-specs`. The full spec syntax is discussed in detail in :ref:`sec-specs`.
.. _compiler-config:
----------------------
Compiler configuration
----------------------
Spack has the ability to build packages with multiple compilers and
compiler versions. Spack searches for compilers on your machine
automatically the first time it is run. It does this by inspecting
your ``PATH``.
.. _spack-compilers:
^^^^^^^^^^^^^^^^^^^
``spack compilers``
^^^^^^^^^^^^^^^^^^^
You can see which compilers spack has found by running ``spack
compilers`` or ``spack compiler list``:
.. code-block:: console
$ spack compilers
==> Available compilers
-- gcc ---------------------------------------------------------
gcc@4.9.0 gcc@4.8.0 gcc@4.7.0 gcc@4.6.2 gcc@4.4.7
gcc@4.8.2 gcc@4.7.1 gcc@4.6.3 gcc@4.6.1 gcc@4.1.2
-- intel -------------------------------------------------------
intel@15.0.0 intel@14.0.0 intel@13.0.0 intel@12.1.0 intel@10.0
intel@14.0.3 intel@13.1.1 intel@12.1.5 intel@12.0.4 intel@9.1
intel@14.0.2 intel@13.1.0 intel@12.1.3 intel@11.1
intel@14.0.1 intel@13.0.1 intel@12.1.2 intel@10.1
-- clang -------------------------------------------------------
clang@3.4 clang@3.3 clang@3.2 clang@3.1
-- pgi ---------------------------------------------------------
pgi@14.3-0 pgi@13.2-0 pgi@12.1-0 pgi@10.9-0 pgi@8.0-1
pgi@13.10-0 pgi@13.1-1 pgi@11.10-0 pgi@10.2-0 pgi@7.1-3
pgi@13.6-0 pgi@12.8-0 pgi@11.1-0 pgi@9.0-4 pgi@7.0-6
Any of these compilers can be used to build Spack packages. More on
how this is done is in :ref:`sec-specs`.
.. _spack-compiler-add:
^^^^^^^^^^^^^^^^^^^^^^
``spack compiler add``
^^^^^^^^^^^^^^^^^^^^^^
An alias for ``spack compiler find``.
.. _spack-compiler-find:
^^^^^^^^^^^^^^^^^^^^^^^
``spack compiler find``
^^^^^^^^^^^^^^^^^^^^^^^
If you do not see a compiler in this list, but you want to use it with
Spack, you can simply run ``spack compiler find`` with the path to
where the compiler is installed. For example:
.. code-block:: console
$ spack compiler find /usr/local/tools/ic-13.0.079
==> Added 1 new compiler to /Users/gamblin2/.spack/compilers.yaml
intel@13.0.079
Or you can run ``spack compiler find`` with no arguments to force
auto-detection. This is useful if you do not know where compilers are
installed, but you know that new compilers have been added to your
``PATH``. For example, using dotkit, you might do this:
.. code-block:: console
$ module load gcc-4.9.0
$ spack compiler find
==> Added 1 new compiler to /Users/gamblin2/.spack/compilers.yaml
gcc@4.9.0
This loads the environment module for gcc-4.9.0 to add it to
``PATH``, and then it adds the compiler to Spack.
.. _spack-compiler-info:
^^^^^^^^^^^^^^^^^^^^^^^
``spack compiler info``
^^^^^^^^^^^^^^^^^^^^^^^
If you want to see specifics on a particular compiler, you can run
``spack compiler info`` on it:
.. code-block:: console
$ spack compiler info intel@15
intel@15.0.0:
cc = /usr/local/bin/icc-15.0.090
cxx = /usr/local/bin/icpc-15.0.090
f77 = /usr/local/bin/ifort-15.0.090
fc = /usr/local/bin/ifort-15.0.090
modules = []
operating system = centos6
This shows which C, C++, and Fortran compilers were detected by Spack.
Notice also that we didn't have to be too specific about the
version. We just said ``intel@15``, and information about the only
matching Intel compiler was displayed.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Manual compiler configuration
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If auto-detection fails, you can manually configure a compiler by
editing your ``~/.spack/compilers.yaml`` file. You can do this by running
``spack config edit compilers``, which will open the file in your ``$EDITOR``.
Each compiler configuration in the file looks like this:
.. code-block:: yaml
compilers:
- compiler:
modules = []
operating_system: centos6
paths:
cc: /usr/local/bin/icc-15.0.024-beta
cxx: /usr/local/bin/icpc-15.0.024-beta
f77: /usr/local/bin/ifort-15.0.024-beta
fc: /usr/local/bin/ifort-15.0.024-beta
spec: intel@15.0.0:
For compilers, like ``clang``, that do not support Fortran, put
``None`` for ``f77`` and ``fc``:
.. code-block:: yaml
paths:
cc: /usr/bin/clang
cxx: /usr/bin/clang++
f77: None
fc: None
spec: clang@3.3svn:
Once you save the file, the configured compilers will show up in the
list displayed by ``spack compilers``.
You can also add compiler flags to manually configured compilers. The
valid flags are ``cflags``, ``cxxflags``, ``fflags``, ``cppflags``,
``ldflags``, and ``ldlibs``. For example:
.. code-block:: yaml
compilers:
- compiler:
modules = []
operating_system: OS
paths:
cc: /usr/local/bin/icc-15.0.024-beta
cxx: /usr/local/bin/icpc-15.0.024-beta
f77: /usr/local/bin/ifort-15.0.024-beta
fc: /usr/local/bin/ifort-15.0.024-beta
parameters:
cppflags: -O3 -fPIC
spec: intel@15.0.0:
These flags will be treated by spack as if they were enterred from
the command line each time this compiler is used. The compiler wrappers
then inject those flags into the compiler command. Compiler flags
enterred from the command line will be discussed in more detail in the
following section.
.. _sec-specs: .. _sec-specs:
-------------------- --------------------
@ -945,51 +824,17 @@ versions are now filtered out.
Integration with module systems Integration with module systems
------------------------------- -------------------------------
Spack provides some integration with `Environment Modules
<http://modules.sourceforge.net/>`_ to make it easier to use the
packages it installs. If your system does not already have
Environment Modules, see :ref:`InstallEnvironmentModules`.
.. note:: .. note::
Environment module support is currently experimental and should not Spack also supports `Dotkit
be considered a stable feature of Spack. In particular, the <https://computing.llnl.gov/?set=jobs&page=dotkit>`_, which is used
interface and/or generated module names may change in future by some systems. If you system does not already have a module
versions. system installed, you should use Environment Modules or LMod.
Spack provides some integration with
`Environment Modules <http://modules.sourceforge.net/>`__
and `Dotkit <https://computing.llnl.gov/?set=jobs&page=dotkit>`_ to make
it easier to use the packages it installed.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Installing Environment Modules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
In order to use Spack's generated environment modules, you must have
installed the *Environment Modules* package. On many Linux
distributions, this can be installed from the vendor's repository:
.. code-block:: sh
$ yum install environment-modules # (Fedora/RHEL/CentOS)
$ apt-get install environment-modules # (Ubuntu/Debian)
If your Linux distribution does not have
Environment Modules, you can get it with Spack:
.. code-block:: console
$ spack install environment-modules
In this case to activate it automatically you need to add the following two
lines to your ``.bashrc`` profile (or similar):
.. code-block:: sh
MODULES_HOME=`spack location -i environment-modules`
source ${MODULES_HOME}/Modules/init/bash
If you use a Unix shell other than ``bash``, modify the commands above
accordingly and source the appropriate file in
``${MODULES_HOME}/Modules/init/``.
.. TODO : Add a similar section on how to install dotkit ?
^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^
Spack and module systems Spack and module systems
@ -1196,9 +1041,36 @@ of module files:
"""Set up the compile and runtime environments for a package.""" """Set up the compile and runtime environments for a package."""
pass pass
""""""""""""""""" .. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
"""Set up the environment of packages that depend on this one"""
pass
As briefly stated in the comments, the first method lets you customize the
module file content for the package you are currently writing, the second
allows for modifications to your dependees module file. In both cases one
needs to fill ``run_env`` with the desired list of environment modifications.
""""""""""""""""""""""""""""""""""""""""""""""""
Example : ``builtin/packages/python/package.py``
""""""""""""""""""""""""""""""""""""""""""""""""
The ``python`` package that comes with the ``builtin`` Spack repository
overrides ``setup_dependent_environment`` in the following way:
.. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, extension_spec):
if extension_spec.package.extends(self.spec):
run_env.prepend_path('PYTHONPATH', os.path.join(extension_spec.prefix, self.site_packages_dir))
to insert the appropriate ``PYTHONPATH`` modifications in the module
files of python packages.
^^^^^^^^^^^^^^^^^
Recursive Modules Recursive Modules
""""""""""""""""" ^^^^^^^^^^^^^^^^^
In some cases, it is desirable to load not just a module, but also all In some cases, it is desirable to load not just a module, but also all
the modules it depends on. This is not required for most modules the modules it depends on. This is not required for most modules
@ -1207,18 +1079,30 @@ packages use RPATH to find their dependencies: this can be true in
particular for Python extensions, which are currently *not* built with particular for Python extensions, which are currently *not* built with
RPATH. RPATH.
Modules may be loaded recursively with the ``load`` command's Scripts to load modules recursively may be made with the command:
``--dependencies`` or ``-r`` argument:
.. code-block:: console .. code-block:: console
$ spack load --dependencies <spec> ... $ spack module loads --dependencies <spec>
More than one spec may be placed on the command line here. An equivalent alternative is:
""""""""""""""""""""""""""""""""" .. code-block :: console
$ source <( spack module loads --dependencies <spec> )
.. warning::
The ``spack load`` command does not currently accept the
``--dependencies`` flag. Use ``spack module loads`` instead, for
now.
.. See #1662
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Module Commands for Shell Scripts Module Commands for Shell Scripts
""""""""""""""""""""""""""""""""" ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Although Spack is flexible, the ``module`` command is much faster. Although Spack is flexible, the ``module`` command is much faster.
This could become an issue when emitting a series of ``spack load`` This could become an issue when emitting a series of ``spack load``
@ -1228,7 +1112,7 @@ cut-and-pasted into a shell script. For example:
.. code-block:: console .. code-block:: console
$ spack module find tcl --dependencies --shell py-numpy git $ spack module loads --dependencies py-numpy git
# bzip2@1.0.6%gcc@4.9.3=linux-x86_64 # bzip2@1.0.6%gcc@4.9.3=linux-x86_64
module load bzip2-1.0.6-gcc-4.9.3-ktnrhkrmbbtlvnagfatrarzjojmkvzsx module load bzip2-1.0.6-gcc-4.9.3-ktnrhkrmbbtlvnagfatrarzjojmkvzsx
# ncurses@6.0%gcc@4.9.3=linux-x86_64 # ncurses@6.0%gcc@4.9.3=linux-x86_64
@ -1261,42 +1145,31 @@ cut-and-pasted into a shell script. For example:
module load git-2.8.0-rc2-gcc-4.9.3-3bib4hqtnv5xjjoq5ugt3inblt4xrgkd module load git-2.8.0-rc2-gcc-4.9.3-3bib4hqtnv5xjjoq5ugt3inblt4xrgkd
The script may be further edited by removing unnecessary modules. The script may be further edited by removing unnecessary modules.
This script may be directly executed in bash via:
.. code-block:: sh
source < (spack module find tcl --dependencies --shell py-numpy git) ^^^^^^^^^^^^^^^
Module Prefixes
^^^^^^^^^^^^^^^
^^^^^^^^^^^^^^^^^^^^^^^^^ On some systems, modules are automatically prefixed with a certain
Regenerating Module files string; ``spack module loads`` needs to know about that prefix when it
^^^^^^^^^^^^^^^^^^^^^^^^^ issues ``module load`` commands. Add the ``--prefix`` option to your
``spack module loads`` commands if this is necessary.
.. code-block:: python For example, consider the following on one system:
def setup_dependent_environment(self, spack_env, run_env, dependent_spec): ..code-block:: console
"""Set up the environment of packages that depend on this one"""
pass
As briefly stated in the comments, the first method lets you customize the $ module avail
module file content for the package you are currently writing, the second linux-SuSE11-x86_64/antlr-2.7.7-gcc-5.3.0-bdpl46y
allows for modifications to your dependees module file. In both cases one
needs to fill ``run_env`` with the desired list of environment modifications.
"""""""""""""""""""""""""""""""""""""""""""""""" $ spack module loads antlr # WRONG!
Example : ``builtin/packages/python/package.py`` # antlr@2.7.7%gcc@5.3.0~csharp+cxx~java~python arch=linux-SuSE11-x86_64
"""""""""""""""""""""""""""""""""""""""""""""""" module load antlr-2.7.7-gcc-5.3.0-bdpl46y
The ``python`` package that comes with the ``builtin`` Spack repository $ spack module loads --prefix linux-SuSE11-x86_64/ antlr
overrides ``setup_dependent_environment`` in the following way: # antlr@2.7.7%gcc@5.3.0~csharp+cxx~java~python arch=linux-SuSE11-x86_64
module load linux-SuSE11-x86_64/antlr-2.7.7-gcc-5.3.0-bdpl46y
.. code-block:: python
def setup_dependent_environment(self, spack_env, run_env, extension_spec):
if extension_spec.package.extends(self.spec):
run_env.prepend_path('PYTHONPATH', os.path.join(extension_spec.prefix, self.site_packages_dir))
to insert the appropriate ``PYTHONPATH`` modifications in the module
files of python packages.
^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^
Configuration files Configuration files
@ -1461,23 +1334,14 @@ load two or more versions of the same software at the same time.
The ``conflict`` option is ``tcl`` specific The ``conflict`` option is ``tcl`` specific
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
Regenerating module files Regenerating Module files
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
Sometimes you may need to regenerate the modules files. For example, Module and dotkit files are generated when packages are installed, and
if newer, fancier module support is added to Spack at some later date, are placed in the directory ``share/spack/modules`` under the Spack
you may want to regenerate all the modules to take advantage of these root. The command ``spack refresh`` will regenerate them all without
new features. re-building the packages; for example, if module format or options
have changed:
.. _spack-module:
""""""""""""""""""""""""
``spack module refresh``
""""""""""""""""""""""""
Running ``spack module refresh`` will remove the
``share/spack/modules`` and ``share/spack/dotkit`` directories, then
regenerate all module and dotkit files from scratch:
.. code-block:: console .. code-block:: console
@ -1485,117 +1349,6 @@ regenerate all module and dotkit files from scratch:
==> Regenerating tcl module files. ==> Regenerating tcl module files.
==> Regenerating dotkit module files. ==> Regenerating dotkit module files.
----------------
Filesystem Views
----------------
.. Maybe this is not the right location for this documentation.
The Spack installation area allows for many package installation trees
to coexist and gives the user choices as to what versions and variants
of packages to use. To use them, the user must rely on a way to
aggregate a subset of those packages. The section on Environment
Modules gives one good way to do that which relies on setting various
environment variables. An alternative way to aggregate is through
**filesystem views**.
A filesystem view is a single directory tree which is the union of the
directory hierarchies of the individual package installation trees
that have been included. The files of the view's installed packages
are brought into the view by symbolic or hard links back to their
location in the original Spack installation area. As the view is
formed, any clashes due to a file having the exact same path in its
package installation tree are handled in a first-come-first-served
basis and a warning is printed. Packages and their dependencies can
be both added and removed. During removal, empty directories will be
purged. These operations can be limited to pertain to just the
packages listed by the user or to exclude specific dependencies and
they allow for software installed outside of Spack to coexist inside
the filesystem view tree.
By its nature, a filesystem view represents a particular choice of one
set of packages among all the versions and variants that are available
in the Spack installation area. It is thus equivalent to the
directory hiearchy that might exist under ``/usr/local``. While this
limits a view to including only one version/variant of any package, it
provides the benefits of having a simpler and traditional layout which
may be used without any particular knowledge that its packages were
built by Spack.
Views can be used for a variety of purposes including:
* A central installation in a traditional layout, eg ``/usr/local`` maintained over time by the sysadmin.
* A self-contained installation area which may for the basis of a top-level atomic versioning scheme, eg ``/opt/pro`` vs ``/opt/dev``.
* Providing an atomic and monolithic binary distribution, eg for delivery as a single tarball.
* Producing ephemeral testing or developing environments.
^^^^^^^^^^^^^^^^^^^^^^
Using Filesystem Views
^^^^^^^^^^^^^^^^^^^^^^
A filesystem view is created and packages are linked in by the ``spack
view`` command's ``symlink`` and ``hardlink`` sub-commands. The
``spack view remove`` command can be used to unlink some or all of the
filesystem view.
The following example creates a filesystem view based
on an installed ``cmake`` package and then removes from the view the
files in the ``cmake`` package while retaining its dependencies.
.. code-block:: console
$ spack view --verbose symlink myview cmake@3.5.2
==> Linking package: "ncurses"
==> Linking package: "zlib"
==> Linking package: "openssl"
==> Linking package: "cmake"
$ ls myview/
bin doc etc include lib share
$ ls myview/bin/
captoinfo clear cpack ctest infotocap openssl tabs toe tset
ccmake cmake c_rehash infocmp ncurses6-config reset tic tput
$ spack view --verbose --dependencies false rm myview cmake@3.5.2
==> Removing package: "cmake"
$ ls myview/bin/
captoinfo c_rehash infotocap openssl tabs toe tset
clear infocmp ncurses6-config reset tic tput
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Limitations of Filesystem Views
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
This section describes some limitations that should be considered in
using filesystems views.
Filesystem views are merely organizational. The binary executable
programs, shared libraries and other build products found in a view
are mere links into the "real" Spack installation area. If a view is
built with symbolic links it requires the Spack-installed package to
be kept in place. Building a view with hardlinks removes this
requirement but any internal paths (eg, rpath or ``#!`` interpreter
specifications) will still require the Spack-installed package files
to be in place.
.. FIXME: reference the relocation work of Hegner and Gartung.
As described above, when a view is built only a single instance of a
file may exist in the unified filesystem tree. If more than one
package provides a file at the same path (relative to its own root)
then it is the first package added to the view that "wins". A warning
is printed and it is up to the user to determine if the conflict
matters.
It is up to the user to assure a consistent view is produced. In
particular if the user excludes packages, limits the following of
dependencies or removes packages the view may become inconsistent. In
particular, if two packages require the same sub-tree of dependencies,
removing one package (recursively) will remove its dependencies and
leave the other package broken.
.. _extensions: .. _extensions:
--------------------------- ---------------------------
@ -1864,144 +1617,6 @@ This issue typically manifests with the error below:
A nicer error message is TBD in future versions of Spack. A nicer error message is TBD in future versions of Spack.
.. _cray-support:
-------------
Spack on Cray
-------------
Spack differs slightly when used on a Cray system. The architecture spec
can differentiate between the front-end and back-end processor and operating system.
For example, on Edison at NERSC, the back-end target processor
is "Ivy Bridge", so you can specify to use the back-end this way:
.. code-block:: console
$ spack install zlib target=ivybridge
You can also use the operating system to build against the back-end:
.. code-block:: console
$ spack install zlib os=CNL10
Notice that the name includes both the operating system name and the major
version number concatenated together.
Alternatively, if you want to build something for the front-end,
you can specify the front-end target processor. The processor for a login node
on Edison is "Sandy bridge" so we specify on the command line like so:
.. code-block:: console
$ spack install zlib target=sandybridge
And the front-end operating system is:
.. code-block:: console
$ spack install zlib os=SuSE11
^^^^^^^^^^^^^^^^^^^^^^^
Cray compiler detection
^^^^^^^^^^^^^^^^^^^^^^^
Spack can detect compilers using two methods. For the front-end, we treat
everything the same. The difference lies in back-end compiler detection.
Back-end compiler detection is made via the Tcl module avail command.
Once it detects the compiler it writes the appropriate PrgEnv and compiler
module name to compilers.yaml and sets the paths to each compiler with Cray\'s
compiler wrapper names (i.e. cc, CC, ftn). During build time, Spack will load
the correct PrgEnv and compiler module and will call appropriate wrapper.
The compilers.yaml config file will also differ. There is a
modules section that is filled with the compiler's Programming Environment
and module name. On other systems, this field is empty []:
.. code-block:: yaml
- compiler:
modules:
- PrgEnv-intel
- intel/15.0.109
As mentioned earlier, the compiler paths will look different on a Cray system.
Since most compilers are invoked using cc, CC and ftn, the paths for each
compiler are replaced with their respective Cray compiler wrapper names:
.. code-block:: yaml
paths:
cc: cc
cxx: CC
f77: ftn
fc: ftn
As opposed to an explicit path to the compiler executable. This allows Spack
to call the Cray compiler wrappers during build time.
For more on compiler configuration, check out :ref:`compiler-config`.
Spack sets the default Cray link type to dynamic, to better match other
other platforms. Individual packages can enable static linking (which is the
default outside of Spack on cray systems) using the ``-static`` flag.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Setting defaults and using Cray modules
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
If you want to use default compilers for each PrgEnv and also be able
to load cray external modules, you will need to set up a ``packages.yaml``.
Here's an example of an external configuration for cray modules:
.. code-block:: yaml
packages:
mpi:
modules:
mpich@7.3.1%gcc@5.2.0 arch=cray_xc-haswell-CNL10: cray-mpich
mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-haswell-CNL10: cray-mpich
This tells Spack that for whatever package that depends on mpi, load the
cray-mpich module into the environment. You can then be able to use whatever
environment variables, libraries, etc, that are brought into the environment
via module load.
You can set the default compiler that Spack can use for each compiler type.
If you want to use the Cray defaults, then set them under ``all:`` in packages.yaml.
In the compiler field, set the compiler specs in your order of preference.
Whenever you build with that compiler type, Spack will concretize to that version.
Here is an example of a full packages.yaml used at NERSC
.. code-block:: yaml
packages:
mpi:
modules:
mpich@7.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge: cray-mpich
mpich@7.3.1%intel@16.0.0.109 arch=cray_xc-SuSE11-ivybridge: cray-mpich
buildable: False
netcdf:
modules:
netcdf@4.3.3.1%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge: cray-netcdf
netcdf@4.3.3.1%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge: cray-netcdf
buildable: False
hdf5:
modules:
hdf5@1.8.14%gcc@5.2.0 arch=cray_xc-CNL10-ivybridge: cray-hdf5
hdf5@1.8.14%intel@16.0.0.109 arch=cray_xc-CNL10-ivybridge: cray-hdf5
buildable: False
all:
compiler: [gcc@5.2.0, intel@16.0.0.109]
Here we tell spack that whenever we want to build with gcc use version 5.2.0 or
if we want to build with intel compilers, use version 16.0.0.109. We add a spec
for each compiler type for each cray modules. This ensures that for each
compiler on our system we can use that external module.
For more on external packages check out the section :ref:`sec-external_packages`.
------------ ------------
Getting Help Getting Help

181
lib/spack/docs/case_studies.rst
View file

@ -1,181 +0,0 @@
=======================================
Using Spack for CMake-based Development
=======================================
These are instructions on how to use Spack to aid in the development
of a CMake-based project. Spack is used to help find the dependencies
for the project, configure it at development time, and then package it
it in a way that others can install. Using Spack for CMake-based
development consists of three parts:
#. Setting up the CMake build in your software
#. Writing the Spack Package
#. Using it from Spack.
--------------------------
Setting Up the CMake Build
--------------------------
You should follow standard CMake conventions in setting up your
software, your CMake build should NOT depend on or require Spack to
build. See here for an example:
https://github.com/citibeth/icebin
Note that there's one exception here to the rule I mentioned above.
In ``CMakeLists.txt``, I have the following line:
.. code-block:: none
include_directories($ENV{CMAKE_TRANSITIVE_INCLUDE_PATH})
This is a hook into Spack, and it ensures that all transitive
dependencies are included in the include path. It's not needed if
everything is in one tree, but it is (sometimes) in the Spack world;
when running without Spack, it has no effect.
Note that this "feature" is controversial, could break with future
versions of GNU ld, and probably not the best to use. The best
practice is that you make sure that anything you #include is listed as
a dependency in your CMakeLists.txt.
To be more specific: if you #inlcude something from package A and an
installed HEADER FILE in A #includes something from package B, then
you should also list B as a dependency in your CMake build. If you
depend on A but header files exported by A do NOT #include things from
B, then you do NOT need to list B as a dependency --- even if linking
to A links in libB.so as well.
I also recommend that you set up your CMake build to use RPATHs
correctly. Not only is this a good idea and nice, but it also ensures
that your package will build the same with or without ``spack
install``.
-------------------------
Writing the Spack Package
-------------------------
Now that you have a CMake build, you want to tell Spack how to
configure it. This is done by writing a Spack package for your
software. See here for example:
https://github.com/citibeth/spack/blob/efischer/develop/var/spack/repos/builtin/packages/icebin/package.py
You need to subclass ``CMakePackage``, as is done in this example.
This enables advanced features of Spack for helping you in configuring
your software (keep reading...). Instead of an ``install()`` method
used when subclassing ``Package``, you write ``configure_args()``.
See here for more info on how this works:
https://github.com/LLNL/spack/pull/543/files
NOTE: if your software is not publicly available, you do not need to
set the URL or version. Or you can set up bogus URLs and
versions... whatever causes Spack to not crash.
-------------------
Using it from Spack
-------------------
Now that you have a Spack package, you can get Spack to setup your
CMake project for you. Use the following to setup, configure and
build your project:
.. code-block:: console
$ cd myproject
$ spack spconfig myproject@local
$ mkdir build; cd build
$ ../spconfig.py ..
$ make
$ make install
Everything here should look pretty familiar here from a CMake
perspective, except that ``spack spconfig`` creates the file
``spconfig.py``, which calls CMake with arguments appropriate for your
Spack configuration. Think of it as the equivalent to running a bunch
of ``spack location -i`` commands. You will run ``spconfig.py``
instead of running CMake directly.
If your project is publicly available (eg on GitHub), then you can
ALSO use this setup to "just install" a release version without going
through the manual configuration/build step. Just do:
#. Put tag(s) on the version(s) in your GitHub repo you want to be release versions.
#. Set the ``url`` in your ``package.py`` to download a tarball for
the appropriate version. (GitHub will give you a tarball for any
version in the repo, if you tickle it the right way). For example:
https://github.com/citibeth/icebin/tarball/v0.1.0
Set up versions as appropriate in your ``package.py``. (Manually
download the tarball and run ``md5sum`` to determine the
appropriate checksum for it).
#. Now you should be able to say ``spack install myproject@version``
and things "just work."
NOTE... in order to use the features outlined in this post, you
currently need to use the following branch of Spack:
https://github.com/citibeth/spack/tree/efischer/develop
There is a pull request open on this branch (
https://github.com/LLNL/spack/pull/543 ) and we are working to get it
integrated into the main ``develop`` branch.
------------------------
Activating your Software
------------------------
Once you've built your software, you will want to load it up. You can
use ``spack load mypackage@local`` for that in your ``.bashrc``, but
that is slow. Try stuff like the following instead:
The following command will load the Spack-installed packages needed
for basic Python use of IceBin:
.. code-block:: console
$ module load `spack module find tcl icebin netcdf cmake@3.5.1`
$ module load `spack module find --dependencies tcl py-basemap py-giss`
You can speed up shell startup by turning these into ``module load`` commands.
#. Cut-n-paste the script ``make_spackenv``:
.. code-block:: sh
#!/bin/sh
#
# Generate commands to load the Spack environment
SPACKENV=$HOME/spackenv.sh
spack module find --shell tcl git icebin@local ibmisc netcdf cmake@3.5.1 > $SPACKENV
spack module find --dependencies --shell tcl py-basemap py-giss >> $SPACKENV
#. Add the following to your ``.bashrc`` file:
.. code-block:: sh
source $HOME/spackenv.sh
# Preferentially use your checked-out Python source
export PYTHONPATH=$HOME/icebin/pylib:$PYTHONPATH
#. Run ``sh make_spackenv`` whenever your Spack installation changes (including right now).
-----------
Giving Back
-----------
If your software is publicly available, you should submit the
``package.py`` for it as a pull request to the main Spack GitHub
project. This will ensure that anyone can install your software
(almost) painlessly with a simple ``spack install`` command. See here
for how that has turned into detailed instructions that have
successfully enabled collaborators to install complex software:
https://github.com/citibeth/icebin/blob/develop/README.rst

83
lib/spack/docs/configuration.rst
View file

@ -132,6 +132,65 @@ The ``buildable`` does not need to be paired with external packages.
It could also be used alone to forbid packages that may be It could also be used alone to forbid packages that may be
buggy or otherwise undesirable. buggy or otherwise undesirable.
.. _system-packages:
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
False Paths for System Packages
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sometimes, the externally-installed package one wishes to use with
Spack comes with the Operating System and is installed in a standard
place --- ``/usr``, for example. Many other packages are there as
well. If Spack adds it to build paths, then some packages might
pick up dependencies from ``/usr`` than the intended Spack version.
In order to avoid this problem, it is advisable to specify a fake path
in ``packages.yaml``, thereby preventing Spack from adding the real
path to compiler command lines. This will work becuase compilers
normally search standard system paths, even if they are not on the
command line. For example:
.. code-block:: yaml
packages:
# Recommended for security reasons
# Do not install OpenSSL as non-root user.
openssl:
paths:
openssl@system: /false/path
version: [system]
buildable: False
^^^^^^^^^^^^^^^^^^^^^^^^^^
Extracting System Packages
^^^^^^^^^^^^^^^^^^^^^^^^^^
In some cases, using false paths for system packages will not work.
Some builds need to run binaries out of their dependencies, not just
access their libraries: the build needs to know the real location of
the system package.
In this case, one can create a Spack-like single-package tree by
creating symlinks to the files related to just that package.
Depending on the OS, it is possible to obtain a list of the files in a
single OS-installed package. For example, on RedHat / Fedora:
.. code-block:: console
$ repoquery --list openssl-devel
...
/usr/lib/libcrypto.so
/usr/lib/libssl.so
/usr/lib/pkgconfig/libcrypto.pc
/usr/lib/pkgconfig/libssl.pc
/usr/lib/pkgconfig/openssl.pc
...
Spack currently does not provide an automated way to create a symlink
tree to these files.
.. _concretization-preferences: .. _concretization-preferences:
-------------------------- --------------------------
@ -190,27 +249,3 @@ The syntax for the ``provider`` section differs slightly from other
concretization rules. A provider lists a value that packages may concretization rules. A provider lists a value that packages may
``depend_on`` (e.g, mpi) and a list of rules for fulfilling that ``depend_on`` (e.g, mpi) and a list of rules for fulfilling that
dependency. dependency.
---------
Profiling
---------
Spack has some limited built-in support for profiling, and can report
statistics using standard Python timing tools. To use this feature,
supply ``-p`` to Spack on the command line, before any subcommands.
.. _spack-p:
^^^^^^^^^^^^^^^^^^^
``spack --profile``
^^^^^^^^^^^^^^^^^^^
``spack --profile`` output looks like this:
.. command-output:: spack --profile graph --deptype=nobuild dyninst
:ellipsis: 25
The bottom of the output shows the top most time consuming functions,
slowest on top. The profiling support is from Python's built-in tool,
`cProfile
<https://docs.python.org/2/library/profile.html#module-cProfile>`_.

24
lib/spack/docs/developer_guide.rst
View file

@ -324,3 +324,27 @@ Developer commands
^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
``spack test`` ``spack test``
^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^
---------
Profiling
---------
Spack has some limited built-in support for profiling, and can report
statistics using standard Python timing tools. To use this feature,
supply ``-p`` to Spack on the command line, before any subcommands.
.. _spack-p:
^^^^^^^^^^^^^^^^^^^
``spack --profile``
^^^^^^^^^^^^^^^^^^^
``spack --profile`` output looks like this:
.. command-output:: spack --profile graph dyninst
:ellipsis: 25
The bottom of the output shows the top most time consuming functions,
slowest on top. The profiling support is from Python's built-in tool,
`cProfile
<https://docs.python.org/2/library/profile.html#module-cProfile>`_.

1080
lib/spack/docs/getting_started.rst
View file

File diff suppressed because it is too large Load diff

25
lib/spack/docs/index.rst
View file

@ -37,25 +37,34 @@ package:
If you're new to spack and want to start using it, see :doc:`getting_started`, If you're new to spack and want to start using it, see :doc:`getting_started`,
or refer to the full manual below. or refer to the full manual below.
-----------------
Table of Contents
-----------------
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
:caption: Tutorials
features features
getting_started getting_started
basic_usage basic_usage
packaging_guide workflows
mirrors
.. toctree::
:maxdepth: 2
:caption: Reference Manual
configuration configuration
developer_guide mirrors
case_studies
command_index
package_list package_list
command_index
.. toctree::
:maxdepth: 2
:caption: Contributing to Spack
packaging_guide
developer_guide
API Docs <spack> API Docs <spack>
================== ==================
Indices and tables Indices and tables
================== ==================

306
lib/spack/docs/packaging_guide.rst
View file

@ -373,6 +373,107 @@ some examples:
In general, you won't have to remember this naming convention because In general, you won't have to remember this naming convention because
:ref:`spack-create` and :ref:`spack-edit` handle the details for you. :ref:`spack-create` and :ref:`spack-edit` handle the details for you.
-----------------
Trusted Downloads
-----------------
Spack verifies that the source code it downloads is not corrupted or
compromised; or at least, that it is the same version the author of
the Spack package saw when the package was created. If Spack uses a
download method it can verify, we say the download method is
*trusted*. Trust is important for *all downloads*: Spack
has no control over the security of the various sites from which it
downloads source code, and can never assume that any particular site
hasn't been compromised.
Trust is established in different ways for different download methods.
For the most common download method --- a single-file tarball --- the
tarball is checksummed. Git downloads using ``commit=`` are trusted
implicitly, as long as a hash is specified.
Spack also provides untrusted download methods: tarball URLs may be
supplied without a checksum, or Git downloads may specify a branch or
tag instead of a hash. If the user does not control or trust the
source of an untrusted download, it is a security risk. Unless otherwise
specified by the user for special cases, Spack should by default use
*only* trusted download methods.
Unfortunately, Spack does not currently provide that guarantee. It
does provide the following mechanisms for safety:
#. By default, Spack will only install a tarball package if it has a
checksum and that checksum matches. You can override this with
``spack install --no-checksum``.
#. Numeric versions are almost always tarball downloads, whereas
non-numeric versions not named ``develop`` frequently download
untrusted branches or tags from a version control system. As long
as a package has at least one numeric version, and no non-numeric
version named ``develop``, Spack will prefer it over any
non-numeric versions.
^^^^^^^^^
Checksums
^^^^^^^^^
For tarball downloads, Spack can currently support checksums using the
MD5, SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 algorithms. It
determines the algorithm to use based on the hash length.
-----------------------
Package Version Numbers
-----------------------
Most Spack versions are numeric, a tuple of integers; for example,
``apex@0.1``, ``ferret@6.96`` or ``py-netcdf@1.2.3.1``. Spack knows
how to compare and sort numeric versions.
Some Spack versions involve slight extensions of numeric syntax; for
example, ``py-sphinx-rtd-theme@0.1.10a0``. In this case, numbers are
always considered to be "newer" than letters. This is for consistency
with `RPM <https://bugzilla.redhat.com/show_bug.cgi?id=50977>`.
Spack versions may also be arbitrary non-numeric strings; any string
here will suffice; for example, ``@develop``, ``@master``, ``@local``.
The following rules determine the sort order of numeric
vs. non-numeric versions:
#. The non-numeric versions ``@develop`` is considered greatest (newest).
#. Numeric versions are all less than ``@develop`` version, and are
sorted numerically.
#. All other non-numeric versions are less than numeric versions, and
are sorted alphabetically.
The logic behind this sort order is two-fold:
#. Non-numeric versions are usually used for special cases while
developing or debugging a piece of software. Keeping most of them
less than numeric versions ensures that Spack choose numeric
versions by default whenever possible.
#. The most-recent development version of a package will usually be
newer than any released numeric versions. This allows the
``develop`` version to satisfy dependencies like ``depends_on(abc,
when="@x.y.z:")``
^^^^^^^^^^^^^
Date Versions
^^^^^^^^^^^^^
If you wish to use dates as versions, it is best to use the format
``@date-yyyy-mm-dd``. This will ensure they sort in the correct
order. If you want your date versions to be numeric (assuming they
don't conflict with other numeric versions), you can use just
``yyyy.mm.dd``.
Alternately, you might use a hybrid release-version / date scheme.
For example, ``@1.3.2016.08.31`` would mean the version from the
``1.3`` branch, as of August 31, 2016.
------------------- -------------------
Adding new versions Adding new versions
------------------- -------------------
@ -459,19 +560,6 @@ it executable, then runs it with some arguments.
installer = Executable(self.stage.archive_file) installer = Executable(self.stage.archive_file)
installer('--prefix=%s' % prefix, 'arg1', 'arg2', 'etc.') installer('--prefix=%s' % prefix, 'arg1', 'arg2', 'etc.')
^^^^^^^^^
Checksums
^^^^^^^^^
Spack uses a checksum to ensure that the downloaded package version is
not corrupted or compromised. This is especially important when
fetching from insecure sources, like unencrypted http. By default, a
package will *not* be installed if it doesn't pass a checksum test
(though you can override this with ``spack install --no-checksum``).
Spack can currently support checksums using the MD5, SHA-1, SHA-224,
SHA-256, SHA-384, and SHA-512 algorithms. It determines the algorithm
to use based on the hash length.
^^^^^^^^^^^^^ ^^^^^^^^^^^^^
``spack md5`` ``spack md5``
@ -584,39 +672,6 @@ call to your package with parameters indicating the repository URL and
any branch, tag, or revision to fetch. See below for the parameters any branch, tag, or revision to fetch. See below for the parameters
you'll need for each VCS system. you'll need for each VCS system.
^^^^^^^^^^^^^^^^^^^^^^^^^
Repositories and versions
^^^^^^^^^^^^^^^^^^^^^^^^^
The package author is responsible for coming up with a sensible name
for each version to be fetched from a repository. For example, if
you're fetching from a tag like ``v1.0``, you might call that ``1.0``.
If you're fetching a nameless git commit or an older subversion
revision, you might give the commit an intuitive name, like ``develop``
for a development version, or ``some-fancy-new-feature`` if you want
to be more specific.
In general, it's recommended to fetch tags or particular
commits/revisions, NOT branches or the repository mainline, as
branches move forward over time and you aren't guaranteed to get the
same thing every time you fetch a particular version. Life isn't
always simple, though, so this is not strictly enforced.
When fetching from from the branch corresponding to the development version
(often called ``master``, ``trunk``, or ``dev``), it is recommended to
call this version ``develop``. Spack has special treatment for this version so
that ``@develop`` will satisfy dependencies like
``depends_on(abc, when="@x.y.z:")``. In other words, ``@develop`` is
greater than any other version. The rationale is that certain features or
options first appear in the development branch. Therefore if a package author
wants to keep the package on the bleeding edge and provide support for new
features, it is advised to use ``develop`` for such a version which will
greatly simplify writing dependencies and version-related conditionals.
In some future release, Spack may support extrapolating repository
versions as it does for tarball URLs, but currently this is not
supported.
.. _git-fetch: .. _git-fetch:
^^^ ^^^
@ -642,8 +697,7 @@ Default branch
... ...
version('develop', git='https://github.com/example-project/example.git') version('develop', git='https://github.com/example-project/example.git')
This is not recommended, as the contents of the default branch This download method is untrusted, and is not recommended.
change over time.
Tags Tags
To fetch from a particular tag, use the ``tag`` parameter along with To fetch from a particular tag, use the ``tag`` parameter along with
@ -654,6 +708,8 @@ Tags
version('1.0.1', git='https://github.com/example-project/example.git', version('1.0.1', git='https://github.com/example-project/example.git',
tag='v1.0.1') tag='v1.0.1')
This download method is untrusted, and is not recommended.
Branches Branches
To fetch a particular branch, use ``branch`` instead: To fetch a particular branch, use ``branch`` instead:
@ -662,8 +718,7 @@ Branches
version('experimental', git='https://github.com/example-project/example.git', version('experimental', git='https://github.com/example-project/example.git',
branch='experimental') branch='experimental')
This is not recommended, as the contents of branches change over This download method is untrusted, and is not recommended.
time.
Commits Commits
Finally, to fetch a particular commit, use ``commit``: Finally, to fetch a particular commit, use ``commit``:
@ -681,6 +736,9 @@ Commits
version('2014-10-08', git='https://github.com/example-project/example.git', version('2014-10-08', git='https://github.com/example-project/example.git',
commit='9d38cd') commit='9d38cd')
This download method *is trusted*. It is the recommended way to
securely download from a Git repository.
It may be useful to provide a saner version for commits like this, It may be useful to provide a saner version for commits like this,
e.g. you might use the date as the version, as done above. Or you e.g. you might use the date as the version, as done above. Or you
could just use the abbreviated commit hash. It's up to the package could just use the abbreviated commit hash. It's up to the package
@ -696,19 +754,24 @@ Submodules
version('1.0.1', git='https://github.com/example-project/example.git', version('1.0.1', git='https://github.com/example-project/example.git',
tag='v1.0.1', submdoules=True) tag='v1.0.1', submdoules=True)
^^^^^^^^^^
Installing
^^^^^^^^^^
You can fetch and install any of the versions above as you'd expect, .. _github-fetch:
by using ``@<version>`` in a spec:
.. code-block:: console """"""
GitHub
""""""
$ spack install example@2014-10-08 If a project is hosted on GitHub, *any* valid Git branch, tag or hash
may be downloaded as a tarball. This is accomplished simply by
constructing an appropriate URL. Spack can checksum any package
downloaded this way, thereby producing a trusted download. For
example, the following downloads a particular hash, and then applies a
checksum.
Git and other VCS versions will show up in the list of versions when .. code-block:: python
a user runs ``spack info <package name>``.
version('1.9.5.1.1', 'd035e4bc704d136db79b43ab371b27d2',
url='https://www.github.com/jswhit/pyproj/tarball/0be612cc9f972e38b50a90c946a9b353e2ab140f')
.. _hg-fetch: .. _hg-fetch:
@ -726,8 +789,7 @@ Default
version('develop', hg='https://jay.grs.rwth-aachen.de/hg/example') version('develop', hg='https://jay.grs.rwth-aachen.de/hg/example')
Note that this is not recommended; try to fetch a particular This download method is untrusted, and is not recommended.
revision instead.
Revisions Revisions
Add ``hg`` and ``revision`` parameters: Add ``hg`` and ``revision`` parameters:
@ -737,6 +799,8 @@ Revisions
version('1.0', hg='https://jay.grs.rwth-aachen.de/hg/example', version('1.0', hg='https://jay.grs.rwth-aachen.de/hg/example',
revision='v1.0') revision='v1.0')
This download method is untrusted, and is not recommended.
Unlike ``git``, which has special parameters for different types of Unlike ``git``, which has special parameters for different types of
revisions, you can use ``revision`` for branches, tags, and commits revisions, you can use ``revision`` for branches, tags, and commits
when you fetch with Mercurial. when you fetch with Mercurial.
@ -759,7 +823,7 @@ Fetching the head
version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk') version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk')
This is not recommended, as the head will move forward over time. This download method is untrusted, and is not recommended.
Fetching a revision Fetching a revision
To fetch a particular revision, add a ``revision`` to the To fetch a particular revision, add a ``revision`` to the
@ -770,6 +834,8 @@ Fetching a revision
version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk', version('develop', svn='https://outreach.scidac.gov/svn/libmonitor/trunk',
revision=128) revision=128)
This download method is untrusted, and is not recommended.
Subversion branches are handled as part of the directory structure, so Subversion branches are handled as part of the directory structure, so
you can check out a branch or tag by changing the ``url``. you can check out a branch or tag by changing the ``url``.
@ -1345,31 +1411,34 @@ Additionally, dependencies may be specified for specific use cases:
The dependency types are: The dependency types are:
* **"build"**: The dependency package is made available during the * **"build"**: made available during the project's build. The package will
package's build. While the package is built, the dependency be added to ``PATH``, the compiler include paths, and ``PYTHONPATH``.
package's install directory will be added to ``PATH``, the Other projects which depend on this one will not have these modified
compiler include and library paths, as well as ``PYTHONPATH``. (building project X doesn't need project Y's build dependencies).
This only applies during this package's build; other packages * **"link"**: the project is linked to by the project. The package will be
which depend on this one will not know about the dependency added to the current package's ``rpath``.
package. In other words, building another project Y doesn't know * **"run"**: the project is used by the project at runtime. The package will
about this project X's build dependencies. be added to ``PATH`` and ``PYTHONPATH``.
* **"link"**: The dependency package is linked against by this
package, presumably via shared libraries. The dependency package
will be added to this package's run-time library search path
``rpath``.
* **"run"**: The dependency package is used by this package at run
time. The dependency package will be added to both ``PATH`` and
``PYTHONPATH`` at run time, but not during build time. **"link"**
and **"run"** are similar in that they both describe a dependency
that exists when the package is used, but they differ in the
mechanism: **"link"** is via shared libraries, and **"run"** via
an explicit search.
If not specified, ``type`` is assumed to be ``("build", "link")``. Additional hybrid dependency types are (note the lack of quotes):
This is the common case for compiled language usage. Also available
are the aliases ``"alldeps"`` for all dependency types combined, and * **<not specified>**: ``type`` assumed to be ``("build",
``"nolink"`` (``("build", "run")``) for use by dependencies which are "link")``. This is the common case for compiled language usage.
not expressed via a linker (e.g., Python or Lua module loading). * **alldeps**: All dependency types. **Note:** No quotes here
* **nolink**: Equal to ``("build", "run")``, for use by dependencies
that are not expressed via a linker (e.g., Python or Lua module
loading). **Note:** No quotes here
"""""""""""""""""""
Dependency Formulas
"""""""""""""""""""
This section shows how to write appropriate ``depends_on()``
declarations for some common cases.
* Python 2 only: ``depends_on('python@:2.8')``
* Python 2.7 only: ``depends_on('python@2.7:2.8')``
* Python 3 only: ``depends_on('python@3:')``
.. _setup-dependent-environment: .. _setup-dependent-environment:
@ -1458,6 +1527,17 @@ Now, the ``py-numpy`` package can be used as an argument to ``spack
activate``. When it is activated, all the files in its prefix will be activate``. When it is activated, all the files in its prefix will be
symbolically linked into the prefix of the python package. symbolically linked into the prefix of the python package.
Some packages produce a Python extension, but are only compatible with
Python 3, or with Python 2. In those cases, a ``depends_on()``
declaration should be made in addition to the ``extends()``
declaration:
.. code-block:: python
class Icebin(Package):
extends('python', when='+python')
depends_on('python@3:', when='+python')
Many packages produce Python extensions for *some* variants, but not Many packages produce Python extensions for *some* variants, but not
others: they should extend ``python`` only if the appropriate others: they should extend ``python`` only if the appropriate
variant(s) are selected. This may be accomplished with conditional variant(s) are selected. This may be accomplished with conditional
@ -1817,6 +1897,46 @@ See the :ref:`concretization-preferences` section for more details.
.. _install-method: .. _install-method:
------------------
Inconsistent Specs
------------------
Suppose a user needs to install package C, which depends on packages A
and B. Package A builds a library with a Python2 extension, and
package B builds a library with a Python3 extension. Packages A and B
cannot be loaded together in the same Python runtime:
.. code-block:: python
class A(Package):
variant('python', default=True, 'enable python bindings')
depends_on('python@2.7', when='+python')
def install(self, spec, prefix):
# do whatever is necessary to enable/disable python
# bindings according to variant
class B(Package):
variant('python', default=True, 'enable python bindings')
depends_on('python@3.2:', when='+python')
def install(self, spec, prefix):
# do whatever is necessary to enable/disable python
# bindings according to variant
Package C needs to use the libraries from packages A and B, but does
not need either of the Python extensions. In this case, package C
should simply depend on the ``~python`` variant of A and B:
.. code-block:: python
class C(Package):
depends_on('A~python')
depends_on('B~python')
This may require that A or B be built twice, if the user wishes to use
the Python extensions provided by them: once for ``+python`` and once
for ``~python``. Other than using a little extra disk space, that
solution has no serious problems.
----------------------------------- -----------------------------------
Implementing the ``install`` method Implementing the ``install`` method
----------------------------------- -----------------------------------
@ -3027,15 +3147,15 @@ might write:
CXXFLAGS += -I$DWARF_PREFIX/include CXXFLAGS += -I$DWARF_PREFIX/include
CXXFLAGS += -L$DWARF_PREFIX/lib CXXFLAGS += -L$DWARF_PREFIX/lib
---------------------------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Build System Configuration Support Build System Configuration Support
---------------------------------- ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Imagine a developer creating a CMake or Autotools-based project in a local Imagine a developer creating a CMake or Autotools-based project in a
directory, which depends on libraries A-Z. Once Spack has installed local directory, which depends on libraries A-Z. Once Spack has
those dependencies, one would like to run ``cmake`` with appropriate installed those dependencies, one would like to run ``cmake`` with
command line and environment so CMake can find them. The ``spack appropriate command line and environment so CMake can find them. The
setup`` command does this conveniently, producing a CMake ``spack setup`` command does this conveniently, producing a CMake
configuration that is essentially the same as how Spack *would have* configuration that is essentially the same as how Spack *would have*
configured the project. This can be demonstrated with a usage configured the project. This can be demonstrated with a usage
example: example:

1208
lib/spack/docs/workflows.rst Normal file
View file

File diff suppressed because it is too large Load diff

5
lib/spack/spack/directives.py
View file

@ -212,7 +212,10 @@ def _depends_on(pkg, spec, when=None, type=None):
@directive(('dependencies', '_deptypes')) @directive(('dependencies', '_deptypes'))
def depends_on(pkg, spec, when=None, type=None): def depends_on(pkg, spec, when=None, type=None):
"""Creates a dict of deps with specs defining when they apply.""" """Creates a dict of deps with specs defining when they apply.
This directive is to be used inside a Package definition to declare
that the package requires other packages to be built first.
@see The section "Dependency specs" in the Spack Packaging Guide."""
_depends_on(pkg, spec, when=when, type=type) _depends_on(pkg, spec, when=when, type=type)

9
lib/spack/spack/package.py
View file

@ -482,8 +482,13 @@ def nearest_url(self, version):
# TODO: move this out of here and into some URL extrapolation module? # TODO: move this out of here and into some URL extrapolation module?
def url_for_version(self, version): def url_for_version(self, version):
""" """Returns a URL from which the specified version of this package
Returns a URL that you can download a new version of this package from. may be downloaded.
version: class Version
The version for which a URL is sought.
See Class Version (version.py)
""" """
if not isinstance(version, Version): if not isinstance(version, Version):
version = Version(version) version = Version(version)