diff --git a/lib/spack/docs/basic_usage.rst b/lib/spack/docs/basic_usage.rst index ae7a3762c6..0a80fa7e2f 100644 --- a/lib/spack/docs/basic_usage.rst +++ b/lib/spack/docs/basic_usage.rst @@ -1,13 +1,14 @@ .. _basic-usage: +=========== Basic usage -===================== +=========== The ``spack`` command has many *subcommands*. You'll only need a small subset of them for typical usage. Note that Spack colorizes output. ``less -R`` should be used with -Spack to maintian this colorization. Eg:: +Spack to maintain this colorization. E.g.:: spack find | less -R @@ -16,8 +17,9 @@ It is recommend that the following be put in your ``.bashrc`` file:: alias less='less -R' +-------------------------- Listing available packages ------------------------------- +-------------------------- To install software with Spack, you need to know what software is available. You can see a list of available package names at the @@ -25,8 +27,9 @@ available. You can see a list of available package names at the .. _spack-list: +^^^^^^^^^^^^^^ ``spack list`` -~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ The ``spack list`` command prints out a list of all of the packages Spack can install: @@ -59,8 +62,9 @@ All packages whose names contain documentation case insensitive: .. _spack-info: +^^^^^^^^^^^^^^ ``spack info`` -~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ To get more information on a particular package from `spack list`, use `spack info`. Just supply the name of a package: @@ -77,8 +81,9 @@ viruses. .. _spack-versions: +^^^^^^^^^^^^^^^^^^ ``spack versions`` -~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^ To see *more* available versions of a package, run ``spack versions``. For example: @@ -95,14 +100,15 @@ by scraping it directly from package web pages. Depending on the package and how its releases are organized, Spack may or may not be able to find remote versions. - +--------------------------- Installing and uninstalling ------------------------------- +--------------------------- .. _spack-install: +^^^^^^^^^^^^^^^^^ ``spack install`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^ ``spack install`` will install any package shown by ``spack list``. For example, To install the latest version of the ``mpileaks`` @@ -142,8 +148,9 @@ that the packages is installed: The last line, with the ``[+]``, indicates where the package is installed. +^^^^^^^^^^^^^^^^^^^^^^^^^^^ Building a specific version -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^ Spack can also build *specific versions* of a package. To do this, just add ``@`` after the package name, followed by a version: @@ -172,8 +179,9 @@ customize an installation in :ref:`sec-specs`. .. _spack-uninstall: +^^^^^^^^^^^^^^^^^^^ ``spack uninstall`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^ To uninstall a package, type ``spack uninstall ``. This will ask the user for confirmation, and in case will completely remove the directory in which the package was installed. @@ -216,17 +224,18 @@ You may force uninstall a package with the `--force` option but you risk breaking other installed packages. In general, it is safer to remove dependent packages *before* removing their dependencies or use the `--dependents` option. - +------------------------- Seeing installed packages ------------------------------------ +------------------------- We know that ``spack list`` shows you the names of available packages, but how do you figure out which are installed? .. _spack-find: +^^^^^^^^^^^^^^ ``spack find`` -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ ``spack find`` shows the *specs* of installed packages. A spec is like a name, but it has a version, compiler, architecture, and build @@ -367,11 +376,11 @@ will find every installed package with a 'debug' compile-time option enabled. The full spec syntax is discussed in detail in :ref:`sec-specs`. - .. _compiler-config: +---------------------- Compiler configuration ------------------------------------ +---------------------- Spack has the ability to build packages with multiple compilers and compiler versions. Spack searches for compilers on your machine @@ -380,8 +389,9 @@ your path. .. _spack-compilers: +^^^^^^^^^^^^^^^^^^^ ``spack compilers`` -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^ You can see which compilers spack has found by running ``spack compilers`` or ``spack compiler list``:: @@ -408,15 +418,17 @@ how this is done is in :ref:`sec-specs`. .. _spack-compiler-add: +^^^^^^^^^^^^^^^^^^^^^^ ``spack compiler add`` -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^ An alias for ``spack compiler find``. .. _spack-compiler-find: +^^^^^^^^^^^^^^^^^^^^^^^ ``spack compiler find`` -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^ If you do not see a compiler in this list, but you want to use it with Spack, you can simply run ``spack compiler find`` with the path to @@ -441,8 +453,9 @@ This loads the environment module for gcc-4.9.0 to add it to .. _spack-compiler-info: +^^^^^^^^^^^^^^^^^^^^^^^ ``spack compiler info`` -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^ If you want to see specifics on a particular compiler, you can run ``spack compiler info`` on it:: @@ -459,9 +472,9 @@ Notice also that we didn\'t have to be too specific about the version. We just said ``intel@15``, and information about the only matching Intel compiler was displayed. - +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Manual compiler configuration -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If auto-detection fails, you can manually configure a compiler by editing your ``~/.spack/compilers.yaml`` file. You can do this by running @@ -518,8 +531,9 @@ following section. .. _sec-specs: +-------------------- Specs & dependencies -------------------------- +-------------------- We know that ``spack install``, ``spack uninstall``, and other commands take a package name with an optional version specifier. In @@ -609,7 +623,6 @@ some *other* version of ``mpich``. In general, such a configuration would likely behave unexpectedly at runtime, and Spack enforces this to ensure a consistent runtime environment. - The point of specs is to abstract this full DAG from Spack users. If a user does not care about the DAG at all, she can refer to mpileaks by simply writing ``mpileaks``. If she knows that ``mpileaks`` @@ -646,8 +659,9 @@ could depend on ``mpich@1.2:`` if it can only build with version Below are more details about the specifiers that you can add to specs. +^^^^^^^^^^^^^^^^^ Version specifier -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^ A version specifier comes somewhere after a package name and starts with ``@``. It can be a single version, e.g. ``@1.0``, ``@3``, or @@ -669,9 +683,9 @@ policies set for the particular Spack installation. Details about how versions are compared and how Spack determines if one version is less than another are discussed in the developer guide. - +^^^^^^^^^^^^^^^^^^ Compiler specifier -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^ A compiler specifier comes somewhere after a package name and starts with ``%``. It tells Spack what compiler(s) a particular package @@ -688,9 +702,9 @@ name or compiler specifier to their left in the spec. If the compiler spec is omitted, Spack will choose a default compiler based on site policies. - +^^^^^^^^ Variants -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^ Variants are named options associated with a particular package. They are optional, as each package must provide default values for each variant it @@ -742,9 +756,9 @@ variants using the backwards compatibility syntax and uses only ``~`` for disabled boolean variants. We allow ``-`` and spaces on the command line is provided for convenience and legibility. - +^^^^^^^^^^^^^^ Compiler Flags -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ Compiler flags are specified using the same syntax as non-boolean variants, but fulfill a different purpose. While the function of a variant is set by @@ -764,9 +778,9 @@ in gnu autotools. If all flags are set, the order is ``$cppflags $cflags|$cxxflags $ldflags command $ldlibs`` for C and C++ and ``$fflags $cppflags $ldflags command $ldlibs`` for fortran. - +^^^^^^^^^^^^^^^^^^^^^^^ Architecture specifiers -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^ The architecture can be specified by using the reserved words ``target`` and/or ``os`` (``target=x86-64 os=debian7``). You can also @@ -781,11 +795,11 @@ Spack will autodetect what kind of operating system is on your machine as well as the processor. For more information on how the architecture can be used on Cray machines, check here :ref:`cray-support` - .. _sec-virtual-dependencies: +-------------------- Virtual dependencies -------------------------- +-------------------- The dependence graph for ``mpileaks`` we saw above wasn't *quite* accurate. ``mpileaks`` uses MPI, which is an interface that has many @@ -828,9 +842,9 @@ version of a virtual package, e.g. if an application needs MPI-2 functions, it can depend on ``mpi@2:`` to indicate that it needs some implementation that provides MPI-2 functions. - +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Constraining virtual packages -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ When installing a package that depends on a virtual package, you can opt to specify the particular provider you want to use, or you can let @@ -857,8 +871,9 @@ any MPI implementation will do. If another package depends on error. Likewise, if you try to plug in some package that doesn't provide MPI, Spack will raise an error. +^^^^^^^^^^^^^^^^^^^^^^^^ Specifying Specs by Hash -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^ Complicated specs can become cumbersome to enter on the command line, especially when many of the qualifications are necessary to @@ -876,8 +891,9 @@ Note that this will not work to reinstall a depencency uninstalled by .. _spack-providers: +^^^^^^^^^^^^^^^^^^^ ``spack providers`` -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^ You can see what packages provide a particular virtual package using ``spack providers``. If you wanted to see what packages provide @@ -895,6 +911,7 @@ versions are now filtered out. .. _shell-support: +------------------------------- Integration with module systems ------------------------------- @@ -910,10 +927,9 @@ Spack provides some integration with and `Dotkit `_ to make it easier to use the packages it installed. - - +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Installing Environment Modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ In order to use Spack's generated environment modules, you must have installed the *Environment Modules* package. On many Linux @@ -931,7 +947,6 @@ Environment Modules, you can get it with Spack: spack install environment-modules - In this case to activate it automatically you need to add the following two lines to your ``.bashrc`` profile (or similar): @@ -944,11 +959,12 @@ If you use a Unix shell other than ``bash``, modify the commands above accordingly and source the appropriate file in ``${MODULES_HOME}/Modules/init/``. - .. TODO : Add a similar section on how to install dotkit ? +^^^^^^^^^^^^^^^^^^^^^^^^ Spack and module systems -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^ + You can enable shell support by sourcing some files in the ``/share/spack`` directory. @@ -986,9 +1002,9 @@ The directories are automatically added to your ``MODULEPATH`` and ``DK_NODE`` environment variables when you enable Spack's `shell support `_. - +^^^^^^^^^^^^^^^^^^^^^^^ Using Modules & Dotkits -~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^ If you have shell support enabled you should be able to run either ``module avail`` or ``use -l spack`` to see what modules/dotkits have @@ -1093,8 +1109,9 @@ a package, you can type unload or unuse similarly: only available if you have enabled Spack's shell support *and* you have dotkit or modules installed on your machine. +^^^^^^^^^^^^^^^^^^^^^^ Ambiguous module names -~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^ If a spec used with load/unload or use/unuse is ambiguous (i.e. more than one installed package matches it), then Spack will warn you: @@ -1118,8 +1135,9 @@ used ``gcc``. You could therefore just type: To identify just the one built with the Intel compiler. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Module files generation and customization -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Environment Modules and Dotkit files are generated when packages are installed, and are placed in the following directories under the Spack root: @@ -1134,8 +1152,10 @@ The content that gets written in each module file can be customized in two ways: 1. overriding part of the ``spack.Package`` API within a ``package.py`` 2. writing dedicated configuration files +^^^^^^^^^^^^^^^^^^^^^^^^ Override ``Package`` API ^^^^^^^^^^^^^^^^^^^^^^^^ + There are currently two methods in ``spack.Package`` that may affect the content of module files: @@ -1145,9 +1165,9 @@ of module files: """Set up the compile and runtime environments for a package.""" pass - +""""""""""""""""" Recursive Modules -`````````````````` +""""""""""""""""" In some cases, it is desirable to load not just a module, but also all the modules it depends on. This is not required for most modules @@ -1165,8 +1185,9 @@ Modules may be loaded recursively with the ``load`` command's More than one spec may be placed on the command line here. +""""""""""""""""""""""""""""""""" Module Commands for Shell Scripts -`````````````````````````````````` +""""""""""""""""""""""""""""""""" Although Spack is flexible, the ``module`` command is much faster. This could become an issue when emitting a series of ``spack load`` @@ -1215,9 +1236,9 @@ This script may be directly executed in bash via source <( spack module find tcl --dependencies --shell py-numpy git ) - +^^^^^^^^^^^^^^^^^^^^^^^^^ Regenerating Module files -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: python @@ -1230,6 +1251,7 @@ module file content for the package you are currently writing, the second allows for modifications to your dependees module file. In both cases one needs to fill ``run_env`` with the desired list of environment modifications. +"""""""""""""""""""""""""""""""""""""""""""""""" Example : ``builtin/packages/python/package.py`` """""""""""""""""""""""""""""""""""""""""""""""" @@ -1246,6 +1268,7 @@ overrides ``setup_dependent_environment`` in the following way: to insert the appropriate ``PYTHONPATH`` modifications in the module files of python packages. +^^^^^^^^^^^^^^^^^^^ Configuration files ^^^^^^^^^^^^^^^^^^^ @@ -1275,8 +1298,10 @@ For each module system that can be enabled a finer configuration is possible: The structure under the ``tcl`` and ``dotkit`` keys is almost equal, and will be showcased in the following by some examples. +""""""""""""""""""""""""""""""""""""""" Select module files by spec constraints """"""""""""""""""""""""""""""""""""""" + Using spec syntax it's possible to have different customizations for different groups of module files. @@ -1316,6 +1341,7 @@ what will happen is that: first, no matter where they appear in the configuration file. All the other spec constraints are instead evaluated top to bottom. +"""""""""""""""""""""""""""""""""""""""" Filter modifications out of module files """""""""""""""""""""""""""""""""""""""" @@ -1335,6 +1361,7 @@ will generate dotkit module files that will not contain modifications to either ``CPATH`` or ``LIBRARY_PATH`` and environment module files that instead will contain those modifications. +""""""""""""""""""""" Autoload dependencies """"""""""""""""""""" @@ -1357,6 +1384,7 @@ dependencies. ``prerequisites`` that accepts the same values and will add ``prereq`` statements instead of automatically loading other modules. +"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" Blacklist or whitelist the generation of specific module files """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""" @@ -1375,6 +1403,7 @@ A configuration file like: will skip module file generation for anything that satisfies ``%gcc@4.4.7``, with the exception of specs that satisfy ``gcc`` or ``llvm``. +"""""""""""""""""""""""""""""""""""""""""""""""" Customize the naming scheme and insert conflicts """""""""""""""""""""""""""""""""""""""""""""""" @@ -1400,6 +1429,7 @@ load two or more versions of the same software at the same time. .. note:: The ``conflict`` option is ``tcl`` specific +^^^^^^^^^^^^^^^^^^^^^^^^^ Regenerating module files ^^^^^^^^^^^^^^^^^^^^^^^^^ @@ -1410,6 +1440,7 @@ new features. .. _spack-module: +"""""""""""""""""""""""" ``spack module refresh`` """""""""""""""""""""""" @@ -1426,8 +1457,9 @@ regenerate all module and dotkit files from scratch: .. _extensions: +---------------- Filesystem Views -------------------------------- +---------------- .. Maybe this is not the right location for this documentation. @@ -1469,8 +1501,9 @@ Views can be used for a variety of purposes including: - Providing an atomic and monolithic binary distribution, eg for delivery as a single tarball. - Producing ephemeral testing or developing environments. +^^^^^^^^^^^^^^^^^^^^^^ Using Filesystem Views -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^ A filesystem view is created and packages are linked in by the ``spack view`` command's ``symlink`` and ``hardlink`` sub-commands. The @@ -1483,7 +1516,6 @@ files in the ``cmake`` package while retaining its dependencies. .. code-block:: sh - $ spack view -v symlink myview cmake@3.5.2 ==> Linking package: "ncurses" ==> Linking package: "zlib" @@ -1504,9 +1536,9 @@ files in the ``cmake`` package while retaining its dependencies. captoinfo c_rehash infotocap openssl tabs toe tset clear infocmp ncurses6-config reset tic tput - +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Limitations of Filesystem Views -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This section describes some limitations that should be considered in using filesystems views. @@ -1536,12 +1568,9 @@ particular, if two packages require the same sub-tree of dependencies, removing one package (recursively) will remove its dependencies and leave the other package broken. - - - - +--------------------------- Extensions & Python support ------------------------------------- +--------------------------- Spack's installation model assumes that each package will live in its own install prefix. However, certain packages are typically installed @@ -1563,8 +1592,9 @@ an *extension*. Suppose you have Python installed like so: .. _spack-extensions: +^^^^^^^^^^^^^^^^^^^^ ``spack extensions`` -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^ You can find extensions for your Python installation like this: @@ -1618,8 +1648,9 @@ directly when you run ``python``: ImportError: No module named numpy >>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Extensions & Environment Modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ There are two ways to get ``numpy`` working in Python. The first is to use :ref:`shell-support`. You can simply ``use`` or ``load`` the @@ -1643,9 +1674,9 @@ or, for dotkit: Now ``import numpy`` will succeed for as long as you keep your current session open. - +^^^^^^^^^^^^^^^^^^^^^ Activating Extensions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^ It is often desirable to have certain packages *always* available as part of a Python installation. Spack offers a more permanent solution @@ -1655,8 +1686,9 @@ installation: .. _spack-activate: +^^^^^^^^^^^^^^^^^^ ``spack activate`` -^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^ .. code-block:: sh @@ -1700,7 +1732,6 @@ packages listed as activated: -- linux-debian7-x86_64 / gcc@4.4.7 -------------------------------- py-nose@1.3.4 py-numpy@1.9.1 py-setuptools@11.3.1 - Now, when a user runs python, ``numpy`` will be available for import *without* the user having to explicitly loaded. ``python@2.7.8`` now acts like a system Python installation with ``numpy`` installed inside @@ -1718,8 +1749,10 @@ into the same prefix. Users who want a different version of a package can still get it by using environment modules, but they will have to explicitly load their preferred version. +^^^^^^^^^^^^^^^^^^^^^ ``spack activate -f`` -^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^^ + If, for some reason, you want to activate a package *without* its dependencies, you can use ``spack activate -f``: @@ -1730,8 +1763,9 @@ dependencies, you can use ``spack activate -f``: .. _spack-deactivate: +^^^^^^^^^^^^^^^^^^^^ ``spack deactivate`` -^^^^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^^^ We've seen how activating an extension can be used to set up a default version of a Python module. Obviously, you may want to change that at @@ -1751,8 +1785,9 @@ several variants: spack deactivate -a python +----------------------- Filesystem requirements --------------------------- +----------------------- Spack currently needs to be run from a filesystem that supports ``flock`` locking semantics. Nearly all local filesystems and recent @@ -1797,11 +1832,16 @@ This issue typically manifests with the error below: A nicer error message is TBD in future versions of Spack. +<<<<<<< c36f13e44dae52086ca48e4f16b514642bc91106 .. _cray-support: +======= +.. _spack-cray: +>>>>>>> Fix #1604 and fix #1605, title underline problems +------------- Spack on Cray ------------------------------ +------------- Spack differs slightly when used on a Cray system. The architecture spec can differentiate between the front-end and back-end processor and operating system. @@ -1835,10 +1875,9 @@ And the front-end operating system is: spack install zlib os=SuSE11 - - +^^^^^^^^^^^^^^^^^^^^^^^ Cray compiler detection -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^ Spack can detect compilers using two methods. For the front-end, we treat everything the same. The difference lies in back-end compiler detection. @@ -1880,8 +1919,9 @@ Spack sets the default Cray link type to dynamic, to better match other other platforms. Individual packages can enable static linking (which is the default outside of Spack on cray systems) using the -static flag. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Setting defaults and using Cray modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ If you want to use default compilers for each PrgEnv and also be able to load cray external modules, you will need to set up a packages.yaml. @@ -1934,16 +1974,17 @@ if we want to build with intel compilers, use version 16.0.0.109. We add a spec for each compiler type for each cray modules. This ensures that for each compiler on our system we can use that external module. - For more on external packages check out the section :ref:`sec-external_packages`. +------------ Getting Help ------------------------ +------------ .. _spack-help: +^^^^^^^^^^^^^^ ``spack help`` -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ If you don't find what you need here, the ``help`` subcommand will print out out a list of *all* of ``spack``'s options and subcommands: diff --git a/lib/spack/docs/configuration.rst b/lib/spack/docs/configuration.rst index d39c932021..bce6005a7c 100644 --- a/lib/spack/docs/configuration.rst +++ b/lib/spack/docs/configuration.rst @@ -1,12 +1,14 @@ .. _configuration: +============= Configuration -=================================== +============= .. _temp-space: +--------------- Temporary space ----------------------------- +--------------- .. warning:: Temporary space configuration will eventually be moved to configuration files, but currently these settings are in @@ -55,8 +57,10 @@ directory is. .. _sec-external_packages: +----------------- External Packages ----------------------------- +----------------- + Spack can be configured to use externally-installed packages rather than building its own packages. This may be desirable if machines ship with system packages, such as a customized MPI @@ -126,9 +130,9 @@ The ``buildable`` does not need to be paired with external packages. It could also be used alone to forbid packages that may be buggy or otherwise undesirable. - +-------------------------- Concretization Preferences --------------------------------- +-------------------------- Spack can be configured to prefer certain compilers, package versions, depends_on, and variants during concretization. @@ -136,7 +140,6 @@ The preferred configuration can be controlled via the ``~/.spack/packages.yaml`` file for user configuations, or the ``etc/spack/packages.yaml`` site configuration. - Here's an example packages.yaml file that sets preferred packages: .. code-block:: sh @@ -152,7 +155,6 @@ Here's an example packages.yaml file that sets preferred packages: providers: mpi: [mvapich, mpich, openmpi] - At a high level, this example is specifying how packages should be concretized. The opencv package should prefer using gcc 4.9 and be built with debug options. The gperftools package should prefer version @@ -185,9 +187,9 @@ concretization rules. A provider lists a value that packages may ``depend_on`` (e.g, mpi) and a list of rules for fulfilling that dependency. - +--------- Profiling ------------------- +--------- Spack has some limited built-in support for profiling, and can report statistics using standard Python timing tools. To use this feature, @@ -195,8 +197,9 @@ supply ``-p`` to Spack on the command line, before any subcommands. .. _spack-p: +^^^^^^^^^^^^ ``spack -p`` -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^ ``spack -p`` output looks like this: diff --git a/lib/spack/docs/developer_guide.rst b/lib/spack/docs/developer_guide.rst index d28fe4b2a5..5c712149b5 100644 --- a/lib/spack/docs/developer_guide.rst +++ b/lib/spack/docs/developer_guide.rst @@ -1,7 +1,8 @@ .. _developer_guide: +=============== Developer Guide -===================== +=============== This guide is intended for people who want to work on Spack itself. If you just want to develop packages, see the :ref:`packaging-guide`. @@ -11,8 +12,9 @@ It is assumed that you've read the :ref:`basic-usage` and concepts discussed there. If you're not, we recommend reading those first. +-------- Overview ------------------------ +-------- Spack is designed with three separate roles in mind: @@ -63,9 +65,9 @@ building the software off to the package object. The rest of this document describes all the pieces that come together to make that happen. - +------------------- Directory Structure -------------------------- +------------------- So that you can familiarize yourself with the project, we'll start with a high level view of Spack's directory structure:: @@ -123,15 +125,16 @@ use the thing that's supposed to spare them from the details of big, complicated packages. The end result is that Spack works out of the box: clone it and add ``bin`` to your PATH and you're ready to go. - +-------------- Code Structure -------------------------- +-------------- This section gives an overview of the various Python modules in Spack, grouped by functionality. +^^^^^^^^^^^^^^^^^^^^^^^ Package-related modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^ :mod:`spack.package` Contains the :class:`Package ` class, which @@ -158,9 +161,9 @@ Package-related modules decorator, which allows :ref:`multimethods ` in packages. - +^^^^^^^^^^^^^^^^^^^^ Spec-related modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^ :mod:`spack.spec` Contains :class:`Spec ` and :class:`SpecParser @@ -208,9 +211,9 @@ Spec-related modules Not yet implemented. Should eventually have architecture descriptions for cross-compiling. - +^^^^^^^^^^^^^^^^^ Build environment -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^ :mod:`spack.stage` Handles creating temporary directories for builds. @@ -224,15 +227,17 @@ Build environment Create more implementations of this to change the hierarchy and naming scheme in ``$spack_prefix/opt`` +^^^^^^^^^^^^^^^^^ Spack Subcommands -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^ :mod:`spack.cmd` Each module in this package implements a Spack subcommand. See :ref:`writing commands ` for details. +^^^^^^^^^^ Unit tests -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^ :mod:`spack.test` Implements Spack's test suite. Add a module and put its name in @@ -242,8 +247,9 @@ Unit tests This is a fake package hierarchy used to mock up packages for Spack's test suite. +^^^^^^^^^^^^^ Other Modules -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^ :mod:`spack.globals` Includes global settings for Spack. the default policy classes for @@ -269,13 +275,13 @@ Other Modules :class:`SpackError `, the base class for Spack's exception hierarchy. - +------------ Spec objects -------------------------- +------------ +--------------- Package objects -------------------------- - +--------------- Most spack commands look something like this: @@ -286,34 +292,37 @@ look something like this: #. Instantiate a package based on the spec, and #. Call methods (e.g., ``install()``) on the package object. - - The information in Package files is used at all stages in this process. - Conceptually, packages are overloaded. They contain +------------- Stage objects -------------------------- +------------- .. _writing-commands: +---------------- Writing commands -------------------------- +---------------- +---------- Unit tests -------------------------- +---------- +------------ Unit testing -------------------------- - +------------ +------------------ Developer commands -------------------------- +------------------ +^^^^^^^^^^^^^ ``spack doc`` -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^ +^^^^^^^^^^^^^^ ``spack test`` -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ diff --git a/lib/spack/docs/features.rst b/lib/spack/docs/features.rst index 27a3b4b435..d746e8ad50 100644 --- a/lib/spack/docs/features.rst +++ b/lib/spack/docs/features.rst @@ -1,13 +1,15 @@ +================ Feature overview -================== +================ This is a high-level overview of features that make Spack different from other `package managers `_ and `port systems `_. +--------------------------- Simple package installation ----------------------------- +--------------------------- Installing the default version of a package is simple. This will install the latest version of the ``mpileaks`` package and all of its dependencies: @@ -16,8 +18,9 @@ the latest version of the ``mpileaks`` package and all of its dependencies: $ spack install mpileaks +-------------------------------- Custom versions & configurations -------------------------------------------- +-------------------------------- Spack allows installation to be customized. Users can specify the version, build compiler, compile-time options, and cross-compile @@ -47,9 +50,9 @@ Users can specify as many or few options as they care about. Spack will fill in the unspecified values with sensible defaults. The two listed syntaxes for variants are identical when the value is boolean. - +---------------------- Customize dependencies -------------------------------------- +---------------------- Spack allows *dependencies* of a particular installation to be customized extensively. Suppose that ``mpileaks`` depends indirectly @@ -61,23 +64,25 @@ configurations for the dependencies: # Install mpileaks and link it with specific versions of libelf and libdwarf $ spack install mpileaks@1.1.2 %gcc@4.7.3 +debug ^libelf@0.8.12 ^libdwarf@20130729+debug - +------------------------ Non-destructive installs -------------------------------------- +------------------------ Spack installs every unique package/dependency configuration into its own prefix, so new installs will not break existing ones. +------------------------------- Packages can peacefully coexist -------------------------------------- +------------------------------- Spack avoids library misconfiguration by using ``RPATH`` to link dependencies. When a user links a library or runs a program, it is tied to the dependencies it was built with, so there is no need to manipulate ``LD_LIBRARY_PATH`` at runtime. +------------------------- Creating packages is easy -------------------------------------- +------------------------- To create a new packages, all Spack needs is a URL for the source archive. The ``spack create`` command will create a boilerplate diff --git a/lib/spack/docs/getting_started.rst b/lib/spack/docs/getting_started.rst index 2c5b68ea65..47f66b429b 100644 --- a/lib/spack/docs/getting_started.rst +++ b/lib/spack/docs/getting_started.rst @@ -1,8 +1,10 @@ +=============== Getting Started -==================== +=============== +-------- Download --------------------- +-------- Getting spack is easy. You can clone it from the `github repository `_ using this command: @@ -35,8 +37,9 @@ For a richer experience, use Spack's `shell support This automatically adds Spack to your ``PATH``. +------------ Installation --------------------- +------------ You don't need to install Spack; it's ready to run as soon as you clone it from git. diff --git a/lib/spack/docs/index.rst b/lib/spack/docs/index.rst index a5bbd4e23b..603245b5dc 100644 --- a/lib/spack/docs/index.rst +++ b/lib/spack/docs/index.rst @@ -3,8 +3,9 @@ You can adapt this file completely to your liking, but it should at least contain the root `toctree` directive. +=================== Spack Documentation -================================= +=================== Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms @@ -36,8 +37,9 @@ package: If you're new to spack and want to start using it, see :doc:`getting_started`, or refer to the full manual below. +----------------- Table of Contents ---------------------- +----------------- .. toctree:: :maxdepth: 2 @@ -54,6 +56,7 @@ Table of Contents package_list API Docs +================== Indices and tables ================== diff --git a/lib/spack/docs/mirrors.rst b/lib/spack/docs/mirrors.rst index 583575a565..41e039b838 100644 --- a/lib/spack/docs/mirrors.rst +++ b/lib/spack/docs/mirrors.rst @@ -1,7 +1,8 @@ .. _mirrors: +======= Mirrors -============================ +======= Some sites may not have access to the internet for fetching packages. These sites will need a local repository of tarballs from which they @@ -51,8 +52,9 @@ contains tarballs for each package, named after each package. .. _spack-mirror: +---------------- ``spack mirror`` ----------------------------- +---------------- Mirrors are managed with the ``spack mirror`` command. The help for ``spack mirror`` looks like this:: @@ -79,8 +81,9 @@ control the URL(s) from which Spack downloads its packages. .. _spack-mirror-create: +----------------------- ``spack mirror create`` ----------------------------- +----------------------- You can create a mirror using the ``spack mirror create`` command, assuming you're on a machine where you can access the internet. @@ -89,7 +92,6 @@ The command will iterate through all of Spack's packages and download the safe ones into a directory structure like the one above. Here is what it looks like: - .. code-block:: bash $ spack mirror create libelf libdwarf @@ -124,8 +126,9 @@ what it looks like: Once this is done, you can tar up the ``spack-mirror-2014-06-24`` directory and copy it over to the machine you want it hosted on. +^^^^^^^^^^^^^^^^^^^ Custom package sets -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^ Normally, ``spack mirror create`` downloads all the archives it has checksums for. If you want to only create a mirror for a subset of @@ -138,8 +141,9 @@ command:: Will create a mirror for libelf versions greater than or equal to 0.8.12 and boost versions greater than or equal to 1.44. +^^^^^^^^^^^^ Mirror files -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^ If you have a *very* large number of packages you want to mirror, you can supply a file with specs in it, one per line:: @@ -158,8 +162,9 @@ your site. .. _spack-mirror-add: +-------------------- ``spack mirror add`` ----------------------------- +-------------------- Once you have a mirror, you need to let spack know about it. This is relatively simple. First, figure out the URL for the mirror. If it's @@ -183,8 +188,9 @@ Each mirror has a name so that you can refer to it again later. .. _spack-mirror-list: +--------------------- ``spack mirror list`` ----------------------------- +--------------------- To see all the mirrors Spack knows about, run ``spack mirror list``:: @@ -193,8 +199,9 @@ To see all the mirrors Spack knows about, run ``spack mirror list``:: .. _spack-mirror-remove: +----------------------- ``spack mirror remove`` ----------------------------- +----------------------- To remove a mirror by name:: @@ -202,8 +209,9 @@ To remove a mirror by name:: $ spack mirror list ==> No mirrors configured. +----------------- Mirror precedence ----------------------------- +----------------- Adding a mirror really adds a line in ``~/.spack/mirrors.yaml``:: @@ -217,8 +225,9 @@ search the topmost mirror first and the bottom-most mirror last. .. _caching: +------------------- Local Default Cache ----------------------------- +------------------- Spack caches resources that are downloaded as part of installs. The cache is a valid spack mirror: it uses the same directory structure and naming scheme diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 34fcb1e101..c6ac6dd7ef 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -1,7 +1,8 @@ .. _packaging-guide: +=============== Packaging Guide -===================== +=============== This guide is intended for developers or administrators who want to package software so that Spack can install it. It assumes that you @@ -28,13 +29,15 @@ ubiquitous in the scientific software community. Second, it's a modern language and has many powerful features to help make package writing easy. +--------------------------- Creating & editing packages ----------------------------------- +--------------------------- .. _spack-create: +^^^^^^^^^^^^^^^^ ``spack create`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^ The ``spack create`` command creates a directory with the package name and generates a ``package.py`` file with a boilerplate package template from a URL. @@ -194,8 +197,9 @@ Before going into details, we'll cover a few more basics. .. _spack-edit: +^^^^^^^^^^^^^^ ``spack edit`` -~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ One of the easiest ways to learn to write packages is to look at existing ones. You can edit a package file by name with the ``spack @@ -222,8 +226,9 @@ using ``spack create`` or ``spack edit -f``: .. _spack-edit-f: +^^^^^^^^^^^^^^^^^ ``spack edit -f`` -~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^ ``spack edit -f`` can be used to create a new, minimal boilerplate package: @@ -258,9 +263,9 @@ to fill in yourself: This is useful when ``spack create`` cannot figure out the name and version of your package from the archive URL. - +---------------------------- Naming & directory structure --------------------------------------- +---------------------------- .. note:: @@ -274,8 +279,9 @@ live in Spack's directory structure. In general, `spack-create`_ and `spack-edit`_ handle creating package files for you, so you can skip most of the details here. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``var/spack/repos/builtin/packages`` -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ A Spack installation directory is structured like a standard UNIX install prefix (``bin``, ``lib``, ``include``, ``var``, ``opt``, @@ -298,9 +304,9 @@ package lives in:: Alongside the ``package.py`` file, a package may contain extra directories or files (like patches) that it needs to build. - +^^^^^^^^^^^^^ Package Names -~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^ Packages are named after the directory containing ``package.py``. It is preferred, but not required, that the directory, and thus the package name, are @@ -346,8 +352,9 @@ difficult to support these options. So, you can name a package ``3proxy`` or ``_foo`` and Spack won't care. It just needs to see that name in the package spec. +^^^^^^^^^^^^^^^^^^^ Package class names -~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^ Spack loads ``package.py`` files dynamically, and it needs to find a special class name in the file for the load to succeed. The **class @@ -368,9 +375,9 @@ some examples: In general, you won't have to remember this naming convention because `spack-create`_ and `spack-edit`_ handle the details for you. - +------------------- Adding new versions ------------------------- +------------------- The most straightforward way to add new versions to your package is to add a line like this in the package class: @@ -385,8 +392,9 @@ add a line like this in the package class: Versions should be listed with the newest version first. +^^^^^^^^^^^^ Version URLs -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^ By default, each version's URL is extrapolated from the ``url`` field in the package. For example, Spack is smart enough to download @@ -423,8 +431,9 @@ construct the new one for ``8.2.1``. When you supply a custom URL for a version, Spack uses that URL *verbatim* and does not perform extrapolation. +^^^^^^^^^^^^^^^^^^^^^^^^ Skipping the expand step -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^ Spack normally expands archives automatically after downloading them. If you want to skip this step (e.g., for self-extracting @@ -452,8 +461,9 @@ it executable, then runs it with some arguments. installer = Executable(self.stage.archive_file) installer('--prefix=%s' % prefix, 'arg1', 'arg2', 'etc.') +^^^^^^^^^ Checksums -~~~~~~~~~~~~~~~~~ +^^^^^^^^^ Spack uses a checksum to ensure that the downloaded package version is not corrupted or compromised. This is especially important when @@ -465,8 +475,9 @@ Spack can currently support checksums using the MD5, SHA-1, SHA-224, SHA-256, SHA-384, and SHA-512 algorithms. It determines the algorithm to use based on the hash length. +^^^^^^^^^^^^^ ``spack md5`` -^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^ If you have one or more files to checksum, you can use the ``spack md5`` command to do it: @@ -495,8 +506,9 @@ version of this process. .. _spack-checksum: +^^^^^^^^^^^^^^^^^^ ``spack checksum`` -^^^^^^^^^^^^^^^^^^^^^^ +^^^^^^^^^^^^^^^^^^ If you want to add new versions to a package you've already created, this is automated with the ``spack checksum`` command. Here's an @@ -559,8 +571,9 @@ versions. See the documentation on `attribute_list_url`_ and .. _vcs-fetch: +------------------------------ Fetching from VCS repositories --------------------------------------- +------------------------------ For some packages, source code is provided in a Version Control System (VCS) repository rather than in a tarball. Spack can fetch packages @@ -573,8 +586,9 @@ call to your package with parameters indicating the repository URL and any branch, tag, or revision to fetch. See below for the parameters you'll need for each VCS system. +^^^^^^^^^^^^^^^^^^^^^^^^^ Repositories and versions -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^ The package author is responsible for coming up with a sensible name for each version to be fetched from a repository. For example, if @@ -607,8 +621,9 @@ supported. .. _git-fetch: +^^^ Git -~~~~~~~~~~~~~~~~~~~~ +^^^ Git fetching is enabled with the following parameters to ``version``: @@ -683,9 +698,9 @@ Submodules version('1.0.1', git='https://github.com/example-project/example.git', tag='v1.0.1', submdoules=True) - +^^^^^^^^^^ Installing -^^^^^^^^^^^^^^ +^^^^^^^^^^ You can fetch and install any of the versions above as you'd expect, by using ``@`` in a spec: @@ -700,8 +715,9 @@ a user runs ``spack info ``. .. _hg-fetch: +^^^^^^^^^ Mercurial -~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^ Fetching with mercurial works much like `git `_, but you use the ``hg`` parameter. @@ -733,8 +749,9 @@ example@`` command-line syntax. .. _svn-fetch: +^^^^^^^^^^ Subversion -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^ To fetch with subversion, use the ``svn`` and ``revision`` parameters: @@ -759,6 +776,7 @@ Fetching a revision Subversion branches are handled as part of the directory structure, so you can check out a branch or tag by changing the ``url``. +------------------------------------------------- Expanding additional resources in the source tree ------------------------------------------------- @@ -779,6 +797,7 @@ Based on the keywords present among the arguments the appropriate ``FetchStrateg will be used for the resource. The keyword ``destination`` is relative to the source root of the package and should point to where the resource is to be expanded. +------------------------------------------------------ Automatic caching of files fetched during installation ------------------------------------------------------ @@ -789,42 +808,48 @@ reinstalled on account of a change in the hashing scheme. .. _license: +----------------- Licensed software ------------------------------------------- +----------------- In order to install licensed software, Spack needs to know a few more details about a package. The following class attributes should be defined. +^^^^^^^^^^^^^^^^^^^^ ``license_required`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^ Boolean. If set to ``True``, this software requires a license. If set to ``False``, all of the following attributes will be ignored. Defaults to ``False``. +^^^^^^^^^^^^^^^^^^^ ``license_comment`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^ String. Contains the symbol used by the license manager to denote a comment. Defaults to ``#``. +^^^^^^^^^^^^^^^^^ ``license_files`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^ List of strings. These are files that the software searches for when looking for a license. All file paths must be relative to the installation directory. More complex packages like Intel may require multiple licenses for individual components. Defaults to the empty list. +^^^^^^^^^^^^^^^^ ``license_vars`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^ List of strings. Environment variables that can be set to tell the software where to look for a license if it is not in the usual location. Defaults to the empty list. +^^^^^^^^^^^^^^^ ``license_url`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^ String. A URL pointing to license setup instructions for the software. Defaults to the empty string. @@ -909,8 +934,9 @@ documentation. .. _patching: +------- Patches ------------------------------------------- +------- Depending on the host architecture, package version, known bugs, or other issues, you may need to patch your software to get it to build @@ -918,8 +944,9 @@ correctly. Like many other package systems, spack allows you to store patches alongside your package files and apply them to source code after it's downloaded. +^^^^^^^^^ ``patch`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^ You can specify patches in your package file with the ``patch()`` function. ``patch`` looks like this: @@ -999,8 +1026,9 @@ from the URL and then applied to your source code. applies cleanly with ``-p1``, but if you're using a patch you didn't create yourself, ``level`` can be handy. +^^^^^^^^^^^^^^^^^^^^^ ``patch()`` functions -~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^ In addition to supplying patch files, you can write a custom function to patch a package's source. For example, the ``py-pyside`` package @@ -1048,8 +1076,9 @@ if you run install, hit ctrl-C, and run install again, the code in the patch function is only run once. Also, you can tell Spack to run only the patching part of the build using the :ref:`spack-patch` command. +--------------- Handling RPATHs ----------------------------- +--------------- Spack installs each package in a way that ensures that all of its dependencies are found when it runs. It does this using `RPATHs @@ -1093,9 +1122,9 @@ RPATHs in Spack are handled in one of three ways: links. You can see this how this is used in the :ref:`PySide example ` above. - +-------------------- Finding new versions ----------------------------- +-------------------- You've already seen the ``homepage`` and ``url`` package attributes: @@ -1121,8 +1150,9 @@ Spack to find tarballs elsewhere. .. _attribute_list_url: +^^^^^^^^^^^^ ``list_url`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^ When spack tries to find available versions of packages (e.g. with `spack checksum `_), it spiders the parent directory @@ -1156,8 +1186,9 @@ the ``list_url``, because that is where links to old versions are: .. _attribute_list_depth: +^^^^^^^^^^^^^^ ``list_depth`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ ``libdwarf`` and many other packages have a listing of available versions on a single webpage, but not all do. For example, ``mpich`` @@ -1187,8 +1218,9 @@ when spidering the page. .. _attribute_parallel: +--------------- Parallel builds ------------------- +--------------- By default, Spack will invoke ``make()`` with a ``-j `` argument, so that builds run in parallel. It figures out how many @@ -1240,11 +1272,11 @@ you set ``parallel`` to ``False`` at the package level, then each call to ``make()`` will be sequential by default, but packagers can call ``make(parallel=True)`` to override it. - .. _dependencies: +------------ Dependencies ------------------------------- +------------ We've covered how to build a simple package, but what if one package relies on another package to build? How do you express that in a @@ -1271,8 +1303,9 @@ Spack makes this relatively easy. Let's take a look at the def install(self, spec, prefix): ... +^^^^^^^^^^^^^^^^ ``depends_on()`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^ The highlighted ``depends_on('libelf')`` call tells Spack that it needs to build and install the ``libelf`` package before it builds @@ -1280,8 +1313,9 @@ needs to build and install the ``libelf`` package before it builds guaranteed that ``libelf`` has been built and installed successfully, so you can rely on it for your libdwarf build. +^^^^^^^^^^^^^^^^ Dependency specs -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^ ``depends_on`` doesn't just take the name of another package. It takes a full spec. This means that you can restrict the versions or @@ -1345,8 +1379,9 @@ Lua module loading). .. _setup-dependent-environment: +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``setup_dependent_environment()`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Spack provides a mechanism for dependencies to provide variables that can be used in their dependents' build. Any package can declare a @@ -1412,11 +1447,11 @@ Python's ``setup_dependent_environment`` method also sets up some other variables, creates a directory, and sets up the ``PYTHONPATH`` so that dependent packages can find their dependencies at build time. - .. _packaging_extensions: +---------- Extensions -------------------------- +---------- Spack's support for package extensions is documented extensively in :ref:`extensions`. This section documents how to make your own @@ -1482,10 +1517,9 @@ linked in at activation time. ``depends_on('python')`` and ``extends(python)`` in the same package. ``extends`` implies ``depends_on``. - - +^^^^^^^^^^^^^^^^^^^^^^^^^ Activation & deactivation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^ Spack's ``Package`` class has default ``activate`` and ``deactivate`` implementations that handle symbolically linking extensions' prefixes @@ -1535,9 +1569,9 @@ Deactivate behaves similarly to activate, but it unlinks files: Both of these methods call some custom functions in the Python package. See the source for Spack's Python package for details. - +^^^^^^^^^^^^^^^^^^^^ Activation arguments -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^ You may have noticed that the ``activate`` function defined above takes keyword arguments. These are the keyword arguments from @@ -1552,11 +1586,11 @@ The only keyword argument supported by default is the ``ignore`` argument, which can take a regex, list of regexes, or a predicate to determine which files *not* to symlink during activation. - .. _virtual-dependencies: +-------------------- Virtual dependencies ------------------------------ +-------------------- In some cases, more than one package can satisfy another package's dependency. One way this can happen is if a package depends on a @@ -1577,8 +1611,9 @@ similar package files, e.g., ``foo``, ``foo-mvapich``, ``foo-mpich``, but Spack avoids this explosion of package files by providing support for *virtual dependencies*. +^^^^^^^^^^^^ ``provides`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^ In Spack, ``mpi`` is handled as a *virtual package*. A package like ``mpileaks`` can depend on it just like any other package, by @@ -1614,8 +1649,9 @@ The ``provides("mpi")`` call tells Spack that the ``mpich`` package can be used to satisfy the dependency of any package that ``depends_on('mpi')``. +^^^^^^^^^^^^^^^^^^^^ Versioned Interfaces -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^ Just as you can pass a spec to ``depends_on``, so can you pass a spec to ``provides`` to add constraints. This allows Spack to support the @@ -1636,8 +1672,9 @@ This says that ``mpich2`` provides MPI support *up to* version 2, but if a package ``depends_on("mpi@3")``, then Spack will *not* build that package with ``mpich2``. +^^^^^^^^^^^^^^^^^ ``provides when`` -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^ The same package may provide different versions of an interface depending on *its* version. Above, we simplified the ``provides`` @@ -1678,11 +1715,11 @@ Suppose a user invokes ``spack install`` like this: Spack will fail with a constraint violation, because the version of MPICH requested is too low for the ``mpi`` requirement in ``foo``. - .. _abstract-and-concrete: +------------------------- Abstract & concrete specs ------------------------------------------- +------------------------- Now that we've seen how spec constraints can be specified :ref:`on the command line ` and within package definitions, we can talk @@ -1716,7 +1753,6 @@ DAG, based on the constraints above:: ^libelf@0.8.11 ^mpi - .. graphviz:: digraph { @@ -1727,7 +1763,6 @@ DAG, based on the constraints above:: dyninst -> "libelf@0.8.11" } - This diagram shows a spec DAG output as a tree, where successive levels of indentation represent a depends-on relationship. In the above DAG, we can see some packages annotated with their constraints, @@ -1735,8 +1770,9 @@ and some packages with no annotations at all. When there are no annotations, it means the user doesn't care what configuration of that package is built, just so long as it works. +^^^^^^^^^^^^^^ Concretization -~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ An abstract spec is useful for the user, but you can't install an abstract spec. Spack has to take the abstract spec and "fill in" the @@ -1776,8 +1812,9 @@ the preferences of their own users. .. _spack-spec: +^^^^^^^^^^^^^^ ``spack spec`` -~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ For an arbitrary spec, you can see the result of concretization by running ``spack spec``. For example: @@ -1796,9 +1833,9 @@ running ``spack spec``. For example: This is useful when you want to know exactly what Spack will do when you ask for a particular spec. - +^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``Concretization Policies`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^ A user may have certain preferences for how packages should be concretized on their system. For example, one user may prefer packages @@ -1810,8 +1847,9 @@ for more details. .. _install-method: +----------------------------------- Implementing the ``install`` method ------------------------------------------- +----------------------------------- The last element of a package is its ``install()`` method. This is where the real work of installation happens, and it's the main part of @@ -1825,15 +1863,12 @@ the package you'll need to customize for each piece of software. be built, and a ``prefix``: the path to the directory where the software should be installed. - Spack provides wrapper functions for ``configure`` and ``make`` so that you can call them in a similar way to how you'd call a shell command. In reality, these are Python functions. Spack provides these functions to make writing packages more natural. See the section on :ref:`shell wrappers `. - - Now that the metadata is out of the way, we can move on to the ``install()`` method. When a user runs ``spack install``, Spack fetches an archive for the correct version of the software, expands @@ -1883,8 +1918,9 @@ information. .. _install-environment: +----------------------- The install environment --------------------------- +----------------------- In general, you should not have to do much differently in your install method than you would when installing a package on the command line. @@ -1901,8 +1937,9 @@ custom Makefiles, you may need to add logic to modify the makefiles. The remainder of the section covers the way Spack's build environment works. +^^^^^^^^^^^^^^^^^^^^^ Environment variables -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^ Spack sets a number of standard environment variables that serve two purposes: @@ -1970,8 +2007,9 @@ below. .. _compiler-wrappers: +^^^^^^^^^^^^^^^^^^^^^ Compiler interceptors -~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^ As mentioned, ``CC``, ``CXX``, ``F77``, and ``FC`` are set to point to Spack's compiler wrappers. These are simply called ``cc``, ``c++``, @@ -2037,8 +2075,10 @@ and/or ``ldlibs``). They do not override the canonical autotools flags with the same names (but in ALL-CAPS) that may be passed into the build by particularly challenging package scripts. +^^^^^^^^^^^^^^ Compiler flags -~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ + In rare circumstances such as compiling and running small unit tests, a package developer may need to know what are the appropriate compiler flags to enable features like ``OpenMP``, ``c++11``, ``c++14`` and alike. To that end the @@ -2054,8 +2094,10 @@ package supports additional variants like variant('openmp', default=True, description="Enable OpenMP support.") +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Message Parsing Interface (MPI) -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + It is common for high performance computing software/packages to use ``MPI``. As a result of conretization, a given package can be built using different implementations of MPI such as ``Openmpi``, ``MPICH`` or ``IntelMPI``. @@ -2070,9 +2112,10 @@ Package developers are advised to use these variables, for example ``self.spec[' instead of hard-coding ``join_path(self.spec['mpi'].prefix.bin, 'mpicc')`` for the reasons outlined above. - +^^^^^^^^^^^^^^^^^^^^^^^^^ Blas and Lapack libraries -~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^ + Different packages provide implementation of ``Blas`` and ``Lapack`` routines. The names of the resulting static and/or shared libraries differ from package to package. In order to make ``install()`` method indifferent to the @@ -2083,9 +2126,9 @@ applies to packages which provide ``Lapack``. Package developers are advised to use these variables, for example ``spec['blas'].blas_shared_lib`` instead of hard-coding ``join_path(spec['blas'].prefix.lib, 'libopenblas.so')``. - +^^^^^^^^^^^^^^^^^^^^^ Forking ``install()`` -~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^ To give packagers free reign over their install environment, Spack forks a new process each time it invokes a package's ``install()`` @@ -2097,9 +2140,9 @@ dedicated process. .. _prefix-objects: - +----------------- Failing the build ----------------------- +----------------- Sometimes you don't want a package to successfully install unless some condition is true. You can explicitly cause the build to fail from @@ -2110,9 +2153,9 @@ condition is true. You can explicitly cause the build to fail from if spec.architecture.startswith('darwin'): raise InstallError('This package does not build on Mac OS X!') - +-------------- Prefix objects ----------------------- +-------------- Spack passes the ``prefix`` parameter to the install method so that you can pass it to ``configure``, ``cmake``, or some other installer, @@ -2122,7 +2165,6 @@ e.g.: configure('--prefix=' + prefix) - For the most part, prefix objects behave exactly like strings. For packages that do not have their own install target, or for those that implement it poorly (like ``libdwarf``), you may need to manually copy @@ -2142,7 +2184,6 @@ yourself, e.g.: mkdirp(prefix.lib) install('libfoo.a', prefix.lib) - Most of the standard UNIX directory names are attributes on the ``prefix`` object. Here is a full list: @@ -2169,8 +2210,9 @@ Most of the standard UNIX directory names are attributes on the .. _spec-objects: +------------ Spec objects -------------------------- +------------ When ``install`` is called, most parts of the build process are set up for you. The correct version's tarball has been downloaded and @@ -2187,8 +2229,9 @@ special parameters to ``configure``, like need to supply special compiler flags depending on the compiler. All of this information is available in the spec. +^^^^^^^^^^^^^^^^^^^^^^^^ Testing spec constraints -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^ You can test whether your spec is configured a certain way by using the ``satisfies`` method. For example, if you want to check whether @@ -2253,9 +2296,9 @@ the two functions is that ``satisfies()`` tests whether spec constraints overlap at all, while ``in`` tests whether a spec or any of its dependencies satisfy the provided spec. - +^^^^^^^^^^^^^^^^^^^^^^ Accessing Dependencies -~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^ You may need to get at some file or binary that's in the prefix of one of your dependencies. You can do that by sub-scripting the spec: @@ -2277,8 +2320,9 @@ would do with the package's own spec: .. _multimethods: +^^^^^^^^^^^^^^^^^^^^^^^^^^ Multimethods and ``@when`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^ Spack allows you to make multiple versions of instance functions in packages, based on whether the package's spec satisfies particular @@ -2377,8 +2421,9 @@ method (the one without the ``@when`` decorator) will be called. .. _shell-wrappers: +----------------------- Shell command functions ----------------------------- +----------------------- Recall the install method from ``libelf``: @@ -2426,11 +2471,11 @@ to the ``make`` wrapper to disable parallel make. In the ``libelf`` package, this allows us to avoid race conditions in the library's build system. - .. _sanity-checks: +------------------------------- Sanity checking an installation --------------------------------- +------------------------------- By default, Spack assumes that a build has failed if nothing is written to the install prefix, and that it has succeeded if anything @@ -2455,9 +2500,9 @@ like this can falsely report that they were successfully installed if an error occurs before the install is complete but after files have been written to the ``prefix``. - +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ``sanity_check_is_file`` and ``sanity_check_is_dir`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ You can optionally specify *sanity checks* to deal with this problem. Add properties like this to your package: @@ -2482,11 +2527,11 @@ the build will fail and the install prefix will be removed. If they succeed, Spack considers the build succeeful and keeps the prefix in place. - .. _file-manipulation: +--------------------------- File manipulation functions ------------------------------- +--------------------------- Many builds are not perfect. If a build lacks an install target, or if it does not use systems like CMake or autotools, which have standard @@ -2508,9 +2553,9 @@ running: This is already part of the boilerplate for packages created with ``spack create`` or ``spack edit``. - +^^^^^^^^^^^^^^^^^^^ Filtering functions -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^ :py:func:`filter_file(regex, repl, *filenames, **kwargs) ` Works like ``sed`` but with Python regular expression syntax. Takes @@ -2568,9 +2613,9 @@ Filtering functions change_sed_delimiter('@', ';', 'utils/FixMakefile') change_sed_delimiter('@', ';', 'utils/FixMakefile.sed.default') - +^^^^^^^^^^^^^^ File functions -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^ :py:func:`ancestor(dir, n=1) ` Get the n\ :sup:`th` ancestor of the directory ``dir``. @@ -2638,22 +2683,21 @@ File functions The ``create=True`` keyword argument causes the command to create the directory if it does not exist. - :py:func:`touch(path) ` Create an empty file at ``path``. - .. _package-lifecycle: +----------------------- Coding Style Guidelines ---------------------------- +----------------------- The following guidelines are provided, in the interests of making Spack packages work in a consistent manner: - +^^^^^^^^^^^^^ Variant Names -~~~~~~~~~~~~~~ +^^^^^^^^^^^^^ Spack packages with variants similar to already-existing Spack packages should use the same name for their variants. Standard @@ -2671,14 +2715,15 @@ variant names are: If specified in this table, the corresponding default should be used when declaring a variant. - +^^^^^^^^^^^^^ Version Lists -~~~~~~~~~~~~~~ +^^^^^^^^^^^^^ Spack packges should list supported versions with the newest first. +^^^^^^^^^^^^^^^^ Special Versions -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^ The following *special* version names may be used when building a package: @@ -2697,9 +2742,9 @@ The following *special* version names may be used when building a package: * *@local*: Indicates the version was built manually from some source tree of unknown provenance (see ``spack setup``). - +--------------------------- Packaging workflow commands ---------------------------------- +--------------------------- When you are building packages, you will likely not get things completely right the first time. @@ -2729,8 +2774,9 @@ control over the install process. .. _spack-fetch: +^^^^^^^^^^^^^^^ ``spack fetch`` -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^ The first step of ``spack install``. Takes a spec and determines the correct download URL to use for the requested package version, then @@ -2743,8 +2789,9 @@ fetch`` is idempotent and will not download the archive again. .. _spack-stage: +^^^^^^^^^^^^^^^ ``spack stage`` -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^ The second step in ``spack install`` after ``spack fetch``. Expands the downloaded archive in its temporary directory, where it will be @@ -2753,8 +2800,9 @@ already been expanded, ``stage`` is idempotent. .. _spack-patch: +^^^^^^^^^^^^^^^ ``spack patch`` -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^ After staging, Spack applies patches to downloaded packages, if any have been specified in the package file. This command will run the @@ -2766,8 +2814,10 @@ package before patching. .. _spack-restage: +^^^^^^^^^^^^^^^^^ ``spack restage`` -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^ + Restores the source code to pristine state, as it was before building. Does this in one of two ways: @@ -2780,18 +2830,21 @@ Does this in one of two ways: .. _spack-clean: +^^^^^^^^^^^^^^^ ``spack clean`` -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^ + Cleans up temporary files for a particular package, by deleting the expanded/checked out source code *and* any downloaded archive. If ``fetch``, ``stage``, or ``install`` are run again after this, Spack's build process will start from scratch. - .. _spack-purge: +^^^^^^^^^^^^^^^ ``spack purge`` -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^ + Cleans up all of Spack's temporary and cached files. This can be used to recover disk space if temporary files from interrupted or failed installs accumulate in the staging area. @@ -2803,8 +2856,9 @@ running ``spack clean`` for every package you have fetched or staged. When called with ``--cache`` or ``--all`` this will clear all resources :ref:`cached ` during installs. +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Keeping the stage directory on success -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ By default, ``spack install`` will delete the staging area once a package has been successfully built and installed. Use @@ -2818,9 +2872,9 @@ This allows you to inspect the build directory and potentially debug the build. You can use ``purge`` or ``clean`` later to get rid of the unwanted temporary files. - +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Keeping the install prefix on failure -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ By default, ``spack install`` will delete any partially constructed install prefix if anything fails during ``install()``. If you want to @@ -2839,14 +2893,15 @@ to get rid of the install prefix before you build again: spack uninstall -f - +--------------------- Graphing dependencies --------------------------- +--------------------- .. _spack-graph: +^^^^^^^^^^^^^^^ ``spack graph`` -~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^ Spack provides the ``spack graph`` command for graphing dependencies. The command by default generates an ASCII rendering of a spec's @@ -2919,8 +2974,9 @@ example:: This graph can be provided as input to other graphing tools, such as those in `Graphviz `_. +------------------------- Interactive shell support --------------------------- +------------------------- Spack provides some limited shell support to make life easier for packagers. You can enable these commands by sourcing a setup file in @@ -2937,8 +2993,9 @@ For ``csh`` and ``tcsh`` run: .. _spack-cd: +^^^^^^^^^^^^ ``spack cd`` -~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^ ``spack cd`` allows you to quickly cd to pertinent directories in Spack. Suppose you've staged a package but you want to modify it before you @@ -2968,8 +3025,9 @@ the main python source directory of your spack install. .. _spack-env: +^^^^^^^^^^^^^ ``spack env`` -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^ ``spack env`` functions much like the standard unix ``env`` command, but it takes a spec as an argument. You can use it to see the @@ -2993,11 +3051,11 @@ To run commands in a package's build environment, you can simply provided them a This will cd to the build directory and then run ``configure`` in the package's build environment. - .. _spack-location: +^^^^^^^^^^^^^^^^^^ ``spack location`` -~~~~~~~~~~~~~~~~~~~~~~ +^^^^^^^^^^^^^^^^^^ ``spack location`` is the same as ``spack cd`` but it does not require shell support. It simply prints out the path you ask for, rather than @@ -3019,6 +3077,7 @@ might write: CXXFLAGS += -I$DWARF_PREFIX/include CXXFLAGS += -L$DWARF_PREFIX/lib +---------------------------------- Build System Configuration Support ---------------------------------- @@ -3064,8 +3123,9 @@ Notes: to manually re-run ``spconfig.py`` when a ``CMakeLists.txt`` file has changed. +^^^^^^^^^^^^ CMakePackage -~~~~~~~~~~~~ +^^^^^^^^^^^^ In order ot enable ``spack setup`` functionality, the author of ``myproject/package.py`` must subclass from ``CMakePackage`` instead @@ -3091,9 +3151,9 @@ translate variant flags into CMake definitions. For example: If needed, a packager may also override methods defined in ``StagedPackage`` (see below). - +^^^^^^^^^^^^^ StagedPackage -~~~~~~~~~~~~~ +^^^^^^^^^^^^^ ``CMakePackage`` is implemented by subclassing the ``StagedPackage`` superclass, which breaks down the standard ``Package.install()`` @@ -3115,8 +3175,9 @@ and ``install``. Details: ``install()`` method may be accessed via ``self.spec`` and ``self.prefix``. +^^^^^^^^^^^^^ GNU Autotools -~~~~~~~~~~~~~ +^^^^^^^^^^^^^ The ``setup`` functionality is currently only available for CMake-based packages. Extending this functionality to GNU