hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Merge branch 'features/new-docs' into develop

Browse source
This commit is contained in:
Todd Gamblin 2014-10-27 20:44:00 -07:00
commit d542b7b003
3 changed files with 427 additions and 40 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

268
lib/spack/docs/basic_usage.rst
View file

@ -10,25 +10,6 @@ Only a small subset of commands are needed for typical usage.
This section covers a small set of subcommands that should cover most
general use cases for Spack.
Getting Help
-----------------------
``spack help``
~~~~~~~~~~~~~~~~~~~~~~
The ``help`` subcommand will print out out a list of all of
``spack``'s options and subcommands:
.. command-output:: spack help
Adding an argument, e.g. ``spack help <subcommand>``, will print out
usage information for a particular subcommand:
.. command-output:: spack help install
Alternately, you can use ``spack -h`` in place of ``spack help``, or
``spack <subcommand> -h`` to get help on a particular subcommand.
Listing available packages
------------------------------
@ -327,19 +308,19 @@ completely remove the directory in which the package was installed.
spack uninstall mpich
If there are still installed packages that depend on the package to be
uninstalled, spack will issue a warning. In general, it is safer to
remove dependent packages *before* removing their dependencies. Not
doing so risks breaking packages on your system. To remove a package
without regard for its dependencies, run ``spack uninstall -f
<package>`` to override the warning.
uninstalled, spack will refuse to uninstall. If you know what you're
doing, you can override this with ``spack uninstall -f <package>``.
However, running this risks breaking other installed packages. In
general, it is safer to remove dependent packages *before* removing
their dependencies.
A line like ``spack uninstall mpich`` may be ambiguous, if multiple
``mpich`` configurations are installed. For example, if both
``mpich@3.0.2`` and ``mpich@3.1`` are installed, it could refer to
either one, and Spack cannot determine which one to uninstall. Spack
will ask you to provide a version number to remove any ambiguity. For
example, ``spack uninstall mpich@3.1`` is unambiguous in the
above scenario.
will ask you to provide a version number to remove the ambiguity. For
example, ``spack uninstall mpich@3.1`` is unambiguous in the above
scenario.
.. _sec-specs:
@ -657,3 +638,236 @@ add a version specifier to the spec:
Notice that the package versions that provide insufficient MPI
versions are now filtered out.
.. _shell-support:
Environment Modules
-------------------------------
.. note::
Environment module support is currently experimental and should not
be considered a stable feature of Spack. In particular, the
interface and/or generated module names may change in future
versions.
Spack provides some limited integration with environment module
systems to make it easier to use the packages it provides.
You can enable shell support by sourcing some files in the
``/share/spack`` directory.
For ``bash`` or ``ksh``, run:
.. code-block:: sh
. $SPACK_ROOT/share/spack/setup-env.sh
For ``csh`` and ``tcsh`` run:
.. code-block:: csh
setenv SPACK_ROOT /path/to/spack
source $SPACK_ROOT/share/spack/setup-env.csh
You can put the above code in your ``.bashrc`` or ``.cshrc``, and
Spack's shell support will be available on the command line.
-------------------------------
When you install a package with Spack, it automatically generates an
environment module that lets you add the package to your environment.
Currently, Spack supports the generation of `TCL Modules
<http://wiki.tcl.tk/12999>`_ and `Dotkit
<https://computing.llnl.gov/?set=jobs&page=dotkit>`_. Generated
module files for each of these systems can be found in these
directories:
* ``$SPACK_ROOT/share/spack/modules``
* ``$SPACK_ROOT/share/spack/dotkit``
The directories are automatically added to your ``MODULEPATH`` and
``DK_NODE`` environment variables when you enable Spack's `shell
support <shell-support_>`_.
Using Modules & Dotkits
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If you have shell support enabled you should be able to run either
``module avail`` or ``use -l spack`` to see what modules/dotkits have
been installed. Here is sample output of those programs, showing lots
of installed packages.
.. code-block:: sh
$ module avail
------- /g/g21/gamblin2/src/spack/share/spack/modules/chaos_5_x86_64_ib --------
adept-utils@1.0%gcc@4.4.7-5adef8da libelf@0.8.13%gcc@4.4.7
automaded@1.0%gcc@4.4.7-d9691bb0 libelf@0.8.13%intel@15.0.0
boost@1.55.0%gcc@4.4.7 mpc@1.0.2%gcc@4.4.7-559607f5
callpath@1.0.1%gcc@4.4.7-5dce4318 mpfr@3.1.2%gcc@4.4.7
dyninst@8.1.2%gcc@4.4.7-b040c20e mpich@3.0.4%gcc@4.4.7
gcc@4.9.1%gcc@4.4.7-93ab98c5 mpich@3.0.4%gcc@4.9.0
gmp@6.0.0a%gcc@4.4.7 mrnet@4.1.0%gcc@4.4.7-72b7881d
graphlib@2.0.0%gcc@4.4.7 netgauge@2.4.6%gcc@4.9.0-27912b7b
launchmon@1.0.1%gcc@4.4.7 stat@2.1.0%gcc@4.4.7-51101207
libNBC@1.1.1%gcc@4.9.0-27912b7b sundials@2.5.0%gcc@4.9.0-27912b7b
libdwarf@20130729%gcc@4.4.7-b52fac98
.. code-block:: sh
$ use -l spack
spack ----------
adept-utils@1.0%gcc@4.4.7-5adef8da - adept-utils @1.0
automaded@1.0%gcc@4.4.7-d9691bb0 - automaded @1.0
boost@1.55.0%gcc@4.4.7 - boost @1.55.0
callpath@1.0.1%gcc@4.4.7-5dce4318 - callpath @1.0.1
dyninst@8.1.2%gcc@4.4.7-b040c20e - dyninst @8.1.2
gmp@6.0.0a%gcc@4.4.7 - gmp @6.0.0a
libNBC@1.1.1%gcc@4.9.0-27912b7b - libNBC @1.1.1
libdwarf@20130729%gcc@4.4.7-b52fac98 - libdwarf @20130729
libelf@0.8.13%gcc@4.4.7 - libelf @0.8.13
libelf@0.8.13%intel@15.0.0 - libelf @0.8.13
mpc@1.0.2%gcc@4.4.7-559607f5 - mpc @1.0.2
mpfr@3.1.2%gcc@4.4.7 - mpfr @3.1.2
mpich@3.0.4%gcc@4.4.7 - mpich @3.0.4
mpich@3.0.4%gcc@4.9.0 - mpich @3.0.4
netgauge@2.4.6%gcc@4.9.0-27912b7b - netgauge @2.4.6
sundials@2.5.0%gcc@4.9.0-27912b7b - sundials @2.5.0
The names here should look familiar, they're the same ones from
``spack find``. You *can* use the names here directly. For example,
you could type either of these commands to load the callpath module
(assuming dotkit and modules are installed):
.. code-block:: sh
use callpath@1.0.1%gcc@4.4.7-5dce4318
.. code-block:: sh
module load callpath@1.0.1%gcc@4.4.7-5dce4318
Neither of these is particularly pretty, easy to remember, or
easy to type. Luckily, Spack has its own interface for using modules
and dotkits. You can use the same spec syntax you're used to:
========================= ==========================
Modules Dotkit
========================= ==========================
``spack load <spec>`` ``spack use <spec>``
``spack unload <spec>`` ``spack unuse <spec>``
========================= ==========================
And you can use the same shortened names you use everywhere else in
Spack. For example, this will add the ``mpich`` package built with
``gcc`` to your path:
.. code-block:: sh
$ spack install mpich %gcc@4.4.7
# ... wait for install ...
$ spack use mpich %gcc@4.4.7
Prepending: mpich@3.0.4%gcc@4.4.7 (ok)
$ which mpicc
~/src/spack/opt/chaos_5_x86_64_ib/gcc@4.4.7/mpich@3.0.4/bin/mpicc
Or, similarly with modules, you could type:
.. code-block:: sh
$ spack load mpich %gcc@4.4.7
These commands will add appropriate directories to your ``PATH``,
``MANPATH``, and ``LD_LIBRARY_PATH``. When you no longer want to use
a package, you can type unload or unuse similarly:
.. code-block:: sh
$ spack unload mpich %gcc@4.4.7 # modules
$ spack unuse mpich %gcc@4.4.7 # dotkit
.. note::
These ``use``, ``unuse``, ``load``, and ``unload`` subcommands are
only available if you have enabled Spack's shell support *and* you
have dotkit or modules installed on your machine.
Ambiguous module names
~~~~~~~~~~~~~~~~~~~~~~~~
If a spec used with load/unload or use/unuse is ambiguous (i.e. more
than one installed package matches it), then Spack will warn you:
.. code-block:: sh
$ spack load libelf
==> Error: Multiple matches for spec libelf. Choose one:
libelf@0.8.13%gcc@4.4.7=chaos_5_x86_64_ib
libelf@0.8.13%intel@15.0.0=chaos_5_x86_64_ib
You can either type the ``spack load`` command again with a fully
qualified argument, or you can add just enough extra constraints to
identify one package. For example, above, the key differentiator is
that one ``libelf`` is built with the Intel compiler, while the other
used ``gcc``. You could therefore just type:
.. code-block:: sh
$ spack load libelf %intel
To identify just the one built with the Intel compiler.
Regenerating Module files
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Module and dotkit files are generated when packages are installed, and
are placed in the following directories under the Spack root:
* ``$SPACK_ROOT/share/spack/modules``
* ``$SPACK_ROOT/share/spack/dotkit``
Sometimes you may need to regenerate the modules files. For example,
if newer, fancier module support is added to Spack at some later date,
you may want to regenerate all the modules to take advantage of these
new features.
``spack module refresh``
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Running ``spack module refresh`` will remove the
``share/spack/modules`` and ``share/spack/dotkit`` directories, then
regenerate all module and dotkit files from scratch:
.. code-block:: sh
$ spack module refresh
==> Regenerating tcl module files.
==> Regenerating dotkit module files.
Getting Help
-----------------------
``spack help``
~~~~~~~~~~~~~~~~~~~~~~
If you don't find what you need here, the ``help`` subcommand will
print out out a list of *all* of ``spack``'s options and subcommands:
.. command-output:: spack help
Adding an argument, e.g. ``spack help <subcommand>``, will print out
usage information for a particular subcommand:
.. command-output:: spack help install
Alternately, you can use ``spack -h`` in place of ``spack help``, or
``spack <subcommand> -h`` to get help on a particular subcommand.

197
lib/spack/docs/packaging_guide.rst
View file

@ -768,9 +768,9 @@ information about the package, and to determine where to download its
source code.
Spack uses the tarball URL to extrapolate where to find other tarballs
of the same package (e.g. in `spack-checksum`_, but this does not
always work. This section covers ways you can tell Spack to find
tarballs elsewhere.
of the same package (e.g. in `spack checksum <spack-checksum_>`_, but
this does not always work. This section covers ways you can tell
Spack to find tarballs elsewhere.
.. _attribute_list_url:
@ -778,8 +778,9 @@ tarballs elsewhere.
~~~~~~~~~~~~~~~~~~~~~
When spack tries to find available versions of packages (e.g. with
`spack-checksum`_), it spiders the parent directory of the tarball in
the ``url`` attribute. For example, for libelf, the url is:
`spack checksum <spack-checksum_>`_), it spiders the parent directory
of the tarball in the ``url`` attribute. For example, for libelf, the
url is:
.. code-block:: python
@ -1335,8 +1336,8 @@ If your build system does *not* automatically pick these variables up
from the environment, then you can simply pass them on the command
line or use a patch as part of your build process to get the correct
compilers into the project's build system. There are also some file
editing commands you can use -- these are described later in
`filtering-files`_.
editing commands you can use -- these are described later in the
`section on file manipulation <file-manipulation_>`_.
In addition to the compiler variables, these variables are set before
entering ``install()`` so that packages can locate dependencies
@ -1466,9 +1467,28 @@ yourself, e.g.:
Most of the standard UNIX directory names are attributes on the
``prefix`` object. See :py:class:`spack.prefix.Prefix` for a full
list.
``prefix`` object. Here is a full list:
========================= ================================================
Prefix Attribute Location
========================= ================================================
``prefix.bin`` ``$prefix/bin``
``prefix.sbin`` ``$prefix/sbin``
``prefix.etc`` ``$prefix/etc``
``prefix.include`` ``$prefix/include``
``prefix.lib`` ``$prefix/lib``
``prefix.lib64`` ``$prefix/lib64``
``prefix.libexec`` ``$prefix/libexec``
``prefix.share`` ``$prefix/share``
``prefix.doc`` ``$prefix/doc``
``prefix.info`` ``$prefix/info``
``prefix.man`` ``$prefix/man``
``prefix.man[1-8]`` ``$prefix/man/man[1-8]``
``prefix.share_man`` ``$prefix/share/man``
``prefix.share_man[1-8]`` ``$prefix/share/man[1-8]``
========================= ================================================
.. _spec-objects:
@ -1678,11 +1698,10 @@ method (the one without the ``@when`` decorator) will be called.
the way decorators work.
.. _shell-wrappers:
Shell command wrappers
-------------------------
Shell command functions
----------------------------
Recall the install method from ``libelf``:
@ -1730,9 +1749,161 @@ to the ``make`` wrapper to disable parallel make. In the ``libelf``
package, this allows us to avoid race conditions in the library's
build system.
.. _file-manipulation:
File manipulation functions
------------------------------
Many builds are not perfect. If a build lacks an install target, or if
it does not use systems like CMake or autotools, which have standard
ways of setting compilers and options, you may need to edit files or
install some files yourself to get them working with Spack.
You can do this with standard Python code, and Python has rich
libraries with functions for file manipulation and filtering. Spack
also provides a number of convenience functions of its own to make
your life even easier. These functions are described in this section.
All of the functions in this section can be included by simply
running:
.. code-block:: python
from spack import *
This is already part of the boilerplate for packages created with
``spack create`` or ``spack edit``.
Filtering functions
~~~~~~~~~~~~~~~~~~~~~~
:py:func:`filter_file(regex, repl, *filenames, **kwargs) <spack.filter_file>`
Works like ``sed`` but with Python regular expression syntax. Takes
a regular expression, a replacement, and a set of files. ``repl``
can be a raw string or a callable function. If it is a raw string,
it can contain ``\1``, ``\2``, etc. to refer to capture groups in
the regular expression. If it is a callable, it is passed the
Python ``MatchObject`` and should return a suitable replacement
string for the particular match.
Examples:
#. Replacing ``#!/usr/bin/perl`` with ``#!/usr/bin/env perl`` in ``bib2xhtml``:
.. code-block:: python
filter_file(r'#!/usr/bin/perl',
'#!/usr/bin/env perl', join_path(prefix.bin, 'bib2xhtml'))
#. Switching the compilers used by ``mpich``'s MPI wrapper scripts from
``cc``, etc. to the compilers used by the Spack build:
.. code-block:: python
filter_file('CC="cc"', 'CC="%s"' % self.compiler.cc,
join_path(prefix.bin, 'mpicc'))
filter_file('CXX="c++"', 'CXX="%s"' % self.compiler.cxx,
join_path(prefix.bin, 'mpicxx'))
:py:func:`change_sed_delimiter(old_delim, new_delim, *filenames) <spack.change_sed_delim>`
Some packages, like TAU, have a build system that can't install
into directories with, e.g. '@' in the name, because they use
hard-coded ``sed`` commands in their build.
``change_sed_delimiter`` finds all ``sed`` search/replace commands
and change the delimiter. e.g., if the file contains commands
that look like ``s///``, you can use this to change them to
``s@@@``.
Example of changing ``s///`` to ``s@@@`` in TAU:
.. code-block:: python
change_sed_delimiter('@', ';', 'configure')
change_sed_delimiter('@', ';', 'utils/FixMakefile')
change_sed_delimiter('@', ';', 'utils/FixMakefile.sed.default')
File functions
~~~~~~~~~~~~~~~~~~~~~~
:py:func:`ancestor(dir, n=1) <spack.ancestor>`
Get the n\ :sup:`th` ancestor of the directory ``dir``.
:py:func:`can_access(path) <spack.can_access>`
True if we can read and write to the file at ``path``. Same as
native python ``os.access(file_name, os.R_OK|os.W_OK)``.
:py:func:`install(src, dest) <spack.install>`
Install a file to a particular location. For example, install a
header into the ``include`` directory under the install ``prefix``:
.. code-block:: python
install('my-header.h', join_path(prefix.include))
:py:func:`join_path(prefix, *args) <spack.join_path>` Like
``os.path.join``, this joins paths using the OS path separator.
However, this version allows an arbitrary number of arguments, so
you can string together many path components.
:py:func:`mkdirp(*paths) <spack.mkdirp>`
Create each of the directories in ``paths``, creating any parent
directories if they do not exist.
:py:func:`working_dir(dirname, kwargs) <spack.working_dir>`
This is a Python `Context Manager
<https://docs.python.org/2/library/contextlib.html>`_ that makes it
easier to work with subdirectories in builds. You use this with the
Python ``with`` statement to change into a working directory, and
when the with block is done, you change back to the original
directory. Think of it as a safe ``pushd`` / ``popd`` combination,
where ``popd`` is guaranteed to be called at the end, even if
exceptions are thrown.
Example usage:
#. The ``libdwarf`` build first runs ``configure`` and ``make`` in a
subdirectory called ``libdwarf``. It then implements the
installation code itself. This is natural with ``working_dir``:
.. code-block:: python
with working_dir('libdwarf'):
configure("--prefix=" + prefix, "--enable-shared")
make()
install('libdwarf.a', prefix.lib)
#. Many CMake builds require that you build "out of source", that
is, in a subdirectory. You can handle creating and ``cd``'ing to
the subdirectory like the LLVM package does:
.. code-block:: python
with working_dir('spack-build', create=True):
cmake('..',
'-DLLVM_REQUIRES_RTTI=1',
'-DPYTHON_EXECUTABLE=/usr/bin/python',
'-DPYTHON_INCLUDE_DIR=/usr/include/python2.6',
'-DPYTHON_LIBRARY=/usr/lib64/libpython2.6.so',
*std_cmake_args)
make()
make("install")
The ``create=True`` keyword argument causes the command to create
the directory if it does not exist.
:py:func:`touch(path) <spack.touch>`
Create an empty file at ``path``.
.. _pacakge-lifecycle:
Useful Packaging Commands
Package Workflow Commands
---------------------------------
When you are building packages, you will likely not get things

2
lib/spack/llnl/util/filesystem.py
View file

@ -144,6 +144,7 @@ def expand_user(path):
def mkdirp(*paths):
"""Creates a directory, as well as parent directories if needed."""
for path in paths:
if not os.path.exists(path):
os.makedirs(path)
@ -163,6 +164,7 @@ def working_dir(dirname, **kwargs):
def touch(path):
"""Creates an empty file at the specified path."""
with closing(open(path, 'a')) as file:
os.utime(path, None)