Start of basic documentation

This commit is contained in:
Todd Gamblin 2013-12-12 04:25:31 -08:00
parent c7539fe950
commit c9e40b725d
23 changed files with 945 additions and 130 deletions

View file

@ -35,7 +35,7 @@ parser.add_argument('-m', '--mock', action='store_true', dest='mock',
# each command module implements a parser() function, to which we pass its # each command module implements a parser() function, to which we pass its
# subparser for setup. # subparser for setup.
subparsers = parser.add_subparsers(metavar='COMMAND', dest="command") subparsers = parser.add_subparsers(metavar='SUBCOMMAND', dest="command")
import spack.cmd import spack.cmd
for cmd in spack.cmd.commands: for cmd in spack.cmd.commands:

2
lib/spack/docs/.gitignore vendored Normal file
View file

@ -0,0 +1,2 @@
spack*.rst
_build

View file

@ -7,6 +7,9 @@ SPHINXBUILD = sphinx-build
PAPER = PAPER =
BUILDDIR = _build BUILDDIR = _build
export PYTHONPATH = ../../spack
APIDOC_FILES = spack*.rst
# Internal variables. # Internal variables.
PAPEROPT_a4 = -D latex_paper_size=a4 PAPEROPT_a4 = -D latex_paper_size=a4
PAPEROPT_letter = -D latex_paper_size=letter PAPEROPT_letter = -D latex_paper_size=letter
@ -14,10 +17,13 @@ ALLSPHINXOPTS = -d $(BUILDDIR)/doctrees $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
# the i18n builder cannot share the environment and doctrees with the others # the i18n builder cannot share the environment and doctrees with the others
I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) . I18NSPHINXOPTS = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
.PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext .PHONY: help clean html dirhtml singlehtml pickle json htmlhelp qthelp devhelp epub latex latexpdf text man changes linkcheck doctest gettext apidoc
all: html all: html
apidoc:
sphinx-apidoc -T -o . $(PYTHONPATH)/spack
help: help:
@echo "Please use \`make <target>' where <target> is one of" @echo "Please use \`make <target>' where <target> is one of"
@echo " html to make standalone HTML files" @echo " html to make standalone HTML files"
@ -41,9 +47,9 @@ help:
@echo " doctest to run all doctests embedded in the documentation (if enabled)" @echo " doctest to run all doctests embedded in the documentation (if enabled)"
clean: clean:
-rm -rf $(BUILDDIR)/* -rm -rf $(BUILDDIR)/* $(APIDOC_FILES)
html: html: apidoc
$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html $(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html
@echo @echo
@echo "Build finished. The HTML pages are in $(BUILDDIR)/html." @echo "Build finished. The HTML pages are in $(BUILDDIR)/html."

View file

@ -24,7 +24,7 @@
{%- if last_updated %} {%- if last_updated %}
{% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %} {% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
{%- endif %} {%- endif %}
</p> &nbsp;&nbsp;
{% trans %}<a href="https://www.github.com/snide/sphinx_rtd_theme">Sphinx theme</a> provided by <a href="http://readthedocs.org">Read the Docs</a>{% endtrans %} {% trans %}<a href="https://www.github.com/snide/sphinx_rtd_theme">Sphinx theme</a> provided by <a href="http://readthedocs.org">Read the Docs</a>{% endtrans %}
</p>
</footer> </footer>

View file

@ -0,0 +1,233 @@
Basic usage
=====================
Nearly everything you do wtih spack will involve the ``spack``
command. Like many well-known tools (``git``, ``cvs``, ``svn``,
``yum``, ``port``, ``apt-get``, etc.), ``spack`` is generally called
with a *subcommand* indicating the action you want to perform.
Getting Help
-----------------------
``spack help``
~~~~~~~~~~~~~~~~~~~~~~
The first subcommand you should know is ``spack help``. Run with no
arguments, it will give a list of all spack options and subcommands:
.. command-output:: spack help
If you want help on the usage of a particular subcommand, you can pass
it as an argument to ``spack help``:
.. command-output:: spack help install
Alternately, you can use ``spack -h`` in place of ``spack help``, or
``spack <subcommand> -h`` to get help on a particular subcommand.
Viewing available packages
------------------------------
The first thing you will likely want to do with spack is find out what
software is available to install. There are two main commands for
this: ``spack list`` and ``spack info``.
``spack list``
~~~~~~~~~~~~~~~~
The ``spack list`` command does what you might expect. it prints out a
list of all the available packages you can install. Use it like
this:
.. command-output:: spack list
The packages are listed by name in alphabetical order. If you just
want to see *installed* packages, you should use ``spack list -i``
``spack info``
~~~~~~~~~~~~~~~~
To get information on a particular package from the full list, you can
run ``spack info <package name>``. e.g., for ``mpich``:
.. command-output:: spack info mpich
This gives basic information about the package, such as where it can
be downloaded, what other packages it depends on, virtual package
information, and a text description, if one is available. We'll give
more details on dependencies and virtual dependencies later in this
guide.
Installing and uninstalling
------------------------------
``spack install``
~~~~~~~~~~~~~~~~~~~~~
You can install any package from ``spack list``, using ``spack
install``. In the simplest case, if you just want the latest version
and you don't care about any configuration, you can just run ``spack
install <package>``:
.. code-block:: sh
spack install mpileaks
This will fetch the tarball for ``mpileaks``, expand it, verify that
it was donwloaded without errors, build the package, and install it in
its own directory in ``$SPACK_HOME/opt``. If the requested packages
depends on other packages in order to build, then they will also be
fetched and installed.
Spack also allows you to ask for *specific* configurations of a
package. For example, if you want to install something with a
specific version, you can add ``@`` after the package name, followed
by the version you want:
.. code-block:: sh
spack install mpich@3.0.4
You can install as many versions of the same pacakge as you want, and
they will not interfere with each other. Spack installs each package
into its own unique prefix. If you or another user links a library
against soething you install using Spack, it will continue to work
until you explicitly uninstall it.
The version isn't all that you can customize on a spack command line.
Spack can install many configurations, with different versions,
compilers, compiler versions, compile-time options (variants), and
even architectures (e.g., on a machine that requires cross-compiling).
Spack is also unique in that it lets you customize the *dependencies*
you build a package with. That is, you could have two configurations
of the same version of a package: one built with boost 1.39.0, and the
other version built with version 1.43.0.
Spack calls the descriptor used to refer to a particular package
configuration a **spec**. In the command lines above, both
``mpileaks`` and ``mpileaks@3.0.4`` are specs. Specs and their syntax
are covered in more detail in :ref:`sec-specs`.
``spack uninstall``
~~~~~~~~~~~~~~~~~~~~~
To uninstall a package, just type ``spack uninstall <package>``. This
will completely remove the directory in which the package was installed.
.. code-block:: sh
spack uninstall mpich
If there are other installed packages depend on the package you're
uninstalling, spack will issue a warning to this effect. In general,
you should remove the other packages *before* removing the package
they depend on, or you risk breaking packages on your system. If you
still want to remove the package without regard for its dependencies,
you can run ``spack uninstall -f <package>`` to override Spack's
warning.
If you have more than one version of the same package installed, spack
may not be able to figure out which on eyou want uninstalled. For
example, if you have both ``mpich@3.0.2`` and ``mpich@3.1`` installed,
and you type ``spack uninstall mpich``, then Spack will not know which
one you're referring to, and it will ask you to be more specific by
providing a version to differentiate, For example, ``spack uninstall
mpich@3.1`` is unambiguous.
.. _sec-specs:
Specs
-------------------------
Dependencies
-------------------------
Virtual dependencies
-------------------------
Versions, compilers, and architectures
----------------------------------------
``spack versions``
~~~~~~~~~~~~~~~~~~~~~~~~
``spack compilers``
~~~~~~~~~~~~~~~~~~~~~~~~
Architectures
~~~~~~~~~~~~~~~~~~~~~~~~
Spack's specs allow insatllations for multiple architectures to coexist
within the same prefix. It is also intended to support multiple
architecutres for cross-compilation.
Package lifecycle
------------------------------
The ``spack install`` command performs a number of tasks before it
finally installs each package. It downloads an archive, expands it in
a temporary directory, and only then performs the installation. Spack
has several commands that allow finer-grained control over each of
these stages of the build process.
``spack fetch``
~~~~~~~~~~~~~~~~~
This is the first step of ``spack install``. It takes a spec and
determines the correct download URL to use for the requested package
version. It then downloads the archive, checks it against an MD5
checksum, and stores it in a staging directory if the check was
successful. The staging directory will be located under
``$SPACK_HOME/var/spack``.
If run after the archive has already been downloaded, ``spack fetch``
is idempotent and will not download the archive again.
``spack stage``
~~~~~~~~~~~~~~~~~
This is the second step in installation after ``spack fetch``. It
expands the downloaded archive in its temporary directory, where it
will be built by ``spack install``. If the archive has already been
expanded, then this command does nothing.
``spack clean``
~~~~~~~~~~~~~~~~~
This command has several variations, each undoing one of the
installation tasks. They are:
``spack clean``
Runs ``make clean`` in the expanded archive directory. This is useful
if an attempted build failed, and something needs to be changed to get
a package to build. If a particular package does not have a ``make clean``
target, this will do nothing.
``spack clean -w`` or ``spack clean --work``
This deletes the entire build directory and re-expands it from the downloaded
archive. This is useful if a package does not support a proper ``make clean``
target.
``spack clean -d`` or ``spack clean --dist``
This deletes the build directory *and* the downloaded archive. If
``fetch``, ``stage``, or ``install`` are run again after this, the
process will start from scratch, and the archive archive will be
downloaded again. Useful if somehow a bad archive is downloaded
accidentally and needs to be cleaned out of the staging area.
``spack purge``
~~~~~~~~~~~~~~~~~

View file

@ -16,7 +16,10 @@
# If extensions (or modules to document with autodoc) are in another directory, # If extensions (or modules to document with autodoc) are in another directory,
# add these directories to sys.path here. If the directory is relative to the # add these directories to sys.path here. If the directory is relative to the
# documentation root, use os.path.abspath to make it absolute, like shown here. # documentation root, use os.path.abspath to make it absolute, like shown here.
#sys.path.insert(0, os.path.abspath('.')) sys.path.insert(0, os.path.abspath('exts'))
os.environ['PATH'] += os.pathsep + '../../../bin'
todo_include_todos = True
# -- General configuration ----------------------------------------------------- # -- General configuration -----------------------------------------------------
@ -25,7 +28,10 @@
# Add any Sphinx extension module names here, as strings. They can be extensions # Add any Sphinx extension module names here, as strings. They can be extensions
# coming with Sphinx (named 'sphinx.ext.*') or your custom ones. # coming with Sphinx (named 'sphinx.ext.*') or your custom ones.
extensions = ['sphinx.ext.autodoc'] extensions = ['sphinx.ext.autodoc',
'sphinx.ext.graphviz',
'sphinx.ext.todo',
'sphinxcontrib.programoutput']
# Add any paths that contain templates here, relative to this directory. # Add any paths that contain templates here, relative to this directory.
templates_path = ['_templates'] templates_path = ['_templates']

View file

@ -0,0 +1,31 @@
Developer Guide
=====================
Spec objects
-------------------------
Package objects
-------------------------
Stage objects
-------------------------
Writing commands
-------------------------
Unit tests
-------------------------
Unit testing
-------------------------
Developer commands
-------------------------
``spack doc``
~~~~~~~~~~~~~~~~~
``spack test``
~~~~~~~~~~~~~~~~~

View file

@ -0,0 +1,25 @@
Copyright (c) 2010, 2011, 2012 Sebastian Wiesner <lunaryorn@googlemail.com>
All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above copyright
notice, this list of conditions and the following disclaimer in the
documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View file

@ -0,0 +1,9 @@
# -*- coding: utf-8 -*-
"""
sphinxcontrib
~~~~~~~~~~~~~
Contains 3rd party Sphinx extensions.
"""
__import__('pkg_resources').declare_namespace(__name__)

View file

@ -0,0 +1,263 @@
# -*- coding: utf-8 -*-
# Copyright (c) 2010, 2011, 2012, Sebastian Wiesner <lunaryorn@gmail.com>
# All rights reserved.
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are met:
# 1. Redistributions of source code must retain the above copyright notice,
# this list of conditions and the following disclaimer.
# 2. Redistributions in binary form must reproduce the above copyright
# notice, this list of conditions and the following disclaimer in the
# documentation and/or other materials provided with the distribution.
# THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
# AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
# IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
# ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
# LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
# CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
# SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
# INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
# CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
# ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
# POSSIBILITY OF SUCH DAMAGE.
"""
sphinxcontrib.programoutput
===========================
This extension provides a directive to include the output of commands as
literal block while building the docs.
.. moduleauthor:: Sebastian Wiesner <lunaryorn@gmail.com>
"""
from __future__ import (print_function, division, unicode_literals,
absolute_import)
import sys
import os
import shlex
from subprocess import Popen, PIPE, STDOUT
from collections import defaultdict, namedtuple
from docutils import nodes
from docutils.parsers import rst
from docutils.parsers.rst.directives import flag, unchanged, nonnegative_int
__version__ = '0.9'
class program_output(nodes.Element):
pass
def _slice(value):
parts = [int(v.strip()) for v in value.split(',')]
if len(parts) > 2:
raise ValueError('too many slice parts')
return tuple((parts + [None] * 2)[:2])
class ProgramOutputDirective(rst.Directive):
has_content = False
final_argument_whitespace = True
required_arguments = 1
option_spec = dict(shell=flag, prompt=flag, nostderr=flag,
ellipsis=_slice, extraargs=unchanged,
returncode=nonnegative_int, cwd=unchanged)
def run(self):
env = self.state.document.settings.env
node = program_output()
node.line = self.lineno
node['command'] = self.arguments[0]
if self.name == 'command-output':
node['show_prompt'] = True
else:
node['show_prompt'] = 'prompt' in self.options
node['hide_standard_error'] = 'nostderr' in self.options
node['extraargs'] = self.options.get('extraargs', '')
_, cwd = env.relfn2path(self.options.get('cwd', '/'))
node['working_directory'] = cwd
node['use_shell'] = 'shell' in self.options
node['returncode'] = self.options.get('returncode', 0)
if 'ellipsis' in self.options:
node['strip_lines'] = self.options['ellipsis']
return [node]
_Command = namedtuple(
'Command', 'command shell hide_standard_error working_directory')
class Command(_Command):
"""
A command to be executed.
"""
def __new__(cls, command, shell=False, hide_standard_error=False,
working_directory='/'):
if isinstance(command, list):
command = tuple(command)
# `chdir()` resolves symlinks, so we need to resolve them too for
# caching to make sure that different symlinks to the same directory
# don't result in different cache keys. Also normalize paths to make
# sure that identical paths are also equal as strings.
working_directory = os.path.normpath(os.path.realpath(
working_directory))
return _Command.__new__(cls, command, shell, hide_standard_error,
working_directory)
@classmethod
def from_program_output_node(cls, node):
"""
Create a command from a :class:`program_output` node.
"""
extraargs = node.get('extraargs', '')
command = (node['command'] + ' ' + extraargs).strip()
return cls(command, node['use_shell'],
node['hide_standard_error'], node['working_directory'])
def execute(self):
"""
Execute this command.
Return the :class:`~subprocess.Popen` object representing the running
command.
"""
if self.shell:
if sys.version_info[0] < 3 and isinstance(self.command, unicode):
command = self.command.encode(sys.getfilesystemencoding())
else:
command = self.command
else:
if sys.version_info[0] < 3 and isinstance(self.command, unicode):
command = shlex.split(self.command.encode(
sys.getfilesystemencoding()))
elif isinstance(self.command, str):
command = shlex.split(self.command)
else:
command = self.command
return Popen(command, shell=self.shell, stdout=PIPE,
stderr=PIPE if self.hide_standard_error else STDOUT,
cwd=self.working_directory)
def get_output(self):
"""
Get the output of this command.
Return a tuple ``(returncode, output)``. ``returncode`` is the
integral return code of the process, ``output`` is the output as
unicode string, with final trailing spaces and new lines stripped.
"""
process = self.execute()
output = process.communicate()[0].decode(
sys.getfilesystemencoding(), 'replace').rstrip()
return process.returncode, output
def __str__(self):
if isinstance(self.command, tuple):
return repr(list(self.command))
return repr(self.command)
class ProgramOutputCache(defaultdict):
"""
Execute command and cache their output.
This class is a mapping. Its keys are :class:`Command` objects represeting
command invocations. Its values are tuples of the form ``(returncode,
output)``, where ``returncode`` is the integral return code of the command,
and ``output`` is the output as unicode string.
The first time, a key is retrieved from this object, the command is
invoked, and its result is cached. Subsequent access to the same key
returns the cached value.
"""
def __missing__(self, command):
"""
Called, if a command was not found in the cache.
``command`` is an instance of :class:`Command`.
"""
result = command.get_output()
self[command] = result
return result
def run_programs(app, doctree):
"""
Execute all programs represented by ``program_output`` nodes in
``doctree``. Each ``program_output`` node in ``doctree`` is then
replaced with a node, that represents the output of this program.
The program output is retrieved from the cache in
``app.env.programoutput_cache``.
"""
if app.config.programoutput_use_ansi:
# enable ANSI support, if requested by config
from sphinxcontrib.ansi import ansi_literal_block
node_class = ansi_literal_block
else:
node_class = nodes.literal_block
cache = app.env.programoutput_cache
for node in doctree.traverse(program_output):
command = Command.from_program_output_node(node)
try:
returncode, output = cache[command]
except EnvironmentError as error:
error_message = 'Command {0} failed: {1}'.format(command, error)
error_node = doctree.reporter.error(error_message, base_node=node)
node.replace_self(error_node)
else:
if returncode != node['returncode']:
app.warn('Unexpected return code {0} from command {1}'.format(
returncode, command))
# replace lines with ..., if ellipsis is specified
if 'strip_lines' in node:
lines = output.splitlines()
start, stop = node['strip_lines']
lines[start:stop] = ['...']
output = '\n'.join(lines)
if node['show_prompt']:
tmpl = app.config.programoutput_prompt_template
output = tmpl.format(command=node['command'], output=output,
returncode=returncode)
new_node = node_class(output, output)
new_node['language'] = 'text'
node.replace_self(new_node)
def init_cache(app):
"""
Initialize the cache for program output at
``app.env.programoutput_cache``, if not already present (e.g. being
loaded from a pickled environment).
The cache is of type :class:`ProgramOutputCache`.
"""
if not hasattr(app.env, 'programoutput_cache'):
app.env.programoutput_cache = ProgramOutputCache()
def setup(app):
app.add_config_value('programoutput_use_ansi', False, 'env')
app.add_config_value('programoutput_prompt_template',
'$ {command}\n{output}', 'env')
app.add_directive('program-output', ProgramOutputDirective)
app.add_directive('command-output', ProgramOutputDirective)
app.connect(str('builder-inited'), init_cache)
app.connect(str('doctree-read'), run_programs)

View file

@ -0,0 +1,96 @@
Feature Overview
==================
This is an overview of features that make Spack different from other
`package managers <http://en.wikipedia.org/wiki/Package_management_system>`_
and `port systems <http://en.wikipedia.org/wiki/Ports_collection>`_.
Simple package installation
----------------------------
Installing packages is easy with Spack when you just want the default
version. This installs the latest version of mpileaks and all of its
dependencies:
.. code-block:: sh
$ spack install mpileaks
Custom versions & configurations
-------------------------------------------
If there's some aspect of your package that you want to customize, you
can do that too.
.. code-block:: sh
# Install a particular version by appending @
$ spack install mpileaks@1.1.2
# Or your favorite compiler (and its version), with %
$ spack install mpileaks@1.1.2 %gcc@4.7.3
# Add some special compile-time options with +
$ spack install mpileaks@1.1.2 %gcc@4.7.3 +debug
# Cross-compile for a different architecture with =
$ spack install mpileaks@1.1.2 =bgqos_0
Customize dependencies
-------------------------------------
You can customize package dependencies with ``^``. Suppose that
``mpileaks`` depends indirectly on ``libelf`` and ``libdwarf``. Using
``^``, you can add custom configurations for the dependencies, too.
.. code-block:: sh
# Install mpileaks and link it with specific versions of libelf and libdwarf
$ spack install mpileaks@1.1.2 %gcc@4.7.3 +debug ^libelf@0.8.12 ^libdwarf@20130729+debug
Non-destructive installs
-------------------------------------
Spack installs every unique package configuration in its own prefix,
so you can install as many different versions and configurations as
you want. New installs will not break existing ones.
Packages can peacefully coexist
-------------------------------------
Spack uses ``RPATH`` everywhere, so users do not need to customize
``LD_LIBRARY_PATH``. If you use a library or run a program, it will
run the way you built it.
Creating packages is easy
-------------------------------------
To create your own packages, give spack the tarball URL. Spack
creates all the boilerplate for you.
.. code-block:: sh
$ spack create http://scalability.llnl.gov/mpileaks/downloads/mpileaks-1.0.tar.gz
Creates ``mpileaks.py``:
.. code-block:: python
from spack import *
class Mpileaks(Package):
homepage = "http://www.example.com/"
url = "http://scalability.llnl.gov/mpileaks/downloads/mpileaks-1.0.tar.gz"
md5 = "4136d7b4c04df68b686570afa26988ac"
def install(self, prefix):
configure("--prefix=%s" % prefix)
make()
make("install")
Packages are pure python, so you have complete freedom when writing
build code. Spack also provides a number of feature that make it
easier to write packages.

View file

@ -0,0 +1,44 @@
Getting Started
====================
Download
--------------------
Getting spack is easy. Clone it using `git <http://git-scm.com->`_
with the following command:
.. code-block:: sh
$ git clone ssh://git@cz-stash.llnl.gov:7999/scale/spack.git
This will create a directory called ``spack``. We'll assume that the
full path to this directory is in some environment called
``SPACK_HOME``. Add ``$SPACK_HOME/bin`` to your path and you're ready
to go:
.. code-block:: sh
$ export PATH=spack/bin:$SPACK_HOME
$ spack install mpich
In general, most of your interactions with Spack will be through the
``spack`` command.
Install
--------------------
You don't need to install Spack; it's ready to run as soon as you
clone it from git.
You may want to run it out of a prefix other than the git repository
you cloned. The ``spack bootstrap`` command provides this
functionality. To install spack in a new directory, simply type:
.. code-block:: sh
$ spack bootstrap /my/favorite/prefix
This will install a new spack script in /my/favorite/prefix/bin, which
you can use just like you would the regular spack script. Each copy
of spack installs packages into its own ``$PREFIX/opt`` directory.

View file

@ -3,15 +3,39 @@
You can adapt this file completely to your liking, but it should at least You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive. contain the root `toctree` directive.
Welcome to Spack's documentation! Spack Documentation
================================= =================================
Contents: Spack builds and installs software the way you want it. Other tools
let you install the latest version once. Spack lets you install the
versions you want, built with the compilers, libraries, and options
you want. Spack is non-destructive; installing a new version does not
break your old installs. See the :doc:`features` for more highlights.
Get spack and install your first package:
.. code-block:: sh
$ git clone ssh://git@cz-stash.llnl.gov:7999/scale/spack.git
$ cd spack/bin
$ ./spack install mpich
If you're new to spack and want to start using it, see :doc:`getting_started`,
or refer to the full manual below.
Table of Contents
---------------------
.. toctree:: .. toctree::
:maxdepth: 2 :maxdepth: 2
features
getting_started
basic_usage
packaging_guide
site_configuration
developer_guide
API Docs <spack>
Indices and tables Indices and tables
================== ==================
@ -19,4 +43,3 @@ Indices and tables
* :ref:`genindex` * :ref:`genindex`
* :ref:`modindex` * :ref:`modindex`
* :ref:`search` * :ref:`search`

View file

@ -0,0 +1,35 @@
Packaging Guide
=====================
Package files
-------------------------
Dependencies
-------------------------
Virtual dependencies
-------------------------
Virtual dependencies
-------------------------
Packaging commands
-------------------------
``spack edit``
~~~~~~~~~~~~~~~~~~~~
``spack create``
~~~~~~~~~~~~~~~~~~~~
``spack checksum``
~~~~~~~~~~~~~~~~~~~~
``spack graph``
~~~~~~~~~~~~~~~~~~~~

View file

@ -0,0 +1,9 @@
Site-specific configuration
===================================
Temporary space
----------------------------
Concretization policies
----------------------------

View file

@ -17,7 +17,7 @@ def get_origin_url():
return origin_url.strip() return origin_url.strip()
def install_spack(parser, args): def bootstrap(parser, args):
origin_url = get_origin_url() origin_url = get_origin_url()
prefix = args.prefix prefix = args.prefix
@ -28,7 +28,7 @@ def install_spack(parser, args):
files_in_the_way = os.listdir(prefix) files_in_the_way = os.listdir(prefix)
if files_in_the_way: if files_in_the_way:
tty.die("There are already files there! Delete these files before installing spack.", tty.die("There are already files there! Delete these files before boostrapping spack.",
*files_in_the_way) *files_in_the_way)
tty.msg("Installing:", tty.msg("Installing:",
@ -41,5 +41,5 @@ def install_spack(parser, args):
check_call(['git', 'fetch', 'origin', 'master:refs/remotes/origin/master', '-n', '-q']) check_call(['git', 'fetch', 'origin', 'master:refs/remotes/origin/master', '-n', '-q'])
check_call(['git', 'reset', '--hard', 'origin/master', '-q']) check_call(['git', 'reset', '--hard', 'origin/master', '-q'])
tty.msg("Successfully installed spack in %s" % prefix, tty.msg("Successfully created a new spack in %s" % prefix,
"Run %s/bin/spack to use this installation." % prefix) "Run %s/bin/spack to use this installation." % prefix)

View file

@ -9,9 +9,9 @@
def setup_parser(subparser): def setup_parser(subparser):
subparser.add_argument('-c', "--clean", action="store_true", dest='clean', subparser.add_argument('-c', "--clean", action="store_true", dest='clean',
help="run make clean in the stage directory (default)") help="run make clean in the build directory (default)")
subparser.add_argument('-w', "--work", action="store_true", dest='work', subparser.add_argument('-w', "--work", action="store_true", dest='work',
help="delete and re-expand the entire stage directory") help="delete the build directory and re-expand it from its archive.")
subparser.add_argument('-d', "--dist", action="store_true", dest='dist', subparser.add_argument('-d', "--dist", action="store_true", dest='dist',
help="delete the downloaded archive.") help="delete the downloaded archive.")
subparser.add_argument('packages', nargs=argparse.REMAINDER, subparser.add_argument('packages', nargs=argparse.REMAINDER,

View file

@ -35,8 +35,8 @@
class Package(object): class Package(object):
"""This is the superclass for all spack packages. """This is the superclass for all spack packages.
The Package class ***The Package class***
==================
Package is where the bulk of the work of installing packages is done. Package is where the bulk of the work of installing packages is done.
A package defines how to fetch, verfiy (via, e.g., md5), build, and A package defines how to fetch, verfiy (via, e.g., md5), build, and
@ -53,11 +53,13 @@ class Package(object):
in this directory. Spack automatically scans the python files there in this directory. Spack automatically scans the python files there
and figures out which one to import when you invoke it. and figures out which one to import when you invoke it.
An example package **An example package**
====================
Let's look at the cmake package to start with. This package lives in Let's look at the cmake package to start with. This package lives in
$prefix/lib/spack/spack/packages/cmake.py: $prefix/lib/spack/spack/packages/cmake.py:
.. code-block:: python
from spack import * from spack import *
class Cmake(Package): class Cmake(Package):
homepage = 'https://www.cmake.org' homepage = 'https://www.cmake.org'
@ -70,18 +72,20 @@ def install(self, prefix):
make() make()
make('install') make('install')
Naming conventions **Naming conventions**
---------------------
There are two names you should care about: There are two names you should care about:
1. The module name, 'cmake'. 1. The module name, ``cmake``.
- User will refers to this name, e.g. 'spack install cmake'.
- Corresponds to the name of the file, 'cmake.py', and it can
include _, -, and numbers (it can even start with a number).
2. The class name, "Cmake". This is formed by converting -'s or _'s * User will refers to this name, e.g. 'spack install cmake'.
in the module name to camel case. If the name starts with a number, * Corresponds to the name of the file, 'cmake.py', and it can
we prefix the class name with 'Num_'. Examples: include ``_``, ``-``, and numbers (it can even start with a
number).
2. The class name, "Cmake". This is formed by converting `-` or
``_`` in the module name to camel case. If the name starts with
a number, we prefix the class name with ``Num_``. Examples:
Module Name Class Name Module Name Class Name
foo_bar FooBar foo_bar FooBar
@ -91,23 +95,28 @@ def install(self, prefix):
The class name is what spack looks for when it loads a package module. The class name is what spack looks for when it loads a package module.
Required Attributes **Required Attributes**
---------------------
Aside from proper naming, here is the bare minimum set of things you Aside from proper naming, here is the bare minimum set of things you
need when you make a package: need when you make a package:
homepage informational URL, so that users know what they're
homepage
informational URL, so that users know what they're
installing. installing.
url URL of the source archive that spack will fetch. url
URL of the source archive that spack will fetch.
md5 md5 hash of the source archive, so that we can md5
md5 hash of the source archive, so that we can
verify that it was downloaded securely and correctly. verify that it was downloaded securely and correctly.
install() This function tells spack how to build and install the install()
This function tells spack how to build and install the
software it downloaded. software it downloaded.
Optional Attributes **Optional Attributes**
---------------------
You can also optionally add these attributes, if needed: You can also optionally add these attributes, if needed:
list_url list_url
Webpage to scrape for available version strings. Default is the Webpage to scrape for available version strings. Default is the
@ -121,8 +130,8 @@ def install(self, prefix):
your package needs special version formatting in its URL. boost your package needs special version formatting in its URL. boost
is an example of a package that needs this. is an example of a package that needs this.
Creating Packages ***Creating Packages***
===================
As a package creator, you can probably ignore most of the preceding As a package creator, you can probably ignore most of the preceding
information, because you can use the 'spack create' command to do it information, because you can use the 'spack create' command to do it
all automatically. all automatically.
@ -130,8 +139,8 @@ def install(self, prefix):
You as the package creator generally only have to worry about writing You as the package creator generally only have to worry about writing
your install function and specifying dependencies. your install function and specifying dependencies.
spack create **spack create**
----------------
Most software comes in nicely packaged tarballs, like this one: Most software comes in nicely packaged tarballs, like this one:
http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz
@ -147,11 +156,13 @@ def install(self, prefix):
Once this skeleton code is generated, spack pops up the new package in Once this skeleton code is generated, spack pops up the new package in
your $EDITOR so that you can modify the parts that need changes. your $EDITOR so that you can modify the parts that need changes.
Dependencies **Dependencies**
---------------
If your package requires another in order to build, you can specify that If your package requires another in order to build, you can specify that
like this: like this:
.. code-block:: python
class Stackwalker(Package): class Stackwalker(Package):
... ...
depends_on("libdwarf") depends_on("libdwarf")
@ -170,8 +181,8 @@ class Stackwalker(Package):
it will find the dependencies automatically. it will find the dependencies automatically.
The Install Function **The Install Function**
----------------------
The install function is designed so that someone not too terribly familiar The install function is designed so that someone not too terribly familiar
with Python could write a package installer. For example, we put a number with Python could write a package installer. For example, we put a number
of commands in install scope that you can use almost like shell commands. of commands in install scope that you can use almost like shell commands.
@ -195,13 +206,15 @@ class Stackwalker(Package):
them are created and set on the module. them are created and set on the module.
Parallel Builds **Parallel Builds**
-------------------
By default, Spack will run make in parallel when you run make() in your By default, Spack will run make in parallel when you run make() in your
install function. Spack figures out how many cores are available on install function. Spack figures out how many cores are available on
your system and runs make with -j<cores>. If you do not want this behavior, your system and runs make with -j<cores>. If you do not want this behavior,
you can explicitly mark a package not to use parallel make: you can explicitly mark a package not to use parallel make:
.. code-block:: python
class SomePackage(Package): class SomePackage(Package):
... ...
parallel = False parallel = False
@ -210,20 +223,26 @@ class SomePackage(Package):
This changes thd default behavior so that make is sequential. If you still This changes thd default behavior so that make is sequential. If you still
want to build some parts in parallel, you can do this in your install function: want to build some parts in parallel, you can do this in your install function:
.. code-block:: python
make(parallel=True) make(parallel=True)
Likewise, if you do not supply parallel = True in your Package, you can keep Likewise, if you do not supply parallel = True in your Package, you can keep
the default parallel behavior and run make like this when you want a the default parallel behavior and run make like this when you want a
sequential build: sequential build:
.. code-block:: python
make(parallel=False) make(parallel=False)
Package Lifecycle **Package Lifecycle**
==================
This section is really only for developers of new spack commands. This section is really only for developers of new spack commands.
A package's lifecycle over a run of Spack looks something like this: A package's lifecycle over a run of Spack looks something like this:
.. code-block:: python
p = Package() # Done for you by spack p = Package() # Done for you by spack
p.do_fetch() # called by spack commands in spack/cmd. p.do_fetch() # called by spack commands in spack/cmd.
@ -232,6 +251,9 @@ class SomePackage(Package):
p.do_uninstall() p.do_uninstall()
There are also some other commands that clean the build area: There are also some other commands that clean the build area:
.. code-block:: python
p.do_clean() # runs make clean p.do_clean() # runs make clean
p.do_clean_work() # removes the build directory and p.do_clean_work() # removes the build directory and
# re-expands the archive. # re-expands the archive.
@ -244,6 +266,7 @@ class SomePackage(Package):
Package creators override functions like install() (all of them do this), Package creators override functions like install() (all of them do this),
clean() (some of them do this), and others to provide custom behavior. clean() (some of them do this), and others to provide custom behavior.
""" """
# #

View file

@ -68,9 +68,11 @@ def _caller_locals():
del stack del stack
def _ensure_caller_is_spack_package(): def _get_calling_package_name():
"""Make sure that the caller is a spack package. If it's not, """Make sure that the caller is a class definition, and return
raise ScopeError. if it is, return its name.""" the module's name. This is useful for getting the name of
spack packages from inside a relation function.
"""
stack = inspect.stack() stack = inspect.stack()
try: try:
# get calling function name (the relation) # get calling function name (the relation)
@ -85,9 +87,6 @@ def _ensure_caller_is_spack_package():
raise ScopeError(relation) raise ScopeError(relation)
module_name = caller_locals['__module__'] module_name = caller_locals['__module__']
if not module_name.startswith(packages_module()):
raise ScopeError(relation)
base_name = module_name.split('.')[-1] base_name = module_name.split('.')[-1]
return base_name return base_name
@ -121,7 +120,7 @@ def _parse_local_spec(spec_like, pkg_name):
"""Adds a dependencies local variable in the locals of """Adds a dependencies local variable in the locals of
the calling class, based on args. """ the calling class, based on args. """
def depends_on(*specs): def depends_on(*specs):
pkg = _ensure_caller_is_spack_package() pkg = _get_calling_package_name()
dependencies = _caller_locals().setdefault('dependencies', {}) dependencies = _caller_locals().setdefault('dependencies', {})
for string in specs: for string in specs:
@ -136,7 +135,7 @@ def provides(*specs, **kwargs):
'mpi', other packages can declare that they depend on "mpi", and spack 'mpi', other packages can declare that they depend on "mpi", and spack
can use the providing package to satisfy the dependency. can use the providing package to satisfy the dependency.
""" """
pkg = _ensure_caller_is_spack_package() pkg = _get_calling_package_name()
spec_string = kwargs.get('when', pkg) spec_string = kwargs.get('when', pkg)
provider_spec = _parse_local_spec(spec_string, pkg) provider_spec = _parse_local_spec(spec_string, pkg)
@ -170,7 +169,7 @@ class ScopeError(RelationError):
def __init__(self, relation): def __init__(self, relation):
super(ScopeError, self).__init__( super(ScopeError, self).__init__(
relation, relation,
"Cannot inovke '%s' from outside of a Spack package!" % relation) "Must invoke '%s' from inside a class definition!" % relation)
class CircularReferenceError(RelationError): class CircularReferenceError(RelationError):

View file

@ -6,8 +6,10 @@
The syntax looks like this: The syntax looks like this:
spack install mpileaks ^openmpi @1.2:1.4 +debug %intel @12.1 .. code-block:: sh
0 1 2 3 4 5
$ spack install mpileaks ^openmpi @1.2:1.4 +debug %intel @12.1 =bgqos_0
0 1 2 3 4 5 6
The first part of this is the command, 'spack install'. The rest of the The first part of this is the command, 'spack install'. The rest of the
line is a spec for a particular installation of the mpileaks package. line is a spec for a particular installation of the mpileaks package.
@ -37,7 +39,10 @@
if it comes immediately after the compiler name. Otherwise it will be if it comes immediately after the compiler name. Otherwise it will be
associated with the current package spec. associated with the current package spec.
Here is the EBNF grammar for a spec: 6. The architecture to build with. This is needed on machines where
cross-compilation is required
Here is the EBNF grammar for a spec::
spec-list = { spec [ dep-list ] } spec-list = { spec [ dep-list ] }
dep_list = { ^ spec } dep_list = { ^ spec }
@ -80,8 +85,7 @@
"""This map determines the coloring of specs when using color output. """This map determines the coloring of specs when using color output.
We make the fields different colors to enhance readability. We make the fields different colors to enhance readability.
See spack.color for descriptions of the color codes. See spack.color for descriptions of the color codes. """
"""
color_formats = {'%' : '@g', # compiler color_formats = {'%' : '@g', # compiler
'@' : '@c', # version '@' : '@c', # version
'=' : '@m', # architecture '=' : '@m', # architecture
@ -129,8 +133,7 @@ def __call__(self, match):
class Compiler(object): class Compiler(object):
"""The Compiler field represents the compiler or range of compiler """The Compiler field represents the compiler or range of compiler
versions that a package should be built with. Compilers have a versions that a package should be built with. Compilers have a
name and a version list. name and a version list. """
"""
def __init__(self, name, version=None): def __init__(self, name, version=None):
if name not in spack.compilers.supported_compilers(): if name not in spack.compilers.supported_compilers():
raise UnknownCompilerError(name) raise UnknownCompilerError(name)
@ -348,6 +351,7 @@ def package(self):
@property @property
def virtual(self): def virtual(self):
"""Right now, a spec is virtual if no package exists with its name. """Right now, a spec is virtual if no package exists with its name.
TODO: revisit this -- might need to use a separate namespace and TODO: revisit this -- might need to use a separate namespace and
be more explicit about this. be more explicit about this.
Possible idea: just use conventin and make virtual deps all Possible idea: just use conventin and make virtual deps all
@ -463,7 +467,9 @@ def _expand_virtual_packages(self):
and normalize again to include the provider's (potentially virtual) and normalize again to include the provider's (potentially virtual)
dependencies. Repeat until there are no virtual deps. dependencies. Repeat until there are no virtual deps.
TODO: If a provider depends on something that conflicts with .. todo::
If a provider depends on something that conflicts with
other dependencies in the spec being expanded, this can other dependencies in the spec being expanded, this can
produce a conflicting spec. For example, if mpich depends produce a conflicting spec. For example, if mpich depends
on hwloc@:1.3 but something in the spec needs hwloc1.4:, on hwloc@:1.3 but something in the spec needs hwloc1.4:,
@ -625,13 +631,16 @@ def normalize(self):
"""When specs are parsed, any dependencies specified are hanging off """When specs are parsed, any dependencies specified are hanging off
the root, and ONLY the ones that were explicitly provided are there. the root, and ONLY the ones that were explicitly provided are there.
Normalization turns a partial flat spec into a DAG, where: Normalization turns a partial flat spec into a DAG, where:
1) ALL dependencies of the root package are in the DAG.
2) Each node's dependencies dict only contains its direct deps. 1. ALL dependencies of the root package are in the DAG.
3) There is only ONE unique spec for each package in the DAG. 2. Each node's dependencies dict only contains its direct deps.
- This includes virtual packages. If there a non-virtual 3. There is only ONE unique spec for each package in the DAG.
* This includes virtual packages. If there a non-virtual
package that provides a virtual package that is in the spec, package that provides a virtual package that is in the spec,
then we replace the virtual package with the non-virtual one. then we replace the virtual package with the non-virtual one.
4) The spec DAG matches package DAG.
4. The spec DAG matches package DAG.
TODO: normalize should probably implement some form of cycle detection, TODO: normalize should probably implement some form of cycle detection,
to ensure that the spec is actually a DAG. to ensure that the spec is actually a DAG.

View file

@ -18,12 +18,16 @@ class Stage(object):
expanded, and built before being installed. It also handles downloading expanded, and built before being installed. It also handles downloading
the archive. A stage's lifecycle looks like this: the archive. A stage's lifecycle looks like this:
setup() Create the stage directory. setup()
fetch() Fetch a source archive into the stage. Create the stage directory.
expand_archive() Expand the source archive. fetch()
<install> Build and install the archive. This is handled Fetch a source archive into the stage.
by the Package class. expand_archive()
destroy() Remove the stage once the package has been installed. Expand the source archive.
<install>
Build and install the archive. This is handled by the Package class.
destroy()
Remove the stage once the package has been installed.
If spack.use_tmp_stage is True, spack will attempt to create stages If spack.use_tmp_stage is True, spack will attempt to create stages
in a tmp directory. Otherwise, stages are created directly in in a tmp directory. Otherwise, stages are created directly in

View file

@ -9,7 +9,7 @@
A list of Versions and VersionRanges. A list of Versions and VersionRanges.
All of these types support the following operations, which can All of these types support the following operations, which can
be called on any of the types: be called on any of the types::
__eq__, __ne__, __lt__, __gt__, __ge__, __le__, __hash__ __eq__, __ne__, __lt__, __gt__, __ge__, __le__, __hash__
__contains__ __contains__
@ -18,8 +18,6 @@
union union
intersection intersection
concrete concrete
True if the Version, VersionRange or VersionList represents
a single version.
""" """
import os import os
import sys import sys