hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

More packaging documentation.

Browse source
This commit is contained in:
Todd Gamblin 2013-12-26 13:47:13 -08:00
parent a4cda94524
commit 208db9b002
10 changed files with 509 additions and 62 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
lib/spack/docs/_themes/sphinx_rtd_theme/footer.html vendored
View file

@ -13,18 +13,16 @@
<hr/>
<p>
{%- if show_copyright %}
{%- if hasdoc('copyright') %}
{% trans path=pathto('copyright'), copyright=copyright|e %}&copy; <a href="{{ path }}">Copyright</a> {{ copyright }}.{% endtrans %}
{%- else %}
{% trans copyright=copyright|e %}&copy; Copyright {{ copyright }}.{% endtrans %}
{%- endif %}
{%- endif %}
&copy; Copyright 2013,
<a href="https://scalability.llnl.gov/">Lawrence Livermore National Laboratory</a>.
<br/>
Written by Todd Gamblin, <a href="mailto:tgamblin@llnl.gov">tgamblin@llnl.gov</a>
<br/>
{%- if last_updated %}
{% trans last_updated=last_updated|e %}Last updated on {{ last_updated }}.{% endtrans %}
{%- endif %}
&nbsp;&nbsp;
{% trans %}<a href="https://www.github.com/snide/sphinx_rtd_theme">Sphinx theme</a> provided by <a href="http://readthedocs.org">Read the Docs</a>{% endtrans %}
{% trans %}<br/><a href="https://www.github.com/snide/sphinx_rtd_theme">Sphinx theme</a> provided by <a href="http://readthedocs.org">Read the Docs</a>{% endtrans %}
</p>
</footer>

4
lib/spack/docs/conf.py
View file

@ -66,7 +66,7 @@
# General information about the project.
project = u'Spack'
copyright = u'2013, Todd Gamblin'
copyright = u'2013, Lawrence Livermore National Laboratory'
# The version info for the project you're documenting, acts as replacement for
# |version| and |release|, also used in various other places throughout the
@ -149,7 +149,7 @@
# If not '', a 'Last updated on:' timestamp is inserted at every page bottom,
# using the given strftime format.
#html_last_updated_fmt = '%b %d, %Y'
html_last_updated_fmt = '%b %d, %Y'
# If true, SmartyPants will be used to convert quotes and dashes to
# typographically correct entities.

2
lib/spack/docs/developer_guide.rst
View file

@ -1,3 +1,5 @@
.. _developer_guide:
Developer Guide
=====================

454
lib/spack/docs/packaging_guide.rst
View file

@ -4,34 +4,439 @@ Packaging Guide
This guide is intended for developers or administrators who want to
*package* their software so that Spack can install it. We assume that
you have at least some familiarty with Python, and that you've read
the :ref:`guide for regular users <basic_usage>`, especially the part
about *specs*.
the :ref:`basic usage guide <basic_usage>`, especially the part
about :ref:`specs <sec-specs>`.
There are two key parts of Spack:
Package files
-------------------------
#. **specs**: a language for describing builds of software, and
#. **packages**: Python modules that build software according to a
spec.
There are two parts of Spack, a language for describing builds of
software (*specs*), and *packages*: Python modules thatactually build
the software. A package essentially takes a spec and implements it
for a particular piece of software. It allows a developer to
encapsulate build logic for different versions, compilers, and
platforms in one place, and it is designed to make things easy for
you, the packager, as much as possible.
The package allows the developer to encapsulate build logic for
different versions, compilers, and platforms in one place.
Packages in spack live in ``$prefix/lib/spack/spack/packages``:
Packages in Spack are written in pure Python, so you can do anything
in Spack that you can do in Python. Python was chosen as the
implementation language for two reasons. First, Python is getting to
be ubiquitous in the HPC community due to its use in numerical codes.
Second, it's a modern language and has many powerful features to help
make package writing easy.
Finally, we've gone to great lengths to make it *easy* to create
packages. The ``spack create`` command lets you generate a
boilerplate package template from a tarball URL, and ideally you'll
only need to run this once and slightly modify the boilerplate to get
your package working.
This section of the guide goes through the parts of a package, and
then tells you how to make your own. If you're impatient, jump ahead
to :ref:`spack-create`.
Directory Structure
---------------------------
A Spack installation directory is structured like a standard UNIX
install prefix (``bin``, ``lib``, ``include``, ``share``, etc.). Most
of the code for Spack lives in ``$SPACK_ROOT/lib/spack``, and this is
also the top-level include directory for Python code. When Spack
runs, it adds this directory to its ``PYTHONPATH``.
Spack packages live in the ``spack.packages`` Python package, which
means that they need to go in ``$prefix/lib/spack/spack/packages``.
If you list that directory, you'll see all the existing packages:
.. command-output:: cd $SPACK_ROOT/lib/spack/spack/packages; ls *.py
:shell:
:ellipsis: 5
``__init__.py`` contains some utility functions used by Spack to load
packages when they're needed for an installation. All the other files
in the ``packages`` directory are actual Spack packages used to
install software.
Parts of a package
---------------------------
It's probably easiest to learn about packages by looking at an
example. Let's take a look at ``libelf.py``:
.. literalinclude:: ../spack/packages/libelf.py
:linenos:
Package Names
~~~~~~~~~~~~~~~~~~
This package lives in a file called ``libelf.py``, and it contains a
class called ``Libelf``. The ``Libelf`` class extends Spack's
``Package`` class (and this is what makes it a Spack package). The
*file name* is what users need to provide in their package
specs. e.g., if you type any of these:
.. code-block:: sh
spack install libelf
spack install libelf@0.8.13
Spack sees the package name in the spec and looks for a file called
``libelf.py`` in its ``packages`` directory. Likewise, if you say
``spack install docbook-xml``, then Spack looks for a file called
``docbook-xml.py``.
We use the filename for the package name to give packagers more
freedom in naming their packages. Package names can contain letters,
numbers, dashes, and underscores, and there are no other restrictions.
You can name a package ``3proxy`` or ``_foo`` and Spack won't care --
it just needs to see that name in the package spec. Experienced
Python programmers will notice that package names are actually Python
module names, and but they're not necessarily valid Python
identifiers. i.e., you can't actually ``import 3proxy`` in Python.
You'll get a syntax error because the identifier doesn't start with a
letter or underscore. For more details on why this is still ok, see
the :ref:`developer guide<developer_guide>`.
.. literalinclude:: ../spack/packages/libelf.py
:linenos:
:lines: 3
The *class name* is formed by converting words separated by `-` or
``_`` in the file name to camel case. If the name starts with a
number, we prefix the class name with ``Num_``. Here are some
examples:
================= =================
Module Name Class Name
================= =================
``foo_bar`` ``FooBar``
``docbook-xml`` ``DocbookXml``
``FooBar`` ``Foobar``
``3proxy`` ``Num_3proxy``
================= =================
The class name is needed by Spack to properly import a package, but
not for much else. In general, you won't have to remember this naming
convention because ``spack create`` will generate a boilerplate class
for you, and you can just fill in the blanks.
Metadata
~~~~~~~~~~~~~~~~~~~~
Just under the class name is a description of the ``libelf`` package.
In Python, this is called a *docstring*, and it's a multi-line,
triple-quoted (``"""``) string that comes just after the definition of
a class. Spack uses the docstring to generate the description of the
package that is shown when you run ``spack info``. If you don't provide
a description, Spack will just print "None" for the description.
In addition the package description, there are a few fields you'll
need to fill out. They are as follows:
``homepage``
This is the URL where you can learn about the package and get
information. It is displayed to users when they run ``spack info``.
``url``
This is the URL where you can download a distribution tarball of
the pacakge's source code.
``versions``
This is a `dictionary
<http://docs.python.org/2/tutorial/datastructures.html#dictionaries>`_
mapping versions to MD5 hashes. Spack uses the hashes to checksum
archives when it downloads a particular version.
The homepage and URL are required fields, and ``versions`` is not
required but it's recommended. Spack will warn usrs if they try to
install a spec (e.g., ``libelf@0.8.10`` for which there is not a
checksum available. They can force it to download the new version and
install, but it's better to provide checksums so users don't have to
install from an unchecked archive.
Install function
~~~~~~~~~~~~~~~~~~~~~~~
The last element of the ``libelf`` package is its ``install()``
function. This is where the real work of installation happens, and
it's the main part of the package you'll need to customize for each
piece of software.
When a user runs ``spack install``, Spack fetches an archive for the
correct version of the software, expands the archive, and sets the
current working directory to the root directory of the expanded
archive. It then instantiates a package object and calls its
``install()`` method.
Install takes a ``spec`` object and a ``prefix`` path:
.. literalinclude:: ../spack/packages/libelf.py
:start-after: 0.8.12
We'll talk about ``spec`` objects and the types of methods you can
call on them later. The ``prefix`` is the path to the directory where
the package should install the software after it is built.
Inside of the ``install()`` function, things should look pretty
familiar. ``libelf`` uses autotools, so the package first calls
``configure``, passing the prefix and some other package-specific
arguments. It then calls ``make`` and ``make install``.
``configure`` and ``make`` look very similar to commands you'd type in
a shell, but they're actually Python functions. Spack provides these
wrapper functions to allow you to call commands more naturally when
you write packages. This allows spack to provide some special
features, as well. For example, in Spack, ``make`` is parallel by
default. Spack figures out the number of cores on your machine and
passes and appropriate value for ``-j<numjobs>`` to the ``make``
command. In a package file, you can supply a keyword argument,
``parallel=False``, to disable parallel make. We do it here to avoid
some race conditions in ``libelf``\'s ``install`` target. The first
call to ``make()``, which does not have a keyword argument, will still
build in parallel.
We'll go into more detail about shell command functions in later
sections.
.. _spack-create:
Creating Packages Automatically
----------------------------------
``spack create``
~~~~~~~~~~~~~~~~~~~~~
The ``spack create`` command takes the drudgery out of making
packages. It generates boilerplate code that conforms to Spack's idea
of a package should be, so that you can focus on getting your pacakge
working.
All you need is the URL to a tarball you want to package:
.. code-block:: sh
$ spack create http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz
When you run this, Spack will look at the tarball URL, and it will try
to figure out the of the package to be created. It also tries to
figure out what version strings for that package look like. Once that
is done, it tries to find *additional* versions by spidering the
package's webpage. Spack then prompts you to tell it how many
versions you want to download and checksum.
.. code-block:: sh
==> Creating template for package cmake
==> Found 18 versions of cmake.
2.8.12.1 http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz
2.8.12 http://www.cmake.org/files/v2.8/cmake-2.8.12.tar.gz
2.8.11.2 http://www.cmake.org/files/v2.8/cmake-2.8.11.2.tar.gz
2.8.11.1 http://www.cmake.org/files/v2.8/cmake-2.8.11.1.tar.gz
2.8.11 http://www.cmake.org/files/v2.8/cmake-2.8.11.tar.gz
2.8.10.2 http://www.cmake.org/files/v2.8/cmake-2.8.10.2.tar.gz
2.8.10.1 http://www.cmake.org/files/v2.8/cmake-2.8.10.1.tar.gz
2.8.10 http://www.cmake.org/files/v2.8/cmake-2.8.10.tar.gz
2.8.9 http://www.cmake.org/files/v2.8/cmake-2.8.9.tar.gz
...
2.8.0 http://www.cmake.org/files/v2.8/cmake-2.8.0.tar.gz
Include how many checksums in the package file? (default is 5, q to abort)
Spack will automatically download the number of tarballs you specify
(starting with the most recent) and checksum each of them.
Note that you don't need to do everything up front. If your package
is large, you can always choose to download just one tarball for now,
then run :ref:`spack checksum <spack-checksum>` later if you end up wanting more. Let's
say, for now, that you opted to download 3 tarballs:
.. code-block:: sh
Include how many checksums in the package file? (default is 5, q to abort) 3
==> Downloading...
==> Fetching http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz
###################################################################### 98.6%
==> Fetching http://www.cmake.org/files/v2.8/cmake-2.8.12.tar.gz
##################################################################### 96.7%
==> Fetching http://www.cmake.org/files/v2.8/cmake-2.8.11.2.tar.gz
#################################################################### 95.2%
Now Spack generates some boilerplate and open the package file in
your favorite ``$EDITOR``:
.. code-block:: python
:linenos:
# FIXME:
# This is a template package file for Spack. We've conveniently
# put "FIXME" labels next to all the things you'll want to change.
#
# Once you've edited all the FIXME's, delete this whole message,
# save this file, and test out your package like this:
#
# spack install cmake
#
# You can always get back here to change things with:
#
# spack edit cmake
#
# See the spack documentation for more information on building
# packages.
#
from spack import *
class Cmake(Package):
"""FIXME: put a proper description of your package here."""
# FIXME: add a proper url for your package's homepage here.
homepage = "http://www.example.com"
url = "http://www.cmake.org/files/v2.8/cmake-2.8.12.1.tar.gz"
versions = { '2.8.12.1' : '9d38cd4e2c94c3cea97d0e2924814acc',
'2.8.12' : '105bc6d21cc2e9b6aff901e43c53afea',
'2.8.11.2' : '6f5d7b8e7534a5d9e1a7664ba63cf882', }
def install(self, spec, prefix):
# FIXME: Modify the configure line to suit your build system here.
configure("--prefix=%s" % prefix)
# FIXME: Add logic to build and install here
make()
make("install")
The tedious stuff (creating the class, checksumming archives) has been
done for you.
All the things you still need to change are marked with ``FIXME``
labels. The first ``FIXME`` refers to the commented instructions at
the top of the file. You can delete these after reading them. The
rest of them are as follows:
#. Add a description in your package's docstring.
#. Change the homepage to a useful URL (not ``example.com``).
#. Get the ``install()`` method working.
``spack edit``
~~~~~~~~~~~~~~~~~~~~
Once you've created a package, you can go back and edit it using
``spack edit``. For example, this:
.. code-block:: sh
spack edit libelf
will open ``$SPACK_ROOT/lib/spack/spack/packages/libelf.py`` in
``$EDITOR``. If you try to edit a package that doesn't exist, Spack
will recommend using ``spack create``:
.. code-block:: sh
$ spack edit foo
==> Error: No package 'foo'. Use spack create, or supply -f/--force to edit a new file.
And, finally, if you *really* want to skip all the automatic stuff
that ``spack create`` does for you, then you can run ``spack edit
-f/--force``:
$ spack edit -f foo
Which will generate a *very* minimal package structure for you to fill
in:
.. code-block:: python
:linenos:
from spack import *
class Foo(Package):
"""Description"""
homepage = "http://www.example.com"
url = "http://www.example.com/foo-1.0.tar.gz"
versions = { '1.0' : '0123456789abcdef0123456789abcdef' }
def install(self, spec, prefix):
configure("--prefix=%s" % prefix)
make()
make("install")
We recommend using this only when you have to, as it's generally more
work than using ``spack create``.
.. _spack-checksum:
``spack checksum``
~~~~~~~~~~~~~~~~~~~~~~
If you've already created a package and you want to add more version
checksums to it, this is automated with ``spack checksum``. Here's an
example for ``libelf``:
.. code-block:: sh
$ spack checksum libelf
==> Found 16 versions of libelf.
0.8.13 http://www.mr511.de/software/libelf-0.8.13.tar.gz
0.8.12 http://www.mr511.de/software/libelf-0.8.12.tar.gz
0.8.11 http://www.mr511.de/software/libelf-0.8.11.tar.gz
0.8.10 http://www.mr511.de/software/libelf-0.8.10.tar.gz
0.8.9 http://www.mr511.de/software/libelf-0.8.9.tar.gz
0.8.8 http://www.mr511.de/software/libelf-0.8.8.tar.gz
0.8.7 http://www.mr511.de/software/libelf-0.8.7.tar.gz
0.8.6 http://www.mr511.de/software/libelf-0.8.6.tar.gz
0.8.5 http://www.mr511.de/software/libelf-0.8.5.tar.gz
...
0.5.2 http://www.mr511.de/software/libelf-0.5.2.tar.gz
How many would you like to checksum? (default is 5, q to abort)
This does the same thing that ``spack create`` did, it just allows you
to go back and create more checksums for an existing package. It
fetches the tarballs you ask for and prints out a dict ready to copy
and paste into your package file:
.. code-block:: sh
==> Checksummed new versions of libelf:
{
'0.8.13' : '4136d7b4c04df68b686570afa26988ac',
'0.8.12' : 'e21f8273d9f5f6d43a59878dc274fec7',
'0.8.11' : 'e931910b6d100f6caa32239849947fbf',
'0.8.10' : '9db4d36c283d9790d8fa7df1f4d7b4d9',
}
You should be able to add these checksums directly to the versions
field in your package.
Note that for ``spack checksum`` to work, Spack needs to be able to
``import`` your pacakge in Python. That means it can't have any
syntax errors, or the ``import`` will fail. Use this once you've got
your package in working order.
Dependencies
------------------------------
Virtual dependencies
-----------------------------
Install environment
-----------------------------
Package lifecycle
------------------------------
``spack install`` command performs a number of tasks before it finally
installs each package. It downloads an archive, expands it in a
temporary directory, and then performs the installation. Spack has
The ``spack install`` command performs a number of tasks before it
finally installs each package. It downloads an archive, expands it in
a temporary directory, and then performs the installation. Spack has
several commands that allow finer-grained control over each stage of
the build process.
@ -100,27 +505,8 @@ files.
Dependencies
-------------------------
Virtual dependencies
-------------------------
Packaging commands
-------------------------
``spack edit``
~~~~~~~~~~~~~~~~~~~~
``spack create``
~~~~~~~~~~~~~~~~~~~~
``spack checksum``
~~~~~~~~~~~~~~~~~~~~
``spack graph``
~~~~~~~~~~~~~~~~~~~~

16
lib/spack/spack/cmd/__init__.py
View file

@ -80,3 +80,19 @@ def parse_specs(args, **kwargs):
except spack.spec.SpecError, e:
tty.error(e.message)
sys.exit(1)
def elide_list(line_list, max_num=10):
"""Takes a long list and limits it to a smaller number of elements,
replacing intervening elements with '...'. For example::
elide_list([1,2,3,4,5,6], 4)
gives::
[1, 2, 3, '...', 6]
"""
if len(line_list) > max_num:
return line_list[:max_num-1] + ['...'] + line_list[-1:]
else:
return line_list

5
lib/spack/spack/cmd/checksum.py
View file

@ -5,6 +5,7 @@
from pprint import pprint
from subprocess import CalledProcessError
import spack.cmd
import spack.tty as tty
import spack.packages as packages
import spack.util.crypto
@ -47,6 +48,7 @@ def get_checksums(versions, urls, **kwargs):
return zip(versions, hashes)
def checksum(parser, args):
# get the package we're going to generate checksums for
pkg = packages.get(args.package)
@ -68,7 +70,8 @@ def checksum(parser, args):
tty.msg("Found %s versions of %s." % (len(urls), pkg.name),
*["%-10s%s" % (v,u) for v, u in zip(versions, urls)])
*spack.cmd.elide_list(
["%-10s%s" % (v,u) for v, u in zip(versions, urls)]))
print
archives_to_fetch = tty.get_number(
"How many would you like to checksum?", default=5, abort='q')

29
lib/spack/spack/cmd/create.py
View file

@ -5,6 +5,7 @@
from contextlib import closing
import spack
import spack.cmd
import spack.package
import spack.packages as packages
import spack.tty as tty
@ -45,11 +46,11 @@ class ${class_name}(Package):
versions = ${versions}
def install(self, prefix):
def install(self, spec, prefix):
# FIXME: Modify the configure line to suit your build system here.
${configure}
# FIXME:
# FIXME: Add logic to build and install here
make()
make("install")
""")
@ -84,6 +85,15 @@ def __call__(self, stage):
self.configure = '%s\n # %s' % (autotools, cmake)
def make_version_dict(ver_hash_tuples):
max_len = max(len(str(v)) for v,hfg in ver_hash_tuples)
width = max_len + 2
format = "%-" + str(width) + "s : '%s',"
sep = '\n '
return '{ ' + sep.join(format % ("'%s'" % v, h)
for v, h in ver_hash_tuples) + ' }'
def create(parser, args):
url = args.url
@ -118,8 +128,9 @@ def create(parser, args):
else:
urls = [spack.url.substitute_version(url, v) for v in versions]
if len(urls) > 1:
tty.msg("Found %s versions of %s to checksum." % (len(urls), name),
*["%-10s%s" % (v,u) for v, u in zip(versions, urls)])
tty.msg("Found %s versions of %s." % (len(urls), name),
*spack.cmd.elide_list(
["%-10s%s" % (v,u) for v, u in zip(versions, urls)]))
print
archives_to_fetch = tty.get_number(
"Include how many checksums in the package file?",
@ -130,17 +141,13 @@ def create(parser, args):
return
guesser = ConfigureGuesser()
version_hashes = spack.cmd.checksum.get_checksums(
ver_hash_tuples = spack.cmd.checksum.get_checksums(
versions[:archives_to_fetch], urls[:archives_to_fetch],
first_stage_function=guesser)
if not version_hashes:
if not ver_hash_tuples:
tty.die("Could not fetch any tarballs for %s." % name)
sep = '\n '
versions_string = '{ ' + sep.join(
"'%s' : '%s'," % (v, h) for v, h in version_hashes) + ' }'
# Write out a template for the file
with closing(open(pkg_path, "w")) as pkg_file:
pkg_file.write(
@ -149,7 +156,7 @@ def create(parser, args):
configure=guesser.configure,
class_name=class_name,
url=url,
versions=versions_string))
versions=make_version_dict(ver_hash_tuples)))
# If everything checks out, go ahead and edit.
spack.editor(pkg_path)

34
lib/spack/spack/cmd/edit.py
View file

@ -1,11 +1,36 @@
import os
import string
from contextlib import closing
import spack
import spack.packages as packages
import spack.tty as tty
description = "Open package files in $EDITOR"
# When -f is supplied, we'll create a very minimal skeleton.
package_template = string.Template("""\
from spack import *
class ${class_name}(Package):
""\"Description""\"
homepage = "http://www.example.com"
url = "http://www.example.com/${name}-1.0.tar.gz"
versions = { '1.0' : '0123456789abcdef0123456789abcdef' }
def install(self, spec, prefix):
configure("--prefix=%s" % prefix)
make()
make("install")
""")
def setup_parser(subparser):
subparser.add_argument(
'-f', '--force', dest='force', action='store_true',
help="Open a new file in $EDITOR even if package doesn't exist.")
subparser.add_argument(
'name', nargs='?', default=None, help="name of package to edit")
@ -24,8 +49,15 @@ def edit(parser, args):
tty.die("Something's wrong. '%s' is not a file!" % path)
if not os.access(path, os.R_OK|os.W_OK):
tty.die("Insufficient permissions on '%s'!" % path)
elif not args.force:
tty.die("No package '%s'. Use spack create, or supply -f/--force "
"to edit a new file." % name)
else:
tty.die("No package for %s. Use spack create.")
class_name = packages.class_name_for_package_name(name)
with closing(open(path, "w")) as pkg_file:
pkg_file.write(
package_template.substitute(name=name, class_name=class_name))
# If everything checks out, go ahead and edit.
spack.editor(path)

5
lib/spack/spack/package.py
View file

@ -108,10 +108,6 @@ def install(self, spec, prefix):
url
URL of the source archive that spack will fetch.
md5
md5 hash of the source archive, so that we can
verify that it was downloaded securely and correctly.
install()
This function tells spack how to build and install the
software it downloaded.
@ -119,6 +115,7 @@ def install(self, spec, prefix):
**Optional Attributes**
You can also optionally add these attributes, if needed:
list_url
Webpage to scrape for available version strings. Default is the
directory containing the tarball; use this if the default isn't

8
lib/spack/spack/packages/libelf.py
View file

@ -1,10 +1,16 @@
from spack import *
class Libelf(Package):
"""libelf lets you read, modify or create ELF object files in an
architecture-independent way. The library takes care of size
and endian issues, e.g. you can process a file for SPARC
processors on an Intel-based system."""
homepage = "http://www.mr511.de/software/english.html"
url = "http://www.mr511.de/software/libelf-0.8.13.tar.gz"
versions = { '0.8.13' : '4136d7b4c04df68b686570afa26988ac' }
versions = { '0.8.13' : '4136d7b4c04df68b686570afa26988ac',
'0.8.12' : 'e21f8273d9f5f6d43a59878dc274fec7', }
def install(self, spec, prefix):
configure("--prefix=%s" % prefix,