hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Merge branch 'efischer/160311-StagedPackage' of https://github.com/citibeth/spack into citibeth-efischer/160311-StagedPackage

Browse source
This commit is contained in:
Todd Gamblin 2016-07-04 00:27:32 -07:00
commit 890df7153a
5 changed files with 524 additions and 18 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

193
lib/spack/docs/packaging_guide.rst
View file

@ -377,6 +377,8 @@ add a line like this in the package class:
version('8.2.1', '4136d7b4c04df68b686570afa26988ac')
...
Versions should be listed with the newest version first.
Version URLs
~~~~~~~~~~~~~~~~~
@ -385,8 +387,21 @@ in the package. For example, Spack is smart enough to download
version ``8.2.1.`` of the ``Foo`` package above from
``http://example.com/foo-8.2.1.tar.gz``.
If spack *cannot* extrapolate the URL from the ``url`` field, or if
the package doesn't have a ``url`` field, you can add a URL explicitly
If spack *cannot* extrapolate the URL from the ``url`` field by
default, you can write your own URL generation algorithm in place of
the ``url`` declaration. For example:
.. code-block:: python
:linenos:
class Foo(Package):
def url_for_version(self, version):
return 'http://example.com/version_%s/foo-%s.tar.gz' \
% (version, version)
version('8.2.1', '4136d7b4c04df68b686570afa26988ac')
...
If a URL cannot be derived systematically, you can add an explicit URL
for a particular version:
.. code-block:: python
@ -1346,6 +1361,19 @@ Now, the ``py-numpy`` package can be used as an argument to ``spack
activate``. When it is activated, all the files in its prefix will be
symbolically linked into the prefix of the python package.
Many packages produce Python extensions for *some* variants, but not
others: they should extend ``python`` only if the apropriate
variant(s) are selected. This may be accomplished with conditional
``extends()`` declarations:
.. code-block:: python
class FooLib(Package):
variant('python', default=True, description= \
'Build the Python extension Module')
extends('python', when='+python')
...
Sometimes, certain files in one package will conflict with those in
another, which means they cannot both be activated (symlinked) at the
same time. In this case, you can tell Spack to ignore those files
@ -1851,7 +1879,7 @@ discover its dependencies.
If you want to see the environment that a package will build with, or
if you want to run commands in that environment to test them out, you
can use the :ref:```spack env`` <spack-env>` command, documented
can use the :ref:`spack env <spack-env>` command, documented
below.
.. _compiler-wrappers:
@ -2531,6 +2559,59 @@ File functions
.. _package-lifecycle:
Coding Style Guidelines
---------------------------
The following guidelines are provided, in the interests of making
Spack packages work in a consistent manner:
Variant Names
~~~~~~~~~~~~~~
Spack packages with variants similar to already-existing Spack
packages should use the same name for their variants. Standard
variant names are:
======= ======== ========================
Name Default Description
------- -------- ------------------------
shared True Build shared libraries
static Build static libraries
mpi Use MPI
python Build Python extension
------- -------- ------------------------
If specified in this table, the corresponding default should be used
when declaring a variant.
Version Lists
~~~~~~~~~~~~~~
Spack packges should list supported versions with the newest first.
Special Versions
~~~~~~~~~~~~~~~~~
The following *special* version names may be used when building a package:
* *@system*: Indicates a hook to the OS-installed version of the
package. This is useful, for example, to tell Spack to use the
OS-installed version in ``packages.yaml``::
openssl:
paths:
openssl@system: /usr
buildable: False
Certain Spack internals look for the *@system* version and do
appropriate things in that case.
* *@local*: Indicates the version was built manually from some source
tree of unknown provenance (see ``spack setup``).
Packaging workflow commands
---------------------------------
@ -2851,3 +2932,109 @@ might write:
DWARF_PREFIX = $(spack location -i libdwarf)
CXXFLAGS += -I$DWARF_PREFIX/include
CXXFLAGS += -L$DWARF_PREFIX/lib
Build System Configuration Support
----------------------------------
Imagine a developer creating a CMake-based (or Autotools) project in a local
directory, which depends on libraries A-Z. Once Spack has installed
those dependencies, one would like to run ``cmake`` with appropriate
command line and environment so CMake can find them. The ``spack
setup`` command does this conveniently, producing a CMake
configuration that is essentially the same as how Spack *would have*
configured the project. This can be demonstrated with a usage
example:
.. code-block:: bash
cd myproject
spack setup myproject@local
mkdir build; cd build
../spconfig.py ..
make
make install
Notes:
* Spack must have ``myproject/package.py`` in its repository for
this to work.
* ``spack setup`` produces the executable script ``spconfig.py`` in
the local directory, and also creates the module file for the
package. ``spconfig.py`` is normally run from the user's
out-of-source build directory.
* The version number given to ``spack setup`` is arbitrary, just
like ``spack diy``. ``myproject/package.py`` does not need to
have any valid downloadable versions listed (typical when a
project is new).
* spconfig.py produces a CMake configuration that *does not* use the
Spack wrappers. Any resulting binaries *will not* use RPATH,
unless the user has enabled it. This is recommended for
development purposes, not production.
* ``spconfig.py`` is human readable, and can serve as a developer
reference of what dependencies are being used.
* ``make install`` installs the package into the Spack repository,
where it may be used by other Spack packages.
* CMake-generated makefiles re-run CMake in some circumstances. Use
of ``spconfig.py`` breaks this behavior, requiring the developer
to manually re-run ``spconfig.py`` when a ``CMakeLists.txt`` file
has changed.
CMakePackage
~~~~~~~~~~~~
In order ot enable ``spack setup`` functionality, the author of
``myproject/package.py`` must subclass from ``CMakePackage`` instead
of the standard ``Package`` superclass. Because CMake is
standardized, the packager does not need to tell Spack how to run
``cmake; make; make install``. Instead the packager only needs to
create (optional) methods ``configure_args()`` and ``configure_env()``, which
provide the arguments (as a list) and extra environment variables (as
a dict) to provide to the ``cmake`` command. Usually, these will
translate variant flags into CMake definitions. For example:
.. code-block:: python
def configure_args(self):
spec = self.spec
return [
'-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'),
'-DBUILD_PYTHON=%s' % ('YES' if '+python' in spec else 'NO'),
'-DBUILD_GRIDGEN=%s' % ('YES' if '+gridgen' in spec else 'NO'),
'-DBUILD_COUPLER=%s' % ('YES' if '+coupler' in spec else 'NO'),
'-DUSE_PISM=%s' % ('YES' if '+pism' in spec else 'NO')]
If needed, a packager may also override methods defined in
``StagedPackage`` (see below).
StagedPackage
~~~~~~~~~~~~~
``CMakePackage`` is implemented by subclassing the ``StagedPackage``
superclass, which breaks down the standard ``Package.install()``
method into several sub-stages: ``setup``, ``configure``, ``build``
and ``install``. Details:
* Instead of implementing the standard ``install()`` method, package
authors implement the methods for the sub-stages
``install_setup()``, ``install_configure()``,
``install_build()``, and ``install_install()``.
* The ``spack install`` command runs the sub-stages ``configure``,
``build`` and ``install`` in order. (The ``setup`` stage is
not run by default; see below).
* The ``spack setup`` command runs the sub-stages ``setup``
and a dummy install (to create the module file).
* The sub-stage install methods take no arguments (other than
``self``). The arguments ``spec`` and ``prefix`` to the standard
``install()`` method may be accessed via ``self.spec`` and
``self.prefix``.
GNU Autotools
~~~~~~~~~~~~~
The ``setup`` functionality is currently only available for
CMake-based packages. Extending this functionality to GNU
Autotools-based packages would be easy (and should be done by a
developer who actively uses Autotools). Packages that use
non-standard build systems can gain ``setup`` functionality by
subclassing ``StagedPackage`` directly.

4
lib/spack/spack/__init__.py
View file

@ -176,8 +176,10 @@
# TODO: it's not clear where all the stuff that needs to be included in packages
# should live. This file is overloaded for spack core vs. for packages.
#
__all__ = ['Package', 'Version', 'when', 'ver']
__all__ = ['Package', 'StagedPackage', 'CMakePackage', \
'Version', 'when', 'ver']
from spack.package import Package, ExtensionConflictError
from spack.package import StagedPackage, CMakePackage
from spack.version import Version, ver
from spack.multimethod import when

91
lib/spack/spack/cmd/setup.py Normal file
View file

@ -0,0 +1,91 @@
##############################################################################
# Copyright (c) 2016, Lawrence Livermore National Security, LLC.
# Produced at the Lawrence Livermore National Laboratory.
#
# This file is part of Spack.
# Written by Elizabeth Fischer
# LLNL-CODE-647188
#
# For details, see https://github.com/llnl/spack
# Please also see the LICENSE file for our notice and the LGPL.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License (as published by
# the Free Software Foundation) version 2.1 dated February 1999.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
# conditions of the GNU General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with this program; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
##############################################################################
import sys
import os
import argparse
import llnl.util.tty as tty
import spack
import spack.cmd
from spack.cmd.edit import edit_package
from spack.stage import DIYStage
description = "Create a configuration script and module, but don't build."
def setup_parser(subparser):
subparser.add_argument(
'-i', '--ignore-dependencies', action='store_true', dest='ignore_deps',
help="Do not try to install dependencies of requested packages.")
subparser.add_argument(
'-v', '--verbose', action='store_true', dest='verbose',
help="Display verbose build output while installing.")
subparser.add_argument(
'spec', nargs=argparse.REMAINDER,
help="specs to use for install. Must contain package AND verison.")
def setup(self, args):
if not args.spec:
tty.die("spack setup requires a package spec argument.")
specs = spack.cmd.parse_specs(args.spec)
if len(specs) > 1:
tty.die("spack setup only takes one spec.")
# Take a write lock before checking for existence.
with spack.installed_db.write_transaction():
spec = specs[0]
if not spack.repo.exists(spec.name):
tty.warn("No such package: %s" % spec.name)
create = tty.get_yes_or_no("Create this package?", default=False)
if not create:
tty.msg("Exiting without creating.")
sys.exit(1)
else:
tty.msg("Running 'spack edit -f %s'" % spec.name)
edit_package(spec.name, spack.repo.first_repo(), None, True)
return
if not spec.versions.concrete:
tty.die("spack setup spec must have a single, concrete version. Did you forget a package version number?")
spec.concretize()
package = spack.repo.get(spec)
# It's OK if the package is already installed.
# Forces the build to run out of the current directory.
package.stage = DIYStage(os.getcwd())
# TODO: make this an argument, not a global.
spack.do_checksum = False
package.do_install(
keep_prefix=True, # Don't remove install directory, even if you think you should
ignore_deps=args.ignore_deps,
verbose=args.verbose,
keep_stage=True, # don't remove source dir for SETUP.
install_phases = set(['setup', 'provenance']))

208
lib/spack/spack/package.py
View file

@ -37,6 +37,8 @@
import re
import textwrap
import time
import glob
import string
import llnl.util.tty as tty
import spack
@ -50,6 +52,8 @@
import spack.repository
import spack.url
import spack.util.web
from urlparse import urlparse
from StringIO import StringIO
from llnl.util.filesystem import *
from llnl.util.lang import *
@ -58,9 +62,11 @@
from spack.stage import Stage, ResourceStage, StageComposite
from spack.util.compression import allowed_archive
from spack.util.environment import dump_environment
from spack.util.executable import ProcessError
from spack.util.executable import ProcessError, Executable, which
from spack.version import *
from spack import directory_layout
from urlparse import urlparse
"""Allowed URL schemes for spack packages."""
_ALLOWED_URL_SCHEMES = ["http", "https", "ftp", "file", "git"]
@ -867,6 +873,7 @@ def _resource_stage(self, resource):
resource_stage_folder = '-'.join(pieces)
return resource_stage_folder
install_phases = set(['configure', 'build', 'install', 'provenance'])
def do_install(self,
keep_prefix=False,
keep_stage=False,
@ -875,7 +882,8 @@ def do_install(self,
verbose=False,
make_jobs=None,
fake=False,
explicit=False):
explicit=False,
install_phases = install_phases):
"""Called by commands to install a package and its dependencies.
Package implementations should override install() to describe
@ -903,7 +911,7 @@ def do_install(self,
return
# Ensure package is not already installed
if spack.install_layout.check_installed(self.spec):
if 'install' in install_phases and spack.install_layout.check_installed(self.spec):
tty.msg("%s is already installed in %s" % (self.name, self.prefix))
rec = spack.installed_db.get_record(self.spec)
if (not rec.explicit) and explicit:
@ -942,6 +950,10 @@ def build_process():
tty.msg("Building %s" % self.name)
self.stage.keep = keep_stage
self.install_phases = install_phases
self.build_directory = join_path(self.stage.path, 'spack-build')
self.source_directory = self.stage.source_path
with self.stage:
# Run the pre-install hook in the child process after
# the directory is created.
@ -973,19 +985,29 @@ def build_process():
raise e
# Ensure that something was actually installed.
self.sanity_check_prefix()
if 'install' in self.install_phases:
self.sanity_check_prefix()
# Copy provenance into the install directory on success
log_install_path = spack.install_layout.build_log_path(
self.spec)
env_install_path = spack.install_layout.build_env_path(
self.spec)
packages_dir = spack.install_layout.build_packages_path(
self.spec)
if 'provenance' in self.install_phases:
log_install_path = spack.install_layout.build_log_path(
self.spec)
env_install_path = spack.install_layout.build_env_path(
self.spec)
packages_dir = spack.install_layout.build_packages_path(
self.spec)
install(log_path, log_install_path)
install(env_path, env_install_path)
dump_packages(self.spec, packages_dir)
# Remove first if we're overwriting another build
# (can happen with spack setup)
try:
shutil.rmtree(packages_dir) # log_install_path and env_install_path are inside this
except:
pass
install(log_path, log_install_path)
install(env_path, env_install_path)
dump_packages(self.spec, packages_dir)
# Run post install hooks before build stage is removed.
spack.hooks.post_install(self)
@ -1003,6 +1025,18 @@ def build_process():
try:
# Create the install prefix and fork the build process.
spack.install_layout.create_install_directory(self.spec)
except directory_layout.InstallDirectoryAlreadyExistsError:
if 'install' in install_phases:
# Abort install if install directory exists.
# But do NOT remove it (you'd be overwriting someon else's stuff)
tty.warn("Keeping existing install prefix in place.")
raise
else:
# We're not installing anyway, so don't worry if someone
# else has already written in the install directory
pass
try:
spack.build_environment.fork(self, build_process)
except:
# remove the install prefix if anything went wrong during install.
@ -1013,7 +1047,7 @@ def build_process():
"Spack will think this package is installed. " +
"Manually remove this directory to fix:",
self.prefix,
wrap=True)
wrap=False)
raise
# note: PARENT of the build process adds the new package to
@ -1485,6 +1519,152 @@ def _hms(seconds):
parts.append("%.2fs" % s)
return ' '.join(parts)
class StagedPackage(Package):
"""A Package subclass where the install() is split up into stages."""
def install_setup(self):
"""Creates an spack_setup.py script to configure the package later if we like."""
raise InstallError("Package %s provides no install_setup() method!" % self.name)
def install_configure(self):
"""Runs the configure process."""
raise InstallError("Package %s provides no install_configure() method!" % self.name)
def install_build(self):
"""Runs the build process."""
raise InstallError("Package %s provides no install_build() method!" % self.name)
def install_install(self):
"""Runs the install process."""
raise InstallError("Package %s provides no install_install() method!" % self.name)
def install(self, spec, prefix):
if 'setup' in self.install_phases:
self.install_setup()
if 'configure' in self.install_phases:
self.install_configure()
if 'build' in self.install_phases:
self.install_build()
if 'install' in self.install_phases:
self.install_install()
else:
# Create a dummy file so the build doesn't fail.
# That way, the module file will also be created.
with open(os.path.join(prefix, 'dummy'), 'w') as fout:
pass
# stackoverflow.com/questions/12791997/how-do-you-do-a-simple-chmod-x-from-within-python
def make_executable(path):
mode = os.stat(path).st_mode
mode |= (mode & 0o444) >> 2 # copy R bits to X
os.chmod(path, mode)
class CMakePackage(StagedPackage):
def make_make(self):
import multiprocessing
# number of jobs spack will to build with.
jobs = multiprocessing.cpu_count()
if not self.parallel:
jobs = 1
elif self.make_jobs:
jobs = self.make_jobs
make = spack.build_environment.MakeExecutable('make', jobs)
return make
def configure_args(self):
"""Returns package-specific arguments to be provided to the configure command."""
return list()
def configure_env(self):
"""Returns package-specific environment under which the configure command should be run."""
return dict()
def spack_transitive_include_path(self):
return ';'.join(
os.path.join(dep, 'include')
for dep in os.environ['SPACK_DEPENDENCIES'].split(os.pathsep)
)
def install_setup(self):
cmd = [str(which('cmake'))] + \
spack.build_environment.get_std_cmake_args(self) + \
['-DCMAKE_INSTALL_PREFIX=%s' % os.environ['SPACK_PREFIX'],
'-DCMAKE_C_COMPILER=%s' % os.environ['SPACK_CC'],
'-DCMAKE_CXX_COMPILER=%s' % os.environ['SPACK_CXX'],
'-DCMAKE_Fortran_COMPILER=%s' % os.environ['SPACK_FC']] + \
self.configure_args()
env = dict()
env['PATH'] = os.environ['PATH']
env['SPACK_TRANSITIVE_INCLUDE_PATH'] = self.spack_transitive_include_path()
env['CMAKE_PREFIX_PATH'] = os.environ['CMAKE_PREFIX_PATH']
setup_fname = 'spconfig.py'
with open(setup_fname, 'w') as fout:
fout.write(\
r"""#!%s
#
import sys
import os
import subprocess
def cmdlist(str):
return list(x.strip().replace("'",'') for x in str.split('\n') if x)
env = dict(os.environ)
""" % sys.executable)
env_vars = sorted(list(env.keys()))
for name in env_vars:
val = env[name]
if string.find(name, 'PATH') < 0:
fout.write('env[%s] = %s\n' % (repr(name),repr(val)))
else:
if name == 'SPACK_TRANSITIVE_INCLUDE_PATH':
sep = ';'
else:
sep = ':'
fout.write('env[%s] = "%s".join(cmdlist("""\n' % (repr(name),sep))
for part in string.split(val, sep):
fout.write(' %s\n' % part)
fout.write('"""))\n')
fout.write("env['CMAKE_TRANSITIVE_INCLUDE_PATH'] = env['SPACK_TRANSITIVE_INCLUDE_PATH'] # Deprecated\n")
fout.write('\ncmd = cmdlist("""\n')
fout.write('%s\n' % cmd[0])
for arg in cmd[1:]:
fout.write(' %s\n' % arg)
fout.write('""") + sys.argv[1:]\n')
fout.write('\nproc = subprocess.Popen(cmd, env=env)\nproc.wait()\n')
make_executable(setup_fname)
def install_configure(self):
cmake = which('cmake')
with working_dir(self.build_directory, create=True):
os.environ.update(self.configure_env())
os.environ['SPACK_TRANSITIVE_INCLUDE_PATH'] = self.spack_transitive_include_path()
options = self.configure_args() + spack.build_environment.get_std_cmake_args(self)
cmake(self.source_directory, *options)
def install_build(self):
make = self.make_make()
with working_dir(self.build_directory, create=False):
make()
def install_install(self):
make = self.make_make()
with working_dir(self.build_directory, create=False):
make('install')
class FetchError(spack.error.SpackError):
"""Raised when something goes wrong during fetch."""

46
var/spack/repos/builtin/packages/ibmisc/package.py Normal file
View file

@ -0,0 +1,46 @@
from spack import *
class Ibmisc(CMakePackage):
"""Misc. reusable utilities used by IceBin."""
homepage = "https://github.com/citibeth/ibmisc"
url = "https://github.com/citibeth/ibmisc/tarball/123"
version('0.1.0', '12f2a32432a11db48e00217df18e59fa')
variant('everytrace', default=False, description='Report errors through Everytrace')
variant('proj', default=True, description='Compile utilities for PROJ.4 library')
variant('blitz', default=True, description='Compile utilities for Blitz library')
variant('netcdf', default=True, description='Compile utilities for NetCDF library')
variant('boost', default=True, description='Compile utilities for Boost library')
variant('udunits2', default=True, description='Compile utilities for UDUNITS2 library')
variant('googletest', default=True, description='Compile utilities for Google Test library')
variant('python', default=True, description='Compile utilities for use with Python/Cython')
extends('python')
depends_on('eigen')
depends_on('everytrace', when='+everytrace')
depends_on('proj', when='+proj')
depends_on('blitz', when='+blitz')
depends_on('netcdf-cxx4', when='+netcdf')
depends_on('udunits2', when='+udunits2')
depends_on('googletest', when='+googletest')
depends_on('py-cython', when='+python')
depends_on('py-numpy', when='+python')
depends_on('boost', when='+boost')
# Build dependencies
depends_on('cmake')
depends_on('doxygen')
def configure_args(self):
spec = self.spec
return [
'-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'),
'-DUSE_PROJ4=%s' % ('YES' if '+proj' in spec else 'NO'),
'-DUSE_BLITZ=%s' % ('YES' if '+blitz' in spec else 'NO'),
'-DUSE_NETCDF=%s' % ('YES' if '+netcdf' in spec else 'NO'),
'-DUSE_BOOST=%s' % ('YES' if '+boost' in spec else 'NO'),
'-DUSE_UDUNITS2=%s' % ('YES' if '+udunits2' in spec else 'NO'),
'-DUSE_GTEST=%s' % ('YES' if '+googletest' in spec else 'NO')]