hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Merge pull request #869 from brettviren/feature/views

Browse source 
Feature/views
This commit is contained in:
Tom Scogland 2016-06-05 11:31:48 -07:00
commit 30e8e77fb6
2 changed files with 410 additions and 0 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

115
lib/spack/docs/basic_usage.rst
View file

@ -342,6 +342,7 @@ will find every installed package with a 'debug' compile-time option enabled.
The full spec syntax is discussed in detail in :ref:`sec-specs`.
Compiler configuration
-----------------------------------
@ -1320,6 +1321,120 @@ regenerate all module and dotkit files from scratch:
.. _extensions:
Filesystem Views
-------------------------------
.. Maybe this is not the right location for this documentation.
The Spack installation area allows for many package installation trees
to coexist and gives the user choices as to what versions and variants
of packages to use. To use them, the user must rely on a way to
aggregate a subset of those packages. The section on Environment
Modules gives one good way to do that which relies on setting various
environment variables. An alternative way to aggregate is through
**filesystem views**.
A filesystem view is a single directory tree which is the union of the
directory hierarchies of the individual package installation trees
that have been included. The files of the view's installed packages
are brought into the view by symbolic or hard links back to their
location in the original Spack installation area. As the view is
formed, any clashes due to a file having the exact same path in its
package installation tree are handled in a first-come-first-served
basis and a warning is printed. Packages and their dependencies can
be both added and removed. During removal, empty directories will be
purged. These operations can be limited to pertain to just the
packages listed by the user or to exclude specific dependencies and
they allow for software installed outside of Spack to coexist inside
the filesystem view tree.
By its nature, a filesystem view represents a particular choice of one
set of packages among all the versions and variants that are available
in the Spack installation area. It is thus equivalent to the
directory hiearchy that might exist under ``/usr/local``. While this
limits a view to including only one version/variant of any package, it
provides the benefits of having a simpler and traditional layout which
may be used without any particular knowledge that its packages were
built by Spack.
Views can be used for a variety of purposes including:
- A central installation in a traditional layout, eg ``/usr/local`` maintained over time by the sysadmin.
- A self-contained installation area which may for the basis of a top-level atomic versioning scheme, eg ``/opt/pro`` vs ``/opt/dev``.
- Providing an atomic and monolithic binary distribution, eg for delivery as a single tarball.
- Producing ephemeral testing or developing environments.
Using Filesystem Views
~~~~~~~~~~~~~~~~~~~~~~
A filesystem view is created and packages are linked in by the ``spack
view`` command's ``symlink`` and ``hardlink`` sub-commands. The
``spack view remove`` command can be used to unlink some or all of the
filesystem view.
The following example creates a filesystem view based
on an installed ``cmake`` package and then removes from the view the
files in the ``cmake`` package while retaining its dependencies.
.. code-block:: sh
$ spack view -v symlink myview cmake@3.5.2
==> Linking package: "ncurses"
==> Linking package: "zlib"
==> Linking package: "openssl"
==> Linking package: "cmake"
$ ls myview/
bin doc etc include lib share
$ ls myview/bin/
captoinfo clear cpack ctest infotocap openssl tabs toe tset
ccmake cmake c_rehash infocmp ncurses6-config reset tic tput
$ spack view -v -d false rm myview cmake@3.5.2
==> Removing package: "cmake"
$ ls myview/bin/
captoinfo c_rehash infotocap openssl tabs toe tset
clear infocmp ncurses6-config reset tic tput
Limitations of Filesystem Views
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This section describes some limitations that should be considered in
using filesystems views.
Filesystem views are merely organizational. The binary executable
programs, shared libraries and other build products found in a view
are mere links into the "real" Spack installation area. If a view is
built with symbolic links it requires the Spack-installed package to
be kept in place. Building a view with hardlinks removes this
requirement but any internal paths (eg, rpath or ``#!`` interpreter
specifications) will still require the Spack-installed package files
to be in place.
.. FIXME: reference the relocation work of Hegner and Gartung.
As described above, when a view is built only a single instance of a
file may exist in the unified filesystem tree. If more than one
package provides a file at the same path (relative to its own root)
then it is the first package added to the view that "wins". A warning
is printed and it is up to the user to determine if the conflict
matters.
It is up to the user to assure a consistent view is produced. In
particular if the user excludes packages, limits the following of
dependencies or removes packages the view may become inconsistent. In
particular, if two packages require the same sub-tree of dependencies,
removing one package (recursively) will remove its dependencies and
leave the other package broken.
Extensions & Python support
------------------------------------

295
lib/spack/spack/cmd/view.py Normal file
View file

@ -0,0 +1,295 @@
##############################################################################
# Copyright (c) 2013, Lawrence Livermore National Security, LLC.
# Produced at the Lawrence Livermore National Laboratory.
#
# This file is part of Spack.
# Written by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
# LLNL-CODE-647188
#
# For details, see https://github.com/llnl/spack
# Please also see the LICENSE file for our notice and the LGPL.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License (as published by
# the Free Software Foundation) version 2.1 dated February 1999.
#
# This program is distributed in the hope that it will be useful, but
# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
# conditions of the GNU General Public License for more details.
#
# You should have received a copy of the GNU Lesser General Public License
# along with this program; if not, write to the Free Software Foundation,
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
##############################################################################
'''Produce a "view" of a Spack DAG.
A "view" is file hierarchy representing the union of a number of
Spack-installed package file hierarchies. The union is formed from:
- specs resolved from the package names given by the user (the seeds)
- all depenencies of the seeds unless user specifies `--no-depenencies`
- less any specs with names matching the regular expressions given by
`--exclude`
The `view` can be built and tore down via a number of methods (the "actions"):
- symlink :: a file system view which is a directory hierarchy that is
the union of the hierarchies of the installed packages in the DAG
where installed files are referenced via symlinks.
- hardlink :: like the symlink view but hardlinks are used.
- statlink :: a view producing a status report of a symlink or
hardlink view.
The file system view concept is imspired by Nix, implemented by
brett.viren@gmail.com ca 2016.
'''
# Implementation notes:
#
# This is implemented as a visitor pattern on the set of package specs.
#
# The command line ACTION maps to a visitor_*() function which takes
# the set of package specs and any args which may be specific to the
# ACTION.
#
# To add a new view:
# 1. add a new cmd line args sub parser ACTION
# 2. add any action-specific options/arguments, most likely a list of specs.
# 3. add a visitor_MYACTION() function
# 4. add any visitor_MYALIAS assignments to match any command line aliases
import os
import re
import spack
import spack.cmd
import llnl.util.tty as tty
description = "Produce a single-rooted directory view of a spec."
def setup_parser(sp):
setup_parser.parser = sp
sp.add_argument(
'-v', '--verbose', action='store_true', default=False,
help="Display verbose output.")
sp.add_argument(
'-e', '--exclude', action='append', default=[],
help="Exclude packages with names matching the given regex pattern.")
sp.add_argument(
'-d', '--dependencies', choices=['true', 'false', 'yes', 'no'],
default='true',
help="Follow dependencies.")
ssp = sp.add_subparsers(metavar='ACTION', dest='action')
specs_opts = dict(metavar='spec', nargs='+',
help="Seed specs of the packages to view.")
# The action parameterizes the command but in keeping with Spack
# patterns we make it a subcommand.
file_system_view_actions = [
ssp.add_parser(
'symlink', aliases=['add', 'soft'],
help='Add package files to a filesystem view via symbolic links.'),
ssp.add_parser(
'hardlink', aliases=['hard'],
help='Add packages files to a filesystem via via hard links.'),
ssp.add_parser(
'remove', aliases=['rm'],
help='Remove packages from a filesystem view.'),
ssp.add_parser(
'statlink', aliases=['status', 'check'],
help='Check status of packages in a filesystem view.')
]
# All these options and arguments are common to every action.
for act in file_system_view_actions:
act.add_argument('path', nargs=1,
help="Path to file system view directory.")
act.add_argument('specs', **specs_opts)
return
def assuredir(path):
'Assure path exists as a directory'
if not os.path.exists(path):
os.makedirs(path)
def relative_to(prefix, path):
'Return end of `path` relative to `prefix`'
assert 0 == path.find(prefix)
reldir = path[len(prefix):]
if reldir.startswith('/'):
reldir = reldir[1:]
return reldir
def transform_path(spec, path, prefix=None):
'Return the a relative path corresponding to given path spec.prefix'
if os.path.isabs(path):
path = relative_to(spec.prefix, path)
subdirs = path.split(os.path.sep)
if subdirs[0] == '.spack':
lst = ['.spack', spec.name] + subdirs[1:]
path = os.path.join(*lst)
if prefix:
path = os.path.join(prefix, path)
return path
def purge_empty_directories(path):
'''Ascend up from the leaves accessible from `path`
and remove empty directories.'''
for dirpath, subdirs, files in os.walk(path, topdown=False):
for sd in subdirs:
sdp = os.path.join(dirpath, sd)
try:
os.rmdir(sdp)
except OSError:
pass
def filter_exclude(specs, exclude):
'Filter specs given sequence of exclude regex'
to_exclude = [re.compile(e) for e in exclude]
def exclude(spec):
for e in to_exclude:
if e.match(spec.name):
return True
return False
return [s for s in specs if not exclude(s)]
def flatten(seeds, descend=True):
'Normalize and flattend seed specs and descend hiearchy'
flat = set()
for spec in seeds:
if not descend:
flat.add(spec)
continue
flat.update(spec.normalized().traverse())
return flat
def check_one(spec, path, verbose=False):
'Check status of view in path against spec'
dotspack = os.path.join(path, '.spack', spec.name)
if os.path.exists(os.path.join(dotspack)):
tty.info('Package in view: "%s"' % spec.name)
return
tty.info('Package not in view: "%s"' % spec.name)
return
def remove_one(spec, path, verbose=False):
'Remove any files found in `spec` from `path` and purge empty directories.'
if not os.path.exists(path):
return # done, short circuit
dotspack = transform_path(spec, '.spack', path)
if not os.path.exists(dotspack):
if verbose:
tty.info('Skipping nonexistent package: "%s"' % spec.name)
return
if verbose:
tty.info('Removing package: "%s"' % spec.name)
for dirpath, dirnames, filenames in os.walk(spec.prefix):
if not filenames:
continue
targdir = transform_path(spec, dirpath, path)
for fname in filenames:
dst = os.path.join(targdir, fname)
if not os.path.exists(dst):
continue
os.unlink(dst)
def link_one(spec, path, link=os.symlink, verbose=False):
'Link all files in `spec` into directory `path`.'
dotspack = transform_path(spec, '.spack', path)
if os.path.exists(dotspack):
tty.warn('Skipping existing package: "%s"' % spec.name)
return
if verbose:
tty.info('Linking package: "%s"' % spec.name)
for dirpath, dirnames, filenames in os.walk(spec.prefix):
if not filenames:
continue # avoid explicitly making empty dirs
targdir = transform_path(spec, dirpath, path)
assuredir(targdir)
for fname in filenames:
src = os.path.join(dirpath, fname)
dst = os.path.join(targdir, fname)
if os.path.exists(dst):
if '.spack' in dst.split(os.path.sep):
continue # silence these
tty.warn("Skipping existing file: %s" % dst)
continue
link(src, dst)
def visitor_symlink(specs, args):
'Symlink all files found in specs'
path = args.path[0]
assuredir(path)
for spec in specs:
link_one(spec, path, verbose=args.verbose)
visitor_add = visitor_symlink
visitor_soft = visitor_symlink
def visitor_hardlink(specs, args):
'Hardlink all files found in specs'
path = args.path[0]
assuredir(path)
for spec in specs:
link_one(spec, path, os.link, verbose=args.verbose)
visitor_hard = visitor_hardlink
def visitor_remove(specs, args):
'Remove all files and directories found in specs from args.path'
path = args.path[0]
for spec in specs:
remove_one(spec, path, verbose=args.verbose)
purge_empty_directories(path)
visitor_rm = visitor_remove
def visitor_statlink(specs, args):
'Give status of view in args.path relative to specs'
path = args.path[0]
for spec in specs:
check_one(spec, path, verbose=args.verbose)
visitor_status = visitor_statlink
visitor_check = visitor_statlink
def view(parser, args):
'Produce a view of a set of packages.'
# Process common args
seeds = [spack.cmd.disambiguate_spec(s) for s in args.specs]
specs = flatten(seeds, args.dependencies.lower() in ['yes', 'true'])
specs = filter_exclude(specs, args.exclude)
# Execute the visitation.
try:
visitor = globals()['visitor_' + args.action]
except KeyError:
tty.error('Unknown action: "%s"' % args.action)
visitor(specs, args)