Tom Scogland
commit
30e8e77fb6
2 changed files with 410 additions and 0 deletions
|
|
@ -342,6 +342,7 @@ will find every installed package with a 'debug' compile-time option enabled.
|
||||||
The full spec syntax is discussed in detail in :ref:`sec-specs`.
|
The full spec syntax is discussed in detail in :ref:`sec-specs`.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Compiler configuration
|
Compiler configuration
|
||||||
-----------------------------------
|
-----------------------------------
|
||||||
|
|
||||||
|
|
@ -1320,6 +1321,120 @@ regenerate all module and dotkit files from scratch:
|
||||||
|
|
||||||
.. _extensions:
|
.. _extensions:
|
||||||
|
|
||||||
|
Filesystem Views
|
||||||
|
-------------------------------
|
||||||
|
|
||||||
|
.. Maybe this is not the right location for this documentation.
|
||||||
|
|
||||||
|
The Spack installation area allows for many package installation trees
|
||||||
|
to coexist and gives the user choices as to what versions and variants
|
||||||
|
of packages to use. To use them, the user must rely on a way to
|
||||||
|
aggregate a subset of those packages. The section on Environment
|
||||||
|
Modules gives one good way to do that which relies on setting various
|
||||||
|
environment variables. An alternative way to aggregate is through
|
||||||
|
**filesystem views**.
|
||||||
|
|
||||||
|
A filesystem view is a single directory tree which is the union of the
|
||||||
|
directory hierarchies of the individual package installation trees
|
||||||
|
that have been included. The files of the view's installed packages
|
||||||
|
are brought into the view by symbolic or hard links back to their
|
||||||
|
location in the original Spack installation area. As the view is
|
||||||
|
formed, any clashes due to a file having the exact same path in its
|
||||||
|
package installation tree are handled in a first-come-first-served
|
||||||
|
basis and a warning is printed. Packages and their dependencies can
|
||||||
|
be both added and removed. During removal, empty directories will be
|
||||||
|
purged. These operations can be limited to pertain to just the
|
||||||
|
packages listed by the user or to exclude specific dependencies and
|
||||||
|
they allow for software installed outside of Spack to coexist inside
|
||||||
|
the filesystem view tree.
|
||||||
|
|
||||||
|
By its nature, a filesystem view represents a particular choice of one
|
||||||
|
set of packages among all the versions and variants that are available
|
||||||
|
in the Spack installation area. It is thus equivalent to the
|
||||||
|
directory hiearchy that might exist under ``/usr/local``. While this
|
||||||
|
limits a view to including only one version/variant of any package, it
|
||||||
|
provides the benefits of having a simpler and traditional layout which
|
||||||
|
may be used without any particular knowledge that its packages were
|
||||||
|
built by Spack.
|
||||||
|
|
||||||
|
Views can be used for a variety of purposes including:
|
||||||
|
|
||||||
|
- A central installation in a traditional layout, eg ``/usr/local`` maintained over time by the sysadmin.
|
||||||
|
- A self-contained installation area which may for the basis of a top-level atomic versioning scheme, eg ``/opt/pro`` vs ``/opt/dev``.
|
||||||
|
- Providing an atomic and monolithic binary distribution, eg for delivery as a single tarball.
|
||||||
|
- Producing ephemeral testing or developing environments.
|
||||||
|
|
||||||
|
Using Filesystem Views
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
A filesystem view is created and packages are linked in by the ``spack
|
||||||
|
view`` command's ``symlink`` and ``hardlink`` sub-commands. The
|
||||||
|
``spack view remove`` command can be used to unlink some or all of the
|
||||||
|
filesystem view.
|
||||||
|
|
||||||
|
The following example creates a filesystem view based
|
||||||
|
on an installed ``cmake`` package and then removes from the view the
|
||||||
|
files in the ``cmake`` package while retaining its dependencies.
|
||||||
|
|
||||||
|
.. code-block:: sh
|
||||||
|
|
||||||
|
|
||||||
|
$ spack view -v symlink myview cmake@3.5.2
|
||||||
|
==> Linking package: "ncurses"
|
||||||
|
==> Linking package: "zlib"
|
||||||
|
==> Linking package: "openssl"
|
||||||
|
==> Linking package: "cmake"
|
||||||
|
|
||||||
|
$ ls myview/
|
||||||
|
bin doc etc include lib share
|
||||||
|
|
||||||
|
$ ls myview/bin/
|
||||||
|
captoinfo clear cpack ctest infotocap openssl tabs toe tset
|
||||||
|
ccmake cmake c_rehash infocmp ncurses6-config reset tic tput
|
||||||
|
|
||||||
|
$ spack view -v -d false rm myview cmake@3.5.2
|
||||||
|
==> Removing package: "cmake"
|
||||||
|
|
||||||
|
$ ls myview/bin/
|
||||||
|
captoinfo c_rehash infotocap openssl tabs toe tset
|
||||||
|
clear infocmp ncurses6-config reset tic tput
|
||||||
|
|
||||||
|
|
||||||
|
Limitations of Filesystem Views
|
||||||
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
This section describes some limitations that should be considered in
|
||||||
|
using filesystems views.
|
||||||
|
|
||||||
|
Filesystem views are merely organizational. The binary executable
|
||||||
|
programs, shared libraries and other build products found in a view
|
||||||
|
are mere links into the "real" Spack installation area. If a view is
|
||||||
|
built with symbolic links it requires the Spack-installed package to
|
||||||
|
be kept in place. Building a view with hardlinks removes this
|
||||||
|
requirement but any internal paths (eg, rpath or ``#!`` interpreter
|
||||||
|
specifications) will still require the Spack-installed package files
|
||||||
|
to be in place.
|
||||||
|
|
||||||
|
.. FIXME: reference the relocation work of Hegner and Gartung.
|
||||||
|
|
||||||
|
As described above, when a view is built only a single instance of a
|
||||||
|
file may exist in the unified filesystem tree. If more than one
|
||||||
|
package provides a file at the same path (relative to its own root)
|
||||||
|
then it is the first package added to the view that "wins". A warning
|
||||||
|
is printed and it is up to the user to determine if the conflict
|
||||||
|
matters.
|
||||||
|
|
||||||
|
It is up to the user to assure a consistent view is produced. In
|
||||||
|
particular if the user excludes packages, limits the following of
|
||||||
|
dependencies or removes packages the view may become inconsistent. In
|
||||||
|
particular, if two packages require the same sub-tree of dependencies,
|
||||||
|
removing one package (recursively) will remove its dependencies and
|
||||||
|
leave the other package broken.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
Extensions & Python support
|
Extensions & Python support
|
||||||
------------------------------------
|
------------------------------------
|
||||||
|
|
||||||
|
|
|
||||||
295
lib/spack/spack/cmd/view.py
Normal file
295
lib/spack/spack/cmd/view.py
Normal file
|
|
@ -0,0 +1,295 @@
|
||||||
|
##############################################################################
|
||||||
|
# Copyright (c) 2013, Lawrence Livermore National Security, LLC.
|
||||||
|
# Produced at the Lawrence Livermore National Laboratory.
|
||||||
|
#
|
||||||
|
# This file is part of Spack.
|
||||||
|
# Written by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
|
||||||
|
# LLNL-CODE-647188
|
||||||
|
#
|
||||||
|
# For details, see https://github.com/llnl/spack
|
||||||
|
# Please also see the LICENSE file for our notice and the LGPL.
|
||||||
|
#
|
||||||
|
# This program is free software; you can redistribute it and/or modify
|
||||||
|
# it under the terms of the GNU General Public License (as published by
|
||||||
|
# the Free Software Foundation) version 2.1 dated February 1999.
|
||||||
|
#
|
||||||
|
# This program is distributed in the hope that it will be useful, but
|
||||||
|
# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
|
||||||
|
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
|
||||||
|
# conditions of the GNU General Public License for more details.
|
||||||
|
#
|
||||||
|
# You should have received a copy of the GNU Lesser General Public License
|
||||||
|
# along with this program; if not, write to the Free Software Foundation,
|
||||||
|
# Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
|
||||||
|
##############################################################################
|
||||||
|
'''Produce a "view" of a Spack DAG.
|
||||||
|
|
||||||
|
A "view" is file hierarchy representing the union of a number of
|
||||||
|
Spack-installed package file hierarchies. The union is formed from:
|
||||||
|
|
||||||
|
- specs resolved from the package names given by the user (the seeds)
|
||||||
|
|
||||||
|
- all depenencies of the seeds unless user specifies `--no-depenencies`
|
||||||
|
|
||||||
|
- less any specs with names matching the regular expressions given by
|
||||||
|
`--exclude`
|
||||||
|
|
||||||
|
The `view` can be built and tore down via a number of methods (the "actions"):
|
||||||
|
|
||||||
|
- symlink :: a file system view which is a directory hierarchy that is
|
||||||
|
the union of the hierarchies of the installed packages in the DAG
|
||||||
|
where installed files are referenced via symlinks.
|
||||||
|
|
||||||
|
- hardlink :: like the symlink view but hardlinks are used.
|
||||||
|
|
||||||
|
- statlink :: a view producing a status report of a symlink or
|
||||||
|
hardlink view.
|
||||||
|
|
||||||
|
The file system view concept is imspired by Nix, implemented by
|
||||||
|
brett.viren@gmail.com ca 2016.
|
||||||
|
|
||||||
|
'''
|
||||||
|
# Implementation notes:
|
||||||
|
#
|
||||||
|
# This is implemented as a visitor pattern on the set of package specs.
|
||||||
|
#
|
||||||
|
# The command line ACTION maps to a visitor_*() function which takes
|
||||||
|
# the set of package specs and any args which may be specific to the
|
||||||
|
# ACTION.
|
||||||
|
#
|
||||||
|
# To add a new view:
|
||||||
|
# 1. add a new cmd line args sub parser ACTION
|
||||||
|
# 2. add any action-specific options/arguments, most likely a list of specs.
|
||||||
|
# 3. add a visitor_MYACTION() function
|
||||||
|
# 4. add any visitor_MYALIAS assignments to match any command line aliases
|
||||||
|
|
||||||
|
import os
|
||||||
|
import re
|
||||||
|
import spack
|
||||||
|
import spack.cmd
|
||||||
|
import llnl.util.tty as tty
|
||||||
|
|
||||||
|
description = "Produce a single-rooted directory view of a spec."
|
||||||
|
|
||||||
|
|
||||||
|
def setup_parser(sp):
|
||||||
|
setup_parser.parser = sp
|
||||||
|
|
||||||
|
sp.add_argument(
|
||||||
|
'-v', '--verbose', action='store_true', default=False,
|
||||||
|
help="Display verbose output.")
|
||||||
|
sp.add_argument(
|
||||||
|
'-e', '--exclude', action='append', default=[],
|
||||||
|
help="Exclude packages with names matching the given regex pattern.")
|
||||||
|
sp.add_argument(
|
||||||
|
'-d', '--dependencies', choices=['true', 'false', 'yes', 'no'],
|
||||||
|
default='true',
|
||||||
|
help="Follow dependencies.")
|
||||||
|
|
||||||
|
ssp = sp.add_subparsers(metavar='ACTION', dest='action')
|
||||||
|
|
||||||
|
specs_opts = dict(metavar='spec', nargs='+',
|
||||||
|
help="Seed specs of the packages to view.")
|
||||||
|
|
||||||
|
# The action parameterizes the command but in keeping with Spack
|
||||||
|
# patterns we make it a subcommand.
|
||||||
|
file_system_view_actions = [
|
||||||
|
ssp.add_parser(
|
||||||
|
'symlink', aliases=['add', 'soft'],
|
||||||
|
help='Add package files to a filesystem view via symbolic links.'),
|
||||||
|
ssp.add_parser(
|
||||||
|
'hardlink', aliases=['hard'],
|
||||||
|
help='Add packages files to a filesystem via via hard links.'),
|
||||||
|
ssp.add_parser(
|
||||||
|
'remove', aliases=['rm'],
|
||||||
|
help='Remove packages from a filesystem view.'),
|
||||||
|
ssp.add_parser(
|
||||||
|
'statlink', aliases=['status', 'check'],
|
||||||
|
help='Check status of packages in a filesystem view.')
|
||||||
|
]
|
||||||
|
# All these options and arguments are common to every action.
|
||||||
|
for act in file_system_view_actions:
|
||||||
|
act.add_argument('path', nargs=1,
|
||||||
|
help="Path to file system view directory.")
|
||||||
|
act.add_argument('specs', **specs_opts)
|
||||||
|
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
|
def assuredir(path):
|
||||||
|
'Assure path exists as a directory'
|
||||||
|
if not os.path.exists(path):
|
||||||
|
os.makedirs(path)
|
||||||
|
|
||||||
|
|
||||||
|
def relative_to(prefix, path):
|
||||||
|
'Return end of `path` relative to `prefix`'
|
||||||
|
assert 0 == path.find(prefix)
|
||||||
|
reldir = path[len(prefix):]
|
||||||
|
if reldir.startswith('/'):
|
||||||
|
reldir = reldir[1:]
|
||||||
|
return reldir
|
||||||
|
|
||||||
|
|
||||||
|
def transform_path(spec, path, prefix=None):
|
||||||
|
'Return the a relative path corresponding to given path spec.prefix'
|
||||||
|
if os.path.isabs(path):
|
||||||
|
path = relative_to(spec.prefix, path)
|
||||||
|
subdirs = path.split(os.path.sep)
|
||||||
|
if subdirs[0] == '.spack':
|
||||||
|
lst = ['.spack', spec.name] + subdirs[1:]
|
||||||
|
path = os.path.join(*lst)
|
||||||
|
if prefix:
|
||||||
|
path = os.path.join(prefix, path)
|
||||||
|
return path
|
||||||
|
|
||||||
|
|
||||||
|
def purge_empty_directories(path):
|
||||||
|
'''Ascend up from the leaves accessible from `path`
|
||||||
|
and remove empty directories.'''
|
||||||
|
for dirpath, subdirs, files in os.walk(path, topdown=False):
|
||||||
|
for sd in subdirs:
|
||||||
|
sdp = os.path.join(dirpath, sd)
|
||||||
|
try:
|
||||||
|
os.rmdir(sdp)
|
||||||
|
except OSError:
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
def filter_exclude(specs, exclude):
|
||||||
|
'Filter specs given sequence of exclude regex'
|
||||||
|
to_exclude = [re.compile(e) for e in exclude]
|
||||||
|
|
||||||
|
def exclude(spec):
|
||||||
|
for e in to_exclude:
|
||||||
|
if e.match(spec.name):
|
||||||
|
return True
|
||||||
|
return False
|
||||||
|
return [s for s in specs if not exclude(s)]
|
||||||
|
|
||||||
|
|
||||||
|
def flatten(seeds, descend=True):
|
||||||
|
'Normalize and flattend seed specs and descend hiearchy'
|
||||||
|
flat = set()
|
||||||
|
for spec in seeds:
|
||||||
|
if not descend:
|
||||||
|
flat.add(spec)
|
||||||
|
continue
|
||||||
|
flat.update(spec.normalized().traverse())
|
||||||
|
return flat
|
||||||
|
|
||||||
|
|
||||||
|
def check_one(spec, path, verbose=False):
|
||||||
|
'Check status of view in path against spec'
|
||||||
|
dotspack = os.path.join(path, '.spack', spec.name)
|
||||||
|
if os.path.exists(os.path.join(dotspack)):
|
||||||
|
tty.info('Package in view: "%s"' % spec.name)
|
||||||
|
return
|
||||||
|
tty.info('Package not in view: "%s"' % spec.name)
|
||||||
|
return
|
||||||
|
|
||||||
|
|
||||||
|
def remove_one(spec, path, verbose=False):
|
||||||
|
'Remove any files found in `spec` from `path` and purge empty directories.'
|
||||||
|
|
||||||
|
if not os.path.exists(path):
|
||||||
|
return # done, short circuit
|
||||||
|
|
||||||
|
dotspack = transform_path(spec, '.spack', path)
|
||||||
|
if not os.path.exists(dotspack):
|
||||||
|
if verbose:
|
||||||
|
tty.info('Skipping nonexistent package: "%s"' % spec.name)
|
||||||
|
return
|
||||||
|
|
||||||
|
if verbose:
|
||||||
|
tty.info('Removing package: "%s"' % spec.name)
|
||||||
|
for dirpath, dirnames, filenames in os.walk(spec.prefix):
|
||||||
|
if not filenames:
|
||||||
|
continue
|
||||||
|
targdir = transform_path(spec, dirpath, path)
|
||||||
|
for fname in filenames:
|
||||||
|
dst = os.path.join(targdir, fname)
|
||||||
|
if not os.path.exists(dst):
|
||||||
|
continue
|
||||||
|
os.unlink(dst)
|
||||||
|
|
||||||
|
|
||||||
|
def link_one(spec, path, link=os.symlink, verbose=False):
|
||||||
|
'Link all files in `spec` into directory `path`.'
|
||||||
|
|
||||||
|
dotspack = transform_path(spec, '.spack', path)
|
||||||
|
if os.path.exists(dotspack):
|
||||||
|
tty.warn('Skipping existing package: "%s"' % spec.name)
|
||||||
|
return
|
||||||
|
|
||||||
|
if verbose:
|
||||||
|
tty.info('Linking package: "%s"' % spec.name)
|
||||||
|
for dirpath, dirnames, filenames in os.walk(spec.prefix):
|
||||||
|
if not filenames:
|
||||||
|
continue # avoid explicitly making empty dirs
|
||||||
|
|
||||||
|
targdir = transform_path(spec, dirpath, path)
|
||||||
|
assuredir(targdir)
|
||||||
|
|
||||||
|
for fname in filenames:
|
||||||
|
src = os.path.join(dirpath, fname)
|
||||||
|
dst = os.path.join(targdir, fname)
|
||||||
|
if os.path.exists(dst):
|
||||||
|
if '.spack' in dst.split(os.path.sep):
|
||||||
|
continue # silence these
|
||||||
|
tty.warn("Skipping existing file: %s" % dst)
|
||||||
|
continue
|
||||||
|
link(src, dst)
|
||||||
|
|
||||||
|
|
||||||
|
def visitor_symlink(specs, args):
|
||||||
|
'Symlink all files found in specs'
|
||||||
|
path = args.path[0]
|
||||||
|
assuredir(path)
|
||||||
|
for spec in specs:
|
||||||
|
link_one(spec, path, verbose=args.verbose)
|
||||||
|
visitor_add = visitor_symlink
|
||||||
|
visitor_soft = visitor_symlink
|
||||||
|
|
||||||
|
|
||||||
|
def visitor_hardlink(specs, args):
|
||||||
|
'Hardlink all files found in specs'
|
||||||
|
path = args.path[0]
|
||||||
|
assuredir(path)
|
||||||
|
for spec in specs:
|
||||||
|
link_one(spec, path, os.link, verbose=args.verbose)
|
||||||
|
visitor_hard = visitor_hardlink
|
||||||
|
|
||||||
|
|
||||||
|
def visitor_remove(specs, args):
|
||||||
|
'Remove all files and directories found in specs from args.path'
|
||||||
|
path = args.path[0]
|
||||||
|
for spec in specs:
|
||||||
|
remove_one(spec, path, verbose=args.verbose)
|
||||||
|
purge_empty_directories(path)
|
||||||
|
visitor_rm = visitor_remove
|
||||||
|
|
||||||
|
|
||||||
|
def visitor_statlink(specs, args):
|
||||||
|
'Give status of view in args.path relative to specs'
|
||||||
|
path = args.path[0]
|
||||||
|
for spec in specs:
|
||||||
|
check_one(spec, path, verbose=args.verbose)
|
||||||
|
visitor_status = visitor_statlink
|
||||||
|
visitor_check = visitor_statlink
|
||||||
|
|
||||||
|
|
||||||
|
def view(parser, args):
|
||||||
|
'Produce a view of a set of packages.'
|
||||||
|
|
||||||
|
# Process common args
|
||||||
|
seeds = [spack.cmd.disambiguate_spec(s) for s in args.specs]
|
||||||
|
specs = flatten(seeds, args.dependencies.lower() in ['yes', 'true'])
|
||||||
|
specs = filter_exclude(specs, args.exclude)
|
||||||
|
|
||||||
|
# Execute the visitation.
|
||||||
|
try:
|
||||||
|
visitor = globals()['visitor_' + args.action]
|
||||||
|
except KeyError:
|
||||||
|
tty.error('Unknown action: "%s"' % args.action)
|
||||||
|
visitor(specs, args)
|
||||||
Loading…
Reference in a new issue