views: packages can customize how they're added to views (#7152)

Functional updates:

- `python` now creates a copy of the `python` binaries when it is added
  to a view

- Python extensions (packages which subclass `PythonPackage`) rewrite
  their shebang lines to refer to python in the view

- Python packages in the same namespace will not generate conflicts if
  both have `...lib/site-packages/namespace-example/__init__.py`

  - These `__init__` files will also remain when removing any package in
    the namespace until the last package in the namespace is removed


Generally (Updated 2/16):

- Any package can define `add_files_to_view` to customize how it is added
  to a view (and at the moment custom definitions are included for
  `python` and `PythonPackage`)

  - Likewise any package can define `remove_files_from_view` to customize
    which files are removed (e.g. you don't always want to remove the
    namespace `__init__`)

- Any package can define `view_file_conflicts` to customize what it
  considers a merge conflict

- Global activations are handled like views (where the view root is the
  spec prefix of the extendee)

  - Benefit: filesystem-management aspects of activating extensions are
    now placed in views (e.g. now one can hardlink a global activation)

  - Benefit: overriding `Package.activate` is more straightforward (see
    `Python.activate`)

  - Complication: extension packages which have special-purpose logic
    *only* when activated outside of the extendee prefix must check for
    this in their `add_files_to_view` method (see `PythonPackage`)

- `LinkTree` is refactored to have separate methods for copying a
  directory structure and for copying files (since it was found that
  generally packages may want to alter how files are copied but still
  wanted to copy directories in the same way)


TODOs (updated 2/20):

- [x] additional testing (there is some unit testing added at this point
  but more would be useful)

- [x] refactor or reorganize `LinkTree` methods: currently there is a
  separate set of methods for replicating just the directory structure
  without the files, and a set for replicating everything

- [x] Right now external views (i.e. those not used for global
  activations) call `view.add_extension`, but global activations do not
  to avoid some extra work that goes into maintaining external views. I'm
  not sure if addressing that needs to be done here but I'd like to
  clarify it in the comments (UPDATE: for now I have added a TODO and in
  my opinion this can be merged now and the refactor handled later)

- [x] Several method descriptions (e.g. for `Package.activate`) are out
  of date and reference a distinction between global activations and
  views, they need to be updated

- [x] Update aspell package activations

lib/spack/docs/basic_usage.rst

@@ -950,11 +950,11 @@ directly when you run ``python``:
    ImportError: No module named numpy
    >>>
 
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^
-Extensions & Environment Modules
+Using Extensions
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^
 
-There are two ways to get ``numpy`` working in Python.  The first is
+There are three ways to get ``numpy`` working in Python.  The first is
 to use :ref:`shell-support`.  You can simply ``use`` or ``load`` the
 module for the extension, and it will be added to the ``PYTHONPATH``
 in your current shell.
@@ -976,15 +976,26 @@ or, for dotkit:
 Now ``import numpy`` will succeed for as long as you keep your current
 session open.
 
-^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-Activating Extensions
+Activating Extensions in a View
-^^^^^^^^^^^^^^^^^^^^^
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-It is often desirable to have certain packages *always* available as
+The second way to use extensions is to create a view, which merges the
-part of a Python installation.  Spack offers a more permanent solution
+python installation along with the extensions into a single prefix.
-for this case.  Instead of requiring users to load particular
+See :ref:`filesystem-views` for a more in-depth description of views and
-environment modules, you can *activate* the package within the Python
+:ref:`cmd-spack-view` for usage of the ``spack view`` command.
-installation:
+
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Activating Extensions Globally
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+As an alternative to creating a merged prefix with Python and its extensions,
+and prior to support for views, Spack has provided a means to install the
+extension into the Spack installation prefix for the extendee. This has
+typically been useful since extendable packages typically search their own
+installation path for addons by default.
+
+Global activations are performed with the ``spack activate`` command:
 
 .. _cmd-spack-activate:
 
@@ -1044,11 +1055,11 @@ the ``py-numpy`` into the prefix of the ``python`` package.  To the
 python interpreter, it looks like ``numpy`` is installed in the
 ``site-packages`` directory.
 
-The only limitation of activation is that you can only have a *single*
+The only limitation of global activation is that you can only have a *single*
 version of an extension activated at a time.  This is because multiple
 versions of the same extension would conflict if symbolically linked
 into the same prefix.  Users who want a different version of a package
-can still get it by using environment modules, but they will have to
+can still get it by using environment modules or views, but they will have to
 explicitly load their preferred version.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^

lib/spack/docs/packaging_guide.rst

@@ -1854,18 +1854,38 @@ from being linked in at activation time.
    ``depends_on('python')`` and ``extends(python)`` in the same
    package.  ``extends`` implies ``depends_on``.
 
+-----
+Views
+-----
+
+As covered in :ref:`filesystem-views`, the ``spack view`` command can be
+used to symlink a number of packages into a merged prefix. The methods of
+``PackageViewMixin`` can be overridden to customize how packages are added
+to views. Generally this can be used to create copies of specific files rather
+than symlinking them when symlinking does not work. For example, ``Python``
+overrides ``add_files_to_view`` in order to create a copy of the ``python``
+binary since the real path of the Python executable is used to detect
+extensions; as a consequence python extension packages (those inheriting from
+``PythonPackage``) likewise override ``add_files_to_view`` in order to rewrite
+shebang lines which point to the Python interpreter.
+
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 Activation & deactivation
 ^^^^^^^^^^^^^^^^^^^^^^^^^
 
+Adding an extension to a view is referred to as an activation. If the view is
+maintained in the Spack installation prefix of the extendee this is called a
+global activation. Activations may involve updating some centralized state
+that is maintained by the extendee package, so there can be additional work
+for adding extensions compared with non-extension packages.
+
 Spack's ``Package`` class has default ``activate`` and ``deactivate``
 implementations that handle symbolically linking extensions' prefixes
-into the directory of the parent package.  However, extendable
+into a specified view. Extendable packages can override these methods
-packages can override these methods to add custom activate/deactivate
+to add custom activate/deactivate logic of their own.  For example,
-logic of their own.  For example, the ``activate`` and ``deactivate``
+the ``activate`` and ``deactivate`` methods in the Python class handle
-methods in the Python class use the symbolic linking, but they also
+symbolic linking of extensions, but they also handle details surrounding
-handle details surrounding Python's ``.pth`` files, and other aspects
+Python's ``.pth`` files, and other aspects of Python packaging.
-of Python packaging.
 
 Spack's extensions mechanism is designed to be extensible, so that
 other packages (like Ruby, R, Perl, etc.)  can provide their own
@@ -1880,7 +1900,7 @@ Let's look at Python's activate function:
 
 This function is called on the *extendee* (Python).  It first calls
 ``activate`` in the superclass, which handles symlinking the
-extension package's prefix into this package's prefix.  It then does
+extension package's prefix into the specified view.  It then does
 some special handling of the ``easy-install.pth`` file, part of
 Python's setuptools.
 

lib/spack/docs/workflows.rst

@@ -402,31 +402,6 @@ Numpy, core Python, BLAS/LAPACK and anything else needed:
 
    spack module loads --dependencies py-scipy
 
-^^^^^^^^^^^^^^^^^^
-Extension Packages
-^^^^^^^^^^^^^^^^^^
-
-:ref:`packaging_extensions` may be used as an alternative to loading
-Python (and similar systems) packages directly.  If extensions are
-activated, then ``spack load python`` will also load all the
-extensions activated for the given ``python``.  This reduces the need
-for users to load a large number of modules.
-
-However, Spack extensions have two potential drawbacks:
-
-#. Activated packages that involve compiled C extensions may still
-   need their dependencies to be loaded manually.  For example,
-   ``spack load openblas`` might be required to make ``py-numpy``
-   work.
-
-#. Extensions "break" a core feature of Spack, which is that multiple
-   versions of a package can co-exist side-by-side.  For example,
-   suppose you wish to run a Python package in two different
-   environments but the same basic Python --- one with
-   ``py-numpy@1.7`` and one with ``py-numpy@1.8``.  Spack extensions
-   will not support this potential debugging use case.
-
-
 ^^^^^^^^^^^^^^
 Dummy Packages
 ^^^^^^^^^^^^^^
@@ -447,6 +422,8 @@ it.  A disadvantage is the set of packages will be consistent; this
 means you cannot load up two applications this way if they are not
 consistent with each other.
 
+.. _filesystem-views:
+
 ^^^^^^^^^^^^^^^^
 Filesystem Views
 ^^^^^^^^^^^^^^^^
@@ -587,6 +564,29 @@ symlinks.  At any time one can delete ``/path/to/MYVIEW`` or use
 ``spack view`` to manage it surgically.  None of this will affect the
 real Spack install area.
 
+^^^^^^^^^^^^^^^^^^
+Global Activations
+^^^^^^^^^^^^^^^^^^
+
+:ref:`cmd-spack-activate` may be used as an alternative to loading
+Python (and similar systems) packages directly or creating a view.
+If extensions are globally activated, then ``spack load python`` will
+also load all the extensions activated for the given ``python``.
+This reduces the need for users to load a large number of modules.
+
+However, Spack global activations have two potential drawbacks:
+
+#. Activated packages that involve compiled C extensions may still
+   need their dependencies to be loaded manually.  For example,
+   ``spack load openblas`` might be required to make ``py-numpy``
+   work.
+
+#. Global activations "break" a core feature of Spack, which is that
+   multiple versions of a package can co-exist side-by-side.  For example,
+   suppose you wish to run a Python package in two different
+   environments but the same basic Python --- one with
+   ``py-numpy@1.7`` and one with ``py-numpy@1.8``.  Spack extensions
+   will not support this potential debugging use case.
 
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Discussion: Running Binaries

lib/spack/llnl/util/filesystem.py

@@ -79,6 +79,18 @@
 ]
 
 
+def path_contains_subdirectory(path, root):
+    norm_root = os.path.abspath(root).rstrip(os.path.sep) + os.path.sep
+    norm_path = os.path.abspath(path).rstrip(os.path.sep) + os.path.sep
+    return norm_path.startswith(norm_root)
+
+
+def same_path(path1, path2):
+    norm1 = os.path.abspath(path1).rstrip(os.path.sep)
+    norm2 = os.path.abspath(path2).rstrip(os.path.sep)
+    return norm1 == norm2
+
+
 def filter_file(regex, repl, *filenames, **kwargs):
     r"""Like sed, but uses python regular expressions.
 
@@ -281,6 +293,17 @@ def is_exe(path):
     return os.path.isfile(path) and os.access(path, os.X_OK)
 
 
+def get_filetype(path_name):
+    """
+    Return the output of file path_name as a string to identify file type.
+    """
+    file = Executable('file')
+    file.add_default_env('LC_ALL', 'C')
+    output = file('-b', '-h', '%s' % path_name,
+                  output=str, error=str)
+    return output.strip()
+
+
 def mkdirp(*paths):
     """Creates a directory, as well as parent directories if needed."""
     for path in paths:

lib/spack/llnl/util/link_tree.py

@@ -29,6 +29,7 @@
 import filecmp
 
 from llnl.util.filesystem import traverse_tree, mkdirp, touch
+import llnl.util.tty as tty
 
 __all__ = ['LinkTree']
 
@@ -44,37 +45,49 @@ class LinkTree(object):
     Trees comprise symlinks only to files; directries are never
     symlinked to, to prevent the source directory from ever being
     modified.
-
     """
-
     def __init__(self, source_root):
         if not os.path.exists(source_root):
             raise IOError("No such file or directory: '%s'", source_root)
 
         self._root = source_root
 
-    def find_conflict(self, dest_root, **kwargs):
+    def find_conflict(self, dest_root, ignore=None,
+                      ignore_file_conflicts=False):
         """Returns the first file in dest that conflicts with src"""
-        kwargs['follow_nonexisting'] = False
+        ignore = ignore or (lambda x: False)
+        conflicts = self.find_dir_conflicts(dest_root, ignore)
+
+        if not ignore_file_conflicts:
+            conflicts.extend(
+                dst for src, dst
+                in self.get_file_map(dest_root, ignore).items()
+                if os.path.exists(dst))
+
+        if conflicts:
+            return conflicts[0]
+
+    def find_dir_conflicts(self, dest_root, ignore):
+        conflicts = []
+        kwargs = {'follow_nonexisting': False, 'ignore': ignore}
         for src, dest in traverse_tree(self._root, dest_root, **kwargs):
             if os.path.isdir(src):
                 if os.path.exists(dest) and not os.path.isdir(dest):
-                    return dest
+                    conflicts.append("File blocks directory: %s" % dest)
-            elif os.path.exists(dest):
+            elif os.path.exists(dest) and os.path.isdir(dest):
-                return dest
+                conflicts.append("Directory blocks directory: %s" % dest)
-        return None
+        return conflicts
 
-    def merge(self, dest_root, link=os.symlink, **kwargs):
+    def get_file_map(self, dest_root, ignore):
-        """Link all files in src into dest, creating directories
+        merge_map = {}
-           if necessary.
+        kwargs = {'follow_nonexisting': True, 'ignore': ignore}
-           If ignore_conflicts is True, do not break when the target exists but
-           rather return a list of files that could not be linked.
-           Note that files blocking directories will still cause an error.
-        """
-        kwargs['order'] = 'pre'
-        ignore_conflicts = kwargs.get("ignore_conflicts", False)
-        existing = []
         for src, dest in traverse_tree(self._root, dest_root, **kwargs):
+            if not os.path.isdir(src):
+                merge_map[src] = dest
+        return merge_map
+
+    def merge_directories(self, dest_root, ignore):
+        for src, dest in traverse_tree(self._root, dest_root, ignore=ignore):
             if os.path.isdir(src):
                 if not os.path.exists(dest):
                     mkdirp(dest)
@@ -88,31 +101,13 @@ def merge(self, dest_root, link=os.symlink, **kwargs):
                     marker = os.path.join(dest, empty_file_name)
                     touch(marker)
 
-            else:
+    def unmerge_directories(self, dest_root, ignore):
-                if os.path.exists(dest):
+        for src, dest in traverse_tree(
-                    if ignore_conflicts:
+                self._root, dest_root, ignore=ignore, order='post'):
-                        existing.append(src)
-                    else:
-                        raise AssertionError("File already exists: %s" % dest)
-                else:
-                    link(src, dest)
-        if ignore_conflicts:
-            return existing
-
-    def unmerge(self, dest_root, **kwargs):
-        """Unlink all files in dest that exist in src.
-
-        Unlinks directories in dest if they are empty.
-
-        """
-        kwargs['order'] = 'post'
-        for src, dest in traverse_tree(self._root, dest_root, **kwargs):
             if os.path.isdir(src):
-                # Skip non-existing links.
                 if not os.path.exists(dest):
                     continue
-
+                elif not os.path.isdir(dest):
-                if not os.path.isdir(dest):
                     raise ValueError("File blocks directory: %s" % dest)
 
                 # remove directory if it is empty.
@@ -124,11 +119,61 @@ def unmerge(self, dest_root, **kwargs):
                 if os.path.exists(marker):
                     os.remove(marker)
 
-            elif os.path.exists(dest):
+    def merge(self, dest_root, **kwargs):
-                if not os.path.islink(dest):
+        """Link all files in src into dest, creating directories
-                    raise ValueError("%s is not a link tree!" % dest)
+           if necessary.
-                # remove if dest is a hardlink/symlink to src; this will only
+           If ignore_conflicts is True, do not break when the target exists but
-                # be false if two packages are merged into a prefix and have a
+           rather return a list of files that could not be linked.
-                # conflicting file
+           Note that files blocking directories will still cause an error.
-                if filecmp.cmp(src, dest, shallow=True):
+        """
-                    os.remove(dest)
+        ignore_conflicts = kwargs.get("ignore_conflicts", False)
+
+        ignore = kwargs.get('ignore', lambda x: False)
+        conflict = self.find_conflict(
+            dest_root, ignore=ignore, ignore_file_conflicts=ignore_conflicts)
+        if conflict:
+            raise MergeConflictError(conflict)
+
+        self.merge_directories(dest_root, ignore)
+        existing = []
+        merge_file = kwargs.get('merge_file', merge_link)
+        for src, dst in self.get_file_map(dest_root, ignore).items():
+            if os.path.exists(dst):
+                existing.append(dst)
+            else:
+                merge_file(src, dst)
+
+        for c in existing:
+            tty.warn("Could not merge: %s" % c)
+
+    def unmerge(self, dest_root, **kwargs):
+        """Unlink all files in dest that exist in src.
+
+        Unlinks directories in dest if they are empty.
+        """
+        remove_file = kwargs.get('remove_file', remove_link)
+        ignore = kwargs.get('ignore', lambda x: False)
+        for src, dst in self.get_file_map(dest_root, ignore).items():
+            remove_file(src, dst)
+        self.unmerge_directories(dest_root, ignore)
+
+
+def merge_link(src, dest):
+    os.symlink(src, dest)
+
+
+def remove_link(src, dest):
+    if not os.path.islink(dest):
+        raise ValueError("%s is not a link tree!" % dest)
+    # remove if dest is a hardlink/symlink to src; this will only
+    # be false if two packages are merged into a prefix and have a
+    # conflicting file
+    if filecmp.cmp(src, dest, shallow=True):
+        os.remove(dest)
+
+
+class MergeConflictError(Exception):
+
+    def __init__(self, path):
+        super(MergeConflictError, self).__init__(
+            "Package merge blocked by file: %s" % path)

lib/spack/spack/binary_distribution.py

@@ -34,7 +34,7 @@
 import yaml
 
 import llnl.util.tty as tty
-from llnl.util.filesystem import mkdirp, install_tree
+from llnl.util.filesystem import mkdirp, install_tree, get_filetype
 
 import spack.cmd
 import spack.fetch_strategy as fs
@@ -148,7 +148,7 @@ def write_buildinfo_file(prefix, workdir, rel=False):
             #  of files potentially needing relocation
             if relocate.strings_contains_installroot(
                     path_name, spack.store.layout.root):
-                filetype = relocate.get_filetype(path_name)
+                filetype = get_filetype(path_name)
                 if relocate.needs_binary_relocation(filetype, os_id):
                     rel_path_name = os.path.relpath(path_name, prefix)
                     binary_to_relocate.append(rel_path_name)

lib/spack/spack/build_systems/aspell_dict.py

@@ -27,6 +27,7 @@
 from llnl.util.filesystem import filter_file
 from spack.build_systems.autotools import AutotoolsPackage
 from spack.directives import extends
+from spack.package import ExtensionError
 from spack.util.executable import which
 
 
@@ -42,6 +43,17 @@ class AspellDictPackage(AutotoolsPackage):
 
     extends('aspell')
 
+    def view_destination(self, view):
+        aspell_spec = self.spec['aspell']
+        if view.root != aspell_spec.prefix:
+            raise ExtensionError(
+                'aspell does not support non-global extensions')
+        aspell = aspell_spec.command
+        return aspell('dump', 'config', 'dict-dir', output=str).strip()
+
+    def view_source(self):
+        return self.prefix.lib
+
     def patch(self):
         filter_file(r'^dictdir=.*$', 'dictdir=/lib', 'configure')
         filter_file(r'^datadir=.*$', 'datadir=/lib', 'configure')

lib/spack/spack/build_systems/python.py

@@ -25,11 +25,14 @@
 
 import inspect
 import os
+import shutil
 
 from spack.directives import depends_on, extends
 from spack.package import PackageBase, run_after
 
-from llnl.util.filesystem import working_dir
+from llnl.util.filesystem import (working_dir, get_filetype, filter_file,
+                                  path_contains_subdirectory, same_path)
+from llnl.util.lang import match_predicate
 
 
 class PythonPackage(PackageBase):
@@ -116,6 +119,8 @@ def configure(self, spec, prefix):
 
     depends_on('python', type=('build', 'run'))
 
+    py_namespace = None
+
     def setup_file(self):
         """Returns the name of the setup file to use."""
         return 'setup.py'
@@ -403,3 +408,66 @@ def import_module_test(self):
 
     # Check that self.prefix is there after installation
     run_after('install')(PackageBase.sanity_check_prefix)
+
+    def view_file_conflicts(self, view, merge_map):
+        """Report all file conflicts, excepting special cases for python.
+           Specifically, this does not report errors for duplicate
+           __init__.py files for packages in the same namespace.
+        """
+        conflicts = list(dst for src, dst in merge_map.items()
+                         if os.path.exists(dst))
+
+        if conflicts and self.py_namespace:
+            ext_map = view.extensions_layout.extension_map(self.extendee_spec)
+            namespaces = set(
+                x.package.py_namespace for x in ext_map.values())
+            namespace_re = (
+                r'site-packages/{0}/__init__.py'.format(self.py_namespace))
+            find_namespace = match_predicate(namespace_re)
+            if self.py_namespace in namespaces:
+                conflicts = list(
+                    x for x in conflicts if not find_namespace(x))
+
+        return conflicts
+
+    def add_files_to_view(self, view, merge_map):
+        bin_dir = self.spec.prefix.bin
+        python_prefix = self.extendee_spec.prefix
+        global_view = same_path(python_prefix, view.root)
+        for src, dst in merge_map.items():
+            if os.path.exists(dst):
+                continue
+            elif global_view or not path_contains_subdirectory(src, bin_dir):
+                view.link(src, dst)
+            elif not os.path.islink(src):
+                shutil.copy2(src, dst)
+                if 'script' in get_filetype(src):
+                    filter_file(
+                        python_prefix, os.path.abspath(view.root), dst)
+            else:
+                orig_link_target = os.path.realpath(src)
+                new_link_target = os.path.abspath(merge_map[orig_link_target])
+                view.link(new_link_target, dst)
+
+    def remove_files_from_view(self, view, merge_map):
+        ignore_namespace = False
+        if self.py_namespace:
+            ext_map = view.extensions_layout.extension_map(self.extendee_spec)
+            remaining_namespaces = set(
+                spec.package.py_namespace for name, spec in ext_map.items()
+                if name != self.name)
+            if self.py_namespace in remaining_namespaces:
+                namespace_init = match_predicate(
+                    r'site-packages/{0}/__init__.py'.format(self.py_namespace))
+                ignore_namespace = True
+
+        bin_dir = self.spec.prefix.bin
+        global_view = self.extendee_spec.prefix == view.root
+        for src, dst in merge_map.items():
+            if ignore_namespace and namespace_init(dst):
+                continue
+
+            if global_view or not path_contains_subdirectory(src, bin_dir):
+                view.remove_file(src, dst)
+            else:
+                os.remove(dst)

lib/spack/spack/cmd/activate.py

@@ -27,7 +27,7 @@
 import llnl.util.tty as tty
 
 import spack.cmd
-from spack.directory_layout import YamlViewExtensionsLayout
+from spack.filesystem_view import YamlFilesystemView
 
 description = "activate a package extension"
 section = "extensions"
@@ -55,14 +55,17 @@ def activate(parser, args):
     if not spec.package.is_extension:
         tty.die("%s is not an extension." % spec.name)
 
-    layout = spack.store.extensions
+    if args.view:
-    if args.view is not None:
+        target = args.view
-        layout = YamlViewExtensionsLayout(
+    else:
-            args.view, spack.store.layout)
+        target = spec.package.extendee_spec.prefix
 
-    if spec.package.is_activated(extensions_layout=layout):
+    view = YamlFilesystemView(target, spack.store.layout)
+
+    if spec.package.is_activated(view):
         tty.msg("Package %s is already activated." % specs[0].short_spec)
         return
 
-    spec.package.do_activate(extensions_layout=layout,
+    # TODO: refactor FilesystemView.add_extension and use that here (so there
-                             with_dependencies=not args.force)
+    # aren't two ways of activating extensions)
+    spec.package.do_activate(view, with_dependencies=not args.force)

lib/spack/spack/cmd/deactivate.py

@@ -27,7 +27,7 @@
 
 import spack.cmd
 import spack.store
-from spack.directory_layout import YamlViewExtensionsLayout
+from spack.filesystem_view import YamlFilesystemView
 from spack.graph import topological_sort
 
 description = "deactivate a package extension"
@@ -59,25 +59,29 @@ def deactivate(parser, args):
     spec = spack.cmd.disambiguate_spec(specs[0])
     pkg = spec.package
 
-    layout = spack.store.extensions
+    if args.view:
-    if args.view is not None:
+        target = args.view
-        layout = YamlViewExtensionsLayout(
+    elif pkg.is_extension:
-            args.view, spack.store.layout)
+        target = pkg.extendee_spec.prefix
+    elif pkg.extendable:
+        target = spec.prefix
+
+    view = YamlFilesystemView(target, spack.store.layout)
 
     if args.all:
         if pkg.extendable:
             tty.msg("Deactivating all extensions of %s" % pkg.spec.short_spec)
             ext_pkgs = spack.store.db.activated_extensions_for(
-                spec, extensions_layout=layout)
+                spec, view.extensions_layout)
 
             for ext_pkg in ext_pkgs:
                 ext_pkg.spec.normalize()
-                if ext_pkg.is_activated():
+                if ext_pkg.is_activated(view):
-                    ext_pkg.do_deactivate(force=True, extensions_layout=layout)
+                    ext_pkg.do_deactivate(view, force=True)
 
         elif pkg.is_extension:
             if not args.force and \
-               not spec.package.is_activated(extensions_layout=layout):
+               not spec.package.is_activated(view):
                 tty.die("%s is not activated." % pkg.spec.short_spec)
 
             tty.msg("Deactivating %s and all dependencies." %
@@ -90,11 +94,8 @@ def deactivate(parser, args):
                 espec = index[name]
                 epkg = espec.package
                 if epkg.extends(pkg.extendee_spec):
-                    if epkg.is_activated(extensions_layout=layout) or \
+                    if epkg.is_activated(view) or args.force:
-                       args.force:
+                        epkg.do_deactivate(view, force=args.force)
-
-                        epkg.do_deactivate(
-                            force=args.force, extensions_layout=layout)
 
         else:
             tty.die(
@@ -107,7 +108,7 @@ def deactivate(parser, args):
                     "Did you mean 'spack deactivate --all'?")
 
         if not args.force and \
-           not spec.package.is_activated(extensions_layout=layout):
+           not spec.package.is_activated(view):
             tty.die("Package %s is not activated." % specs[0].short_spec)
 
-        spec.package.do_deactivate(force=args.force, extensions_layout=layout)
+        spec.package.do_deactivate(view, force=args.force)

lib/spack/spack/cmd/extensions.py

@@ -31,7 +31,7 @@
 import spack.cmd.find
 import spack.repo
 import spack.store
-from spack.directory_layout import YamlViewExtensionsLayout
+from spack.filesystem_view import YamlFilesystemView
 
 description = "list extensions for package"
 section = "extensions"
@@ -113,10 +113,12 @@ def extensions(parser, args):
             tty.msg("%d extensions:" % len(extensions))
             colify(ext.name for ext in extensions)
 
-    layout = spack.store.extensions
+    if args.view:
-    if args.view is not None:
+        target = args.view
-        layout = YamlViewExtensionsLayout(
+    else:
-            args.view, spack.store.layout)
+        target = spec.prefix
+
+    view = YamlFilesystemView(target, spack.store.layout)
 
     if show_installed:
         #
@@ -137,7 +139,7 @@ def extensions(parser, args):
         #
         # List specs of activated extensions.
         #
-        activated = layout.extension_map(spec)
+        activated = view.extensions_layout.extension_map(spec)
         if show_all:
             print
         if not activated:

lib/spack/spack/database.py

@@ -58,6 +58,7 @@
 import spack.spec
 import spack.util.spack_yaml as syaml
 import spack.util.spack_json as sjson
+from spack.filesystem_view import YamlFilesystemView
 from spack.util.crypto import bit_length
 from spack.directory_layout import DirectoryLayoutError
 from spack.error import SpackError
@@ -823,7 +824,8 @@ def activated_extensions_for(self, extendee_spec, extensions_layout=None):
         the given spec
         """
         if extensions_layout is None:
-            extensions_layout = spack.store.extensions
+            view = YamlFilesystemView(extendee_spec.prefix, spack.store.layout)
+            extensions_layout = view.extensions_layout
         for spec in self.query():
             try:
                 extensions_layout.check_activated(extendee_spec, spec)

lib/spack/spack/directory_layout.py

@@ -128,7 +128,6 @@ class ExtensionsLayout(object):
     """
     def __init__(self, root, **kwargs):
         self.root = root
-        self.link = kwargs.get("link", os.symlink)
 
     def add_extension(self, spec, ext_spec):
         """Add to the list of currently installed extensions."""
@@ -313,14 +312,14 @@ def specs_by_hash(self):
         return by_hash
 
 
-class YamlExtensionsLayout(ExtensionsLayout):
+class YamlViewExtensionsLayout(ExtensionsLayout):
-    """Implements globally activated extensions within a YamlDirectoryLayout.
+    """Maintain extensions within a view.
     """
     def __init__(self, root, layout):
         """layout is the corresponding YamlDirectoryLayout object for which
            we implement extensions.
         """
-        super(YamlExtensionsLayout, self).__init__(root)
+        super(YamlViewExtensionsLayout, self).__init__(root)
         self.layout = layout
         self.extension_file_name = 'extensions.yaml'
 
@@ -354,19 +353,29 @@ def check_activated(self, spec, ext_spec):
             raise NoSuchExtensionError(spec, ext_spec)
 
     def extension_file_path(self, spec):
-        """Gets full path to an installed package's extension file"""
+        """Gets full path to an installed package's extension file, which
+           keeps track of all the extensions for that package which have been
+           added to this view.
+        """
         _check_concrete(spec)
-        return os.path.join(self.layout.metadata_path(spec),
+        normalize_path = lambda p: (
-                            self.extension_file_name)
+            os.path.abspath(p).rstrip(os.path.sep))
+        if normalize_path(spec.prefix) == normalize_path(self.root):
+            # For backwards compatibility, when the root is the extended
+            # package's installation directory, do not include the spec name
+            # as a subdirectory.
+            components = [self.root, self.layout.metadata_dir,
+                          self.extension_file_name]
+        else:
+            components = [self.root, self.layout.metadata_dir, spec.name,
+                          self.extension_file_name]
+        return os.path.join(*components)
 
     def extension_map(self, spec):
         """Defensive copying version of _extension_map() for external API."""
         _check_concrete(spec)
         return self._extension_map(spec).copy()
 
-    def extendee_target_directory(self, extendee):
-        return extendee.prefix
-
     def remove_extension(self, spec, ext_spec):
         _check_concrete(spec)
         _check_concrete(ext_spec)
@@ -419,6 +428,8 @@ def _write_extensions(self, spec, extensions):
 
         # Create a temp file in the same directory as the actual file.
         dirname, basename = os.path.split(path)
+        mkdirp(dirname)
+
         tmp = tempfile.NamedTemporaryFile(
             prefix=basename, dir=dirname, delete=False)
 
@@ -436,23 +447,6 @@ def _write_extensions(self, spec, extensions):
         os.rename(tmp.name, path)
 
 
-class YamlViewExtensionsLayout(YamlExtensionsLayout):
-    """Governs the directory layout present when creating filesystem views in a
-    certain root folder.
-
-    Meant to replace YamlDirectoryLayout when working with filesystem views.
-    """
-
-    def extension_file_path(self, spec):
-        """Gets the full path to an installed package's extension file."""
-        _check_concrete(spec)
-        return os.path.join(self.root, self.layout.metadata_dir, spec.name,
-                            self.extension_file_name)
-
-    def extendee_target_directory(self, extendee):
-        return self.root
-
-
 class DirectoryLayoutError(SpackError):
     """Superclass for directory layout errors."""
 

lib/spack/spack/filesystem_view.py

@@ -22,14 +22,16 @@
 # License along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
+import filecmp
 import functools as ft
 import os
 import re
 import shutil
 import sys
 
-from llnl.util.link_tree import LinkTree
+from llnl.util.link_tree import LinkTree, MergeConflictError
 from llnl.util import tty
+from llnl.util.lang import match_predicate
 
 import spack.spec
 import spack.store
@@ -223,20 +225,9 @@ def add_extension(self, spec):
                      % colorize_spec(spec))
             return True
 
-        try:
+        if not spec.package.is_activated(self):
-            if not spec.package.is_activated(self.extensions_layout):
+            spec.package.do_activate(
-                spec.package.do_activate(
+                self, verbose=self.verbose, with_dependencies=False)
-                    ignore_conflicts=self.ignore_conflicts,
-                    with_dependencies=False,  # already taken care of
-                                              # in add_specs()
-                    verbose=self.verbose,
-                    extensions_layout=self.extensions_layout)
-
-        except ExtensionAlreadyInstalledError:
-            # As we use sets in add_specs(), the order in which packages get
-            # activated is essentially random. So this spec might have already
-            # been activated as dependency of another package -> fail silently
-            pass
 
         # make sure the meta folder is linked as well (this is not done by the
         # extension-activation mechnism)
@@ -274,29 +265,66 @@ def add_standalone(self, spec):
                                         long=False)
                 return False
 
-        tree = LinkTree(spec.prefix)
+        self.merge(spec)
 
-        if not self.ignore_conflicts:
-            conflict = tree.find_conflict(self.root)
-            if conflict is not None:
-                tty.error(self._croot +
-                          "Cannot link package %s, file already exists: %s"
-                          % (spec.name, conflict))
-                return False
-
-        conflicts = tree.merge(self.root, link=self.link,
-                               ignore=ignore_metadata_dir,
-                               ignore_conflicts=self.ignore_conflicts)
         self.link_meta_folder(spec)
 
-        if self.ignore_conflicts:
-            for c in conflicts:
-                tty.warn(self._croot + "Could not link: %s" % c)
-
         if self.verbose:
             tty.info(self._croot + 'Linked package: %s' % colorize_spec(spec))
         return True
 
+    def merge(self, spec, ignore=None):
+        pkg = spec.package
+        view_source = pkg.view_source()
+        view_dst = pkg.view_destination(self)
+
+        tree = LinkTree(view_source)
+
+        ignore = ignore or (lambda f: False)
+        ignore_file = match_predicate(
+            self.layout.hidden_file_paths, ignore)
+
+        # check for dir conflicts
+        conflicts = tree.find_dir_conflicts(view_dst, ignore_file)
+
+        merge_map = tree.get_file_map(view_dst, ignore_file)
+        if not self.ignore_conflicts:
+            conflicts.extend(pkg.view_file_conflicts(self, merge_map))
+
+        if conflicts:
+            raise MergeConflictError(conflicts[0])
+
+        # merge directories with the tree
+        tree.merge_directories(view_dst, ignore_file)
+
+        pkg.add_files_to_view(self, merge_map)
+
+    def unmerge(self, spec, ignore=None):
+        pkg = spec.package
+        view_source = pkg.view_source()
+        view_dst = pkg.view_destination(self)
+
+        tree = LinkTree(view_source)
+
+        ignore = ignore or (lambda f: False)
+        ignore_file = match_predicate(
+            self.layout.hidden_file_paths, ignore)
+
+        merge_map = tree.get_file_map(view_dst, ignore_file)
+        pkg.remove_files_from_view(self, merge_map)
+
+        # now unmerge the directory tree
+        tree.unmerge_directories(view_dst, ignore_file)
+
+    def remove_file(self, src, dest):
+        if not os.path.islink(dest):
+            raise ValueError("%s is not a link tree!" % dest)
+        # remove if dest is a hardlink/symlink to src; this will only
+        # be false if two packages are merged into a prefix and have a
+        # conflicting file
+        if filecmp.cmp(src, dest, shallow=True):
+            os.remove(dest)
+
     def check_added(self, spec):
         assert spec.concrete
         return spec == self.get_spec(spec)
@@ -364,11 +392,11 @@ def remove_extension(self, spec, with_dependents=True):
 
         # The spec might have been deactivated as depdency of another package
         # already
-        if spec.package.is_activated(self.extensions_layout):
+        if spec.package.is_activated(self):
             spec.package.do_deactivate(
+                self,
                 verbose=self.verbose,
-                remove_dependents=with_dependents,
+                remove_dependents=with_dependents)
-                extensions_layout=self.extensions_layout)
         self.unlink_meta_folder(spec)
 
     def remove_standalone(self, spec):
@@ -380,8 +408,7 @@ def remove_standalone(self, spec):
                      'Skipping package not linked in view: %s' % spec.name)
             return
 
-        tree = LinkTree(spec.prefix)
+        self.unmerge(spec)
-        tree.unmerge(self.root, ignore=ignore_metadata_dir)
         self.unlink_meta_folder(spec)
 
         if self.verbose:
@@ -545,7 +572,3 @@ def get_dependencies(specs):
     retval = set()
     set(map(retval.update, (set(s.traverse()) for s in specs)))
     return retval
-
-
-def ignore_metadata_dir(f):
-    return f in spack.store.layout.hidden_file_paths

lib/spack/spack/hooks/extensions.py

@@ -22,6 +22,8 @@
 # License along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
+import spack
+from spack.filesystem_view import YamlFilesystemView
 
 
 def pre_uninstall(spec):
@@ -29,6 +31,9 @@ def pre_uninstall(spec):
     assert spec.concrete
 
     if pkg.is_extension:
-        if pkg.is_activated():
+        target = pkg.extendee_spec.prefix
+        view = YamlFilesystemView(target, spack.store.layout)
+
+        if pkg.is_activated(view):
             # deactivate globally
             pkg.do_deactivate(force=True)

lib/spack/spack/package.py

@@ -75,6 +75,7 @@
 from llnl.util.link_tree import LinkTree
 from llnl.util.tty.log import log_output
 from llnl.util.tty.color import colorize
+from spack.filesystem_view import YamlFilesystemView
 from spack.util.executable import which
 from spack.stage import Stage, ResourceStage, StageComposite
 from spack.util.environment import dump_environment
@@ -260,7 +261,54 @@ def _wrapper(instance, *args, **kwargs):
     return _execute_under_condition
 
 
-class PackageBase(with_metaclass(PackageMeta, object)):
+class PackageViewMixin(object):
+    """This collects all functionality related to adding installed Spack
+    package to views. Packages can customize how they are added to views by
+    overriding these functions.
+    """
+    def view_source(self):
+        """The source root directory that will be added to the view: files are
+        added such that their path relative to the view destination matches
+        their path relative to the view source.
+        """
+        return self.spec.prefix
+
+    def view_destination(self, view):
+        """The target root directory: each file is added relative to this
+        directory.
+        """
+        return view.root
+
+    def view_file_conflicts(self, view, merge_map):
+        """Report any files which prevent adding this package to the view. The
+        default implementation looks for any files which already exist.
+        Alternative implementations may allow some of the files to exist in
+        the view (in this case they would be omitted from the results).
+        """
+        return set(dst for dst in merge_map.values() if os.path.exists(dst))
+
+    def add_files_to_view(self, view, merge_map):
+        """Given a map of package files to destination paths in the view, add
+        the files to the view. By default this adds all files. Alternative
+        implementations may skip some files, for example if other packages
+        linked into the view already include the file.
+        """
+        for src, dst in merge_map.items():
+            if not os.path.exists(dst):
+                view.link(src, dst)
+
+    def remove_files_from_view(self, view, merge_map):
+        """Given a map of package files to files currently linked in the view,
+        remove the files from the view. The default implementation removes all
+        files. Alternative implementations may not remove all files. For
+        example if two packages include the same file, it should only be
+        removed when both packages are removed.
+        """
+        for src, dst in merge_map.items():
+            view.remove_file(src, dst)
+
+
+class PackageBase(with_metaclass(PackageMeta, PackageViewMixin, object)):
     """This is the superclass for all spack packages.
 
     ***The Package class***
@@ -942,13 +990,12 @@ def extends(self, spec):
         s = self.extendee_spec
         return s and spec.satisfies(s)
 
-    def is_activated(self, extensions_layout=None):
+    def is_activated(self, view):
         """Return True if package is activated."""
         if not self.is_extension:
             raise ValueError(
-                "is_extension called on package that is not an extension.")
+                "is_activated called on package that is not an extension.")
-        if extensions_layout is None:
+        extensions_layout = view.extensions_layout
-            extensions_layout = spack.store.extensions
         exts = extensions_layout.extension_map(self.extendee_spec)
         return (self.name in exts) and (exts[self.name] == self.spec)
 
@@ -1979,8 +2026,7 @@ def _sanity_check_extension(self):
             raise ActivationError("%s does not extend %s!" %
                                   (self.name, self.extendee.name))
 
-    def do_activate(self, with_dependencies=True, ignore_conflicts=False,
+    def do_activate(self, view=None, with_dependencies=True, verbose=True):
-                    verbose=True, extensions_layout=None):
         """Called on an extension to invoke the extendee's activate method.
 
         Commands should call this routine, and should not call
@@ -1991,9 +2037,11 @@ def do_activate(self, with_dependencies=True, ignore_conflicts=False,
                     (self.spec.cshort_spec, self.extendee_spec.cshort_spec))
 
         self._sanity_check_extension()
+        if not view:
+            view = YamlFilesystemView(
+                self.extendee_spec.prefix, spack.store.layout)
 
-        if extensions_layout is None:
+        extensions_layout = view.extensions_layout
-            extensions_layout = spack.store.extensions
 
         extensions_layout.check_extension_conflict(
             self.extendee_spec, self.spec)
@@ -2001,17 +2049,13 @@ def do_activate(self, with_dependencies=True, ignore_conflicts=False,
         # Activate any package dependencies that are also extensions.
         if with_dependencies:
             for spec in self.dependency_activations():
-                if not spec.package.is_activated(
+                if not spec.package.is_activated(view):
-                        extensions_layout=extensions_layout):
                     spec.package.do_activate(
-                        with_dependencies=with_dependencies,
+                        view, with_dependencies=with_dependencies,
-                        ignore_conflicts=ignore_conflicts,
+                        verbose=verbose)
-                        verbose=verbose,
-                        extensions_layout=extensions_layout)
 
         self.extendee_spec.package.activate(
-            self, extensions_layout=extensions_layout,
+            self, view, **self.extendee_args)
-            ignore_conflicts=ignore_conflicts, **self.extendee_args)
 
         extensions_layout.add_extension(self.extendee_spec, self.spec)
 
@@ -2025,41 +2069,22 @@ def dependency_activations(self):
         return (spec for spec in self.spec.traverse(root=False, deptype='run')
                 if spec.package.extends(self.extendee_spec))
 
-    def activate(self, extension, ignore_conflicts=False, **kwargs):
+    def activate(self, extension, view, **kwargs):
-        """Make extension package usable by linking all its files to a target
-        provided by the directory layout (depending if the user wants to
-        activate globally or in a specified file system view).
-
-        Package authors can override this method to support other
-        extension mechanisms.  Spack internals (commands, hooks, etc.)
-        should call do_activate() method so that proper checks are
-        always executed.
-
         """
-        extensions_layout = kwargs.get("extensions_layout",
+        Add the extension to the specified view.
-                                       spack.store.extensions)
-        target = extensions_layout.extendee_target_directory(self)
 
-        def ignore(filename):
+        Package authors can override this function to maintain some
-            return (filename in spack.store.layout.hidden_file_paths or
+        centralized state related to the set of activated extensions
-                    kwargs.get('ignore', lambda f: False)(filename))
+        for a package.
 
-        tree = LinkTree(extension.prefix)
+        Spack internals (commands, hooks, etc.) should call
+        do_activate() method so that proper checks are always executed.
+        """
+        view.merge(extension.spec, ignore=kwargs.get('ignore', None))
 
-        conflict = tree.find_conflict(target, ignore=ignore)
+    def do_deactivate(self, view=None, **kwargs):
-        if not conflict:
+        """Remove this extension package from the specified view. Called
-            pass
+        on the extension to invoke extendee's deactivate() method.
-        elif ignore_conflicts:
-            tty.warn("While activating %s, found conflict %s" %
-                     (self.spec.cshort_spec, conflict))
-        else:
-            raise ExtensionConflictError(conflict)
-
-        tree.merge(target, ignore=ignore, link=extensions_layout.link,
-                   ignore_conflicts=ignore_conflicts)
-
-    def do_deactivate(self, **kwargs):
-        """Called on the extension to invoke extendee's deactivate() method.
 
         `remove_dependents=True` deactivates extensions depending on this
         package instead of raising an error.
@@ -2068,8 +2093,11 @@ def do_deactivate(self, **kwargs):
         force = kwargs.get('force', False)
         verbose = kwargs.get("verbose", True)
         remove_dependents = kwargs.get("remove_dependents", False)
-        extensions_layout = kwargs.get("extensions_layout",
+
-                                       spack.store.extensions)
+        if not view:
+            view = YamlFilesystemView(
+                self.extendee_spec.prefix, spack.store.layout)
+        extensions_layout = view.extensions_layout
 
         # Allow a force deactivate to happen.  This can unlink
         # spurious files if something was corrupted.
@@ -2094,13 +2122,11 @@ def do_deactivate(self, **kwargs):
                                        aspec.cshort_spec))
 
         self.extendee_spec.package.deactivate(
-            self,
+            self, view, **self.extendee_args)
-            extensions_layout=extensions_layout,
-            **self.extendee_args)
 
         # redundant activation check -- makes SURE the spec is not
         # still activated even if something was wrong above.
-        if self.is_activated(extensions_layout):
+        if self.is_activated(view):
             extensions_layout.remove_extension(
                 self.extendee_spec, self.spec)
 
@@ -2110,26 +2136,23 @@ def do_deactivate(self, **kwargs):
                 (self.spec.short_spec,
                  self.extendee_spec.cformat("$_$@$+$%@")))
 
-    def deactivate(self, extension, **kwargs):
+    def deactivate(self, extension, view, **kwargs):
-        """Unlinks all files from extension out of this package's install dir
+        """
-        or the corresponding filesystem view.
+        Remove all extension files from the specified view.
 
         Package authors can override this method to support other
         extension mechanisms.  Spack internals (commands, hooks, etc.)
         should call do_deactivate() method so that proper checks are
         always executed.
-
         """
-        extensions_layout = kwargs.get("extensions_layout",
+        view.unmerge(extension.spec, ignore=kwargs.get('ignore', None))
-                                       spack.store.extensions)
-        target = extensions_layout.extendee_target_directory(self)
 
-        def ignore(filename):
+    def view(self):
-            return (filename in spack.store.layout.hidden_file_paths or
+        """Create a view with the prefix of this package as the root.
-                    kwargs.get('ignore', lambda f: False)(filename))
+        Extensions added to this view will modify the installation prefix of
-
+        this package.
-        tree = LinkTree(extension.prefix)
+        """
-        tree.unmerge(target, ignore=ignore)
+        return YamlFilesystemView(self.prefix, spack.store.layout)
 
     def do_restage(self):
         """Reverts expanded/checked out source to a pristine state."""
@@ -2409,13 +2432,6 @@ class ExtensionError(PackageError):
     pass
 
 
-class ExtensionConflictError(ExtensionError):
-
-    def __init__(self, path):
-        super(ExtensionConflictError, self).__init__(
-            "Extension blocked by file: %s" % path)
-
-
 class ActivationError(ExtensionError):
 
     def __init__(self, msg, long_msg=None):

lib/spack/spack/relocate.py

@@ -256,17 +256,6 @@ def modify_macho_object(cur_path, rpaths, deps, idpath,
     return
 
 
-def get_filetype(path_name):
-    """
-    Return the output of file path_name as a string to identify file type.
-    """
-    file = Executable('file')
-    file.add_default_env('LC_ALL', 'C')
-    output = file('-b', '-h', '%s' % path_name,
-                  output=str, error=str)
-    return output.strip()
-
-
 def strings_contains_installroot(path_name, root_dir):
     """
     Check if the file contain the install root string.

lib/spack/spack/store.py

@@ -80,8 +80,6 @@ def __init__(self, root, path_scheme=None, hash_length=None):
         self.db = spack.database.Database(root)
         self.layout = spack.directory_layout.YamlDirectoryLayout(
             root, hash_len=hash_length, path_scheme=path_scheme)
-        self.extensions = spack.directory_layout.YamlExtensionsLayout(
-            root, self.layout)
 
     def reindex(self):
         """Convenience function to reindex the store DB with its own layout."""
@@ -105,4 +103,3 @@ def _store():
 root = llnl.util.lang.LazyReference(lambda: store.root)
 db = llnl.util.lang.LazyReference(lambda: store.db)
 layout = llnl.util.lang.LazyReference(lambda: store.layout)
-extensions = llnl.util.lang.LazyReference(lambda: store.extensions)

lib/spack/spack/test/cmd/activate.py

@@ -0,0 +1,61 @@
+##############################################################################
+# Copyright (c) 2013-2018, Lawrence Livermore National Security, LLC.
+# Produced at the Lawrence Livermore National Laboratory.
+#
+# This file is part of Spack.
+# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
+# LLNL-CODE-647188
+#
+# For details, see https://github.com/spack/spack
+# Please also see the NOTICE and LICENSE files for our notice and the LGPL.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License (as
+# published by the Free Software Foundation) version 2.1, February 1999.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
+# conditions of the GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+##############################################################################
+from spack.main import SpackCommand
+
+activate = SpackCommand('activate')
+deactivate = SpackCommand('deactivate')
+install = SpackCommand('install')
+extensions = SpackCommand('extensions')
+
+
+def test_activate(
+        mock_packages, mock_archive, mock_fetch, config,
+        install_mockery):
+    install('extension1')
+    activate('extension1')
+    output = extensions('--show', 'activated', 'extendee')
+    assert 'extension1' in output
+
+
+def test_deactivate(
+        mock_packages, mock_archive, mock_fetch, config,
+        install_mockery):
+    install('extension1')
+    activate('extension1')
+    deactivate('extension1')
+    output = extensions('--show', 'activated', 'extendee')
+    assert 'extension1' not in output
+
+
+def test_deactivate_all(
+        mock_packages, mock_archive, mock_fetch, config,
+        install_mockery):
+    install('extension1')
+    install('extension2')
+    activate('extension1')
+    activate('extension2')
+    deactivate('--all', 'extendee')
+    output = extensions('--show', 'activated', 'extendee')
+    assert 'extension1' not in output

lib/spack/spack/test/test_activations.py

@@ -24,16 +24,25 @@
 ##############################################################################
 import os
 import pytest
+import sys
 
 import spack.spec
+import spack.package
+from llnl.util.link_tree import MergeConflictError
+from spack.build_systems.python import PythonPackage
 from spack.directory_layout import YamlDirectoryLayout
 from spack.filesystem_view import YamlFilesystemView
+from spack.util.prefix import Prefix
+
+"""This includes tests for customized activation logic for specific packages
+   (e.g. python and perl).
+"""
 
 
-class FakeExtensionPackage(object):
+class FakeExtensionPackage(spack.package.PackageViewMixin):
     def __init__(self, name, prefix):
         self.name = name
-        self.prefix = prefix
+        self.prefix = Prefix(prefix)
         self.spec = FakeSpec(self)
 
 
@@ -42,10 +51,43 @@ def __init__(self, package):
         self.name = package.name
         self.prefix = package.prefix
         self.hash = self.name
+        self.package = package
+        self.concrete = True
 
     def dag_hash(self):
         return self.hash
 
+    def __lt__(self, other):
+        return self.name < other.name
+
+
+class FakePythonExtensionPackage(FakeExtensionPackage):
+    def __init__(self, name, prefix, py_namespace, python_spec):
+        self.py_namespace = py_namespace
+        self.extendee_spec = python_spec
+        super(FakePythonExtensionPackage, self).__init__(name, prefix)
+
+    def add_files_to_view(self, view, merge_map):
+        if sys.version_info >= (3, 0):
+            add_fn = PythonPackage.add_files_to_view
+        else:
+            add_fn = PythonPackage.add_files_to_view.im_func
+        return add_fn(self, view, merge_map)
+
+    def view_file_conflicts(self, view, merge_map):
+        if sys.version_info >= (3, 0):
+            conflicts_fn = PythonPackage.view_file_conflicts
+        else:
+            conflicts_fn = PythonPackage.view_file_conflicts.im_func
+        return conflicts_fn(self, view, merge_map)
+
+    def remove_files_from_view(self, view, merge_map):
+        if sys.version_info >= (3, 0):
+            remove_fn = PythonPackage.remove_files_from_view
+        else:
+            remove_fn = PythonPackage.remove_files_from_view.im_func
+        return remove_fn(self, view, merge_map)
+
 
 def create_dir_structure(tmpdir, dir_structure):
     for fname, children in dir_structure.items():
@@ -102,25 +144,49 @@ def python_and_extension_dirs(tmpdir):
     return str(python_prefix), str(ext_prefix)
 
 
-def test_python_activation(tmpdir):
+@pytest.fixture()
-    # Note the lib directory is based partly on the python version
+def namespace_extensions(tmpdir):
-    python_spec = spack.spec.Spec('python@2.7.12')
+    ext1_dirs = {
-    python_spec._concrete = True
+        'bin/': {
+            'py-ext-tool1': None
+        },
+        'lib/': {
+            'python2.7/': {
+                'site-packages/': {
+                    'examplenamespace/': {
+                        '__init__.py': None,
+                        'ext1_sample.py': None
+                    }
+                }
+            }
+        }
+    }
 
-    python_name = 'python'
+    ext2_dirs = {
-    tmpdir.ensure(python_name, dir=True)
+        'bin/': {
+            'py-ext-tool2': None
+        },
+        'lib/': {
+            'python2.7/': {
+                'site-packages/': {
+                    'examplenamespace/': {
+                        '__init__.py': None,
+                        'ext2_sample.py': None
+                    }
+                }
+            }
+        }
+    }
 
-    python_prefix = str(tmpdir.join(python_name))
+    ext1_name = 'py-extension1'
-    # Set the prefix on the package's spec reference because that is a copy of
+    ext1_prefix = tmpdir.join(ext1_name)
-    # the original spec
+    create_dir_structure(ext1_prefix, ext1_dirs)
-    python_spec.package.spec.prefix = python_prefix
 
-    ext_name = 'py-extension'
+    ext2_name = 'py-extension2'
-    tmpdir.ensure(ext_name, dir=True)
+    ext2_prefix = tmpdir.join(ext2_name)
-    ext_pkg = FakeExtensionPackage(ext_name, str(tmpdir.join(ext_name)))
+    create_dir_structure(ext2_prefix, ext2_dirs)
 
-    python_pkg = python_spec.package
+    return str(ext1_prefix), str(ext2_prefix), 'examplenamespace'
-    python_pkg.activate(ext_pkg)
 
 
 def test_python_activation_with_files(tmpdir, python_and_extension_dirs):
@@ -133,7 +199,7 @@ def test_python_activation_with_files(tmpdir, python_and_extension_dirs):
     ext_pkg = FakeExtensionPackage('py-extension', ext_prefix)
 
     python_pkg = python_spec.package
-    python_pkg.activate(ext_pkg)
+    python_pkg.activate(ext_pkg, python_pkg.view())
 
     assert os.path.exists(os.path.join(python_prefix, 'bin/py-ext-tool'))
 
@@ -159,13 +225,114 @@ def test_python_activation_view(tmpdir, python_and_extension_dirs):
     view = YamlFilesystemView(view_dir, layout)
 
     python_pkg = python_spec.package
-    python_pkg.activate(ext_pkg, extensions_layout=view.extensions_layout)
+    python_pkg.activate(ext_pkg, view)
 
     assert not os.path.exists(os.path.join(python_prefix, 'bin/py-ext-tool'))
 
     assert os.path.exists(os.path.join(view_dir, 'bin/py-ext-tool'))
 
 
+def test_python_ignore_namespace_init_conflict(tmpdir, namespace_extensions):
+    """Test the view update logic in PythonPackage ignores conflicting
+       instances of __init__ for packages which are in the same namespace.
+    """
+    ext1_prefix, ext2_prefix, py_namespace = namespace_extensions
+
+    python_spec = spack.spec.Spec('python@2.7.12')
+    python_spec._concrete = True
+
+    ext1_pkg = FakePythonExtensionPackage(
+        'py-extension1', ext1_prefix, py_namespace, python_spec)
+    ext2_pkg = FakePythonExtensionPackage(
+        'py-extension2', ext2_prefix, py_namespace, python_spec)
+
+    view_dir = str(tmpdir.join('view'))
+    layout = YamlDirectoryLayout(view_dir)
+    view = YamlFilesystemView(view_dir, layout)
+
+    python_pkg = python_spec.package
+    python_pkg.activate(ext1_pkg, view)
+    # Normally handled by Package.do_activate, but here we activate directly
+    view.extensions_layout.add_extension(python_spec, ext1_pkg.spec)
+    python_pkg.activate(ext2_pkg, view)
+
+    f1 = 'lib/python2.7/site-packages/examplenamespace/ext1_sample.py'
+    f2 = 'lib/python2.7/site-packages/examplenamespace/ext2_sample.py'
+    init_file = 'lib/python2.7/site-packages/examplenamespace/__init__.py'
+
+    assert os.path.exists(os.path.join(view_dir, f1))
+    assert os.path.exists(os.path.join(view_dir, f2))
+    assert os.path.exists(os.path.join(view_dir, init_file))
+
+
+def test_python_keep_namespace_init(tmpdir, namespace_extensions):
+    """Test the view update logic in PythonPackage keeps the namespace
+       __init__ file as long as one package in the namespace still
+       exists.
+    """
+    ext1_prefix, ext2_prefix, py_namespace = namespace_extensions
+
+    python_spec = spack.spec.Spec('python@2.7.12')
+    python_spec._concrete = True
+
+    ext1_pkg = FakePythonExtensionPackage(
+        'py-extension1', ext1_prefix, py_namespace, python_spec)
+    ext2_pkg = FakePythonExtensionPackage(
+        'py-extension2', ext2_prefix, py_namespace, python_spec)
+
+    view_dir = str(tmpdir.join('view'))
+    layout = YamlDirectoryLayout(view_dir)
+    view = YamlFilesystemView(view_dir, layout)
+
+    python_pkg = python_spec.package
+    python_pkg.activate(ext1_pkg, view)
+    # Normally handled by Package.do_activate, but here we activate directly
+    view.extensions_layout.add_extension(python_spec, ext1_pkg.spec)
+    python_pkg.activate(ext2_pkg, view)
+    view.extensions_layout.add_extension(python_spec, ext2_pkg.spec)
+
+    f1 = 'lib/python2.7/site-packages/examplenamespace/ext1_sample.py'
+    init_file = 'lib/python2.7/site-packages/examplenamespace/__init__.py'
+
+    python_pkg.deactivate(ext1_pkg, view)
+    view.extensions_layout.remove_extension(python_spec, ext1_pkg.spec)
+
+    assert not os.path.exists(os.path.join(view_dir, f1))
+    assert os.path.exists(os.path.join(view_dir, init_file))
+
+    python_pkg.deactivate(ext2_pkg, view)
+    view.extensions_layout.remove_extension(python_spec, ext2_pkg.spec)
+
+    assert not os.path.exists(os.path.join(view_dir, init_file))
+
+
+def test_python_namespace_conflict(tmpdir, namespace_extensions):
+    """Test the view update logic in PythonPackage reports an error when two
+       python extensions with different namespaces have a conflicting __init__
+       file.
+    """
+    ext1_prefix, ext2_prefix, py_namespace = namespace_extensions
+    other_namespace = py_namespace + 'other'
+
+    python_spec = spack.spec.Spec('python@2.7.12')
+    python_spec._concrete = True
+
+    ext1_pkg = FakePythonExtensionPackage(
+        'py-extension1', ext1_prefix, py_namespace, python_spec)
+    ext2_pkg = FakePythonExtensionPackage(
+        'py-extension2', ext2_prefix, other_namespace, python_spec)
+
+    view_dir = str(tmpdir.join('view'))
+    layout = YamlDirectoryLayout(view_dir)
+    view = YamlFilesystemView(view_dir, layout)
+
+    python_pkg = python_spec.package
+    python_pkg.activate(ext1_pkg, view)
+    view.extensions_layout.add_extension(python_spec, ext1_pkg.spec)
+    with pytest.raises(MergeConflictError):
+        python_pkg.activate(ext2_pkg, view)
+
+
 @pytest.fixture()
 def perl_and_extension_dirs(tmpdir):
     perl_dirs = {
@@ -230,7 +397,7 @@ def test_perl_activation(tmpdir):
     ext_pkg = FakeExtensionPackage(ext_name, str(tmpdir.join(ext_name)))
 
     perl_pkg = perl_spec.package
-    perl_pkg.activate(ext_pkg)
+    perl_pkg.activate(ext_pkg, perl_pkg.view())
 
 
 def test_perl_activation_with_files(tmpdir, perl_and_extension_dirs):
@@ -243,7 +410,7 @@ def test_perl_activation_with_files(tmpdir, perl_and_extension_dirs):
     ext_pkg = FakeExtensionPackage('perl-extension', ext_prefix)
 
     perl_pkg = perl_spec.package
-    perl_pkg.activate(ext_pkg)
+    perl_pkg.activate(ext_pkg, perl_pkg.view())
 
     assert os.path.exists(os.path.join(perl_prefix, 'bin/perl-ext-tool'))
 
@@ -262,7 +429,7 @@ def test_perl_activation_view(tmpdir, perl_and_extension_dirs):
     view = YamlFilesystemView(view_dir, layout)
 
     perl_pkg = perl_spec.package
-    perl_pkg.activate(ext_pkg, extensions_layout=view.extensions_layout)
+    perl_pkg.activate(ext_pkg, view)
 
     assert not os.path.exists(os.path.join(perl_prefix, 'bin/perl-ext-tool'))
 

lib/spack/spack/test/views.py

@@ -0,0 +1,48 @@
+##############################################################################
+# Copyright (c) 2013-2018, Lawrence Livermore National Security, LLC.
+# Produced at the Lawrence Livermore National Laboratory.
+#
+# This file is part of Spack.
+# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
+# LLNL-CODE-647188
+#
+# For details, see https://github.com/spack/spack
+# Please also see the NOTICE and LICENSE files for our notice and the LGPL.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License (as
+# published by the Free Software Foundation) version 2.1, February 1999.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
+# conditions of the GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+##############################################################################
+import os
+
+from spack.spec import Spec
+
+
+def test_global_activation(install_mockery, mock_fetch):
+    """This test ensures that views which are maintained inside of an extendee
+       package's prefix are maintained as expected and are compatible with
+       global activations prior to #7152.
+    """
+    spec = Spec('extension1').concretized()
+    pkg = spec.package
+    pkg.do_install()
+    pkg.do_activate()
+
+    extendee_spec = spec['extendee']
+    extendee_pkg = spec['extendee'].package
+    view = extendee_pkg.view()
+    assert pkg.is_activated(view)
+
+    expected_path = os.path.join(
+        extendee_spec.prefix, '.spack', 'extensions.yaml')
+    assert (view.extensions_layout.extension_file_path(extendee_spec) ==
+            expected_path)

var/spack/repos/builtin/packages/aspell/package.py

@@ -23,9 +23,6 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
 from spack import *
-from llnl.util.link_tree import LinkTree
-import spack.store
-from spack.package import ExtensionError, ExtensionConflictError
 
 
 # See also: AspellDictPackage
@@ -41,47 +38,3 @@ class Aspell(AutotoolsPackage):
     version('0.60.6.1', 'e66a9c9af6a60dc46134fdacf6ce97d7')
 
     patch('darwin.patch', when='platform=darwin')
-
-    # The dictionaries install all their bits into their prefix.lib dir,
-    # we want to link them into aspell's dict-dir.
-    # These are identical to what's in spack/package.py except
-    # for using:
-    #   - extension.prefix.lib instead of extension.prefix in LinkTree()
-    #   - dest_dir instead of self.prefix in tree.(find_conflict|merge)()
-    def activate(self, extension, **kwargs):
-        extensions_layout = kwargs.get("extensions_layout",
-                                       spack.store.extensions)
-        if extensions_layout is not spack.store.extensions:
-            raise ExtensionError(
-                'aspell does not support non-global extensions')
-
-        aspell = which(self.prefix.bin.aspell)
-        dest_dir = aspell('dump', 'config', 'dict-dir', output=str).strip()
-        tree = LinkTree(extension.prefix.lib)
-
-        def ignore(filename):
-            return (filename in spack.store.layout.hidden_file_paths or
-                    kwargs.get('ignore', lambda f: False)(filename))
-
-        conflict = tree.find_conflict(dest_dir, ignore=ignore)
-        if conflict:
-            raise ExtensionConflictError(conflict)
-
-        tree.merge(dest_dir, ignore=ignore)
-
-    def deactivate(self, extension, **kwargs):
-        extensions_layout = kwargs.get("extensions_layout",
-                                       spack.store.extensions)
-        if extensions_layout is not spack.store.extensions:
-            raise ExtensionError(
-                'aspell does not support non-global extensions')
-
-        aspell = which(self.prefix.bin.aspell)
-        dest_dir = aspell('dump', 'config', 'dict-dir', output=str).strip()
-
-        def ignore(filename):
-            return (filename in spack.store.layout.hidden_file_paths or
-                    kwargs.get('ignore', lambda f: False)(filename))
-
-        tree = LinkTree(extension.prefix.lib)
-        tree.unmerge(dest_dir, ignore=ignore)

var/spack/repos/builtin/packages/perl/package.py

@@ -35,7 +35,6 @@
 
 from llnl.util.lang import match_predicate
 
-import spack.store
 from spack import *
 
 
@@ -267,27 +266,23 @@ def perl_ignore(self, ext_pkg, args):
 
         return match_predicate(ignore_arg, patterns)
 
-    def activate(self, ext_pkg, **args):
+    def activate(self, ext_pkg, view, **args):
         ignore = self.perl_ignore(ext_pkg, args)
         args.update(ignore=ignore)
 
-        super(Perl, self).activate(ext_pkg, **args)
+        super(Perl, self).activate(ext_pkg, view, **args)
-
-        extensions_layout = args.get("extensions_layout",
-                                     spack.store.extensions)
 
+        extensions_layout = view.extensions_layout
         exts = extensions_layout.extension_map(self.spec)
         exts[ext_pkg.name] = ext_pkg.spec
 
-    def deactivate(self, ext_pkg, **args):
+    def deactivate(self, ext_pkg, view, **args):
         ignore = self.perl_ignore(ext_pkg, args)
         args.update(ignore=ignore)
 
-        super(Perl, self).deactivate(ext_pkg, **args)
+        super(Perl, self).deactivate(ext_pkg, view, **args)
-
-        extensions_layout = args.get("extensions_layout",
-                                     spack.store.extensions)
 
+        extensions_layout = view.extensions_layout
         exts = extensions_layout.extension_map(self.spec)
         # Make deactivate idempotent
         if ext_pkg.name in exts:

var/spack/repos/builtin/packages/py-backports-shutil-get-terminal-size/package.py

@@ -32,6 +32,8 @@ class PyBackportsShutilGetTerminalSize(PythonPackage):
     homepage = "https://pypi.python.org/pypi/backports.shutil_get_terminal_size"
     url      = "https://pypi.io/packages/source/b/backports.shutil_get_terminal_size/backports.shutil_get_terminal_size-1.0.0.tar.gz"
 
+    py_namespace = 'backports'
+
     version('1.0.0', '03267762480bd86b50580dc19dff3c66')
 
     # newer setuptools version mess with "namespace" packages in an

var/spack/repos/builtin/packages/py-backports-ssl-match-hostname/package.py

@@ -31,4 +31,6 @@ class PyBackportsSslMatchHostname(PythonPackage):
     homepage = "https://pypi.python.org/pypi/backports.ssl_match_hostname"
     url      = "https://pypi.io/packages/source/b/backports.ssl_match_hostname/backports.ssl_match_hostname-3.5.0.1.tar.gz"
 
+    py_namespace = 'backports'
+
     version('3.5.0.1', 'c03fc5e2c7b3da46b81acf5cbacfe1e6')

var/spack/repos/builtin/packages/python/package.py

@@ -27,10 +27,12 @@
 import platform
 import re
 import sys
+import shutil
 
 import llnl.util.tty as tty
 from llnl.util.lang import match_predicate
-from llnl.util.filesystem import force_remove
+from llnl.util.filesystem import (force_remove, get_filetype,
+                                  path_contains_subdirectory)
 
 import spack.store
 import spack.util.spack_json as sjson
@@ -676,33 +678,49 @@ def write_easy_install_pth(self, exts, prefix=None):
                         "sys.path[p:p]=new; "
                         "sys.__egginsert = p+len(new)\n")
 
-    def activate(self, ext_pkg, **args):
+    def activate(self, ext_pkg, view, **args):
         ignore = self.python_ignore(ext_pkg, args)
         args.update(ignore=ignore)
 
-        extensions_layout = args.get("extensions_layout",
+        super(Python, self).activate(ext_pkg, view, **args)
-                                     spack.store.extensions)
-
-        super(Python, self).activate(ext_pkg, **args)
 
+        extensions_layout = view.extensions_layout
         exts = extensions_layout.extension_map(self.spec)
         exts[ext_pkg.name] = ext_pkg.spec
 
-        self.write_easy_install_pth(
+        self.write_easy_install_pth(exts, prefix=view.root)
-            exts,
-            prefix=extensions_layout.extendee_target_directory(self))
 
-    def deactivate(self, ext_pkg, **args):
+    def deactivate(self, ext_pkg, view, **args):
         args.update(ignore=self.python_ignore(ext_pkg, args))
-        super(Python, self).deactivate(ext_pkg, **args)
 
-        extensions_layout = args.get("extensions_layout",
+        super(Python, self).deactivate(ext_pkg, view, **args)
-                                     spack.store.extensions)
 
+        extensions_layout = view.extensions_layout
         exts = extensions_layout.extension_map(self.spec)
         # Make deactivate idempotent
         if ext_pkg.name in exts:
             del exts[ext_pkg.name]
-            self.write_easy_install_pth(
+            self.write_easy_install_pth(exts, prefix=view.root)
-                exts,
+
-                prefix=extensions_layout.extendee_target_directory(self))
+    def add_files_to_view(self, view, merge_map):
+        bin_dir = self.spec.prefix.bin
+        for src, dst in merge_map.items():
+            if not path_contains_subdirectory(src, bin_dir):
+                view.link(src, dst)
+            elif not os.path.islink(src):
+                shutil.copy2(src, dst)
+                if 'script' in get_filetype(src):
+                    filter_file(
+                        self.spec.prefix, os.path.abspath(view.root), dst)
+            else:
+                orig_link_target = os.path.realpath(src)
+                new_link_target = os.path.abspath(merge_map[orig_link_target])
+                view.link(new_link_target, dst)
+
+    def remove_files_from_view(self, view, merge_map):
+        bin_dir = self.spec.prefix.bin
+        for src, dst in merge_map.items():
+            if not path_contains_subdirectory(src, bin_dir):
+                view.remove_file(src, dst)
+            else:
+                os.remove(dst)