hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

remove activate/deactivate support in favor of environments (#29317)

Browse source 
Environments and environment views have taken over the role of `spack activate/deactivate`, and we should deprecate these commands for several reasons:

- Global activation is a really poor idea:
   - Install prefixes should be immutable; since they can have multiple, unrelated dependents; see below
   - Added complexity elsewhere: verification of installations, tarballs for build caches, creation of environment views of packages with unrelated extensions "globally activated"... by removing the feature, it gets easier for people to contribute, and we'd end up with fewer bugs due to edge cases.
- Environment accomplish the same thing for non-global "activation" i.e. `spack view`, but better.

Also we write in the docs:

```
However, Spack global activations have two potential drawbacks:

#. Activated packages that involve compiled C extensions may still
   need their dependencies to be loaded manually.  For example,
   ``spack load openblas`` might be required to make ``py-numpy``
   work.

#. Global activations "break" a core feature of Spack, which is that
   multiple versions of a package can co-exist side-by-side.  For example,
   suppose you wish to run a Python package in two different
   environments but the same basic Python --- one with
   ``py-numpy@1.7`` and one with ``py-numpy@1.8``.  Spack extensions
   will not support this potential debugging use case.
```

Now that environments are established and views can take over the role of activation
non-destructively, we can remove global activation/deactivation.
This commit is contained in:
Harmen Stoppels 2022-11-11 09:50:07 +01:00 committed by GitHub
parent f11778bb02
commit 0f54a63dfd
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
24 changed files with 34 additions and 1708 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

127
lib/spack/docs/basic_usage.rst
View file

@ -1745,13 +1745,13 @@ directly when you run ``python``:
Using Extensions
^^^^^^^^^^^^^^^^
There are four ways to get ``numpy`` working in Python. The first is
There are multiple ways to get ``numpy`` working in Python. The first is
to use :ref:`shell-support`. You can simply ``load`` the extension,
and it will be added to the ``PYTHONPATH`` in your current shell:
and it will be added to the ``PYTHONPATH`` in your current shell, and
Python itself will be available in the ``PATH``:
.. code-block:: console
$ spack load python
$ spack load py-numpy
Now ``import numpy`` will succeed for as long as you keep your current
@ -1777,128 +1777,13 @@ load, you can use the ``spack module tcl|lmod loads`` command to get
the name of the module from the Spack spec.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Activating Extensions in a View
Extensions in an Environment
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Another way to use extensions is to create a view, which merges the
python installation along with the extensions into a single prefix.
See :ref:`configuring_environment_views` for a more in-depth description
of views.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Activating Extensions Globally
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
As an alternative to creating a merged prefix with Python and its extensions,
and prior to support for views, Spack has provided a means to install the
extension into the Spack installation prefix for the extendee. This has
typically been useful since extendable packages typically search their own
installation path for addons by default.
Global activations are performed with the ``spack activate`` command:
.. _cmd-spack-activate:
^^^^^^^^^^^^^^^^^^
``spack activate``
^^^^^^^^^^^^^^^^^^
.. code-block:: console
$ spack activate py-numpy
==> Activated extension py-setuptools@11.3.1%gcc@4.4.7 arch=linux-debian7-x86_64-3c74eb69 for python@2.7.8%gcc@4.4.7.
==> Activated extension py-nose@1.3.4%gcc@4.4.7 arch=linux-debian7-x86_64-5f70f816 for python@2.7.8%gcc@4.4.7.
==> Activated extension py-numpy@1.9.1%gcc@4.4.7 arch=linux-debian7-x86_64-66733244 for python@2.7.8%gcc@4.4.7.
Several things have happened here. The user requested that
``py-numpy`` be activated in the ``python`` installation it was built
with. Spack knows that ``py-numpy`` depends on ``py-nose`` and
``py-setuptools``, so it activated those packages first. Finally,
once all dependencies were activated in the ``python`` installation,
``py-numpy`` was activated as well.
If we run ``spack extensions`` again, we now see the three new
packages listed as activated:
.. code-block:: console
$ spack extensions python
==> python@2.7.8%gcc@4.4.7 arch=linux-debian7-x86_64-703c7a96
==> 36 extensions:
geos py-ipython py-pexpect py-pyside py-sip
py-basemap py-libxml2 py-pil py-pytz py-six
py-biopython py-mako py-pmw py-rpy2 py-sympy
py-cython py-matplotlib py-pychecker py-scientificpython py-virtualenv
py-dateutil py-mpi4py py-pygments py-scikit-learn
py-epydoc py-mx py-pylint py-scipy
py-gnuplot py-nose py-pyparsing py-setuptools
py-h5py py-numpy py-pyqt py-shiboken
==> 12 installed:
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
py-dateutil@2.4.0 py-nose@1.3.4 py-pyside@1.2.2
py-dateutil@2.4.0 py-numpy@1.9.1 py-pytz@2014.10
py-ipython@2.3.1 py-pygments@2.0.1 py-setuptools@11.3.1
py-matplotlib@1.4.2 py-pyparsing@2.0.3 py-six@1.9.0
==> 3 currently activated:
-- linux-debian7-x86_64 / gcc@4.4.7 --------------------------------
py-nose@1.3.4 py-numpy@1.9.1 py-setuptools@11.3.1
Now, when a user runs python, ``numpy`` will be available for import
*without* the user having to explicitly load it. ``python@2.7.8`` now
acts like a system Python installation with ``numpy`` installed inside
of it.
Spack accomplishes this by symbolically linking the *entire* prefix of
the ``py-numpy`` package into the prefix of the ``python`` package. To the
python interpreter, it looks like ``numpy`` is installed in the
``site-packages`` directory.
The only limitation of global activation is that you can only have a *single*
version of an extension activated at a time. This is because multiple
versions of the same extension would conflict if symbolically linked
into the same prefix. Users who want a different version of a package
can still get it by using environment modules or views, but they will have to
explicitly load their preferred version.
^^^^^^^^^^^^^^^^^^^^^^^^^^
``spack activate --force``
^^^^^^^^^^^^^^^^^^^^^^^^^^
If, for some reason, you want to activate a package *without* its
dependencies, you can use ``spack activate --force``:
.. code-block:: console
$ spack activate --force py-numpy
==> Activated extension py-numpy@1.9.1%gcc@4.4.7 arch=linux-debian7-x86_64-66733244 for python@2.7.8%gcc@4.4.7.
.. _cmd-spack-deactivate:
^^^^^^^^^^^^^^^^^^^^
``spack deactivate``
^^^^^^^^^^^^^^^^^^^^
We've seen how activating an extension can be used to set up a default
version of a Python module. Obviously, you may want to change that at
some point. ``spack deactivate`` is the command for this. There are
several variants:
* ``spack deactivate <extension>`` will deactivate a single
extension. If another activated extension depends on this one,
Spack will warn you and exit with an error.
* ``spack deactivate --force <extension>`` deactivates an extension
regardless of packages that depend on it.
* ``spack deactivate --all <extension>`` deactivates an extension and
all of its dependencies. Use ``--force`` to disregard dependents.
* ``spack deactivate --all <extendee>`` deactivates *all* activated
extensions of a package. For example, to deactivate *all* python
extensions, use:
.. code-block:: console
$ spack deactivate --all python
See :ref:`environments` for a more in-depth description
of environment views.
-----------------------
Filesystem requirements

29
lib/spack/docs/contribution_guide.rst
View file

@ -253,27 +253,6 @@ to update them.
multiple runs of ``spack style`` just to re-compute line numbers and
makes it much easier to fix errors directly off of the CI output.
.. warning::
Flake8 and ``pep8-naming`` require a number of dependencies in order
to run. If you installed ``py-flake8`` and ``py-pep8-naming``, the
easiest way to ensure the right packages are on your ``PYTHONPATH`` is
to run::
spack activate py-flake8
spack activate pep8-naming
so that all of the dependencies are symlinked to a central
location. If you see an error message like:
.. code-block:: console
Traceback (most recent call last):
File: "/usr/bin/flake8", line 5, in <module>
from pkg_resources import load_entry_point
ImportError: No module named pkg_resources
that means Flake8 couldn't find setuptools in your ``PYTHONPATH``.
^^^^^^^^^^^^^^^^^^^
Documentation Tests
@ -309,13 +288,9 @@ All of these can be installed with Spack, e.g.
.. code-block:: console
$ spack activate py-sphinx
$ spack activate py-sphinx-rtd-theme
$ spack activate py-sphinxcontrib-programoutput
$ spack load py-sphinx py-sphinx-rtd-theme py-sphinxcontrib-programoutput
so that all of the dependencies are symlinked into that Python's
tree. Alternatively, you could arrange for their library
directories to be added to PYTHONPATH. If you see an error message
so that all of the dependencies are added to PYTHONPATH. If you see an error message
like:
.. code-block:: console

4
lib/spack/docs/environments.rst
View file

@ -233,8 +233,8 @@ packages will be listed as roots of the Environment.
All of the Spack commands that act on the list of installed specs are
Environment-sensitive in this way, including ``install``,
``uninstall``, ``activate``, ``deactivate``, ``find``, ``extensions``,
and more. In the :ref:`environment-configuration` section we will discuss
``uninstall``, ``find``, ``extensions``, and more. In the
:ref:`environment-configuration` section we will discuss
Environment-sensitive commands further.
^^^^^^^^^^^^^^^^^^^^^

61
lib/spack/docs/packaging_guide.rst
View file

@ -2722,67 +2722,6 @@ extensions; as a consequence python extension packages (those inheriting from
``PythonPackage``) likewise override ``add_files_to_view`` in order to rewrite
shebang lines which point to the Python interpreter.
^^^^^^^^^^^^^^^^^^^^^^^^^
Activation & deactivation
^^^^^^^^^^^^^^^^^^^^^^^^^
Adding an extension to a view is referred to as an activation. If the view is
maintained in the Spack installation prefix of the extendee this is called a
global activation. Activations may involve updating some centralized state
that is maintained by the extendee package, so there can be additional work
for adding extensions compared with non-extension packages.
Spack's ``Package`` class has default ``activate`` and ``deactivate``
implementations that handle symbolically linking extensions' prefixes
into a specified view. Extendable packages can override these methods
to add custom activate/deactivate logic of their own. For example,
the ``activate`` and ``deactivate`` methods in the Python class handle
symbolic linking of extensions, but they also handle details surrounding
Python's ``.pth`` files, and other aspects of Python packaging.
Spack's extensions mechanism is designed to be extensible, so that
other packages (like Ruby, R, Perl, etc.) can provide their own
custom extension management logic, as they may not handle modules the
same way that Python does.
Let's look at Python's activate function:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py
:pyobject: Python.activate
:linenos:
This function is called on the *extendee* (Python). It first calls
``activate`` in the superclass, which handles symlinking the
extension package's prefix into the specified view. It then does
some special handling of the ``easy-install.pth`` file, part of
Python's setuptools.
Deactivate behaves similarly to activate, but it unlinks files:
.. literalinclude:: _spack_root/var/spack/repos/builtin/packages/python/package.py
:pyobject: Python.deactivate
:linenos:
Both of these methods call some custom functions in the Python
package. See the source for Spack's Python package for details.
^^^^^^^^^^^^^^^^^^^^
Activation arguments
^^^^^^^^^^^^^^^^^^^^
You may have noticed that the ``activate`` function defined above
takes keyword arguments. These are the keyword arguments from
``extends()``, and they are passed to both activate and deactivate.
This capability allows an extension to customize its own activation by
passing arguments to the extendee. Extendees can likewise implement
custom ``activate()`` and ``deactivate()`` functions to suit their
needs.
The only keyword argument supported by default is the ``ignore``
argument, which can take a regex, list of regexes, or a predicate to
determine which files *not* to symlink during activation.
.. _virtual-dependencies:
--------------------

53
lib/spack/spack/cmd/activate.py
View file

@ -1,53 +0,0 @@
# Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import llnl.util.tty as tty
import spack.cmd
import spack.cmd.common.arguments as arguments
import spack.environment as ev
from spack.filesystem_view import YamlFilesystemView
description = "activate a package extension"
section = "extensions"
level = "long"
def setup_parser(subparser):
subparser.add_argument(
"-f", "--force", action="store_true", help="activate without first activating dependencies"
)
subparser.add_argument("-v", "--view", metavar="VIEW", type=str, help="the view to operate on")
arguments.add_common_arguments(subparser, ["installed_spec"])
def activate(parser, args):
tty.warn(
"spack activate is deprecated in favor of " "environments and will be removed in v0.19.0"
)
specs = spack.cmd.parse_specs(args.spec)
if len(specs) != 1:
tty.die("activate requires one spec. %d given." % len(specs))
spec = spack.cmd.disambiguate_spec(specs[0], ev.active_environment())
if not spec.package.is_extension:
tty.die("%s is not an extension." % spec.name)
if args.view:
target = args.view
else:
target = spec.package.extendee_spec.prefix
view = YamlFilesystemView(target, spack.store.layout)
if spec.package.is_activated(view):
tty.msg("Package %s is already activated." % specs[0].short_spec)
return
# TODO: refactor FilesystemView.add_extension and use that here (so there
# aren't two ways of activating extensions)
spec.package.do_activate(view, with_dependencies=not args.force)

96
lib/spack/spack/cmd/deactivate.py
View file

@ -1,96 +0,0 @@
# Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import llnl.util.tty as tty
import spack.cmd
import spack.cmd.common.arguments as arguments
import spack.environment as ev
import spack.graph
import spack.store
from spack.filesystem_view import YamlFilesystemView
description = "deactivate a package extension"
section = "extensions"
level = "long"
def setup_parser(subparser):
subparser.add_argument(
"-f",
"--force",
action="store_true",
help="run deactivation even if spec is NOT currently activated",
)
subparser.add_argument("-v", "--view", metavar="VIEW", type=str, help="the view to operate on")
subparser.add_argument(
"-a",
"--all",
action="store_true",
help="deactivate all extensions of an extendable package, or "
"deactivate an extension AND its dependencies",
)
arguments.add_common_arguments(subparser, ["installed_spec"])
def deactivate(parser, args):
tty.warn(
"spack deactivate is deprecated in favor of " "environments and will be removed in v0.19.0"
)
specs = spack.cmd.parse_specs(args.spec)
if len(specs) != 1:
tty.die("deactivate requires one spec. %d given." % len(specs))
env = ev.active_environment()
spec = spack.cmd.disambiguate_spec(specs[0], env)
pkg = spec.package
if args.view:
target = args.view
elif pkg.is_extension:
target = pkg.extendee_spec.prefix
elif pkg.extendable:
target = spec.prefix
view = YamlFilesystemView(target, spack.store.layout)
if args.all:
if pkg.extendable:
tty.msg("Deactivating all extensions of %s" % pkg.spec.short_spec)
ext_pkgs = spack.store.db.activated_extensions_for(spec, view.extensions_layout)
for ext_pkg in ext_pkgs:
ext_pkg.spec.normalize()
if ext_pkg.is_activated(view):
ext_pkg.do_deactivate(view, force=True)
elif pkg.is_extension:
if not args.force and not spec.package.is_activated(view):
tty.die("%s is not activated." % pkg.spec.short_spec)
tty.msg("Deactivating %s and all dependencies." % pkg.spec.short_spec)
nodes_in_topological_order = spack.graph.topological_sort(spec)
for espec in reversed(nodes_in_topological_order):
epkg = espec.package
if epkg.extends(pkg.extendee_spec):
if epkg.is_activated(view) or args.force:
epkg.do_deactivate(view, force=args.force)
else:
tty.die("spack deactivate --all requires an extendable package " "or an extension.")
else:
if not pkg.is_extension:
tty.die(
"spack deactivate requires an extension.", "Did you mean 'spack deactivate --all'?"
)
if not args.force and not spec.package.is_activated(view):
tty.die("Package %s is not activated." % spec.short_spec)
spec.package.do_deactivate(view, force=args.force)

22
lib/spack/spack/cmd/extensions.py
View file

@ -14,7 +14,6 @@
import spack.environment as ev
import spack.repo
import spack.store
from spack.filesystem_view import YamlFilesystemView
description = "list extensions for package"
section = "extensions"
@ -38,10 +37,9 @@ def setup_parser(subparser):
"--show",
action="store",
default="all",
choices=("packages", "installed", "activated", "all"),
choices=("packages", "installed", "all"),
help="show only part of output",
)
subparser.add_argument("-v", "--view", metavar="VIEW", type=str, help="the view to operate on")
subparser.add_argument(
"spec",
@ -91,13 +89,6 @@ def extensions(parser, args):
tty.msg("%d extensions:" % len(extensions))
colify(ext.name for ext in extensions)
if args.view:
target = args.view
else:
target = spec.prefix
view = YamlFilesystemView(target, spack.store.layout)
if args.show in ("installed", "all"):
# List specs of installed extensions.
installed = [s.spec for s in spack.store.db.installed_extensions_for(spec)]
@ -109,14 +100,3 @@ def extensions(parser, args):
else:
tty.msg("%d installed:" % len(installed))
cmd.display_specs(installed, args)
if args.show in ("activated", "all"):
# List specs of activated extensions.
activated = view.extensions_layout.extension_map(spec)
if args.show == "all":
print
if not activated:
tty.msg("None activated.")
else:
tty.msg("%d activated:" % len(activated))
cmd.display_specs(activated.values(), args)

18
lib/spack/spack/database.py
View file

@ -53,7 +53,6 @@
InconsistentInstallDirectoryError,
)
from spack.error import SpackError
from spack.filesystem_view import YamlFilesystemView
from spack.util.crypto import bit_length
from spack.version import Version
@ -1379,23 +1378,6 @@ def installed_extensions_for(self, extendee_spec):
if spec.package.extends(extendee_spec):
yield spec.package
@_autospec
def activated_extensions_for(self, extendee_spec, extensions_layout=None):
"""
Return the specs of all packages that extend
the given spec
"""
if extensions_layout is None:
view = YamlFilesystemView(extendee_spec.prefix, spack.store.layout)
extensions_layout = view.extensions_layout
for spec in self.query():
try:
extensions_layout.check_activated(extendee_spec, spec)
yield spec.package
except spack.directory_layout.NoSuchExtensionError:
continue
# TODO: conditional way to do this instead of catching exceptions
def _get_by_hash_local(self, dag_hash, default=None, installed=any):
# hash is a full hash and is in the data somewhere
if dag_hash in self._data:

9
lib/spack/spack/directives.py
View file

@ -468,14 +468,7 @@ def _execute_depends_on(pkg):
@directive(("extendees", "dependencies"))
def extends(spec, type=("build", "run"), **kwargs):
"""Same as depends_on, but allows symlinking into dependency's
prefix tree.
This is for Python and other language modules where the module
needs to be installed into the prefix of the Python installation.
Spack handles this by installing modules into their own prefix,
but allowing ONE module version to be symlinked into a parent
Python install at a time, using ``spack activate``.
"""Same as depends_on, but also adds this package to the extendee list.
keyword arguments can be passed to extends() so that extension
packages can pass parameters to the extendee's extension

211
lib/spack/spack/directory_layout.py
View file

@ -10,10 +10,8 @@
import re
import shutil
import sys
import tempfile
from contextlib import contextmanager
import ruamel.yaml as yaml
import six
import llnl.util.filesystem as fs
@ -389,205 +387,6 @@ def remove_install_directory(self, spec, deprecated=False):
path = os.path.dirname(path)
class ExtensionsLayout(object):
"""A directory layout is used to associate unique paths with specs for
package extensions.
Keeps track of which extensions are activated for what package.
Depending on the use case, this can mean globally activated extensions
directly in the installation folder - or extensions activated in
filesystem views.
"""
def __init__(self, view, **kwargs):
self.view = view
def add_extension(self, spec, ext_spec):
"""Add to the list of currently installed extensions."""
raise NotImplementedError()
def check_activated(self, spec, ext_spec):
"""Ensure that ext_spec can be removed from spec.
If not, raise NoSuchExtensionError.
"""
raise NotImplementedError()
def check_extension_conflict(self, spec, ext_spec):
"""Ensure that ext_spec can be activated in spec.
If not, raise ExtensionAlreadyInstalledError or
ExtensionConflictError.
"""
raise NotImplementedError()
def extension_map(self, spec):
"""Get a dict of currently installed extension packages for a spec.
Dict maps { name : extension_spec }
Modifying dict does not affect internals of this layout.
"""
raise NotImplementedError()
def extendee_target_directory(self, extendee):
"""Specify to which full path extendee should link all files
from extensions."""
raise NotImplementedError
def remove_extension(self, spec, ext_spec):
"""Remove from the list of currently installed extensions."""
raise NotImplementedError()
class YamlViewExtensionsLayout(ExtensionsLayout):
"""Maintain extensions within a view."""
def __init__(self, view, layout):
"""layout is the corresponding YamlDirectoryLayout object for which
we implement extensions.
"""
super(YamlViewExtensionsLayout, self).__init__(view)
self.layout = layout
self.extension_file_name = "extensions.yaml"
# Cache of already written/read extension maps.
self._extension_maps = {}
def add_extension(self, spec, ext_spec):
_check_concrete(spec)
_check_concrete(ext_spec)
# Check whether it's already installed or if it's a conflict.
exts = self._extension_map(spec)
self.check_extension_conflict(spec, ext_spec)
# do the actual adding.
exts[ext_spec.name] = ext_spec
self._write_extensions(spec, exts)
def check_extension_conflict(self, spec, ext_spec):
exts = self._extension_map(spec)
if ext_spec.name in exts:
installed_spec = exts[ext_spec.name]
if ext_spec.dag_hash() == installed_spec.dag_hash():
raise ExtensionAlreadyInstalledError(spec, ext_spec)
else:
raise ExtensionConflictError(spec, ext_spec, installed_spec)
def check_activated(self, spec, ext_spec):
exts = self._extension_map(spec)
if (ext_spec.name not in exts) or (ext_spec != exts[ext_spec.name]):
raise NoSuchExtensionError(spec, ext_spec)
def extension_file_path(self, spec):
"""Gets full path to an installed package's extension file, which
keeps track of all the extensions for that package which have been
added to this view.
"""
_check_concrete(spec)
normalize_path = lambda p: (os.path.abspath(p).rstrip(os.path.sep))
view_prefix = self.view.get_projection_for_spec(spec)
if normalize_path(spec.prefix) == normalize_path(view_prefix):
# For backwards compatibility, when the view is the extended
# package's installation directory, do not include the spec name
# as a subdirectory.
components = [view_prefix, self.layout.metadata_dir, self.extension_file_name]
else:
components = [
view_prefix,
self.layout.metadata_dir,
spec.name,
self.extension_file_name,
]
return os.path.join(*components)
def extension_map(self, spec):
"""Defensive copying version of _extension_map() for external API."""
_check_concrete(spec)
return self._extension_map(spec).copy()
def remove_extension(self, spec, ext_spec):
_check_concrete(spec)
_check_concrete(ext_spec)
# Make sure it's installed before removing.
exts = self._extension_map(spec)
self.check_activated(spec, ext_spec)
# do the actual removing.
del exts[ext_spec.name]
self._write_extensions(spec, exts)
def _extension_map(self, spec):
"""Get a dict<name -> spec> for all extensions currently
installed for this package."""
_check_concrete(spec)
if spec not in self._extension_maps:
path = self.extension_file_path(spec)
if not os.path.exists(path):
self._extension_maps[spec] = {}
else:
by_hash = self.layout.specs_by_hash()
exts = {}
with open(path) as ext_file:
yaml_file = yaml.load(ext_file)
for entry in yaml_file["extensions"]:
name = next(iter(entry))
dag_hash = entry[name]["hash"]
prefix = entry[name]["path"]
if dag_hash not in by_hash:
raise InvalidExtensionSpecError(
"Spec %s not found in %s" % (dag_hash, prefix)
)
ext_spec = by_hash[dag_hash]
if prefix != ext_spec.prefix:
raise InvalidExtensionSpecError(
"Prefix %s does not match spec hash %s: %s"
% (prefix, dag_hash, ext_spec)
)
exts[ext_spec.name] = ext_spec
self._extension_maps[spec] = exts
return self._extension_maps[spec]
def _write_extensions(self, spec, extensions):
path = self.extension_file_path(spec)
if not extensions:
# Remove the empty extensions file
os.remove(path)
return
# Create a temp file in the same directory as the actual file.
dirname, basename = os.path.split(path)
fs.mkdirp(dirname)
tmp = tempfile.NamedTemporaryFile(prefix=basename, dir=dirname, delete=False)
# write tmp file
with tmp:
yaml.dump(
{
"extensions": [
{ext.name: {"hash": ext.dag_hash(), "path": str(ext.prefix)}}
for ext in sorted(extensions.values())
]
},
tmp,
default_flow_style=False,
encoding="utf-8",
)
# Atomic update by moving tmpfile on top of old one.
fs.rename(tmp.name, path)
class DirectoryLayoutError(SpackError):
"""Superclass for directory layout errors."""
@ -644,13 +443,3 @@ def __init__(self, spec, ext_spec, conflict):
"%s cannot be installed in %s because it conflicts with %s"
% (ext_spec.short_spec, spec.short_spec, conflict.short_spec)
)
class NoSuchExtensionError(DirectoryLayoutError):
"""Raised when an extension isn't there on deactivate."""
def __init__(self, spec, ext_spec):
super(NoSuchExtensionError, self).__init__(
"%s cannot be removed from %s because it's not activated."
% (ext_spec.short_spec, spec.short_spec)
)

120
lib/spack/spack/filesystem_view.py
View file

@ -37,10 +37,6 @@
import spack.store
import spack.util.spack_json as s_json
import spack.util.spack_yaml as s_yaml
from spack.directory_layout import (
ExtensionAlreadyInstalledError,
YamlViewExtensionsLayout,
)
from spack.error import SpackError
__all__ = ["FilesystemView", "YamlFilesystemView"]
@ -166,9 +162,6 @@ def add_specs(self, *specs, **kwargs):
"""
Add given specs to view.
The supplied specs might be standalone packages or extensions of
other packages.
Should accept `with_dependencies` as keyword argument (default
True) to indicate wether or not dependencies should be activated as
well.
@ -176,13 +169,7 @@ def add_specs(self, *specs, **kwargs):
Should except an `exclude` keyword argument containing a list of
regexps that filter out matching spec names.
This method should make use of `activate_{extension,standalone}`.
"""
raise NotImplementedError
def add_extension(self, spec):
"""
Add (link) an extension in this view. Does not add dependencies.
This method should make use of `activate_standalone`.
"""
raise NotImplementedError
@ -202,9 +189,6 @@ def remove_specs(self, *specs, **kwargs):
"""
Removes given specs from view.
The supplied spec might be a standalone package or an extension of
another package.
Should accept `with_dependencies` as keyword argument (default
True) to indicate wether or not dependencies should be deactivated
as well.
@ -216,13 +200,7 @@ def remove_specs(self, *specs, **kwargs):
Should except an `exclude` keyword argument containing a list of
regexps that filter out matching spec names.
This method should make use of `deactivate_{extension,standalone}`.
"""
raise NotImplementedError
def remove_extension(self, spec):
"""
Remove (unlink) an extension from this view.
This method should make use of `deactivate_standalone`.
"""
raise NotImplementedError
@ -296,8 +274,6 @@ def __init__(self, root, layout, **kwargs):
msg += " which does not match projections passed manually."
raise ConflictingProjectionsError(msg)
self.extensions_layout = YamlViewExtensionsLayout(self, layout)
self._croot = colorize_root(self._root) + " "
def write_projections(self):
@ -332,38 +308,10 @@ def add_specs(self, *specs, **kwargs):
self.print_conflict(v, s)
return
extensions = set(filter(lambda s: s.package.is_extension, specs))
standalones = specs - extensions
set(map(self._check_no_ext_conflicts, extensions))
# fail on first error, otherwise link extensions as well
if all(map(self.add_standalone, standalones)):
all(map(self.add_extension, extensions))
def add_extension(self, spec):
if not spec.package.is_extension:
tty.error(self._croot + "Package %s is not an extension." % spec.name)
return False
if spec.external:
tty.warn(self._croot + "Skipping external package: %s" % colorize_spec(spec))
return True
if not spec.package.is_activated(self):
spec.package.do_activate(self, verbose=self.verbose, with_dependencies=False)
# make sure the meta folder is linked as well (this is not done by the
# extension-activation mechnism)
if not self.check_added(spec):
self.link_meta_folder(spec)
return True
for s in specs:
self.add_standalone(s)
def add_standalone(self, spec):
if spec.package.is_extension:
tty.error(self._croot + "Package %s is an extension." % spec.name)
return False
if spec.external:
tty.warn(self._croot + "Skipping external package: %s" % colorize_spec(spec))
return True
@ -372,19 +320,6 @@ def add_standalone(self, spec):
tty.warn(self._croot + "Skipping already linked package: %s" % colorize_spec(spec))
return True
if spec.package.extendable:
# Check for globally activated extensions in the extendee that
# we're looking at.
activated = [p.spec for p in spack.store.db.activated_extensions_for(spec)]
if activated:
tty.error(
"Globally activated extensions cannot be used in "
"conjunction with filesystem views. "
"Please deactivate the following specs: "
)
spack.cmd.display_specs(activated, flags=True, variants=True, long=False)
return False
self.merge(spec)
self.link_meta_folder(spec)
@ -533,27 +468,10 @@ def remove_specs(self, *specs, **kwargs):
# Remove the packages from the view
for spec in to_deactivate_sorted:
if spec.package.is_extension:
self.remove_extension(spec, with_dependents=with_dependents)
else:
self.remove_standalone(spec)
self._purge_empty_directories()
def remove_extension(self, spec, with_dependents=True):
"""
Remove (unlink) an extension from this view.
"""
if not self.check_added(spec):
tty.warn(self._croot + "Skipping package not linked in view: %s" % spec.name)
return
if spec.package.is_activated(self):
spec.package.do_deactivate(
self, verbose=self.verbose, remove_dependents=with_dependents
)
self.unlink_meta_folder(spec)
def remove_standalone(self, spec):
"""
Remove (unlink) a standalone package from this view.
@ -575,14 +493,9 @@ def get_projection_for_spec(self, spec):
Relies on the ordering of projections to avoid ambiguity.
"""
spec = spack.spec.Spec(spec)
# Extensions are placed by their extendee, not by their own spec
locator_spec = spec
if spec.package.extendee_spec:
locator_spec = spec.package.extendee_spec
proj = spack.projections.get_projection(self.projections, locator_spec)
proj = spack.projections.get_projection(self.projections, spec)
if proj:
return os.path.join(self._root, locator_spec.format(proj))
return os.path.join(self._root, spec.format(proj))
return self._root
def get_all_specs(self):
@ -712,18 +625,6 @@ def unlink_meta_folder(self, spec):
assert os.path.exists(path)
shutil.rmtree(path)
def _check_no_ext_conflicts(self, spec):
"""
Check that there is no extension conflict for specs.
"""
extendee = spec.package.extendee_spec
try:
self.extensions_layout.check_extension_conflict(extendee, spec)
except ExtensionAlreadyInstalledError:
# we print the warning here because later on the order in which
# packages get activated is not clear (set-sorting)
tty.warn(self._croot + "Skipping already activated package: %s" % spec.name)
class SimpleFilesystemView(FilesystemView):
"""A simple and partial implementation of FilesystemView focused on
@ -842,14 +743,9 @@ def get_projection_for_spec(self, spec):
Relies on the ordering of projections to avoid ambiguity.
"""
spec = spack.spec.Spec(spec)
# Extensions are placed by their extendee, not by their own spec
locator_spec = spec
if spec.package.extendee_spec:
locator_spec = spec.package.extendee_spec
proj = spack.projections.get_projection(self.projections, locator_spec)
proj = spack.projections.get_projection(self.projections, spec)
if proj:
return os.path.join(self._root, locator_spec.format(proj))
return os.path.join(self._root, spec.format(proj))
return self._root

20
lib/spack/spack/hooks/extensions.py
View file

@ -1,20 +0,0 @@
# Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import spack
from spack.filesystem_view import YamlFilesystemView
def pre_uninstall(spec):
pkg = spec.package
assert spec.concrete
if pkg.is_extension:
target = pkg.extendee_spec.prefix
view = YamlFilesystemView(target, spack.store.layout)
if pkg.is_activated(view):
# deactivate globally
pkg.do_deactivate(force=True)

186
lib/spack/spack/package_base.py
View file

@ -1313,19 +1313,6 @@ def extends(self, spec):
s = self.extendee_spec
return s and spec.satisfies(s)
def is_activated(self, view):
"""Return True if package is activated."""
if not self.is_extension:
raise ValueError("is_activated called on package that is not an extension.")
if self.extendee_spec.installed_upstream:
# If this extends an upstream package, it cannot be activated for
# it. This bypasses construction of the extension map, which can
# can fail when run in the context of a downstream Spack instance
return False
extensions_layout = view.extensions_layout
exts = extensions_layout.extension_map(self.extendee_spec)
return (self.name in exts) and (exts[self.name] == self.spec)
def provides(self, vpkg_name):
"""
True if this package provides a virtual package with the specified name
@ -2325,30 +2312,6 @@ def do_deprecate(self, deprecator, link_fn):
"""Deprecate this package in favor of deprecator spec"""
spec = self.spec
# Check whether package to deprecate has active extensions
if self.extendable:
view = spack.filesystem_view.YamlFilesystemView(spec.prefix, spack.store.layout)
active_exts = view.extensions_layout.extension_map(spec).values()
if active_exts:
short = spec.format("{name}/{hash:7}")
m = "Spec %s has active extensions\n" % short
for active in active_exts:
m += " %s\n" % active.format("{name}/{hash:7}")
m += "Deactivate extensions before deprecating %s" % short
tty.die(m)
# Check whether package to deprecate is an active extension
if self.is_extension:
extendee = self.extendee_spec
view = spack.filesystem_view.YamlFilesystemView(extendee.prefix, spack.store.layout)
if self.is_activated(view):
short = spec.format("{name}/{hash:7}")
short_ext = extendee.format("{name}/{hash:7}")
msg = "Spec %s is an active extension of %s\n" % (short, short_ext)
msg += "Deactivate %s to be able to deprecate it" % short
tty.die(msg)
# Install deprecator if it isn't installed already
if not spack.store.db.query(deprecator):
deprecator.package.do_install()
@ -2378,155 +2341,6 @@ def _check_extendable(self):
if not self.extendable:
raise ValueError("Package %s is not extendable!" % self.name)
def _sanity_check_extension(self):
if not self.is_extension:
raise ActivationError("This package is not an extension.")
extendee_package = self.extendee_spec.package
extendee_package._check_extendable()
if not self.extendee_spec.installed:
raise ActivationError("Can only (de)activate extensions for installed packages.")
if not self.spec.installed:
raise ActivationError("Extensions must first be installed.")
if self.extendee_spec.name not in self.extendees:
raise ActivationError("%s does not extend %s!" % (self.name, self.extendee.name))
def do_activate(self, view=None, with_dependencies=True, verbose=True):
"""Called on an extension to invoke the extendee's activate method.
Commands should call this routine, and should not call
activate() directly.
"""
if verbose:
tty.msg(
"Activating extension {0} for {1}".format(
self.spec.cshort_spec, self.extendee_spec.cshort_spec
)
)
self._sanity_check_extension()
if not view:
view = YamlFilesystemView(self.extendee_spec.prefix, spack.store.layout)
extensions_layout = view.extensions_layout
try:
extensions_layout.check_extension_conflict(self.extendee_spec, self.spec)
except spack.directory_layout.ExtensionAlreadyInstalledError as e:
# already installed, let caller know
tty.msg(e.message)
return
# Activate any package dependencies that are also extensions.
if with_dependencies:
for spec in self.dependency_activations():
if not spec.package.is_activated(view):
spec.package.do_activate(
view, with_dependencies=with_dependencies, verbose=verbose
)
self.extendee_spec.package.activate(self, view, **self.extendee_args)
extensions_layout.add_extension(self.extendee_spec, self.spec)
if verbose:
tty.debug(
"Activated extension {0} for {1}".format(
self.spec.cshort_spec, self.extendee_spec.cshort_spec
)
)
def dependency_activations(self):
return (
spec
for spec in self.spec.traverse(root=False, deptype="run")
if spec.package.extends(self.extendee_spec)
)
def activate(self, extension, view, **kwargs):
"""
Add the extension to the specified view.
Package authors can override this function to maintain some
centralized state related to the set of activated extensions
for a package.
Spack internals (commands, hooks, etc.) should call
do_activate() method so that proper checks are always executed.
"""
view.merge(extension.spec, ignore=kwargs.get("ignore", None))
def do_deactivate(self, view=None, **kwargs):
"""Remove this extension package from the specified view. Called
on the extension to invoke extendee's deactivate() method.
`remove_dependents=True` deactivates extensions depending on this
package instead of raising an error.
"""
self._sanity_check_extension()
force = kwargs.get("force", False)
verbose = kwargs.get("verbose", True)
remove_dependents = kwargs.get("remove_dependents", False)
if verbose:
tty.msg(
"Deactivating extension {0} for {1}".format(
self.spec.cshort_spec, self.extendee_spec.cshort_spec
)
)
if not view:
view = YamlFilesystemView(self.extendee_spec.prefix, spack.store.layout)
extensions_layout = view.extensions_layout
# Allow a force deactivate to happen. This can unlink
# spurious files if something was corrupted.
if not force:
extensions_layout.check_activated(self.extendee_spec, self.spec)
activated = extensions_layout.extension_map(self.extendee_spec)
for name, aspec in activated.items():
if aspec == self.spec:
continue
for dep in aspec.traverse(deptype="run"):
if self.spec == dep:
if remove_dependents:
aspec.package.do_deactivate(**kwargs)
else:
msg = (
"Cannot deactivate {0} because {1} is "
"activated and depends on it"
)
raise ActivationError(
msg.format(self.spec.cshort_spec, aspec.cshort_spec)
)
self.extendee_spec.package.deactivate(self, view, **self.extendee_args)
# redundant activation check -- makes SURE the spec is not
# still activated even if something was wrong above.
if self.is_activated(view):
extensions_layout.remove_extension(self.extendee_spec, self.spec)
if verbose:
tty.debug(
"Deactivated extension {0} for {1}".format(
self.spec.cshort_spec, self.extendee_spec.cshort_spec
)
)
def deactivate(self, extension, view, **kwargs):
"""
Remove all extension files from the specified view.
Package authors can override this method to support other
extension mechanisms. Spack internals (commands, hooks, etc.)
should call do_deactivate() method so that proper checks are
always executed.
"""
view.unmerge(extension.spec, ignore=kwargs.get("ignore", None))
def view(self):
"""Create a view with the prefix of this package as the root.
Extensions added to this view will modify the installation prefix of

41
lib/spack/spack/test/cmd/activate.py
View file

@ -1,41 +0,0 @@
# Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import sys
import pytest
from spack.main import SpackCommand
activate = SpackCommand("activate")
deactivate = SpackCommand("deactivate")
install = SpackCommand("install")
extensions = SpackCommand("extensions")
pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
def test_activate(mock_packages, mock_archive, mock_fetch, config, install_mockery):
install("extension1")
activate("extension1")
output = extensions("--show", "activated", "extendee")
assert "extension1" in output
def test_deactivate(mock_packages, mock_archive, mock_fetch, config, install_mockery):
install("extension1")
activate("extension1")
deactivate("extension1")
output = extensions("--show", "activated", "extendee")
assert "extension1" not in output
def test_deactivate_all(mock_packages, mock_archive, mock_fetch, config, install_mockery):
install("extension1")
install("extension2")
activate("extension1")
activate("extension2")
deactivate("--all", "extendee")
output = extensions("--show", "activated", "extendee")
assert "extension1" not in output

19
lib/spack/spack/test/cmd/deprecate.py
View file

@ -15,7 +15,6 @@
uninstall = SpackCommand("uninstall")
deprecate = SpackCommand("deprecate")
find = SpackCommand("find")
activate = SpackCommand("activate")
pytestmark = pytest.mark.skipif(sys.platform == "win32", reason="does not run on windows")
@ -89,24 +88,6 @@ def test_deprecate_deps(mock_packages, mock_archive, mock_fetch, install_mockery
assert sorted(deprecated) == sorted(list(old_spec.traverse()))
def test_deprecate_fails_active_extensions(
mock_packages, mock_archive, mock_fetch, install_mockery
):
"""Tests that active extensions and their extendees cannot be
deprecated."""
install("extendee")
install("extension1")
activate("extension1")
output = deprecate("-yi", "extendee", "extendee@nonexistent", fail_on_error=False)
assert "extension1" in output
assert "Deactivate extensions before deprecating" in output
output = deprecate("-yiD", "extension1", "extension1@notaversion", fail_on_error=False)
assert "extendee" in output
assert "is an active extension of" in output
def test_uninstall_deprecated(mock_packages, mock_archive, mock_fetch, install_mockery):
"""Tests that we can still uninstall deprecated packages."""
install("libelf@0.8.13")

20
lib/spack/spack/test/cmd/extensions.py
View file

@ -35,12 +35,11 @@ def python_database(mock_packages, mutable_database):
def test_extensions(mock_packages, python_database, config, capsys):
ext2 = Spec("py-extension2").concretized()
def check_output(ni, na):
def check_output(ni):
with capsys.disabled():
output = extensions("python")
packages = extensions("-s", "packages", "python")
installed = extensions("-s", "installed", "python")
activated = extensions("-s", "activated", "python")
assert "==> python@2.7.11" in output
assert "==> 2 extensions" in output
assert "py-extension1" in output
@ -50,26 +49,13 @@ def check_output(ni, na):
assert "py-extension1" in packages
assert "py-extension2" in packages
assert "installed" not in packages
assert "activated" not in packages
assert ("%s installed" % (ni if ni else "None")) in output
assert ("%s activated" % (na if na else "None")) in output
assert ("%s installed" % (ni if ni else "None")) in installed
assert ("%s activated" % (na if na else "None")) in activated
check_output(2, 0)
ext2.package.do_activate()
check_output(2, 2)
ext2.package.do_deactivate(force=True)
check_output(2, 1)
ext2.package.do_activate()
check_output(2, 2)
check_output(2)
ext2.package.do_uninstall(force=True)
check_output(1, 1)
check_output(1)
def test_extensions_no_arguments(mock_packages):

82
lib/spack/spack/test/cmd/view.py
View file

@ -12,7 +12,6 @@
from spack.main import SpackCommand
from spack.spec import Spec
activate = SpackCommand("activate")
extensions = SpackCommand("extensions")
install = SpackCommand("install")
view = SpackCommand("view")
@ -135,46 +134,9 @@ def test_view_extension(tmpdir, mock_packages, mock_archive, mock_fetch, config,
assert "extension1@1.0" in all_installed
assert "extension1@2.0" in all_installed
assert "extension2@1.0" in all_installed
global_activated = extensions("--show", "activated", "extendee")
assert "extension1@1.0" not in global_activated
assert "extension1@2.0" not in global_activated
assert "extension2@1.0" not in global_activated
view_activated = extensions("--show", "activated", "-v", viewpath, "extendee")
assert "extension1@1.0" in view_activated
assert "extension1@2.0" not in view_activated
assert "extension2@1.0" not in view_activated
assert os.path.exists(os.path.join(viewpath, "bin", "extension1"))
def test_view_extension_projection(
tmpdir, mock_packages, mock_archive, mock_fetch, config, install_mockery
):
install("extendee@1.0")
install("extension1@1.0")
install("extension1@2.0")
install("extension2@1.0")
viewpath = str(tmpdir.mkdir("view"))
view_projection = {"all": "{name}-{version}"}
projection_file = create_projection_file(tmpdir, view_projection)
view("symlink", viewpath, "--projection-file={0}".format(projection_file), "extension1@1.0")
all_installed = extensions("--show", "installed", "extendee")
assert "extension1@1.0" in all_installed
assert "extension1@2.0" in all_installed
assert "extension2@1.0" in all_installed
global_activated = extensions("--show", "activated", "extendee")
assert "extension1@1.0" not in global_activated
assert "extension1@2.0" not in global_activated
assert "extension2@1.0" not in global_activated
view_activated = extensions("--show", "activated", "-v", viewpath, "extendee")
assert "extension1@1.0" in view_activated
assert "extension1@2.0" not in view_activated
assert "extension2@1.0" not in view_activated
assert os.path.exists(os.path.join(viewpath, "extendee-1.0", "bin", "extension1"))
def test_view_extension_remove(
tmpdir, mock_packages, mock_archive, mock_fetch, config, install_mockery
):
@ -185,10 +147,6 @@ def test_view_extension_remove(
view("remove", viewpath, "extension1@1.0")
all_installed = extensions("--show", "installed", "extendee")
assert "extension1@1.0" in all_installed
global_activated = extensions("--show", "activated", "extendee")
assert "extension1@1.0" not in global_activated
view_activated = extensions("--show", "activated", "-v", viewpath, "extendee")
assert "extension1@1.0" not in view_activated
assert not os.path.exists(os.path.join(viewpath, "bin", "extension1"))
@ -217,46 +175,6 @@ def test_view_extension_conflict_ignored(
assert fin.read() == "1.0"
def test_view_extension_global_activation(
tmpdir, mock_packages, mock_archive, mock_fetch, config, install_mockery
):
install("extendee")
install("extension1@1.0")
install("extension1@2.0")
install("extension2@1.0")
viewpath = str(tmpdir.mkdir("view"))
view("symlink", viewpath, "extension1@1.0")
activate("extension1@2.0")
activate("extension2@1.0")
all_installed = extensions("--show", "installed", "extendee")
assert "extension1@1.0" in all_installed
assert "extension1@2.0" in all_installed
assert "extension2@1.0" in all_installed
global_activated = extensions("--show", "activated", "extendee")
assert "extension1@1.0" not in global_activated
assert "extension1@2.0" in global_activated
assert "extension2@1.0" in global_activated
view_activated = extensions("--show", "activated", "-v", viewpath, "extendee")
assert "extension1@1.0" in view_activated
assert "extension1@2.0" not in view_activated
assert "extension2@1.0" not in view_activated
assert os.path.exists(os.path.join(viewpath, "bin", "extension1"))
assert not os.path.exists(os.path.join(viewpath, "bin", "extension2"))
def test_view_extendee_with_global_activations(
tmpdir, mock_packages, mock_archive, mock_fetch, config, install_mockery
):
install("extendee")
install("extension1@1.0")
install("extension1@2.0")
install("extension2@1.0")
viewpath = str(tmpdir.mkdir("view"))
activate("extension1@2.0")
output = view("symlink", viewpath, "extension1@1.0")
assert "Error: Globally activated extensions cannot be used" in output
def test_view_fails_with_missing_projections_file(tmpdir):
viewpath = str(tmpdir.mkdir("view"))
projection_file = os.path.join(str(tmpdir), "nonexistent")

6
lib/spack/spack/test/packages.py
View file

@ -84,12 +84,6 @@ def test_inheritance_of_patches(self):
# Will error if inheritor package cannot find inherited patch files
s.concretize()
def test_dependency_extensions(self):
s = Spec("extension2")
s.concretize()
deps = set(x.name for x in s.package.dependency_activations())
assert deps == set(["extension1"])
def test_import_class_from_package(self):
from spack.pkg.builtin.mock.mpich import Mpich # noqa: F401

402
lib/spack/spack/test/test_activations.py
View file

@ -1,402 +0,0 @@
# Copyright 2013-2022 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
"""This includes tests for customized activation logic for specific packages
(e.g. python and perl).
"""
import os
import sys
import pytest
from llnl.util.link_tree import MergeConflictError
import spack.package_base
import spack.spec
from spack.directory_layout import DirectoryLayout
from spack.filesystem_view import YamlFilesystemView
pytestmark = pytest.mark.skipif(
sys.platform == "win32",
reason="Python activation not currently supported on Windows",
)
def create_ext_pkg(name, prefix, extendee_spec, monkeypatch):
ext_spec = spack.spec.Spec(name)
ext_spec._concrete = True
ext_spec.package.spec.prefix = prefix
ext_pkg = ext_spec.package
# temporarily override extendee_spec property on the package
monkeypatch.setattr(ext_pkg.__class__, "extendee_spec", extendee_spec)
return ext_pkg
def create_python_ext_pkg(name, prefix, python_spec, monkeypatch, namespace=None):
ext_pkg = create_ext_pkg(name, prefix, python_spec, monkeypatch)
ext_pkg.py_namespace = namespace
return ext_pkg
def create_dir_structure(tmpdir, dir_structure):
for fname, children in dir_structure.items():
tmpdir.ensure(fname, dir=fname.endswith("/"))
if children:
create_dir_structure(tmpdir.join(fname), children)
@pytest.fixture()
def builtin_and_mock_packages():
# These tests use mock_repo packages to test functionality of builtin
# packages for python and perl. To test this we put the mock repo at lower
# precedence than the builtin repo, so we test builtin.perl against
# builtin.mock.perl-extension.
repo_dirs = [spack.paths.packages_path, spack.paths.mock_packages_path]
with spack.repo.use_repositories(*repo_dirs):
yield
@pytest.fixture()
def python_and_extension_dirs(tmpdir, builtin_and_mock_packages):
python_dirs = {"bin/": {"python": None}, "lib/": {"python2.7/": {"site-packages/": None}}}
python_name = "python"
python_prefix = tmpdir.join(python_name)
create_dir_structure(python_prefix, python_dirs)
python_spec = spack.spec.Spec("python@2.7.12")
python_spec._concrete = True
python_spec.package.spec.prefix = str(python_prefix)
ext_dirs = {
"bin/": {"py-ext-tool": None},
"lib/": {"python2.7/": {"site-packages/": {"py-extension1/": {"sample.py": None}}}},
}
ext_name = "py-extension1"
ext_prefix = tmpdir.join(ext_name)
create_dir_structure(ext_prefix, ext_dirs)
easy_install_location = "lib/python2.7/site-packages/easy-install.pth"
with open(str(ext_prefix.join(easy_install_location)), "w") as f:
f.write(
"""path/to/ext1.egg
path/to/setuptools.egg"""
)
return str(python_prefix), str(ext_prefix)
@pytest.fixture()
def namespace_extensions(tmpdir, builtin_and_mock_packages):
ext1_dirs = {
"bin/": {"py-ext-tool1": None},
"lib/": {
"python2.7/": {
"site-packages/": {
"examplenamespace/": {"__init__.py": None, "ext1_sample.py": None}
}
}
},
}
ext2_dirs = {
"bin/": {"py-ext-tool2": None},
"lib/": {
"python2.7/": {
"site-packages/": {
"examplenamespace/": {"__init__.py": None, "ext2_sample.py": None}
}
}
},
}
ext1_name = "py-extension1"
ext1_prefix = tmpdir.join(ext1_name)
create_dir_structure(ext1_prefix, ext1_dirs)
ext2_name = "py-extension2"
ext2_prefix = tmpdir.join(ext2_name)
create_dir_structure(ext2_prefix, ext2_dirs)
return str(ext1_prefix), str(ext2_prefix), "examplenamespace"
def test_python_activation_with_files(
tmpdir, python_and_extension_dirs, monkeypatch, builtin_and_mock_packages
):
python_prefix, ext_prefix = python_and_extension_dirs
python_spec = spack.spec.Spec("python@2.7.12")
python_spec._concrete = True
python_spec.package.spec.prefix = python_prefix
ext_pkg = create_python_ext_pkg("py-extension1", ext_prefix, python_spec, monkeypatch)
python_pkg = python_spec.package
python_pkg.activate(ext_pkg, python_pkg.view())
assert os.path.exists(os.path.join(python_prefix, "bin/py-ext-tool"))
easy_install_location = "lib/python2.7/site-packages/easy-install.pth"
with open(os.path.join(python_prefix, easy_install_location), "r") as f:
easy_install_contents = f.read()
assert "ext1.egg" in easy_install_contents
assert "setuptools.egg" not in easy_install_contents
def test_python_activation_view(
tmpdir, python_and_extension_dirs, builtin_and_mock_packages, monkeypatch
):
python_prefix, ext_prefix = python_and_extension_dirs
python_spec = spack.spec.Spec("python@2.7.12")
python_spec._concrete = True
python_spec.package.spec.prefix = python_prefix
ext_pkg = create_python_ext_pkg("py-extension1", ext_prefix, python_spec, monkeypatch)
view_dir = str(tmpdir.join("view"))
layout = DirectoryLayout(view_dir)
view = YamlFilesystemView(view_dir, layout)
python_pkg = python_spec.package
python_pkg.activate(ext_pkg, view)
assert not os.path.exists(os.path.join(python_prefix, "bin/py-ext-tool"))
assert os.path.exists(os.path.join(view_dir, "bin/py-ext-tool"))
def test_python_ignore_namespace_init_conflict(
tmpdir, namespace_extensions, builtin_and_mock_packages, monkeypatch
):
"""Test the view update logic in PythonPackage ignores conflicting
instances of __init__ for packages which are in the same namespace.
"""
ext1_prefix, ext2_prefix, py_namespace = namespace_extensions
python_spec = spack.spec.Spec("python@2.7.12")
python_spec._concrete = True
ext1_pkg = create_python_ext_pkg(
"py-extension1", ext1_prefix, python_spec, monkeypatch, py_namespace
)
ext2_pkg = create_python_ext_pkg(
"py-extension2", ext2_prefix, python_spec, monkeypatch, py_namespace
)
view_dir = str(tmpdir.join("view"))
layout = DirectoryLayout(view_dir)
view = YamlFilesystemView(view_dir, layout)
python_pkg = python_spec.package
python_pkg.activate(ext1_pkg, view)
# Normally handled by Package.do_activate, but here we activate directly
view.extensions_layout.add_extension(python_spec, ext1_pkg.spec)
python_pkg.activate(ext2_pkg, view)
f1 = "lib/python2.7/site-packages/examplenamespace/ext1_sample.py"
f2 = "lib/python2.7/site-packages/examplenamespace/ext2_sample.py"
init_file = "lib/python2.7/site-packages/examplenamespace/__init__.py"
assert os.path.exists(os.path.join(view_dir, f1))
assert os.path.exists(os.path.join(view_dir, f2))
assert os.path.exists(os.path.join(view_dir, init_file))
def test_python_keep_namespace_init(
tmpdir, namespace_extensions, builtin_and_mock_packages, monkeypatch
):
"""Test the view update logic in PythonPackage keeps the namespace
__init__ file as long as one package in the namespace still
exists.
"""
ext1_prefix, ext2_prefix, py_namespace = namespace_extensions
python_spec = spack.spec.Spec("python@2.7.12")
python_spec._concrete = True
ext1_pkg = create_python_ext_pkg(
"py-extension1", ext1_prefix, python_spec, monkeypatch, py_namespace
)
ext2_pkg = create_python_ext_pkg(
"py-extension2", ext2_prefix, python_spec, monkeypatch, py_namespace
)
view_dir = str(tmpdir.join("view"))
layout = DirectoryLayout(view_dir)
view = YamlFilesystemView(view_dir, layout)
python_pkg = python_spec.package
python_pkg.activate(ext1_pkg, view)
# Normally handled by Package.do_activate, but here we activate directly
view.extensions_layout.add_extension(python_spec, ext1_pkg.spec)
python_pkg.activate(ext2_pkg, view)
view.extensions_layout.add_extension(python_spec, ext2_pkg.spec)
f1 = "lib/python2.7/site-packages/examplenamespace/ext1_sample.py"
init_file = "lib/python2.7/site-packages/examplenamespace/__init__.py"
python_pkg.deactivate(ext1_pkg, view)
view.extensions_layout.remove_extension(python_spec, ext1_pkg.spec)
assert not os.path.exists(os.path.join(view_dir, f1))
assert os.path.exists(os.path.join(view_dir, init_file))
python_pkg.deactivate(ext2_pkg, view)
view.extensions_layout.remove_extension(python_spec, ext2_pkg.spec)
assert not os.path.exists(os.path.join(view_dir, init_file))
def test_python_namespace_conflict(
tmpdir, namespace_extensions, monkeypatch, builtin_and_mock_packages
):
"""Test the view update logic in PythonPackage reports an error when two
python extensions with different namespaces have a conflicting __init__
file.
"""
ext1_prefix, ext2_prefix, py_namespace = namespace_extensions
other_namespace = py_namespace + "other"
python_spec = spack.spec.Spec("python@2.7.12")
python_spec._concrete = True
ext1_pkg = create_python_ext_pkg(
"py-extension1", ext1_prefix, python_spec, monkeypatch, py_namespace
)
ext2_pkg = create_python_ext_pkg(
"py-extension2", ext2_prefix, python_spec, monkeypatch, other_namespace
)
view_dir = str(tmpdir.join("view"))
layout = DirectoryLayout(view_dir)
view = YamlFilesystemView(view_dir, layout)
python_pkg = python_spec.package
python_pkg.activate(ext1_pkg, view)
view.extensions_layout.add_extension(python_spec, ext1_pkg.spec)
with pytest.raises(MergeConflictError):
python_pkg.activate(ext2_pkg, view)
@pytest.fixture()
def perl_and_extension_dirs(tmpdir, builtin_and_mock_packages):
perl_dirs = {
"bin/": {"perl": None},
"lib/": {"site_perl/": {"5.24.1/": {"x86_64-linux/": None}}},
}
perl_name = "perl"
perl_prefix = tmpdir.join(perl_name)
create_dir_structure(perl_prefix, perl_dirs)
perl_spec = spack.spec.Spec("perl@5.24.1")
perl_spec._concrete = True
perl_spec.package.spec.prefix = str(perl_prefix)
ext_dirs = {
"bin/": {"perl-ext-tool": None},
"lib/": {"site_perl/": {"5.24.1/": {"x86_64-linux/": {"TestExt/": {}}}}},
}
ext_name = "perl-extension"
ext_prefix = tmpdir.join(ext_name)
create_dir_structure(ext_prefix, ext_dirs)
return str(perl_prefix), str(ext_prefix)
def test_perl_activation(tmpdir, builtin_and_mock_packages, monkeypatch):
# Note the lib directory is based partly on the perl version
perl_spec = spack.spec.Spec("perl@5.24.1")
perl_spec._concrete = True
perl_name = "perl"
tmpdir.ensure(perl_name, dir=True)
perl_prefix = str(tmpdir.join(perl_name))
# Set the prefix on the package's spec reference because that is a copy of
# the original spec
perl_spec.package.spec.prefix = perl_prefix
ext_name = "perl-extension"
tmpdir.ensure(ext_name, dir=True)
ext_pkg = create_ext_pkg(ext_name, str(tmpdir.join(ext_name)), perl_spec, monkeypatch)
perl_pkg = perl_spec.package
perl_pkg.activate(ext_pkg, perl_pkg.view())
def test_perl_activation_with_files(
tmpdir, perl_and_extension_dirs, monkeypatch, builtin_and_mock_packages
):
perl_prefix, ext_prefix = perl_and_extension_dirs
perl_spec = spack.spec.Spec("perl@5.24.1")
perl_spec._concrete = True
perl_spec.package.spec.prefix = perl_prefix
ext_pkg = create_ext_pkg("perl-extension", ext_prefix, perl_spec, monkeypatch)
perl_pkg = perl_spec.package
perl_pkg.activate(ext_pkg, perl_pkg.view())
assert os.path.exists(os.path.join(perl_prefix, "bin/perl-ext-tool"))
def test_perl_activation_view(
tmpdir, perl_and_extension_dirs, monkeypatch, builtin_and_mock_packages
):
perl_prefix, ext_prefix = perl_and_extension_dirs
perl_spec = spack.spec.Spec("perl@5.24.1")
perl_spec._concrete = True
perl_spec.package.spec.prefix = perl_prefix
ext_pkg = create_ext_pkg("perl-extension", ext_prefix, perl_spec, monkeypatch)
view_dir = str(tmpdir.join("view"))
layout = DirectoryLayout(view_dir)
view = YamlFilesystemView(view_dir, layout)
perl_pkg = perl_spec.package
perl_pkg.activate(ext_pkg, view)
assert not os.path.exists(os.path.join(perl_prefix, "bin/perl-ext-tool"))
assert os.path.exists(os.path.join(view_dir, "bin/perl-ext-tool"))
def test_is_activated_upstream_extendee(tmpdir, builtin_and_mock_packages, monkeypatch):
"""When an extendee is installed upstream, make sure that the extension
spec is never considered to be globally activated for it.
"""
extendee_spec = spack.spec.Spec("python")
extendee_spec._concrete = True
python_name = "python"
tmpdir.ensure(python_name, dir=True)
python_prefix = str(tmpdir.join(python_name))
# Set the prefix on the package's spec reference because that is a copy of
# the original spec
extendee_spec.package.spec.prefix = python_prefix
monkeypatch.setattr(extendee_spec.__class__, "installed_upstream", True)
ext_name = "py-extension1"
tmpdir.ensure(ext_name, dir=True)
ext_pkg = create_ext_pkg(ext_name, str(tmpdir.join(ext_name)), extendee_spec, monkeypatch)
# The view should not be checked at all if the extendee is installed
# upstream, so use 'None' here
mock_view = None
assert not ext_pkg.is_activated(mock_view)

21
lib/spack/spack/test/views.py
View file

@ -3,7 +3,6 @@
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import os
import sys
import pytest
@ -13,26 +12,6 @@
from spack.spec import Spec
@pytest.mark.skipif(sys.platform == "win32", reason="Not supported on Windows (yet)")
def test_global_activation(install_mockery, mock_fetch):
"""This test ensures that views which are maintained inside of an extendee
package's prefix are maintained as expected and are compatible with
global activations prior to #7152.
"""
spec = Spec("extension1").concretized()
pkg = spec.package
pkg.do_install()
pkg.do_activate()
extendee_spec = spec["extendee"]
extendee_pkg = spec["extendee"].package
view = extendee_pkg.view()
assert pkg.is_activated(view)
expected_path = os.path.join(extendee_spec.prefix, ".spack", "extensions.yaml")
assert view.extensions_layout.extension_file_path(extendee_spec) == expected_path
@pytest.mark.skipif(sys.platform == "win32", reason="Not supported on Windows (yet)")
def test_remove_extensions_ordered(install_mockery, mock_fetch, tmpdir):
view_dir = str(tmpdir.join("view"))

29
lib/spack/spack/verify.py
View file

@ -162,39 +162,12 @@ def check_spec_manifest(spec):
results.add_error(prefix, "manifest corrupted")
return results
# Get extensions active in spec
view = spack.filesystem_view.YamlFilesystemView(prefix, spack.store.layout)
active_exts = view.extensions_layout.extension_map(spec).values()
ext_file = ""
if active_exts:
# No point checking contents of this file as it is the only source of
# truth for that information.
ext_file = view.extensions_layout.extension_file_path(spec)
def is_extension_artifact(p):
if os.path.islink(p):
if any(os.readlink(p).startswith(e.prefix) for e in active_exts):
# This file is linked in by an extension. Belongs to extension
return True
elif os.path.isdir(p) and p not in manifest:
if all(is_extension_artifact(os.path.join(p, f)) for f in os.listdir(p)):
return True
return False
for root, dirs, files in os.walk(prefix):
for entry in list(dirs + files):
path = os.path.join(root, entry)
# Do not check links from prefix to active extension
# TODO: make this stricter for non-linux systems that use symlink
# permissions
# Do not check directories that only exist for extensions
if is_extension_artifact(path):
continue
# Do not check manifest file. Can't store your own hash
# Nothing to check for ext_file
if path == manifest_file or path == ext_file:
if path == manifest_file:
continue
data = manifest.pop(path, {})

22
share/spack/spack-completion.bash
View file

@ -337,16 +337,7 @@ _spack() {
then
SPACK_COMPREPLY="-h --help -H --all-help --color -c --config -C --config-scope -d --debug --timestamp --pdb -e --env -D --env-dir -E --no-env --use-env-repo -k --insecure -l --enable-locks -L --disable-locks -m --mock -b --bootstrap -p --profile --sorted-profile --lines -v --verbose --stacktrace --backtrace -V --version --print-shell-vars"
else
SPACK_COMPREPLY="activate add arch audit blame bootstrap build-env buildcache cd change checksum ci clean clone commands compiler compilers concretize config containerize create deactivate debug dependencies dependents deprecate dev-build develop diff docs edit env extensions external fetch find gc gpg graph help info install license list load location log-parse maintainers make-installer mark mirror module patch pkg providers pydoc python reindex remove rm repo resource restage solve spec stage style tags test test-env tutorial undevelop uninstall unit-test unload url verify versions view"
fi
}
_spack_activate() {
if $list_options
then
SPACK_COMPREPLY="-h --help -f --force -v --view"
else
_installed_packages
SPACK_COMPREPLY="add arch audit blame bootstrap build-env buildcache cd change checksum ci clean clone commands compiler compilers concretize config containerize create debug dependencies dependents deprecate dev-build develop diff docs edit env extensions external fetch find gc gpg graph help info install license list load location log-parse maintainers make-installer mark mirror module patch pkg providers pydoc python reindex remove rm repo resource restage solve spec stage style tags test test-env tutorial undevelop uninstall unit-test unload url verify versions view"
fi
}
@ -838,15 +829,6 @@ _spack_create() {
fi
}
_spack_deactivate() {
if $list_options
then
SPACK_COMPREPLY="-h --help -f --force -v --view -a --all"
else
_installed_packages
fi
}
_spack_debug() {
if $list_options
then
@ -1039,7 +1021,7 @@ _spack_env_depfile() {
_spack_extensions() {
if $list_options
then
SPACK_COMPREPLY="-h --help -l --long -L --very-long -d --deps -p --paths -s --show -v --view"
SPACK_COMPREPLY="-h --help -l --long -L --very-long -d --deps -p --paths -s --show"
else
_extensions
fi

22
var/spack/repos/builtin/packages/perl/package.py
View file

@ -482,28 +482,6 @@ def perl_ignore(self, ext_pkg, args):
return match_predicate(ignore_arg, patterns)
def activate(self, ext_pkg, view, **args):
ignore = self.perl_ignore(ext_pkg, args)
args.update(ignore=ignore)
super(Perl, self).activate(ext_pkg, view, **args)
extensions_layout = view.extensions_layout
exts = extensions_layout.extension_map(self.spec)
exts[ext_pkg.name] = ext_pkg.spec
def deactivate(self, ext_pkg, view, **args):
ignore = self.perl_ignore(ext_pkg, args)
args.update(ignore=ignore)
super(Perl, self).deactivate(ext_pkg, view, **args)
extensions_layout = view.extensions_layout
exts = extensions_layout.extension_map(self.spec)
# Make deactivate idempotent
if ext_pkg.name in exts:
del exts[ext_pkg.name]
@property
def command(self):
"""Returns the Perl command, which may vary depending on the version

108
var/spack/repos/builtin/packages/python/package.py
View file

@ -18,7 +18,7 @@
is_nonsymlink_exe_with_shebang,
path_contains_subdirectory,
)
from llnl.util.lang import dedupe, match_predicate
from llnl.util.lang import dedupe
from spack.build_environment import dso_suffix, stat_suffix
from spack.package import *
@ -1401,10 +1401,6 @@ def include(self):
return path.replace(prefix, "")
return os.path.join("include", "python{}".format(self.version.up_to(2)))
@property
def easy_install_file(self):
return join_path(self.purelib, "easy-install.pth")
def setup_run_environment(self, env):
env.prepend_path("CPATH", os.pathsep.join(self.spec["python"].headers.directories))
@ -1526,108 +1522,6 @@ def setup_dependent_package(self, module, dependent_spec):
mkdirp(module.python_platlib)
mkdirp(module.python_purelib)
# ========================================================================
# Handle specifics of activating and deactivating python modules.
# ========================================================================
def python_ignore(self, ext_pkg, args):
"""Add some ignore files to activate/deactivate args."""
ignore_arg = args.get("ignore", lambda f: False)
# Always ignore easy-install.pth, as it needs to be merged.
patterns = [r"(site|dist)-packages/easy-install\.pth$"]
# Ignore pieces of setuptools installed by other packages.
# Must include directory name or it will remove all site*.py files.
if ext_pkg.name != "py-setuptools":
patterns.extend(
[
r"bin/easy_install[^/]*$",
r"(site|dist)-packages/setuptools[^/]*\.egg$",
r"(site|dist)-packages/setuptools\.pth$",
r"(site|dist)-packages/site[^/]*\.pyc?$",
r"(site|dist)-packages/__pycache__/site[^/]*\.pyc?$",
]
)
if ext_pkg.name != "py-pygments":
patterns.append(r"bin/pygmentize$")
if ext_pkg.name != "py-numpy":
patterns.append(r"bin/f2py[0-9.]*$")
return match_predicate(ignore_arg, patterns)
def write_easy_install_pth(self, exts, prefix=None):
if not prefix:
prefix = self.prefix
paths = []
unique_paths = set()
for ext in sorted(exts.values()):
easy_pth = join_path(ext.prefix, self.easy_install_file)
if not os.path.isfile(easy_pth):
continue
with open(easy_pth) as f:
for line in f:
line = line.rstrip()
# Skip lines matching these criteria
if not line:
continue
if re.search(r"^(import|#)", line):
continue
if ext.name != "py-setuptools" and re.search(r"setuptools.*egg$", line):
continue
if line not in unique_paths:
unique_paths.add(line)
paths.append(line)
main_pth = join_path(prefix, self.easy_install_file)
if not paths:
if os.path.isfile(main_pth):
os.remove(main_pth)
else:
with open(main_pth, "w") as f:
f.write("import sys; sys.__plen = len(sys.path)\n")
for path in paths:
f.write("{0}\n".format(path))
f.write(
"import sys; new=sys.path[sys.__plen:]; "
"del sys.path[sys.__plen:]; "
"p=getattr(sys,'__egginsert',0); "
"sys.path[p:p]=new; "
"sys.__egginsert = p+len(new)\n"
)
def activate(self, ext_pkg, view, **args):
ignore = self.python_ignore(ext_pkg, args)
args.update(ignore=ignore)
super(Python, self).activate(ext_pkg, view, **args)
extensions_layout = view.extensions_layout
exts = extensions_layout.extension_map(self.spec)
exts[ext_pkg.name] = ext_pkg.spec
self.write_easy_install_pth(exts, prefix=view.get_projection_for_spec(self.spec))
def deactivate(self, ext_pkg, view, **args):
args.update(ignore=self.python_ignore(ext_pkg, args))
super(Python, self).deactivate(ext_pkg, view, **args)
extensions_layout = view.extensions_layout
exts = extensions_layout.extension_map(self.spec)
# Make deactivate idempotent
if ext_pkg.name in exts:
del exts[ext_pkg.name]
self.write_easy_install_pth(exts, prefix=view.get_projection_for_spec(self.spec))
def add_files_to_view(self, view, merge_map, skip_if_exists=True):
bin_dir = self.spec.prefix.bin if sys.platform != "win32" else self.spec.prefix
for src, dst in merge_map.items():