hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Support parallel environment builds (#18131)

Browse source 
As of #13100, Spack installs the dependencies of a _single_ spec in parallel.
Environments, when installed, can only get parallelism from each individual
spec, as they're installed in order.  This PR makes entire environments build
in parallel by extending Spack's package installer to accept multiple root
specs.  The install command and Environment class have been updated to use
the new parallel install method.

The specs and kwargs for each *uninstalled* package (when not force-replacing
installations) of an environment are collected, passed to the `PackageInstaller`,
and processed using a single build queue.

This introduces a `BuildRequest` class to track install arguments, and it
significantly cleans up the code used to track package ids during installation.
Package ids in the build queue are now just DAG hashes as you would expect,

Other tasks:

- [x] Finish updating the unit tests based on `PackageInstaller`'s use of
      `BuildRequest` and the associated changes
- [x] Change `environment.py`'s `install_all` to use the `PackageInstaller` directly
- [x] Change the `install` command to leverage the new installation process for multiple specs
- [x] Change install output messages for external packages, e.g.:
       `[+] /usr` -> `[+] /usr (external bzip2-1.0.8-<dag-hash>`
- [x] Fix incomplete environment install's view setup/update and not confirming all 
       packages are installed (?)
- [x] Ensure externally installed package dependencies are properly accounted for in 
       remaining build tasks
- [x] Add tests for coverage (if insufficient and can identity the appropriate, uncovered non-comment lines)
- [x] Add documentation
- [x] Resolve multi-compiler environment install issues
- [x] Fix issue with environment installation reporting (restore CDash/JUnit reports)
This commit is contained in:
Tamara Dahlgren 2020-11-17 02:41:07 -08:00 committed by GitHub
parent 423e80af23
commit 6fa6af1070
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
17 changed files with 1223 additions and 557 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

32
lib/spack/docs/basic_usage.rst
View file

@ -132,32 +132,28 @@ If ``mpileaks`` depends on other packages, Spack will install the
dependencies first. It then fetches the ``mpileaks`` tarball, expands dependencies first. It then fetches the ``mpileaks`` tarball, expands
it, verifies that it was downloaded without errors, builds it, and it, verifies that it was downloaded without errors, builds it, and
installs it in its own directory under ``$SPACK_ROOT/opt``. You'll see installs it in its own directory under ``$SPACK_ROOT/opt``. You'll see
a number of messages from spack, a lot of build output, and a message a number of messages from Spack, a lot of build output, and a message
that the packages is installed: that the package is installed. Add one or more debug options (``-d``)
to get increasingly detailed output.
.. code-block:: console .. code-block:: console
$ spack install mpileaks $ spack install mpileaks
==> Installing mpileaks ... dependency build output ...
==> mpich is already installed in ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpich@3.0.4. ==> Installing mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2
==> callpath is already installed in ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/callpath@1.0.2-5dce4318. ==> No binary for mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2 found: installing from source
==> adept-utils is already installed in ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/adept-utils@1.0-5adef8da. ==> mpileaks: Executing phase: 'autoreconf'
==> Trying to fetch from https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz ==> mpileaks: Executing phase: 'configure'
######################################################################## 100.0% ==> mpileaks: Executing phase: 'build'
==> Staging archive: ~/spack/var/spack/stage/mpileaks@1.0%gcc@4.4.7 arch=linux-debian7-x86_64-59f6ad23/mpileaks-1.0.tar.gz ==> mpileaks: Executing phase: 'install'
==> Created stage in ~/spack/var/spack/stage/mpileaks@1.0%gcc@4.4.7 arch=linux-debian7-x86_64-59f6ad23. [+] ~/spack/opt/linux-rhel7-broadwell/gcc-8.1.0/mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2
==> No patches needed for mpileaks.
==> Building mpileaks.
... build output ...
==> Successfully installed mpileaks.
Fetch: 2.16s. Build: 9.82s. Total: 11.98s.
[+] ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpileaks@1.0-59f6ad23
The last line, with the ``[+]``, indicates where the package is The last line, with the ``[+]``, indicates where the package is
installed. installed.
Add the debug option -- ``spack install -d mpileaks`` -- to get additional
output.
^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Building a specific version Building a specific version
^^^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^

47
lib/spack/docs/environments.rst
View file

@ -191,44 +191,24 @@ Environment has been activated. Similarly, the ``install`` and
==> 0 installed packages ==> 0 installed packages
$ spack install zlib@1.2.11 $ spack install zlib@1.2.11
==> Installing zlib ==> Installing zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv
==> Searching for binary cache of zlib ==> No binary for zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv found: installing from source
==> Warning: No Spack mirrors are currently configured ==> zlib: Executing phase: 'install'
==> No binary for zlib found: installing from source [+] ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv
==> Fetching http://zlib.net/fossils/zlib-1.2.11.tar.gz
######################################################################## 100.0%
==> Staging archive: /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur/zlib-1.2.11.tar.gz
==> Created stage in /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
==> No patches needed for zlib
==> Building zlib [Package]
==> Executing phase: 'install'
==> Successfully installed zlib
Fetch: 0.36s. Build: 11.58s. Total: 11.93s.
[+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
$ spack env activate myenv $ spack env activate myenv
$ spack find $ spack find
==> In environment myenv ==> In environment myenv
==> No root specs ==> No root specs
==> 0 installed packages ==> 0 installed packages
$ spack install zlib@1.2.8 $ spack install zlib@1.2.8
==> Installing zlib ==> Installing zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x
==> Searching for binary cache of zlib ==> No binary for zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x found: installing from source
==> Warning: No Spack mirrors are currently configured ==> zlib: Executing phase: 'install'
==> No binary for zlib found: installing from source [+] ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x
==> Fetching http://zlib.net/fossils/zlib-1.2.8.tar.gz ==> Updating view at ~/spack/var/spack/environments/myenv/.spack-env/view
######################################################################## 100.0%
==> Staging archive: /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7/zlib-1.2.8.tar.gz
==> Created stage in /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
==> No patches needed for zlib
==> Building zlib [Package]
==> Executing phase: 'install'
==> Successfully installed zlib
Fetch: 0.26s. Build: 2.08s. Total: 2.35s.
[+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
$ spack find $ spack find
==> In environment myenv ==> In environment myenv
@ -236,15 +216,17 @@ Environment has been activated. Similarly, the ``install`` and
zlib@1.2.8 zlib@1.2.8
==> 1 installed package ==> 1 installed package
-- linux-rhel7-x86_64 / gcc@4.9.3 ------------------------------- -- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
zlib@1.2.8 zlib@1.2.8
$ despacktivate $ despacktivate
$ spack find $ spack find
==> 2 installed packages ==> 2 installed packages
-- linux-rhel7-x86_64 / gcc@4.9.3 ------------------------------- -- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
zlib@1.2.8 zlib@1.2.11 zlib@1.2.8 zlib@1.2.11
Note that when we installed the abstract spec ``zlib@1.2.8``, it was Note that when we installed the abstract spec ``zlib@1.2.8``, it was
presented as a root of the Environment. All explicitly installed presented as a root of the Environment. All explicitly installed
packages will be listed as roots of the Environment. packages will be listed as roots of the Environment.
@ -349,6 +331,9 @@ installed specs using the ``-c`` (``--concretized``) flag.
==> 0 installed packages ==> 0 installed packages
.. _installing-environment:
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^
Installing an Environment Installing an Environment
^^^^^^^^^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^

49
lib/spack/docs/packaging_guide.rst
View file

@ -1778,8 +1778,18 @@ RPATHs in Spack are handled in one of three ways:
Parallel builds Parallel builds
--------------- ---------------
Spack supports parallel builds on an individual package and at the
installation level. Package-level parallelism is established by the
``--jobs`` option and its configuration and package recipe equivalents.
Installation-level parallelism is driven by the DAG(s) of the requested
package or packages.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Package-level build parallelism
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
By default, Spack will invoke ``make()``, or any other similar tool, By default, Spack will invoke ``make()``, or any other similar tool,
with a ``-j <njobs>`` argument, so that builds run in parallel. with a ``-j <njobs>`` argument, so those builds run in parallel.
The parallelism is determined by the value of the ``build_jobs`` entry The parallelism is determined by the value of the ``build_jobs`` entry
in ``config.yaml`` (see :ref:`here <build-jobs>` for more details on in ``config.yaml`` (see :ref:`here <build-jobs>` for more details on
how this value is computed). how this value is computed).
@ -1827,6 +1837,43 @@ you set ``parallel`` to ``False`` at the package level, then each call
to ``make()`` will be sequential by default, but packagers can call to ``make()`` will be sequential by default, but packagers can call
``make(parallel=True)`` to override it. ``make(parallel=True)`` to override it.
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Install-level build parallelism
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Spack supports the concurrent installation of packages within a Spack
instance across multiple processes using file system locks. This
parallelism is separate from the package-level achieved through build
systems' use of the ``-j <njobs>`` option. With install-level parallelism,
processes coordinate the installation of the dependencies of specs
provided on the command line and as part of an environment build with
only **one process** being allowed to install a given package at a time.
Refer to :ref:`Dependencies` for more information on dependencies and
:ref:`installing-environment` for how to install an environment.
Concurrent processes may be any combination of interactive sessions and
batch jobs. Which means a ``spack install`` can be running in a terminal
window while a batch job is running ``spack install`` on the same or
overlapping dependencies without any process trying to re-do the work of
another.
For example, if you are using SLURM, you could launch an installation
of ``mpich`` using the following command:
.. code-block:: console
$ srun -N 2 -n 8 spack install -j 4 mpich@3.3.2
This will create eight concurrent four-job installation on two different
nodes.
.. note::
The effective parallelism will be based on the maximum number of
packages that can be installed at the same time, which will limited
by the number of packages with no (remaining) uninstalled dependencies.
.. _dependencies: .. _dependencies:
------------ ------------

2
lib/spack/env/cc vendored
View file

@ -22,7 +22,7 @@
# This is an array of environment variables that need to be set before # This is an array of environment variables that need to be set before
# the script runs. They are set by routines in spack.build_environment # the script runs. They are set by routines in spack.build_environment
# as part of spack.package.Package.do_install(). # as part of the package installation process.
parameters=( parameters=(
SPACK_ENV_PATH SPACK_ENV_PATH
SPACK_DEBUG_LOG_DIR SPACK_DEBUG_LOG_DIR

61
lib/spack/spack/cmd/install.py
View file

@ -20,6 +20,7 @@
import spack.paths import spack.paths
import spack.report import spack.report
from spack.error import SpackError from spack.error import SpackError
from spack.installer import PackageInstaller
description = "build and install packages" description = "build and install packages"
@ -29,7 +30,7 @@
def update_kwargs_from_args(args, kwargs): def update_kwargs_from_args(args, kwargs):
"""Parse cli arguments and construct a dictionary """Parse cli arguments and construct a dictionary
that will be passed to Package.do_install API""" that will be passed to the package installer."""
kwargs.update({ kwargs.update({
'fail_fast': args.fail_fast, 'fail_fast': args.fail_fast,
@ -238,23 +239,29 @@ def default_log_file(spec):
return fs.os.path.join(dirname, basename) return fs.os.path.join(dirname, basename)
def install_spec(cli_args, kwargs, abstract_spec, spec): def install_specs(cli_args, kwargs, specs):
"""Do the actual installation.""" """Do the actual installation.
Args:
cli_args (Namespace): argparse namespace with command arguments
kwargs (dict): keyword arguments
specs (list of tuples): list of (abstract, concrete) spec tuples
"""
# handle active environment, if any
env = ev.get_env(cli_args, 'install')
try: try:
# handle active environment, if any
env = ev.get_env(cli_args, 'install')
if env: if env:
with env.write_transaction(): for abstract, concrete in specs:
concrete = env.concretize_and_add( with env.write_transaction():
abstract_spec, spec) concrete = env.concretize_and_add(abstract, concrete)
env.write(regenerate_views=False) env.write(regenerate_views=False)
env._install(concrete, **kwargs) env.install_all(cli_args, **kwargs)
with env.write_transaction():
env.regenerate_views()
else: else:
spec.package.do_install(**kwargs) installs = [(concrete.package, kwargs) for _, concrete in specs]
builder = PackageInstaller(installs)
builder.install()
except spack.build_environment.InstallError as e: except spack.build_environment.InstallError as e:
if cli_args.show_log_on_error: if cli_args.show_log_on_error:
e.print_context() e.print_context()
@ -280,6 +287,10 @@ def install(parser, args, **kwargs):
parser.print_help() parser.print_help()
return return
reporter = spack.report.collect_info(args.log_format, args)
if args.log_file:
reporter.filename = args.log_file
if not args.spec and not args.specfiles: if not args.spec and not args.specfiles:
# if there are no args but an active environment # if there are no args but an active environment
# then install the packages from it. # then install the packages from it.
@ -294,8 +305,17 @@ def install(parser, args, **kwargs):
# once, as it can be slow. # once, as it can be slow.
env.write(regenerate_views=False) env.write(regenerate_views=False)
tty.msg("Installing environment %s" % env.name) specs = env.all_specs()
env.install_all(args) if not args.log_file and not reporter.filename:
reporter.filename = default_log_file(specs[0])
reporter.specs = specs
tty.msg("Installing environment {0}".format(env.name))
with reporter:
env.install_all(args, **kwargs)
tty.debug("Regenerating environment views for {0}"
.format(env.name))
with env.write_transaction(): with env.write_transaction():
# It is not strictly required to synchronize view regeneration # It is not strictly required to synchronize view regeneration
# but doing so can prevent redundant work in the filesystem. # but doing so can prevent redundant work in the filesystem.
@ -319,17 +339,13 @@ def install(parser, args, **kwargs):
spack.config.set('config:checksum', False, scope='command_line') spack.config.set('config:checksum', False, scope='command_line')
# Parse cli arguments and construct a dictionary # Parse cli arguments and construct a dictionary
# that will be passed to Package.do_install API # that will be passed to the package installer
update_kwargs_from_args(args, kwargs) update_kwargs_from_args(args, kwargs)
if args.run_tests: if args.run_tests:
tty.warn("Deprecated option: --run-tests: use --test=all instead") tty.warn("Deprecated option: --run-tests: use --test=all instead")
# 1. Abstract specs from cli # 1. Abstract specs from cli
reporter = spack.report.collect_info(args.log_format, args)
if args.log_file:
reporter.filename = args.log_file
abstract_specs = spack.cmd.parse_specs(args.spec) abstract_specs = spack.cmd.parse_specs(args.spec)
tests = False tests = False
if args.test == 'all' or args.run_tests: if args.test == 'all' or args.run_tests:
@ -401,5 +417,4 @@ def install(parser, args, **kwargs):
# overwrite all concrete explicit specs from this build # overwrite all concrete explicit specs from this build
kwargs['overwrite'] = [spec.dag_hash() for spec in specs] kwargs['overwrite'] = [spec.dag_hash() for spec in specs]
for abstract, concrete in zip(abstract_specs, specs): install_specs(args, kwargs, zip(abstract_specs, specs))
install_spec(args, kwargs, abstract, concrete)

60
lib/spack/spack/environment.py
View file

@ -39,6 +39,7 @@
from spack.variant import UnknownVariantError from spack.variant import UnknownVariantError
import spack.util.lock as lk import spack.util.lock as lk
from spack.util.path import substitute_path_variables from spack.util.path import substitute_path_variables
from spack.installer import PackageInstaller
#: environment variable used to indicate the active environment #: environment variable used to indicate the active environment
spack_env_var = 'SPACK_ENV' spack_env_var = 'SPACK_ENV'
@ -1371,24 +1372,7 @@ def _get_overwrite_specs(self):
return ret return ret
def install(self, user_spec, concrete_spec=None, **install_args): def _install_log_links(self, spec):
"""Install a single spec into an environment.
This will automatically concretize the single spec, but it won't
affect other as-yet unconcretized specs.
"""
concrete = self.concretize_and_add(user_spec, concrete_spec)
self._install(concrete, **install_args)
def _install(self, spec, **install_args):
# "spec" must be concrete
package = spec.package
install_args['overwrite'] = install_args.get(
'overwrite', []) + self._get_overwrite_specs()
package.do_install(**install_args)
if not spec.external: if not spec.external:
# Make sure log directory exists # Make sure log directory exists
log_path = self.log_path log_path = self.log_path
@ -1402,19 +1386,22 @@ def _install(self, spec, **install_args):
os.remove(build_log_link) os.remove(build_log_link)
os.symlink(spec.package.build_log_path, build_log_link) os.symlink(spec.package.build_log_path, build_log_link)
def install_all(self, args=None): def install_all(self, args=None, **install_args):
"""Install all concretized specs in an environment. """Install all concretized specs in an environment.
Note: this does not regenerate the views for the environment; Note: this does not regenerate the views for the environment;
that needs to be done separately with a call to write(). that needs to be done separately with a call to write().
Args:
args (Namespace): argparse namespace with command arguments
install_args (dict): keyword install arguments
""" """
# If "spack install" is invoked repeatedly for a large environment # If "spack install" is invoked repeatedly for a large environment
# where all specs are already installed, the operation can take # where all specs are already installed, the operation can take
# a large amount of time due to repeatedly acquiring and releasing # a large amount of time due to repeatedly acquiring and releasing
# locks, this does an initial check across all specs within a single # locks, this does an initial check across all specs within a single
# DB read transaction to reduce time spent in this case. # DB read transaction to reduce time spent in this case.
tty.debug('Assessing installation status of environment packages')
specs_to_install = [] specs_to_install = []
with spack.store.db.read_transaction(): with spack.store.db.read_transaction():
for concretized_hash in self.concretized_order: for concretized_hash in self.concretized_order:
@ -1426,14 +1413,43 @@ def install_all(self, args=None):
# If it's a dev build it could need to be reinstalled # If it's a dev build it could need to be reinstalled
specs_to_install.append(spec) specs_to_install.append(spec)
if not specs_to_install:
tty.msg('All of the packages are already installed')
return
tty.debug('Processing {0} uninstalled specs'
.format(len(specs_to_install)))
install_args['overwrite'] = install_args.get(
'overwrite', []) + self._get_overwrite_specs()
installs = []
for spec in specs_to_install: for spec in specs_to_install:
# Parse cli arguments and construct a dictionary # Parse cli arguments and construct a dictionary
# that will be passed to Package.do_install API # that will be passed to the package installer
kwargs = dict() kwargs = dict()
if install_args:
kwargs.update(install_args)
if args: if args:
spack.cmd.install.update_kwargs_from_args(args, kwargs) spack.cmd.install.update_kwargs_from_args(args, kwargs)
self._install(spec, **kwargs) installs.append((spec.package, kwargs))
try:
builder = PackageInstaller(installs)
builder.install()
finally:
# Ensure links are set appropriately
for spec in specs_to_install:
if spec.package.installed:
try:
self._install_log_links(spec)
except OSError as e:
tty.warn('Could not install log links for {0}: {1}'
.format(spec.name, str(e)))
with self.write_transaction():
self.regenerate_views()
def all_specs(self): def all_specs(self):
"""Return all specs, even those a user spec would shadow.""" """Return all specs, even those a user spec would shadow."""

802
lib/spack/spack/installer.py
View file

File diff suppressed because it is too large Load diff

41
lib/spack/spack/package.py
View file

@ -53,8 +53,7 @@
from six import string_types from six import string_types
from six import with_metaclass from six import with_metaclass
from spack.filesystem_view import YamlFilesystemView from spack.filesystem_view import YamlFilesystemView
from spack.installer import \ from spack.installer import PackageInstaller, InstallError
install_args_docstring, PackageInstaller, InstallError
from spack.stage import stage_prefix, Stage, ResourceStage, StageComposite from spack.stage import stage_prefix, Stage, ResourceStage, StageComposite
from spack.util.package_hash import package_hash from spack.util.package_hash import package_hash
from spack.version import Version from spack.version import Version
@ -1591,17 +1590,45 @@ def do_install(self, **kwargs):
Package implementations should override install() to describe Package implementations should override install() to describe
their build process. their build process.
Args:""" Args:
cache_only (bool): Fail if binary package unavailable.
dirty (bool): Don't clean the build environment before installing.
explicit (bool): True if package was explicitly installed, False
if package was implicitly installed (as a dependency).
fail_fast (bool): Fail if any dependency fails to install;
otherwise, the default is to install as many dependencies as
possible (i.e., best effort installation).
fake (bool): Don't really build; install fake stub files instead.
force (bool): Install again, even if already installed.
install_deps (bool): Install dependencies before installing this
package
install_source (bool): By default, source is not installed, but
for debugging it might be useful to keep it around.
keep_prefix (bool): Keep install prefix on failure. By default,
destroys it.
keep_stage (bool): By default, stage is destroyed only if there
are no exceptions during build. Set to True to keep the stage
even with exceptions.
restage (bool): Force spack to restage the package source.
skip_patch (bool): Skip patch stage of build if True.
stop_before (InstallPhase): stop execution before this
installation phase (or None)
stop_at (InstallPhase): last installation phase to be executed
(or None)
tests (bool or list or set): False to run no tests, True to test
all packages, or a list of package names to run tests for some
use_cache (bool): Install from binary package, if available.
verbose (bool): Display verbose build output (by default,
suppresses it)
"""
# Non-transitive dev specs need to keep the dev stage and be built from # Non-transitive dev specs need to keep the dev stage and be built from
# source every time. Transitive ones just need to be built from source. # source every time. Transitive ones just need to be built from source.
dev_path_var = self.spec.variants.get('dev_path', None) dev_path_var = self.spec.variants.get('dev_path', None)
if dev_path_var: if dev_path_var:
kwargs['keep_stage'] = True kwargs['keep_stage'] = True
builder = PackageInstaller(self) builder = PackageInstaller([(self, kwargs)])
builder.install(**kwargs) builder.install()
do_install.__doc__ += install_args_docstring
def unit_test_check(self): def unit_test_check(self):
"""Hook for unit tests to assert things about package internals. """Hook for unit tests to assert things about package internals.

4
lib/spack/spack/report.py
View file

@ -44,8 +44,8 @@ def fetch_package_log(pkg):
class InfoCollector(object): class InfoCollector(object):
"""Decorates PackageInstaller._install_task, which is called by """Decorates PackageInstaller._install_task, which is called via
PackageBase.do_install for each spec, to collect information PackageBase.do_install for individual specs, to collect information
on the installation of certain specs. on the installation of certain specs.
When exiting the context this change will be rolled-back. When exiting the context this change will be rolled-back.

55
lib/spack/spack/test/buildrequest.py Normal file
View file

@ -0,0 +1,55 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
import pytest
import spack.installer as inst
import spack.repo
import spack.spec
def test_build_request_errors(install_mockery):
with pytest.raises(ValueError, match='must be a package'):
inst.BuildRequest('abc', {})
pkg = spack.repo.get('trivial-install-test-package')
with pytest.raises(ValueError, match='must have a concrete spec'):
inst.BuildRequest(pkg, {})
def test_build_request_basics(install_mockery):
spec = spack.spec.Spec('dependent-install')
spec.concretize()
assert spec.concrete
# Ensure key properties match expectations
request = inst.BuildRequest(spec.package, {})
assert not request.pkg.stop_before_phase
assert not request.pkg.last_phase
assert request.spec == spec.package.spec
# Ensure key default install arguments are set
assert 'install_package' in request.install_args
assert 'install_deps' in request.install_args
def test_build_request_strings(install_mockery):
"""Tests of BuildRequest repr and str for coverage purposes."""
# Using a package with one dependency
spec = spack.spec.Spec('dependent-install')
spec.concretize()
assert spec.concrete
# Ensure key properties match expectations
request = inst.BuildRequest(spec.package, {})
# Cover __repr__
irep = request.__repr__()
assert irep.startswith(request.__class__.__name__)
# Cover __str__
istr = str(request)
assert "package=dependent-install" in istr
assert "install_args=" in istr

20
lib/spack/spack/test/buildtask.py
View file

@ -12,17 +12,22 @@
def test_build_task_errors(install_mockery): def test_build_task_errors(install_mockery):
with pytest.raises(ValueError, match='must be a package'): with pytest.raises(ValueError, match='must be a package'):
inst.BuildTask('abc', False, 0, 0, 0, []) inst.BuildTask('abc', None, False, 0, 0, 0, [])
pkg = spack.repo.get('trivial-install-test-package') pkg = spack.repo.get('trivial-install-test-package')
with pytest.raises(ValueError, match='must have a concrete spec'): with pytest.raises(ValueError, match='must have a concrete spec'):
inst.BuildTask(pkg, False, 0, 0, 0, []) inst.BuildTask(pkg, None, False, 0, 0, 0, [])
spec = spack.spec.Spec('trivial-install-test-package') spec = spack.spec.Spec('trivial-install-test-package')
spec.concretize() spec.concretize()
assert spec.concrete assert spec.concrete
with pytest.raises(ValueError, match='must have a build request'):
inst.BuildTask(spec.package, None, False, 0, 0, 0, [])
request = inst.BuildRequest(spec.package, {})
with pytest.raises(inst.InstallError, match='Cannot create a build task'): with pytest.raises(inst.InstallError, match='Cannot create a build task'):
inst.BuildTask(spec.package, False, 0, 0, inst.STATUS_REMOVED, []) inst.BuildTask(spec.package, request, False, 0, 0, inst.STATUS_REMOVED,
[])
def test_build_task_basics(install_mockery): def test_build_task_basics(install_mockery):
@ -31,7 +36,10 @@ def test_build_task_basics(install_mockery):
assert spec.concrete assert spec.concrete
# Ensure key properties match expectations # Ensure key properties match expectations
task = inst.BuildTask(spec.package, False, 0, 0, inst.STATUS_ADDED, []) request = inst.BuildRequest(spec.package, {})
task = inst.BuildTask(spec.package, request, False, 0, 0,
inst.STATUS_ADDED, [])
assert task.explicit # package was "explicitly" requested
assert task.priority == len(task.uninstalled_deps) assert task.priority == len(task.uninstalled_deps)
assert task.key == (task.priority, task.sequence) assert task.key == (task.priority, task.sequence)
@ -51,7 +59,9 @@ def test_build_task_strings(install_mockery):
assert spec.concrete assert spec.concrete
# Ensure key properties match expectations # Ensure key properties match expectations
task = inst.BuildTask(spec.package, False, 0, 0, inst.STATUS_ADDED, []) request = inst.BuildRequest(spec.package, {})
task = inst.BuildTask(spec.package, request, False, 0, 0,
inst.STATUS_ADDED, [])
# Cover __repr__ # Cover __repr__
irep = task.__repr__() irep = task.__repr__()

16
lib/spack/spack/test/cmd/dev_build.py
View file

@ -8,7 +8,7 @@
import spack.spec import spack.spec
import llnl.util.filesystem as fs import llnl.util.filesystem as fs
import spack.environment as ev import spack.environment as ev
from spack.main import SpackCommand, SpackCommandError from spack.main import SpackCommand
dev_build = SpackCommand('dev-build') dev_build = SpackCommand('dev-build')
install = SpackCommand('install') install = SpackCommand('install')
@ -99,13 +99,15 @@ def test_dev_build_before_until(tmpdir, mock_packages, install_mockery):
dev_build('-u', 'edit', '-b', 'edit', dev_build('-u', 'edit', '-b', 'edit',
'dev-build-test-install@0.0.0') 'dev-build-test-install@0.0.0')
with pytest.raises(SpackCommandError): bad_phase = 'phase_that_does_not_exist'
dev_build('-u', 'phase_that_does_not_exist', not_allowed = 'is not a valid phase'
'dev-build-test-install@0.0.0') out = dev_build('-u', bad_phase, 'dev-build-test-install@0.0.0')
assert bad_phase in out
assert not_allowed in out
with pytest.raises(SpackCommandError): out = dev_build('-b', bad_phase, 'dev-build-test-install@0.0.0')
dev_build('-b', 'phase_that_does_not_exist', assert bad_phase in out
'dev-build-test-install@0.0.0') assert not_allowed in out
def print_spack_cc(*args): def print_spack_cc(*args):

55
lib/spack/spack/test/cmd/env.py
View file

@ -185,17 +185,53 @@ def setup_error(pkg, env):
assert "Warning: couldn't get environment settings" in err assert "Warning: couldn't get environment settings" in err
def test_env_install_same_spec_twice(install_mockery, mock_fetch, capfd): def test_env_install_same_spec_twice(install_mockery, mock_fetch):
env('create', 'test') env('create', 'test')
e = ev.read('test') e = ev.read('test')
with capfd.disabled(): with e:
with e: # The first installation outputs the package prefix, updates the view
# The first installation outputs the package prefix out = install('cmake-client')
install('cmake-client') assert 'Updating view at' in out
# The second installation attempt will also update the view
out = install('cmake-client') # The second installation reports all packages already installed
assert 'Updating view at' in out out = install('cmake-client')
assert 'already installed' in out
def test_env_install_two_specs_same_dep(
install_mockery, mock_fetch, tmpdir, capsys):
"""Test installation of two packages that share a dependency with no
connection and the second specifying the dependency as a 'build'
dependency.
"""
path = tmpdir.join('spack.yaml')
with tmpdir.as_cwd():
with open(str(path), 'w') as f:
f.write("""\
env:
specs:
- a
- depb
""")
env('create', 'test', 'spack.yaml')
with ev.read('test'):
with capsys.disabled():
out = install()
# Ensure both packages reach install phase processing and are installed
out = str(out)
assert 'depb: Executing phase:' in out
assert 'a: Executing phase:' in out
depb = spack.repo.path.get_pkg_class('depb')
assert depb.installed, 'Expected depb to be installed'
a = spack.repo.path.get_pkg_class('a')
assert a.installed, 'Expected a to be installed'
def test_remove_after_concretize(): def test_remove_after_concretize():
@ -2094,7 +2130,8 @@ def test_cant_install_single_spec_when_concretizing_together():
e.concretization = 'together' e.concretization = 'together'
with pytest.raises(ev.SpackEnvironmentError, match=r'cannot install'): with pytest.raises(ev.SpackEnvironmentError, match=r'cannot install'):
e.install('zlib') e.concretize_and_add('zlib')
e.install_all()
def test_duplicate_packages_raise_when_concretizing_together(): def test_duplicate_packages_raise_when_concretizing_together():

33
lib/spack/spack/test/cmd/install.py
View file

@ -38,7 +38,7 @@
def noop_install(monkeypatch): def noop_install(monkeypatch):
def noop(*args, **kwargs): def noop(*args, **kwargs):
pass pass
monkeypatch.setattr(spack.package.PackageBase, 'do_install', noop) monkeypatch.setattr(spack.installer.PackageInstaller, 'install', noop)
def test_install_package_and_dependency( def test_install_package_and_dependency(
@ -321,15 +321,19 @@ def test_install_from_file(spec, concretize, error_code, tmpdir):
with specfile.open('w') as f: with specfile.open('w') as f:
spec.to_yaml(f) spec.to_yaml(f)
err_msg = 'does not contain a concrete spec' if error_code else ''
# Relative path to specfile (regression for #6906) # Relative path to specfile (regression for #6906)
with fs.working_dir(specfile.dirname): with fs.working_dir(specfile.dirname):
# A non-concrete spec will fail to be installed # A non-concrete spec will fail to be installed
install('-f', specfile.basename, fail_on_error=False) out = install('-f', specfile.basename, fail_on_error=False)
assert install.returncode == error_code assert install.returncode == error_code
assert err_msg in out
# Absolute path to specfile (regression for #6983) # Absolute path to specfile (regression for #6983)
install('-f', str(specfile), fail_on_error=False) out = install('-f', str(specfile), fail_on_error=False)
assert install.returncode == error_code assert install.returncode == error_code
assert err_msg in out
@pytest.mark.disable_clean_stage_check @pytest.mark.disable_clean_stage_check
@ -615,12 +619,14 @@ def test_build_warning_output(tmpdir, mock_fetch, install_mockery, capfd):
assert 'foo.c:89: warning: some weird warning!' in msg assert 'foo.c:89: warning: some weird warning!' in msg
def test_cache_only_fails(tmpdir, mock_fetch, install_mockery): def test_cache_only_fails(tmpdir, mock_fetch, install_mockery, capfd):
# libelf from cache fails to install, which automatically removes the # libelf from cache fails to install, which automatically removes the
# the libdwarf build task and flags the package as failed to install. # the libdwarf build task
err_msg = 'Installation of libdwarf failed' with capfd.disabled():
with pytest.raises(spack.installer.InstallError, match=err_msg): out = install('--cache-only', 'libdwarf')
install('--cache-only', 'libdwarf')
assert 'Failed to install libelf' in out
assert 'Skipping build of libdwarf' in out
# Check that failure prefix locks are still cached # Check that failure prefix locks are still cached
failure_lock_prefixes = ','.join(spack.store.db._prefix_failures.keys()) failure_lock_prefixes = ','.join(spack.store.db._prefix_failures.keys())
@ -828,6 +834,7 @@ def test_cache_install_full_hash_match(
mirror_url = 'file://{0}'.format(mirror_dir.strpath) mirror_url = 'file://{0}'.format(mirror_dir.strpath)
s = Spec('libdwarf').concretized() s = Spec('libdwarf').concretized()
package_id = spack.installer.package_id(s.package)
# Install a package # Install a package
install(s.name) install(s.name)
@ -843,9 +850,9 @@ def test_cache_install_full_hash_match(
# Make sure we get the binary version by default # Make sure we get the binary version by default
install_output = install('--no-check-signature', s.name, output=str) install_output = install('--no-check-signature', s.name, output=str)
expect_msg = 'Extracting {0} from binary cache'.format(s.name) expect_extract_msg = 'Extracting {0} from binary cache'.format(package_id)
assert expect_msg in install_output assert expect_extract_msg in install_output
uninstall('-y', s.name) uninstall('-y', s.name)
@ -855,9 +862,7 @@ def test_cache_install_full_hash_match(
# Check that even if the full hash changes, we install from binary when # Check that even if the full hash changes, we install from binary when
# we don't explicitly require the full hash to match # we don't explicitly require the full hash to match
install_output = install('--no-check-signature', s.name, output=str) install_output = install('--no-check-signature', s.name, output=str)
expect_msg = 'Extracting {0} from binary cache'.format(s.name) assert expect_extract_msg in install_output
assert expect_msg in install_output
uninstall('-y', s.name) uninstall('-y', s.name)
@ -865,7 +870,7 @@ def test_cache_install_full_hash_match(
# installs from source. # installs from source.
install_output = install('--require-full-hash-match', s.name, output=str) install_output = install('--require-full-hash-match', s.name, output=str)
expect_msg = 'No binary for {0} found: installing from source'.format( expect_msg = 'No binary for {0} found: installing from source'.format(
s.name) package_id)
assert expect_msg in install_output assert expect_msg in install_output

2
lib/spack/spack/test/install.py
View file

@ -507,7 +507,7 @@ def test_unconcretized_install(install_mockery, mock_fetch, mock_packages):
"""Test attempts to perform install phases with unconcretized spec.""" """Test attempts to perform install phases with unconcretized spec."""
spec = Spec('trivial-install-test-package') spec = Spec('trivial-install-test-package')
with pytest.raises(ValueError, match="only install concrete packages"): with pytest.raises(ValueError, match='must have a concrete spec'):
spec.package.do_install() spec.package.do_install()
with pytest.raises(ValueError, match="only patch concrete packages"): with pytest.raises(ValueError, match="only patch concrete packages"):

479
lib/spack/spack/test/installer.py
View file

@ -59,34 +59,52 @@ def _true(*args, **kwargs):
return True return True
def create_build_task(pkg): def create_build_task(pkg, install_args={}):
""" """
Create a built task for the given (concretized) package Create a built task for the given (concretized) package
Args: Args:
pkg (PackageBase): concretized package associated with the task pkg (PackageBase): concretized package associated with the task
install_args (dict): dictionary of kwargs (or install args)
Return: Return:
(BuildTask) A basic package build task (BuildTask) A basic package build task
""" """
return inst.BuildTask(pkg, False, 0, 0, inst.STATUS_ADDED, []) request = inst.BuildRequest(pkg, install_args)
return inst.BuildTask(pkg, request, False, 0, 0, inst.STATUS_ADDED, [])
def create_installer(spec_name): def create_installer(installer_args):
""" """
Create an installer for the named spec Create an installer using the concretized spec for each arg
Args: Args:
spec_name (str): Name of the explicit install spec installer_args (list of tuples): the list of (spec name, kwargs) tuples
Return: Return:
spec (Spec): concretized spec
installer (PackageInstaller): the associated package installer installer (PackageInstaller): the associated package installer
""" """
spec = spack.spec.Spec(spec_name) const_arg = [(spec.package, kwargs) for spec, kwargs in installer_args]
spec.concretize() return inst.PackageInstaller(const_arg)
assert spec.concrete
return spec, inst.PackageInstaller(spec.package)
def installer_args(spec_names, kwargs={}):
"""Return a the installer argument with each spec paired with kwargs
Args:
spec_names (list of str): list of spec names
kwargs (dict or None): install arguments to apply to all of the specs
Returns:
list of (spec, kwargs): the installer constructor argument
"""
arg = []
for name in spec_names:
spec = spack.spec.Spec(name)
spec.concretize()
assert spec.concrete
arg.append((spec, kwargs))
return arg
@pytest.mark.parametrize('sec,result', [ @pytest.mark.parametrize('sec,result', [
@ -99,6 +117,21 @@ def test_hms(sec, result):
assert inst._hms(sec) == result assert inst._hms(sec) == result
def test_get_dependent_ids(install_mockery, mock_packages):
# Concretize the parent package, which handle dependency too
spec = spack.spec.Spec('a')
spec.concretize()
assert spec.concrete
pkg_id = inst.package_id(spec.package)
# Grab the sole dependency of 'a', which is 'b'
dep = spec.dependencies()[0]
# Ensure the parent package is a dependent of the dependency package
assert pkg_id in inst.get_dependent_ids(dep)
def test_install_msg(monkeypatch): def test_install_msg(monkeypatch):
"""Test results of call to install_msg based on debug level.""" """Test results of call to install_msg based on debug level."""
name = 'some-package' name = 'some-package'
@ -190,7 +223,9 @@ def _spec(spec, preferred_mirrors=None):
spec = spack.spec.Spec('a').concretized() spec = spack.spec.Spec('a').concretized()
assert inst._process_binary_cache_tarball(spec.package, spec, False, False) assert inst._process_binary_cache_tarball(spec.package, spec, False, False)
assert 'Extracting a from binary cache' in capfd.readouterr()[0] out = capfd.readouterr()[0]
assert 'Extracting a' in out
assert 'from binary cache' in out
def test_try_install_from_binary_cache(install_mockery, mock_packages, def test_try_install_from_binary_cache(install_mockery, mock_packages,
@ -216,18 +251,9 @@ def _mirrors_for_spec(spec, force, full_hash_match=False):
assert 'add a spack mirror to allow download' in str(captured) assert 'add a spack mirror to allow download' in str(captured)
def test_installer_init_errors(install_mockery):
"""Test to ensure cover installer constructor errors."""
with pytest.raises(ValueError, match='must be a package'):
inst.PackageInstaller('abc')
pkg = spack.repo.get('trivial-install-test-package')
with pytest.raises(ValueError, match='Can only install concrete'):
inst.PackageInstaller(pkg)
def test_installer_repr(install_mockery): def test_installer_repr(install_mockery):
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
installer = create_installer(const_arg)
irep = installer.__repr__() irep = installer.__repr__()
assert irep.startswith(installer.__class__.__name__) assert irep.startswith(installer.__class__.__name__)
@ -236,7 +262,8 @@ def test_installer_repr(install_mockery):
def test_installer_str(install_mockery): def test_installer_str(install_mockery):
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
installer = create_installer(const_arg)
istr = str(installer) istr = str(installer)
assert "#tasks=0" in istr assert "#tasks=0" in istr
@ -244,20 +271,33 @@ def test_installer_str(install_mockery):
assert "failed (0)" in istr assert "failed (0)" in istr
def test_installer_last_phase_error(install_mockery, capsys): def test_check_before_phase_error(install_mockery):
spec = spack.spec.Spec('trivial-install-test-package') pkg = spack.repo.get('trivial-install-test-package')
spec.concretize() pkg.stop_before_phase = 'beforephase'
assert spec.concrete with pytest.raises(inst.BadInstallPhase) as exc_info:
with pytest.raises(SystemExit): inst._check_last_phase(pkg)
installer = inst.PackageInstaller(spec.package)
installer.install(stop_at='badphase')
captured = capsys.readouterr() err = str(exc_info.value)
assert 'is not an allowed phase' in str(captured) assert 'is not a valid phase' in err
assert pkg.stop_before_phase in err
def test_check_last_phase_error(install_mockery):
pkg = spack.repo.get('trivial-install-test-package')
pkg.stop_before_phase = None
pkg.last_phase = 'badphase'
with pytest.raises(inst.BadInstallPhase) as exc_info:
inst._check_last_phase(pkg)
err = str(exc_info.value)
assert 'is not a valid phase' in err
assert pkg.last_phase in err
def test_installer_ensure_ready_errors(install_mockery): def test_installer_ensure_ready_errors(install_mockery):
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
installer = create_installer(const_arg)
spec = installer.build_requests[0].pkg.spec
fmt = r'cannot be installed locally.*{0}' fmt = r'cannot be installed locally.*{0}'
# Force an external package error # Force an external package error
@ -290,7 +330,9 @@ def test_ensure_locked_err(install_mockery, monkeypatch, tmpdir, capsys):
def _raise(lock, timeout): def _raise(lock, timeout):
raise RuntimeError(mock_err_msg) raise RuntimeError(mock_err_msg)
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
installer = create_installer(const_arg)
spec = installer.build_requests[0].pkg.spec
monkeypatch.setattr(ulk.Lock, 'acquire_read', _raise) monkeypatch.setattr(ulk.Lock, 'acquire_read', _raise)
with tmpdir.as_cwd(): with tmpdir.as_cwd():
@ -304,14 +346,17 @@ def _raise(lock, timeout):
def test_ensure_locked_have(install_mockery, tmpdir, capsys): def test_ensure_locked_have(install_mockery, tmpdir, capsys):
"""Test _ensure_locked when already have lock.""" """Test _ensure_locked when already have lock."""
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
installer = create_installer(const_arg)
spec = installer.build_requests[0].pkg.spec
pkg_id = inst.package_id(spec.package)
with tmpdir.as_cwd(): with tmpdir.as_cwd():
# Test "downgrade" of a read lock (to a read lock) # Test "downgrade" of a read lock (to a read lock)
lock = lk.Lock('./test', default_timeout=1e-9, desc='test') lock = lk.Lock('./test', default_timeout=1e-9, desc='test')
lock_type = 'read' lock_type = 'read'
tpl = (lock_type, lock) tpl = (lock_type, lock)
installer.locks[installer.pkg_id] = tpl installer.locks[pkg_id] = tpl
assert installer._ensure_locked(lock_type, spec.package) == tpl assert installer._ensure_locked(lock_type, spec.package) == tpl
# Test "upgrade" of a read lock without read count to a write # Test "upgrade" of a read lock without read count to a write
@ -341,7 +386,9 @@ def test_ensure_locked_have(install_mockery, tmpdir, capsys):
def test_ensure_locked_new_lock( def test_ensure_locked_new_lock(
install_mockery, tmpdir, lock_type, reads, writes): install_mockery, tmpdir, lock_type, reads, writes):
pkg_id = 'a' pkg_id = 'a'
spec, installer = create_installer(pkg_id) const_arg = installer_args([pkg_id], {})
installer = create_installer(const_arg)
spec = installer.build_requests[0].pkg.spec
with tmpdir.as_cwd(): with tmpdir.as_cwd():
ltype, lock = installer._ensure_locked(lock_type, spec.package) ltype, lock = installer._ensure_locked(lock_type, spec.package)
assert ltype == lock_type assert ltype == lock_type
@ -359,7 +406,9 @@ def _pl(db, spec, timeout):
return lock return lock
pkg_id = 'a' pkg_id = 'a'
spec, installer = create_installer(pkg_id) const_arg = installer_args([pkg_id], {})
installer = create_installer(const_arg)
spec = installer.build_requests[0].pkg.spec
monkeypatch.setattr(spack.database.Database, 'prefix_lock', _pl) monkeypatch.setattr(spack.database.Database, 'prefix_lock', _pl)
@ -529,52 +578,65 @@ def _raise_except(path):
def test_check_deps_status_install_failure(install_mockery, monkeypatch): def test_check_deps_status_install_failure(install_mockery, monkeypatch):
spec, installer = create_installer('a') const_arg = installer_args(['a'], {})
installer = create_installer(const_arg)
request = installer.build_requests[0]
# Make sure the package is identified as failed # Make sure the package is identified as failed
monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true) monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
with pytest.raises(inst.InstallError, match='install failure'): with pytest.raises(inst.InstallError, match='install failure'):
installer._check_deps_status() installer._check_deps_status(request)
def test_check_deps_status_write_locked(install_mockery, monkeypatch): def test_check_deps_status_write_locked(install_mockery, monkeypatch):
spec, installer = create_installer('a') const_arg = installer_args(['a'], {})
installer = create_installer(const_arg)
request = installer.build_requests[0]
# Ensure the lock is not acquired # Ensure the lock is not acquired
monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', _not_locked) monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', _not_locked)
with pytest.raises(inst.InstallError, match='write locked by another'): with pytest.raises(inst.InstallError, match='write locked by another'):
installer._check_deps_status() installer._check_deps_status(request)
def test_check_deps_status_external(install_mockery, monkeypatch): def test_check_deps_status_external(install_mockery, monkeypatch):
spec, installer = create_installer('a') const_arg = installer_args(['a'], {})
installer = create_installer(const_arg)
request = installer.build_requests[0]
# Mock the known dependent, b, as external so assumed to be installed # Mock the known dependent, b, as external so assumed to be installed
monkeypatch.setattr(spack.spec.Spec, 'external', True) monkeypatch.setattr(spack.spec.Spec, 'external', True)
installer._check_deps_status() installer._check_deps_status(request)
assert 'b' in installer.installed assert list(installer.installed)[0].startswith('b')
def test_check_deps_status_upstream(install_mockery, monkeypatch): def test_check_deps_status_upstream(install_mockery, monkeypatch):
spec, installer = create_installer('a') const_arg = installer_args(['a'], {})
installer = create_installer(const_arg)
request = installer.build_requests[0]
# Mock the known dependent, b, as installed upstream # Mock the known dependent, b, as installed upstream
monkeypatch.setattr(spack.package.PackageBase, 'installed_upstream', True) monkeypatch.setattr(spack.package.PackageBase, 'installed_upstream', True)
installer._check_deps_status() installer._check_deps_status(request)
assert 'b' in installer.installed assert list(installer.installed)[0].startswith('b')
def test_add_bootstrap_compilers(install_mockery, monkeypatch): def test_add_bootstrap_compilers(install_mockery, monkeypatch):
from collections import defaultdict
def _pkgs(pkg): def _pkgs(pkg):
spec = spack.spec.Spec('mpi').concretized() spec = spack.spec.Spec('mpi').concretized()
return [(spec.package, True)] return [(spec.package, True)]
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
installer = create_installer(const_arg)
request = installer.build_requests[0]
all_deps = defaultdict(set)
monkeypatch.setattr(inst, '_packages_needed_to_bootstrap_compiler', _pkgs) monkeypatch.setattr(inst, '_packages_needed_to_bootstrap_compiler', _pkgs)
installer._add_bootstrap_compilers(spec.package) installer._add_bootstrap_compilers(request.pkg, request, all_deps)
ids = list(installer.build_tasks) ids = list(installer.build_tasks)
assert len(ids) == 1 assert len(ids) == 1
@ -584,33 +646,40 @@ def _pkgs(pkg):
def test_prepare_for_install_on_installed(install_mockery, monkeypatch): def test_prepare_for_install_on_installed(install_mockery, monkeypatch):
"""Test of _prepare_for_install's early return for installed task path.""" """Test of _prepare_for_install's early return for installed task path."""
spec, installer = create_installer('dependent-install') const_arg = installer_args(['dependent-install'], {})
task = create_build_task(spec.package) installer = create_installer(const_arg)
request = installer.build_requests[0]
install_args = {'keep_prefix': True, 'keep_stage': True, 'restage': False}
task = create_build_task(request.pkg, install_args)
installer.installed.add(task.pkg_id) installer.installed.add(task.pkg_id)
monkeypatch.setattr(inst.PackageInstaller, '_ensure_install_ready', _noop) monkeypatch.setattr(inst.PackageInstaller, '_ensure_install_ready', _noop)
installer._prepare_for_install(task, True, True, False) installer._prepare_for_install(task)
def test_installer_init_queue(install_mockery): def test_installer_init_requests(install_mockery):
"""Test of installer queue functions.""" """Test of installer initial requests."""
spec_name = 'dependent-install'
with spack.config.override('config:install_missing_compilers', True): with spack.config.override('config:install_missing_compilers', True):
spec, installer = create_installer('dependent-install') const_arg = installer_args([spec_name], {})
installer._init_queue(True, True) installer = create_installer(const_arg)
ids = list(installer.build_tasks) # There is only one explicit request in this case
assert len(ids) == 2 assert len(installer.build_requests) == 1
assert 'dependency-install' in ids request = installer.build_requests[0]
assert 'dependent-install' in ids assert request.pkg.name == spec_name
def test_install_task_use_cache(install_mockery, monkeypatch): def test_install_task_use_cache(install_mockery, monkeypatch):
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
task = create_build_task(spec.package) installer = create_installer(const_arg)
request = installer.build_requests[0]
task = create_build_task(request.pkg)
monkeypatch.setattr(inst, '_install_from_cache', _true) monkeypatch.setattr(inst, '_install_from_cache', _true)
installer._install_task(task) installer._install_task(task)
assert spec.package.name in installer.installed assert request.pkg_id in installer.installed
def test_install_task_add_compiler(install_mockery, monkeypatch, capfd): def test_install_task_add_compiler(install_mockery, monkeypatch, capfd):
@ -619,8 +688,9 @@ def test_install_task_add_compiler(install_mockery, monkeypatch, capfd):
def _add(_compilers): def _add(_compilers):
tty.msg(config_msg) tty.msg(config_msg)
spec, installer = create_installer('a') const_arg = installer_args(['a'], {})
task = create_build_task(spec.package) installer = create_installer(const_arg)
task = create_build_task(installer.build_requests[0].pkg)
task.compiler = True task.compiler = True
# Preclude any meaningful side-effects # Preclude any meaningful side-effects
@ -638,7 +708,8 @@ def _add(_compilers):
def test_release_lock_write_n_exception(install_mockery, tmpdir, capsys): def test_release_lock_write_n_exception(install_mockery, tmpdir, capsys):
"""Test _release_lock for supposed write lock with exception.""" """Test _release_lock for supposed write lock with exception."""
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
installer = create_installer(const_arg)
pkg_id = 'test' pkg_id = 'test'
with tmpdir.as_cwd(): with tmpdir.as_cwd():
@ -652,10 +723,30 @@ def test_release_lock_write_n_exception(install_mockery, tmpdir, capsys):
assert msg in out assert msg in out
@pytest.mark.parametrize('installed', [True, False])
def test_push_task_skip_processed(install_mockery, installed):
"""Test to ensure skip re-queueing a processed package."""
const_arg = installer_args(['a'], {})
installer = create_installer(const_arg)
assert len(list(installer.build_tasks)) == 0
# Mark the package as installed OR failed
task = create_build_task(installer.build_requests[0].pkg)
if installed:
installer.installed.add(task.pkg_id)
else:
installer.failed[task.pkg_id] = None
installer._push_task(task)
assert len(list(installer.build_tasks)) == 0
def test_requeue_task(install_mockery, capfd): def test_requeue_task(install_mockery, capfd):
"""Test to ensure cover _requeue_task.""" """Test to ensure cover _requeue_task."""
spec, installer = create_installer('a') const_arg = installer_args(['a'], {})
task = create_build_task(spec.package) installer = create_installer(const_arg)
task = create_build_task(installer.build_requests[0].pkg)
installer._requeue_task(task) installer._requeue_task(task)
@ -663,9 +754,12 @@ def test_requeue_task(install_mockery, capfd):
assert len(ids) == 1 assert len(ids) == 1
qtask = installer.build_tasks[ids[0]] qtask = installer.build_tasks[ids[0]]
assert qtask.status == inst.STATUS_INSTALLING assert qtask.status == inst.STATUS_INSTALLING
assert qtask.sequence > task.sequence
assert qtask.attempts == task.attempts + 1
out = capfd.readouterr()[0] out = capfd.readouterr()[0]
assert 'Installing a in progress by another process' in out assert 'Installing a' in out
assert ' in progress by another process' in out
def test_cleanup_all_tasks(install_mockery, monkeypatch): def test_cleanup_all_tasks(install_mockery, monkeypatch):
@ -676,7 +770,9 @@ def _mktask(pkg):
def _rmtask(installer, pkg_id): def _rmtask(installer, pkg_id):
raise RuntimeError('Raise an exception to test except path') raise RuntimeError('Raise an exception to test except path')
spec, installer = create_installer('a') const_arg = installer_args(['a'], {})
installer = create_installer(const_arg)
spec = installer.build_requests[0].pkg.spec
# Cover task removal happy path # Cover task removal happy path
installer.build_tasks['a'] = _mktask(spec.package) installer.build_tasks['a'] = _mktask(spec.package)
@ -704,7 +800,9 @@ def _chgrp(path, group):
monkeypatch.setattr(prefs, 'get_package_group', _get_group) monkeypatch.setattr(prefs, 'get_package_group', _get_group)
monkeypatch.setattr(fs, 'chgrp', _chgrp) monkeypatch.setattr(fs, 'chgrp', _chgrp)
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
installer = create_installer(const_arg)
spec = installer.build_requests[0].pkg.spec
fs.touchp(spec.prefix) fs.touchp(spec.prefix)
metadatadir = spack.store.layout.metadata_path(spec) metadatadir = spack.store.layout.metadata_path(spec)
@ -725,7 +823,8 @@ def test_cleanup_failed_err(install_mockery, tmpdir, monkeypatch, capsys):
def _raise_except(lock): def _raise_except(lock):
raise RuntimeError(msg) raise RuntimeError(msg)
spec, installer = create_installer('trivial-install-test-package') const_arg = installer_args(['trivial-install-test-package'], {})
installer = create_installer(const_arg)
monkeypatch.setattr(lk.Lock, 'release_write', _raise_except) monkeypatch.setattr(lk.Lock, 'release_write', _raise_except)
pkg_id = 'test' pkg_id = 'test'
@ -741,7 +840,9 @@ def _raise_except(lock):
def test_update_failed_no_dependent_task(install_mockery): def test_update_failed_no_dependent_task(install_mockery):
"""Test _update_failed with missing dependent build tasks.""" """Test _update_failed with missing dependent build tasks."""
spec, installer = create_installer('dependent-install') const_arg = installer_args(['dependent-install'], {})
installer = create_installer(const_arg)
spec = installer.build_requests[0].pkg.spec
for dep in spec.traverse(root=False): for dep in spec.traverse(root=False):
task = create_build_task(dep.package) task = create_build_task(dep.package)
@ -751,7 +852,8 @@ def test_update_failed_no_dependent_task(install_mockery):
def test_install_uninstalled_deps(install_mockery, monkeypatch, capsys): def test_install_uninstalled_deps(install_mockery, monkeypatch, capsys):
"""Test install with uninstalled dependencies.""" """Test install with uninstalled dependencies."""
spec, installer = create_installer('dependent-install') const_arg = installer_args(['dependent-install'], {})
installer = create_installer(const_arg)
# Skip the actual installation and any status updates # Skip the actual installation and any status updates
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _noop) monkeypatch.setattr(inst.PackageInstaller, '_install_task', _noop)
@ -759,7 +861,7 @@ def test_install_uninstalled_deps(install_mockery, monkeypatch, capsys):
monkeypatch.setattr(inst.PackageInstaller, '_update_failed', _noop) monkeypatch.setattr(inst.PackageInstaller, '_update_failed', _noop)
msg = 'Cannot proceed with dependent-install' msg = 'Cannot proceed with dependent-install'
with pytest.raises(spack.installer.InstallError, match=msg): with pytest.raises(inst.InstallError, match=msg):
installer.install() installer.install()
out = str(capsys.readouterr()) out = str(capsys.readouterr())
@ -768,30 +870,47 @@ def test_install_uninstalled_deps(install_mockery, monkeypatch, capsys):
def test_install_failed(install_mockery, monkeypatch, capsys): def test_install_failed(install_mockery, monkeypatch, capsys):
"""Test install with failed install.""" """Test install with failed install."""
spec, installer = create_installer('b') const_arg = installer_args(['b'], {})
installer = create_installer(const_arg)
# Make sure the package is identified as failed # Make sure the package is identified as failed
monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true) monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
# Skip the actual installation though it should never get there installer.install()
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _noop)
msg = 'Installation of b failed'
with pytest.raises(spack.installer.InstallError, match=msg):
installer.install()
out = str(capsys.readouterr()) out = str(capsys.readouterr())
assert 'Warning: b failed to install' in out assert installer.build_requests[0].pkg_id in out
assert 'failed to install' in out
def test_install_failed_not_fast(install_mockery, monkeypatch, capsys):
"""Test install with failed install."""
const_arg = installer_args(['a'], {'fail_fast': False})
installer = create_installer(const_arg)
# Make sure the package is identified as failed
monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
installer.install()
out = str(capsys.readouterr())
assert 'failed to install' in out
assert 'Skipping build of a' in out
def test_install_fail_on_interrupt(install_mockery, monkeypatch): def test_install_fail_on_interrupt(install_mockery, monkeypatch):
"""Test ctrl-c interrupted install.""" """Test ctrl-c interrupted install."""
err_msg = 'mock keyboard interrupt' spec_name = 'a'
err_msg = 'mock keyboard interrupt for {0}'.format(spec_name)
def _interrupt(installer, task, **kwargs): def _interrupt(installer, task, **kwargs):
raise KeyboardInterrupt(err_msg) if task.pkg.name == spec_name:
raise KeyboardInterrupt(err_msg)
else:
installer.installed.add(task.pkg.name)
spec, installer = create_installer('a') const_arg = installer_args([spec_name], {})
installer = create_installer(const_arg)
# Raise a KeyboardInterrupt error to trigger early termination # Raise a KeyboardInterrupt error to trigger early termination
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _interrupt) monkeypatch.setattr(inst.PackageInstaller, '_install_task', _interrupt)
@ -799,41 +918,112 @@ def _interrupt(installer, task, **kwargs):
with pytest.raises(KeyboardInterrupt, match=err_msg): with pytest.raises(KeyboardInterrupt, match=err_msg):
installer.install() installer.install()
assert 'b' in installer.installed # ensure dependency of a is 'installed'
assert spec_name not in installer.installed
def test_install_fail_single(install_mockery, monkeypatch):
"""Test expected results for failure of single package."""
spec_name = 'a'
err_msg = 'mock internal package build error for {0}'.format(spec_name)
class MyBuildException(Exception):
pass
def _install(installer, task, **kwargs):
if task.pkg.name == spec_name:
raise MyBuildException(err_msg)
else:
installer.installed.add(task.pkg.name)
const_arg = installer_args([spec_name], {})
installer = create_installer(const_arg)
# Raise a KeyboardInterrupt error to trigger early termination
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
with pytest.raises(MyBuildException, match=err_msg):
installer.install()
assert 'b' in installer.installed # ensure dependency of a is 'installed'
assert spec_name not in installer.installed
def test_install_fail_multi(install_mockery, monkeypatch):
"""Test expected results for failure of multiple packages."""
spec_name = 'c'
err_msg = 'mock internal package build error'
class MyBuildException(Exception):
pass
def _install(installer, task, **kwargs):
if task.pkg.name == spec_name:
raise MyBuildException(err_msg)
else:
installer.installed.add(task.pkg.name)
const_arg = installer_args([spec_name, 'a'], {})
installer = create_installer(const_arg)
# Raise a KeyboardInterrupt error to trigger early termination
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
with pytest.raises(inst.InstallError, match='Installation request failed'):
installer.install()
assert 'a' in installer.installed # ensure the the second spec installed
assert spec_name not in installer.installed
def test_install_fail_fast_on_detect(install_mockery, monkeypatch, capsys): def test_install_fail_fast_on_detect(install_mockery, monkeypatch, capsys):
"""Test fail_fast install when an install failure is detected.""" """Test fail_fast install when an install failure is detected."""
spec, installer = create_installer('a') const_arg = installer_args(['b'], {'fail_fast': False})
const_arg.extend(installer_args(['c'], {'fail_fast': True}))
installer = create_installer(const_arg)
pkg_ids = [inst.package_id(spec.package) for spec, _ in const_arg]
# Make sure the package is identified as failed # Make sure all packages are identified as failed
# #
# This will prevent b from installing, which will cause the build of a # This will prevent b from installing, which will cause the build of a
# to be skipped. # to be skipped.
monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true) monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
with pytest.raises(spack.installer.InstallError): with pytest.raises(inst.InstallError, match='after first install failure'):
installer.install(fail_fast=True) installer.install()
out = str(capsys.readouterr()) assert pkg_ids[0] in installer.failed, 'Expected b to be marked as failed'
assert 'Skipping build of a' in out assert pkg_ids[1] not in installer.failed, \
'Expected no attempt to install c'
out = capsys.readouterr()[1]
assert '{0} failed to install'.format(pkg_ids[0]) in out
def _test_install_fail_fast_on_except_patch(installer, **kwargs):
"""Helper for test_install_fail_fast_on_except."""
# This is a module-scope function and not a local function because it
# needs to be pickleable.
raise RuntimeError('mock patch failure')
def test_install_fail_fast_on_except(install_mockery, monkeypatch, capsys): def test_install_fail_fast_on_except(install_mockery, monkeypatch, capsys):
"""Test fail_fast install when an install failure results from an error.""" """Test fail_fast install when an install failure results from an error."""
err_msg = 'mock patch failure' const_arg = installer_args(['a'], {'fail_fast': True})
installer = create_installer(const_arg)
def _patch(installer, task, **kwargs):
raise RuntimeError(err_msg)
spec, installer = create_installer('a')
# Raise a non-KeyboardInterrupt exception to trigger fast failure. # Raise a non-KeyboardInterrupt exception to trigger fast failure.
# #
# This will prevent b from installing, which will cause the build of a # This will prevent b from installing, which will cause the build of a
# to be skipped. # to be skipped.
monkeypatch.setattr(spack.package.PackageBase, 'do_patch', _patch) monkeypatch.setattr(
spack.package.PackageBase,
'do_patch',
_test_install_fail_fast_on_except_patch
)
with pytest.raises(spack.installer.InstallError, matches=err_msg): with pytest.raises(inst.InstallError, match='mock patch failure'):
installer.install(fail_fast=True) installer.install()
out = str(capsys.readouterr()) out = str(capsys.readouterr())
assert 'Skipping build of a' in out assert 'Skipping build of a' in out
@ -844,7 +1034,8 @@ def test_install_lock_failures(install_mockery, monkeypatch, capfd):
def _requeued(installer, task): def _requeued(installer, task):
tty.msg('requeued {0}' .format(task.pkg.spec.name)) tty.msg('requeued {0}' .format(task.pkg.spec.name))
spec, installer = create_installer('b') const_arg = installer_args(['b'], {})
installer = create_installer(const_arg)
# Ensure never acquire a lock # Ensure never acquire a lock
monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', _not_locked) monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', _not_locked)
@ -852,9 +1043,6 @@ def _requeued(installer, task):
# Ensure don't continually requeue the task # Ensure don't continually requeue the task
monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued) monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued)
# Skip the actual installation though should never reach it
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _noop)
installer.install() installer.install()
out = capfd.readouterr()[0] out = capfd.readouterr()[0]
expected = ['write locked', 'read locked', 'requeued'] expected = ['write locked', 'read locked', 'requeued']
@ -864,22 +1052,21 @@ def _requeued(installer, task):
def test_install_lock_installed_requeue(install_mockery, monkeypatch, capfd): def test_install_lock_installed_requeue(install_mockery, monkeypatch, capfd):
"""Cover basic install handling for installed package.""" """Cover basic install handling for installed package."""
def _install(installer, task, **kwargs): const_arg = installer_args(['b'], {})
tty.msg('{0} installing'.format(task.pkg.spec.name)) b, _ = const_arg[0]
installer = create_installer(const_arg)
b_pkg_id = inst.package_id(b.package)
def _prep(installer, task, keep_prefix, keep_stage, restage): def _prep(installer, task):
installer.installed.add('b') installer.installed.add(b_pkg_id)
tty.msg('{0} is installed' .format(task.pkg.spec.name)) tty.msg('{0} is installed' .format(b_pkg_id))
# also do not allow the package to be locked again # also do not allow the package to be locked again
monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked',
_not_locked) _not_locked)
def _requeued(installer, task): def _requeued(installer, task):
tty.msg('requeued {0}' .format(task.pkg.spec.name)) tty.msg('requeued {0}' .format(inst.package_id(task.pkg)))
# Skip the actual installation though should never reach it
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
# Flag the package as installed # Flag the package as installed
monkeypatch.setattr(inst.PackageInstaller, '_prepare_for_install', _prep) monkeypatch.setattr(inst.PackageInstaller, '_prepare_for_install', _prep)
@ -887,10 +1074,8 @@ def _requeued(installer, task):
# Ensure don't continually requeue the task # Ensure don't continually requeue the task
monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued) monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued)
spec, installer = create_installer('b')
installer.install() installer.install()
assert 'b' not in installer.installed assert b_pkg_id not in installer.installed
out = capfd.readouterr()[0] out = capfd.readouterr()[0]
expected = ['is installed', 'read locked', 'requeued'] expected = ['is installed', 'read locked', 'requeued']
@ -902,14 +1087,11 @@ def test_install_read_locked_requeue(install_mockery, monkeypatch, capfd):
"""Cover basic read lock handling for uninstalled package with requeue.""" """Cover basic read lock handling for uninstalled package with requeue."""
orig_fn = inst.PackageInstaller._ensure_locked orig_fn = inst.PackageInstaller._ensure_locked
def _install(installer, task, **kwargs):
tty.msg('{0} installing'.format(task.pkg.spec.name))
def _read(installer, lock_type, pkg): def _read(installer, lock_type, pkg):
tty.msg('{0}->read locked {1}' .format(lock_type, pkg.spec.name)) tty.msg('{0}->read locked {1}' .format(lock_type, pkg.spec.name))
return orig_fn(installer, 'read', pkg) return orig_fn(installer, 'read', pkg)
def _prep(installer, task, keep_prefix, keep_stage, restage): def _prep(installer, task):
tty.msg('preparing {0}' .format(task.pkg.spec.name)) tty.msg('preparing {0}' .format(task.pkg.spec.name))
assert task.pkg.spec.name not in installer.installed assert task.pkg.spec.name not in installer.installed
@ -919,16 +1101,14 @@ def _requeued(installer, task):
# Force a read lock # Force a read lock
monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', _read) monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', _read)
# Skip the actual installation though should never reach it
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
# Flag the package as installed # Flag the package as installed
monkeypatch.setattr(inst.PackageInstaller, '_prepare_for_install', _prep) monkeypatch.setattr(inst.PackageInstaller, '_prepare_for_install', _prep)
# Ensure don't continually requeue the task # Ensure don't continually requeue the task
monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued) monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued)
spec, installer = create_installer('b') const_arg = installer_args(['b'], {})
installer = create_installer(const_arg)
installer.install() installer.install()
assert 'b' not in installer.installed assert 'b' not in installer.installed
@ -939,28 +1119,55 @@ def _requeued(installer, task):
assert exp in ln assert exp in ln
def test_install_dir_exists(install_mockery, monkeypatch, capfd): def test_install_dir_exists(install_mockery, monkeypatch):
"""Cover capture of install directory exists error.""" """Cover capture of install directory exists error."""
err = 'Mock directory exists error' def _install(installer, task):
raise dl.InstallDirectoryAlreadyExistsError(task.pkg.prefix)
def _install(installer, task, **kwargs): # Ensure raise the desired exception
raise dl.InstallDirectoryAlreadyExistsError(err) monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
const_arg = installer_args(['b'], {})
installer = create_installer(const_arg)
err = 'already exists'
with pytest.raises(dl.InstallDirectoryAlreadyExistsError, match=err):
installer.install()
b, _ = const_arg[0]
assert inst.package_id(b.package) in installer.installed
def test_install_dir_exists_multi(install_mockery, monkeypatch, capfd):
"""Cover capture of install directory exists error for multiple specs."""
def _install(installer, task):
raise dl.InstallDirectoryAlreadyExistsError(task.pkg.prefix)
# Skip the actual installation though should never reach it # Skip the actual installation though should never reach it
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install) monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
spec, installer = create_installer('b') # Use two packages to ensure multiple specs
const_arg = installer_args(['b', 'c'], {})
installer = create_installer(const_arg)
with pytest.raises(dl.InstallDirectoryAlreadyExistsError, match=err): with pytest.raises(inst.InstallError, match='Installation request failed'):
installer.install() installer.install()
assert 'b' in installer.installed err = capfd.readouterr()[1]
assert 'already exists' in err
for spec, install_args in const_arg:
pkg_id = inst.package_id(spec.package)
assert pkg_id in installer.installed
def test_install_skip_patch(install_mockery, mock_fetch): def test_install_skip_patch(install_mockery, mock_fetch):
"""Test the path skip_patch install path.""" """Test the path skip_patch install path."""
spec, installer = create_installer('b') spec_name = 'b'
const_arg = installer_args([spec_name],
{'fake': False, 'skip_patch': True})
installer = create_installer(const_arg)
installer.install(fake=False, skip_patch=True) installer.install()
assert 'b' in installer.installed spec, install_args = const_arg[0]
assert inst.package_id(spec.package) in installer.installed

22
var/spack/repos/builtin.mock/packages/depb/package.py Normal file
View file

@ -0,0 +1,22 @@
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
# Spack Project Developers. See the top-level COPYRIGHT file for details.
#
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
from spack import *
class Depb(AutotoolsPackage):
"""Simple package with one build dependency"""
homepage = "http://www.example.com"
url = "http://www.example.com/a-1.0.tar.gz"
version('1.0', '0123456789abcdef0123456789abcdef')
depends_on('b')
def install(self, spec, prefix):
# sanity_check_prefix requires something in the install directory
# Test requires overriding the one provided by `AutotoolsPackage`
mkdirp(prefix.bin)