Support parallel environment builds (#18131)
As of #13100, Spack installs the dependencies of a _single_ spec in parallel. Environments, when installed, can only get parallelism from each individual spec, as they're installed in order. This PR makes entire environments build in parallel by extending Spack's package installer to accept multiple root specs. The install command and Environment class have been updated to use the new parallel install method. The specs and kwargs for each *uninstalled* package (when not force-replacing installations) of an environment are collected, passed to the `PackageInstaller`, and processed using a single build queue. This introduces a `BuildRequest` class to track install arguments, and it significantly cleans up the code used to track package ids during installation. Package ids in the build queue are now just DAG hashes as you would expect, Other tasks: - [x] Finish updating the unit tests based on `PackageInstaller`'s use of `BuildRequest` and the associated changes - [x] Change `environment.py`'s `install_all` to use the `PackageInstaller` directly - [x] Change the `install` command to leverage the new installation process for multiple specs - [x] Change install output messages for external packages, e.g.: `[+] /usr` -> `[+] /usr (external bzip2-1.0.8-<dag-hash>` - [x] Fix incomplete environment install's view setup/update and not confirming all packages are installed (?) - [x] Ensure externally installed package dependencies are properly accounted for in remaining build tasks - [x] Add tests for coverage (if insufficient and can identity the appropriate, uncovered non-comment lines) - [x] Add documentation - [x] Resolve multi-compiler environment install issues - [x] Fix issue with environment installation reporting (restore CDash/JUnit reports)
This commit is contained in:
Tamara Dahlgren
GitHub
parent
423e80af23
commit
6fa6af1070
17 changed files with 1223 additions and 557 deletions
|
|
@ -132,32 +132,28 @@ If ``mpileaks`` depends on other packages, Spack will install the
|
|||
dependencies first. It then fetches the ``mpileaks`` tarball, expands
|
||||
it, verifies that it was downloaded without errors, builds it, and
|
||||
installs it in its own directory under ``$SPACK_ROOT/opt``. You'll see
|
||||
a number of messages from spack, a lot of build output, and a message
|
||||
that the packages is installed:
|
||||
a number of messages from Spack, a lot of build output, and a message
|
||||
that the package is installed. Add one or more debug options (``-d``)
|
||||
to get increasingly detailed output.
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ spack install mpileaks
|
||||
==> Installing mpileaks
|
||||
==> mpich is already installed in ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpich@3.0.4.
|
||||
==> callpath is already installed in ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/callpath@1.0.2-5dce4318.
|
||||
==> adept-utils is already installed in ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/adept-utils@1.0-5adef8da.
|
||||
==> Trying to fetch from https://github.com/hpc/mpileaks/releases/download/v1.0/mpileaks-1.0.tar.gz
|
||||
######################################################################## 100.0%
|
||||
==> Staging archive: ~/spack/var/spack/stage/mpileaks@1.0%gcc@4.4.7 arch=linux-debian7-x86_64-59f6ad23/mpileaks-1.0.tar.gz
|
||||
==> Created stage in ~/spack/var/spack/stage/mpileaks@1.0%gcc@4.4.7 arch=linux-debian7-x86_64-59f6ad23.
|
||||
==> No patches needed for mpileaks.
|
||||
==> Building mpileaks.
|
||||
|
||||
... build output ...
|
||||
|
||||
==> Successfully installed mpileaks.
|
||||
Fetch: 2.16s. Build: 9.82s. Total: 11.98s.
|
||||
[+] ~/spack/opt/linux-debian7-x86_64/gcc@4.4.7/mpileaks@1.0-59f6ad23
|
||||
... dependency build output ...
|
||||
==> Installing mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2
|
||||
==> No binary for mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2 found: installing from source
|
||||
==> mpileaks: Executing phase: 'autoreconf'
|
||||
==> mpileaks: Executing phase: 'configure'
|
||||
==> mpileaks: Executing phase: 'build'
|
||||
==> mpileaks: Executing phase: 'install'
|
||||
[+] ~/spack/opt/linux-rhel7-broadwell/gcc-8.1.0/mpileaks-1.0-ph7pbnhl334wuhogmugriohcwempqry2
|
||||
|
||||
The last line, with the ``[+]``, indicates where the package is
|
||||
installed.
|
||||
|
||||
Add the debug option -- ``spack install -d mpileaks`` -- to get additional
|
||||
output.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Building a specific version
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
|
|
|||
|
|
@ -191,44 +191,24 @@ Environment has been activated. Similarly, the ``install`` and
|
|||
==> 0 installed packages
|
||||
|
||||
$ spack install zlib@1.2.11
|
||||
==> Installing zlib
|
||||
==> Searching for binary cache of zlib
|
||||
==> Warning: No Spack mirrors are currently configured
|
||||
==> No binary for zlib found: installing from source
|
||||
==> Fetching http://zlib.net/fossils/zlib-1.2.11.tar.gz
|
||||
######################################################################## 100.0%
|
||||
==> Staging archive: /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur/zlib-1.2.11.tar.gz
|
||||
==> Created stage in /spack/var/spack/stage/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
|
||||
==> No patches needed for zlib
|
||||
==> Building zlib [Package]
|
||||
==> Executing phase: 'install'
|
||||
==> Successfully installed zlib
|
||||
Fetch: 0.36s. Build: 11.58s. Total: 11.93s.
|
||||
[+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.11-3r4cfkmx3wwfqeof4bc244yduu2mz4ur
|
||||
==> Installing zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv
|
||||
==> No binary for zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv found: installing from source
|
||||
==> zlib: Executing phase: 'install'
|
||||
[+] ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.11-q6cqrdto4iktfg6qyqcc5u4vmfmwb7iv
|
||||
|
||||
$ spack env activate myenv
|
||||
|
||||
$ spack find
|
||||
==> In environment myenv
|
||||
==> No root specs
|
||||
|
||||
==> 0 installed packages
|
||||
|
||||
$ spack install zlib@1.2.8
|
||||
==> Installing zlib
|
||||
==> Searching for binary cache of zlib
|
||||
==> Warning: No Spack mirrors are currently configured
|
||||
==> No binary for zlib found: installing from source
|
||||
==> Fetching http://zlib.net/fossils/zlib-1.2.8.tar.gz
|
||||
######################################################################## 100.0%
|
||||
==> Staging archive: /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7/zlib-1.2.8.tar.gz
|
||||
==> Created stage in /spack/var/spack/stage/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
|
||||
==> No patches needed for zlib
|
||||
==> Building zlib [Package]
|
||||
==> Executing phase: 'install'
|
||||
==> Successfully installed zlib
|
||||
Fetch: 0.26s. Build: 2.08s. Total: 2.35s.
|
||||
[+] /spack/opt/spack/linux-rhel7-x86_64/gcc-4.9.3/zlib-1.2.8-y2t6kq3s23l52yzhcyhbpovswajzi7f7
|
||||
==> Installing zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x
|
||||
==> No binary for zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x found: installing from source
|
||||
==> zlib: Executing phase: 'install'
|
||||
[+] ~/spack/opt/spack/linux-rhel7-broadwell/gcc-8.1.0/zlib-1.2.8-yfc7epf57nsfn2gn4notccaiyxha6z7x
|
||||
==> Updating view at ~/spack/var/spack/environments/myenv/.spack-env/view
|
||||
|
||||
$ spack find
|
||||
==> In environment myenv
|
||||
|
|
@ -236,15 +216,17 @@ Environment has been activated. Similarly, the ``install`` and
|
|||
zlib@1.2.8
|
||||
|
||||
==> 1 installed package
|
||||
-- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
|
||||
-- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
|
||||
zlib@1.2.8
|
||||
|
||||
$ despacktivate
|
||||
|
||||
$ spack find
|
||||
==> 2 installed packages
|
||||
-- linux-rhel7-x86_64 / gcc@4.9.3 -------------------------------
|
||||
-- linux-rhel7-broadwell / gcc@8.1.0 ----------------------------
|
||||
zlib@1.2.8 zlib@1.2.11
|
||||
|
||||
|
||||
Note that when we installed the abstract spec ``zlib@1.2.8``, it was
|
||||
presented as a root of the Environment. All explicitly installed
|
||||
packages will be listed as roots of the Environment.
|
||||
|
|
@ -349,6 +331,9 @@ installed specs using the ``-c`` (``--concretized``) flag.
|
|||
|
||||
==> 0 installed packages
|
||||
|
||||
|
||||
.. _installing-environment:
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Installing an Environment
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
|
|
|||
|
|
@ -1778,8 +1778,18 @@ RPATHs in Spack are handled in one of three ways:
|
|||
Parallel builds
|
||||
---------------
|
||||
|
||||
Spack supports parallel builds on an individual package and at the
|
||||
installation level. Package-level parallelism is established by the
|
||||
``--jobs`` option and its configuration and package recipe equivalents.
|
||||
Installation-level parallelism is driven by the DAG(s) of the requested
|
||||
package or packages.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Package-level build parallelism
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
By default, Spack will invoke ``make()``, or any other similar tool,
|
||||
with a ``-j <njobs>`` argument, so that builds run in parallel.
|
||||
with a ``-j <njobs>`` argument, so those builds run in parallel.
|
||||
The parallelism is determined by the value of the ``build_jobs`` entry
|
||||
in ``config.yaml`` (see :ref:`here <build-jobs>` for more details on
|
||||
how this value is computed).
|
||||
|
|
@ -1827,6 +1837,43 @@ you set ``parallel`` to ``False`` at the package level, then each call
|
|||
to ``make()`` will be sequential by default, but packagers can call
|
||||
``make(parallel=True)`` to override it.
|
||||
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
Install-level build parallelism
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
|
||||
Spack supports the concurrent installation of packages within a Spack
|
||||
instance across multiple processes using file system locks. This
|
||||
parallelism is separate from the package-level achieved through build
|
||||
systems' use of the ``-j <njobs>`` option. With install-level parallelism,
|
||||
processes coordinate the installation of the dependencies of specs
|
||||
provided on the command line and as part of an environment build with
|
||||
only **one process** being allowed to install a given package at a time.
|
||||
Refer to :ref:`Dependencies` for more information on dependencies and
|
||||
:ref:`installing-environment` for how to install an environment.
|
||||
|
||||
Concurrent processes may be any combination of interactive sessions and
|
||||
batch jobs. Which means a ``spack install`` can be running in a terminal
|
||||
window while a batch job is running ``spack install`` on the same or
|
||||
overlapping dependencies without any process trying to re-do the work of
|
||||
another.
|
||||
|
||||
For example, if you are using SLURM, you could launch an installation
|
||||
of ``mpich`` using the following command:
|
||||
|
||||
.. code-block:: console
|
||||
|
||||
$ srun -N 2 -n 8 spack install -j 4 mpich@3.3.2
|
||||
|
||||
This will create eight concurrent four-job installation on two different
|
||||
nodes.
|
||||
|
||||
.. note::
|
||||
|
||||
The effective parallelism will be based on the maximum number of
|
||||
packages that can be installed at the same time, which will limited
|
||||
by the number of packages with no (remaining) uninstalled dependencies.
|
||||
|
||||
|
||||
.. _dependencies:
|
||||
|
||||
------------
|
||||
|
|
|
|||
2
lib/spack/env/cc
vendored
2
lib/spack/env/cc
vendored
|
|
@ -22,7 +22,7 @@
|
|||
|
||||
# This is an array of environment variables that need to be set before
|
||||
# the script runs. They are set by routines in spack.build_environment
|
||||
# as part of spack.package.Package.do_install().
|
||||
# as part of the package installation process.
|
||||
parameters=(
|
||||
SPACK_ENV_PATH
|
||||
SPACK_DEBUG_LOG_DIR
|
||||
|
|
|
|||
|
|
@ -20,6 +20,7 @@
|
|||
import spack.paths
|
||||
import spack.report
|
||||
from spack.error import SpackError
|
||||
from spack.installer import PackageInstaller
|
||||
|
||||
|
||||
description = "build and install packages"
|
||||
|
|
@ -29,7 +30,7 @@
|
|||
|
||||
def update_kwargs_from_args(args, kwargs):
|
||||
"""Parse cli arguments and construct a dictionary
|
||||
that will be passed to Package.do_install API"""
|
||||
that will be passed to the package installer."""
|
||||
|
||||
kwargs.update({
|
||||
'fail_fast': args.fail_fast,
|
||||
|
|
@ -238,23 +239,29 @@ def default_log_file(spec):
|
|||
return fs.os.path.join(dirname, basename)
|
||||
|
||||
|
||||
def install_spec(cli_args, kwargs, abstract_spec, spec):
|
||||
"""Do the actual installation."""
|
||||
def install_specs(cli_args, kwargs, specs):
|
||||
"""Do the actual installation.
|
||||
|
||||
Args:
|
||||
cli_args (Namespace): argparse namespace with command arguments
|
||||
kwargs (dict): keyword arguments
|
||||
specs (list of tuples): list of (abstract, concrete) spec tuples
|
||||
"""
|
||||
|
||||
try:
|
||||
# handle active environment, if any
|
||||
env = ev.get_env(cli_args, 'install')
|
||||
if env:
|
||||
with env.write_transaction():
|
||||
concrete = env.concretize_and_add(
|
||||
abstract_spec, spec)
|
||||
env.write(regenerate_views=False)
|
||||
env._install(concrete, **kwargs)
|
||||
with env.write_transaction():
|
||||
env.regenerate_views()
|
||||
else:
|
||||
spec.package.do_install(**kwargs)
|
||||
|
||||
try:
|
||||
if env:
|
||||
for abstract, concrete in specs:
|
||||
with env.write_transaction():
|
||||
concrete = env.concretize_and_add(abstract, concrete)
|
||||
env.write(regenerate_views=False)
|
||||
env.install_all(cli_args, **kwargs)
|
||||
else:
|
||||
installs = [(concrete.package, kwargs) for _, concrete in specs]
|
||||
builder = PackageInstaller(installs)
|
||||
builder.install()
|
||||
except spack.build_environment.InstallError as e:
|
||||
if cli_args.show_log_on_error:
|
||||
e.print_context()
|
||||
|
|
@ -280,6 +287,10 @@ def install(parser, args, **kwargs):
|
|||
parser.print_help()
|
||||
return
|
||||
|
||||
reporter = spack.report.collect_info(args.log_format, args)
|
||||
if args.log_file:
|
||||
reporter.filename = args.log_file
|
||||
|
||||
if not args.spec and not args.specfiles:
|
||||
# if there are no args but an active environment
|
||||
# then install the packages from it.
|
||||
|
|
@ -294,8 +305,17 @@ def install(parser, args, **kwargs):
|
|||
# once, as it can be slow.
|
||||
env.write(regenerate_views=False)
|
||||
|
||||
tty.msg("Installing environment %s" % env.name)
|
||||
env.install_all(args)
|
||||
specs = env.all_specs()
|
||||
if not args.log_file and not reporter.filename:
|
||||
reporter.filename = default_log_file(specs[0])
|
||||
reporter.specs = specs
|
||||
|
||||
tty.msg("Installing environment {0}".format(env.name))
|
||||
with reporter:
|
||||
env.install_all(args, **kwargs)
|
||||
|
||||
tty.debug("Regenerating environment views for {0}"
|
||||
.format(env.name))
|
||||
with env.write_transaction():
|
||||
# It is not strictly required to synchronize view regeneration
|
||||
# but doing so can prevent redundant work in the filesystem.
|
||||
|
|
@ -319,17 +339,13 @@ def install(parser, args, **kwargs):
|
|||
spack.config.set('config:checksum', False, scope='command_line')
|
||||
|
||||
# Parse cli arguments and construct a dictionary
|
||||
# that will be passed to Package.do_install API
|
||||
# that will be passed to the package installer
|
||||
update_kwargs_from_args(args, kwargs)
|
||||
|
||||
if args.run_tests:
|
||||
tty.warn("Deprecated option: --run-tests: use --test=all instead")
|
||||
|
||||
# 1. Abstract specs from cli
|
||||
reporter = spack.report.collect_info(args.log_format, args)
|
||||
if args.log_file:
|
||||
reporter.filename = args.log_file
|
||||
|
||||
abstract_specs = spack.cmd.parse_specs(args.spec)
|
||||
tests = False
|
||||
if args.test == 'all' or args.run_tests:
|
||||
|
|
@ -401,5 +417,4 @@ def install(parser, args, **kwargs):
|
|||
# overwrite all concrete explicit specs from this build
|
||||
kwargs['overwrite'] = [spec.dag_hash() for spec in specs]
|
||||
|
||||
for abstract, concrete in zip(abstract_specs, specs):
|
||||
install_spec(args, kwargs, abstract, concrete)
|
||||
install_specs(args, kwargs, zip(abstract_specs, specs))
|
||||
|
|
|
|||
|
|
@ -39,6 +39,7 @@
|
|||
from spack.variant import UnknownVariantError
|
||||
import spack.util.lock as lk
|
||||
from spack.util.path import substitute_path_variables
|
||||
from spack.installer import PackageInstaller
|
||||
|
||||
#: environment variable used to indicate the active environment
|
||||
spack_env_var = 'SPACK_ENV'
|
||||
|
|
@ -1371,24 +1372,7 @@ def _get_overwrite_specs(self):
|
|||
|
||||
return ret
|
||||
|
||||
def install(self, user_spec, concrete_spec=None, **install_args):
|
||||
"""Install a single spec into an environment.
|
||||
|
||||
This will automatically concretize the single spec, but it won't
|
||||
affect other as-yet unconcretized specs.
|
||||
"""
|
||||
concrete = self.concretize_and_add(user_spec, concrete_spec)
|
||||
|
||||
self._install(concrete, **install_args)
|
||||
|
||||
def _install(self, spec, **install_args):
|
||||
# "spec" must be concrete
|
||||
package = spec.package
|
||||
|
||||
install_args['overwrite'] = install_args.get(
|
||||
'overwrite', []) + self._get_overwrite_specs()
|
||||
package.do_install(**install_args)
|
||||
|
||||
def _install_log_links(self, spec):
|
||||
if not spec.external:
|
||||
# Make sure log directory exists
|
||||
log_path = self.log_path
|
||||
|
|
@ -1402,19 +1386,22 @@ def _install(self, spec, **install_args):
|
|||
os.remove(build_log_link)
|
||||
os.symlink(spec.package.build_log_path, build_log_link)
|
||||
|
||||
def install_all(self, args=None):
|
||||
def install_all(self, args=None, **install_args):
|
||||
"""Install all concretized specs in an environment.
|
||||
|
||||
Note: this does not regenerate the views for the environment;
|
||||
that needs to be done separately with a call to write().
|
||||
|
||||
Args:
|
||||
args (Namespace): argparse namespace with command arguments
|
||||
install_args (dict): keyword install arguments
|
||||
"""
|
||||
|
||||
# If "spack install" is invoked repeatedly for a large environment
|
||||
# where all specs are already installed, the operation can take
|
||||
# a large amount of time due to repeatedly acquiring and releasing
|
||||
# locks, this does an initial check across all specs within a single
|
||||
# DB read transaction to reduce time spent in this case.
|
||||
tty.debug('Assessing installation status of environment packages')
|
||||
specs_to_install = []
|
||||
with spack.store.db.read_transaction():
|
||||
for concretized_hash in self.concretized_order:
|
||||
|
|
@ -1426,14 +1413,43 @@ def install_all(self, args=None):
|
|||
# If it's a dev build it could need to be reinstalled
|
||||
specs_to_install.append(spec)
|
||||
|
||||
if not specs_to_install:
|
||||
tty.msg('All of the packages are already installed')
|
||||
return
|
||||
|
||||
tty.debug('Processing {0} uninstalled specs'
|
||||
.format(len(specs_to_install)))
|
||||
|
||||
install_args['overwrite'] = install_args.get(
|
||||
'overwrite', []) + self._get_overwrite_specs()
|
||||
|
||||
installs = []
|
||||
for spec in specs_to_install:
|
||||
# Parse cli arguments and construct a dictionary
|
||||
# that will be passed to Package.do_install API
|
||||
# that will be passed to the package installer
|
||||
kwargs = dict()
|
||||
if install_args:
|
||||
kwargs.update(install_args)
|
||||
if args:
|
||||
spack.cmd.install.update_kwargs_from_args(args, kwargs)
|
||||
|
||||
self._install(spec, **kwargs)
|
||||
installs.append((spec.package, kwargs))
|
||||
|
||||
try:
|
||||
builder = PackageInstaller(installs)
|
||||
builder.install()
|
||||
finally:
|
||||
# Ensure links are set appropriately
|
||||
for spec in specs_to_install:
|
||||
if spec.package.installed:
|
||||
try:
|
||||
self._install_log_links(spec)
|
||||
except OSError as e:
|
||||
tty.warn('Could not install log links for {0}: {1}'
|
||||
.format(spec.name, str(e)))
|
||||
|
||||
with self.write_transaction():
|
||||
self.regenerate_views()
|
||||
|
||||
def all_specs(self):
|
||||
"""Return all specs, even those a user spec would shadow."""
|
||||
|
|
|
|||
File diff suppressed because it is too large
Load diff
|
|
@ -53,8 +53,7 @@
|
|||
from six import string_types
|
||||
from six import with_metaclass
|
||||
from spack.filesystem_view import YamlFilesystemView
|
||||
from spack.installer import \
|
||||
install_args_docstring, PackageInstaller, InstallError
|
||||
from spack.installer import PackageInstaller, InstallError
|
||||
from spack.stage import stage_prefix, Stage, ResourceStage, StageComposite
|
||||
from spack.util.package_hash import package_hash
|
||||
from spack.version import Version
|
||||
|
|
@ -1591,17 +1590,45 @@ def do_install(self, **kwargs):
|
|||
Package implementations should override install() to describe
|
||||
their build process.
|
||||
|
||||
Args:"""
|
||||
Args:
|
||||
cache_only (bool): Fail if binary package unavailable.
|
||||
dirty (bool): Don't clean the build environment before installing.
|
||||
explicit (bool): True if package was explicitly installed, False
|
||||
if package was implicitly installed (as a dependency).
|
||||
fail_fast (bool): Fail if any dependency fails to install;
|
||||
otherwise, the default is to install as many dependencies as
|
||||
possible (i.e., best effort installation).
|
||||
fake (bool): Don't really build; install fake stub files instead.
|
||||
force (bool): Install again, even if already installed.
|
||||
install_deps (bool): Install dependencies before installing this
|
||||
package
|
||||
install_source (bool): By default, source is not installed, but
|
||||
for debugging it might be useful to keep it around.
|
||||
keep_prefix (bool): Keep install prefix on failure. By default,
|
||||
destroys it.
|
||||
keep_stage (bool): By default, stage is destroyed only if there
|
||||
are no exceptions during build. Set to True to keep the stage
|
||||
even with exceptions.
|
||||
restage (bool): Force spack to restage the package source.
|
||||
skip_patch (bool): Skip patch stage of build if True.
|
||||
stop_before (InstallPhase): stop execution before this
|
||||
installation phase (or None)
|
||||
stop_at (InstallPhase): last installation phase to be executed
|
||||
(or None)
|
||||
tests (bool or list or set): False to run no tests, True to test
|
||||
all packages, or a list of package names to run tests for some
|
||||
use_cache (bool): Install from binary package, if available.
|
||||
verbose (bool): Display verbose build output (by default,
|
||||
suppresses it)
|
||||
"""
|
||||
# Non-transitive dev specs need to keep the dev stage and be built from
|
||||
# source every time. Transitive ones just need to be built from source.
|
||||
dev_path_var = self.spec.variants.get('dev_path', None)
|
||||
if dev_path_var:
|
||||
kwargs['keep_stage'] = True
|
||||
|
||||
builder = PackageInstaller(self)
|
||||
builder.install(**kwargs)
|
||||
|
||||
do_install.__doc__ += install_args_docstring
|
||||
builder = PackageInstaller([(self, kwargs)])
|
||||
builder.install()
|
||||
|
||||
def unit_test_check(self):
|
||||
"""Hook for unit tests to assert things about package internals.
|
||||
|
|
|
|||
|
|
@ -44,8 +44,8 @@ def fetch_package_log(pkg):
|
|||
|
||||
|
||||
class InfoCollector(object):
|
||||
"""Decorates PackageInstaller._install_task, which is called by
|
||||
PackageBase.do_install for each spec, to collect information
|
||||
"""Decorates PackageInstaller._install_task, which is called via
|
||||
PackageBase.do_install for individual specs, to collect information
|
||||
on the installation of certain specs.
|
||||
|
||||
When exiting the context this change will be rolled-back.
|
||||
|
|
|
|||
55
lib/spack/spack/test/buildrequest.py
Normal file
55
lib/spack/spack/test/buildrequest.py
Normal file
|
|
@ -0,0 +1,55 @@
|
|||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
||||
import pytest
|
||||
|
||||
import spack.installer as inst
|
||||
import spack.repo
|
||||
import spack.spec
|
||||
|
||||
|
||||
def test_build_request_errors(install_mockery):
|
||||
with pytest.raises(ValueError, match='must be a package'):
|
||||
inst.BuildRequest('abc', {})
|
||||
|
||||
pkg = spack.repo.get('trivial-install-test-package')
|
||||
with pytest.raises(ValueError, match='must have a concrete spec'):
|
||||
inst.BuildRequest(pkg, {})
|
||||
|
||||
|
||||
def test_build_request_basics(install_mockery):
|
||||
spec = spack.spec.Spec('dependent-install')
|
||||
spec.concretize()
|
||||
assert spec.concrete
|
||||
|
||||
# Ensure key properties match expectations
|
||||
request = inst.BuildRequest(spec.package, {})
|
||||
assert not request.pkg.stop_before_phase
|
||||
assert not request.pkg.last_phase
|
||||
assert request.spec == spec.package.spec
|
||||
|
||||
# Ensure key default install arguments are set
|
||||
assert 'install_package' in request.install_args
|
||||
assert 'install_deps' in request.install_args
|
||||
|
||||
|
||||
def test_build_request_strings(install_mockery):
|
||||
"""Tests of BuildRequest repr and str for coverage purposes."""
|
||||
# Using a package with one dependency
|
||||
spec = spack.spec.Spec('dependent-install')
|
||||
spec.concretize()
|
||||
assert spec.concrete
|
||||
|
||||
# Ensure key properties match expectations
|
||||
request = inst.BuildRequest(spec.package, {})
|
||||
|
||||
# Cover __repr__
|
||||
irep = request.__repr__()
|
||||
assert irep.startswith(request.__class__.__name__)
|
||||
|
||||
# Cover __str__
|
||||
istr = str(request)
|
||||
assert "package=dependent-install" in istr
|
||||
assert "install_args=" in istr
|
||||
|
|
@ -12,17 +12,22 @@
|
|||
|
||||
def test_build_task_errors(install_mockery):
|
||||
with pytest.raises(ValueError, match='must be a package'):
|
||||
inst.BuildTask('abc', False, 0, 0, 0, [])
|
||||
inst.BuildTask('abc', None, False, 0, 0, 0, [])
|
||||
|
||||
pkg = spack.repo.get('trivial-install-test-package')
|
||||
with pytest.raises(ValueError, match='must have a concrete spec'):
|
||||
inst.BuildTask(pkg, False, 0, 0, 0, [])
|
||||
inst.BuildTask(pkg, None, False, 0, 0, 0, [])
|
||||
|
||||
spec = spack.spec.Spec('trivial-install-test-package')
|
||||
spec.concretize()
|
||||
assert spec.concrete
|
||||
with pytest.raises(ValueError, match='must have a build request'):
|
||||
inst.BuildTask(spec.package, None, False, 0, 0, 0, [])
|
||||
|
||||
request = inst.BuildRequest(spec.package, {})
|
||||
with pytest.raises(inst.InstallError, match='Cannot create a build task'):
|
||||
inst.BuildTask(spec.package, False, 0, 0, inst.STATUS_REMOVED, [])
|
||||
inst.BuildTask(spec.package, request, False, 0, 0, inst.STATUS_REMOVED,
|
||||
[])
|
||||
|
||||
|
||||
def test_build_task_basics(install_mockery):
|
||||
|
|
@ -31,7 +36,10 @@ def test_build_task_basics(install_mockery):
|
|||
assert spec.concrete
|
||||
|
||||
# Ensure key properties match expectations
|
||||
task = inst.BuildTask(spec.package, False, 0, 0, inst.STATUS_ADDED, [])
|
||||
request = inst.BuildRequest(spec.package, {})
|
||||
task = inst.BuildTask(spec.package, request, False, 0, 0,
|
||||
inst.STATUS_ADDED, [])
|
||||
assert task.explicit # package was "explicitly" requested
|
||||
assert task.priority == len(task.uninstalled_deps)
|
||||
assert task.key == (task.priority, task.sequence)
|
||||
|
||||
|
|
@ -51,7 +59,9 @@ def test_build_task_strings(install_mockery):
|
|||
assert spec.concrete
|
||||
|
||||
# Ensure key properties match expectations
|
||||
task = inst.BuildTask(spec.package, False, 0, 0, inst.STATUS_ADDED, [])
|
||||
request = inst.BuildRequest(spec.package, {})
|
||||
task = inst.BuildTask(spec.package, request, False, 0, 0,
|
||||
inst.STATUS_ADDED, [])
|
||||
|
||||
# Cover __repr__
|
||||
irep = task.__repr__()
|
||||
|
|
|
|||
|
|
@ -8,7 +8,7 @@
|
|||
import spack.spec
|
||||
import llnl.util.filesystem as fs
|
||||
import spack.environment as ev
|
||||
from spack.main import SpackCommand, SpackCommandError
|
||||
from spack.main import SpackCommand
|
||||
|
||||
dev_build = SpackCommand('dev-build')
|
||||
install = SpackCommand('install')
|
||||
|
|
@ -99,13 +99,15 @@ def test_dev_build_before_until(tmpdir, mock_packages, install_mockery):
|
|||
dev_build('-u', 'edit', '-b', 'edit',
|
||||
'dev-build-test-install@0.0.0')
|
||||
|
||||
with pytest.raises(SpackCommandError):
|
||||
dev_build('-u', 'phase_that_does_not_exist',
|
||||
'dev-build-test-install@0.0.0')
|
||||
bad_phase = 'phase_that_does_not_exist'
|
||||
not_allowed = 'is not a valid phase'
|
||||
out = dev_build('-u', bad_phase, 'dev-build-test-install@0.0.0')
|
||||
assert bad_phase in out
|
||||
assert not_allowed in out
|
||||
|
||||
with pytest.raises(SpackCommandError):
|
||||
dev_build('-b', 'phase_that_does_not_exist',
|
||||
'dev-build-test-install@0.0.0')
|
||||
out = dev_build('-b', bad_phase, 'dev-build-test-install@0.0.0')
|
||||
assert bad_phase in out
|
||||
assert not_allowed in out
|
||||
|
||||
|
||||
def print_spack_cc(*args):
|
||||
|
|
|
|||
|
|
@ -185,18 +185,54 @@ def setup_error(pkg, env):
|
|||
assert "Warning: couldn't get environment settings" in err
|
||||
|
||||
|
||||
def test_env_install_same_spec_twice(install_mockery, mock_fetch, capfd):
|
||||
def test_env_install_same_spec_twice(install_mockery, mock_fetch):
|
||||
env('create', 'test')
|
||||
|
||||
e = ev.read('test')
|
||||
with capfd.disabled():
|
||||
with e:
|
||||
# The first installation outputs the package prefix
|
||||
install('cmake-client')
|
||||
# The second installation attempt will also update the view
|
||||
# The first installation outputs the package prefix, updates the view
|
||||
out = install('cmake-client')
|
||||
assert 'Updating view at' in out
|
||||
|
||||
# The second installation reports all packages already installed
|
||||
out = install('cmake-client')
|
||||
assert 'already installed' in out
|
||||
|
||||
|
||||
def test_env_install_two_specs_same_dep(
|
||||
install_mockery, mock_fetch, tmpdir, capsys):
|
||||
"""Test installation of two packages that share a dependency with no
|
||||
connection and the second specifying the dependency as a 'build'
|
||||
dependency.
|
||||
"""
|
||||
path = tmpdir.join('spack.yaml')
|
||||
|
||||
with tmpdir.as_cwd():
|
||||
with open(str(path), 'w') as f:
|
||||
f.write("""\
|
||||
env:
|
||||
specs:
|
||||
- a
|
||||
- depb
|
||||
""")
|
||||
|
||||
env('create', 'test', 'spack.yaml')
|
||||
|
||||
with ev.read('test'):
|
||||
with capsys.disabled():
|
||||
out = install()
|
||||
|
||||
# Ensure both packages reach install phase processing and are installed
|
||||
out = str(out)
|
||||
assert 'depb: Executing phase:' in out
|
||||
assert 'a: Executing phase:' in out
|
||||
|
||||
depb = spack.repo.path.get_pkg_class('depb')
|
||||
assert depb.installed, 'Expected depb to be installed'
|
||||
|
||||
a = spack.repo.path.get_pkg_class('a')
|
||||
assert a.installed, 'Expected a to be installed'
|
||||
|
||||
|
||||
def test_remove_after_concretize():
|
||||
e = ev.create('test')
|
||||
|
|
@ -2094,7 +2130,8 @@ def test_cant_install_single_spec_when_concretizing_together():
|
|||
e.concretization = 'together'
|
||||
|
||||
with pytest.raises(ev.SpackEnvironmentError, match=r'cannot install'):
|
||||
e.install('zlib')
|
||||
e.concretize_and_add('zlib')
|
||||
e.install_all()
|
||||
|
||||
|
||||
def test_duplicate_packages_raise_when_concretizing_together():
|
||||
|
|
|
|||
|
|
@ -38,7 +38,7 @@
|
|||
def noop_install(monkeypatch):
|
||||
def noop(*args, **kwargs):
|
||||
pass
|
||||
monkeypatch.setattr(spack.package.PackageBase, 'do_install', noop)
|
||||
monkeypatch.setattr(spack.installer.PackageInstaller, 'install', noop)
|
||||
|
||||
|
||||
def test_install_package_and_dependency(
|
||||
|
|
@ -321,15 +321,19 @@ def test_install_from_file(spec, concretize, error_code, tmpdir):
|
|||
with specfile.open('w') as f:
|
||||
spec.to_yaml(f)
|
||||
|
||||
err_msg = 'does not contain a concrete spec' if error_code else ''
|
||||
|
||||
# Relative path to specfile (regression for #6906)
|
||||
with fs.working_dir(specfile.dirname):
|
||||
# A non-concrete spec will fail to be installed
|
||||
install('-f', specfile.basename, fail_on_error=False)
|
||||
out = install('-f', specfile.basename, fail_on_error=False)
|
||||
assert install.returncode == error_code
|
||||
assert err_msg in out
|
||||
|
||||
# Absolute path to specfile (regression for #6983)
|
||||
install('-f', str(specfile), fail_on_error=False)
|
||||
out = install('-f', str(specfile), fail_on_error=False)
|
||||
assert install.returncode == error_code
|
||||
assert err_msg in out
|
||||
|
||||
|
||||
@pytest.mark.disable_clean_stage_check
|
||||
|
|
@ -615,12 +619,14 @@ def test_build_warning_output(tmpdir, mock_fetch, install_mockery, capfd):
|
|||
assert 'foo.c:89: warning: some weird warning!' in msg
|
||||
|
||||
|
||||
def test_cache_only_fails(tmpdir, mock_fetch, install_mockery):
|
||||
def test_cache_only_fails(tmpdir, mock_fetch, install_mockery, capfd):
|
||||
# libelf from cache fails to install, which automatically removes the
|
||||
# the libdwarf build task and flags the package as failed to install.
|
||||
err_msg = 'Installation of libdwarf failed'
|
||||
with pytest.raises(spack.installer.InstallError, match=err_msg):
|
||||
install('--cache-only', 'libdwarf')
|
||||
# the libdwarf build task
|
||||
with capfd.disabled():
|
||||
out = install('--cache-only', 'libdwarf')
|
||||
|
||||
assert 'Failed to install libelf' in out
|
||||
assert 'Skipping build of libdwarf' in out
|
||||
|
||||
# Check that failure prefix locks are still cached
|
||||
failure_lock_prefixes = ','.join(spack.store.db._prefix_failures.keys())
|
||||
|
|
@ -828,6 +834,7 @@ def test_cache_install_full_hash_match(
|
|||
mirror_url = 'file://{0}'.format(mirror_dir.strpath)
|
||||
|
||||
s = Spec('libdwarf').concretized()
|
||||
package_id = spack.installer.package_id(s.package)
|
||||
|
||||
# Install a package
|
||||
install(s.name)
|
||||
|
|
@ -843,9 +850,9 @@ def test_cache_install_full_hash_match(
|
|||
|
||||
# Make sure we get the binary version by default
|
||||
install_output = install('--no-check-signature', s.name, output=str)
|
||||
expect_msg = 'Extracting {0} from binary cache'.format(s.name)
|
||||
expect_extract_msg = 'Extracting {0} from binary cache'.format(package_id)
|
||||
|
||||
assert expect_msg in install_output
|
||||
assert expect_extract_msg in install_output
|
||||
|
||||
uninstall('-y', s.name)
|
||||
|
||||
|
|
@ -855,9 +862,7 @@ def test_cache_install_full_hash_match(
|
|||
# Check that even if the full hash changes, we install from binary when
|
||||
# we don't explicitly require the full hash to match
|
||||
install_output = install('--no-check-signature', s.name, output=str)
|
||||
expect_msg = 'Extracting {0} from binary cache'.format(s.name)
|
||||
|
||||
assert expect_msg in install_output
|
||||
assert expect_extract_msg in install_output
|
||||
|
||||
uninstall('-y', s.name)
|
||||
|
||||
|
|
@ -865,7 +870,7 @@ def test_cache_install_full_hash_match(
|
|||
# installs from source.
|
||||
install_output = install('--require-full-hash-match', s.name, output=str)
|
||||
expect_msg = 'No binary for {0} found: installing from source'.format(
|
||||
s.name)
|
||||
package_id)
|
||||
|
||||
assert expect_msg in install_output
|
||||
|
||||
|
|
|
|||
|
|
@ -507,7 +507,7 @@ def test_unconcretized_install(install_mockery, mock_fetch, mock_packages):
|
|||
"""Test attempts to perform install phases with unconcretized spec."""
|
||||
spec = Spec('trivial-install-test-package')
|
||||
|
||||
with pytest.raises(ValueError, match="only install concrete packages"):
|
||||
with pytest.raises(ValueError, match='must have a concrete spec'):
|
||||
spec.package.do_install()
|
||||
|
||||
with pytest.raises(ValueError, match="only patch concrete packages"):
|
||||
|
|
|
|||
|
|
@ -59,34 +59,52 @@ def _true(*args, **kwargs):
|
|||
return True
|
||||
|
||||
|
||||
def create_build_task(pkg):
|
||||
def create_build_task(pkg, install_args={}):
|
||||
"""
|
||||
Create a built task for the given (concretized) package
|
||||
|
||||
Args:
|
||||
pkg (PackageBase): concretized package associated with the task
|
||||
install_args (dict): dictionary of kwargs (or install args)
|
||||
|
||||
Return:
|
||||
(BuildTask) A basic package build task
|
||||
"""
|
||||
return inst.BuildTask(pkg, False, 0, 0, inst.STATUS_ADDED, [])
|
||||
request = inst.BuildRequest(pkg, install_args)
|
||||
return inst.BuildTask(pkg, request, False, 0, 0, inst.STATUS_ADDED, [])
|
||||
|
||||
|
||||
def create_installer(spec_name):
|
||||
def create_installer(installer_args):
|
||||
"""
|
||||
Create an installer for the named spec
|
||||
Create an installer using the concretized spec for each arg
|
||||
|
||||
Args:
|
||||
spec_name (str): Name of the explicit install spec
|
||||
installer_args (list of tuples): the list of (spec name, kwargs) tuples
|
||||
|
||||
Return:
|
||||
spec (Spec): concretized spec
|
||||
installer (PackageInstaller): the associated package installer
|
||||
"""
|
||||
spec = spack.spec.Spec(spec_name)
|
||||
const_arg = [(spec.package, kwargs) for spec, kwargs in installer_args]
|
||||
return inst.PackageInstaller(const_arg)
|
||||
|
||||
|
||||
def installer_args(spec_names, kwargs={}):
|
||||
"""Return a the installer argument with each spec paired with kwargs
|
||||
|
||||
Args:
|
||||
spec_names (list of str): list of spec names
|
||||
kwargs (dict or None): install arguments to apply to all of the specs
|
||||
|
||||
Returns:
|
||||
list of (spec, kwargs): the installer constructor argument
|
||||
"""
|
||||
arg = []
|
||||
for name in spec_names:
|
||||
spec = spack.spec.Spec(name)
|
||||
spec.concretize()
|
||||
assert spec.concrete
|
||||
return spec, inst.PackageInstaller(spec.package)
|
||||
arg.append((spec, kwargs))
|
||||
return arg
|
||||
|
||||
|
||||
@pytest.mark.parametrize('sec,result', [
|
||||
|
|
@ -99,6 +117,21 @@ def test_hms(sec, result):
|
|||
assert inst._hms(sec) == result
|
||||
|
||||
|
||||
def test_get_dependent_ids(install_mockery, mock_packages):
|
||||
# Concretize the parent package, which handle dependency too
|
||||
spec = spack.spec.Spec('a')
|
||||
spec.concretize()
|
||||
assert spec.concrete
|
||||
|
||||
pkg_id = inst.package_id(spec.package)
|
||||
|
||||
# Grab the sole dependency of 'a', which is 'b'
|
||||
dep = spec.dependencies()[0]
|
||||
|
||||
# Ensure the parent package is a dependent of the dependency package
|
||||
assert pkg_id in inst.get_dependent_ids(dep)
|
||||
|
||||
|
||||
def test_install_msg(monkeypatch):
|
||||
"""Test results of call to install_msg based on debug level."""
|
||||
name = 'some-package'
|
||||
|
|
@ -190,7 +223,9 @@ def _spec(spec, preferred_mirrors=None):
|
|||
spec = spack.spec.Spec('a').concretized()
|
||||
assert inst._process_binary_cache_tarball(spec.package, spec, False, False)
|
||||
|
||||
assert 'Extracting a from binary cache' in capfd.readouterr()[0]
|
||||
out = capfd.readouterr()[0]
|
||||
assert 'Extracting a' in out
|
||||
assert 'from binary cache' in out
|
||||
|
||||
|
||||
def test_try_install_from_binary_cache(install_mockery, mock_packages,
|
||||
|
|
@ -216,18 +251,9 @@ def _mirrors_for_spec(spec, force, full_hash_match=False):
|
|||
assert 'add a spack mirror to allow download' in str(captured)
|
||||
|
||||
|
||||
def test_installer_init_errors(install_mockery):
|
||||
"""Test to ensure cover installer constructor errors."""
|
||||
with pytest.raises(ValueError, match='must be a package'):
|
||||
inst.PackageInstaller('abc')
|
||||
|
||||
pkg = spack.repo.get('trivial-install-test-package')
|
||||
with pytest.raises(ValueError, match='Can only install concrete'):
|
||||
inst.PackageInstaller(pkg)
|
||||
|
||||
|
||||
def test_installer_repr(install_mockery):
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
irep = installer.__repr__()
|
||||
assert irep.startswith(installer.__class__.__name__)
|
||||
|
|
@ -236,7 +262,8 @@ def test_installer_repr(install_mockery):
|
|||
|
||||
|
||||
def test_installer_str(install_mockery):
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
istr = str(installer)
|
||||
assert "#tasks=0" in istr
|
||||
|
|
@ -244,20 +271,33 @@ def test_installer_str(install_mockery):
|
|||
assert "failed (0)" in istr
|
||||
|
||||
|
||||
def test_installer_last_phase_error(install_mockery, capsys):
|
||||
spec = spack.spec.Spec('trivial-install-test-package')
|
||||
spec.concretize()
|
||||
assert spec.concrete
|
||||
with pytest.raises(SystemExit):
|
||||
installer = inst.PackageInstaller(spec.package)
|
||||
installer.install(stop_at='badphase')
|
||||
def test_check_before_phase_error(install_mockery):
|
||||
pkg = spack.repo.get('trivial-install-test-package')
|
||||
pkg.stop_before_phase = 'beforephase'
|
||||
with pytest.raises(inst.BadInstallPhase) as exc_info:
|
||||
inst._check_last_phase(pkg)
|
||||
|
||||
captured = capsys.readouterr()
|
||||
assert 'is not an allowed phase' in str(captured)
|
||||
err = str(exc_info.value)
|
||||
assert 'is not a valid phase' in err
|
||||
assert pkg.stop_before_phase in err
|
||||
|
||||
|
||||
def test_check_last_phase_error(install_mockery):
|
||||
pkg = spack.repo.get('trivial-install-test-package')
|
||||
pkg.stop_before_phase = None
|
||||
pkg.last_phase = 'badphase'
|
||||
with pytest.raises(inst.BadInstallPhase) as exc_info:
|
||||
inst._check_last_phase(pkg)
|
||||
|
||||
err = str(exc_info.value)
|
||||
assert 'is not a valid phase' in err
|
||||
assert pkg.last_phase in err
|
||||
|
||||
|
||||
def test_installer_ensure_ready_errors(install_mockery):
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
spec = installer.build_requests[0].pkg.spec
|
||||
|
||||
fmt = r'cannot be installed locally.*{0}'
|
||||
# Force an external package error
|
||||
|
|
@ -290,7 +330,9 @@ def test_ensure_locked_err(install_mockery, monkeypatch, tmpdir, capsys):
|
|||
def _raise(lock, timeout):
|
||||
raise RuntimeError(mock_err_msg)
|
||||
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
spec = installer.build_requests[0].pkg.spec
|
||||
|
||||
monkeypatch.setattr(ulk.Lock, 'acquire_read', _raise)
|
||||
with tmpdir.as_cwd():
|
||||
|
|
@ -304,14 +346,17 @@ def _raise(lock, timeout):
|
|||
|
||||
def test_ensure_locked_have(install_mockery, tmpdir, capsys):
|
||||
"""Test _ensure_locked when already have lock."""
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
spec = installer.build_requests[0].pkg.spec
|
||||
pkg_id = inst.package_id(spec.package)
|
||||
|
||||
with tmpdir.as_cwd():
|
||||
# Test "downgrade" of a read lock (to a read lock)
|
||||
lock = lk.Lock('./test', default_timeout=1e-9, desc='test')
|
||||
lock_type = 'read'
|
||||
tpl = (lock_type, lock)
|
||||
installer.locks[installer.pkg_id] = tpl
|
||||
installer.locks[pkg_id] = tpl
|
||||
assert installer._ensure_locked(lock_type, spec.package) == tpl
|
||||
|
||||
# Test "upgrade" of a read lock without read count to a write
|
||||
|
|
@ -341,7 +386,9 @@ def test_ensure_locked_have(install_mockery, tmpdir, capsys):
|
|||
def test_ensure_locked_new_lock(
|
||||
install_mockery, tmpdir, lock_type, reads, writes):
|
||||
pkg_id = 'a'
|
||||
spec, installer = create_installer(pkg_id)
|
||||
const_arg = installer_args([pkg_id], {})
|
||||
installer = create_installer(const_arg)
|
||||
spec = installer.build_requests[0].pkg.spec
|
||||
with tmpdir.as_cwd():
|
||||
ltype, lock = installer._ensure_locked(lock_type, spec.package)
|
||||
assert ltype == lock_type
|
||||
|
|
@ -359,7 +406,9 @@ def _pl(db, spec, timeout):
|
|||
return lock
|
||||
|
||||
pkg_id = 'a'
|
||||
spec, installer = create_installer(pkg_id)
|
||||
const_arg = installer_args([pkg_id], {})
|
||||
installer = create_installer(const_arg)
|
||||
spec = installer.build_requests[0].pkg.spec
|
||||
|
||||
monkeypatch.setattr(spack.database.Database, 'prefix_lock', _pl)
|
||||
|
||||
|
|
@ -529,52 +578,65 @@ def _raise_except(path):
|
|||
|
||||
|
||||
def test_check_deps_status_install_failure(install_mockery, monkeypatch):
|
||||
spec, installer = create_installer('a')
|
||||
const_arg = installer_args(['a'], {})
|
||||
installer = create_installer(const_arg)
|
||||
request = installer.build_requests[0]
|
||||
|
||||
# Make sure the package is identified as failed
|
||||
monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
|
||||
|
||||
with pytest.raises(inst.InstallError, match='install failure'):
|
||||
installer._check_deps_status()
|
||||
installer._check_deps_status(request)
|
||||
|
||||
|
||||
def test_check_deps_status_write_locked(install_mockery, monkeypatch):
|
||||
spec, installer = create_installer('a')
|
||||
const_arg = installer_args(['a'], {})
|
||||
installer = create_installer(const_arg)
|
||||
request = installer.build_requests[0]
|
||||
|
||||
# Ensure the lock is not acquired
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', _not_locked)
|
||||
|
||||
with pytest.raises(inst.InstallError, match='write locked by another'):
|
||||
installer._check_deps_status()
|
||||
installer._check_deps_status(request)
|
||||
|
||||
|
||||
def test_check_deps_status_external(install_mockery, monkeypatch):
|
||||
spec, installer = create_installer('a')
|
||||
const_arg = installer_args(['a'], {})
|
||||
installer = create_installer(const_arg)
|
||||
request = installer.build_requests[0]
|
||||
|
||||
# Mock the known dependent, b, as external so assumed to be installed
|
||||
monkeypatch.setattr(spack.spec.Spec, 'external', True)
|
||||
installer._check_deps_status()
|
||||
assert 'b' in installer.installed
|
||||
installer._check_deps_status(request)
|
||||
assert list(installer.installed)[0].startswith('b')
|
||||
|
||||
|
||||
def test_check_deps_status_upstream(install_mockery, monkeypatch):
|
||||
spec, installer = create_installer('a')
|
||||
const_arg = installer_args(['a'], {})
|
||||
installer = create_installer(const_arg)
|
||||
request = installer.build_requests[0]
|
||||
|
||||
# Mock the known dependent, b, as installed upstream
|
||||
monkeypatch.setattr(spack.package.PackageBase, 'installed_upstream', True)
|
||||
installer._check_deps_status()
|
||||
assert 'b' in installer.installed
|
||||
installer._check_deps_status(request)
|
||||
assert list(installer.installed)[0].startswith('b')
|
||||
|
||||
|
||||
def test_add_bootstrap_compilers(install_mockery, monkeypatch):
|
||||
from collections import defaultdict
|
||||
|
||||
def _pkgs(pkg):
|
||||
spec = spack.spec.Spec('mpi').concretized()
|
||||
return [(spec.package, True)]
|
||||
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
request = installer.build_requests[0]
|
||||
all_deps = defaultdict(set)
|
||||
|
||||
monkeypatch.setattr(inst, '_packages_needed_to_bootstrap_compiler', _pkgs)
|
||||
installer._add_bootstrap_compilers(spec.package)
|
||||
installer._add_bootstrap_compilers(request.pkg, request, all_deps)
|
||||
|
||||
ids = list(installer.build_tasks)
|
||||
assert len(ids) == 1
|
||||
|
|
@ -584,33 +646,40 @@ def _pkgs(pkg):
|
|||
|
||||
def test_prepare_for_install_on_installed(install_mockery, monkeypatch):
|
||||
"""Test of _prepare_for_install's early return for installed task path."""
|
||||
spec, installer = create_installer('dependent-install')
|
||||
task = create_build_task(spec.package)
|
||||
const_arg = installer_args(['dependent-install'], {})
|
||||
installer = create_installer(const_arg)
|
||||
request = installer.build_requests[0]
|
||||
|
||||
install_args = {'keep_prefix': True, 'keep_stage': True, 'restage': False}
|
||||
task = create_build_task(request.pkg, install_args)
|
||||
installer.installed.add(task.pkg_id)
|
||||
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_ensure_install_ready', _noop)
|
||||
installer._prepare_for_install(task, True, True, False)
|
||||
installer._prepare_for_install(task)
|
||||
|
||||
|
||||
def test_installer_init_queue(install_mockery):
|
||||
"""Test of installer queue functions."""
|
||||
def test_installer_init_requests(install_mockery):
|
||||
"""Test of installer initial requests."""
|
||||
spec_name = 'dependent-install'
|
||||
with spack.config.override('config:install_missing_compilers', True):
|
||||
spec, installer = create_installer('dependent-install')
|
||||
installer._init_queue(True, True)
|
||||
const_arg = installer_args([spec_name], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
ids = list(installer.build_tasks)
|
||||
assert len(ids) == 2
|
||||
assert 'dependency-install' in ids
|
||||
assert 'dependent-install' in ids
|
||||
# There is only one explicit request in this case
|
||||
assert len(installer.build_requests) == 1
|
||||
request = installer.build_requests[0]
|
||||
assert request.pkg.name == spec_name
|
||||
|
||||
|
||||
def test_install_task_use_cache(install_mockery, monkeypatch):
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
task = create_build_task(spec.package)
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
request = installer.build_requests[0]
|
||||
task = create_build_task(request.pkg)
|
||||
|
||||
monkeypatch.setattr(inst, '_install_from_cache', _true)
|
||||
installer._install_task(task)
|
||||
assert spec.package.name in installer.installed
|
||||
assert request.pkg_id in installer.installed
|
||||
|
||||
|
||||
def test_install_task_add_compiler(install_mockery, monkeypatch, capfd):
|
||||
|
|
@ -619,8 +688,9 @@ def test_install_task_add_compiler(install_mockery, monkeypatch, capfd):
|
|||
def _add(_compilers):
|
||||
tty.msg(config_msg)
|
||||
|
||||
spec, installer = create_installer('a')
|
||||
task = create_build_task(spec.package)
|
||||
const_arg = installer_args(['a'], {})
|
||||
installer = create_installer(const_arg)
|
||||
task = create_build_task(installer.build_requests[0].pkg)
|
||||
task.compiler = True
|
||||
|
||||
# Preclude any meaningful side-effects
|
||||
|
|
@ -638,7 +708,8 @@ def _add(_compilers):
|
|||
|
||||
def test_release_lock_write_n_exception(install_mockery, tmpdir, capsys):
|
||||
"""Test _release_lock for supposed write lock with exception."""
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
pkg_id = 'test'
|
||||
with tmpdir.as_cwd():
|
||||
|
|
@ -652,10 +723,30 @@ def test_release_lock_write_n_exception(install_mockery, tmpdir, capsys):
|
|||
assert msg in out
|
||||
|
||||
|
||||
@pytest.mark.parametrize('installed', [True, False])
|
||||
def test_push_task_skip_processed(install_mockery, installed):
|
||||
"""Test to ensure skip re-queueing a processed package."""
|
||||
const_arg = installer_args(['a'], {})
|
||||
installer = create_installer(const_arg)
|
||||
assert len(list(installer.build_tasks)) == 0
|
||||
|
||||
# Mark the package as installed OR failed
|
||||
task = create_build_task(installer.build_requests[0].pkg)
|
||||
if installed:
|
||||
installer.installed.add(task.pkg_id)
|
||||
else:
|
||||
installer.failed[task.pkg_id] = None
|
||||
|
||||
installer._push_task(task)
|
||||
|
||||
assert len(list(installer.build_tasks)) == 0
|
||||
|
||||
|
||||
def test_requeue_task(install_mockery, capfd):
|
||||
"""Test to ensure cover _requeue_task."""
|
||||
spec, installer = create_installer('a')
|
||||
task = create_build_task(spec.package)
|
||||
const_arg = installer_args(['a'], {})
|
||||
installer = create_installer(const_arg)
|
||||
task = create_build_task(installer.build_requests[0].pkg)
|
||||
|
||||
installer._requeue_task(task)
|
||||
|
||||
|
|
@ -663,9 +754,12 @@ def test_requeue_task(install_mockery, capfd):
|
|||
assert len(ids) == 1
|
||||
qtask = installer.build_tasks[ids[0]]
|
||||
assert qtask.status == inst.STATUS_INSTALLING
|
||||
assert qtask.sequence > task.sequence
|
||||
assert qtask.attempts == task.attempts + 1
|
||||
|
||||
out = capfd.readouterr()[0]
|
||||
assert 'Installing a in progress by another process' in out
|
||||
assert 'Installing a' in out
|
||||
assert ' in progress by another process' in out
|
||||
|
||||
|
||||
def test_cleanup_all_tasks(install_mockery, monkeypatch):
|
||||
|
|
@ -676,7 +770,9 @@ def _mktask(pkg):
|
|||
def _rmtask(installer, pkg_id):
|
||||
raise RuntimeError('Raise an exception to test except path')
|
||||
|
||||
spec, installer = create_installer('a')
|
||||
const_arg = installer_args(['a'], {})
|
||||
installer = create_installer(const_arg)
|
||||
spec = installer.build_requests[0].pkg.spec
|
||||
|
||||
# Cover task removal happy path
|
||||
installer.build_tasks['a'] = _mktask(spec.package)
|
||||
|
|
@ -704,7 +800,9 @@ def _chgrp(path, group):
|
|||
monkeypatch.setattr(prefs, 'get_package_group', _get_group)
|
||||
monkeypatch.setattr(fs, 'chgrp', _chgrp)
|
||||
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
spec = installer.build_requests[0].pkg.spec
|
||||
|
||||
fs.touchp(spec.prefix)
|
||||
metadatadir = spack.store.layout.metadata_path(spec)
|
||||
|
|
@ -725,7 +823,8 @@ def test_cleanup_failed_err(install_mockery, tmpdir, monkeypatch, capsys):
|
|||
def _raise_except(lock):
|
||||
raise RuntimeError(msg)
|
||||
|
||||
spec, installer = create_installer('trivial-install-test-package')
|
||||
const_arg = installer_args(['trivial-install-test-package'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
monkeypatch.setattr(lk.Lock, 'release_write', _raise_except)
|
||||
pkg_id = 'test'
|
||||
|
|
@ -741,7 +840,9 @@ def _raise_except(lock):
|
|||
|
||||
def test_update_failed_no_dependent_task(install_mockery):
|
||||
"""Test _update_failed with missing dependent build tasks."""
|
||||
spec, installer = create_installer('dependent-install')
|
||||
const_arg = installer_args(['dependent-install'], {})
|
||||
installer = create_installer(const_arg)
|
||||
spec = installer.build_requests[0].pkg.spec
|
||||
|
||||
for dep in spec.traverse(root=False):
|
||||
task = create_build_task(dep.package)
|
||||
|
|
@ -751,7 +852,8 @@ def test_update_failed_no_dependent_task(install_mockery):
|
|||
|
||||
def test_install_uninstalled_deps(install_mockery, monkeypatch, capsys):
|
||||
"""Test install with uninstalled dependencies."""
|
||||
spec, installer = create_installer('dependent-install')
|
||||
const_arg = installer_args(['dependent-install'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
# Skip the actual installation and any status updates
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _noop)
|
||||
|
|
@ -759,7 +861,7 @@ def test_install_uninstalled_deps(install_mockery, monkeypatch, capsys):
|
|||
monkeypatch.setattr(inst.PackageInstaller, '_update_failed', _noop)
|
||||
|
||||
msg = 'Cannot proceed with dependent-install'
|
||||
with pytest.raises(spack.installer.InstallError, match=msg):
|
||||
with pytest.raises(inst.InstallError, match=msg):
|
||||
installer.install()
|
||||
|
||||
out = str(capsys.readouterr())
|
||||
|
|
@ -768,30 +870,47 @@ def test_install_uninstalled_deps(install_mockery, monkeypatch, capsys):
|
|||
|
||||
def test_install_failed(install_mockery, monkeypatch, capsys):
|
||||
"""Test install with failed install."""
|
||||
spec, installer = create_installer('b')
|
||||
const_arg = installer_args(['b'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
# Make sure the package is identified as failed
|
||||
monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
|
||||
|
||||
# Skip the actual installation though it should never get there
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _noop)
|
||||
|
||||
msg = 'Installation of b failed'
|
||||
with pytest.raises(spack.installer.InstallError, match=msg):
|
||||
installer.install()
|
||||
|
||||
out = str(capsys.readouterr())
|
||||
assert 'Warning: b failed to install' in out
|
||||
assert installer.build_requests[0].pkg_id in out
|
||||
assert 'failed to install' in out
|
||||
|
||||
|
||||
def test_install_failed_not_fast(install_mockery, monkeypatch, capsys):
|
||||
"""Test install with failed install."""
|
||||
const_arg = installer_args(['a'], {'fail_fast': False})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
# Make sure the package is identified as failed
|
||||
monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
|
||||
|
||||
installer.install()
|
||||
|
||||
out = str(capsys.readouterr())
|
||||
assert 'failed to install' in out
|
||||
assert 'Skipping build of a' in out
|
||||
|
||||
|
||||
def test_install_fail_on_interrupt(install_mockery, monkeypatch):
|
||||
"""Test ctrl-c interrupted install."""
|
||||
err_msg = 'mock keyboard interrupt'
|
||||
spec_name = 'a'
|
||||
err_msg = 'mock keyboard interrupt for {0}'.format(spec_name)
|
||||
|
||||
def _interrupt(installer, task, **kwargs):
|
||||
if task.pkg.name == spec_name:
|
||||
raise KeyboardInterrupt(err_msg)
|
||||
else:
|
||||
installer.installed.add(task.pkg.name)
|
||||
|
||||
spec, installer = create_installer('a')
|
||||
const_arg = installer_args([spec_name], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
# Raise a KeyboardInterrupt error to trigger early termination
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _interrupt)
|
||||
|
|
@ -799,41 +918,112 @@ def _interrupt(installer, task, **kwargs):
|
|||
with pytest.raises(KeyboardInterrupt, match=err_msg):
|
||||
installer.install()
|
||||
|
||||
assert 'b' in installer.installed # ensure dependency of a is 'installed'
|
||||
assert spec_name not in installer.installed
|
||||
|
||||
|
||||
def test_install_fail_single(install_mockery, monkeypatch):
|
||||
"""Test expected results for failure of single package."""
|
||||
spec_name = 'a'
|
||||
err_msg = 'mock internal package build error for {0}'.format(spec_name)
|
||||
|
||||
class MyBuildException(Exception):
|
||||
pass
|
||||
|
||||
def _install(installer, task, **kwargs):
|
||||
if task.pkg.name == spec_name:
|
||||
raise MyBuildException(err_msg)
|
||||
else:
|
||||
installer.installed.add(task.pkg.name)
|
||||
|
||||
const_arg = installer_args([spec_name], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
# Raise a KeyboardInterrupt error to trigger early termination
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
|
||||
|
||||
with pytest.raises(MyBuildException, match=err_msg):
|
||||
installer.install()
|
||||
|
||||
assert 'b' in installer.installed # ensure dependency of a is 'installed'
|
||||
assert spec_name not in installer.installed
|
||||
|
||||
|
||||
def test_install_fail_multi(install_mockery, monkeypatch):
|
||||
"""Test expected results for failure of multiple packages."""
|
||||
spec_name = 'c'
|
||||
err_msg = 'mock internal package build error'
|
||||
|
||||
class MyBuildException(Exception):
|
||||
pass
|
||||
|
||||
def _install(installer, task, **kwargs):
|
||||
if task.pkg.name == spec_name:
|
||||
raise MyBuildException(err_msg)
|
||||
else:
|
||||
installer.installed.add(task.pkg.name)
|
||||
|
||||
const_arg = installer_args([spec_name, 'a'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
# Raise a KeyboardInterrupt error to trigger early termination
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
|
||||
|
||||
with pytest.raises(inst.InstallError, match='Installation request failed'):
|
||||
installer.install()
|
||||
|
||||
assert 'a' in installer.installed # ensure the the second spec installed
|
||||
assert spec_name not in installer.installed
|
||||
|
||||
|
||||
def test_install_fail_fast_on_detect(install_mockery, monkeypatch, capsys):
|
||||
"""Test fail_fast install when an install failure is detected."""
|
||||
spec, installer = create_installer('a')
|
||||
const_arg = installer_args(['b'], {'fail_fast': False})
|
||||
const_arg.extend(installer_args(['c'], {'fail_fast': True}))
|
||||
installer = create_installer(const_arg)
|
||||
pkg_ids = [inst.package_id(spec.package) for spec, _ in const_arg]
|
||||
|
||||
# Make sure the package is identified as failed
|
||||
# Make sure all packages are identified as failed
|
||||
#
|
||||
# This will prevent b from installing, which will cause the build of a
|
||||
# to be skipped.
|
||||
monkeypatch.setattr(spack.database.Database, 'prefix_failed', _true)
|
||||
|
||||
with pytest.raises(spack.installer.InstallError):
|
||||
installer.install(fail_fast=True)
|
||||
with pytest.raises(inst.InstallError, match='after first install failure'):
|
||||
installer.install()
|
||||
|
||||
out = str(capsys.readouterr())
|
||||
assert 'Skipping build of a' in out
|
||||
assert pkg_ids[0] in installer.failed, 'Expected b to be marked as failed'
|
||||
assert pkg_ids[1] not in installer.failed, \
|
||||
'Expected no attempt to install c'
|
||||
|
||||
out = capsys.readouterr()[1]
|
||||
assert '{0} failed to install'.format(pkg_ids[0]) in out
|
||||
|
||||
|
||||
def _test_install_fail_fast_on_except_patch(installer, **kwargs):
|
||||
"""Helper for test_install_fail_fast_on_except."""
|
||||
# This is a module-scope function and not a local function because it
|
||||
# needs to be pickleable.
|
||||
raise RuntimeError('mock patch failure')
|
||||
|
||||
|
||||
def test_install_fail_fast_on_except(install_mockery, monkeypatch, capsys):
|
||||
"""Test fail_fast install when an install failure results from an error."""
|
||||
err_msg = 'mock patch failure'
|
||||
|
||||
def _patch(installer, task, **kwargs):
|
||||
raise RuntimeError(err_msg)
|
||||
|
||||
spec, installer = create_installer('a')
|
||||
const_arg = installer_args(['a'], {'fail_fast': True})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
# Raise a non-KeyboardInterrupt exception to trigger fast failure.
|
||||
#
|
||||
# This will prevent b from installing, which will cause the build of a
|
||||
# to be skipped.
|
||||
monkeypatch.setattr(spack.package.PackageBase, 'do_patch', _patch)
|
||||
monkeypatch.setattr(
|
||||
spack.package.PackageBase,
|
||||
'do_patch',
|
||||
_test_install_fail_fast_on_except_patch
|
||||
)
|
||||
|
||||
with pytest.raises(spack.installer.InstallError, matches=err_msg):
|
||||
installer.install(fail_fast=True)
|
||||
with pytest.raises(inst.InstallError, match='mock patch failure'):
|
||||
installer.install()
|
||||
|
||||
out = str(capsys.readouterr())
|
||||
assert 'Skipping build of a' in out
|
||||
|
|
@ -844,7 +1034,8 @@ def test_install_lock_failures(install_mockery, monkeypatch, capfd):
|
|||
def _requeued(installer, task):
|
||||
tty.msg('requeued {0}' .format(task.pkg.spec.name))
|
||||
|
||||
spec, installer = create_installer('b')
|
||||
const_arg = installer_args(['b'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
# Ensure never acquire a lock
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', _not_locked)
|
||||
|
|
@ -852,9 +1043,6 @@ def _requeued(installer, task):
|
|||
# Ensure don't continually requeue the task
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued)
|
||||
|
||||
# Skip the actual installation though should never reach it
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _noop)
|
||||
|
||||
installer.install()
|
||||
out = capfd.readouterr()[0]
|
||||
expected = ['write locked', 'read locked', 'requeued']
|
||||
|
|
@ -864,22 +1052,21 @@ def _requeued(installer, task):
|
|||
|
||||
def test_install_lock_installed_requeue(install_mockery, monkeypatch, capfd):
|
||||
"""Cover basic install handling for installed package."""
|
||||
def _install(installer, task, **kwargs):
|
||||
tty.msg('{0} installing'.format(task.pkg.spec.name))
|
||||
const_arg = installer_args(['b'], {})
|
||||
b, _ = const_arg[0]
|
||||
installer = create_installer(const_arg)
|
||||
b_pkg_id = inst.package_id(b.package)
|
||||
|
||||
def _prep(installer, task, keep_prefix, keep_stage, restage):
|
||||
installer.installed.add('b')
|
||||
tty.msg('{0} is installed' .format(task.pkg.spec.name))
|
||||
def _prep(installer, task):
|
||||
installer.installed.add(b_pkg_id)
|
||||
tty.msg('{0} is installed' .format(b_pkg_id))
|
||||
|
||||
# also do not allow the package to be locked again
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked',
|
||||
_not_locked)
|
||||
|
||||
def _requeued(installer, task):
|
||||
tty.msg('requeued {0}' .format(task.pkg.spec.name))
|
||||
|
||||
# Skip the actual installation though should never reach it
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
|
||||
tty.msg('requeued {0}' .format(inst.package_id(task.pkg)))
|
||||
|
||||
# Flag the package as installed
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_prepare_for_install', _prep)
|
||||
|
|
@ -887,10 +1074,8 @@ def _requeued(installer, task):
|
|||
# Ensure don't continually requeue the task
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued)
|
||||
|
||||
spec, installer = create_installer('b')
|
||||
|
||||
installer.install()
|
||||
assert 'b' not in installer.installed
|
||||
assert b_pkg_id not in installer.installed
|
||||
|
||||
out = capfd.readouterr()[0]
|
||||
expected = ['is installed', 'read locked', 'requeued']
|
||||
|
|
@ -902,14 +1087,11 @@ def test_install_read_locked_requeue(install_mockery, monkeypatch, capfd):
|
|||
"""Cover basic read lock handling for uninstalled package with requeue."""
|
||||
orig_fn = inst.PackageInstaller._ensure_locked
|
||||
|
||||
def _install(installer, task, **kwargs):
|
||||
tty.msg('{0} installing'.format(task.pkg.spec.name))
|
||||
|
||||
def _read(installer, lock_type, pkg):
|
||||
tty.msg('{0}->read locked {1}' .format(lock_type, pkg.spec.name))
|
||||
return orig_fn(installer, 'read', pkg)
|
||||
|
||||
def _prep(installer, task, keep_prefix, keep_stage, restage):
|
||||
def _prep(installer, task):
|
||||
tty.msg('preparing {0}' .format(task.pkg.spec.name))
|
||||
assert task.pkg.spec.name not in installer.installed
|
||||
|
||||
|
|
@ -919,16 +1101,14 @@ def _requeued(installer, task):
|
|||
# Force a read lock
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_ensure_locked', _read)
|
||||
|
||||
# Skip the actual installation though should never reach it
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
|
||||
|
||||
# Flag the package as installed
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_prepare_for_install', _prep)
|
||||
|
||||
# Ensure don't continually requeue the task
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_requeue_task', _requeued)
|
||||
|
||||
spec, installer = create_installer('b')
|
||||
const_arg = installer_args(['b'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
installer.install()
|
||||
assert 'b' not in installer.installed
|
||||
|
|
@ -939,28 +1119,55 @@ def _requeued(installer, task):
|
|||
assert exp in ln
|
||||
|
||||
|
||||
def test_install_dir_exists(install_mockery, monkeypatch, capfd):
|
||||
def test_install_dir_exists(install_mockery, monkeypatch):
|
||||
"""Cover capture of install directory exists error."""
|
||||
err = 'Mock directory exists error'
|
||||
def _install(installer, task):
|
||||
raise dl.InstallDirectoryAlreadyExistsError(task.pkg.prefix)
|
||||
|
||||
def _install(installer, task, **kwargs):
|
||||
raise dl.InstallDirectoryAlreadyExistsError(err)
|
||||
# Ensure raise the desired exception
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
|
||||
|
||||
const_arg = installer_args(['b'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
err = 'already exists'
|
||||
with pytest.raises(dl.InstallDirectoryAlreadyExistsError, match=err):
|
||||
installer.install()
|
||||
|
||||
b, _ = const_arg[0]
|
||||
assert inst.package_id(b.package) in installer.installed
|
||||
|
||||
|
||||
def test_install_dir_exists_multi(install_mockery, monkeypatch, capfd):
|
||||
"""Cover capture of install directory exists error for multiple specs."""
|
||||
def _install(installer, task):
|
||||
raise dl.InstallDirectoryAlreadyExistsError(task.pkg.prefix)
|
||||
|
||||
# Skip the actual installation though should never reach it
|
||||
monkeypatch.setattr(inst.PackageInstaller, '_install_task', _install)
|
||||
|
||||
spec, installer = create_installer('b')
|
||||
# Use two packages to ensure multiple specs
|
||||
const_arg = installer_args(['b', 'c'], {})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
with pytest.raises(dl.InstallDirectoryAlreadyExistsError, match=err):
|
||||
with pytest.raises(inst.InstallError, match='Installation request failed'):
|
||||
installer.install()
|
||||
|
||||
assert 'b' in installer.installed
|
||||
err = capfd.readouterr()[1]
|
||||
assert 'already exists' in err
|
||||
for spec, install_args in const_arg:
|
||||
pkg_id = inst.package_id(spec.package)
|
||||
assert pkg_id in installer.installed
|
||||
|
||||
|
||||
def test_install_skip_patch(install_mockery, mock_fetch):
|
||||
"""Test the path skip_patch install path."""
|
||||
spec, installer = create_installer('b')
|
||||
spec_name = 'b'
|
||||
const_arg = installer_args([spec_name],
|
||||
{'fake': False, 'skip_patch': True})
|
||||
installer = create_installer(const_arg)
|
||||
|
||||
installer.install(fake=False, skip_patch=True)
|
||||
installer.install()
|
||||
|
||||
assert 'b' in installer.installed
|
||||
spec, install_args = const_arg[0]
|
||||
assert inst.package_id(spec.package) in installer.installed
|
||||
|
|
|
|||
22
var/spack/repos/builtin.mock/packages/depb/package.py
Normal file
22
var/spack/repos/builtin.mock/packages/depb/package.py
Normal file
|
|
@ -0,0 +1,22 @@
|
|||
# Copyright 2013-2020 Lawrence Livermore National Security, LLC and other
|
||||
# Spack Project Developers. See the top-level COPYRIGHT file for details.
|
||||
#
|
||||
# SPDX-License-Identifier: (Apache-2.0 OR MIT)
|
||||
|
||||
from spack import *
|
||||
|
||||
|
||||
class Depb(AutotoolsPackage):
|
||||
"""Simple package with one build dependency"""
|
||||
|
||||
homepage = "http://www.example.com"
|
||||
url = "http://www.example.com/a-1.0.tar.gz"
|
||||
|
||||
version('1.0', '0123456789abcdef0123456789abcdef')
|
||||
|
||||
depends_on('b')
|
||||
|
||||
def install(self, spec, prefix):
|
||||
# sanity_check_prefix requires something in the install directory
|
||||
# Test requires overriding the one provided by `AutotoolsPackage`
|
||||
mkdirp(prefix.bin)
|
||||
Loading…
Reference in a new issue