LLVM is the only package that explicitly sets the "termlib" variant
of ncurses and it specifies +termlib. ncurses defaults to ~termlib;
if a package depends on LLVM and ncurses, there is a concretizer bug
that incorrectly detects a constraint conflict (see #267). Setting
+termlib as the default is a stopgap measure to avoid this conflict.
If other packages were to explicitly request ~termlib in the future,
the same issue would come up again (and could not be resolved by
adjusting the default of "termlib").
Setting +termlib on ncurses moves some symbols into a separate
"libtinfo". Not all packages may be able to detect libtinfo properly
so may require an update; vim, samtools, and libedit have been
updated to use ncurses+termlib (in the case of libedit, the only
necessary action was to add a newer version where the build system
was updated to check libtinfo).
* Flake8 OK
* Update var/spack/repos/builtin/packages/py-basis-set-exchange/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-basis-set-exchange/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Added missing dependencies proposed by @adamjstewart
* Without py-versioneer
* Added py-versioneer
* Python2 for bse
* Python build error
* Update var/spack/repos/builtin/packages/py-basis-set-exchange/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Removed py-versioneer, according to @adamjstewart
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
- When compiling qrupdate with `FFLAGS=-fdefault-integer-8` it can be perfectly used for larger problem dimensions.
- Improved the readability of the file with the added rules.
* Update py-bx-python package
- add update to py-bx-python
- switch to pypi downloads
- set dependencies
* Update var/spack/repos/builtin/packages/py-bx-python/package.py
I had initially pulled version 0.8.6 and then updated that to 0.8.8 but missed the change in the python specs between those two versions.
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Fix version 0.7.4
- set dependency on python2
- add dependency on py-python-lzo
- add py-python-lzo package
- set py-numpy dependency to correspond to latest version that works
with python2
* Add constraint for py-six dependency
* Update var/spack/repos/builtin/packages/py-bx-python/package.py
Ah, I had that `when` clause in and then took it out as it did not seem to be needed. I guess it is always better to be more explicit.
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Remove py-numpy constraint
Let the concretizer catch the conflict with python2 and py-numpy
versions.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fixes & additional variant(s) for HepMC3
* Syntax
* Restore recipe for HepMC2
* Remove FIXME
* Update package.py
* Apply suggestions from code review
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-jsonschema
* Update dependencies
* Update dependencies + flake8
* Add py-pyrsistent package
* Update package.py
* Update var/spack/repos/builtin/packages/py-jsonschema/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update package.py
* Apply suggestions from code review
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Apply suggestions from code review
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add a variant to QE that suppresses upstream patching. Need in order to do ddependency patching.
* QE variant fails to build often. Set default variant to False as a user friendly change.
* QMCPACK converter patch collides with internal QE patches. Deactivate internal patches when performing dependency patching.
* Clearer description of QE patch variant that is also flake8 compliant.
* Add extra version of py-sqlalchemy
* Update package.py
* Update package.py
* Update package.py
* Update package.py
* Update package.py
* Apply suggestions from code review
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The hashing logic looks for function calls that are Spack directives.
It expects that when a Spack directive is used that it is referenced
directly by name, and that the directive function is not itself
retrieved by calling another function. When the hashing logic
encountered a function call where the function was determined
dynamically, it would fail (attempting to access a name attribute
that does not happen to exist in this case).
This updates the hashing logic to filter out function calls where the
function is determined dynamically when looking for uses of Spack
directives.
* Add extra version of py-terminado
* Update package.py
* Update var/spack/repos/builtin/packages/py-terminado/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-widgetsnbextension
* Update dependency version
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-widgetsnbextension/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-urllib3
* Update package.py
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-urllib3/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of vc
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/vc/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Mark conflicts with binutils on darwin
* Explicitly require binutils bootstrapping and mark conflict with nvptx
* Disable gold variant by default on darwin
* igv: adding package igv
* removing some remaining initial boilerplate
* changing path construction to be more correct
* adding in type for java dep, also forgot about prefix.bin etc
* Added IRPF90 package
* PEP8
* SHA256
* Update var/spack/repos/builtin/packages/py-irpf90/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
libfabric used to install fabtests only when installed
using --test. fabtests has tools that are useful on a
running system, so they should be installed always.
* Rewrote the build/install part to always install
fabtests alongside libfabric.
* Updated a few fabtests resources.
* Updated the test related stuff. Works for most versions
now.
* Include tcp and udp fabrics so that the test suite works.
Change hpctoolkit's dependency on libunwind from 2018.10.12 to 1.4:.
In libunwind, 2018.10.12 is going away in favor of 1.4-rc1 (they're
nearly identical commits).
Remove the 'gpu' version. This was a temporary branch that is now
folded into master.
* py-notebook: make py-setuptools a run dependency
The py-setuptools dependency in py-notebook needs to be a run
dependency. The following message is received if it is not in the run
environment.
Traceback (most recent call last): File "/opt/ssoft/apps/2020.1/linu
x-centos7-sandybridge/gcc-9.2.0/py-notebook-6.0.1-6usbn4c/bin/jupyter-notebook",
line 6, in <module>
from pkg_resources import load_entry_point
Module NotFoundError: No module named 'pkg_resources'
* Remove extraneous whitespace
* Add extra version of py-jedi
* Update package.py
* Update package.py
Correct dependency types
* Update var/spack/repos/builtin/packages/py-jedi/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-jedi/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-jedi/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-parso package
* Remove boilerplate from py-parso
* Flake-8
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-isort
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-isort/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-isort/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-isort/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-isort/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-dask
* Update package.py
* Add extra dependencies for py-dask+distributed
* Update package.py
* Update var/spack/repos/builtin/packages/py-heapdict/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-heapdict/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update package.py
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Flake-8
* Add patch step for py-distributed
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Fixes#9394Closes#13217.
## Background
Spack provides the ability to enable/disable parallel builds through two options: package `parallel` and configuration `build_jobs`. This PR changes the algorithm to allow multiple, simultaneous processes to coordinate the installation of the same spec (and specs with overlapping dependencies.).
The `parallel` (boolean) property sets the default for its package though the value can be overridden in the `install` method.
Spack's current parallel builds are limited to build tools supporting `jobs` arguments (e.g., `Makefiles`). The number of jobs actually used is calculated as`min(config:build_jobs, # cores, 16)`, which can be overridden in the package or on the command line (i.e., `spack install -j <# jobs>`).
This PR adds support for distributed (single- and multi-node) parallel builds. The goals of this work include improving the efficiency of installing packages with many dependencies and reducing the repetition associated with concurrent installations of (dependency) packages.
## Approach
### File System Locks
Coordination between concurrent installs of overlapping packages to a Spack instance is accomplished through bottom-up dependency DAG processing and file system locks. The runs can be a combination of interactive and batch processes affecting the same file system. Exclusive prefix locks are required to install a package while shared prefix locks are required to check if the package is installed.
Failures are communicated through a separate exclusive prefix failure lock, for concurrent processes, combined with a persistent store, for separate, related build processes. The resulting file contains the failing spec to facilitate manual debugging.
### Priority Queue
Management of dependency builds changed from reliance on recursion to use of a priority queue where the priority of a spec is based on the number of its remaining uninstalled dependencies.
Using a queue required a change to dependency build exception handling with the most visible issue being that the `install` method *must* install something in the prefix. Consequently, packages can no longer get away with an install method consisting of `pass`, for example.
## Caveats
- This still only parallelizes a single-rooted build. Multi-rooted installs (e.g., for environments) are TBD in a future PR.
Tasks:
- [x] Adjust package lock timeout to correspond to value used in the demo
- [x] Adjust database lock timeout to reduce contention on startup of concurrent
`spack install <spec>` calls
- [x] Replace (test) package's `install: pass` methods with file creation since post-install
`sanity_check_prefix` will otherwise error out with `Install failed .. Nothing was installed!`
- [x] Resolve remaining existing test failures
- [x] Respond to alalazo's initial feedback
- [x] Remove `bin/demo-locks.py`
- [x] Add new tests to address new coverage issues
- [x] Replace built-in package's `def install(..): pass` to "install" something
(i.e., only `apple-libunwind`)
- [x] Increase code coverage
* ENH: add catch2 CMake install
* add a variant allowing catch2 to be installed
via CMake, which is useful for generating a .cmake
config file for consumption by other projects
* Catch2: Simplify Package
- CMake install is also single-header for new releases
- testing triggered by Spack's test mechanism
- default to CMake build (better than simple copy, which is
just for old releases to be installed)
* Catch: Remove Variant
We can control all installs with CMake to be quick and complete.
Old versions prior to 1.7.0 will be manually installed, as the
`make install` target is missing in those.
Releases 1.7.0-1.9.3 do not expose control over test builds.
* openPMD-api: Catch Lost single_header
... variant is gone :)
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
* Finish the HepMC3 spackage update
- Update CMake requirement per latest master
- Account for Python variant, add python dependency if used
- Account for ROOT I/O variant, add ROOT dependency if used
* Please flake8
* Update and simplify julia package
The current Spack Julia package potentially installs a few julia
packages, with the installation being controlled by variants. There are
a couple of problems with this.
First, Julia handles packages very differently from systems such as R
and Python. Julia requires write access to the repository directories in
order for user installs of packages to work. If spack installs julia
packages then there will be a repository, DEPOT_PATH, in the
installation directory. If spack is used on an individual basis this
would work but would mean that package data is written to the spack
installation directory after installation. If spack is used to provide
packages for end users then user installs of julia packages will fail
due to lack of write access to the repository in the installation
directory. It seems best for spack to just install julia without any
julia packages, and drop the configuration for those packages.
Second, having spack install package as variants seems to be counter to
how spack works for other extendable systems, like R and Python. Julia
should be an extendable package in spack but it is not clear how to make
that work. As pointed out above, installing user packages requires write
access to the julia repositories, including the one in the install
directory. Essentially, a user package installation will try to update
metadata for *all* julia repositories. Furthermore, each of those
repositories, assuming one per package with spack, would need to have
the Project.toml files merged to present the package stack to julia.
Again, it seems best for spack to just install julia itself and not try
to install julia packages, at least at this time. A good discussion on
this can be found at
https://discourse.julialang.org/t/how-does-one-set-up-a-centralized-julia-installation/13922.
This PR does the following:
- adds versions 1.2.0 and 1.3.1
- removes variants that correspond to julia packages
- changes python to build dependency as it seems to only be needed for
LLVM
- the new versions can use Python-3
- removes dependencies for packages
- adds a conflict statement for Intel compiler, with comment
- add a setup_build_environment method to find GCC libraries
- make formatting consistent
- adds JULIA_CPU_TARGET option to correspond with target to help with
running julia image on hardware older than build host
- added intel build options, for when they can be used
- removed code for installing packages
- removed code for julia config that was needed for packages
Note that versions below 0.5.1 fail to build, with or without these
changes. It is not clear why that is.
* Update var/spack/repos/builtin/packages/julia/package.py
Yes, need to use correct grammar even in the midst of numbers and symbols. Good catch!
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* More cleanup of Julia package
This commit does more cleanup and sets more constraints.
- Removed release-0.4 and release-0.5. I am not sure if those are
actually useful but they are quite old and there are released versions
from the same timeframe.
- Remove the binutils variant.
- Made cmake a build dependency for versions >= 1.
- Added git as a dependency for @master.
- Limit curl dependency to released versions.
- Do not use external curl for master. When I checked, using the
external version failed but the internal curl worked.
- Versions <= 0.5.0 need an older version of openssl.
- Set conflicts directive for cxx variant.
- Added conflicts directive for needing +mkl with Intel compiler.
- Removed configuration settings as these prevented julia from working
properly in all cases that I looked at.
* Fix flake8 error
Remove 'import sys' that is no longer used.
* More dependency tweaks
This commit sets further version constraints on dependencies. It really
looks like julia requires its internal dependencies more over time.
- curl only up to 0.5.0
- openssl only up to 0.5.0
- override with system curl up to version 0.5.0
* Fix spec for curl certs
Only depending on curl through 0.5.0.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-cryptography
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-cryptography/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-cryptography/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-cryptography/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Flake-8
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
