* Update py-bx-python package
- add update to py-bx-python
- switch to pypi downloads
- set dependencies
* Update var/spack/repos/builtin/packages/py-bx-python/package.py
I had initially pulled version 0.8.6 and then updated that to 0.8.8 but missed the change in the python specs between those two versions.
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Fix version 0.7.4
- set dependency on python2
- add dependency on py-python-lzo
- add py-python-lzo package
- set py-numpy dependency to correspond to latest version that works
with python2
* Add constraint for py-six dependency
* Update var/spack/repos/builtin/packages/py-bx-python/package.py
Ah, I had that `when` clause in and then took it out as it did not seem to be needed. I guess it is always better to be more explicit.
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Remove py-numpy constraint
Let the concretizer catch the conflict with python2 and py-numpy
versions.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fixes & additional variant(s) for HepMC3
* Syntax
* Restore recipe for HepMC2
* Remove FIXME
* Update package.py
* Apply suggestions from code review
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-jsonschema
* Update dependencies
* Update dependencies + flake8
* Add py-pyrsistent package
* Update package.py
* Update var/spack/repos/builtin/packages/py-jsonschema/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update package.py
* Apply suggestions from code review
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Apply suggestions from code review
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add a variant to QE that suppresses upstream patching. Need in order to do ddependency patching.
* QE variant fails to build often. Set default variant to False as a user friendly change.
* QMCPACK converter patch collides with internal QE patches. Deactivate internal patches when performing dependency patching.
* Clearer description of QE patch variant that is also flake8 compliant.
* Add extra version of py-sqlalchemy
* Update package.py
* Update package.py
* Update package.py
* Update package.py
* Update package.py
* Apply suggestions from code review
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
The hashing logic looks for function calls that are Spack directives.
It expects that when a Spack directive is used that it is referenced
directly by name, and that the directive function is not itself
retrieved by calling another function. When the hashing logic
encountered a function call where the function was determined
dynamically, it would fail (attempting to access a name attribute
that does not happen to exist in this case).
This updates the hashing logic to filter out function calls where the
function is determined dynamically when looking for uses of Spack
directives.
* Add extra version of py-terminado
* Update package.py
* Update var/spack/repos/builtin/packages/py-terminado/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-widgetsnbextension
* Update dependency version
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-widgetsnbextension/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-urllib3
* Update package.py
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-urllib3/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of vc
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/vc/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Mark conflicts with binutils on darwin
* Explicitly require binutils bootstrapping and mark conflict with nvptx
* Disable gold variant by default on darwin
* igv: adding package igv
* removing some remaining initial boilerplate
* changing path construction to be more correct
* adding in type for java dep, also forgot about prefix.bin etc
* Added IRPF90 package
* PEP8
* SHA256
* Update var/spack/repos/builtin/packages/py-irpf90/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
libfabric used to install fabtests only when installed
using --test. fabtests has tools that are useful on a
running system, so they should be installed always.
* Rewrote the build/install part to always install
fabtests alongside libfabric.
* Updated a few fabtests resources.
* Updated the test related stuff. Works for most versions
now.
* Include tcp and udp fabrics so that the test suite works.
Change hpctoolkit's dependency on libunwind from 2018.10.12 to 1.4:.
In libunwind, 2018.10.12 is going away in favor of 1.4-rc1 (they're
nearly identical commits).
Remove the 'gpu' version. This was a temporary branch that is now
folded into master.
* py-notebook: make py-setuptools a run dependency
The py-setuptools dependency in py-notebook needs to be a run
dependency. The following message is received if it is not in the run
environment.
Traceback (most recent call last): File "/opt/ssoft/apps/2020.1/linu
x-centos7-sandybridge/gcc-9.2.0/py-notebook-6.0.1-6usbn4c/bin/jupyter-notebook",
line 6, in <module>
from pkg_resources import load_entry_point
Module NotFoundError: No module named 'pkg_resources'
* Remove extraneous whitespace
* Add extra version of py-jedi
* Update package.py
* Update package.py
Correct dependency types
* Update var/spack/repos/builtin/packages/py-jedi/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-jedi/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-jedi/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Add py-parso package
* Remove boilerplate from py-parso
* Flake-8
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-isort
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-isort/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-isort/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-isort/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-isort/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-dask
* Update package.py
* Add extra dependencies for py-dask+distributed
* Update package.py
* Update var/spack/repos/builtin/packages/py-heapdict/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-heapdict/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update package.py
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update var/spack/repos/builtin/packages/py-distributed/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Flake-8
* Add patch step for py-distributed
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Fixes#9394Closes#13217.
## Background
Spack provides the ability to enable/disable parallel builds through two options: package `parallel` and configuration `build_jobs`. This PR changes the algorithm to allow multiple, simultaneous processes to coordinate the installation of the same spec (and specs with overlapping dependencies.).
The `parallel` (boolean) property sets the default for its package though the value can be overridden in the `install` method.
Spack's current parallel builds are limited to build tools supporting `jobs` arguments (e.g., `Makefiles`). The number of jobs actually used is calculated as`min(config:build_jobs, # cores, 16)`, which can be overridden in the package or on the command line (i.e., `spack install -j <# jobs>`).
This PR adds support for distributed (single- and multi-node) parallel builds. The goals of this work include improving the efficiency of installing packages with many dependencies and reducing the repetition associated with concurrent installations of (dependency) packages.
## Approach
### File System Locks
Coordination between concurrent installs of overlapping packages to a Spack instance is accomplished through bottom-up dependency DAG processing and file system locks. The runs can be a combination of interactive and batch processes affecting the same file system. Exclusive prefix locks are required to install a package while shared prefix locks are required to check if the package is installed.
Failures are communicated through a separate exclusive prefix failure lock, for concurrent processes, combined with a persistent store, for separate, related build processes. The resulting file contains the failing spec to facilitate manual debugging.
### Priority Queue
Management of dependency builds changed from reliance on recursion to use of a priority queue where the priority of a spec is based on the number of its remaining uninstalled dependencies.
Using a queue required a change to dependency build exception handling with the most visible issue being that the `install` method *must* install something in the prefix. Consequently, packages can no longer get away with an install method consisting of `pass`, for example.
## Caveats
- This still only parallelizes a single-rooted build. Multi-rooted installs (e.g., for environments) are TBD in a future PR.
Tasks:
- [x] Adjust package lock timeout to correspond to value used in the demo
- [x] Adjust database lock timeout to reduce contention on startup of concurrent
`spack install <spec>` calls
- [x] Replace (test) package's `install: pass` methods with file creation since post-install
`sanity_check_prefix` will otherwise error out with `Install failed .. Nothing was installed!`
- [x] Resolve remaining existing test failures
- [x] Respond to alalazo's initial feedback
- [x] Remove `bin/demo-locks.py`
- [x] Add new tests to address new coverage issues
- [x] Replace built-in package's `def install(..): pass` to "install" something
(i.e., only `apple-libunwind`)
- [x] Increase code coverage
* ENH: add catch2 CMake install
* add a variant allowing catch2 to be installed
via CMake, which is useful for generating a .cmake
config file for consumption by other projects
* Catch2: Simplify Package
- CMake install is also single-header for new releases
- testing triggered by Spack's test mechanism
- default to CMake build (better than simple copy, which is
just for old releases to be installed)
* Catch: Remove Variant
We can control all installs with CMake to be quick and complete.
Old versions prior to 1.7.0 will be manually installed, as the
`make install` target is missing in those.
Releases 1.7.0-1.9.3 do not expose control over test builds.
* openPMD-api: Catch Lost single_header
... variant is gone :)
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
* Finish the HepMC3 spackage update
- Update CMake requirement per latest master
- Account for Python variant, add python dependency if used
- Account for ROOT I/O variant, add ROOT dependency if used
* Please flake8
* Update and simplify julia package
The current Spack Julia package potentially installs a few julia
packages, with the installation being controlled by variants. There are
a couple of problems with this.
First, Julia handles packages very differently from systems such as R
and Python. Julia requires write access to the repository directories in
order for user installs of packages to work. If spack installs julia
packages then there will be a repository, DEPOT_PATH, in the
installation directory. If spack is used on an individual basis this
would work but would mean that package data is written to the spack
installation directory after installation. If spack is used to provide
packages for end users then user installs of julia packages will fail
due to lack of write access to the repository in the installation
directory. It seems best for spack to just install julia without any
julia packages, and drop the configuration for those packages.
Second, having spack install package as variants seems to be counter to
how spack works for other extendable systems, like R and Python. Julia
should be an extendable package in spack but it is not clear how to make
that work. As pointed out above, installing user packages requires write
access to the julia repositories, including the one in the install
directory. Essentially, a user package installation will try to update
metadata for *all* julia repositories. Furthermore, each of those
repositories, assuming one per package with spack, would need to have
the Project.toml files merged to present the package stack to julia.
Again, it seems best for spack to just install julia itself and not try
to install julia packages, at least at this time. A good discussion on
this can be found at
https://discourse.julialang.org/t/how-does-one-set-up-a-centralized-julia-installation/13922.
This PR does the following:
- adds versions 1.2.0 and 1.3.1
- removes variants that correspond to julia packages
- changes python to build dependency as it seems to only be needed for
LLVM
- the new versions can use Python-3
- removes dependencies for packages
- adds a conflict statement for Intel compiler, with comment
- add a setup_build_environment method to find GCC libraries
- make formatting consistent
- adds JULIA_CPU_TARGET option to correspond with target to help with
running julia image on hardware older than build host
- added intel build options, for when they can be used
- removed code for installing packages
- removed code for julia config that was needed for packages
Note that versions below 0.5.1 fail to build, with or without these
changes. It is not clear why that is.
* Update var/spack/repos/builtin/packages/julia/package.py
Yes, need to use correct grammar even in the midst of numbers and symbols. Good catch!
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* More cleanup of Julia package
This commit does more cleanup and sets more constraints.
- Removed release-0.4 and release-0.5. I am not sure if those are
actually useful but they are quite old and there are released versions
from the same timeframe.
- Remove the binutils variant.
- Made cmake a build dependency for versions >= 1.
- Added git as a dependency for @master.
- Limit curl dependency to released versions.
- Do not use external curl for master. When I checked, using the
external version failed but the internal curl worked.
- Versions <= 0.5.0 need an older version of openssl.
- Set conflicts directive for cxx variant.
- Added conflicts directive for needing +mkl with Intel compiler.
- Removed configuration settings as these prevented julia from working
properly in all cases that I looked at.
* Fix flake8 error
Remove 'import sys' that is no longer used.
* More dependency tweaks
This commit sets further version constraints on dependencies. It really
looks like julia requires its internal dependencies more over time.
- curl only up to 0.5.0
- openssl only up to 0.5.0
- override with system curl up to version 0.5.0
* Fix spec for curl certs
Only depending on curl through 0.5.0.
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-cryptography
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-cryptography/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-cryptography/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-cryptography/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Flake-8
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* py-hpcbench: new package
* obey the flake8
* address comments, fix versions.
* Update var/spack/repos/builtin/packages/py-hpcbench/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-graphviz
* Update package.py
* Update package.py
* Update var/spack/repos/builtin/packages/py-graphviz/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-ipywidgets
* Update var/spack/repos/builtin/packages/py-ipywidgets/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update package.py
* Update var/spack/repos/builtin/packages/py-ipywidgets/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ipywidgets/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ipywidgets/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ipywidgets/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ipywidgets/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-ipywidgets/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-defusedxml
* Update package.py
* Update var/spack/repos/builtin/packages/py-defusedxml/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-logilab-common
* Update package.py
* Update var/spack/repos/builtin/packages/py-logilab-common/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
From ROOT cmake output:
```
-- Checking for module 'davix>=0.6.4'
-- Found davix, version 0.6.8
CMake Warning at cmake/modules/SearchInstalledSoftware.cmake:960 (message):
Davix versions 0.6.8 to 0.7.0 have a bug and do not work with ROOT, please
upgrade to 0.7.1 or later.
```
* an argument 'buf_size' of 'h5fget_file_image_c' should be intent(out).
* correct format errors
* some modifications based on the comments from the reviewer
* Add new version of cairo
* Update var/spack/repos/builtin/packages/cairo/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add new version of ccache; update URL
* Update var/spack/repos/builtin/packages/ccache/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Fix patch applicability
* Combine patches for missing qt3 headers and remove krell variant
The variant should have always been applied.
* Restrict QT patches to actual applicable versions/situations
- I researched the associated patches so now their `when=` should more
closely match when they're actually needed.
- I sorted the patch order so they're grouped by version requirement
- I renamed the patches so they're listed by version requirements
* Added new default tau version: 2.29. Added explicit zlib build requirement. Set up environment to use use elf and libz
* Changed zlib to link dependency. Removed elf library path load (wasn't able to reproduce the need for this)
* Add extra version of py-matplotlib
* Update dependency
* Update package.py
* Update var/spack/repos/builtin/packages/py-matplotlib/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-matplotlib/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Add extra version of py-lxml
* Extra variants for py-lxml
* Update var/spack/repos/builtin/packages/py-lxml/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-lxml/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-lxml/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update var/spack/repos/builtin/packages/py-lxml/package.py
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Use separate build directory for gzip
At least on mac systems (perhaps because of a case sensitivity issue?)
gzip fails to build inside the source tree:
```
config.status: linking /var/folders/fy/x2xtwh1n7fn0_0q2kk29xkv9vvmbqb/T/spack-stage/s3j/spack-stage-gzip-1.10-iatwtuk2l5xgwmuh4pwu5bf27yezpydj/spack-src/GNUmakefile to GNUmakefile
config.status: executing depfiles commands
==> Executing phase: 'build'
==> [2020-02-14-09:32:45.502913] 'make' '-j12'
make: GNUmakefile: Too many levels of symbolic links
make: stat: GNUmakefile: Too many levels of symbolic links
make: *** No rule to make target `GNUmakefile'. Stop.
```
* Simplify build directory and add gmake dependency
Libmng only needs gzip to compress man files for distribution, so it
builds fine without it. The spack
gzip currently fails to compile.
```
config.status: linking /var/folders/fy/x2xtwh1n7fn0_0q2kk29xkv9vvmbqb/T/spack-stage/s3j/spack-stage-gzip-1.10-iatwtuk2l5xgwmuh4pwu5bf27yezpydj/spack-src/GNUmakefile to GNUmakefile
config.status: executing depfiles commands
==> Executing phase: 'build'
==> [2020-02-14-09:32:45.502913] 'make' '-j12'
make: GNUmakefile: Too many levels of symbolic links
make: stat: GNUmakefile: Too many levels of symbolic links
make: *** No rule to make target `GNUmakefile'. Stop.
```
* Modify Flang NVidia GPU variant to make use of built-in CudaPackage
* Add OpenMP Offload patch if March 2019 compiler is selected.
* Flang parallel build has a race condition.
* llvm-flang now uses built-in CudaPackage.
* Add variant for different build releases.
* Fix OpenMP target offload for NVidia GPUs.
* Additional commong flags that are needed with comments.
* NVidia BC required for libomp target requires special treatment. Use clang built in previous step to re-compile libomptarget.
* Add a new package: Metall
* Fix errors in metall/package.py
* Update var/spack/repos/builtin/packages/metall/package.py
Change to https style URL
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
* Update in metall/package.py. Change Metall to depend on Boost always
* Update in metall/package.py. Change to install Boost with the default variants
* Update metall/package.py. Removed a comment
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Add the OpenBLAS variant `+consistentFPCSR`, by default `False`, which adds the compile definition `CONSISTENT_FPCSR=1` as documented in OpenBLAS `Makefile.rule`.
This PR adds an updated version to the r-rhtslib package as well as fix
the build.
- add patches to use compiler flags from R
- add variables for bzip2 and xz dependencies
- use the spack Makeconf file when building the in-tree htslib
- make patchelf available to allow R to remove reference to temporary
installation directory in htslib shared object
- Add new version of r-rsamtools as the r-rsamtools and r-rhtlib
packages are closely paired.
* Fix run environment
Trying to install Avizo, i get "Error: NameError: name 'run_env' is not defined". Correcting it to be just "env"
* fix identation
Starting with 2020, the tar files are named v2020.0.tar.gz,
v2020.1.tar.gz, etc, not 2020_U1.tar.gz.
https://github.com/intel/tbb/releases
The previous commit (7a10478708) fixed the checksum mismatch, but
didn't update url_for_version (my bad).
UnifyFS no longer has an option to depend on numa. This removes the
numa variant, dependency, and associated conflict.
This commit also
- renames the `pmpi` variant to the more appropriate `auto-mount`
- changes the preferred version to the most recent release
