Commit graph

26212 commits

Author SHA1 Message Date
Alberto Invernizzi
f40f1b5c7c
Fix for spack stage command not extracting packages in custom paths (#30448) 2022-05-15 12:13:42 +02:00
Michael Kuhn
edd3cf0b17
qt: add 5.15.4 (#30656) 2022-05-14 23:34:06 -06:00
Michael Kuhn
ff03e2ef4c
uninstall: fix dependency check (#30674)
The dependency check currently checks whether there are only build
dependencies left for a particular package. However, the database also
contains uninstalled packages, which can cause the check to fail.

For instance, with `bison` and `flex` having already been uninstalled,
`m4` will have the following dependents:
```
bison ('build', 'run')--> m4
flex ('build',)--> m4
libnl ('build',)--> m4
```
`bison` and `flex` should be ignored in this case because they are not
installed anymore.

Fixes #30673
2022-05-14 18:01:29 -07:00
Zack Galbreath
bee311edf3
Update GitLab environment variable name (#30671)
Use the IAM credentials that correspond to our new binary mirror
(s3://spack-binaries vs. s3://spack-binaries-develop)
2022-05-14 16:33:32 -06:00
Jen Herting
73b69cfeec
[py-pyworld] Limiting numpy version. See: https://zenn.dev/ymd_h/articles/934a90e1468a05 (#30670) 2022-05-14 09:22:31 -05:00
Cameron Smith
4a1041dbc3
CEED v5.0 release (#29710)
* ceed50: add ceed 5.0.0 and pumi 2.2.7

* libceed-0.10

* ceed50: add omegah

* omega-h: mpi and cuda builds work

* omega-h: fix style

* New package: libfms

* New version: gslib@1.0.7

CEED: add some TODO items for the 5.0 release

* ceed: variant name consistent with package name

* LAGHOS: allow newer versions of MFEM to be used with v3.1

* LIBCEED: add missing 'install' target in 'install_targets'

* CEED: address some TODO items + some tweaks

* MFEM: add new variant for FMS (libfms)

* CEED: v5.0.0 depends on 'libfms' and 'mfem+fms'

* RATEL: add missing 'install' target in 'install_targets'

* CEED: add dependency for v5.0.0 on Ratel v0.1.2

* CEED: add Nek-related dependencies for ceed@5.0.0

* CEED: v5.0.0 depends on MAGMA v2.6.2

* libCEED: set the `CUDA_ARCH` makefile parameter

* libCEED: set the `HIP_ARCH` makefile parameter

Co-authored-by: Jed Brown <jed@jedbrown.org>
Co-authored-by: Veselin Dobrev <dobrev@llnl.gov>
Co-authored-by: Veselin Dobrev <v-dobrev@users.noreply.github.com>
2022-05-13 18:29:02 -07:00
Jen Herting
ccab7bf4fd
New package: py-pyworld (#28641)
* espnet first build with depends

* added cython>=0.24.0' and type='build'

* [py-pyworld] updated copyright

Co-authored-by: Sid Pendelberry <sid@rit.edu>
2022-05-13 17:19:58 -05:00
John W. Parent
e24e71be6a
Preserve Permissions on .zip extraction (#30407)
#24556 merged in support for Python's .zip file support via ZipFile.
However as per #30200 ZipFile does not preserve file permissions of
the extracted contents. This PR returns to using the `unzip`
executable on non-Windows systems (as was the case before #24556)
and now uses `tar` on Windows to extract .zip files.
2022-05-13 13:38:05 -07:00
kwryankrattiger
72d83a6f94
Ascent: Patch 0.8.0 for finding ADIOS2. (#30609) 2022-05-13 13:26:53 -07:00
Todd Gamblin
5cb40cbcd2 directory_layout: remove outdated checks for old DAG hash
We previously had checks in `directory_layout` to check for build-dependency
conflicts when we weren't storing build dependencies.  We don't need
those anymore; we can just rely on the DAG hash now that it includes everything
we know about each spec.

- [x] Remove vestigial code for checking installed spec against concrete spec
      in `ensure_installed()`
- [x] Remove `SpecHashCollisionError` -- if specs have the same hash now, they're
      the same as far as `DirectoryLayout` should be concerned.
- [x] Convert spec comparison to `dag_hash()` comparison when adding extensions.
2022-05-13 10:45:12 -07:00
Todd Gamblin
c93e465134 full hash: fix uninstall and gc with full hash DB
The database now stores full hashes, so we need to adjust the criteria we use to
determine if something can be uninstalled. Specifically, it's ok to uninstall thing that
have remaining build-only dependents.
2022-05-13 10:45:12 -07:00
Todd Gamblin
521c206030 concretizer: enable hash reuse with full hash
With the original DAG hash, we did not store build dependencies in the database, but
with the full DAG hash, we do. Previously, we'd never tell the concretizer about build
dependencies of things used by hash, because we never had them. Now, we have to avoid
telling the concretizer about them, or they'll unnecessarily constrain build
dependencies for new concretizations.

- [x] Make database track all dependencies included in the `dag_hash`
- [x] Modify spec_clauses so that build dependency information is optional
      and off by default.
- [x] `spack diff` asks `spec_clauses` for build dependencies for completeness
- [x] Modify `concretize.lp` so that reuse optimization doesn't affect fresh
      installations.
- [x] Modify concretizer setup so that it does *not* prioritize installed versions
      over package versions. We don't need this with reuse, so they're low priority.
- [x] Fix `test_installed_deps` for full hash and new concretizer (does not work
      for old concretizer with full hash -- leave this for later if we need it)
- [x] Move `test_installed_deps` mock packages to `builtin.mock` for easier debugging
      with `spack -m`.
- [x] Fix `test_reuse_installed_packages_when_package_def_changes` for full hash
2022-05-13 10:45:12 -07:00
Todd Gamblin
15eb98368d bugfix: tests trying to ignore package changes should use build_hash
- [x] update test to use `build_hash` instead of `dag_hash`, as we're testing for
      graph structure, and specifically NOT testing for package changes.
- [x] make hash descriptors callable on specs to simplify syntax for invoking them
- [x] make `Spec.spec_hash()` public
2022-05-13 10:45:12 -07:00
Todd Gamblin
7c1d566959 Remove all uses of runtime_hash; document lockfile formats and fix tests
This removes all but one usage of runtime hash. The runtime hash was being used to write
historical lockfiles for tests, but we don't need it for that; we can just save those
lockfiles.

- [x] add legacy lockfiles for v1, v2, v3
- [x] fix bugs with v1 lockfile tests (the dummy lockfile we were writing was not actually
      a v1 lockfile because it used the new spec file format).
- [x] remove all but one runtime_hash usage -- that one needs a small rework of the
      concretizer to really fix, as it's about separate concretization of build
      dependencies.
- [x] Document the history of the lockfile format in `environment/__init__.py`
2022-05-13 10:45:12 -07:00
Todd Gamblin
7ab46e26b5 content_hash(): make it work on abstract specs
Some test cases had to be modified in a kludgy way so that abstract specs made
concrete would have versions on them. We shouldn't *need* to do this, as the
only reason we care is because the content hash has to be able to get an archive
for a version.

This modifies the content hash so that it can be called on abstract specs,
including only relevant content.

This does NOT add a partial content hash to the DAG hash, as we do not really
want that -- we don't need in-memory spec hashes to need to load package files.
It just makes `Package.content_hash()` less prickly and tests easier to
understand.
2022-05-13 10:45:12 -07:00
Todd Gamblin
6db215dd89 spec: fix serialization, avoid double call to node_dict_with_hashes 2022-05-13 10:45:12 -07:00
Todd Gamblin
72b38851eb hashes: revert spack monitor hash changes to preserve original protocol
`spack monitor` expects a field called `spec_full_hash`, so we shouldn't change that.
Instead, we can pass a `dag_hash` (which is now the full hash) but not change the field
name.
2022-05-13 10:45:12 -07:00
Todd Gamblin
9d9e970367 fix full hash calls in spack graph 2022-05-13 10:45:12 -07:00
Todd Gamblin
283a4e6068 remove no longer needed full hash check 2022-05-13 10:45:12 -07:00
Todd Gamblin
d20cc7b124 spec: remove hashes_final as it's no longer needed.
`hashes_final` was used to indicate when a spec was concrete but possibly lacked
`full_hash` or `build_hash` fields. This was only necessary because older Spacks
didn't generate them, and we want to avoid recomputing them, as we likely do not
have the same package files as existed at concretization time.

Now, we don't need to do that -- there is only the DAG hash and specs are either
concrete and have a `dag_hash`, or not concrete and have no `dag_hash`. There's
no middle ground.
2022-05-13 10:45:12 -07:00
Scott Wittenburg
0dd373846f gitlab ci: switch over to new bucket for all stacks 2022-05-13 10:45:12 -07:00
Scott Wittenburg
c202953528 gitlab ci: Docstring methods or make them private 2022-05-13 10:45:12 -07:00
Scott Wittenburg
be0e3f4458 binary_distribution: Refactor index generation into smaller methods 2022-05-13 10:45:12 -07:00
Scott Wittenburg
fd3bb5177b env: Use order of roots to resolve DAG hash conflicts in legacy lockfiles 2022-05-13 10:45:12 -07:00
Scott Wittenburg
9de61c0197 env: enforce predictable ordering when reading lockfile
Without some enforcement of spec ordering, python 2 produced
different results in the affected test than did python 3.  This
change makes the arbitrary but reproducible decision to sort
the specs by their lockfile key alphabetically.
2022-05-13 10:45:12 -07:00
Scott Wittenburg
84cfb3b7fe spec: fix infinite recursion when computing package hash
Issue described in the following PR comment:

https://github.com/spack/spack/pull/28504#issuecomment-1051835568

Solution described in subsequent comment:

https://github.com/spack/spack/pull/28504#issuecomment-1053986132
2022-05-13 10:45:12 -07:00
Scott Wittenburg
cb0d12b9d5 Fix how environments are read from lockfile 2022-05-13 10:45:12 -07:00
Scott Wittenburg
f6e7c0b740 hashes: remove full_hash and build_hash from spack 2022-05-13 10:45:12 -07:00
Scott Wittenburg
512645ff2e environment: key by dag_hash instead of build_hash 2022-05-13 10:45:12 -07:00
Scott Wittenburg
32a2c22b2b tests: fix failing test_hash_change
The full hash appears twice in the spec dict now, replacing just
the value replaces it under "hash" and "full_hash".  Only replace
the one that appears after "full_hash".

I'm actually not sure what purpose this test served, so maybe it
could be removed, as it may be testing some distinction between
full and dag hash which no longer exists.
2022-05-13 10:45:12 -07:00
Todd Gamblin
e02020c80a Include all deps and package content in the dag_hash()
For a long time, Spack has used a coarser hash to identify packages
than it likely should. Packages are identified by `dag_hash()`, which
includes only link and run dependencies. Build dependencies are
stripped before hashing, and we have notincluded hashes of build
artifacts or the `package.py` files used to build.  This means the
DAG hash actually doesn't represent all the things Spack can build,
and it reduces reproducibility.

We did this because, in the early days, users were (rightly) annoyed
when a new version of CMake, autotools, or some other build dependency
would necessitate a rebuild of their entire stack. Coarsening the hash
avoided this issue and enabled a modicum of stability when only reusing
packages by hash match.

Now that we have `--reuse`, we don't need to be so careful. Users can
avoid unnecessary rebuilds much more easily, and we can add more
provenance to the spec without worrying that frequent hash changes
will cause too many rebuilds.

This commit starts the refactor with the following major change:

- [x] Make `Spec.dag_hash()` include build, run, and link
      dependencides and the package hash (it is now equivalent to
      `full_hash()`).

It also adds a couple of bugfixes for problems discovered during
the switch:

- [x] Don't add a `package_hash()` in `to_node_dict()` unless
      the spec is concrete (fixes breaks on abstract specs)

- [x] Don't add source ids to the package hash for packages without
      a known fetch strategy (may mock packages are like this)

- [x] Change how `Spec.patches` is memoized. Using
      `llnl.util.lang.memoized` on `Spec` objects causes specs to
      be stored in a `dict`, which means they need a hash.  But,
      `dag_hash()` now includes patch `sha256`'s via the package
      hash, which can lead to infinite recursion
2022-05-13 10:45:12 -07:00
Massimiliano Culpo
d900ac2003
Reuse concretization by default (#30396)
* Enable reuse by default in Spack
* Update documentation to match new default
* Configure pipelines not to reuse software
2022-05-13 09:11:10 -07:00
Teodor Nikolov
faa277778e
sarus: fix dependency on boost (#30659)
The program_options variant is disabled by default in Boost and
has to be enabled explicitly.
2022-05-13 08:41:41 -06:00
Harmen Stoppels
b60d3dcd29
libtree: add v3.0.4, v3.1.1 (#30658) 2022-05-13 11:35:06 +00:00
Todd Gamblin
e0bed2d6a7
tutorial stack: allow deprecated versions (#30648)
For tutorial builds, we should continue to allow deprecated builds to be installed. We
can update them as needed when we update the tutorial, but we don't need to correct them
immediately on deprecation in CI.

- [x] add `deprecated:true` to tutorial `spack.yaml` config.
2022-05-13 04:09:39 -06:00
Massimiliano Culpo
745c191d73
Gitlab pipelines: add a small legend in the logs to interpret "[x]" (#30643) 2022-05-12 21:46:35 +02:00
Chuck Atkins
42e9430fbc
silo: Cleanup the HDF5 dependency specs (#30593) 2022-05-12 15:43:11 -04:00
Chuck Atkins
f11572166f
vtk: fix hdf5 version specification (#30592) 2022-05-12 21:16:57 +02:00
Adam J. Stewart
aa6665d5ee
py-scikit-learn: add v1.1.0 (#30649) 2022-05-12 12:50:26 -06:00
Simon Pintarelli
3e8f31a068
atompaw: add version 4.2.0.0 (#30650) 2022-05-12 18:09:30 +00:00
Massimiliano Culpo
3625ea4726
Revert "atompaw: add new package (#30619)" (#30647)
This reverts commit 12b0278f08.
2022-05-12 19:47:47 +02:00
snehring
245b95223d
soapdenovo-trans: add v1.0.5 (#30629) 2022-05-12 19:14:49 +02:00
Max Zeyen
e00c8a7d98
code-server: add new versions (#30646) 2022-05-12 10:08:02 -07:00
Adam J. Stewart
97792f04e9
py-netcdf4: add v1.5.8 (#30598) 2022-05-12 09:02:27 -05:00
Frédéric Simonis
ca069f6906
precice: add v2.4.0 (#30633) 2022-05-12 15:57:53 +02:00
Qian Jianhua
35a91bdd72
fakexrandr: remove 'ldconfig' from Makefile (#30632) 2022-05-12 14:55:45 +02:00
Mikael Simberg
c866a50446
Add patch for compiling llvm ~omp_as_runtime (#30583) 2022-05-12 05:49:39 -06:00
Harmen Stoppels
3033abb5bd
Add cuda 11.7 compat bounds for gcc/clang (#30639) 2022-05-12 10:41:30 +00:00
Harmen Stoppels
d37f439557
pigz: add v2.7 (#30640) 2022-05-12 10:39:08 +00:00
dlkuehn
18710936f1
platypus: limit install to Python2 (#30624)
Co-authored-by: David Kuehn <las_dkuehn@iastate.edu>
2022-05-12 09:30:45 +02:00