Despite trying very hard to keep dicts out of our hash algorithm, we seem
to still accidentally add them in ways that the tests can't catch. This
can cause errors when hashes are not computed deterministically.
This fixes an error we saw with Python 3.5, where dictionary iteration
order is random. In this instance, we saw a bug when reading Spack
environment lockfiles -- The load would fail like this:
```
...
File "/sw/spack/lib/spack/spack/environment.py", line 1249, in concretized_specs
yield (s, self.specs_by_hash[h])
KeyError: 'qcttqplkwgxzjlycbs4rfxxladnt423p'
```
This was because the hashes differed depending on whether we wrote `path`
or `module` first when recomputing the build hash as part of reading a
Spack lockfile. We can fix it by ensuring a determistic iteration order.
- [x] Fix two places (one that caused an issue, and one that did
not... yet) where our to_node_dict-like methods were using regular python
dicts.
- [x] Also add a check that statically analyzes our to_node_dict
functions and flags any that use Python dicts.
The test found the two errors fixed here, specifically:
```
E AssertionError: assert [] == ['Use syaml_dict instead of ...pack/spack/spec.py:1495:28']
E Right contains more items, first extra item: 'Use syaml_dict instead of dict at /Users/gamblin2/src/spack/lib/spack/spack/spec.py:1495:28'
E Full diff:
E - []
E + ['Use syaml_dict instead of dict at '
E + '/Users/gamblin2/src/spack/lib/spack/spack/spec.py:1495:28']
```
and
```
E AssertionError: assert [] == ['Use syaml_dict instead of ...ack/architecture.py:359:15']
E Right contains more items, first extra item: 'Use syaml_dict instead of dict at /Users/gamblin2/src/spack/lib/spack/spack/architecture.py:359:15'
E Full diff:
E - []
E + ['Use syaml_dict instead of dict at '
E + '/Users/gamblin2/src/spack/lib/spack/spack/architecture.py:359:15']
```
Rework Spack's continuous integration workflow to be environment-based.
- Add the `spack ci` command, which replaces the many scripts in `bin/`
- `spack ci` decouples the CI workflow from the spack instance:
- CI is defined in a spack environment
- environment is in its own (single) git repository, separate from Spack
- spack instance used to run the pipeline is up to the user
- A new `gitlab-ci` section in environments allows users to configure how
specs in the environment should be mapped to runners
- Compilers can be bootstrapped in the new pipeline workflow
- Add extensive documentation on pipelines (see `pipelines.rst` for further details)
- Add extensive tests for pipeline code
* Update and fix samtools
This PR adds samtools-1.10 and sets the htlib directory so that the
spack built htslib can be used. This PR also arranges the dependencies
so that the htslib sequence is grouped on its own. Finally, the bzip2
dependency is removed and python and perl run dependencies are added.
* Fix samtools when built with ncurses+termlib
* The CI flake8 tests require lowercase variable
Interestingly, this did not show up when I ran `spack flake8` locally.
* Reorder GNU mirrors (#14395)
As @adamjstewart commented in #14395, GNU suggests to use
their mirror. So reorder the mirror to the top.
GNU Doc: https://www.gnu.org/prep/ftp.en.html
* Use spack.util.url.join for URLs in GNU mirrors (#14395)
One should not use os.path.join for URLs. This does only
work on POSIX systems.
Instead use spack.util.url.join.
So every part in spack uses the same url joining method.
Unfortunately UCX 1.7.0 is appearing in RPMS before it's officially released.
There's a problem with Open MPI 4.0.x where x < 3 and this version of UCX,
namely that the UCT BTL fails to compile.
See https://github.com/open-mpi/ompi/issues/7128
This patch works around the problem by disabling the build of the UCT BTL
for releases 4.0.0 to 4.0.2.
add hppritcha (me) as maintainer
Signed-off-by: Howard Pritchard <howardp@lanl.gov>
* Add +cfitsio variant to wcslib dependency
* Replace ncurses dependency with readline dependency
casacore explicitly may depend on readline, not ncurses
* Add workaround for casacore's readline dependency
casacore optionally depends upon readline, but it's CMakeLists.txt provides no
user control over whether or not readline becomes a dependency. As readline is
often present by default on systems, it's better for this package to explicitly
depend on readline in order to prevent linking to whatever system version of the
library happens to be found during the build process. This should be considered
a workaround until casacore's CMakeLists.txt is fixed.
* Apply workaround for casacore's dependency on SOFA
Similar to the issues with casacore's readline dependency, casacore's optional
dependency on SOFA does not provide the user with a means of controlling the
dependency during build time. Unlike the readline library, the SOFA library is
unlikely to exist on most systems by default. As the SOFA dependency is only
optionally used for testing casacore, requiring it by default is not a good
workaround. Until casacore's CMakeLists.txt is fixed, this variant has been
removed to avoid unexpected library dependencies in the installed package.
* Add newer casacore versions
* Add mpokorny to maintainer field
- The suite-sparse author publishes new versions starting with 5.5.0 on GitHub, see https://github.com/DrTimothyAldenDavis/SuiteSparse/releases and http://faculty.cse.tamu.edu/davis/SuiteSparse/
- change spack to download from there
- updated sha256 checksums from GitHub for all available releases
- For versions 5.4.0, 5.5.0, 5.6.0 there is a slightly different compilation necessary: first `make default` then `make install`.
Summary of the version changes (+ added, -removed [because not available on GitHub]):
```
+ 5.6.0
+ 5.5.0
+ 5.4.0
5.3.0
5.2.0
+ 5.1.2
5.1.0
+ 5.0.0
+ 4.5.6
4.5.5
- 4.5.4
4.5.3
- 4.5.1
```
to disabled use of libunwind. Without this mesa fails to build
using recent Cray compilers - cce 9 and higher - on aarch64 systems.
Signed-off-by: Howard Pritchard <hppritcha@gmail.com>
It seems that stable versions of perl also install a `perlX.Y.Z` binary.
However, it seems that this binary can hang if used in conjunction with
Spack's sbang workaround, as observed during automake's build.
* Update py-csvkit
This PR updates the py-csvkit package. This version requires a python
stack based on agate and this PR includes the new dependency packages.
- py-agate-dbf
- py-agate-excel
- py-agate-sql
- py-agate
- py-dbfread
- py-isodate
- py-leather
- py-parsedatetime
- py-python-slugify
- py-pytimeparse
- py-text-unidecode
* Replace the copy/pasted apostrophes
Python 2 can not process the copy/pasted apostrophes so replace them
with standard single quote character.
* Add version constraints on dependencies
* Add version contraint to graphviz patches
This PR restricts the graphviz version that the patches for building
with the Intel compiler apply to. The two patches that were needed for
building graphviz-2.40.1 with the Intel compiler are not needed for
graphviz-2.42.2.
* Adjust the qt dependencies
The qt5 patch is only needed for graphviz-2.40.1. However, that version
will only compile with GCC-6 or greater.
* add variant for enabling testing
* add variant for enabling testing
* enable tests and clean up other options
* enable tests and clean up other options
* add numbered versions
* add numbered versions
* updates to avoid enable_tests variant; correct versioning
* updates to avoid enable_tests variant; correct versioning
* fixes for style
* appropriate partitioners are enabled if 'all' is specified - so no need to check in spec
* revert accidental change to copyright
* remove erroneously re-introduced line about tests
* new spack recipe for build Jali - unstructured mesh infrastructure for multiphysics applications
* remove the +parallel condition for mstk, update 1.1.1 sha256sum and whitespace cleanup
* reformat description
* cut down description
* Fixes:
1. MPI_THREAD_MULTIPLE problem with OpenMPI and UCX.
Changes:
1. OpenMPI provides two new depends_on options which result in UCX being compiled with multiple threads support. One implicit when OpenMPI 3.x is used, MPI_THREAD_MULTIPLE is enabled by default, and one explicit for OpenMPI <= 2.x, MPI_THREAD_MULTIPLE is disabled by default.
2. Extends UCX package to allow "Enable thread support in UCP and UCT" option.
3. Adds sha256 sums of UCX releases 1.6.1 and 1.2.0.
More details:
Fixes the issue with OpenMPI where programs which use MPI_THREAD_MULTIPLE will fail to execute because UCP worker didn't support it.
During the OpenMPI package installation it's the +thread_multiple spec was not propagated to UCX nor UCX handled it at all.
Now, the OpenMPI package is capable of handling +thread_multiple spec when UCX is request and the UCX package correctly handles +thread_multiple and compiles with the --enable-mt option.
Error message during runtime:
pml_ucx.c:226 Error: UCP worker does not support MPI_THREAD_MULTIPLE
* Adapts check of specs to read better and is the suggested form in the docs.
* Explicitly disables multithreading of UCX if +thread_multiple option is not used.
* Rework texlive package to install from source
This PR reworks the texlive package so that it installs from versioned
source distibution files. This is preferred over installing the binary
package for several reasons. For the binary installation:
1. Each component is downloaded, so can not use a spack mirror.
2. Changes in components over time are not reflected in spack hash.
3. Some of the binaries do not run due to glibc issues, depending on OS.
This PR keeps the binary installation as an option but it should be
considered deprecated, and probably rewmoved at some point.
This PR depends on zziplib from PR #14318.
* Fix flake8 issues
One of the perl scripts was encoded with ISO-8859-1, which caused the
sbang replacement process to fail when spack uses python3. This PR
converts the ps_scan script to UTF-8 encoding.
This PR converts ISO-8859 encoding to UTF-8 encoding for three scripts in
repeatmasker.
- the main RepeatMasker script
- SimpleBatcher.pm
- wublastToCrossmatch.pl
The ISO-8859 encoding prevented the sbang replacement of long paths when
spack uses python3.
