Commit graph

28753 commits

Author SHA1 Message Date
Harmen Stoppels
1d4919924d
Add in-place RPATH replacement (#27610)
Whenever the rpath string actually _grows_, it falls back to patchelf,
when it stays the same length or gets shorter, we update it in-place,
padded with null bytes.

This PR only deals with absolute -> absolute rpath replacement. We don't
use `_build_tarball(relative=True)` in our CI. If `relative` then it falls
back to the old replacement code.

With this PR, relocation time goes down significantly, likely because patchelf
does some odd things with mmap, causing lots of overhead. Example:

- `binutils`: 700MB installed, goes from `1.91s` to `0.57s`, or `3.4x` faster.
   Relocation time: 27% -> 10% of total install time
- `llvm`: 6.8GB installed, goes from `28.56s` to `5.38`, or `5.3x` faster.
   Relocation time: 44% -> 13% of total install time

The bottleneck is now decompression.

Note: I'm somewhat confused about the "relative rpath" code paths. Right
now this PR only deals with absolute -> absolute replacement. As far as
I understand, if you embrace relative rpaths when uploading to the
buildcache, the whole point is you _don't_ want to patch rpaths on
install? So it seems fine to not expand `$ORIGIN` again imho.
2022-11-04 02:30:53 -07:00
Jordan Galby
b1f896e6c7
Fix non-parallel make under depfile jobserver (#32862)
When a package asks for non-parallel make, we need to force `make -j1` because just doing `make` will run in parallel under jobserver (e.g. `spack env depfile`).

We now always add `-j1` when asked for a non-parallel execution (even if there is no jobserver).

And each `MakeExecutable` can now ask for jobserver support or not. For example: the default `ninja` does not support jobserver so spack applies the default `-j`, but `ninja@kitware` or `ninja-fortran` does, so spack doesn't add `-j`.

Tips: you can run `SPACK_INSTALL_FLAGS=-j1 make -f spack-env-depfile.make -j8` to avoid massive job-spawning because of build tools that don't support jobserver (ninja).
2022-11-04 02:05:22 -07:00
Satish Balay
34969b7072
petsc: fix configure option to use double-hyphen (#33685) 2022-11-03 20:22:09 -06:00
Lucas
72b11c1883
Adding gegelati library package (#33686)
* testing ssh key
* test
* LR : Creating the packge to install the gegelati app
* LR : Gegelati, a TPG C++ library added and fully tested
* LR : adjusting for fork
* LR: taking out the boilerplates
* LR: taking out the rest
2022-11-03 18:34:00 -07:00
wspear
b3e41736e6
Add version 2.32 to tau package (#33702) 2022-11-03 18:24:23 -07:00
Todd Gamblin
9c5c327a6f
propagation: don't count propagated variant values as non-default (#33687)
We try to avoid non-default variant values in the concretizer, but this doesn't make
sense for variants forced to take some non-default value by variant propagation.
Counting this as a penalty effectively biases the concretizer for small specs dependency
graphs -- something we try very hard to avoid elsewhere because it can lead to very
strange decisions.

Example: with the penalty, `spack spec hdf5` will choose the default `openmpi` as its
`mpi` provider, but `spack spec hdf5 ~~shared` will choose `mpich` because it has to set
fewer non-default variant values because `mpich`'s DAG is smaller. That's not a good
reason to prefer a non-default virtual provider.

To fix this, if the user explicitly requests a non-default value to be propagated, there
shouldn't be a penalty. Variant values set on the CLI already don't count as default; we
just need to extend that to propagated values.
2022-11-03 18:26:03 -06:00
Greg Becker
79fcc0848f
Update glvis for new builder interface (#33699) 2022-11-03 17:10:39 -07:00
Harmen Stoppels
b52be75978
Experimental binding of shared ELF libraries (#31948)
Adds another post install hook that loops over the install prefix, looking for shared libraries type of ELF files, and sets the soname to their own absolute paths.

The idea being, whenever somebody links against those libraries, the linker copies the soname (which is the absolute path to the library) as a "needed" library, so that at runtime the dynamic loader realizes the needed library is a path which should be loaded directly without searching.

As a result:

1. rpaths are not used for the fixed/static list of needed libraries in the dynamic section (only for _actually_ dynamically loaded libraries through `dlopen`), which largely solves the issue that Spack's rpaths are a heuristic (`<prefix>/lib` and `<prefix>/lib64` might not be where libraries really are...)
2. improved startup times (no library search required)
2022-11-03 17:34:00 -06:00
Adrien Cotte
7fc49c42ee
Add new versions to py-clustershell (#33694) 2022-11-03 17:11:28 -05:00
snehring
5b39059472
tassel: adding version 5.2.86 (#33697) 2022-11-03 15:02:36 -07:00
snehring
e8cc1a60ea
spades: adding version 3.15.5 (#33698) 2022-11-03 14:44:31 -07:00
eugeneswalker
d2d01ea488
flux-core: allow ncurses >= 6.2 (#33599) 2022-11-03 14:44:02 -07:00
Zack Galbreath
ccc716f617
Limit the number of parallel jobs launched by Tensile (#33692) 2022-11-03 15:26:03 -06:00
Scott Wittenburg
b55509ffa8
gitlab: Prune untouched specs less aggressively (#33669)
Untouched spec pruning was added to reduce the number of specs
developers see getting rebuilt in their PR pipelines that they
don't understand.  Because the state of the develop mirror lags
quite far behind the tip of the develop branch, PRs often find
they need to rebuild things untouched by their PR.

Untouched spec pruning was previously implemented by finding all
specs in the environment with names of packages touched by the PR,
traversing in both directions the DAGS of those specs, and adding
all dependencies as well as dependents to a list of concrete specs
that should not be considered for pruning.

We found that this heuristic results in too many pruned specs, and
that dependents of touched specs must have all their dependencies
added to the list of specs that should not be considered for pruning.
2022-11-03 13:33:52 -06:00
Miroslav Stoyanov
08bee718a2
new cmake requirement (#33679) 2022-11-03 11:28:08 -07:00
Nicolas Cornu
da020d1bb8
Bump HighFive to v2.5.0 (#33691)
* Bump HighFive to v2.5.0
* Adding myself as maintainers
* fix format with black
2022-11-03 11:34:15 -06:00
Cyrus Harrison
ca93c8b57a
fides: remove unneeded variants (#32521) 2022-11-03 11:33:59 -06:00
Greg Sjaardema
16acd25053
SEACAS: further refactor (#33673)
* SEACAS: Update package.py to handle new SEACAS project name

The base project name for the SEACAS project has changed from
"SEACASProj" to "SEACAS" as of @2022-10-14, so the package
needed to be updated to use the new project name when needed.

The refactor also changes several:
    "-DSome_CMAKE_Option:BOOL=ON"
to
   define("Some_CMAKE_Option", True)

* SEACAS: Additional refactorings

* Replaced all cmake "-Dsomething=other" lines with either `define`
or `define_from_variant` functions.

Consolidated the application (fortran, legacy, all) enabling lines
into loops over the code names.  Easier to see categorization of
applications and also to add/move/remove an application

Reordered some lines; general cleanup and restructuring.

* Address flake8 issues

* Remove trailing whitespace

* Reformat using black
2022-11-03 11:42:25 -04:00
snehring
9925f3b779
isescan: add version 1.7.2.3 (#33675) 2022-11-03 09:29:59 -06:00
Mikael Simberg
68a5fe84a7
Add pika 0.10.0 (#33659)
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-11-03 12:28:38 +01:00
Harmen Stoppels
243dfe91e9
Use spack.traverse.traverse_nodes where useful (#33677) 2022-11-03 11:34:24 +01:00
Jose E. Roman
30da20c1bc
New patch release SLEPc 3.18.1 (#33661) 2022-11-03 03:58:16 -06:00
Massimiliano Culpo
0d82688903
Update metadata for bootstrapping (#33665) 2022-11-03 09:05:03 +00:00
Michael Kuhn
707e56dea8
glib: add 2.74.1 (#33650) 2022-11-03 02:50:20 -06:00
Sinan
2264b75ca0
add new package: py-pylatex (#33573)
* add new package: py-pylatex

* fix bugs

* add extras indicated in setup.py

* Update var/spack/repos/builtin/packages/py-pylatex/package.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* Update var/spack/repos/builtin/packages/py-pylatex/package.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* improvements

* remove git merge related lines

* tidy

* Update var/spack/repos/builtin/packages/py-pylatex/package.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* remove variant

* [@spackbot] updating style on behalf of Sinan81

Co-authored-by: sbulut <sbulut@3vgeomatics.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Sinan81 <Sinan81@users.noreply.github.com>
2022-11-02 22:38:17 -06:00
Greg Becker
c716c6ca95
Bugfix for spec objects modified by flag handlers (#33682)
This issue was introduced in #29761:

```
==> Installing ncurses-6.3-22hz6q6cvo3ep2uhrs3erpp2kogxncbn
==> No binary for ncurses-6.3-22hz6q6cvo3ep2uhrs3erpp2kogxncbn found: installing from source
==> Using cached archive: /spack/var/spack/cache/_source-cache/archive/97/97fc51ac2b085d4cde31ef4d2c3122c21abc217e9090a43a30fc5ec21684e059.tar.gz
==> No patches needed for ncurses
==> ncurses: Executing phase: 'autoreconf'
==> ncurses: Executing phase: 'configure'
==> ncurses: Executing phase: 'build'
==> ncurses: Executing phase: 'install'
==> Error: AttributeError: 'str' object has no attribute 'propagate'

The 'ncurses' package cannot find an attribute while trying to build from sources. This might be due to a change in Spack's package format to support multiple build-systems for a single package. You can fix this by updating the build recipe, and you can also report the issue as a bug. More information at https://spack.readthedocs.io/en/latest/packaging_guide.html#installation-procedure

/spack/lib/spack/spack/build_environment.py:1075, in _setup_pkg_and_run:
       1072        tb_string = traceback.format_exc()
       1073
       1074        # build up some context from the offending package so we can
  >>   1075        # show that, too.
       1076        package_context = get_package_context(tb)
       1077
       1078        logfile = None
```

It turns out this was caused by a bug that had been around much longer, in which the flags were passed by reference to the flag_handler, and the flag_handler was modifying the spec object, not just the flags given to the build system. The scope of this bug was limited by the forking model in Spack, which is how it went under the radar for so long.

PR includes regression test.
2022-11-02 21:09:43 -07:00
Harmen Stoppels
b652fe72d7
remove deptype_query remnants and fix incorrect deptypes kwarg (#33670)
* remove deptype_query remnants
* deptypes -> deptype

These arguments haven't existed since 2017, but `traverse` now fails on unknown **kwargs, so they have finally popped up.
2022-11-02 14:24:59 -07:00
Robert Pavel
4d5f2e3a37
Initial Version of Hypar Spackage (#33647)
Initial spackage for Hypar proxy app
2022-11-02 14:10:19 -07:00
Martin Diehl
4c535a2037
update: damask3.0.0-alpha7 (#33634)
* damask3.0.0-alpha7
* [@spackbot] updating style on behalf of MarDiehl

Co-authored-by: MarDiehl <MarDiehl@users.noreply.github.com>
2022-11-02 12:23:19 -07:00
Weiqun Zhang
2a516aadb1
amrex: add v22.11 (#33671) 2022-11-02 11:59:50 -07:00
Carson Woods
0661c1f531
Package: add new package py-fitter (#33652)
* Add new python package

* Fix isort style issues

* Add additional dependencies

* Add additional dependencies

* Remove comments

* Add additional explicit dependencies
2022-11-02 12:42:10 -06:00
Sinan
4b549560f9
package_qgis_fix_pythonpath (#33655)
* package_qgis_fix_pythonpath

* check if bindings enabled also

Co-authored-by: sbulut <sbulut@3vgeomatics.com>
2022-11-02 13:40:05 -05:00
Greg Sjaardema
d4ea74bf80
SEACAS: Update package.py to handle new SEACAS project name (#33646)
The base project name for the SEACAS project has changed from
"SEACASProj" to "SEACAS" as of @2022-10-14, so the package
needed to be updated to use the new project name when needed.

The refactor also changes several:
    "-DSome_CMAKE_Option:BOOL=ON"
to
   define("Some_CMAKE_Option", True)
2022-11-02 11:50:15 -05:00
Paul R. C. Kent
cff94f8e71
llvm: add 15.0.3, 15.0.4 (#33651)
* v1503

* v1504
2022-11-02 10:46:08 -06:00
Gregory Becker
aa4f478ab8 propagation: improve performance
This updates the propagation logic used in `concretize.lp` to avoid rules with `path()`
in the body and instead base propagation around `depends_on()`.
2022-11-02 09:43:57 -07:00
Kayla Butler
bc209c470d flags/variants: Add ++/~~/== syntax for propagation to dependencies
Currently, compiler flags and variants are inconsistent: compiler flags set for a
package are inherited by its dependencies, while variants are not. We should have these
be consistent by allowing for inheritance to be enabled or disabled for both variants
and compiler flags.

- [x] Make new (spec language) operators
- [x] Apply operators to variants and compiler flags
- [x] Conflicts currently result in an unsatisfiable spec
      (i.e., you can't propagate two conflicting values)

What I propose is using two of the currently used sigils to symbolized that the variant
or compiler flag will be inherited:

Example syntax:
- `package ++variant`
      enabled variant that will be propagated to dependencies
- `package +variant`
      enabled variant that will NOT be propagated to dependencies
- `package ~~variant`
      disabled variant that will be propagated to dependencies
- `package ~variant`
      disabled variant that will NOT be propagated to dependencies
- `package cflags==True`
      `cflags` will be propagated to dependencies
- `package cflags=True`
      `cflags` will NOT be propagated to dependencies

Syntax for string-valued variants is similar to compiler flags.
2022-11-02 09:43:57 -07:00
Filippo Spiga
ae99829af4
armpl-gcc: Pull RHEL8 package when OS is Rocky8 (#33641) 2022-11-02 16:09:28 +01:00
Todd Gamblin
a4d978be59
tests: fix group membership check in sbang tests. (#33658)
Fixes an issue on the RHEL8 UBI container where this test would fail because `gr_mem`
was empty for every entry in the `grp` DB.

You have to check *both* the `pwd` database (which has primary groups) and `grp` (which
has other gorups) to do this correctly.

- [x] update `llnl.util.filesystem.group_ids()` to do this
- [x] use it in the `sbang` test
2022-11-02 11:00:16 +01:00
Harmen Stoppels
4bad9f9b13
Consolidate DAG traversal in traverse.py, support DFS/BFS (#33406)
This PR introduces breadth-first traversal, and moves depth-first traversal
logic out of Spec's member functions, into `traverse.py`.

It introduces a high-level API with three main methods:

```python
spack.traverse.traverse_edges(specs, kwargs...)
spack.traverse.traverse_nodes(specs, kwags...)
spack.traverse.traverse_tree(specs, kwargs...)
```

with the usual `root`, `order`, `cover`, `direction`, `deptype`, `depth`, `key`,
`visited` kwargs for the first two.

What's new is that `order="breadth"` is added for breadth-first traversal.

The lower level API is not exported, but is certainly useful for advanced use
cases. The lower level API includes visitor classes for direction reversal and
edge pruning, which can be used to create more advanced traversal methods,
especially useful when the `deptype` is not constant but depends on the node
or depth. 

---

There's a couple nice use-cases for breadth-first traversal:

- Sometimes roots have to be handled differently (e.g. follow build edges of
  roots but not of deps). BFS ensures that root nodes are always discovered at
  depth 0, instead of at any depth > 1 as a dep of another root.
- When printing a tree, it would be nice to reduce indent levels so it fits in the 
  terminal, and ensure that e.g. `zlib` is not printed at indent level 10 as a 
  dependency of a build dep of a build dep -- rather if it's a direct dep of my
  package, I wanna see it at depth 1. This basically requires one breadth-first
  traversal to construct a tree, which can then be printed with depth-first traversal.
- In environments in general, it's sometimes inconvenient to have a double
  loop: first over the roots then over each root's deps, and maintain your own
  `visited` set outside. With BFS, you can simply init the queue with the
  environment root specs and it Just Works. [Example here](3ec7304699/lib/spack/spack/environment/environment.py (L1815-L1816))
2022-11-01 23:04:47 -07:00
eugeneswalker
353e31e72a
rempi %oneapi: -Wno-error=implicit-function-declaration (#33654) 2022-11-01 17:53:53 -06:00
Vicente Bolea
328addd43d
ParaView: add v5.11.0-RC2 (#33486)
Co-authored-by: eugeneswalker <38933153+eugeneswalker@users.noreply.github.com>
2022-11-01 15:49:57 -07:00
Greg Becker
f696f02a46
Unit tests: make unit tests work for aarch64 machines (#33625)
Currently, many tests hardcode to older versions of gcc for comparisons of
concretization among compiler versions. Those versions are too old to concretize for
`aarch64`-family targets, which leads to failing tests on `aarch64`.

This PR fixes those tests by updating the compiler versions used for testing.

Currently, many tests hardcode the expected architecture result in concretization to the
`x86_64` family of architectures.

This PR generalizes the tests that can be generalized, to cover multiple architecture
families. For those that test specific relationships among `x86_64`-family targets, it
ensures that concretization uses the `x86_64`-family targets in those cases.

Currently, many tests rely on the fact that `AutotoolsPackage` imposes no dependencies
on the inheriting package. That is not true on `aarch64`-family architectures.

This PR ensures that the fact `AutotoolsPackage` on `aarch64` pulls in a dependency on
`gnuconfig` is ignored when testing for the appropriate relationships among dependencies

Additionally, 5 tests currently prompt the user for input when `gpg` is available in the
user's path. This PR fixes that issue. And 7 tests fail currently when the user has a
yubikey available. This PR fixes the incorrect gpg argument causing those issues.
2022-11-01 16:25:55 -06:00
Greg Sjaardema
e0265745bc
Update command option for example (#33321)
The `spack info <package>` command does not show the `Virtual Packages:` output unless the `--virtuals` command option is passed.  Before this changes, the information that the command is supposed to be illustrating is not shown in the example and is confusing.
2022-11-01 15:29:50 -06:00
Massimiliano Culpo
75360bdc21
Allow target requirements in packages.yaml (#32528)
This PR solves the issue reported in #32471 specifically for targets and operating systems, 
by avoiding to add a default platform to anonymous specs.
2022-11-01 22:11:49 +01:00
Harmen Stoppels
230e96fbb8
Add elf parsing utility function (#33628)
Introduces `spack.util.elf.parse_elf(file_handle)`
2022-11-01 19:42:06 +00:00
Michael Kuhn
6b3ea94630
perl: add 5.36.0 (#33336) 2022-11-01 12:13:53 -07:00
Michael Kuhn
6dd2bb258c
xz: add 5.2.7 (#33338) 2022-11-01 20:02:11 +01:00
Cody Balos
1abcc8caf7
sundials: add v6.4.1, new ginkgo and kokkos variants, plus some fixes (#33644)
* sundials: add v6.4.1, new ginkgo and kokkos variants, plus some fixes
* add missing kokkos/kokkos-kernels variant
* add v6.4.0
2022-11-01 12:45:53 -06:00
Massimiliano Culpo
23aef6bb94
Let pytest-cov create the xml directly (#33619)
`coverage` sometimes failed to combine, even if there were multiple reports.
2022-11-01 19:04:45 +01:00
liuyangzhuan
973b43b1c1
superlu-dist: fix rocm variant for the master branch (#33624)
* superlu-dist: fix rocm variant for the master branch
* simplify superlu-dist
* add mpi include to HIP_HIPCC_FLAGS

Co-authored-by: liuyangzhuan <liuyangzhuan@users.noreply.github.com>
2022-11-01 10:28:01 -07:00