Commit graph

992 commits

Author SHA1 Message Date
Alec Scott
8f48bf7195
rust: add v1.75.0 & v1.74.0, merge related variants into +dev, add rust-analyzer (#41903)
* Add rust-analyzer as variant to rust build

* Expose cargo module only when +cargo

* rust: add v1.74.0 and v1.75.0 and remove variants in favor of +dev

* [@spackbot] updating style on behalf of alecbcs

* Fix variant typo

---------

Co-authored-by: alecbcs <alecbcs@users.noreply.github.com>
2024-01-04 11:08:20 -08:00
Todd Gamblin
3118647802
Update copyright year to 2024 (#41919)
It was time to run `spack license update-copyright-year` again.
2024-01-02 09:21:30 +01:00
kwryankrattiger
a1e86781bd
CI: Fix timing search paths to ignore bootstrap (#41893) 2023-12-28 20:43:12 -08:00
kwryankrattiger
d52fda6015
CI: Fix timing search paths to ignore bootstrap (#40677) 2023-12-28 16:24:37 -08:00
Todd Gamblin
24d12c632c
spack gc: add options for environments and build dependencies (#41731)
This adds a few options to `spack gc`.

One to give you a little more control over dependencies:

* `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well.

And two more to make working with environments easier:

* `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment.

* `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.:

      spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #...

- [x] rework `unused_specs()` method on DB to add options for roots and deptypes
- [x] add `all_hashes()` method on DB
- [x] rework `spack gc` command to add 3 more options
- [x] tests
2023-12-26 14:52:10 -08:00
Harmen Stoppels
7cb873fb87
Revert "CI: Disable downloading artifacts from upstream jobs (#41432)" (#41843)
This reverts commit a43156a861.
2023-12-23 00:21:19 +01:00
Harmen Stoppels
8371bb4e19 gcc-runtime: add separate package for gcc runtime libs
The gcc-runtime package adds a separate node for gcc's dynamic runtime
libraries.

This should help with:

1. binary caches where rpaths for compiler support libs cannot be
   relocated because the compiler is missing on the target system
2. creating "minimal" container images

The package is versioned like `gcc` (in principle it could be
unversioned, but Spack doesn't always guarantee not mixing compilers)
2023-12-21 12:22:58 -08:00
Harmen Stoppels
e7f897f959
ci: use "strong preference" idiom for compilers (#41806)
to avoid duplication of conflicts / requirements in config
2023-12-21 12:50:35 +01:00
Harmen Stoppels
16e27ba4a6
spack buildcache push --tag: create container image with multiple roots (#41077)
This PR adds a flag `--tag/-t` to `buildcache push`, which you can use like

```
$ spack mirror add my-oci-registry oci://example.com/hello/world
$ spack -e my_env buildcache push --base-image ubuntu:22.04 --tag my_custom_tag my-oci-registry
```

and lets users ship a full, installed environment as a minimal container image where each image layer is one Spack package, on top of a base image of choice. The image can then be used as

```
$ docker run -it --rm example.com/hello/world:my_custom_tag
```

Apart from environments, users can also pick arbitrary installed spec from their database, for instance:

```
$ spack buildcache push --base-image ubuntu:22.04 --tag some_specs my-oci-registry gcc@12 cmake
$ docker run -it --rm example.com/hello/world:some_specs
```

It has many advantages over `spack containerize`:

1. No external tools required (`docker`, `buildah`, ...)
2. Creates images from locally installed Spack packages (No need to rebuild inside `docker build`, where troubleshooting build failures is notoriously hard)
3. No need for multistage builds (Spack just tarballs existing installations of runtime deps)
4. Reduced storage size / composability: when pushing multiple environments with common specs, container image layers are shared.
5. Automatic build cache: later `spack install` of the env elsewhere speeds up since the containerized environment is a build cache
2023-12-20 11:31:41 +01:00
Peter Scheibel
5d50ad3941
"spack diff": add ignore option for dependencies (#41711)
* add trim function to `Spec` and `--ignore` option to 'spack diff'

Allows user to compare two specs while ignoring the sub-DAG of a particular dependency, e.g.

spack diff --ignore=mpi --ignore=zlib trilinos/abcdef trilinos/fedcba

to focus on differences closer to the root of the software stack
2023-12-19 16:37:44 -08:00
kwryankrattiger
a43156a861
CI: Disable downloading artifacts from upstream jobs (#41432)
* CI: Disable downloading artifacts from upstream jobs

* CI: Default .base-jobs are `when:manual`
2023-12-19 15:53:28 -07:00
Harmen Stoppels
4f8b856145
e4s: add julia (#41768) 2023-12-19 18:17:54 +01:00
Massimiliano Culpo
2ef8d09fc7
spack config get/blame: with no args, show entire config
This PR changes the default behavior of `spack config get` and `spack config blame`
to print a flattened version of the entire spack configuration, including any active 
environment, if the commands are invoked with no section arguments.

The new behavior is used in Gitlab CI to help debug CI configuration, but it can also
be useful when asking for more information in issues, or when simply debugging Spack.
2023-12-19 01:26:53 -08:00
Peter Scheibel
14c7bfe9ce
spack develop: convert to config (#35273)
Convert the 'develop' section of an environment to a dedicated configuration section.
This means for example that instead of having to define `develop` specs in the
`spack.yaml`, the environment can `include:` another `develop.yaml` configuration
which specifies which specs should be developed in the environment.

This change is not expected to be disruptive given that existing environment `spack.yaml`
files will conform to the new schema.

(Update 11/28/2023) I have implemented the `develop`/`undevelop` commands in terms
of more-generic modification functions added to the `config` module: `change_or_add`
and `update_all`. It is assumed that the semantics added here (described in 11/18 update)
would be desirable to extend to other config update actions (e.g. adding compilers, 
changing package requirements, adding mirrors).

(Update 11/18/2023) I have updated this such that `spack develop`, and
`spack undevelop` to potentially modify all writable scopes, like 
https://github.com/spack/spack/pull/41147. https://github.com/spack/spack/pull/35307
will be useful for modifying included scopes, but generally speaking specifying a 
`--scope` will not be required for `spack develop`: `spack develop` will add new 
develop specs to whatever scope already has develop specs defined, or to the
highest-priority writable scope (which should be the env scope).

TODOs:

- [x] If you `spack undevelop` a package which is mentioned at multiple layers of
      configuration, then currently this would only modify one of them. That's not
      technically a new issue (has always existed for configuration modification), but
      may be confusing to users when presented via an interface other than `spack config set`
- [x] Need to add (or confirm) the ability to modify individual config files by providing
      a path (rather than using a scope identifier as a key to retrieve associated config).
- [x] `spack develop` adds new develop specs to the scope that defines them
      (potentially skipping higher priority scopes to e.g. augment included scope files)

---------

Co-authored-by: scheibelp <scheibelp@users.noreply.github.com>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-12-18 00:47:53 -08:00
Harmen Stoppels
8c29e90fa9
Build cache: make signed/unsigned a mirror property (#41507)
* Add `signed` property to mirror config

* make unsigned a tri-state: true/false overrides mirror config, none takes mirror config

* test commands

* Document this

* add a test
2023-12-11 15:14:59 -06:00
Jordan Galby
bb03ce7281
Do not use depfile in bootstrap (#41458)
- we don't have a fallback if make is not installed
- we assume file system locking works
- we don't verify that make is gnu make (bootstrapping fails on FreeBSD as a result)
- there are some weird race conditions in writing spack.yaml on concurrent spack install
- the view is updated after every package install instead of post environment install.
2023-12-07 10:09:49 +00:00
Harmen Stoppels
3cefd73fcc
spack buildcache check: use same interface as push (#41378) 2023-12-05 12:44:50 +01:00
Adam J. Stewart
ab50aa61db
py-keras: add v3.0.0 (#41356)
* py-keras: add v3.0.0

* Older keras actually requires protobuf

* Correct url_for_version

* Capitalization is important

* Keep pil and pydot deps
2023-12-01 18:23:58 +00:00
Tom Scogland
dd607d11d5
developer tools stack try 2 (#40921)
* developer tools stack try 2

This version is actually in use locally and has largely stabilized, at
least on x86.  Some packages are still a challenge on ppc64le, but maybe
worth keeping this working as a set.

* add packages, try to get container with newer gcc

* remove reuse: true

* try to get cmake to build on medium, 25 minutes is too long

* add lsd package and add to dev tools stack

* clean up fzf dependency and sorting

* Update share/spack/gitlab/cloud_pipelines/stacks/developer_tools/spack.yaml

* cuda: add 12.3.0 (#40827)

* Switch to dashes

* yet more underscores

---------

Co-authored-by: Paul R. C. Kent <kentpr@ornl.gov>
2023-11-30 18:32:21 +00:00
Luc Berger
450f938056
kokkos: add v4.2.00 (#41203)
* Kokkos: adding version 4.2.00 to the package
* Kokkos: adding AMD GPU arch
* kokkos@4.2.00 +sycl: patch numeric traits unit test

---------

Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2023-11-29 22:47:15 +00:00
eugeneswalker
430b2dff5c
e4s ci: disable gpu test stack (#41296) 2023-11-28 18:02:00 +01:00
Alec Scott
18efd808da
GoPackage: add new build system for Go packages (#41164)
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
Co-authored-by: Wouter Deconinck <wdconinc@gmail.com>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2023-11-28 10:33:46 +01:00
Alec Scott
c482534c1d
CargoPackage: add new build system for Cargo packages (#41192)
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
2023-11-27 20:15:16 +00:00
Andrey Perestoronin
3d744e7c95
intel-oneapi 2024.0.0: added new version to packages (#41135)
* oneapi 2024.0.0 release

* oneapi v2 directory support and some cleanups

* sycl abi change requires 2024 compilers for packages that use sycl

---------

Co-authored-by: Robert Cohn <robert.s.cohn@intel.com>
2023-11-27 13:13:04 -05:00
eugeneswalker
2870b6002c
e4s oneapi stack: turn on +sycl: ginkgo, heffte, petsc, upcxx, warpx (#41157)
* e4s oneapi stack: turn on +sycl: ginkgo, heffte, petsc, upcxx, warpx

* comment out warpx; build fails; add note
2023-11-18 22:21:20 -08:00
Massimiliano Culpo
2ac128a3ad
Add papyrus to the list of broken tests (#40923)
* Disable papyrus in the neoverse v1 pipeline
   See https://gitlab.spack.io/spack/spack/-/jobs/8983875
   The job is hanging on tests for 6 hrs.
* Add papyrus to broken tests instead of removing it
2023-11-14 07:37:29 -08:00
Greg Becker
13abfb7013
spack deconcretize command (#38803)
We have two ways to concretize now:
* `spack concretize` concretizes only the root specs that are not concrete in the environment.
* `spack concretize -f` eliminates all cached concretization data and reconcretizes the *entire* environment.

This PR adds `spack deconcretize`, which eliminates cached concretization data for a spec.  This allows
users greater control over what is preserved from their `spack.lock` file and what is reused when not
using `spack concretize -f`.  If you want to update a spec installed in your environment, you can call
`spack deconcretize` on it, and that spec and any relevant dependents will be removed from the lock file.

`spack concretize` has two options:
* `--root`: limits deconcretized specs to *specific* roots in the environment. You can use this to
  deconcretize exactly one root in a `unify: false` environment.  i.e., if `foo` root is a dependent
  of `bar`, both roots, `spack deconcretize bar` will *not* deconcretize `foo`.
* `--all`: deconcretize *all* specs that match the input spec. By default `spack deconcretize`
  will complain about multiple matches, like `spack uninstall`.
2023-11-10 14:55:35 -08:00
Todd Gamblin
f0ced1af42
info: rework spack info command to display variants better (#40998)
This changes variant display to use a much more legible format, and to use screen space
much better (particularly on narrow terminals). It also adds color the variant display
to match other parts of `spack info`.

Descriptions and variant value lists that were frequently squished into a tiny column
before now have closer to the full terminal width.

This change also preserves any whitespace formatting present in `package.py`, so package
maintainers can make easer-to-read descriptions of variant values if they want. For
example, `gasnet` has had a nice description of the `conduits` variant for a while, but
it was wrapped and made illegible by `spack info`. That is now fixed and the original
newlines are kept.

Conditional variants are grouped by their when clauses by default, but if you do not
like the grouping, you can display all the variants in order with `--variants-by-name`.
I'm not sure when people will prefer this, but it makes it easier to tell that a
particular variant is/isn't there. I do think grouping by `when` is the better default.
2023-11-10 12:31:28 +01:00
eugeneswalker
383ec19a0c
Revert "Deactivate Cray sles, due to unavailable runner (#40291)" (#40910)
This reverts commit 4b06862a7f.
2023-11-09 12:24:18 -08:00
Massimiliano Culpo
afc693645a
tcl: filter compiler wrappers to avoid pointing to Spack (#40946) 2023-11-08 19:38:41 +01:00
Greg Becker
0b344e0fd3
tutorial stack: update for changes to the basics section for SC23 (#40942) 2023-11-07 23:46:57 -08:00
Harmen Stoppels
c1f134e2a0
tutorial: use lmod@8.7.18 because @8.7.19: has bugs (#40939) 2023-11-07 23:04:45 +01:00
Harmen Stoppels
3a5c1eb5f3
tutorial pipeline: force gcc@12.3.0 (#40937) 2023-11-07 20:53:44 +01:00
Harmen Stoppels
4ce80b95f3
spack compiler find --[no]-mixed-toolchain (#40902)
Currently there's some hacky logic in the AppleClang compiler that makes
it also accept `gfortran` as a fortran compiler if `flang` is not found.

This is guarded by `if sys.platform` checks s.t. it only applies to
Darwin.

But on Linux the feature of detecting mixed toolchains is highly
requested too, cause it's rather annoying to run into a failed build of
`openblas` after dozens of minutes of compiling its dependencies, just
because clang doesn't have a fortran compiler.

In particular in CI where the system compilers may change during system
updates, it's typically impossible to fix compilers in a hand-written
compilers.yaml config file: the config will almost certainly be outdated
sooner or later, and maintaining one config file per target machine and
writing logic to select the correct config is rather undesirable too.

---

This PR introduces a flag `spack compiler find --mixed-toolchain` that
fills out missing `fc` and `f77` entries in `clang` / `apple-clang` by
picking the best matching `gcc`.

It is enabled by default on macOS, but not on Linux, matching current
behavior of `spack compiler find`.

The "best matching gcc" logic and compiler path updates are identical to
how compiler path dictionaries are currently flattened "horizontally"
(per compiler id). This just adds logic to do the same "vertically"
(across different compiler ids).

So, with this change on Ubuntu 22.04:

```
$ spack compiler find --mixed-toolchain
==> Added 6 new compilers to /home/harmen/.spack/linux/compilers.yaml
    gcc@13.1.0  gcc@12.3.0  gcc@11.4.0  gcc@10.5.0  clang@16.0.0  clang@15.0.7
==> Compilers are defined in the following files:
    /home/harmen/.spack/linux/compilers.yaml

```

you finally get:

```
compilers:
- compiler:
    spec: clang@=15.0.7
    paths:
      cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    flags: {}
    operating_system: ubuntu23.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: clang@=16.0.0
    paths:
      cc: /usr/bin/clang-16
      cxx: /usr/bin/clang++-16
      f77: /usr/bin/gfortran
      fc: /usr/bin/gfortran
    flags: {}
    operating_system: ubuntu23.04
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
```

The "best gcc" is automatically default system gcc, since it has no
suffixes / prefixes.
2023-11-06 15:17:31 -08:00
Harmen Stoppels
3c641c8509
spack env activate: create & activate default environment without args (#40756)
This PR implements the concept of "default environment", which doesn't have to be
created explicitly. The aim is to lower the barrier for adopting environments.

To (create and) activate the default environment, run

```
$ spack env activate
```

This mimics the behavior of

```
$ cd
```

which brings you to your home directory.

This is not a breaking change, since `spack env activate` without arguments
currently errors. It is similar to the already existing `spack env activate --temp`
command which always creates an env in a temporary directory, the difference
is that the default environment is a managed / named environment named `default`.

The name `default` is not a reserved name, it's just that `spack env activate`
creates it for you if you don't have it already.

With this change, you can get started with environments faster:

```
$ spack env activate [--prompt]
$ spack install --add x y z
```

instead of

```
$ spack env create default
==> Created environment 'default in /Users/harmenstoppels/spack/var/spack/environments/default
==> You can activate this environment with:
==>   spack env activate default
$ spack env activate [--prompt] default 
$ spack install --add x y z
```

Notice that Spack supports switching (but not stacking) environments, so the
parallel with `cd` is pretty clear:

```
$ spack env activate named_env
$ spack env status
==> In environment named_env
$ spack env activate
$ spack env status
==> In environment default
```
2023-11-05 22:53:26 -08:00
Todd Gamblin
f6b23b4653
bugfix: compress aliases for first command in completion (#40890)
This completes to `spack concretize`:

```
spack conc<tab>
```

but this still gets hung up on the difference between `concretize` and `concretise`:

```
spack -e . conc<tab>
```

We were checking `"$COMP_CWORD" = 1`, which tracks the word on the command line
including any flags and their args, but we should track `"$COMP_CWORD_NO_FLAGS" = 1` to
figure out if the arg we're completing is the first real command.
2023-11-05 10:15:37 +00:00
Tamara Dahlgren
c9dfb9b0fd
Environments: Add support for including definitions files (#33960)
This PR adds support for including separate definitions from `spack.yaml`.

Supporting the inclusion of files with definitions enables user to make
curated/standardized collections of packages that can re-used by others.
2023-11-05 00:47:06 -07:00
Massimiliano Culpo
f51dad976e
hdf5-vol-async: better specify dependency condition (#40882) 2023-11-04 20:31:52 +01:00
eugeneswalker
fd22d109a6
sundials +sycl: add cxxflags=-fsycl via flag_handler (#40845) 2023-11-04 08:55:19 -05:00
eugeneswalker
d4a1618e07
tau: update 2.33 hash, add syscall variant (#40851)
Co-authored-by: wspear <wjspear@gmail.com>
2023-11-03 07:58:00 +01:00
Harmen Stoppels
8b0ab67de4
depfile: deal with empty / non-concrete env (#40816) 2023-11-02 16:04:35 +01:00
Harmen Stoppels
343ed8a3fa
force color in subshell if not SPACK_COLOR (#40782) 2023-10-31 22:27:00 +01:00
Harmen Stoppels
6933e1c3cb
ci: bump tutorial image and toolchain (#40795) 2023-10-31 12:58:33 +01:00
Harmen Stoppels
160bfd881d
tutorial: replace zlib -> gmake to avoid deprecated versions (#40769) 2023-10-31 10:04:53 +01:00
Harmen Stoppels
b1b8500eba
ci: print colored specs in concretization progress (#40711) 2023-10-30 15:29:27 +01:00
eugeneswalker
ed7274a4d0
e4s ci stacks: add exago specs (#40712)
* e4s ci: add exago +cuda, +rocm builds

* exago: rename 5-18-2022-snapshot to snapshot.5-18-2022

* disable exago +rocm for non-external rocm ci install

* note that hiop +rocm fails to find hip libraries when they are spack-installed
2023-10-27 11:15:11 -07:00
Harmen Stoppels
195f965076
OCI buildcache (#38358)
Credits to @ChristianKniep for advocating the idea of OCI image layers
being identical to spack buildcache tarballs.

With this you can configure an OCI registry as a buildcache:

```console 
$ spack mirror add my_registry oci://user/image # Dockerhub

$ spack mirror add my_registry oci://ghcr.io/haampie/spack-test # GHCR

$ spack mirror set --push --oci-username ... --oci-password ... my_registry  # set login credentials
```

which should result in this config:

```yaml
mirrors:
  my_registry:
    url: oci://ghcr.io/haampie/spack-test
    push:
      access_pair: [<username>, <password>]
```

It can be used like any other registry

```
spack buildcache push my_registry [specs...]
```

It will upload the Spack tarballs in parallel, as well as manifest + config
files s.t. the binaries are compatible with `docker pull` or `skopeo copy`.

In fact, a base image can be added to get a _runnable_ image:

```console
$ spack buildcache push --base-image ubuntu:23.04 my_registry python
Pushed ... as [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack

$ docker run --rm -it [image]:python-3.11.2-65txfcpqbmpawclvtasuog4yzmxwaoia.spack
```

which should really be a game changer for sharing binaries.

Further, all content-addressable blobs that are downloaded and verified
will be cached in Spack's download cache. This should make repeated
`push` commands faster, as well as `push` followed by a separate
`update-index` command.

An end to end example of how to use this in Github Actions is here:

**https://github.com/haampie/spack-oci-buildcache-example**


TODO:

- [x] Generate environment modifications in config so PATH is set up
- [x] Enrich config with Spack's `spec` json (this is allowed in the OCI specification)
- [x] When ^ is done, add logic to create an index in say `<image>:index` by fetching all config files (using OCI distribution discovery API)
- [x] Add logic to use object storage in an OCI registry in `spack install`.
- [x] Make the user pick the base image for generated OCI images.
- [x] Update buildcache install logic to deal with absolute paths in tarballs
- [x] Merge with `spack buildcache` command
- [x] Merge #37441 (included here)
- [x] Merge #39077 (included here)
- [x] #39187 + #39285
- [x] #39341
- [x] Not a blocker: #35737 fixes correctness run env for the generated container images

NOTE:

1. `oci://` is unfortunately taken, so it's being abused in this PR to mean "oci type mirror". `skopeo` uses `docker://` which I'd like to avoid, given that classical docker v1 registries are not supported.
2. this is currently `https`-only, given that basic auth is used to login. I _could_ be convinced to allow http, but I'd prefer not to, given that for a `spack buildcache push` command multiple domains can be involved (auth server, source of base image, destination registry). Right now, no urllib http handler is added, so redirects to https and auth servers with http urls will simply result in a hard failure.

CAVEATS:

1. Signing is not implemented in this PR. `gpg --clearsign` is not the nicest solution, since (a) the spec.json is merged into the image config, which must be valid json, and (b) it would be better to sign the manifest (referencing both config/spec file and tarball) using more conventional image signing tools
2. `spack.binary_distribution.push` is not yet implemented for the OCI buildcache, only `spack buildcache push` is. This is because I'd like to always push images + deps to the registry, so that it's `docker pull`-able, whereas in `spack ci` we really wanna push an individual package without its deps to say `pr-xyz`, while its deps reside in some `develop` buildcache.
3. The `push -j ...` flag only works for OCI buildcache, not for others
2023-10-27 15:30:04 +02:00
Harmen Stoppels
51275df0b1
ci: spack compiler find should list extra config scopes (#40727)
otherwise it detected pre-configured compilers in an potentially different way.
2023-10-27 09:43:01 +02:00
Xavier Delaruelle
751b64cbcd
modules: no --delim option if separator is colon character (#39010)
Update Tcl modulefile template to simplify generated `append-path`,
`prepend-path` and `remove-path` commands and improve their readability.

If path element delimiter is colon character, do not set the `--delim`
option as it is the default delimiter value.
2023-10-26 15:55:49 +02:00
Harmen Stoppels
1c8073c21f
spack checksum: show long flags in usage output (#40407) 2023-10-26 14:48:35 +02:00
Harmen Stoppels
c0a4be156c
ci: don't put compilers in config (#40700)
* ci: don't register detectable compilers

Cause they go out of sync...

* remove intel compiler, it can be detected too

* Do not run spack compiler find since compilers are registered in concretize job already

* trilinos: work around +stokhos +cuda +superlu-dist bug due to EMPTY macro
2023-10-25 11:55:04 -07:00
Harmen Stoppels
0c30418732
ci: darwin aarch64 use apple-clang-15 tag (#40706) 2023-10-25 17:35:47 +02:00
Vicente Bolea
d0982115b3
Adios2: add kokkos variant (#40623)
* adios2: update variants and dependencies

* adios2: add kokkos rocm|cuda|sycl variant

* e4s oneapi ci stack: add adios2 +sycl

* e4s ci stack: add adios2 +rocm

* [@spackbot] updating style on behalf of vicentebolea

* Apply suggestions from code review

* adios2: fixed cuda variant

* update ecp-data-vis-sdk

* Update share/spack/gitlab/cloud_pipelines/stacks/e4s-power/spack.yaml

---------

Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
Co-authored-by: vicentebolea <vicentebolea@users.noreply.github.com>
2023-10-23 13:01:57 -07:00
Harmen Stoppels
27a0425e5d
concretize separately: show concretization time per spec as they concretize when verbose (#40634) 2023-10-20 17:09:19 +02:00
Harmen Stoppels
4bade7ef96
gromacs +cp2k: build in CI (#40494)
* gromacs +cp2k: build in CI

* libxsmm: x86 only

* attempt to fix dbcsr + new mpich

* use c11 standard

* gromacs: does not depend on dbcsr

* cp2k: build with cmake in CI, s.t. dbcsr is a separate package

* cp2k: cmake patches for config files and C/C++ std

* cp2k: remove unnecessary constraints due to patch
2023-10-20 16:20:20 +02:00
Harmen Stoppels
72b36ac144
Improve setup build / run / test environment (#35737)
This adds a `SetupContext` class which is responsible for setting
package.py module globals, and computing the changes to environment
variables for the build, test or run context.

The class uses `effective_deptypes` which takes a list of specs (e.g. single
item of a spec to build, or a list of environment roots) and a context
(build, run, test), and outputs a flat list of specs that affect the
environment together with a flag in what way they do so. This list is
topologically ordered from root to leaf, so that one can be assured that
dependents override variables set by dependencies, not the other way
around.

This is used to replace the logic in `modifications_from_dependencies`,
which has several issues: missing calls to `setup_run_environment`, and
the order in which operations are applied.

Further, it should improve performance a bit in certain cases, since
`effective_deptypes` run in O(v + e) time, whereas `spack env activate`
currently can take up to O(v^2 + e) time due to loops over roots. Each
edge in the DAG is visited once by calling `effective_deptypes` with
`env.concrete_roots()`.

By marking and propagating flags through the DAG, this commit also fixes
a bug where Spack wouldn't call `setup_run_environment` for runtime
dependencies of link dependencies. And this PR ensures that Spack
correctly sets up the runtime environment of direct build dependencies.

Regarding test dependencies: in a build context they are are build-time
test deps, whereas in a test context they are install-time test deps.
Since there are no means to distinguish the build/install type test deps,
they're both.

Further changes:

- all `package.py` module globals are guaranteed to be set before any of the
  `setup_(dependent)_(run|build)_env` functions is called
- traversal order during setup: first the group of externals, then the group
  of non-externals, with specs in each group traversed topological (dependencies
  are setup before dependents)
- modules: only ever call `setup_dependent_run_environment` of *direct* link/run
   type deps
- the marker in `set_module_variables_for_package` is dropped, since we should
  call the method once per spec. This allows us to set only a cheap subset of
  globals on the module: for example it's not necessary to compute the expensive
  `cmake_args` and w/e if the spec under consideration is not the root node to be
  built.
- `spack load`'s `--only` is deprecated (it has no effect now), and `spack load x`
  now means: do everything that's required for `x` to work at runtime, which
  requires runtime deps to be setup -- just like `spack env activate`.
- `spack load` no longer loads build deps (of build deps) ...
- `spack env activate` on partially installed or broken environments: this is all
  or nothing now. If some spec errors during setup of its runtime env, you'll only
  get the unconditional variables + a warning that says the runtime changes for
  specs couldn't be applied.
- Remove traversal in upward direction from `setup_dependent_*` in packages.
  Upward traversal may iterate to specs that aren't children of the roots
  (e.g. zlib / python have hundreds of dependents, only a small fraction is
  reachable from the roots. Packages should only modify the direct dependent
  they receive as an argument)
2023-10-19 20:44:05 +02:00
Scott Wittenburg
46c1a8e4c6
gitlab ci: Rework how mirrors are configured (#39939)
Improve how mirrors are used in gitlab ci, where we have until now thought
of them as only a string.

By configuring ci mirrors ahead of time using the proposed mirror templates,
and by taking advantage of the expressiveness that spack now has for mirrors,
this PR will allow us to easily switch the protocol/url we use for fetching
binary dependencies.

This change also deprecates some gitlab functionality and marks it for
removal in Spack 0.23:

    - arguments to "spack ci generate":
        * --buildcache-destination
        * --copy-to
    - gitlab configuration options:
        * enable-artifacts-buildcache
        * temporary-storage-url-prefix
2023-10-19 11:04:59 -05:00
Harmen Stoppels
681639985a
ci: remove incorrect compilers.yaml (#40610) 2023-10-19 16:11:42 +02:00
Harmen Stoppels
294e659ae8
AutotoolsPackage / MakefilePackage: add gmake build dependency (#40380) 2023-10-18 19:56:54 +02:00
Harmen Stoppels
bd165ebc4d
Support spack env activate --with-view <name> <env> (#40549)
Currently `spack env activate --with-view` exists, but is a no-op.

So, it is not too much of a breaking change to make this redundant flag
accept a value `spack env activate --with-view <name>` which activates
a particular view by name.

The view name is stored in `SPACK_ENV_VIEW`.

This also fixes an issue where deactivating a view that was activated
with `--without-view` possibly removes entries from PATH, since now we
keep track of whether the default view was "enabled" or not.
2023-10-17 15:40:48 +02:00
Massimiliano Culpo
3935e047c6
Remove deprecated "extra_instructions" option for containers (#40365) 2023-10-12 12:12:15 +02:00
Massimiliano Culpo
861bb4d35a
Update bootstrap buildcache to support Python 3.12 (#40404)
* Add support for Python 3.12
* Use optimized build of clingo
2023-10-11 19:03:17 +02:00
eugeneswalker
37df8bfc73
e4s arm stack: duplicate and target neoverse v1 (#40369)
* e4s arm stack: duplicate and target both neoverse n1, v1

* remove neoverse_n1 target until issue #40397 is resolved
2023-10-10 14:32:51 -07:00
Alex Richert
7a358c9005
Change 'exit' to 'return' in setup-env.sh (#36137)
* Change 'exit' to 'return' in `setup-env.sh` to avoid losing shell in some cases when sourcing twice.
2023-10-06 16:19:19 -07:00
Massimiliano Culpo
e20c05fcdf
Make "minimal" the default duplicate strategy (#39621)
* Allow branching out of the "generic build" unification set

For cases like the one in https://github.com/spack/spack/pull/39661
we need to relax rules on unification sets.

The issue is that, right now, nodes in the "generic build" unification
set are unified together with their build dependencies. This was done
out of caution to avoid the risk of circular dependencies, which would
ultimately cause a very slow solve.

For build-tools like Cython, however, the build dependencies is masked
by a long chain of "build, run" dependencies that belong in the
"generic build" unification space.

To allow splitting on cases like this, we relax the rule disallowing
branching out of the "generic build" unification set.

* Fix issue with pure build virtual dependencies

Pure build virtual dependencies were not accounted properly in the
list of possible virtuals. This caused some facts connecting virtuals
to the corresponding providers to not be emitted, and in the end
lead to unsat problems.

* Fixed a few issues in packages

py-gevent: restore dependency on py-cython@3
jsoncpp: fix typo in build dependency
ecp-data-vis-sdk: update spack.yaml and cmake recipe
py-statsmodels: add v0.13.5

* Make dependency on "blt" of type "build"
2023-10-06 10:24:21 +02:00
eugeneswalker
e0e6ff5a68
Revert "cray rhel: disable due to runner issues (#40324)" (#40335)
This reverts commit bf7f54449b.
2023-10-05 10:32:52 -07:00
Harmen Stoppels
bf7f54449b
cray rhel: disable due to runner issues (#40324) 2023-10-05 08:45:33 +02:00
Scott Wittenburg
7bf6780de2
ci: pull E4S images from github instead of dockerhub (#40307) 2023-10-04 09:55:06 +02:00
eugeneswalker
91178d40f3
e4s arm: disable bricks due to target=aarch64 not being respected (#40308) 2023-10-04 09:51:14 +02:00
eugeneswalker
8746c75db0
e4s ci stacks: sync with e4s-23.08 (#40263)
* e4s amd64 gcc ci stack: sync with e4s-23.08

* e4s amd64 oneapi ci stack: sync with e4s-23.08

* e4s ppc64 gcc ci stack: sync with e4s-23.08

* add new ci stack: e4s amd64 gcc w/ external rocm

* add new ci stack: e4s arm gcc ci

* updates

* py-scipy: -fvisibility issue is resolved in 2023.1.0: #39464

* paraview oneapi fails

* comment out pkgs that fail to build on power

* fix arm stack name

* fix cabana +cuda specification

* comment out failing spces

* visit fails build on arm

* comment out slepc arm builds due to make issue

* comment out failing dealii arm builds
2023-10-03 08:12:51 -07:00
Massimiliano Culpo
4b06862a7f
Deactivate Cray sles, due to unavailable runner (#40291)
This reverts commit 0274091204.
2023-10-03 11:06:36 +02:00
G-Ragghianti
1b75651af6
Implemented +sycl for slate (#39927)
* Implemented +sycl for slate

* style

* style

* add slate +sycl to ci

* slate +sycl: explicitly specify +mpi

* comment out slate+sycl+mpi in e4s oneapi stack

* removing requirement of intel-oneapi-mpi

* Added slate+sycl to e4s oneapi stack

* Removing obsolete comment

---------

Co-authored-by: eugeneswalker <eugenesunsetwalker@gmail.com>
2023-10-02 07:35:00 -07:00
eugeneswalker
2d509dc3eb
py-scipy: -fvisibility issue is resolved in 2023.1.0: (#39464)
* py-scipy: -fvisibility issue is resolved in 2023.1.0:

* e4s oneapi ci: add py-scipy
2023-10-01 17:29:21 +00:00
eugeneswalker
68dbd25f5f
e4s cray sles ci: expand spec list (#40162)
* e4s cray sles stack: expand spec list

* remove unnecessary packages:trilinos:one_of
2023-09-30 21:32:33 -07:00
eugeneswalker
08a9345fcc
e4s ci: add packages: drishti, dxt-explorer (#39597)
* e4s ci: add packages: drishti, dxt-explorer

* e4s oneapi ci: comment out dxt-explorer until r%oneapi issue #40257 is resolved
2023-09-30 06:15:58 +09:00
Massimiliano Culpo
210d221357
Test package detection in a systematic way (#18175)
This PR adds a new audit sub-command to check that detection of relevant packages
is performed correctly in a few scenarios mocking real use-cases. The data for each 
package being tested is in a YAML file called detection_test.yaml alongside the 
corresponding package.py file.

This is to allow encoding detection tests for compilers and other widely used tools, 
in preparation for compilers as dependencies.
2023-09-29 10:24:42 +02:00
eugeneswalker
0274091204
Revert "e4s cray sles: suspend ci until machine issues resolved (#40219)" (#40238)
This reverts commit 5165524ca6.
2023-09-29 07:25:50 +02:00
eugeneswalker
5165524ca6
e4s cray sles: suspend ci until machine issues resolved (#40219) 2023-09-27 04:51:34 +02:00
eugeneswalker
95b737d923
trilinos +shylu: patch to resolve trilinos issue #12048 (#40169)
* trilinos +shylu: patch to resolve trilinos issue #12048

* e4s ci: build xyce ^trilinos+shylu

---------

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
2023-09-24 16:28:03 +09:00
eugeneswalker
daf95227bf
e4s cray rhel stack: expand to full spec list (#40111)
* e4s cray rhel stack: expand to full spec list

* comment out gasnet; require %gcc for unzip

* require openblas@0.3.20 to get around %cce error; follow up with issue report

* comment out failing specs;

* comment out axom and xyce due to errors

* improve clarity of failing specs
2023-09-22 16:04:20 +00:00
eugeneswalker
2755706115
Revert "e4s-cray: drop in ci as there are no runners (#40122)" (#40144)
This reverts commit bd8d0324a9.
2023-09-21 22:38:26 -06:00
Harmen Stoppels
501d322264
e4s: drop python 3.8 preference (#40123) 2023-09-21 15:07:21 +02:00
Harmen Stoppels
bd8d0324a9
e4s-cray: drop in ci as there are no runners (#40122) 2023-09-21 09:54:06 +00:00
Harmen Stoppels
c57e2140c2
ci: dont use nightly image tag (#40070) 2023-09-19 17:53:28 +02:00
Massimiliano Culpo
3b4ca0374e
Use process pool executors for web-crawling and retrieving archives (#39888)
Fix a race condition when searching urls, and updating a shared
set '_visited'.
2023-09-19 15:32:59 +02:00
Harmen Stoppels
7d33c36a30
ci: drop redundant packages.yaml (#40068) 2023-09-19 11:47:48 +02:00
eugeneswalker
3045ed0e43
Revert "gitlab ci: temporarily disable darwin aarch64 (#39974)" (#40052)
This reverts commit 2c13361b09.
2023-09-18 12:48:32 -06:00
Scott Wittenburg
cbabdf283c
gitlab ci: submit the correct audience for protected publish jobs (#40046) 2023-09-15 20:09:18 +00:00
Scott Wittenburg
579df768ca
gitlab ci: Put Cray binaries in public mirrors (#39938) 2023-09-14 15:11:29 -06:00
Dan LaManna
6a249944f5
Add OIDC tokens to gitlab-ci jobs (#39813)
* Add OIDC tokens to gitlab-ci jobs

This should allow us to start issuing just-in-time generated
credentials for CI jobs that need to modify binary mirrors. The "aud"
claim of the token describes what the token is allowed to do. The
claim is verified against a set of rules on the IAM role using signed
information from GitLab. See spack-infrastructure for the claim
verification logic.

---------

Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2023-09-14 09:59:25 -06:00
Harmen Stoppels
6838ee6bb8
Add efficient deptype flag and spack.deptypes module (#39472)
This commit replaces the internal representation of deptypes with `int`, which is more compact
and faster to operate with.

Double loops like:
```
any(x in ys for x in xs)
```
are replaced by constant operations bool(xs & ys), where xs and ys are dependency types. 

Global constants are exposed for convenience in `spack.deptypes`
2023-09-14 12:25:24 +02:00
Massimiliano Culpo
d50f296d4f
data-vis-sdk: make llvm~libomptarget the default (#40007)
Currently the configuration of the pipeline is such that
there are multiple "optimal" solutions. This is due to
the pipeline making ~lld the default for LLVM, but leaving
+libomptarget from package.py

Since LLVM recipe has a:

conflicts("~lld", "+libomptarget")

clingo is forced to change one of the two defaults, and
the penalty for doing so is the same for each. Hence, it
is random whether we'll get +libomptarget+lld
or ~libomptarget~lld.

This fixes it by changing also the default for libomptarget.
2023-09-14 09:19:47 +02:00
Harmen Stoppels
2c13361b09
gitlab ci: temporarily disable darwin aarch64 (#39974)
as there are no or few runners available
2023-09-13 11:11:33 +02:00
Scott Wittenburg
fa9acb6a98
ci: Tag reindex job to match the image architecture (#39958) 2023-09-12 16:33:56 -04:00
Tamara Dahlgren
c10ff27600
CI/Add superlu-dist to broken stand-alone tests due to hang (#38908)
* Add superlu-dist to broken stand-alone CI tests
* Revert "disable superlu test (#38894)"
  This reverts commit e16397b5d8.
2023-09-12 11:09:25 -07:00
Scott Wittenburg
516f0461b8
gitlab ci: Use image with awscli for rebuild-index job (#39636)
Instead of pointing to the image on DockerHub, which rate limits us and
causes pipeline failures durying busy times, use the version at ghcr.

And we might as well use the ghcr version everywhere else too.
2023-09-11 18:03:11 +02:00
Todd Gamblin
45d149c7d3
bugfix: only complete aliases for potential aliases (#39887)
Smart alias completion introduced in #39499 wasn't as smart as it needed to be, and
would complete any invalid command prefix and some env names with alias names.

- [x] don't complete aliases if there are no potential completions
      e.g., don't convert `spack isnotacommand` -> `spack concretize`

- [x] don't complete with an aliases if we're not looking at a top-level subcommand.
2023-09-08 12:51:26 +02:00
Massimiliano Culpo
39b9f214a8
Speed-up spack external find execution (#39843)
* Perform external spec detection with multiple workers

The logic to perform external spec detection has been refactored
into classes. These classes use the GoF "template" pattern to account
for the small differences between searching for "executables" and
for "libraries", while unifying the larger part of the algorithm.

A ProcessPoolExecutor is used to parallelize the work.

* Speed-up external find by tagging detectable packages automatically

Querying packages by tag is much faster than inspecting the repository,
since tags are cached. This commit adds a "detectable" tag to every
package that implements the detection protocol, and external detection
uses it to search for packages.

* Pass package names instead of package classes to workers

The slowest part of the search is importing the Python modules
associated with candidate packages. The import is done serially
before we distribute the work to the pool of executors.

This commit pushes the import of the Python module to the job
performed by the workers, and passes just the name of the packages
to the executors.

In this way imports can be done in parallel.

* Rework unit-tests for Windows

Some unit tests were doing a full e2e run of a command
just to check a input handling. Make the test more
focused by just stressing a specific function.

Mark as xfailed 2 tests on Windows, that will be fixed
by a PR in the queue. The tests are failing because we
monkeypatch internals in the parent process, but the
monkeypatching is not done in the "spawned" child
process.
2023-09-08 09:25:50 +02:00
kwryankrattiger
8ec1657136
CI Timing Statistics (#38598)
* Write timing information for installs from cache

* CI: aggregate and upload install_times.json to artifacts

* CI: Don't change root directory for artifact generation

* Flat event based timer variation

Event based timer allows for easily starting and stopping timers without
wiping sub-timer data. It also requires less branching logic when
tracking time.

The json output is non-hierarchical in this version and hierarchy is
less rigidly enforced between starting and stopping.

* Add and write timers for top level install

* Update completion

* remove unused subtimer api

* Fix unit tests

* Suppress timing summary option

* Save timers summaries to user_data artifacts

* Remove completion from fish

* Move spack python to script section

* Write timer correctly for non-cache installs

* Re-add hash to timer file

* Fish completion updates

* Fix null timer yield value

* fix type hints

* Remove timer-summary-file option

* Add "." in front of non-package timer name

---------

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-09-07 15:41:31 -05:00
Harmen Stoppels
d80bc70481
gitlab: set cache and config dir in top-level file (#39829) 2023-09-06 08:16:35 +02:00