Commit graph

450 commits

Author SHA1 Message Date
Greg Becker
9345bf81b9 Add option to minimize full debug cores. include warning message about performance (#27970)
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-12-23 16:02:09 +01:00
Harmen Stoppels
654f6839eb ci: run style unit tests only if we target develop (#27472)
Some tests assume the base branch is develop, but this branch may not
have been checked out.
2021-12-23 16:02:09 +01:00
Harmen Stoppels
d862507bcf Fix overloaded argparse keys (#27379)
Commands should not reuse option names defined in main.
2021-12-23 16:02:09 +01:00
Todd Gamblin
e13e697067
commands: spack load --list alias for spack find --loaded (#27184)
See #25249 and https://github.com/spack/spack/pull/27159#issuecomment-958163679.
This adds `spack load --list` as an alias for `spack find --loaded`.  The new command is
not as powerful as `spack find --loaded`, as you can't combine it with all the queries or
formats that `spack find` provides.  However, it is more intuitively located in the command
structure in that it appears in the output of `spack load --help`.

The idea here is that people can use `spack load --list`  for simple stuff but fall back to
`spack find --loaded` if they need more.

- add help to `spack load --list` that references `spack find`
- factor some parts of `spack find` out to be called from `spack load`
- add shell tests
- update docs

Co-authored-by: Peter Josef Scheibel <scheibel1@llnl.gov>
Co-authored-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>
2021-11-05 00:58:29 -07:00
Massimiliano Culpo
31dfad9c16 spack install: add --reuse argument 2021-11-05 00:15:47 -07:00
Massimiliano Culpo
e2744fafa1 spack concretize: add --reuse argument 2021-11-05 00:15:47 -07:00
Massimiliano Culpo
290f57c779 spack spec: add --reuse argument 2021-11-05 00:15:47 -07:00
Massimiliano Culpo
da57b8775f Update command completion 2021-11-05 00:15:47 -07:00
Tamara Dahlgren
080f1872b8
Add the spack tutorial environment as a cloud pipeline stack (#27137) 2021-11-04 15:14:46 -07:00
Massimiliano Culpo
78c08fccd5
Bootstrap GnuPG (#24003)
* GnuPG: allow bootstrapping from buildcache and sources

* Add a test to bootstrap GnuPG from binaries

* Disable bootstrapping in tests

* Add e2e test to bootstrap GnuPG from sources on Ubuntu

* Add e2e test to bootstrap GnuPG on macOS
2021-11-02 23:15:24 -07:00
Michael Kuhn
1e26e25bc8
spack arch: add --generic argument (#27061)
The `--generic` argument allows printing the best generic target for the
current machine. This can be quite handy when wanting to find the
generic architecture to use when building a shared software stack for
multiple machines.
2021-11-02 10:19:23 +01:00
Tamara Dahlgren
9d3d7c68fb
Add tag filters to spack test list (#26842) 2021-11-02 10:00:21 +01:00
Tamara Dahlgren
d4cecd9ab2
feature: add "spack tags" command (#26136)
This PR adds a "spack tags" command to output package tags or 
(available) packages with those tags. It also ensures each package
is listed in the tag cache ONLY ONCE per tag.
2021-11-01 20:40:29 +00:00
Chuck Atkins
15d407c674
ci: Enable more packages in the DVSDK CI pipeline (#27025)
* ci: Enable more packages in the DVSDK CI pipeline

* doxygen: Add conflicts for gcc bugs

* dray: Add version constraints for api breakage with newer deps
2021-11-01 08:54:50 -07:00
Scott Wittenburg
f2a36bdf14
pipelines: llvm kills the xlarge, use huge (#27079) 2021-10-29 22:01:35 +00:00
Harmen Stoppels
49034abd76
Fix exit codes posix shell wrapper (#27012)
* Correct exit code in sh wrapper

* Fix tests

* SC2069
2021-10-29 08:10:22 +00:00
Harmen Stoppels
574395af93
Fix exit codes in fish (#27028) 2021-10-29 01:10:31 +00:00
Todd Gamblin
233dabbd4f bugfix: config edit should work with a malformed spack.yaml
If you don't format `spack.yaml` correctly, `spack config edit` still fails and
you have to edit your `spack.yaml` manually.

- [x] Add some code to `_main()` to defer `ConfigFormatError` when loading the
  environment, until we know what command is being run.

- [x] Make `spack config edit` use `SPACK_ENV` instead of the config scope
  object to find `spack.yaml`, so it can work even if the environment is bad.

Co-authored-by: scheibelp <scheibel1@llnl.gov>
2021-10-28 15:37:44 -07:00
Harmen Stoppels
6d030ba137
Deactivate previous env before activating new one (#25409)
* Deactivate previous env before activating new one

Currently on develop you can run `spack env activate` multiple times to switch
between environments, but they leave traces, even though Spack only supports
one active environment at a time.

Currently:

```console
$ spack env create a
$ spack env create b
$ spack env activate -p a
[a] $ spack env activate -p b
[b] [a] $ spack env activate -p b
[a] [b] [a] $ spack env activate -p a
[a] [b] [c] $ echo $MANPATH | tr ":" "\n"
/path/to/environments/a/.spack-env/view/share/man
/path/to/environments/a/.spack-env/view/man
/path/to/environments/b/.spack-env/view/share/man
/path/to/environments/b/.spack-env/view/man
```

This PR fixes that:

```console
$ spack env activate -p a
[a] $ spack env activate -p b
[b] $ spack env activate -p a
[a] $ echo $MANPATH | tr ":" "\n"
/path/to/environments/a/.spack-env/view/share/man
/path/to/environments/a/.spack-env/view/man
```
2021-10-28 11:39:25 -07:00
Tom Scogland
87e456d59c
spack setup-env.sh: make zsh loading async compatible, and ~10x faster (in some cases) (#26120)
Currently spack is a bit of a bad actor as a zsh plugin, and it was my
fault.  The autoload and compinit should really be handled by the user,
as was made abundantly clear when I found spack was doing completion
initialization for *all* of my plugins due to a deferred setup that was
getting messed up by it.

Making this conditional took spack load time from 1.5 seconds (with
module loading disabled) to 0.029 seconds. I can actually afford to load
spack by default with this change in.

Hopefully someday we'll do proper zsh completion support, but for now
this helps a lot.

* use zsh hist expansion in place of dirname
* only run (bash)compinit if compdef/complete missing
* add zsh compiled files to .gitignore
* move changes to .in file, because spack
2021-10-28 11:32:59 -07:00
Massimiliano Culpo
3d5444fdd8
Remove documentation tests from GitHub Actions (#26981)
We moved documentation tests to readthedocs since a while,
so remove the one on GitHub.
2021-10-27 19:02:52 +02:00
Massimiliano Culpo
6063600a7b
containerize: pin the Spack version used in a container (#21910)
This PR permits to specify the `url` and `ref` of the Spack instance used in a container recipe simply by expanding the YAML schema as outlined in #20442:
```yaml
container:
  images:
    os: amazonlinux:2
    spack:
      ref: develop
      resolve_sha: true
```
The `resolve_sha` option, if true, verifies the `ref` by cloning the Spack repository in a temporary directory and transforming any tag or branch name to a commit sha. When this new ability is leveraged an additional "bootstrap" stage is added, which builds an image with Spack setup and ready to install software. The Spack repository to be used can be customized with the `url` keyword under `spack`.

Modifications:
- [x] Permit to pin the version of Spack, either by branch or tag or sha
- [x] Added a few new OSes (centos:8, amazonlinux:2, ubuntu:20.04, alpine:3, cuda:11.2.1)
- [x] Permit to print the bootstrap image as a standalone
- [x] Add documentation on the new part of the schema
- [x] Add unit tests for different use cases
2021-10-25 13:09:27 -07:00
eugeneswalker
0baec34dd7
E4S amd64 CI: add parsec (#26906) 2021-10-22 15:02:47 -06:00
eugeneswalker
3720d06e26
update E4S CI environments in preparation for 21.11 release (#26826)
* update E4S CI environments in preparation for 21.11 release

* e4s ci env: use clingo
2021-10-21 07:06:02 -07:00
Tamara Dahlgren
cc8b6ca69f
Add --preferred and --latest tospack checksum (#25830) 2021-10-20 13:38:55 +00:00
Massimiliano Culpo
4c082a5357
Relax os constraints in e4s pipelines (#26547) 2021-10-19 10:51:37 -05:00
Massimiliano Culpo
79c92062a8
Gitlab pipelines: use images from the Spack organization (#26796) 2021-10-19 14:38:39 +02:00
Harmen Stoppels
c0c9ab113e
Add spack env activate --temp (#25388)
Creates an environment in a temporary directory and activates it, which
is useful for a quick ephemeral environment:

```
$ spack env activate -p --temp
[spack-1a203lyg] $ spack add zlib
==> Adding zlib to environment /tmp/spack-1a203lyg
==> Updating view at /tmp/spack-1a203lyg/.spack-env/view
```
2021-10-11 06:56:03 -04:00
Harmen Stoppels
f28b08bf02
Remove unused --dependencies flag (#25731) 2021-10-11 10:16:11 +02:00
Scott Wittenburg
9f09156923 Retry pipeline generation jobs in certain cases 2021-10-01 10:12:37 -07:00
Scott Wittenburg
d11156f361 Add DAG scheduling to child pipelines 2021-10-01 10:12:37 -07:00
Scott Wittenburg
4637c51c7f Service jobs do not need an active environment 2021-10-01 10:12:37 -07:00
Scott Wittenburg
d233c20725 Activate concrete env dir in all service jobs 2021-10-01 10:12:37 -07:00
Scott Wittenburg
5f527075bf Use same image to build e4s as to generate it 2021-10-01 10:12:37 -07:00
Scott Wittenburg
f6fcfd2acc Use same image to build dvsdk as to generate it 2021-10-01 10:12:37 -07:00
Scott Wittenburg
ae092915ac Use default runner image for radiuss 2021-10-01 10:12:37 -07:00
Massimiliano Culpo
8ade8a77dd
Build container images on Github Actions and push to multiple registries (#26247)
Modifications:
- Modify the workflow to build container images without pushing when the workflow file itself is modified
- Strip the leading ghcr.io/spack/ from env.container env.versioned to prepare pushing to multiple registries
- Fixed CentOS 7 and Amazon Linux builds
- Login and push to Docker Hub as well as Github Action
- Add a badge to README.md with the status of docker images
2021-09-30 23:34:47 +02:00
Scott Wittenburg
45b70d9798
Pipelines: Disable ppc builds until we have resources or make it smaller (#26238) 2021-09-24 08:24:36 -07:00
Harmen Stoppels
a6bb3a66ea
Remove centos:6 image references (#26095)
This was EOL November 30th, 2020. I believe the "builds" are failing on
develop because of it.
2021-09-24 09:47:49 +02:00
Massimiliano Culpo
c52426ea7a
Make clingo the default solver (#25502)
Modifications:
- [x] Change `defaults/config.yaml`
- [x] Add a fix for bootstrapping patchelf from sources if `compilers.yaml` is empty
- [x] Make `SPACK_TEST_SOLVER=clingo` the default for unit-tests
- [x] Fix package failures in the e4s pipeline

Caveats:
1. CentOS 6 still uses the original concretizer as it can't connect to the buildcache due to issues with `ssl` (bootstrapping from sources requires a C++14 capable compiler)
1. I had to update the image tag for GitlabCI in e699f14.  
1. libtool v2.4.2 has been deprecated and other packages received some update
2021-09-14 22:44:16 -07:00
Scott Wittenburg
bf7c12b4df
Pipelines: (Re)enable E4S on Power stack (#25921)
Pipelines: (Re)enable E4S on Power stack
2021-09-14 14:55:50 -06:00
Chuck Atkins
060582a21d
ci: Add ecp-data-vis-sdk CI pipeline (#22179)
* ci: Add a minimal subset of the ECP Data & Vis SDK CI pipeline

* ci: Expand the ECP Data & Vis SDK pipeline with more variants
2021-09-13 11:34:13 -06:00
Scott Wittenburg
81962f100c
Pipelines: build kokkos-kernels on bigger instance (#25845) 2021-09-10 08:27:42 -07:00
Nathan Hanford
d83f7110d5
specs: move to new spec.json format with build provenance (#22845)
This is a major rework of Spack's core core `spec.yaml` metadata format.  It moves from `spec.yaml` to `spec.json` for speed, and it changes the format in several ways. Specifically:

1. The spec format now has a `_meta` section with a version (now set to version `2`).  This will simplify major changes like this one in the future.
2. The node list in spec dictionaries is no longer keyed by name. Instead, it is a list of records with no required key. The name, hash, etc. are fields in the dictionary records like any other.
3. Dependencies can be keyed by any hash (`hash`, `full_hash`, `build_hash`).
4. `build_spec` provenance from #20262 is included in the spec format. This means that, for spliced specs, we preserve the *full* provenance of how to build, and we can reproduce a spliced spec from the original builds that produced it.

**NOTE**: Because we have switched the spec format, this PR changes Spack's hashing algorithm.  This means that after this commit, Spack will think a lot of things need rebuilds.

There are two major benefits this PR provides:
* The switch to JSON format speeds up Spack significantly, as Python's builtin JSON implementation is orders of magnitude faster than YAML. 
* The new Spec format will soon allow us to represent DAGs with potentially multiple versions of the same dependency -- e.g., for build dependencies or for compilers-as-dependencies.  This PR lays the necessary groundwork for those features.

The old `spec.yaml` format continues to be supported, but is now considered a legacy format, and Spack will opportunistically convert these to the new `spec.json` format.
2021-09-09 01:48:30 -07:00
Todd Gamblin
c309adb4b3
url stats: add --show-issues option (#25792)
* tests: make `spack url [stats|summary]` work on mock packages

Mock packages have historically had mock hashes, but this means they're also invalid
as far as Spack's hash detection is concerned.

- [x] convert all hashes in mock package to md5 or sha256
- [x] ensure that all mock packages have a URL
- [x] ignore some special cases with multiple VCS fetchers

* url stats: add `--show-issues` option

`spack url stats` tells us how many URLs are using what protocol, type of checksum,
etc., but it previously did not tell us which packages and URLs had the issues. This
adds a `--show-issues` option to show URLs with insecure (`http`) URLs or `md5` hashes
(which are now deprecated by NIST).
2021-09-08 07:59:06 -07:00
Scott Wittenburg
a358358aa7
Update pinned OpenSSL version to 1.1.1l (#25787)
Update to the latest version of openssl, as the previous one (1.1.1k) is
now deprecated, so spack can no longer rebuild it from source.
2021-09-06 16:46:41 +00:00
Vanessasaurus
8e61f54260
start of work to add spack audit packages-https checker (#25670)
This PR will add a new audit, specifically for spack package homepage urls (and eventually
other kinds I suspect) to see if there is an http address that can be changed to https.

Usage is as follows:

```bash
$ spack audit packages-https <package>
```
And in list view:

```bash
$ spack audit list
generic:
  Generic checks relying on global variables

configs:
  Sanity checks on compilers.yaml
  Sanity checks on packages.yaml

packages:
  Sanity checks on specs used in directives

packages-https:
  Sanity checks on https checks of package urls, etc.
```

I think it would be unwise to include with packages, because when run for all, since we do requests it takes a long time. I also like the idea of more well scoped checks - likely there will be other addresses for http/https within a package that we eventually check. For now, there are two error cases - one is when an https url is tried but there is some SSL error (or other error that means we cannot update to https):

```bash
$ spack audit packages-https zoltan
PKG-HTTPS-DIRECTIVES: 1 issue found
1. Error with attempting https for "zoltan": 
    <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: Hostname mismatch, certificate is not valid for 'www.cs.sandia.gov'. (_ssl.c:1125)>
```
This is either not fixable, or could be fixed with a change to the url or (better) contacting the site owners to ask about some certificate or similar.

The second case is when there is an http that needs to be https, which is a huge issue now, but hopefully not after this spack PR.

```bash
$ spack audit packages-https xman
Package "xman" uses http but has a valid https endpoint.
```

And then when a package is fixed:

```bash
$ spack audit packages-https zlib
PKG-HTTPS-DIRECTIVES: 0 issues found.
```
And that's mostly it. :)

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-09-02 08:46:27 +02:00
David Beckingsale
156edffec2
Update versions for RAJA, CHAI, Umpire and camp (#25528) 2021-09-01 17:58:47 -07:00
Scott Wittenburg
b34f289796
Pipelines: disable power builds (#25704) 2021-08-30 17:19:42 -07:00
Harmen Stoppels
12e87ebf14
Fix fish test "framework" (#25242)
Remove broken test, see #21699
2021-08-27 10:52:00 +02:00