Commit graph

461 commits

Author SHA1 Message Date
victorusu
18615b1485
Add setdefault option to tcl module (#14686)
This commit introduces the command

spack module tcl setdefault <package>

similar to the one already available for lmod

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-12-17 10:05:32 +01:00
Christian Goll
bd0ffa8a3c
Added opensuse/leap:15 to spack containerize (#27837)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-12-16 09:47:15 +00:00
Greg Becker
5319b6e3b1
Add option to minimize full debug cores. include warning message about performance (#27970)
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-12-15 08:52:53 +01:00
Massimiliano Culpo
d17511a806
Refactor "spack buildcache" command (#27776)
This PR is meant to move code with "business logic" from `spack.cmd.buildcache` to appropriate core modules[^1]. 

Modifications:
- [x] Add `spack.binary_distribution.push` to create a binary package from a spec and push it to a mirror
- [x] Add `spack.binary_distribution.install_root_node` to install only the root node of a concrete spec from a buildcache (may check the sha256 sum if it is passed in as input)
- [x] Add `spack.binary_distribution.install_single_spec` to install a single concrete spec from a buildcache
- [x] Add `spack.binary_distribution.download_single_spec` to download a single concrete spec from a buildcache to a local destination
- [x] Add `Spec.from_specfile` that construct a spec given the path of a JSON or YAML spec file
- [x] Removed logic from `spack.cmd.buildcache`
- [x] Removed calls to `spack.cmd.buildcache` in `spack.bootstrap`
- [x] Deprecate `spack buildcache copy` with a message that says it will be removed in v0.19.0

[^1]: The rationale is that commands should be lightweight wrappers of the core API, since that helps with both testing and scripting (easier mocking and no need to invoke `SpackCommand`s in a script).
2021-12-10 10:23:14 +01:00
eugeneswalker
b9a2e71a8b
e4s ci: expand +cuda specs (#27272) 2021-12-08 16:13:45 -08:00
kwryankrattiger
203ccdd976
LLVM: Revert build_llvm_dylib to llvm_dylib (#27761) 2021-12-03 23:47:05 +01:00
Massimiliano Culpo
270ba10962
spack flake8: remove deprecated command (#27290)
The "spack flake8" command wwas deprecated in favor
of "spack style". The deprecation wwarning is in the
0.17.X series, so removing it for v0.18.x
2021-11-24 14:20:11 -08:00
Joseph Snyder
d759612523
Add connection specification to mirror creation (#24244)
* Add connection specification to mirror creation

This allows each mirror to contain information about the credentials
used to access it.

Update command and tests based on comments

Switch to only "long form" flags for the s3 connection information.
Use the "any" function instead of checking for an empty list when looking
for s3 connection information.

Split test to use the access token separately from the access id and key.
Use long flag form in test.

Add endpoint_url to available S3 options.

Extend the special parameters for an S3 mirror to accept the
endpoint_url parameter.

Add a test.

* Add connection information per URL not per mirror

Expand the mirror-based connection information to be per-URL.
This will allow a user to specify different S3 connection information
for both the fetch and the push URLs.

Add a parameter for "profile", another way of storing the id/secret pair.

* Switch from "access_profile" to "profile"
2021-11-19 15:28:34 -05:00
Michael Davis
3375db12a5
Adding --reuse to dev-build command. (#27487) 2021-11-19 09:25:45 +01:00
Harmen Stoppels
8f7640dbef
ci: run style unit tests only if we target develop (#27472)
Some tests assume the base branch is develop, but this branch may not
have been checked out.
2021-11-18 13:00:39 +01:00
Harmen Stoppels
40a6ac62d3
llvm: introduce [build/link]_llvm_dylib (#27450)
Apart from building a single dylib for LLVM, users should also be able
to link tools against it.
2021-11-17 20:10:59 -07:00
Massimiliano Culpo
c5dd3265ed
Update the Public key of the tutorial (#27370) 2021-11-12 11:46:02 +01:00
Harmen Stoppels
9637ed05f5
Fix overloaded argparse keys (#27379)
Commands should not reuse option names defined in main.
2021-11-11 23:34:18 -08:00
eugeneswalker
a35d3b895b
e4s ci: enable tau +mpi +python variants (#27273) 2021-11-11 17:14:31 -08:00
Todd Gamblin
e13e697067
commands: spack load --list alias for spack find --loaded (#27184)
See #25249 and https://github.com/spack/spack/pull/27159#issuecomment-958163679.
This adds `spack load --list` as an alias for `spack find --loaded`.  The new command is
not as powerful as `spack find --loaded`, as you can't combine it with all the queries or
formats that `spack find` provides.  However, it is more intuitively located in the command
structure in that it appears in the output of `spack load --help`.

The idea here is that people can use `spack load --list`  for simple stuff but fall back to
`spack find --loaded` if they need more.

- add help to `spack load --list` that references `spack find`
- factor some parts of `spack find` out to be called from `spack load`
- add shell tests
- update docs

Co-authored-by: Peter Josef Scheibel <scheibel1@llnl.gov>
Co-authored-by: Richarda Butler <39577672+RikkiButler20@users.noreply.github.com>
2021-11-05 00:58:29 -07:00
Massimiliano Culpo
31dfad9c16 spack install: add --reuse argument 2021-11-05 00:15:47 -07:00
Massimiliano Culpo
e2744fafa1 spack concretize: add --reuse argument 2021-11-05 00:15:47 -07:00
Massimiliano Culpo
290f57c779 spack spec: add --reuse argument 2021-11-05 00:15:47 -07:00
Massimiliano Culpo
da57b8775f Update command completion 2021-11-05 00:15:47 -07:00
Tamara Dahlgren
080f1872b8
Add the spack tutorial environment as a cloud pipeline stack (#27137) 2021-11-04 15:14:46 -07:00
Massimiliano Culpo
78c08fccd5
Bootstrap GnuPG (#24003)
* GnuPG: allow bootstrapping from buildcache and sources

* Add a test to bootstrap GnuPG from binaries

* Disable bootstrapping in tests

* Add e2e test to bootstrap GnuPG from sources on Ubuntu

* Add e2e test to bootstrap GnuPG on macOS
2021-11-02 23:15:24 -07:00
Michael Kuhn
1e26e25bc8
spack arch: add --generic argument (#27061)
The `--generic` argument allows printing the best generic target for the
current machine. This can be quite handy when wanting to find the
generic architecture to use when building a shared software stack for
multiple machines.
2021-11-02 10:19:23 +01:00
Tamara Dahlgren
9d3d7c68fb
Add tag filters to spack test list (#26842) 2021-11-02 10:00:21 +01:00
Tamara Dahlgren
d4cecd9ab2
feature: add "spack tags" command (#26136)
This PR adds a "spack tags" command to output package tags or 
(available) packages with those tags. It also ensures each package
is listed in the tag cache ONLY ONCE per tag.
2021-11-01 20:40:29 +00:00
Chuck Atkins
15d407c674
ci: Enable more packages in the DVSDK CI pipeline (#27025)
* ci: Enable more packages in the DVSDK CI pipeline

* doxygen: Add conflicts for gcc bugs

* dray: Add version constraints for api breakage with newer deps
2021-11-01 08:54:50 -07:00
Scott Wittenburg
f2a36bdf14
pipelines: llvm kills the xlarge, use huge (#27079) 2021-10-29 22:01:35 +00:00
Harmen Stoppels
49034abd76
Fix exit codes posix shell wrapper (#27012)
* Correct exit code in sh wrapper

* Fix tests

* SC2069
2021-10-29 08:10:22 +00:00
Harmen Stoppels
574395af93
Fix exit codes in fish (#27028) 2021-10-29 01:10:31 +00:00
Todd Gamblin
233dabbd4f bugfix: config edit should work with a malformed spack.yaml
If you don't format `spack.yaml` correctly, `spack config edit` still fails and
you have to edit your `spack.yaml` manually.

- [x] Add some code to `_main()` to defer `ConfigFormatError` when loading the
  environment, until we know what command is being run.

- [x] Make `spack config edit` use `SPACK_ENV` instead of the config scope
  object to find `spack.yaml`, so it can work even if the environment is bad.

Co-authored-by: scheibelp <scheibel1@llnl.gov>
2021-10-28 15:37:44 -07:00
Harmen Stoppels
6d030ba137
Deactivate previous env before activating new one (#25409)
* Deactivate previous env before activating new one

Currently on develop you can run `spack env activate` multiple times to switch
between environments, but they leave traces, even though Spack only supports
one active environment at a time.

Currently:

```console
$ spack env create a
$ spack env create b
$ spack env activate -p a
[a] $ spack env activate -p b
[b] [a] $ spack env activate -p b
[a] [b] [a] $ spack env activate -p a
[a] [b] [c] $ echo $MANPATH | tr ":" "\n"
/path/to/environments/a/.spack-env/view/share/man
/path/to/environments/a/.spack-env/view/man
/path/to/environments/b/.spack-env/view/share/man
/path/to/environments/b/.spack-env/view/man
```

This PR fixes that:

```console
$ spack env activate -p a
[a] $ spack env activate -p b
[b] $ spack env activate -p a
[a] $ echo $MANPATH | tr ":" "\n"
/path/to/environments/a/.spack-env/view/share/man
/path/to/environments/a/.spack-env/view/man
```
2021-10-28 11:39:25 -07:00
Tom Scogland
87e456d59c
spack setup-env.sh: make zsh loading async compatible, and ~10x faster (in some cases) (#26120)
Currently spack is a bit of a bad actor as a zsh plugin, and it was my
fault.  The autoload and compinit should really be handled by the user,
as was made abundantly clear when I found spack was doing completion
initialization for *all* of my plugins due to a deferred setup that was
getting messed up by it.

Making this conditional took spack load time from 1.5 seconds (with
module loading disabled) to 0.029 seconds. I can actually afford to load
spack by default with this change in.

Hopefully someday we'll do proper zsh completion support, but for now
this helps a lot.

* use zsh hist expansion in place of dirname
* only run (bash)compinit if compdef/complete missing
* add zsh compiled files to .gitignore
* move changes to .in file, because spack
2021-10-28 11:32:59 -07:00
Massimiliano Culpo
3d5444fdd8
Remove documentation tests from GitHub Actions (#26981)
We moved documentation tests to readthedocs since a while,
so remove the one on GitHub.
2021-10-27 19:02:52 +02:00
Massimiliano Culpo
6063600a7b
containerize: pin the Spack version used in a container (#21910)
This PR permits to specify the `url` and `ref` of the Spack instance used in a container recipe simply by expanding the YAML schema as outlined in #20442:
```yaml
container:
  images:
    os: amazonlinux:2
    spack:
      ref: develop
      resolve_sha: true
```
The `resolve_sha` option, if true, verifies the `ref` by cloning the Spack repository in a temporary directory and transforming any tag or branch name to a commit sha. When this new ability is leveraged an additional "bootstrap" stage is added, which builds an image with Spack setup and ready to install software. The Spack repository to be used can be customized with the `url` keyword under `spack`.

Modifications:
- [x] Permit to pin the version of Spack, either by branch or tag or sha
- [x] Added a few new OSes (centos:8, amazonlinux:2, ubuntu:20.04, alpine:3, cuda:11.2.1)
- [x] Permit to print the bootstrap image as a standalone
- [x] Add documentation on the new part of the schema
- [x] Add unit tests for different use cases
2021-10-25 13:09:27 -07:00
eugeneswalker
0baec34dd7
E4S amd64 CI: add parsec (#26906) 2021-10-22 15:02:47 -06:00
eugeneswalker
3720d06e26
update E4S CI environments in preparation for 21.11 release (#26826)
* update E4S CI environments in preparation for 21.11 release

* e4s ci env: use clingo
2021-10-21 07:06:02 -07:00
Tamara Dahlgren
cc8b6ca69f
Add --preferred and --latest tospack checksum (#25830) 2021-10-20 13:38:55 +00:00
Massimiliano Culpo
4c082a5357
Relax os constraints in e4s pipelines (#26547) 2021-10-19 10:51:37 -05:00
Massimiliano Culpo
79c92062a8
Gitlab pipelines: use images from the Spack organization (#26796) 2021-10-19 14:38:39 +02:00
Harmen Stoppels
c0c9ab113e
Add spack env activate --temp (#25388)
Creates an environment in a temporary directory and activates it, which
is useful for a quick ephemeral environment:

```
$ spack env activate -p --temp
[spack-1a203lyg] $ spack add zlib
==> Adding zlib to environment /tmp/spack-1a203lyg
==> Updating view at /tmp/spack-1a203lyg/.spack-env/view
```
2021-10-11 06:56:03 -04:00
Harmen Stoppels
f28b08bf02
Remove unused --dependencies flag (#25731) 2021-10-11 10:16:11 +02:00
Scott Wittenburg
9f09156923 Retry pipeline generation jobs in certain cases 2021-10-01 10:12:37 -07:00
Scott Wittenburg
d11156f361 Add DAG scheduling to child pipelines 2021-10-01 10:12:37 -07:00
Scott Wittenburg
4637c51c7f Service jobs do not need an active environment 2021-10-01 10:12:37 -07:00
Scott Wittenburg
d233c20725 Activate concrete env dir in all service jobs 2021-10-01 10:12:37 -07:00
Scott Wittenburg
5f527075bf Use same image to build e4s as to generate it 2021-10-01 10:12:37 -07:00
Scott Wittenburg
f6fcfd2acc Use same image to build dvsdk as to generate it 2021-10-01 10:12:37 -07:00
Scott Wittenburg
ae092915ac Use default runner image for radiuss 2021-10-01 10:12:37 -07:00
Massimiliano Culpo
8ade8a77dd
Build container images on Github Actions and push to multiple registries (#26247)
Modifications:
- Modify the workflow to build container images without pushing when the workflow file itself is modified
- Strip the leading ghcr.io/spack/ from env.container env.versioned to prepare pushing to multiple registries
- Fixed CentOS 7 and Amazon Linux builds
- Login and push to Docker Hub as well as Github Action
- Add a badge to README.md with the status of docker images
2021-09-30 23:34:47 +02:00
Scott Wittenburg
45b70d9798
Pipelines: Disable ppc builds until we have resources or make it smaller (#26238) 2021-09-24 08:24:36 -07:00
Harmen Stoppels
a6bb3a66ea
Remove centos:6 image references (#26095)
This was EOL November 30th, 2020. I believe the "builds" are failing on
develop because of it.
2021-09-24 09:47:49 +02:00