* gitlab: remove requests for unreferenced packages
The packages removed in this commit are not built by any of
our current GitLab CI stacks.
* gitlab: update memory requests for "huge" packages
* gitlab: reduce memory requests for overprovisioned packages
* gitlab: more memory for py-torch (again)
* gitlab: update memory but keep CPU the same
It is useful to enable/disable stacks in order to handle turning
specific stacks on/off based on runner availability, stack stability,
testing requirements, etc.
The disabled stack list takes precedence over the enable stack list. The
assumption is that stacks that are disabled are so due to some
functionality missing or broken for that stack.
The enable stack list implicitly disables all stacks not listed in the
enable list.
The main script body is over-written for power. Putting thet timing
aggregation in the after script allows it to be called on all of the
current pipelines.
This "breaks" the deprecated schema by allowing unknown attributes
to the attributes section of the job types. The breaking change here is
that deprecated stacks will no longer ignore attributes that are unknown
but rather assume the new CI schema behavior of injecting them into the
generated CI configuration. This change is required to secure
authentication in Spack CI.
* gitlab: remove commented-out duplicate entries
* gitlab: reclassify some packages from "huge" to "large"
Our observed max memory usage for these packages is as follows:
hipblas: 7.7G
qt: 6.6G
visit: 9.7G
All of these should fit within a "large" request (currently 12G).
* gitlab: remove pango from list of huge packages
This package is not currently built by any of our CI stacks.
* gitlab: update requests for high memory packages
Refine resource requests for memory-intensive packages based on
max memory usage data.
Currently requirements allow to express "strong preferences" and "conflicts" from
configuration using a convoluted syntax:
```yaml
packages:
zlib-ng:
require:
# conflict on %clang
- one_of: ["%clang", "@:"]
# Strong preference for +shared
- any_of: ["+shared", "@:"]
```
This PR adds syntactic sugar so that the same can be written as:
```yaml
packages:
zlib-ng:
conflict:
- "%clang"
prefer:
- "+shared"
```
Preferences written in this way are "stronger" that the ones documented at:
- https://spack.readthedocs.io/en/latest/packages_yaml.html#package-preferences
* e4s ci: use latest intel/hpckit 2024 based image
* use latest container image: ecpe4s/ubuntu22.04-runner-amd64-oneapi-2024.0.0:2023.12.01
* comment out failing specs
* update to use patched container
* remove generalized package preference for intel-oneapi-mkl@2023
* change packages commented out
Like `spack change` for specs in environments, this can e.g. replace `examplespec+debug` with `examplespec~debug` in a `require:` section.
Example behavior for a config like:
```
packages:
foo:
require:
- spec: +debug
```
* `spack config change packages:foo:require:~debug` replaces `+debug` with `~debug`
* `spack config change packages:foo:require:@1.1` adds a requirement to the list
* `spack config change packages:bar:require:~debug` adds a requirement
The current `mkdir {{ paths.environment }}` will generate an error if:
* `{{ paths.environment }}` already exists, or
* `{{ paths.environment }}` is nested in non-existing dirs.
Adding `-p` to the command will make this robust to both possibilities.
Set noclobber bash option when writing manifest.
Add `--create` option to `env activate` to allow users to create and activate in one command.
---------
Co-authored-by: Wouter Deconinck <wdconinc@gmail.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: psakievich <psakievich@users.noreply.github.com>
* setup-env: Fix back and forth between two instances
* setup-env.csh: Fix SPACK_ROOT when switch to a different instance
i.e. Always look for the current SPACK_ROOT
* setup-env: Update comments
This adds options to `spack list` that allow you to list only packages from specific
repositories/namespaces, e.g.:
```console
spack list -r builtin
```
only lists packages from the `builtin` repo, while:
```console
spack list -r myrepo -r myrepo2
```
would list packages from `myrepo` and `myrepo2`, but not from `builtin`. Note that you
can use the same argument multiple times.
You can use either `-r` / `--repo` or `-N` / `--namespace`. `-N` is there to match the
corresponding option on `spack find`.
- [x] add `-r` / `--repo` / `-N` / `--namespace` argument
- [x] add test
* Add rust-analyzer as variant to rust build
* Expose cargo module only when +cargo
* rust: add v1.74.0 and v1.75.0 and remove variants in favor of +dev
* [@spackbot] updating style on behalf of alecbcs
* Fix variant typo
---------
Co-authored-by: alecbcs <alecbcs@users.noreply.github.com>
This adds a few options to `spack gc`.
One to give you a little more control over dependencies:
* `-b` / `--keep-build-dependencies`: By default, `spack gc` considers build dependencies to be "no longer needed" once their dependents are installed. With this option, we'll keep build dependencies of needed installations as well.
And two more to make working with environments easier:
* `-E` / `--except-any-environment`: Garbage collect anything NOT needed by an environment. `spack gc -E` and `spack gc -bE` are now easy ways to get rid of everytihng not used by some environment.
* `-e` / `--except-environment` `ENV`: Instead of considering all environments, garbage collect everything not needed by a *specific* environment. Note that you can use this with `-E` to add directory environments to the list of considered envs, e.g.:
spack gc -E -e /path/to/direnv1 -e /path/to/direnv2 #...
- [x] rework `unused_specs()` method on DB to add options for roots and deptypes
- [x] add `all_hashes()` method on DB
- [x] rework `spack gc` command to add 3 more options
- [x] tests
The gcc-runtime package adds a separate node for gcc's dynamic runtime
libraries.
This should help with:
1. binary caches where rpaths for compiler support libs cannot be
relocated because the compiler is missing on the target system
2. creating "minimal" container images
The package is versioned like `gcc` (in principle it could be
unversioned, but Spack doesn't always guarantee not mixing compilers)
This PR adds a flag `--tag/-t` to `buildcache push`, which you can use like
```
$ spack mirror add my-oci-registry oci://example.com/hello/world
$ spack -e my_env buildcache push --base-image ubuntu:22.04 --tag my_custom_tag my-oci-registry
```
and lets users ship a full, installed environment as a minimal container image where each image layer is one Spack package, on top of a base image of choice. The image can then be used as
```
$ docker run -it --rm example.com/hello/world:my_custom_tag
```
Apart from environments, users can also pick arbitrary installed spec from their database, for instance:
```
$ spack buildcache push --base-image ubuntu:22.04 --tag some_specs my-oci-registry gcc@12 cmake
$ docker run -it --rm example.com/hello/world:some_specs
```
It has many advantages over `spack containerize`:
1. No external tools required (`docker`, `buildah`, ...)
2. Creates images from locally installed Spack packages (No need to rebuild inside `docker build`, where troubleshooting build failures is notoriously hard)
3. No need for multistage builds (Spack just tarballs existing installations of runtime deps)
4. Reduced storage size / composability: when pushing multiple environments with common specs, container image layers are shared.
5. Automatic build cache: later `spack install` of the env elsewhere speeds up since the containerized environment is a build cache
* add trim function to `Spec` and `--ignore` option to 'spack diff'
Allows user to compare two specs while ignoring the sub-DAG of a particular dependency, e.g.
spack diff --ignore=mpi --ignore=zlib trilinos/abcdef trilinos/fedcba
to focus on differences closer to the root of the software stack
This PR changes the default behavior of `spack config get` and `spack config blame`
to print a flattened version of the entire spack configuration, including any active
environment, if the commands are invoked with no section arguments.
The new behavior is used in Gitlab CI to help debug CI configuration, but it can also
be useful when asking for more information in issues, or when simply debugging Spack.
Convert the 'develop' section of an environment to a dedicated configuration section.
This means for example that instead of having to define `develop` specs in the
`spack.yaml`, the environment can `include:` another `develop.yaml` configuration
which specifies which specs should be developed in the environment.
This change is not expected to be disruptive given that existing environment `spack.yaml`
files will conform to the new schema.
(Update 11/28/2023) I have implemented the `develop`/`undevelop` commands in terms
of more-generic modification functions added to the `config` module: `change_or_add`
and `update_all`. It is assumed that the semantics added here (described in 11/18 update)
would be desirable to extend to other config update actions (e.g. adding compilers,
changing package requirements, adding mirrors).
(Update 11/18/2023) I have updated this such that `spack develop`, and
`spack undevelop` to potentially modify all writable scopes, like
https://github.com/spack/spack/pull/41147. https://github.com/spack/spack/pull/35307
will be useful for modifying included scopes, but generally speaking specifying a
`--scope` will not be required for `spack develop`: `spack develop` will add new
develop specs to whatever scope already has develop specs defined, or to the
highest-priority writable scope (which should be the env scope).
TODOs:
- [x] If you `spack undevelop` a package which is mentioned at multiple layers of
configuration, then currently this would only modify one of them. That's not
technically a new issue (has always existed for configuration modification), but
may be confusing to users when presented via an interface other than `spack config set`
- [x] Need to add (or confirm) the ability to modify individual config files by providing
a path (rather than using a scope identifier as a key to retrieve associated config).
- [x] `spack develop` adds new develop specs to the scope that defines them
(potentially skipping higher priority scopes to e.g. augment included scope files)
---------
Co-authored-by: scheibelp <scheibelp@users.noreply.github.com>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
* Add `signed` property to mirror config
* make unsigned a tri-state: true/false overrides mirror config, none takes mirror config
* test commands
* Document this
* add a test
- we don't have a fallback if make is not installed
- we assume file system locking works
- we don't verify that make is gnu make (bootstrapping fails on FreeBSD as a result)
- there are some weird race conditions in writing spack.yaml on concurrent spack install
- the view is updated after every package install instead of post environment install.
* developer tools stack try 2
This version is actually in use locally and has largely stabilized, at
least on x86. Some packages are still a challenge on ppc64le, but maybe
worth keeping this working as a set.
* add packages, try to get container with newer gcc
* remove reuse: true
* try to get cmake to build on medium, 25 minutes is too long
* add lsd package and add to dev tools stack
* clean up fzf dependency and sorting
* Update share/spack/gitlab/cloud_pipelines/stacks/developer_tools/spack.yaml
* cuda: add 12.3.0 (#40827)
* Switch to dashes
* yet more underscores
---------
Co-authored-by: Paul R. C. Kent <kentpr@ornl.gov>
