Commit graph

6094 commits

Author SHA1 Message Date
Todd Gamblin
b260234faf editing: add higher-precedence SPACK_EDITOR environment variable
Other tools like git support `GIT_EDITOR` which takes higher precedence than the
standard `VISUAL` or `EDITOR` variables. This adds similar support for Spack, in the
`SPACK_EDITOR` env var.

- [x] consolidate editor code from hooks into `spack.util.editor`
- [x] add more editor tests
- [x] add support for `SPACK_EDITOR`
- [x] add a documentation section for controlling the editor and reference it
2023-04-18 16:23:00 -07:00
Todd Gamblin
2f30da1762 refactor: unify use of spack.util.editor
Code from `spack.util.editor` was duplicated into our licensing hook in #11968. We
really only want one place where editor search logic is implemented. This consolidates
the logic into `spack.util.editor`, including a special case to run `gvim` with `-f`.

- [x] consolidate editor search logic in spack.util.editor
- [x] add tests for licensing case, where `Executable` is used instead of `os.execv`
- [x] make `_exec_func` argument of `editor()` into public `exec_fn` arg
- [x] add type annotations
2023-04-18 16:23:00 -07:00
Massimiliano Culpo
d92c21ec97
Fix compilation on Cray (target: any) (#37011)
fixes #36628

Fix using compilers that declare "target: any" in their
configuration. This should happen only on Cray with the
module based programming environment.
2023-04-18 15:47:52 -07:00
Jonathon Anderson
08fd8c8d0a
spack ci: preserve custom attributes in build jobs (#36651)
* Simplify test/cmd/ci.py::test_ci_generate_with_custom_scripts

* Rearrange the build-job logic in generate_gitlab_ci_yaml

* Preserve all unknown attributes in build jobs

* Slip tests for custom attributes in the tests for other job types

* Support custom artifacts

* [@spackbot] updating style on behalf of blue42u

* Don't bother sorting needs

---------

Co-authored-by: blue42u <blue42u@users.noreply.github.com>
2023-04-18 15:40:43 -05:00
Todd Gamblin
6845f41d67
Revert addition of SPACK_EDITOR pending review.
This reverts commit d8a26905ee.
This reverts commit 1ee049ccc3.

These were spuriously pushed to `develop`.
2023-04-18 03:57:24 -07:00
Todd Gamblin
1ee049ccc3
editing: add higher-precedence SPACK_EDITOR environment variable
Other tools like git support `GIT_EDITOR` which takes higher precedence than the
standard `VISUAL` or `EDITOR` variables. This adds similar support for Spack, in the
`SPACK_EDITOR` env var.

- [x] consolidate editor code from hooks into `spack.util.editor`
- [x] add more editor tests
- [x] add support for `SPACK_EDITOR`
- [x] add a documentation section for controlling the editor and reference it
2023-04-18 03:42:56 -07:00
Todd Gamblin
d8a26905ee
refactor: unify use of spack.util.editor
Code from `spack.util.editor` was duplicated into our licensing hook in #11968. We
really only want one place where editor search logic is implemented. This consolidates
the logic into `spack.util.editor`, including a special case to run `gvim` with `-f`.

- [x] consolidate editor search logic in spack.util.editor
- [x] add tests for licensing case, where `Executable` is used instead of `os.execv`
- [x] make `_exec_func` argument of `editor()` into public `exec_fn` arg
- [x] add type annotations
2023-04-18 03:00:04 -07:00
Greg Becker
480b7f397e
Allow users to remove items from hierarchy per-path (#31351)
* lmod modules: allow users to remove items from hierarchy per-spec

This allows MPI wrappers that depend on MPI to be removed from the MPI portion of
the hierarchy and be made available when the appropriate compiler is loaded.

module load gcc
module load mpi-wrapper  # implicitly loads mpi
module load hdf5

This allows users to treat an mpi wrapper like an mpi program
2023-04-17 22:17:11 -07:00
Harmen Stoppels
381c0af988
Revert "move depfile logic into its own module, separate traversal logic from model (#36911)" (#36985)
This reverts commit a676f706a8.
2023-04-17 20:58:38 +02:00
Harmen Stoppels
a676f706a8
move depfile logic into its own module, separate traversal logic from model (#36911) 2023-04-17 15:27:01 +02:00
Doug Jacobsen
690394fabc
Change environment modifications to escape with double quotes (#36789)
This commit changes the environment modifications class to escape
strings with double quotes instead of single quotes.

Single quotes prevent the expansion of enviornment variables that are
nested within environment variable definitions.
2023-04-14 10:13:17 -07:00
Scott Wittenburg
bfa94c5781
gitlab ci: Better tagging of "service" jobs (#36846)
- Tag non-rebuild jobs to target a cheaper (and more highly available)
subset of runners.

- Add missing resource requests to these jobs as well.
2023-04-14 09:03:12 -06:00
Massimiliano Culpo
9ec289857c
netcdf: fix bugs introduced with multiple build systems split (#36825)
Fixes #36689

- The "base" builder class should be last in the MRO
- `filter_compiler_wrappers` needs to be moved to builders
- Decorating a function from a mixin class require using
   the correct metaclass for the mixin
2023-04-14 10:59:12 +02:00
Wouter Deconinck
ff319e9863
Resolve <include-fragment> tags e.g. in github release pages (#36674)
This aims to resolve #34164 by resolving the <include-fragment> tags
that GitHub has started using for their release pages, see
https://github.github.io/include-fragment-element/.

This feels a bit hacky but intended as a starting point for discussion.
After reading a page during spidering, it first parses for
include-fragments, gets them all, and treats them all as separate pages.
Then it looks for href links in both the page itself and the fragments.

Co-authored-by: Alec Scott <alec@bcs.sh>
2023-04-13 20:26:26 +02:00
Massimiliano Culpo
d918ae0bde
containerize: strip binaries in a less aggressive way (#36683) 2023-04-13 17:09:34 +02:00
John W. Parent
530669346a
Windows/MSVC: propagate all VCVARS changes to Spack env (#36582)
MSVC compilers rely on vcvars environment setup scripts to establish
build environement variables neccesary for all projects to build
successfully. Prior to this we were only piping LIB, INCLUDE, and PATH
change through.

Instead we need to propegate all changes to the env variables made by
VCVARs in order to establish robust support for the MSVC compiler.
This most significantly impacts projects that need to be build with
NMake and MSBuild
2023-04-11 19:04:34 -04:00
markus-ferrell
3edb044706
Windows testing: reenable tests for "spack dependents" (#36786)
All the tests worked out of the box. This just removes the skip statements.
2023-04-11 14:30:40 -07:00
Harmen Stoppels
1c3961bdd0
Remove a unit-test that monkey-patches os.stat (#36757)
"test_create_stage_root_bad_uid" started failing as pytest updated to v7.3.0
2023-04-11 14:02:35 +02:00
kwryankrattiger
b2310f9e64
Ci backwards compat (#36045)
* CI: Fixup docs for bootstrap.

* CI: Add compatibility shim

* Add an update method for CI

Update requires manually renaming section to `ci`. After
this patch, updating and using the deprecated `gitlab-ci` section
should be possible.

* Fix typos in generate warnings

* Fixup CI schema validation

* Add unit tests for legacy CI

* Add deprecated CI stack for continuous testing

* Allow updating gitlab-ci section directly with env update

* Make warning give good advice for updating gitlab-ci

* Fix typo in CI name

* Remove white space

* Remove unneeded component of deprected-ci
2023-04-10 16:46:45 -05:00
Massimiliano Culpo
a7b2196eab
Fix incorrect reformatting of spack.yaml (#36698)
* Extract a method to warn when the manifest is not up-to-date

* Extract methods to update the repository and ensure dir exists

* Simplify further the write method, add failing unit-test

* Fix the function computing YAML equivalence between two instances
2023-04-07 13:37:28 +02:00
Massimiliano Culpo
0b9694575f
spack install: fail if --log-file and not --log-format (#36684)
fixes #34551

"spack install" now fails if a user passed the --logfile option without specifying a log format.
2023-04-06 09:09:18 -07:00
Massimiliano Culpo
3e1c6b27a4
Update archspec to HEAD of develop (#36657) 2023-04-05 13:23:42 -04:00
Jonathon Anderson
78f5b2a2c6
Add workflow:rules:always to spack ci output (#36011) 2023-04-04 10:03:58 -05:00
Harmen Stoppels
dc1399386c
Make spack config update work on environments (#36542)
Previously `spack -e bla config update <section>` would treat the
environment config scope as standard config file instead of a single
file config scope. This fixes that.
2023-04-04 10:19:05 +02:00
Harmen Stoppels
e9a1d0a157
filter __spack_path_placeholder__ in generated module files after buildcache install (#36611)
* filter __spack_path_placeholder__ in generated module files after buildcache install

* fix windows
2023-04-04 09:45:43 +02:00
Xavier Delaruelle
7a77ecbdb6
modules: remove default symlink on uninstall (#36454)
When app is uninstalled, if it matches a default, then remove the
default symlink targeting its modulefile.

Until now, when a default were uninstalled, the default symlink were
left pointing to a nonexistent modulefile.
2023-04-03 22:54:18 +02:00
Massimiliano Culpo
f91968cf6f
Improve Dockerfile recipe generation (#35187)
- Update default image to Ubuntu 22.04 (previously was still Ubuntu 18.04)
- Optionally use depfiles to install the environment within the container
- Allow extending Dockerfile Jinja2 template
- Allow extending Singularity definition file Jinja2 template
- Deprecate previous options to add extra instructions
2023-04-03 21:05:19 +02:00
Massimiliano Culpo
9d68100891
Rework error handling within the ASP logic program (#36536)
* Reduce effort on grounding by employing cardinality constraints

If we use a cardinality constraint instead of a rule
using pair of values, we'll end up grounding 1 rule
instead of all the possible pair combinations of the
allowed values.

* Display all errors from concretization, instead of just one

If clingo produces multiple "error" facts, we now print all
of them in the error message. Before we were printing just
the one with the least priority.

Consolidate a few common patterns in concretize.lp to ensure
that certain node attributes have one and only one value
assigned.

All errors are displayed, so use a single criterion
instead of three.

* Account for weights in concretize.lp

To recover the optimization order we had before, account
for weights of errors when minimizing.

The priority is mapped to powers of 10, so to effectively
get back the same results as with priorities.
2023-04-03 19:23:29 +02:00
Xavier Delaruelle
7e4927b892
modules: correctly detect explicit installation (#36533)
When generating modulefile, correctly detect software installation asked
by user as explicit installation.

Explicit installation status were previously fetched from database
record of spec, which was only set after modulefile generation.

Code is updated to pass down the explicit status of software
installation to the object that generates modulefiles.

Fixes #34730.
Fixes #12105.

A value for the explicit argument has to be set when creating a new
installation, but for operations on existing installation, this value is
retrieved from database. Such operations are: module rm, module refresh,
module setdefaults or when get_module function is used.

Update on the way tests that mimics an installation, thus explicit
argument has to be set under such situation.
2023-04-03 11:19:18 +02:00
John W. Parent
a8e2961010
Allow configurable stage names (#36509)
Add `config:stage_name` which is a Spec format string that can
customize the names of stages created by Spack. This was primarily
created to allow generating shorter stage names on Windows (along
with `config:build_stage`, this can be used to create stages with
short absolute paths).

By default, this is not set and the prior name stage format is used.

This also removes the username component that is always added to
Stage paths on Windows (if users want to include this, they can
add it to the `build_stage`).
2023-03-31 11:46:47 -07:00
Harmen Stoppels
46bbce1922
compiler wrapper: fix -Xlinker parsing (#35929)
* compiler wrapper: fix -Xlinker parsing
* handle the case of -rpath without value; avoid that we drop the flag
* also handle the -Xlinker -rpath -Xlinker without further args case...
* fix test
* get rid of global $rp var, reduce branching
2023-03-31 09:47:24 -07:00
Harmen Stoppels
b0e7b8c794
typehint a few globals (#36544) 2023-03-31 08:32:56 +02:00
Massimiliano Culpo
e1a104e3a2
Add type-hints to spack.bootstrap (#36491) 2023-03-30 22:12:18 +02:00
Massimiliano Culpo
16404034dc
Fix a couple of minor bugs with ASP weights (#36522)
Reorder versions so that deprecated ones are last. 

Account for default not used when the variant exists.
2023-03-30 01:08:57 -04:00
Todd Gamblin
d76a8b7de7
retry: bugfix: package requirements with git commits (#35057) (#36347)
- [x] Specs that define 'new' versions in the require: section need to generate associated facts to indicate that those versions are valid.
- [x] add test to verify success with unknown versions.
- [x] remove unneeded check that was leading to extra complexity and test
      failures (at this point, all `hash=version` does not require listing out that version
      in `packages.yaml`)
- [x] unique index for origin (dont reuse 0)

Co-authored-by: Peter Josef Scheibel <scheibel1@llnl.gov>
2023-03-28 11:18:54 -07:00
Harmen Stoppels
d862edcce0
macos: set new ad-hoc signature with codesign after binary string replacement (#35585) 2023-03-28 00:33:35 -04:00
Harmen Stoppels
5072e48dab
Add llnl.util.filesystem.find_first (#36083)
Add a `find_first` method that locates one instance of a file
that matches a specified pattern by recursively searching a directory
tree. Unlike other `find` methods, this only locates one file at most,
so can use optimizations that avoid searching the entire tree:
Typically the relevant files are at low depth, so it makes sense to
locate files through iterative deepening and early exit.
2023-03-27 09:42:16 -07:00
Xavier Delaruelle
ea60220a84
modules: enhance help message (#36410)
Update tcl and lmod modulefile template to provide more information on
help message (name, version and target) like done on whatis for lmod
modulefiles.
2023-03-27 10:48:25 +02:00
Xavier Delaruelle
c4923fe3b3
modules: add support for append_flags/remove_flags (#36402)
Adapt tcl and lmod modulefile templates to generate append-path or
remove-path commands in modulefile when respectively append_flags or
remove_flags commands are defined in package for run environment.

Fixes #10299.
2023-03-24 15:38:24 -04:00
Xavier Delaruelle
906151075d
modules tcl: simplify env modification block in template (#36334)
Simplify environment modification block in modulefile Tcl template by
always setting a path delimiter to the prepend-path, append-path and
remove-path commands.

Remove --delim option to the setenv command as this command does not
allow such option.

Update test_prepend_path_separator test to explicitly check the 6
path-like commands that should be present in generated modulefile.
2023-03-24 10:28:10 +01:00
Harmen Stoppels
118d8e4f57
unit tests: don't hard-code arch in compiler config (#36360)
This breaks when testing on non-x86_64 machines outside CI
2023-03-23 23:22:45 +01:00
Massimiliano Culpo
b0e54bc0ac
Fix regression on compiler constraint (#36342)
fixes #36339

We were missing a rule that enforced a match between
the `node_compiler` and the compiler used to satisfy
a requirement.

Fix compiler with custom, made up version too
2023-03-23 20:43:13 +01:00
Harmen Stoppels
4dc9d9f60e
Revert "Bugfix: package requirements with git commits (#35057)" (#36341)
This reverts commit 3d597e29be.
2023-03-23 12:10:46 +01:00
Peter Scheibel
3d597e29be
Bugfix: package requirements with git commits (#35057)
* Specs that define 'new' versions in the require: section need to generate
  associated facts to indicate that those versions are valid.

* add test to verify success with unknown versions.
2023-03-23 01:58:20 -07:00
Xavier Delaruelle
47d710dc4d
modules tcl: switch default all:autoload from none to direct (#36269)
Since environment-modules has support for autoloading since 4.2,
and Spack-builds of it enable it by default, use the same autoload
default for tcl as lmod.
2023-03-23 07:49:17 +01:00
Leopold Talirz
311d3be18e
docs: mention cuda multi-arch capability (#36321) 2023-03-22 16:52:53 -04:00
Peter Scheibel
c3e41153ac
Package requirements: allow single specs in requirement lists (#36258)
If you have a "require:" section in your packages config, and you
use it to specify a list of requirements, the list elements can
now include strings (before this, each element in the list had to
be a `one_of` or `any_of` specification, which is awkward if you
wanted to apply just one spec with no alternatives).
2023-03-20 12:30:33 -07:00
Harmen Stoppels
88d78025a6
spack install: simplify behavior when inside environments (#35206)
Example one:

```
spack install --add x y z
```

is equivalent to

```
spack add x y z
spack concretize
spack install --only-concrete
```

where `--only-concrete` installs without modifying spack.yaml/spack.lock

Example two:

```
spack install
```

concretizes current spack.yaml if outdated and installs all specs.

Example three:

```
spack install x y z
```

concretizes current spack.yaml if outdated and installs *only* concrete
specs in the environment that match abstract specs `x`, `y`, or `z`.
2023-03-20 13:51:30 +01:00
Adam J. Stewart
5dc8ed2694
Remove unused ignore parameter of extends() directive (#35588)
The `ignore` parameter was only used for `spack activate/deactivate`, and it isn't used
by Spack Environments which have their own handling of file conflicts. We should remove it.

Everything that handles `ignore=` was removed in #29317 and included in 0.19, when we
removed `spack activate` and `spack deactivate` in favor of environments.  So all of these
usages removed here were already being ignored by Spack.
2023-03-20 07:22:59 -04:00
Harmen Stoppels
b8e5fc061d
ci.py: remove redundant wrapper around get (#36188) 2023-03-20 10:56:19 +01:00