Add a `find_first` method that locates one instance of a file
that matches a specified pattern by recursively searching a directory
tree. Unlike other `find` methods, this only locates one file at most,
so can use optimizations that avoid searching the entire tree:
Typically the relevant files are at low depth, so it makes sense to
locate files through iterative deepening and early exit.
Update tcl and lmod modulefile template to provide more information on
help message (name, version and target) like done on whatis for lmod
modulefiles.
Adapt tcl and lmod modulefile templates to generate append-path or
remove-path commands in modulefile when respectively append_flags or
remove_flags commands are defined in package for run environment.
Fixes#10299.
Simplify environment modification block in modulefile Tcl template by
always setting a path delimiter to the prepend-path, append-path and
remove-path commands.
Remove --delim option to the setenv command as this command does not
allow such option.
Update test_prepend_path_separator test to explicitly check the 6
path-like commands that should be present in generated modulefile.
fixes#36339
We were missing a rule that enforced a match between
the `node_compiler` and the compiler used to satisfy
a requirement.
Fix compiler with custom, made up version too
* Specs that define 'new' versions in the require: section need to generate
associated facts to indicate that those versions are valid.
* add test to verify success with unknown versions.
Since environment-modules has support for autoloading since 4.2,
and Spack-builds of it enable it by default, use the same autoload
default for tcl as lmod.
If you have a "require:" section in your packages config, and you
use it to specify a list of requirements, the list elements can
now include strings (before this, each element in the list had to
be a `one_of` or `any_of` specification, which is awkward if you
wanted to apply just one spec with no alternatives).
Example one:
```
spack install --add x y z
```
is equivalent to
```
spack add x y z
spack concretize
spack install --only-concrete
```
where `--only-concrete` installs without modifying spack.yaml/spack.lock
Example two:
```
spack install
```
concretizes current spack.yaml if outdated and installs all specs.
Example three:
```
spack install x y z
```
concretizes current spack.yaml if outdated and installs *only* concrete
specs in the environment that match abstract specs `x`, `y`, or `z`.
The `ignore` parameter was only used for `spack activate/deactivate`, and it isn't used
by Spack Environments which have their own handling of file conflicts. We should remove it.
Everything that handles `ignore=` was removed in #29317 and included in 0.19, when we
removed `spack activate` and `spack deactivate` in favor of environments. So all of these
usages removed here were already being ignored by Spack.
Adapt tcl modulefile template to call "module load" on autoload
dependency without testing if this dependency is already loaded or not.
The is-loaded test is not necessary, as module commands know how to cope
with an already loaded module. With environment-modules 4.2+ (released
in 2018) it is also important to have this "module load" command even if
dependency is already loaded in order to record that the modulefile
declares such dependency. This is important if you want to keep a
consistent environment when a dependent module is unloaded.
The "Autoloading" verbose message is also removed as recent module
commands will report such information to the user (depending on the
verbosity configured for the module command).
Such change has been test successfully with Modules 3.2 (EL7), 4.5 (EL8)
and 5.2 (latest) and also with Lmod 7 and 8 (as it is mentionned in
Spack docs that Lmod can be used along with tcl modules). Dependencies
are correctly loaded or unloaded, whether they are loaded/unloaded or
not.
This change fixes Tcl quoting issue introduced in #32853.
Fixes#19155.
In the Windows filesystem logic for creating a symlink, we intend to
fall back to a copy when the symlink cannot be created (for some
configuration settings on Windows it is not possible for the user
to create a symlink). It turns out we were overly-broad in which
exceptions lead to this fallback, and the subsequent copy would
also fail: at least one case where this occurred is when we
attempted to create a symlink that already existed.
The updated logic expressly avoids falling back to a copy when the
file/symlink already exists.
* ASP-based solver: use satisfies instead of intersects
They are semantically equivalent for concrete versions,
but the GitVersion.intersects implementation is buggy
* Mitigation for git version bug
fixes#36134
This commit works around the issue in #36134, by using
GitVersion.satisfies instead of GitVersion.intersects
There are still underlying issues when trying to infer the
"reference version" when no explicit one is given, but:
1. They are not reproducible with our synthetic repo
2. They occur only when the `git.<xxx>` form of Git version
is used
Here we just work around the user facing issue and ensure
the tests are correct with our synthetic repository.
This PR does 2 unrelated things:
1. It changes the encoding of the compilers
2. It tweaks the heuristic for the solves in a0d88179074f81d13a3fad629a43d86334e7f982
Both were initially motivated by trying to get a performance gain but, while 2 showed significant speed-ups[^1], 1 instead didn't. I kept it anyhow, since I think the code related to compilers is more consolidated with the new encoding and we might get some performance improvement out of it if we can base our errors on the `node_compiler(Package, CompilerID)` atoms instead of `attrs`.
[^1]: In general the changes in the heuristic brought a ~10% speed-up on the tests I did. I'll post detailed results below.
Add a warning about compilers.yaml that is triggered if there are multiple compilers with the same spec, os and
target (since they can't be selected by users with the spec syntax only).
Since GPG clear-sign cannot deal with lines longer than 19995 characters
and doesn't even error but simply truncates those linese (don't ask me
why...), we have to be careful not to hit that line limit when reducing
the filesize.
So, instead this PR sets the indent level to 0 and drops the whitespace
after `: `, which still reduces file size by 50% or so.
Adds builders appropriate for building these packages on Windows.
It is intended that builds on other platforms are unaffected (e.g.
they build with Autotools as before on Linux).
... and use colors in disambiguate message for clarity.
This commit avoids the loop:
```
for root in roots:
for dep in deps(root):
...
```
instead it ensures each node is visited once and only once.
Also adds a small optimization when searching for concrete specs, since
we can assume uniqueness of dag hash, so it's fine to early exit.
This adds a new mode for `concretizer:reuse` called `dependencies`,
which only reuses dependencies. Currently, `spack install foo` will
reuse older versions of `foo`, which might be surprising to users.
* CI configuration boilerplate reduction and refactor
Configuration:
- New notation for list concatenation (prepend/append)
- New notation for string concatenation (prepend/append)
- Break out configuration files for: ci.yaml, cdash.yaml, view.yaml
- Spack CI section refactored to improve self-consistency and
composability
- Scripts are now lists of lists and/or lists of strings
- Job attributes are now listed under precedence ordered list that are
composed/merged using Spack config merge rules.
- "service-jobs" are identified explicitly rather than as a batch
CI:
- Consolidate common, platform, and architecture configurations for all CI stacks into composable configuration files
- Make padding consistent across all stacks (256)
- Merge all package -> runner mappings to be consistent across all
stacks
Unit Test:
- Refactor CI module unit-tests for refactor configuration
Docs:
- Add docs for new notations in configuration.rst
- Rewrite docs on CI pipelines to be consistent with refactored CI
workflow
* Script verbose environ, dev bootstrap
* Port #35409
When untouched spec pruning is enabled, specs possibly affected
by a change cannot be pruned from a pipeline.
Previously spack looked at all specs matching changed package
names, and traversed dependents of each, all the way to the
environment root, to compute the set of environment specs
possibly affected by a change (and thus, not candidates for
pruning).
With this PR, when untouched spec pruning is enabled, a new
environment variable can control how far towards the root spack
traverses to compute the set of specs possibly affected by a
change. SPACK_UNTOUCHED_PRUNING_DEPENDENT_DEPTH can be set
to any numeric value before the "spack ci generate" command
is called to control this traversal depth parameter. Setting
it to "0" traverses only touched specs, setting it to "1"
traverses only touched specs and their direct dependents, and
so on. Omitting the variable results in the previous behavior
of traversing all the way to the root. Setting it to a negative
value means no traversal is done, and always yields an empty
set of possibly affected specs (which would result in the max
pruning possible).
Currently `spack buildcache create` creates compressed tarballs that
differ between each invocation, thanks to:
1. The gzip header containing mtime set to time.time()
2. The generated buildinfo file which has a different mtime every time.
To avoid this, you have to explicitly construct GZipFile yourself, since
the Python API doesn't expose the mtime arg, and we have to manually
create the tarinfo object for the buildinfo metadata file.
Normalize mode: regular files & hardlinks executable by user, dirs, symlinks: set 0o755 permissions in tarfile; other files use 0o644
This commit formalizes `satisfies(lhs, rhs, strict=True/False)`
and splits it into two functions: `satisfies(lhs, rhs)` and
`intersects(lhs, rhs)`.
- `satisfies(lhs, rhs)` means: all concrete specs matching the
left hand side also match the right hand side
- `intersects(lhs, rhs)` means: there exist concrete specs
matching both lhs and rhs.
`intersects` now has the property that it's commutative,
which previously was not guaranteed.
For abstract specs, `intersects(lhs, rhs)` implies that
`constrain(lhs, rhs)` works.
What's *not* done in this PR is ensuring that
`intersects(concrete, abstract)` returns false when the
abstract spec has additional properties not present in the
concrete spec, but `constrain(concrete, abstract)` will
raise an error.
To accomplish this, some semantics have changed, as well
as bugfixes to ArchSpec:
- GitVersion is now interpreted as a more constrained
version
- Compiler flags are interpreted as strings since their
order is important
- Abstract specs respect variant type (bool / multivalued)
Two fixes:
1. `-Wl,a,b,c,d` is a comma separated list of linker arguments, we
incorrectly assume key/value pairs, which runs into issues with for
example `-Wl,--enable-new-dtags,-rpath,/x`
2. `-Xlinker,xxx` is not a think, so it shouldn't be parsed.
Currently, if two compilers with the same spec differ on the flags, the concretizer will:
1. mix both sets of flags for the spec in the ASP program
2. error noting that the set of flags from the compiler (both of them) doesn't match the set from the lower priority compiler
This PR fixes both -- only flags from the highest priority compiler with a given spec are considered.
`mypy` only understands `sys.platform == "win32"`, not indirect assignments of that
value to things like `is_windows`. If we don't use the accepted platform checks, `mypy`
registers many Windows-only symbols as not present on Linux, when it should skip the
checks for platform-specific code.
Update `spack.util.environment` to remove legacy idioms.
* Remove kwargs from method signature and use a class for traces
* Uppercase a few global variables
* spack.util.environment: add type-hints
* Improve docstrings
* Fixed most style issues reported by pylint
a shared library /lib64/libcxi.so, which seems to also appear on other
non-slingshot systems. This patch also checks to make sure that there
is a Cray programming enviornment in /opt/cray/pe in addition to the
shared library.
