Spack merges ranges and concrete versions if they have non-empty
intersection. That is not enough for adjacent version ranges.
This commit ensures that disjoint ranges in version lists are simplified
if their union is not disjoint:
```python
"@1.0:2.0,2.1,2.2:3,4:6" # simplifies to "@1.0:6"
```
Refactoring `SpackSolverSetup` is a bit easier with type annotations, so I started
adding some. This adds annotations for the (many) instance variables on
`SpackSolverSetup` as well as a few other places.
This also refactors `condition()` to reduce redundancy and to allow
`_get_condition_id()` to be called independently of the larger condition
function.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Some builds on Windows break when encountering paths with spaces. This
reencodes some paths in Windows 8.3 filename format (when on Windows):
this serves as an equivalent identifier for the file, but in a form that
does not have spaces.
8.3 filenames are also truncated in length, which could be helpful, but
that is not the primary intended purpose of using this format.
Overall
* nmake/msbuild packages do this generally for the install prefix
* curl/perl require additional modifications (as written now, each package
may require calls to `windows_sfn` to work when the Spack
root/install/staging prefixes contain spaces)
Some items for follow-up:
* Spack itself does not create paths with spaces "on top" of whatever
the user configures or where it is placed (e.g. the Spack root, the
staging directory, etc.), so it might be possible to edit some of these
paths once and avoid a proliferation of individual `windows_sfn`
calls in individual packages.
* This approach may result in the insertion of 8.3-style paths into
build artifacts (on Windows), handling this may require additional
bookkeeping (e.g. when relocating).
* Move spec_list into its own file, instead of __init__.py
* Remove spack.schema.spack
This module was introduced in #33960 It's almost an exact duplicate of
spack.schema.env, and is not used anywhere.
* Fix typo
* Add support for clang in oneapi packages with OpenMP
* Add fallback search for libomp in OneApi package with OpenMP threading
* Add requires for the compiler when using threads=openmp in intel-oneapi-mkl
* Cosmetic changes to messages in oneapi.py
* Update error message in oneapi.py
Co-authored-by: Robert Cohn <rscohn2@gmail.com>
* Update another error message in oneapi.py
Co-authored-by: Robert Cohn <rscohn2@gmail.com>
* Inline helper error function in oneapi.py
* Update one more error message in oneapi.py
* Wrap long line in oneapi.py
---------
Co-authored-by: Robert Cohn <rscohn2@gmail.com>
* allow packages to request no submodules be updated when self.submodules is a
callable function
* Extend the test added in Allow more fine-grained control over what submodules are
updated: part 2 #27293 to include this case
* Update the type signature for the submodules arg of version() in directives.py
---------
Co-authored-by: tjfulle <tjfulle@users.noreply.github.com>
* cmake: Enable CMAKE_EXPORT_COMPILE_COMMANDS
Enabling this option causes CMake to generate a compile_commands.json file
containing a compilation database that can be used to drive third-party tools.
CMAKE_EXPORT_COMPILE_COMMANDS only exists for CMake >= 3.5
Exporting compilation databases is only supported for Makefile and Ninja
generators, so check these conditions as well.
CMAKE_EXPORT_COMPILE_COMMANDS is only enabled in supported configurations
This fixes bugs, performance issues, and removes no longer necessary code.
Short version:
1. Creating views from Python extensions would error if the Spack `opt` dir itself was in some symlinked directory. Use of `realpath` would expand those, and keying into `merge_map` would fail.
2. Creating views from Python extensions (and Python itself, potentially) could fail if the `bin/` dir contains symlinks pointing outside the package prefix -- Spack keyed into `merge_map[target_of_symlink]` incorrectly.
3. In the `python` package the `remove_files_from_view` function was broken after a breaking API change two years ago (#24355). However, the entire function body was redundant anyways, so solved it by removing it.
4. Notions of "global view" (i.e. python extensions being linked into Python's own prefix instead of into a view) are completely outdated, and removed. It used to be supported but was removed years ago.
5. Views for Python extension would _always_ copy non-symlinks in `./bin/*`, which is a big mistake, since all we care about is rewriting shebangs of scripts; we don't want to copy binaries. Now we first check if the file is executable, and then read two bytes to check if it has a shebang, and only if so, copy the entire file and patch up shebangs.
The bug fixes for (1) and (2) basically consist of getting rid of `realpath` entirely, and instead simply keep track of file identifiers of files that are copied/modified in the view. Only after patching up regular files do we iterate over symlinks and check if they target one of those. If so, retarget it to the modified file in the view.
These 7 hooks were not used.
- Six of them related to install phases were unused after `spack`
`monitor` was removed, and the code seems to have bit rotten as there
were reports they were not (always?) triggered when they should.
- The post environment one was made redundant after spack install for
environment started following the common code path for generating
module files in #42147.
It should not be a breaking change to remove, since users cannot define
hooks in extensions, they would have to fork Spack.
If we ever _were_ to make those hooks extendable outside of core Spack,
it would also be better to start with fewer rather than more, cause
everything you expose gets relied upon...
Removing those also allows us to rethink what hooks we really need, and
in particular it seems like we need a hook that runs post install also when
the spec is inserted into the database.
The lack of a rule to avoid enforcing requirements on multi-valued variants, when the condition activating the environment was not met, resulted in multiple optimal solutions. The fix is to prevent imposing a requirement if the when= rule activating it is not met.
The section was highly outdated as it referred to old defaults, and
failed to mention `hide_implicits: true`.
This commit restructures it, moves some deeply nested sections a level
up, and promotes `hide_implicits: true` + `autoload: direct` before
talking about `exclude`.
* Registry queries can fail due to simultaneous access from other
processes. Wrap some of these accesses and retry when this may
be the cause (the generated exceptions don't allow pinpointing
this as the reason, but we add logic to identify cases which
are definitely unrecoverable, and retry if it is not one of
these).
* Add make recursion optioal for most registry search functions;
disable recursive search in case where it was originally always
recursive.
Fix two separate problems:
1. We want to always visit parents before children while creating views
(when it comes to ignoring conflicts, the first instance generated in
the view is chosen, and we want the parent instance to have precedence).
Our preorder traversal does not guarantee that, but our topological-
order traversal does.
2. For copy style views with packages x depending on y, where
<x-prefix>/foo is a symlink to <y-prefix>/foo, we want to guarantee
that:
* A conflict is not registered
* <y-prefix>/foo is chosen (otherwise, the "foo" symlink would become
self-referential if relocated relative to the view root)
Note that
* This is an exception to [1] (in this case the dependency instance
overrides the dependent)
* Prior to this change, if "foo" was ignored as a conflict, it was
possible to create this self-referential symlink
Add tests for each of these cases
Sometimes the logs are too long and the copy & paste command is not
shown. In that case I'd like to just copy the failing GitLab job URL in
my browser to `spack reproduce-build <url>`.
Currently, the `SpackSolverSetup` and the `PyclingoDriver` are more coupled than necessary:
1. The driver object needs a setup object to be injected during a solve,
2. And the setup object will get a reference back to the driver
This design is necessary because we use the low-level `clingo.backend` interface to setup our problem. This interface though is meant to bypass the grounder and add symbols directly in the grounded table, which is a feature we don't currently use.
The PR simplifies the encoding by having the setup object returning the problem-specific facts / rules as a list of strings, and the driver ingesting them using the [clingo.Control.add](https://potassco.org/clingo/python-api/5.6/clingo/control.html#clingo.control.Control.add) method. This removes any use of the low level interface.
Using this encoding makes it easy to hash the output of the setup phase, since it is returned as a string.
This "breaks" the deprecated schema by allowing unknown attributes
to the attributes section of the job types. The breaking change here is
that deprecated stacks will no longer ignore attributes that are unknown
but rather assume the new CI schema behavior of injecting them into the
generated CI configuration. This change is required to secure
authentication in Spack CI.
Improve naming, so it's clear file "extensions" are not taken in the
`PurePath(path).suffix` sense as the original function name suggests,
but rather that the files are opened and their magic bytes are
classified.
Add type hints.
Fix a bug where `stream.read(num_bytes)` was run on the compressed
stream instead of the uncompressed stream, which can potentially break
detection of tar.bz2 files.
Ensure that when peeking into streams for magic bytes, they are reset to
their original position upon return.
Use new API in `spack logs`.
Relocation of `PT_INTERP` in ELF files already happens to work from long to short path, thanks to generic binary relocation (i.e. find and replace). This PR improves it:
1. Adds logic to grow `PT_INTERP` strings through patchelf (which is only useful if the interpreter and rpath paths are the _only_ paths in the binary that need to be relocated)
2. Makes shrinking `PT_INTERP` cleaner. Before this PR when you would use Spack-built glibc as link dep, and relocate
executables using its dynamic linker, you'd end up with
```
$ file exe
exe: ELF 64-bit LSD pie executable, ..., interpreter /////////////////////////////////////////////////path/to/glibc/lib/ld-linux.so
```
With this PR you get something sensible:
```
$ file exe
exe: ELF 64-bit LSD pie executable, ..., interpreter /path/to/glibc/lib/ld-linux.so
```
When Spack cannot modify the interpreter or rpath strings in-place, it errors out without modifying the file, and leaves both tasks to patchelf instead.
Also add type hints to `elf.py`.
Certain versions of ifx (the majority of those available) have an issue
where they are not compatible with TMP directories with dot chars
This precludes their use with CMake.
Remap TMP to point to the stage directory rather than whatever the TMP
default is
Add the empty deptype `spack.deptypes.NONE`.
Test the case `traverse_nodes(deptype=spack.deptypes.NONE)` to not
traverse dependencies, only de-duplicate.
Use the construct in environment views that otherwise would branch on
whether deps are enabled or not.
Previously, for abstract specs like:
```
foo ^[virtuals=a] bar ^[virtuals=b] bar
```
the second requirement was silently discarded on concretization. Now they're merged, and the abstract spec is equivalent to:
```
foo ^[virtuals=a,b] bar
```
CMake may write and read from `~/.cmake` through `export(...)` and read `find_package(...)` respectively. We don't want this as it may influence the build in a non-deterministic way, so disable it for all versions of `cmake`.
Fixes a bug where Spack did not generate module files of non-roots during
spack install with an active environment.
The reason being that Environment.new_installs only contained roots.
This PR:
Drops special casing of automatic module generation in post-install hooks
When `use_view`, compute environment variable modifications like always, and
applies a view projection to them afterwards, like we do for spack env activate.
This ensures we don't have to delay module generation until after the view is
created.
Fixes another bug in use_view where prefixes of dependencies would not be
projected -- previously Spack would only temporarily set the current spec's prefix.
Removes the one and only use of the post_env_write hook (but doesn't drop it to
make this backportable w/o changes)
Previously `std_args` was called on non-roots in a build context, which is redundant, and also leads to issues when `std_args` expects build deps of the `pkg` to be installed.
* Environments: fix environment config
* Don't change the lockfile manifest path
* Update activate's comments to tie the manifest to the configuration
* Add spec_list override method
* Remove type checking from 'activate' since already have built-in check
* Refactor global methods tied to the manifest to be in its class
* Restore Environment's 'env_subdir_path' and pass its config_stage_dir to EnvironmentManifestFile
* Restore global env_subdir_path for use by Environment and EnvironmentManifestFile
Currently requirements allow to express "strong preferences" and "conflicts" from
configuration using a convoluted syntax:
```yaml
packages:
zlib-ng:
require:
# conflict on %clang
- one_of: ["%clang", "@:"]
# Strong preference for +shared
- any_of: ["+shared", "@:"]
```
This PR adds syntactic sugar so that the same can be written as:
```yaml
packages:
zlib-ng:
conflict:
- "%clang"
prefer:
- "+shared"
```
Preferences written in this way are "stronger" that the ones documented at:
- https://spack.readthedocs.io/en/latest/packages_yaml.html#package-preferences
`spack install` early exit behavior was sometimes convenient, except
that it had and has bugs:
1. prior bug: we didn't mark env roots of already installed specs as
explicit in the db
2. current bug: `spack install --overwrite` is ignored
So this PR simplifies by letting the installer do its thing even if
everything is supposedly installed.
This commit ensures that CMake packages that also have Python as a build/link dep get a couple defines for the Python path so that CMake's builtin `FindPython3`, `FindPython`, `FindPythonInterp` modules can locate Python correctly.
The main problem with those CMake modules is that they first search for Python versions known at the time of release, meaning that old CMake maybe find older system Python 3.8 even though Python 3.11 comes first in `CMAKE_PREFIX_PATH` and `PATH`.
Package maintainers can opt out of this by overriding the `find_python_hints = False` attribute in the package class.
The piece of code that is removed in this PR predates environment views.
Spack would symlink build logs in `<env>/.spack-env/logs/*`, but this is
redundant because:
1. Views already add `<prefix>/.spack` (and there's logic there to avoid
clashes)
2. The code was broken anyways: it would only symlink the logs of
environment roots, not their deps, even if they were just built.
If users disable views, I'm pretty sure they're not waiting for
`.spack-env/logs` either. So, imo we can delete this code, and it was
probably overlooked in the past.
For a requirement like
```
packages:
foo:
require:
- "+debug"
```
(not `one_of:`, `any_of:`, or `spec:`)
`spack config change` would ignore the string. This was particularly evident if toggling a variant for a previously unmentioned package:
```
$ spack config change packages:foo:require:+debug
$ spack config change packages:foo:require:~debug
```
This fixes that and adds a test for it.
* Reduce the size on disk for logs
This PR does two things:
1. Store a compressed `spack-build-out.txt.gz`
2. Get rid of phase logs, as they are duplicates of
`spack-build-out.txt`
The logs are not compressed in the stage dir, so on build failure the
workflow for users is no different.
It's just that on install the logs are rarely used, and if needed, users
can easily `gzip -d` or `zgrep` them.
In the case of GCC installs, the compressed logs are <5% of the original
size, which is typically dozens of MBs.
* get rid of "backwards compat" of file names in stage dirs
Sbangs don't exist on Native Windows, and the hook is causing errors
due to the file comparison + behavior of os.rename on Windows. Skip
the hook on Windows.
Like `spack change` for specs in environments, this can e.g. replace `examplespec+debug` with `examplespec~debug` in a `require:` section.
Example behavior for a config like:
```
packages:
foo:
require:
- spec: +debug
```
* `spack config change packages:foo:require:~debug` replaces `+debug` with `~debug`
* `spack config change packages:foo:require:@1.1` adds a requirement to the list
* `spack config change packages:bar:require:~debug` adds a requirement
As observed in #40944, when using `spack config add <path>`, the `path` might
contain keys that are enclosed in quotes.
This was broken in https://github.com/spack/spack/pull/39831, which assumed that
only the value (if present, the final element of the path) would use quotes.
This preserves the primary intended behavior of #39931 (allowing ":" in values when
using `spack config add`) while also allowing quotes on keys.
This has complicated the function `process_config_path`, but:
* It is not used outside of `config.py`
* The docstring has been updated to account for this
* Created an object to formalize the DSL, added a test for that, and
refactored parsing to make use of regular expressions as well.
* Updated the parsing and also updated the `config_path_dsl` test with an explicit check.
At a higher level, split the parsing to check if something is either a key or not:
* in the first case, it is covered by a regex
* in the second, it may be a YAML value, but in that case it would have to be the last
entry of x:y:z, so in that case I attempt to use the YAML handling logic to parse it as such
Spack packages may not have a public download option, and can implement
`download_instr` to inform users how to obtain the artifacts needed to
build. `spack checksum` however did not account for this and would print
out a confusing error message when invoked on such packages ("Could not
find any remote versions").
This PR updates the error message to output the manual download instructions
if `spack checksum` is invoked on a package with `manual_download = True`.
Currently when you repeatedly create a bootstrap mirror that includes
`clingo-bootstrap@spack` you get different tarballs every time.
This is a general problem with mirroring checkouts from version control
as tarballs. I think it's best to create tarballs ourselves, since that way we
have more control over its contents.
This PR ensures normalized tarballs like we do for build caches:
- normalize file permissions (in fact that was already inspired by git, so
should be good)
- normalized file creation/modification time (timestamp 0)
- uid / guid = 0, no usernames
- normalized gzip header
- dir entries are ordered by `(is_dir, name)` where strings are not locale aware ;)
- POSIX says st_mode of symlinks is unspecified, so work around it and
force mode to `0o755`
Explicitly requested namespaces are annotated during
the setup phase, and used to retrieve the correct package
class.
An attribute for the namespace has been added for each node.
Currently, a single namespace per package is allowed
during concretization.
Add `--create` option to `env activate` to allow users to create and activate in one command.
---------
Co-authored-by: Wouter Deconinck <wdconinc@gmail.com>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: psakievich <psakievich@users.noreply.github.com>
* Bump the build cache layout version from 1 to 2
* Version to lists parent directories of the prefix in the tarball too, which is required from some container runtimes
* Move in vs. satisfies to a note and mention special cases of in
* Address feedback: oveoverlap -> intersect
* Re-word the satisfies versus in note.
---------
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
This fixes an issue where pkg.stage throws because a patch cannot be found,
but the patch is redundant because the spec is reused from a build cache and
will be installed from existing binaries.
