* Remove mesa18 and libosmesa
mesa18 was introduced in #19528 as a way to maintain the old
autotools build of mesa separate from the new meson build.
We could add a second build system to mesa, but since mesa18 has
been deprecated for a long time, we'll just remove it.
libosmesa was used to multiplex the gl provider between mesa18
and mesa, and is thus unecessary. Remove it to reduce complexity
in the graphical stack.
* Remove references to mesa18 and libosmesa
* vtk: rework dependency on gl and osmesa
* memsurfer: rework dependency on vtk
* visit: minimal fix to avoid having both osmesa and glx
`glibc` and `musl` provide a basic implementation of `iconv` (`iconv`,
`iconv_open`, `iconv_close`), but in practice the installation may be
missing the character encoding methods to make them usable. On Fedora
for example, users need to
```yum install glibc-gconv-extra```
to get the character encodings that `gettext` requires during configure,
namely EUC-JP. Users may not have permissions to install the missing
parts of glibc.
Since Spack can install `libiconv`, it is simpler to use that by
default.
Some logic to detect what libc the c / cxx compilers use by default,
based on `-dynamic-linker`.
The function `compiler.default_libc()` returns a `Spec` of the form
`glibc@x.y` or `musl@x.y` with the `external_path` property set.
The idea is this can be injected as a dependency.
If we can't run the dynamic linker directly, fall back to `ldd` relative
to the prefix computed from `ld.so.`
This PR allows the user to specify a path to a custom cert file (or directory) in
Spack's config:
```yaml
# This is where custom certs for proxy/firewall are stored.
# It can be a path or environment variable. To match ssl env configuration
# the default is the environment variable SSL_CERT_FILE
ssl_certs: $SSL_CERT_FILE
```
`config:ssl_certs` can be a path to a file or a directory, or it can be and environment
variable that resolves to one of those. When it posts to something valid, Spack will
update the ssl context to include custom certs, and fetching via `urllib` and `curl`
will trust the provided certs.
This should resolve many issues with fetching behind corporate firewalls.
---------
Co-authored-by: psakievich <psakievich@users.noreply.github.com>
Co-authored-by: Alec Scott <alec@bcs.sh>
* Allow compilers to function across compatible OS's
* Add documentation in the default yaml
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Co-authored-by: Gregory Becker <becker33@llnl.gov>
This PR adds:
- A new runtime for `%oneapi` compilers, called `intel-oneapi-runtime`
- Information to both `gcc-runtime` and `intel-oneapi-runtime`, to ensure
that we don't mix compilers using different soname for either `libgfortran`
or `libifcore`
To do so, the following internal mechanisms have been implemented:
- Possibility to inject virtual dependencies from the `runtime_constraints`
callback on packages
Information has been added to `gcc-runtime` to provide the correct soname
under different conditions on its `%gcc`.
Rules injected into the solver looks like:
```prolog
% Add a dependency on 'gfortran@5' for nodes compiled with gcc@=13.2.0 and using the 'fortran' language
attr("dependency_holds", node(ID, Package), "gfortran", "link") :-
attr("node", node(ID, Package)),
attr("node_compiler", node(ID, Package), "gcc"),
attr("node_compiler_version", node(ID, Package), "gcc", "13.2.0"),
not external(node(ID, Package)),
not runtime(Package),
attr("language", node(ID, Package), "fortran").
attr("virtual_node", node(RuntimeID, "gfortran")) :-
attr("depends_on", node(ID, Package), ProviderNode, "link"),
provider(ProviderNode, node(RuntimeID, "gfortran")),
attr("node", node(ID, Package)),
attr("node_compiler", node(ID, Package), "gcc"),
attr("node_compiler_version", node(ID, Package), "gcc", "13.2.0"),
not external(node(ID, Package)),
not runtime(Package),
attr("language", node(ID, Package), "fortran").
attr("node_version_satisfies", node(RuntimeID, "gfortran"), "5") :-
attr("depends_on", node(ID, Package), ProviderNode, "link"),
provider(ProviderNode, node(RuntimeID, "gfortran")),
attr("node", node(ID, Package)),
attr("node_compiler", node(ID, Package), "gcc"),
attr("node_compiler_version", node(ID, Package), "gcc", "13.2.0"),
not external(node(ID, Package)),
not runtime(Package),
attr("language", node(ID, Package), "fortran").
```
Add a new config section: `config:aliases`, which is a dictionary mapping aliases
to commands.
For instance:
```yaml
config:
aliases:
sp: spec -I
```
will define a new command `sp` that will execute `spec` with the `-I`
argument.
Aliases cannot override existing commands, and this is ensured with a test.
We cannot currently alias subcommands. Spack will warn about any aliases
containing a space, but will not error, which leaves room for subcommand
aliases in the future.
---------
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
* Allow branching out of the "generic build" unification set
For cases like the one in https://github.com/spack/spack/pull/39661
we need to relax rules on unification sets.
The issue is that, right now, nodes in the "generic build" unification
set are unified together with their build dependencies. This was done
out of caution to avoid the risk of circular dependencies, which would
ultimately cause a very slow solve.
For build-tools like Cython, however, the build dependencies is masked
by a long chain of "build, run" dependencies that belong in the
"generic build" unification space.
To allow splitting on cases like this, we relax the rule disallowing
branching out of the "generic build" unification set.
* Fix issue with pure build virtual dependencies
Pure build virtual dependencies were not accounted properly in the
list of possible virtuals. This caused some facts connecting virtuals
to the corresponding providers to not be emitted, and in the end
lead to unsat problems.
* Fixed a few issues in packages
py-gevent: restore dependency on py-cython@3
jsoncpp: fix typo in build dependency
ecp-data-vis-sdk: update spack.yaml and cmake recipe
py-statsmodels: add v0.13.5
* Make dependency on "blt" of type "build"
In the HPC package manager, we want the fastest `zlib` implementation by default. `zlib-ng` is up to 4x faster than stock `zlib`, and it can do things like take advantage of AVX-512 instructions. This PR makes `zlib-ng` the default `zlib-api` provider (`zlib-api` was introduced earlier, in #37372).
As far as I can see, the only issues you can encounter are:
1. Build issues with packages that heavily rely on `zlib` internals. In Gitlab CI only one out of hundreds of packages had that issue (it extended zlib with deflate stuff, and used its own copy of zlib sources).
2. Packages that like to detect `zlib-ng` separately and rely on `zlib-ng` internals. The only issue I've found with this among the hundreds of packages built in CI is `perl` trying to report more specific zlib-ng version details, and relied on some internals that got refactored. But yeah... that warrants a patch / conflict and is nothing special.
At runtime, you cannot really have any issues, given that zlib and zlib-ng export the exact same symbols (and zlib-ng tests this in their CI).
You can't really have issues with externals when using zlib-ng either. The only type of issue is when system zlib is rather new, and not marked as external; if another external uses new symbols, and Spack builds an older zlib/zlib-ng, then the external might not find the new symbols. But this is a configuration issue, and it's not an issue caused by zlib-ng, as the same would happen with older Spack zlib.
* zlib-api: use zlib-ng +compat by default
* make a trivial change to zlib-ng to trigger a rebuild
* add `haampie` as maintainer
The "concretizer" section has been extended with a "duplicates:strategy"
attribute, that can take three values:
- "none": only 1 node per package
- "minimal": allow multiple nodes opf specific packages
- "full": allow full duplication for a build tool
Allow the following formats:
```yaml
mirrors:
name: <url>
```
```yaml
mirrors:
name:
url: s3://xyz
access_pair: [x, y]
```
```yaml
mirrors:
name:
fetch: http://xyz
push:
url: s3://xyz
access_pair: [x, y]
```
And reserve two new properties to indicate the mirror type (e.g.
mirror.spack.io is a source mirror, not a binary cache)
```yaml
mirrors:
spack-public:
source: true
binary: false
url: https://mirror.spack.io
```
Refactor `TermTitle` into `InstallStatus` and use it to show progress
information both in the terminal title as well as inline. This also
turns on the terminal title status by default.
The inline output will look like the following after this change:
```
==> Installing m4-1.4.19-w2fxrpuz64zdq63woprqfxxzc3tzu7p3 [4/4]
```
* add a virtual dependency name instead of complete package name
* add OneAPI components as providers of virtual packages
* Revert the default of tbb
---------
Co-authored-by: Nisarg Patel <nisarg.patel@lrz.de>
* Disable module generation by default (#35564)
a) It's used by site administrators, so it's niche
b) If it's used by site administrators, they likely need to modify the config anyhow, so the default config only serves as an example to get started
c) it's too arbitrary to enable tcl, but disable lmod
* Remove leftover from old module file schema
* Warn if module file config is detected and generation is disabled
---------
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Add `config:stage_name` which is a Spec format string that can
customize the names of stages created by Spack. This was primarily
created to allow generating shorter stage names on Windows (along
with `config:build_stage`, this can be used to create stages with
short absolute paths).
By default, this is not set and the prior name stage format is used.
This also removes the username component that is always added to
Stage paths on Windows (if users want to include this, they can
add it to the `build_stage`).
Corrects libs detection with a more specific root, otherwise there
can be inconsistencies between version of WGL requested and the
version picked up by `find_libraries`.
Corrects headers detection - win-sdk, win-wdk, and WGL headers all
exist under the same directory, so we can compute the headers for WGL
without querying the spec for win-sdk (which causes errors).
This commit also removes the `plat` variant of `wgl`, which is
redundant with the Spec's target.
Since environment-modules has support for autoloading since 4.2,
and Spack-builds of it enable it by default, use the same autoload
default for tcl as lmod.
This adds a new mode for `concretizer:reuse` called `dependencies`,
which only reuses dependencies. Currently, `spack install foo` will
reuse older versions of `foo`, which might be surprising to users.
* Allow users to specify root env dir
Environments managed by spack have some advantages over anonymous Environments
but they are tucked away inside spack's directory tree. This PR gives
users the ability to specify where the environments should live.
See #32823
This is also taken as an opportunity to ensure that all references are to "managed environments",
rather than "named environments". Prior to this PR some references to the latter persisted.
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
Co-authored-by: Gregory Becker <becker33@llnl.gov>
a) It's used by site administrators, so it's niche
b) If it's used by site administrators, they likely need to modify the config anyhow, so the default config only serves as an example to get started
c) it's too arbitrary to enable tcl, but disable lmod
When running many concurrent spack install processes that need to write
to the db, Spack regularly times out. This is because writing to the DB
after another process has written to it requires deserialization of the
db, mutating it in memory, and serializing it again, which takes some
time. On top of that, I believe there's a 1 second retry when a write
lock cannot be obtained, so I think this means only 3 processes can
really write to the DB at the same time before timing out.
* Update packages config to indicate that MSVC is the preferred compiler
* Update packages config to indicate that msmpi is the preferred MPI provider
* Fix msmpi external detection
Using `-Werror` is good practice for development and testing, but causes us a great
deal of heartburn supporting multiple compiler versions, especially as newer compiler
versions add warnings for released packages. This PR adds support for suppressing
`-Werror` through spack's compiler wrappers. There are currently three modes for
the `flags:keep_werror` setting:
* `none`: (default) cancel all `-Werror`, `-Werror=*` and `-Werror-*` flags by
converting them to `-Wno-error[=]*` flags
* `specific`: preserve explicitly selected warnings as errors, such as
`-Werror=format-truncation`, but reverse the blanket `-Werror`
* `all`: keeps all `-Werror` flags
These can be set globally in config.yaml, through the config command-line flags, or
overridden by a particular package (some packages use Werror as a proxy for determining
support for other compiler features). We chose to use this approach because:
1. removing `-Werror` flags entirely broke *many* build systems, especially autoconf
based ones, because of things like checking `-Werror=feature` and making the
assumption that if that did not error other flags related to that feature would also work
2. Attempting to preserve `-Werror` in some phases but not others caused similar issues
3. The per-package setting came about because some packages, even with all these
protections, still use `-Werror` unsafely. Currently there are roughly 3 such packages
known.
`spack env create` enables a view by default (in a weird hidden
directory, but well...). This is asking for trouble with the other
default of `concretizer:unify:false`, since having different flavors of
the same spec in an environment, leads to collision errors when
generating the view.
A change of defaults would improve user experience:
However, `unify:true` makes most sense, since any time the issue is
brought up in Slack, the user changes the concretization config, since
it wasn't the intention to have different flavors of the same spec, and
install times are decreased.
Further we improve the docs and drop the duplicate root spec limitation
Adds another post install hook that loops over the install prefix, looking for shared libraries type of ELF files, and sets the soname to their own absolute paths.
The idea being, whenever somebody links against those libraries, the linker copies the soname (which is the absolute path to the library) as a "needed" library, so that at runtime the dynamic loader realizes the needed library is a path which should be loaded directly without searching.
As a result:
1. rpaths are not used for the fixed/static list of needed libraries in the dynamic section (only for _actually_ dynamically loaded libraries through `dlopen`), which largely solves the issue that Spack's rpaths are a heuristic (`<prefix>/lib` and `<prefix>/lib64` might not be where libraries really are...)
2. improved startup times (no library search required)
Changes to improve locating shared libraries on Windows, which in
turn enables the use of Clingo. This PR attempts to establish a
proper distinction between linking on Windows vs. Linux/Mac: on
Windows, linking is always done with .lib files (never .dll files).
This somewhat complicates the model since the Spec.lib method could
return libraries that were used for both linking and loading, but
since these are not always the same on Windows, it was decided to
treat Spec.libs as being for link-time libraries. Additional functions
are added to help dependents locate run-time libraries.
* Clingo is now the default concretizer on Windows
* Clingo is now the concretizer used for unit tests on Windows
* Fix a permissions issue that can occur while moving Git files during
fetching/staging
* Packages can now implement "win_add_library_dependent" to register
files/directories that include libraries that would need to link
to dependency dlls
* Packages can now implement "win_add_rpath" to register the locations
of dlls that dependents would want to load
* "Spec.libs" on Windows is updated to return link-time libraries
(i.e. .lib files, rather than .dll files)
* PackageBase.rpath on Windows is now updated to return the most-likely
locations where .dlls will be found (which is generally in the bin/
directory)
"spack install" will not update the binary index if given a concrete
spec, which causes it to fall back to direct fetches when a simple
index update would have helped. For S3 buckets in particular, this
significantly and needlessly slows down the install process.
This commit alters the logic so that the binary index is updated
whenever a by-hash lookup fails. The lookup is attempted again with
the updated index before falling back to direct fetches. To avoid
updating too frequently (potentially once for each spec being
installed), BinaryCacheIndex.update now includes a "cooldown"
option, and when this option is enabled it will not update more
than once in a cooldown window (set in config.yaml).
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
`LD_LIBRARY_PATH` can break system executables (e.g., when an enviornment is loaded) and isn't necessary thanks to `RPATH`s. Packages that require `LD_LIBRARY_PATH` can set this in `setup_run_environment`.
- [x] Prefix inspections no longer set `LD_LIBRARY_PATH` by default
- [x] Document changes and workarounds for people who want `LD_LIBRARY_PATH`
