Addresses #12804
This PR adds the creation of the remaining (16) templates to ensure we can create them with expected content. The goal is to facilitate catching during testing.
Spack doesn't need `requests`, and neither does `jsonschema`, but
`jsonschema` tries to import it, and it'll succeed if `requests` is on
your machine (which is likely, given how popular it is). This commit
removes the import to improve Spack's startup time a bit.
On a mac with SSD, the import of requests is ~28% of Spack's startup time
when run as `spack --print-shell-vars sh,modules` (.069 / .25 seconds),
which is what `setup-env.sh` runs.
On a Linux cluster where Python is mounted from NFS, this reduces
`setup-env.sh` source time from ~1s to .75s.
Note: This issue will be eliminated if we upgrade to a newer `jsonschema`
(we'd need to drop Python 2.6 for that). See
https://github.com/Julian/jsonschema/pull/388.
- This is needed to support Cray machines -- we need an architecture
mic_knl > x86_64
- We used Cray's naming scheme for this target to make it work seamlessly
with the module-based detection sccheme on Cray. mic_knl is pretty
much dead, so this will be the last succh target. We will need to work
wtih Cray and other vendors in the future.
Seamless translation from 'target=<generic>' to either
- target.family == <generic> (in methods)
- 'target=<generic>:' (in directives)
Also updated docs to show ranges in directives.
Spack can now:
- label ppc64, ppc64le, x86_64, etc. builds with specific
microarchitecture-specific names, like 'haswell', 'skylake' or
'icelake'.
- detect the host architecture of a machine from /proc/cpuinfo or similar
tools.
- Understand which microarchitectures are compatible with which (for
binary reuse)
- Understand which compiler flags are needed (for GCC, so far) to build
binaries for particular microarchitectures.
All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets. The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.
The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines. Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.
Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.
Specifics:
- Add microarchitectures.json with descriptions of architectures
- Relaxed semantic of compiler's "target" attribute. Before this change
the semantic to check if a compiler could be viable for a given target
was exact match. This made sense as the finest granularity of targets
was architecture families. As now we can target micro-architectures,
this commit changes the semantic by interpreting as the architecture
family what is stored in the compiler's "target" attribute. A compiler
is then a viable choice if the target being concretized belongs to the
same family. Similarly when a new compiler is detected the architecture
family is stored in the "target" attribute.
- Make Spack's `cc` compiler wrapper inject target-specific flags on the
command line
- Architecture concretization updated to use the same algorithm as
compiler concretization
- Micro-architecture features, vendor, generation etc. are included in
the package hash. Generic architectures, such as x86_64 or ppc64, are
still dumped using the name only.
- If the compiler for a target is not supported exit with an intelligible
error message. If the compiler support is unknown don't try to use
optimization flags.
- Support and define feature aliases (e.g., sse3 -> ssse3) in
microarchitectures.json and on Microarchitecture objects. Feature
aliases are defined in targets.json and map a name (the "alias") to a
list of rules that must be met for the test to be successful. The rules
that are available can be extended later using a decorator.
- Implement subset semantics for comparing microarchitectures (treat
microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
a) can all be false.
- Implement logic to automatically demote the default target if the
compiler being used is too old to optimize for it. Updated docs to make
this behavior explicit. This avoids surprising the user if the default
compiler is older than the host architecture.
This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type. A more careful
refactor that takes into account the type system might be due later.
Co-authored-by: Gregory Becker <becker33.llnl.gov>
Add llnl.util.cpu_name, with initial support for detecting different
microarchitectures on Linux. This also adds preliminary changes for
compiler support and variants to control the optimizatoin levels by
target.
This does not yet include translations of targets to particular
compilers; that is left to another PR.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Move verbose messages to debug level
get_patchelf should return None for test platform as well because create_buildinfo invokes patchelf to get rpaths.
Update command-line (CLI) parsing to understand references to yaml
files that store Spack specs. Where a file reference is encountered,
the full Spec in the file will be read in. A file reference may
appear anywhere that a spec could appear before. For example, if you
write "spack spec -y openmpi > openmpi.yaml" you may then install the
spec using the yaml file by running "spack install ./openmpi.yaml";
you can also refer to dependencies in this way (e.g.
"spack install foo^./openmpi.yaml").
There are two requirements for file references:
* A file path entered on the CLI must include a "/" even if the file
exists in your current working directory. For example, if you
create an openmpi.yaml file as above and run
"spack install openmpi.yaml" from the same directory, it will
report an error.
* A file path entered on the CLI must end with ".yaml"
This commit adds error messages to clearly inform the user of both
violations.
* implicit_rpaths are now removed from compilers.yaml config and are always instantiated dynamically, this occurs one time in the build_environment module
* per-compiler list required libraries (e.g. libstdc++, libgfortran) and whitelist directories from rpaths including those libraries. Remove non-whitelisted implicit rpaths. Some libraries default for all compilers.
* reintroduce 'implicit_rpaths' as a config variable that can be used to disable Spack insertion of compiler RPATHs generated at build time.
Fixes#12829
This adds a variant to the util-linux package that controls whether it
builds its own libuuid. Variant defaults to True. It enables other
packages to choose to get libuuid from the libuuid package instead.
This also changes the cryptsetup package to build util-linux with
~libuuid (so it uses an explicitly-Spack-built instance of libuuid
instead).
Fixes#12732Fixes#12767c22a145 added automatic detection and RPATHing of compiler libraries
to Spack builds. However, in cases where the parsing/detection logic
fails this was terminating the build. This makes the compiler library
detection "best-effort" and reports an issue when the detection fails
rather than terminating the build.
