The gcc compiler can be configured to use `ld.gold` by default. It will
then call `ld.gold` explicitly when linking. When so, spack need to have
a ld.gold wrapper in PATH to inject rpaths link flags etc...
Also I wouldn't be surprised to see some package calling `ld.gold`
directly.
As for ld.gold, the argument could be made that we want to support any
package that could call ld.lld.
For configure (e.g. for hdf5) to pass, this option needs to be pulled out when invoked in ccld mode.
I thought it had fixed the issue but I still saw it after that. After some digging, my guess is that I was able
to get hdf5 to build with ifort instead of ifx. Lot of overlapping changes occurring at the time, as it were.
There are still outstanding issues building hdf5 with ifx, and Intel is looking into what appears to be a
compiler bug, but this manifests during build and is likely a separate issue.
I have verified that the making the edit in 'ccld' mode removes the -loopopt=0 and enables hdf5 to pass
configure. It should be fine to make the edit in 'ld' mode as well, but I have not tested that and didn't
include an -or- condition for it.
In the face of two consecutive spaces in the command line, the compiler wrapper would skip all remaining arguments, causing problems building py-scipy with Intel compiler. This PR solves the problem.
* Fixed compiler wrapper in the face of extra spaces between arguments
Co-authored-by: Elizabeth Fischer <elizabeth.fischer@alaska.edu>
Autoconf before 2.70 will erroneously pass ifx's -loopopt argument to the
linker, requiring all packages to use autoconf 2.70 or newer to use ifx.
This is a hotfix enabling ifx to be used in Spack. Instead of bothering
to upgrade autoconf for every package, we'll just strip out the
problematic flag if we're in `ld` mode.
- [x] Add a conditional to the `cc` wrapper to skip `-loopopt` in `ld`
mode. This can probably be generalized in the future to strip more
things (e.g., via an environment variable we can constrol from
Spack) but it's good enough for now.
- [x] Add a test ensuring that `-loopopt` arguments are stripped in link
mode, but not in compile mode.
Some compilers, such as the NV compilers, do not recognize -isystem
dir when specified without a space.
Works: -isystem ../include
Does not work: -isystem../include
This PR updates the compiler wrapper to include the space with -isystem.
- [x] add `concretize.lp`, `spack.yaml`, etc. to licensed files
- [x] update all licensed files to say 2013-2021 using
`spack license update-copyright-year`
- [x] appease mypy with some additions to package.py that needed
for oneapi.py
As of #13100, Spack installs the dependencies of a _single_ spec in parallel.
Environments, when installed, can only get parallelism from each individual
spec, as they're installed in order. This PR makes entire environments build
in parallel by extending Spack's package installer to accept multiple root
specs. The install command and Environment class have been updated to use
the new parallel install method.
The specs and kwargs for each *uninstalled* package (when not force-replacing
installations) of an environment are collected, passed to the `PackageInstaller`,
and processed using a single build queue.
This introduces a `BuildRequest` class to track install arguments, and it
significantly cleans up the code used to track package ids during installation.
Package ids in the build queue are now just DAG hashes as you would expect,
Other tasks:
- [x] Finish updating the unit tests based on `PackageInstaller`'s use of
`BuildRequest` and the associated changes
- [x] Change `environment.py`'s `install_all` to use the `PackageInstaller` directly
- [x] Change the `install` command to leverage the new installation process for multiple specs
- [x] Change install output messages for external packages, e.g.:
`[+] /usr` -> `[+] /usr (external bzip2-1.0.8-<dag-hash>`
- [x] Fix incomplete environment install's view setup/update and not confirming all
packages are installed (?)
- [x] Ensure externally installed package dependencies are properly accounted for in
remaining build tasks
- [x] Add tests for coverage (if insufficient and can identity the appropriate, uncovered non-comment lines)
- [x] Add documentation
- [x] Resolve multi-compiler environment install issues
- [x] Fix issue with environment installation reporting (restore CDash/JUnit reports)
* enable flatcc to be built with gcc/9.X.X
* add static option for building libyogrt
* cleanup
* Initial working version
* rework new oneapi wrappers
* tested and removed my initials from source
* cleanup
* Update __init__.py
* remove whitespace
* working now with mods for testing, detection. Detection for oneapi is working, but entry needs to be modified to add link path for libimf.so. Cleared cruft for old Intel versions
* fixed some formatting
* cleanup
* flake8 cleanup
* flake8
* fixed syntax of compiler version detection tests
* fixed syntax of compiler version detection tests
modified: detection.py
* fix typo
* fixes for compilers tests
* remove erroneous tests for outdated -std= flags, remove ifx version check (output won't parse)
Co-authored-by: Frank Willmore <willmore@anl.gov>
* Adding AOCC compiler to SPACK community
The AOCC compiler system offers a high level of advanced optimizations, multi-threading and processor support that includes global optimization, vectorization, inter-procedural analyses, loop transformations, and code generation. AMD also provides highly optimized libraries, which extract the optimal performance from each x86 processor core when utilized. The AOCC Compiler Suite simplifies and accelerates development and tuning for x86 applications.
* Added unit tests for detection and flags for AOCC
* Addressed reviewers comments w.r.t version checks and url,checksum related line lengths
Co-authored-by: Test User <spack@example.com>
* Add nvhpc compiler definition: "spack compiler add" will now look
for instances of the NVIDIA HPC SDK compiler executables
(nvc, nvc++, nvfortran) in supplied paths
* Add the nvhpc package which installs the nvhpc compiler
* Add testing for nvhpc detection and C++-standard/pic flags
Co-authored-by: Scott McMillan <smcmillan@nvidia.com>
If the Spack compiler wrapper encounters any "-isystem" option, then
when adding include directories for Spack dependencies, Spack will
use "-isystem" instead of "-I". This prevents Spack-generated "-I"
options from overriding the "-isystem" options generated by the build
system. To ensure that build-system "-isystem" directories are
searched first, Spack places all of its inserted "-isystem"
directories after.
The new ordering of -isystem includes is:
* -isystem from build system (not system directories)
* -isystem from Spack
* -isystem from build system (for directories like /usr/include)
The prior order of "-I" arguments is preserved (although as of this
commit Spack no longer generates -I if -isystem is detected):
* -I from build system (not system directories)
* -I from Spack (only if there are no "-isystem" options)
* -I from build system (for directories like /usr/include)
Spack's fflags are meant for both f77 and fc. Therefore, they must
be passed as FFLAGS and FCFLAGS to the configure scripts of
Autotools-based packages.
Add a new entry in `config.yaml`:
config:
shared_linking: 'rpath'
If this variable is set to `rpath` (the default) Spack will set RPATH in ELF binaries. If set to `runpath` it will set RUNPATH.
Details:
* Spack cc wrapper explicitly adds `--disable-new-dtags` when linking
* cc wrapper also strips `--enable-new-dtags` from the compile line
when disabling (and vice versa)
* We specifically do *not* add any dtags flags on macOS, which uses
Mach-O binaries, not ELF, so there's no RUNPATH)
Spack can now:
- label ppc64, ppc64le, x86_64, etc. builds with specific
microarchitecture-specific names, like 'haswell', 'skylake' or
'icelake'.
- detect the host architecture of a machine from /proc/cpuinfo or similar
tools.
- Understand which microarchitectures are compatible with which (for
binary reuse)
- Understand which compiler flags are needed (for GCC, so far) to build
binaries for particular microarchitectures.
All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets. The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.
The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines. Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.
Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.
Specifics:
- Add microarchitectures.json with descriptions of architectures
- Relaxed semantic of compiler's "target" attribute. Before this change
the semantic to check if a compiler could be viable for a given target
was exact match. This made sense as the finest granularity of targets
was architecture families. As now we can target micro-architectures,
this commit changes the semantic by interpreting as the architecture
family what is stored in the compiler's "target" attribute. A compiler
is then a viable choice if the target being concretized belongs to the
same family. Similarly when a new compiler is detected the architecture
family is stored in the "target" attribute.
- Make Spack's `cc` compiler wrapper inject target-specific flags on the
command line
- Architecture concretization updated to use the same algorithm as
compiler concretization
- Micro-architecture features, vendor, generation etc. are included in
the package hash. Generic architectures, such as x86_64 or ppc64, are
still dumped using the name only.
- If the compiler for a target is not supported exit with an intelligible
error message. If the compiler support is unknown don't try to use
optimization flags.
- Support and define feature aliases (e.g., sse3 -> ssse3) in
microarchitectures.json and on Microarchitecture objects. Feature
aliases are defined in targets.json and map a name (the "alias") to a
list of rules that must be met for the test to be successful. The rules
that are available can be extended later using a decorator.
- Implement subset semantics for comparing microarchitectures (treat
microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
a) can all be false.
- Implement logic to automatically demote the default target if the
compiler being used is too old to optimize for it. Updated docs to make
this behavior explicit. This avoids surprising the user if the default
compiler is older than the host architecture.
This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type. A more careful
refactor that takes into account the type system might be due later.
Co-authored-by: Gregory Becker <becker33.llnl.gov>
Add llnl.util.cpu_name, with initial support for detecting different
microarchitectures on Linux. This also adds preliminary changes for
compiler support and variants to control the optimizatoin levels by
target.
This does not yet include translations of targets to particular
compilers; that is left to another PR.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Uses code from CMake to detect implicit link paths from compilers
System paths are filtered out of implicit link paths
Implicit link paths added to compiler config and object under `implicit_rpaths`
Implicit link paths added as rpaths to compile line through env/cc wrapper
Authored by: "Ben Boeckel <ben.boeckel@kitware.com>"
Co-authored by: "Peter Scheibel <scheibel1@llnl.gov>"
Co-authored by: "Gregory Becker <becker33@llnl.gov>"
* Add Fujitsu compiler to Spack.
* Fixes for flake8
* Chenges location of FCC to subdirectory called case-insensitive
* Add compiler tests for Fujitsu compiler
* Modify the logic of taking compiler version for new version of Fujitsu compiler
Fixes#11335
Update the Spack compiler wrappers to add the headerpad_max_install_names
linker flag on MacOS. This allows the install_name_tool to rewrite
the RPATH entry of the binary to be longer if needed. This is
primarily useful for creating and distributing binary caches of
packages (i.e. using the "spack buildcache" command); binary caches
created on MacOS before this commit may not successfully relocate
(if the target root path is larger).
Fixes#11070#11010
Spack attempts to intercede on behalf of all compiler invocations for
a build. This involves adding its wrappers to PATH. Cray systems
include a "ftn" executable and Spack was only redirecting this call
when the Spec was built with cce. This updates the compiler wrappers
to add "ftn" in all cases.
This restores the use of Package.headers when computing -I options
for building a package that was added in #8136 and reverted in
#10604. #8136 used utility logic that located all header files in
an installation prefix, and calculated the -I options as the
immediate roots containing those header files.
In some cases, for a package containing a directory structure like
prefix/
include/
ex1.h
subdir/
ex2.h
dependents may expect to include ex2.h relative to 'include', and
adding 'prefix/include/subdir' as a -I was causing errors,
in particular if ex2.h has the same name as a system header.
This updates header utility logic to by default return the base
"include" directory when it exists, rather than subdirectories.
It also makes it possible for package implementers to override
Package.headers to return the subdirectory when it is required
(for example with libxml2).
fixes#10601
Due to a bug this attribute is wrong for packages that use directories
as namespaces. For instance it will add "<boost-prefix>/include/boost"
instead of "<boost-prefix>/include" to the include path.
As a minor addition a few loops in the compiler wrappers have been
simplified.
Fixes#7855Closes#8070Closes#2645
When searching for library directories (e.g. to add "-L" arguments to
the compiler wrapper) Spack was only trying the "lib/" and "lib64/"
directories for each dependency install prefix; this missed cases
where packages would install libraries to subdirectories and also was
not customizable. This PR makes use of the ".headers" and ".libs"
properties for more-advanced location of header/library directories.
Since packages can override the default behavior of ".headers" and
".libs", it also allows package writers to customize.
The following environment variables which used to be set by Spack
for a package build have been removed:
* Remove SPACK_PREFIX and SPACK_DEPENDENCIES environment variables as
they are no-longer used
* Remove SPACK_INSTALL environment variable: it was not used before
this PR
* Remove Cray CC compilers causing problems on case-insensitive filesystems
* cray -> cce
* Ensure that compiler-specific directory comes first in build-env
* Point to compiler-specific symlinks
* Initial compiler support
* added arm.py
* Changed licence to Arm suggested header
* Changed licence to the same as clang.py
Main author of file is Nick Forrington <Nick.Forrington@arm.com>
Minor changes by Srinath Vadlamani <srinath.vadlamani@arm.com>
* compilers: add arm compiler detection to Spack
- added arm.py with support for detecting `armclang` and `armflang`
Co-authored-by: Srinath Vadlamani <srinath.vadlamani@arm.com>
* Changed to using get get_compiler_version
* linking to general cc for arm compiler
* For arm compiler add CFLAGS to use compiler-rt rtlib.
* Escape for special characters in rexep
* Cleaned up for Flake8 to pass.
* libcompiler-rt should be part of the LDFLAGS not CFLAGS
* fixed m4 when using clang to used LDFLAGS. Fixed comments for arm.py to display compiler --version output with # NOAQ for flakes pass.
* added arm compilers
* proper linked names
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
- core and most packages are (Apache-2.0 OR MIT)
- a very small number of remaining packages are LGPL-2.1-only
- cc cleanup caused a parsing regression in flag handling
- We added proper quoting to array expansions, but flag variables were
never actually converted to arrays. Old code relied on this.
This commit:
- Adds reads to convert flags to arrays.
- Makes the cc test check for improper space handling to prevent future
regressions.
- flags were prepended in reverse order to args, but this makes it hard
to see what order they'll be in on the final command line.
- add them in the order they'll appear to make cc easier to maintain.
- simplify code for assembling the command line
- fix separator used in SPACK_SYSTEM_DIRS test
- This corrects most of the issues found by shellcheck
- This also uses ':' as the delimiter for SPACK_SYSTEM_DIRS, for
consistency with other variables.
- filtering using sed causes most builds to slow down quite a bit, as the
compiler wrapper has to run sed many times, and *it* runs many times
- do the system directory parsing directly in bash
- Adding -L and -Wl,-rpath to compile-only command lines ("cc mode" or
"-c") causes clang (if not also other compilers) to emit warnings that
confuse configure systems.
- Clang will print warnings about unused command-line arguments.
- This fix ensures that -L and -Wl,-rpath are not added if the compile
line is just building an object file with -c
- This also cleans up the cc script in several places.
Spack currently prepends include paths, library paths, and rpaths to the
compile line. This causes problems when a header or library in the package
has the same name as one exported by one of its dependencies. The
*dependency's* header will be preferred over the package's, which is not
what most builds expect. This also breaks some of our production codes.
This restores the original cc behavior (from *very* early Spack) of parsing
compiler arguments out by type (`-L`, `-I`, `-Wl,-rpath`) and reconstituting
the full command at the end.
`<includes> <other_args> <library dirs> <rpaths>`
This differs from the original behavior in one significant way, though: it
*appends* the library arguments so that dependency libraries do not shadow
those in the build.
This is safe because semantics aren't affected by *interleaving* `-I`, `-L`,
and `-Wl,-rpath` arguments with others, only with each other (so the order of
two `-L` args affects the search path, but we search for all libraries on the
command line using the same search path).
We preserve the following:
1. Any system directory in the paths will be listed last.
2. The root package's include/library/RPATH flags come before flags of the
same type for any dependency.
3. Order will be preserved within flags passed by the build (except system
paths, which are moved to be last)
4. Flags for dependencies will appear between the root flags and the system
flags, and the flags for any dependency will come before those for *its*
dependencies (this is for completeness -- we already guarantee this in
`build_environment.py`)
Spack currently prepends include paths, library paths, and rpaths to the compile line. This causes problems when a header or library in the package has the same name as one exported by one of its dependencies. The *dependency's* header will be preferred over the package's, which is not what most builds expect. This also breaks some of our production codes.
This restores the original cc behavior (from *very* early Spack) of parsing compiler arguments out by type (`-L`, `-I`, `-Wl,-rpath`) and reconstituting the full command at the end.
`<includes> <other_args> <library dirs> <rpaths>`
This differs from the original behavior in one significant way, though: it *appends* the library arguments so that dependency libraries do not shadow those in the build.
This is safe because semantics aren't affected by *interleaving* `-I`, `-L`, and `-Wl,-rpath` arguments with others, only with each other (so the order fo two `-L` args affects the search path, but we search for all libraries on the command line using the same search path).
We preserve the following:
1. Any system directory in the paths will be listed last.
2. The root package's include/library/RPATH flags come before flags of the same type for any dependency.
3. Order will be preserved within flags passed by the build (except system paths, which are moved to be last)
4. Flags for dependencies will appear between the root flags and the system flags, and the flags for any dependency will come before those for *its* dependencies (this is for completeness -- we already guarantee this in `build_environment.py`)
If the user sets "ccache: true" in spack's config.yaml, Spack will use an available
ccache executable when compiling c/c++ code. This feature is disabled by default
(i.e. "ccache: false") and the documentation is updated with how to enable
ccache support
The name of the debug log written by the cc compiler wrapper was given
by Spec.short_spec, which includes the architecture. Somewhere along
the line Spec.format started adding spaces around the architecture
property so the filename started including spaces; the cc wrapper
script appears to ignore this, so files like spack-cc-bzip2-....in.log
(which record the wrapped compiler invocations) were not being
generated. This uses a different format string from the spec to
generate the wrapper log file names (which does not include spaces).
