libuuid is currently contained in util-linux, libuuid and uuid. This
change introduces a new virtual provider `uuid` and renames the existing
`uuid` package to `ossp-uuid`.
util-linux's libuuid is provided in the form of a separate package
util-linux-uuid to make sure that packages depending on uuid and
util-linux can use a separate uuid implementation, which the concretizer
does not allow if libuuid is contained in util-linux.
Users can add test() methods to their packages to run smoke tests on
installations with the new `spack test` command (the old `spack test` is
now `spack unit-test`). spack test is environment-aware, so you can
`spack install` an environment and then run `spack test run` to run smoke
tests on all of its packages. Historical test logs can be perused with
`spack test results`. Generic smoke tests for MPI implementations, C,
C++, and Fortran compilers as well as specific smoke tests for 18
packages.
Inside the test method, individual tests can be run separately (and
continue to run best-effort after a test failure) using the `run_test`
method. The `run_test` method encapsulates finding test executables,
running and checking return codes, checking output, and error handling.
This handles the following trickier aspects of testing with direct
support in Spack's package API:
- [x] Caching source or intermediate build files at build time for
use at test time.
- [x] Test dependencies,
- [x] packages that require a compiler for testing (such as library only
packages).
See the packaging guide for more details on using Spack testing support.
Included is support for package.py files for virtual packages. This does
not change the Spack interface, but is a major change in internals.
Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Co-authored-by: wspear <wjspear@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Since #11598 sbang has been installed within the install_tree. This doesn’t play
nicely with install_tree padding, since sbang can’t do its job if it is installed in a
long path (this is the whole point of sbang).
This PR changes the padding specification. Instead of $padding inside paths,
we now have a separate `padding:` field in the `install_tree` configuration.
Previously, the `install_tree` looked like this:
```
/path/to/opt/spack_padding_padding_padding_padding_padding/
bin/
sbang
.spack-db/
...
linux-rhel7-x86_64/
...
```
```
This PR updates things to look like this:
/path/to/opt/
bin/
sbang
spack_padding_padding_padding_padding_padding/
.spack-db/
...
linux-rhel7-x86_64/
...
So padding is added at the start of all install prefixes *within* the unpadded
root. The database and all installations still go under the padded root.
This ensures that `sbang` is in the shorted possible path while also allowing
us to make long paths for relocatable binaries.
* [NEW] Added amdfftw, amdlibflame and amdscalapack recipes
Updated base fftw, libflame and netlib-scalapack recipes
to accommodate the above listed AMD Optimizing CPU Libraries
which are a set of numerical routines optimized for AMD platforms.
Updated amdblis spack recipe
amdblis:
1. updated with amdblis 2.2 release
amdfftw:
1. "--enable-single" now work as synonym for "--enable-float"
amdlibflame:
1. Added enable_or_disable_threads() to set value for "--enable-multithreading" flag
Libflame:
1. Added enable_or_disable_threads() to set value for "--enable-multithreading" flag
2. Corrected invocation of "enable_or_disable('threads')"
Change-Id: I9da0a2c2c4e2075b7fa2776e7cfe6548a2e0b32f
* Added amd-toolchain-support as maintainers
Added team github account amd-toolchain-support
as maintainers for all the recipes owned by
AMD Optimizing CPU Libraries (AOCL) team
Change-Id: I9a7969bd48fc42cfbb88dd7bd93e0802c6138582
* Incorporated review comments
Updated packages.yaml with aocl components
Handled Flake8 test failures
Change-Id: I0a03f02d8c9f326b2434ec907958c3de3a8e18eb
* Readded accidental removal of stream recipe
amdfftw:
1. Updated the aocc clang selection as per spack standards
fftw:
1. Currently apple-clang section is redundant,
already it is handled in the conflict checks.
Change-Id: Idef4a3f61717eb81f321e0cd16e7ba9619eac846
* Fix for style and docs/validate (pull_request) test
unnumbered format placeholders from {} to {0}
Change-Id: If67a3374177ec067573e5504462d257712fafc05
* changed compiler references to Spack's compiler wrapper:spack_cc, spack_cxx, spack_fc
Change-Id: I7ae29c978fff16e37773913f14c84df232499763
* Removed 'single' variant from amdfftw recipe
Instead of conflict for apple-clang + openmp, handled this senario
via below available feature:
depends_on('llvm-openmp', when='%apple-clang +openmp')
Change-Id: I701b23d83e822a500ca3aaf2b60cc9ace09e13dc
* Added relevant info for users who prefers to use single precision
Change-Id: I3506e21da428ddef5fb7895b5aaed32c2a061ef6
* Minor changes on fftw, amdfftw and libflame
amdfftw:
1. Removed escape symbol to the single quotes
2. Rewording the conflict line from Recommended
to Required
fftw:
1. Reorded to following recommended sections:
versions, variants, dependencies, providers,
patches
libflame:
1. Added provides entry for 5.1.0 version
Change-Id: I21ebff99b6dfde031763154693ecb3f1fa47b476
* Removed single quote from amdfftw docstring to fix style failures
Change-Id: Ife939a5a2f5ccbc8879b730c7bebfe2fcfef9332
* Adding AOCC compiler to SPACK community
The AOCC compiler system offers a high level of advanced optimizations, multi-threading and processor support that includes global optimization, vectorization, inter-procedural analyses, loop transformations, and code generation. AMD also provides highly optimized libraries, which extract the optimal performance from each x86 processor core when utilized. The AOCC Compiler Suite simplifies and accelerates development and tuning for x86 applications.
* Added unit tests for detection and flags for AOCC
* Addressed reviewers comments w.r.t version checks and url,checksum related line lengths
Co-authored-by: Test User <spack@example.com>
The YAML config for paths and modules of external packages has
changed: the new format allows a single spec to load multiple
modules. Spack will automatically convert from the old format
when reading the configs (the updates do not add new essential
properties, so this change in Spack is backwards-compatible).
With this update, Spack cannot modify existing configs/environments
without updating them (e.g. “spack config add” will fail if the
configuration is in a format that predates this PR). The user is
prompted to do this explicitly and commands are provided. All
config scopes can be updated at once. Each environment must be
updated one at a time.
* Separate Apple Clang from LLVM Clang
Apple Clang is a compiler of its own. All places
referring to "-apple" suffix have been updated.
* Hack to use a dash in 'apple-clang'
To be able to use autodoc from Sphinx we need
a valid Python name for the module that contains
Apple's Clang code.
* Updated packages to account for the existence of apple-clang
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
* Added unit test for XCode related functions
Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
This commit removes the DYLD_LIBRARY_PATH variable from the default
modules.yaml for darwin. The rationale behind deleting this
environment variable is that paths in this environment variable take
precedence over the default locations of libraries (usually the
install path of the library), which can lead to linking errors in some
circumstances. For example, executables intended to link with Apple's
system BLAS and LAPACK will instead link to a spack-installed
implementation (e.g., OpenBLAS), causing runtime errors.
These errors are resolved by instead relying on paths set in
DYLD_FALLBACK_LIBRARY_PATH, which is lower in precedence than default
locations of libraries.
Spack currently cannot run as a background process uninterrupted because some of the logging functions used in the install method (especially to create the dynamic verbosity toggle with the v key) cause the OS to issue a SIGTTOU to Spack when it's backgrounded.
This PR puts the necessary gatekeeping in place so that Spack doesn't do anything that will cause a signal to stop the process when operating as a background process.
fixes#11555
Every path in CPATH is equivalent to a -I path to the compiler,
while every path in *_INCLUDE_PATH is equivalent to -isystem.
The latter avoids the noise due to warnings coming from 3rd party
libraries that a project depends on.
Added INCLUDE env variable (Intel Fortran, .mod files)
connect_timeout can be used to increase the time Spack waits for the
server to answer. This can be used to work around slow connections or
servers.
Fixes#14700
Fixes#9394Closes#13217.
## Background
Spack provides the ability to enable/disable parallel builds through two options: package `parallel` and configuration `build_jobs`. This PR changes the algorithm to allow multiple, simultaneous processes to coordinate the installation of the same spec (and specs with overlapping dependencies.).
The `parallel` (boolean) property sets the default for its package though the value can be overridden in the `install` method.
Spack's current parallel builds are limited to build tools supporting `jobs` arguments (e.g., `Makefiles`). The number of jobs actually used is calculated as`min(config:build_jobs, # cores, 16)`, which can be overridden in the package or on the command line (i.e., `spack install -j <# jobs>`).
This PR adds support for distributed (single- and multi-node) parallel builds. The goals of this work include improving the efficiency of installing packages with many dependencies and reducing the repetition associated with concurrent installations of (dependency) packages.
## Approach
### File System Locks
Coordination between concurrent installs of overlapping packages to a Spack instance is accomplished through bottom-up dependency DAG processing and file system locks. The runs can be a combination of interactive and batch processes affecting the same file system. Exclusive prefix locks are required to install a package while shared prefix locks are required to check if the package is installed.
Failures are communicated through a separate exclusive prefix failure lock, for concurrent processes, combined with a persistent store, for separate, related build processes. The resulting file contains the failing spec to facilitate manual debugging.
### Priority Queue
Management of dependency builds changed from reliance on recursion to use of a priority queue where the priority of a spec is based on the number of its remaining uninstalled dependencies.
Using a queue required a change to dependency build exception handling with the most visible issue being that the `install` method *must* install something in the prefix. Consequently, packages can no longer get away with an install method consisting of `pass`, for example.
## Caveats
- This still only parallelizes a single-rooted build. Multi-rooted installs (e.g., for environments) are TBD in a future PR.
Tasks:
- [x] Adjust package lock timeout to correspond to value used in the demo
- [x] Adjust database lock timeout to reduce contention on startup of concurrent
`spack install <spec>` calls
- [x] Replace (test) package's `install: pass` methods with file creation since post-install
`sanity_check_prefix` will otherwise error out with `Install failed .. Nothing was installed!`
- [x] Resolve remaining existing test failures
- [x] Respond to alalazo's initial feedback
- [x] Remove `bin/demo-locks.py`
- [x] Add new tests to address new coverage issues
- [x] Replace built-in package's `def install(..): pass` to "install" something
(i.e., only `apple-libunwind`)
- [x] Increase code coverage
Add a configuration option to suppress gpg warnings during binary
package verification. This only suppresses warnings: a gpg failure
will still fail the install. This allows users who have already
explicitly trusted the gpg key they are using to avoid seeing
repeated warnings that it is self-signed.
Add a new entry in `config.yaml`:
config:
shared_linking: 'rpath'
If this variable is set to `rpath` (the default) Spack will set RPATH in ELF binaries. If set to `runpath` it will set RUNPATH.
Details:
* Spack cc wrapper explicitly adds `--disable-new-dtags` when linking
* cc wrapper also strips `--enable-new-dtags` from the compile line
when disabling (and vice versa)
* We specifically do *not* add any dtags flags on macOS, which uses
Mach-O binaries, not ELF, so there's no RUNPATH)
While the build environment already takes share/pkgconfig into account,
the generated module files etc. only consider lib/pkgconfig and
lib64/pkgconfig.
Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.
A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.
* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator
* When cleaning the stage root, only remove directories that appear
to be used for staging Spack packages. Previously Spack was clearing
all directories in the stage root, which could remove content not
related to Spack if the user chose a staging root which contains
files/directories not managed by Spack.
* The documentation is updated with warnings about choosing a stage
directory that is only managed by Spack (although generally the
check added in this PR for "spack clean" should avoid removing
content that was not created by Spack)
* The default stage directory (in config.yaml) is now
$tempdir/$user/spack-stage and the logic is updated to omit the
$user portion of this path if $tempdir already contains a $user
directory.
* When creating stage root assign user read/write permissions to all
directories in the path under $user. Previously Spack was assigning
the permissions of the first existing parent directory
Fixes#11163
The goal of this work is to simplify stage directory structures by eliminating use of symbolic links. This means, among other things, that` $spack/var/spack/stage` will no longer be the core staging directory. Instead, the first accessible `config:build_stage` path will be used.
Spack will no longer automatically append `spack-stage` (or the like) to configured build stage directories so the onus of distinguishing the directory from other work -- so the other work is not automatically removed with a `spack clean` operation -- falls on the user.
The original packaging, with openfoam as a virtual and with
openfoam-com, openfoam-org and openfoam-extend as
providers, adds an obfuscation layer without any benefit.
- Rename `openfoam-com` to `openfoam`, and remove the
`openfoam` virtual package.
* config:build_jobs now controls the number of parallel jobs to spawn during
builds, but cannot ever exceed the number of cores on the machine.
* The default is set to 16 or the number of available cores, whatever
is lowest.
* Updated docs to reflect the changes done to limit parallel builds
* Add Fujitsu compiler to Spack.
* Fixes for flake8
* Chenges location of FCC to subdirectory called case-insensitive
* Add compiler tests for Fujitsu compiler
* Modify the logic of taking compiler version for new version of Fujitsu compiler
* mesa: re-writen to new meson build system
* xorg-server: Remove unneeded mesa dependency
* glx: provide the virtual GLX package
* Convert most "mesa" dependencies to use virtual gl and glx dependencies
* mesa: Switch to always building from the git repo instead of a tarball
* Use gl and glx version ranges instead of exact versions
* glx: provide a default package
* add auto to swr variant
* Revert "add auto to swr variant"
This reverts commit 2e7cd8a8aca07767c00657d0d09985ddd6306883.
* update mesa package to version 19.0
* set +glx by default only when running on linux
* fix style issues in geant4 package
* remove virtual package "egl"
* add cxxstd variant
* add CMake constraints based on platform/version
* add older versions
* update boost dependency version constraints (which are closely
tied to mysql version) and update boost cxxstd choice to be
the same as the cxxstd chosen for mysql
* add client-only support (including a patch for 5.5.x)
* record the mysql package as a provider of the mysql-client virtual
* mariadb-c-client is a new package using the distinct, LGPL,
MySQL-compatible client library from mariadb.com. It provides the
virtual package mariadb-client
* mariadb is recorded as a provider of the mariadb-client virtual
* The mysql-client virtual package is also added, and mariadb-c-client
is recorded as a provider for it
* Add binary package of the IBM Java SDK for big and little-endian
powerpc (power7, 8 and 9). The jdk and openjdk packages only install
on x86_64.
* Add ibm-java as a java provider
* The jdk and openjdk packages only install on x86_64. Add conflicts
for ppc64 and ppc64le to jdk and openjdk.
* Create option to build missing compilers and add them to config before installing packages that use them
* Clean up kwarg passing for do_install, put compiler bootstrapping in separate method
This spack command adds a new schema for a file which describes the
builder containers available, along with the compilers availabe on
each builder. The release-jobs command then generates the .gitlab-ci.yml
file by first expanding the release spec set, concretizing each spec
(in an appropriate docker container if --this-machine-only argument is
not provided on command line), and then combining and staging all the
concrete specs as jobs to be run by gitlab.
* Remove /nfs/tmp2 from default configuration
* /nfs/tmp2 is going away from LC... and doesn’t exist for the rest of the world.
* update documentation to remove /nfs/tmp2 as well
