Commit graph

150 commits

Author SHA1 Message Date
Harmen Stoppels
8446bebdd9
Revert "Separable module configurations (#22588)" (#23674)
This reverts commit cefbe48c89.
2021-05-17 06:42:48 -07:00
Greg Becker
cefbe48c89
Separable module configurations (#22588)
Currently, module configurations are inconsistent because modulefiles are generated with the configs for the active environment, but are shared among all environments (and spack outside any environment).

This PR fixes that by allowing Spack environments (or other spack config scopes) to define additional sets of modules to generate. Each set of modules can enable either lmod or tcl modules, and contains all of the previously available module configuration. The user defines the name of each module set -- the set configured in Spack by default is named "default", and is the one returned by module manipulation commands in the absence of user intervention.

As part of this change, the module roots configuration moved from the `config` section to inside each module configuration.

Additionally, it adds a feature that the modulefiles for an environment can be configured to be relative to an environment view rather than the underlying prefix. This will not be enabled by default, as it should only be enabled within an environment and for non-default views constructed with separate projections per-spec.

TODO:
- [x] code changes to support multiple module sets
- [x] code changes to support modules relative to a view
- [x] Tests for multiple module configurations
- [x] Tests for modules relative to a view
- [x] Backwards compatibility for module roots from config section
- [x] Backwards compatibility for default module set without the name specified
- [x] Tests for backwards compatibility
2021-05-14 15:03:28 -07:00
Harmen Stoppels
1db6cd5d16
Make -j flag less exceptional (#22360)
* Make -j flag less exceptional

The -j flag in spack behaves differently from make, ctest, ninja, etc,
because it caps the number of jobs to an arbitrary number 16.
Spack will behave like other tools if `spack install` uses a reasonable
default, and `spack install -j <num>` *overrides* that default.

This will be particularly useful for Spack usage outside of a traditional
HPC context and for HPC centers that encourage users to compile on
login nodes with many cores instead of on compute nodes, which has
become increasingly common as individual nodes have more cores.

This maintains the existing default value of min(num_cpus, 16). However, 
as it is right now, Spack does a poor job at determining the number of 
cpus on linux, since it doesn't take cgroups into account. This is
particularly problematic when using distributed builds with slurm. This PR
also introduces `spack.util.cpus.cpus_available()` to consolidate
knowledge on determining the number of available cores, and improves
core detection for linux. This should also improve core detection for Docker/
Kubernetes, which also use cgroups.
2021-03-30 12:03:50 -07:00
Massimiliano Culpo
796588b2fd
zig: add new package at v0.7.1 (#22046) 2021-03-03 13:30:41 +01:00
Harmen Stoppels
0664b90751
Drop compiler variables from spack load (#21699)
Drops:

* C_INCLUDE_PATH
* CPLUS_INCLUDE_PATH
* LIBRARY_PATH
* INCLUDE

We already decided to use C_INCLUDE_PATH, CPLUS_INCLUDE_PATH, INCLUDE over CPATH here:

https://github.com/spack/spack/pull/14749

However, none of these flags apply to Fortran on Linux. So for consistency it seems better to make the user use -I and -L flags by hand or through pkgconfig.
2021-02-23 13:35:19 +01:00
Adam J. Stewart
7ccb9992a6
Procedure to deprecate old versions of software (#20767)
* Procedure to deprecate old versions of software

* Add documentation

* Fix bug in logic

* Update tab completion

* Deprecate legacy packages

* Deprecate old mxnet as well

* More explicit docs
2021-02-09 13:51:18 -05:00
Laurent Aphecetche
e0b7ca47d8
config: (darwin only) change prefix of external libuuid (#21480) 2021-02-04 14:01:40 -06:00
Adam J. Stewart
5964cb49d9
Use system libuuid on macOS (#20608) 2020-12-30 10:13:34 -06:00
Michael Kuhn
c22141f444
Introduce virtual provider uuid (#18322)
libuuid is currently contained in util-linux, libuuid and uuid. This
change introduces a new virtual provider `uuid` and renames the existing
`uuid` package to `ossp-uuid`.

util-linux's libuuid is provided in the form of a separate package
util-linux-uuid to make sure that packages depending on uuid and
util-linux can use a separate uuid implementation, which the concretizer
does not allow if libuuid is contained in util-linux.
2020-12-29 10:12:26 -06:00
Chuck Atkins
7d79db2de7
Fix Mesa GLES conflicts (#20184) 2020-11-30 23:54:24 +01:00
Greg Becker
77b2e578ec
spack test (#15702)
Users can add test() methods to their packages to run smoke tests on
installations with the new `spack test` command (the old `spack test` is
now `spack unit-test`). spack test is environment-aware, so you can
`spack install` an environment and then run `spack test run` to run smoke
tests on all of its packages. Historical test logs can be perused with
`spack test results`. Generic smoke tests for MPI implementations, C,
C++, and Fortran compilers as well as specific smoke tests for 18
packages.

Inside the test method, individual tests can be run separately (and
continue to run best-effort after a test failure) using the `run_test`
method. The `run_test` method encapsulates finding test executables,
running and checking return codes, checking output, and error handling.

This handles the following trickier aspects of testing with direct
support in Spack's package API:

- [x] Caching source or intermediate build files at build time for
      use at test time.
- [x] Test dependencies,
- [x] packages that require a compiler for testing (such as library only
      packages).

See the packaging guide for more details on using Spack testing support.
Included is support for package.py files for virtual packages. This does
not change the Spack interface, but is a major change in internals.

Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
Co-authored-by: wspear <wjspear@gmail.com>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-11-18 02:39:02 -08:00
Todd Gamblin
14ab63f97c concretizer: add a configuration option to use new or old concretizer
- [x] spec.py can call out to the new concretizer
- [x] config.yaml now has an option to choose a concretizer (original, clingo)
2020-11-17 10:04:13 -08:00
Greg Becker
fafff0c6c0
move sbang to unpadded install tree root (#19640)
Since #11598 sbang has been installed within the install_tree. This doesn’t play
nicely with install_tree padding, since sbang can’t do its job if it is installed in a
long path (this is the whole point of sbang).

This PR changes the padding specification.  Instead of $padding inside paths,
we now have a separate `padding:` field in the `install_tree` configuration.

Previously, the `install_tree` looked like this:

```
    /path/to/opt/spack_padding_padding_padding_padding_padding/
        bin/
            sbang
        .spack-db/
            ...
        linux-rhel7-x86_64/
            ...
```

```
This PR updates things to look like this:

    /path/to/opt/
        bin/
            sbang
        spack_padding_padding_padding_padding_padding/
            .spack-db/
                ...
            linux-rhel7-x86_64/
                ...

So padding is added at the start of all install prefixes *within* the unpadded
root.  The database and all installations still go under the padded root.

This ensures that `sbang` is in the shorted possible path while also allowing
us to make long paths for relocatable binaries.
2020-11-12 16:08:55 -08:00
vijay kallesh
51a9473bf8
[NEW] Added amdfftw, amdlibflame and amdscalapack recipes (#19457)
* [NEW] Added amdfftw, amdlibflame and amdscalapack recipes

Updated base fftw, libflame and netlib-scalapack recipes
to accommodate the above listed AMD Optimizing CPU Libraries
which are a set of numerical routines optimized for AMD platforms.

Updated amdblis spack recipe

amdblis:
	1. updated with amdblis 2.2 release
amdfftw:
	1. "--enable-single" now work as synonym for "--enable-float"
amdlibflame:
	1. Added enable_or_disable_threads() to set value for "--enable-multithreading" flag
Libflame:
	1. Added enable_or_disable_threads() to set value for "--enable-multithreading" flag
	2. Corrected invocation of "enable_or_disable('threads')"

Change-Id: I9da0a2c2c4e2075b7fa2776e7cfe6548a2e0b32f

* Added amd-toolchain-support as maintainers

Added team github account amd-toolchain-support
as maintainers for all the recipes owned by
AMD Optimizing CPU Libraries (AOCL) team

Change-Id: I9a7969bd48fc42cfbb88dd7bd93e0802c6138582

* Incorporated review comments

Updated packages.yaml with aocl components
Handled Flake8 test failures

Change-Id: I0a03f02d8c9f326b2434ec907958c3de3a8e18eb

* Readded accidental removal of stream recipe

amdfftw:
	1. Updated the aocc clang selection as per spack standards
fftw:
	1. Currently apple-clang section is redundant,
	already it is handled in the conflict checks.

Change-Id: Idef4a3f61717eb81f321e0cd16e7ba9619eac846

* Fix for style and docs/validate (pull_request) test

unnumbered format placeholders from {} to {0}

Change-Id: If67a3374177ec067573e5504462d257712fafc05

* changed compiler references to Spack's compiler wrapper:spack_cc, spack_cxx, spack_fc

Change-Id: I7ae29c978fff16e37773913f14c84df232499763

* Removed 'single' variant from amdfftw recipe

Instead of conflict for apple-clang + openmp, handled this senario
via below available feature:
depends_on('llvm-openmp', when='%apple-clang +openmp')

Change-Id: I701b23d83e822a500ca3aaf2b60cc9ace09e13dc

* Added relevant info for users who prefers to use single precision

Change-Id: I3506e21da428ddef5fb7895b5aaed32c2a061ef6

* Minor changes on fftw, amdfftw and libflame

amdfftw:
	1. Removed escape symbol to the single quotes
	2. Rewording the conflict line from Recommended
	to Required

fftw:
	1. Reorded to following recommended sections:
	versions, variants, dependencies, providers,
	patches

libflame:
	1. Added provides entry for 5.1.0 version

Change-Id: I21ebff99b6dfde031763154693ecb3f1fa47b476

* Removed single quote from amdfftw docstring to fix style failures

Change-Id: Ife939a5a2f5ccbc8879b730c7bebfe2fcfef9332
2020-10-31 11:57:17 -05:00
Chuck Atkins
aaa1afe9ec
mesa: Retire older autotools mesa as mesa18 and create current meson mesa (#19528) 2020-10-30 11:16:17 -04:00
GaneshPrasadMA
0253f0af29
Adding AOCC compiler to SPACK community (#19345)
* Adding AOCC compiler to SPACK community

The AOCC compiler system offers a high level of advanced optimizations, multi-threading and processor support that includes global optimization, vectorization, inter-procedural analyses, loop transformations, and code generation. AMD also provides highly optimized libraries, which extract the optimal performance from each x86 processor core when utilized. The AOCC Compiler Suite simplifies and accelerates development and tuning for x86 applications.

* Added unit tests for detection and flags for AOCC

* Addressed reviewers comments w.r.t version checks and url,checksum related line lengths

Co-authored-by: Test User <spack@example.com>
2020-10-20 10:50:09 -05:00
Andrew Gaspar
ac7c3a1305
make py-pillow the default for pil (#19251)
py-pillow-simd is not portable to other architectures.
2020-10-10 12:55:07 -05:00
Andrew Gaspar
6aa9866b79
Add yacc provider and add dependency to swig (#19087)
* Add byacc dependency to swig when building an autoconf version

* Add yacc provider

Removed extra sycl provider default
2020-10-01 16:09:24 -04:00
Greg Becker
f616422fd7
refactor install_tree to use projections format (#18341)
* refactor install_tree to use projections format

* Add update method for config.yaml

* add test for config update config
2020-09-25 11:15:49 -05:00
Adam J. Stewart
098beee295
Pillow-SIMD: use as default PIL provider (#18097)
* Pillow-SIMD: use as default PIL provider

* Fix concretization of pil

* Fix build of older versions of pillow
2020-09-03 10:39:35 -07:00
Massimiliano Culpo
193e8333fa Update packages.yaml format and support configuration updates
The YAML config for paths and modules of external packages has
changed: the new format allows a single spec to load multiple
modules. Spack will automatically convert from the old format
when reading the configs (the updates do not add new essential
properties, so this change in Spack is backwards-compatible).

With this update, Spack cannot modify existing configs/environments
without updating them (e.g. “spack config add” will fail if the
configuration is in a format that predates this PR). The user is
prompted to do this explicitly and commands are provided. All
config scopes can be updated at once. Each environment must be
updated one at a time.
2020-08-10 11:59:05 -07:00
Chuck Atkins
547c71ad78
Revert "Add libglvnd packages/Add EGL support (#14572)" (#17682)
This reverts commit 573489db71.
2020-07-23 17:41:48 -04:00
Omar Padron
573489db71
Add libglvnd packages/Add EGL support (#14572)
* add new package: "libglvnd-frontend"

* add +glvnd variant to opengl package

* add +glvnd variant to mesa package

* add +egl variant to paraview package

* add libglvnd-frontend entries to default packages config

* fix style

* add default providers for glvnd virtuals

add default providers for glvnd-gl, glvnd-glx, and glvnd-egl

* WIP: rough start to external OpenGL documentation

* rename libglvnd-frontend package and backend virtual dependencies

* update documentation

* fix ligvnd-be-* typos

* fix libglvnd-fe package class name

* fix doc parse error
2020-07-13 11:32:36 -04:00
Peter Scheibel
ce9d30f80f
add public spack mirror (#17077) 2020-07-08 15:59:24 -07:00
Massimiliano Culpo
14599f09be
Separate Apple Clang from LLVM Clang (#17110)
* Separate Apple Clang from LLVM Clang

Apple Clang is a compiler of its own. All places
referring to "-apple" suffix have been updated.

* Hack to use a dash in 'apple-clang'

To be able to use autodoc from Sphinx we need
a valid Python name for the module that contains
Apple's Clang code.

* Updated packages to account for the existence of apple-clang

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* Added unit test for XCode related functions

Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-06-25 11:18:48 -05:00
Sergey Kosukhin
ff44a6d03f
hdf: new version, variants and refactoring (#16469)
* hdf: new version, variants and refactoring.

* libc provides rpc

* Fix szip-related configure argument.

* Update dependent packages.
2020-05-25 19:20:04 -05:00
Michael Kuhn
f12ae1bcd0
Fix typo for allow_sgid (#16806)
fixes #14425

The config: prefix should be included in the actual option name and
makes it impossible to change this option.
2020-05-25 18:49:35 +02:00
iarspider
e2d42672b7
Config option to disable setting S_ISGID bit when creating installation directory (#14479)
* Add config option to disable setting S_ISGID bit when creating installation directory. 

Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch>
2020-05-07 17:21:53 -07:00
Geoffrey Malcolm Oxberry
1f5ef5c876 darwin: cut DYLD_LIBRARY_PATH from default modules
This commit removes the DYLD_LIBRARY_PATH variable from the default
modules.yaml for darwin. The rationale behind deleting this
environment variable is that paths in this environment variable take
precedence over the default locations of libraries (usually the
install path of the library), which can lead to linking errors in some
circumstances. For example, executables intended to link with Apple's
system BLAS and LAPACK will instead link to a spack-installed
implementation (e.g., OpenBLAS), causing runtime errors.

These errors are resolved by instead relying on paths set in
DYLD_FALLBACK_LIBRARY_PATH, which is lower in precedence than default
locations of libraries.
2020-04-16 16:17:35 -07:00
iarspider
27e4a4750d
Hack to select iconv implementation - libiconv vs. libc iconv (#15437)
(re-do of #15213 due to changes in gnupg recipe)
2020-03-28 09:10:31 -05:00
Greg Becker
75fafcece9
multiprocessing: allow Spack to run uninterrupted in background (#14682)
Spack currently cannot run as a background process uninterrupted because some of the logging functions used in the install method (especially to create the dynamic verbosity toggle with the v key) cause the OS to issue a SIGTTOU to Spack when it's backgrounded.

This PR puts the necessary gatekeeping in place so that Spack doesn't do anything that will cause a signal to stop the process when operating as a background process.
2020-03-20 12:22:32 -07:00
Massimiliano Culpo
e3f97b37e6
Module files won't use CPATH by default, but language specific vars (#14749)
fixes #11555

Every path in CPATH is equivalent to a -I path to the compiler,
while every path in *_INCLUDE_PATH is equivalent to -isystem.
The latter avoids the noise due to warnings coming from 3rd party
libraries that a project depends on.

Added INCLUDE env variable (Intel Fortran, .mod files)
2020-03-17 08:36:56 +01:00
Michael Kuhn
7325c20794 config: Add a new option connect_timeout
connect_timeout can be used to increase the time Spack waits for the
server to answer. This can be used to work around slow connections or
servers.

Fixes #14700
2020-02-27 10:52:52 -08:00
Tamara Dahlgren
f2aca86502
Distributed builds (#13100)
Fixes #9394
Closes #13217.

## Background
Spack provides the ability to enable/disable parallel builds through two options: package `parallel` and configuration `build_jobs`.  This PR changes the algorithm to allow multiple, simultaneous processes to coordinate the installation of the same spec (and specs with overlapping dependencies.).

The `parallel` (boolean) property sets the default for its package though the value can be overridden in the `install` method.

Spack's current parallel builds are limited to build tools supporting `jobs` arguments (e.g., `Makefiles`).  The number of jobs actually used is calculated as`min(config:build_jobs, # cores, 16)`, which can be overridden in the package or on the command line (i.e., `spack install -j <# jobs>`).

This PR adds support for distributed (single- and multi-node) parallel builds.  The goals of this work include improving the efficiency of installing packages with many dependencies and reducing the repetition associated with concurrent installations of (dependency) packages.

## Approach
### File System Locks
Coordination between concurrent installs of overlapping packages to a Spack instance is accomplished through bottom-up dependency DAG processing and file system locks.  The runs can be a combination of interactive and batch processes affecting the same file system.  Exclusive prefix locks are required to install a package while shared prefix locks are required to check if the package is installed.

Failures are communicated through a separate exclusive prefix failure lock, for concurrent processes, combined with a persistent store, for separate, related build processes.  The resulting file contains the failing spec to facilitate manual debugging.

### Priority Queue
Management of dependency builds changed from reliance on recursion to use of a priority queue where the priority of a spec is based on the number of its remaining uninstalled dependencies.  

Using a queue required a change to dependency build exception handling with the most visible issue being that the `install` method *must* install something in the prefix.  Consequently, packages can no longer get away with an install method consisting of `pass`, for example.

## Caveats
- This still only parallelizes a single-rooted build.  Multi-rooted installs (e.g., for environments) are TBD in a future PR.

Tasks:
- [x] Adjust package lock timeout to correspond to value used in the demo
- [x] Adjust database lock timeout to reduce contention on startup of concurrent
    `spack install <spec>` calls
- [x] Replace (test) package's `install: pass` methods with file creation since post-install 
    `sanity_check_prefix` will otherwise error out with `Install failed .. Nothing was installed!`
- [x] Resolve remaining existing test failures
- [x] Respond to alalazo's initial feedback
- [x] Remove `bin/demo-locks.py`
- [x] Add new tests to address new coverage issues
- [x] Replace built-in package's `def install(..): pass` to "install" something
    (i.e., only `apple-libunwind`)
- [x] Increase code coverage
2020-02-19 00:04:22 -08:00
Federico Ficarelli
de84bd3f24
hipsycl: new package and new 'sycl' virtual package (#14051) 2020-02-13 07:32:39 +01:00
Greg Becker
c587c76537 Config option to allow gpg warning suppression (#13743)
Add a configuration option to suppress gpg warnings during binary
package verification. This only suppresses warnings: a gpg failure
will still fail the install. This allows users who have already
explicitly trusted the gpg key they are using to avoid seeing
repeated warnings that it is self-signed.
2019-11-14 16:22:01 -08:00
Massimiliano Culpo
b29eb4212e Users can configure use of RPATH or RUNPATH (#9168)
Add a new entry in `config.yaml`:

    config:
        shared_linking: 'rpath'

If this variable is set to `rpath` (the default) Spack will set RPATH in ELF binaries. If set to `runpath` it will set RUNPATH.

Details:
* Spack cc wrapper explicitly adds `--disable-new-dtags` when linking
* cc wrapper also strips `--enable-new-dtags` from the compile line
    when disabling (and vice versa)
* We specifically do *not* add any dtags flags on macOS, which uses
    Mach-O binaries, not ELF, so there's no RUNPATH)
2019-10-23 13:22:24 -07:00
Michael Kuhn
bdb92f2d91 Consistently support pkg-config files in share subdirectory (#12838)
While the build environment already takes share/pkgconfig into account,
the generated module files etc. only consider lib/pkgconfig and
lib64/pkgconfig.
2019-10-05 21:03:35 -05:00
Massimiliano Culpo
76b9c56110 Remove support for generating dotkit files (#11986)
Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.

A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.

* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator
2019-10-02 22:15:01 -07:00
Scott Wittenburg
5323a5cff9 Remove CombinatorialSpecSet in favor of environments + stacks 2019-09-13 22:57:15 -07:00
Tamara Dahlgren
c141e99e06 Use spack/user-specific stage root by default; stage cleaning (#12516)
* When cleaning the stage root, only remove directories that appear
  to be used for staging Spack packages. Previously Spack was clearing
  all directories in the stage root, which could remove content not
  related to Spack if the user chose a staging root which contains
  files/directories not managed by Spack.
* The documentation is updated with warnings about choosing a stage
  directory that is only managed by Spack (although generally the
  check added in this PR for "spack clean" should avoid removing
  content that was not created by Spack)
* The default stage directory (in config.yaml) is now
  $tempdir/$user/spack-stage and the logic is updated to omit the
  $user portion of this path if $tempdir already contains a $user
  directory.
* When creating stage root assign user read/write permissions to all
  directories in the path under $user. Previously Spack was assigning
  the permissions of the first existing parent directory
2019-09-03 16:31:27 -07:00
Tamara Dahlgren
0ea6e0f817
features: Remove stage symlinks (#12072)
Fixes #11163

The goal of this work is to simplify stage directory structures by eliminating use of symbolic links. This means, among other things, that` $spack/var/spack/stage` will no longer be the core staging directory. Instead, the first accessible `config:build_stage` path will be used.

Spack will no longer automatically append `spack-stage` (or the like) to configured build stage directories so the onus of distinguishing the directory from other work -- so the other work is not automatically removed with a `spack clean` operation -- falls on the user.
2019-08-19 10:31:24 -07:00
Elizabeth Fischer
fff99a8a3a Make OpenJDK the default (#12106) 2019-07-25 10:25:26 -07:00
Mark Olesen
5fd3b5acda openfoam: rename openfoam-com to openfoam (#11876)
The original packaging, with openfoam as a virtual and with
openfoam-com, openfoam-org and openfoam-extend as
providers, adds an obfuscation layer without any benefit.

- Rename `openfoam-com` to `openfoam`, and remove the 
  `openfoam` virtual package.
2019-07-01 22:54:36 -07:00
Geoffrey Oxberry
a52ed34fa4 ninja{-fortran}: make pkgs nonvirtual (fix #11628) (#11629) 2019-06-10 12:53:30 -04:00
Massimiliano Culpo
01ece824e1 Cap the maximum number of build jobs (#11373)
* config:build_jobs now controls the number of parallel jobs to spawn during
builds, but cannot ever exceed the number of cores on the machine.

* The default is set to 16 or the number of available cores, whatever
is lowest.

* Updated docs to reflect the changes done to limit parallel builds
2019-05-28 06:42:04 -07:00
t-karatsu
1e9bb8c512 Add Fujitsu compiler to Spack. (#11287)
* Add Fujitsu compiler to Spack.

* Fixes for flake8

* Chenges location of FCC to subdirectory called case-insensitive

* Add compiler tests for Fujitsu compiler

* Modify the logic of taking compiler version for new version of Fujitsu compiler
2019-05-21 20:31:14 -05:00
Chuck Atkins
251f3f5058
ninja: drop python as a run dep (#11347)
* ninja: Make python a build-only dependency

* ninja, ninja-fortran: allow the packages to provide eachother
2019-05-07 13:07:24 -04:00
Chuck Atkins
9f848205e4 mesa: re-written to new meson build system (#10482)
* mesa: re-writen to new meson build system

* xorg-server: Remove unneeded mesa dependency

* glx: provide the virtual GLX package

* Convert most "mesa" dependencies to use virtual gl and glx dependencies

* mesa: Switch to always building from the git repo instead of a tarball

* Use gl and glx version ranges instead of exact versions

* glx: provide a default package

* add auto to swr variant

* Revert "add auto to swr variant"

This reverts commit 2e7cd8a8aca07767c00657d0d09985ddd6306883.

* update mesa package to version 19.0

* set +glx by default only when running on linux

* fix style issues in geant4 package

* remove virtual package "egl"
2019-05-01 17:57:33 -05:00
Chris Green
763549fe7c mysql: Support client-only, cxxstd and more versions (#10911)
* add cxxstd variant
* add CMake constraints based on platform/version
* add older versions
* update boost dependency version constraints (which are closely
  tied to mysql version) and update boost cxxstd choice to be
  the same as the cxxstd chosen for mysql
* add client-only support (including a patch for 5.5.x)
* record the mysql package as a provider of the mysql-client virtual
2019-03-26 13:19:06 -05:00
Chris Green
6de6d5396c New package: mariadb-c-client (#7729)
* mariadb-c-client is a new package using the distinct, LGPL,
  MySQL-compatible client library from mariadb.com. It provides the
  virtual package mariadb-client
* mariadb is recorded as a provider of the mariadb-client virtual
* The mysql-client virtual package is also added, and mariadb-c-client
  is recorded as a provider for it
2019-03-15 14:12:30 -05:00
Mark W. Krentel
3ed6e78338 New package: ibm-java (#10777)
* Add binary package of the IBM Java SDK for big and little-endian
  powerpc (power7, 8 and 9).  The jdk and openjdk packages only install
  on x86_64.
* Add ibm-java as a java provider
* The jdk and openjdk packages only install on x86_64. Add conflicts
  for ppc64 and ppc64le to jdk and openjdk.
2019-03-12 18:05:47 -05:00
Greg Becker
f4d4322a41
Create option to build compilers as needed (#10761)
* Create option to build missing compilers and add them to config before installing packages that use them
* Clean up kwarg passing for do_install, put compiler bootstrapping in separate method
2019-03-07 17:30:48 -08:00
Scott Wittenburg
a6e8e889b2 release workflow: Add spack command for generating the .gitlab-ci.yml
This spack command adds a new schema for a file which describes the
builder containers available, along with the compilers availabe on
each builder.  The release-jobs command then generates the .gitlab-ci.yml
file by first expanding the release spec set, concretizing each spec
(in an appropriate docker container if --this-machine-only argument is
not provided on command line), and then combining and staging all the
concrete specs as jobs to be run by gitlab.
2019-02-21 15:37:35 -06:00
Elsa Gonsiorowski, PhD
74a5d92f07 Remove /nfs/tmp2 from default build_stage locations (#10170)
* Remove /nfs/tmp2 from default configuration
* /nfs/tmp2 is going away from LC... and doesn’t exist for the rest of the world.
* update documentation to remove /nfs/tmp2 as well
2018-12-21 02:03:54 -08:00
Adam J. Stewart
c227e1f02e Minor changes to Basic Settings docs for SC18 (#9809)
Grammar/spelling issues
2018-11-11 23:10:05 -06:00
Nichols A. Romero
236b34f3df Introduce fftw-api virtual package for Intel-MKL and FFTW (#9618)
* Introduce FFTW2 and FFT3 providers for Intel-MKL and FFTW Spack packages.

* make fftw default package for fftw-api virtual package

* virtual package test assertion now provides location of default virtual packages.

* Change name of virtual package to fftw-api and used versioned interface.
2018-11-09 08:16:11 -08:00
Michael Kuhn
7070d9fcd4 Allow setting language of compiler messages (#9486)
* Add a build_language config.yaml option which controls the language
  of compiler messages
* build_language defaults to "C", in which case the compiler messages
  will be in English. This allows Spack log parsing to detect and
  highlight error messages (since the regular expressions to find
  error messages are in English)
* The user can use the default language in their environment by setting
  the build_language config variable to null or ''
2018-10-18 12:55:58 -07:00
Greg Becker
d1a5113cfe permissions: add permission configuration to packages.yaml (#8773)
Spack can now be configured to assign permissions to the files installed by a package.

In the `packages.yaml` file under `permissions`, the attributes `read`, `write`, and `group` control the package permissions. These attributes can be set per-package, or for all packages under `all`. If permissions are set under `all` and for a specific package, the package-specific settings take precedence.  The `read` and `write` attributes take one of `user`, `group`, and `world`.

   packages:
    all:
      permissions:
        write: group
        group: spack
    my_app:
      permissions:
        read: group
        group: my_team
2018-10-11 14:29:07 -07:00
Greg Becker
7ece1a8676 Correct name for install_path_scheme in defaults config.yaml (#9482)
Default config.yaml mis-named `install_path_scheme` as `directory_layout`.
This commit corrects the error.
2018-10-10 18:54:58 -07:00
Peter Scheibel
28c0dd9148
Increase and customize lock timeouts (#9219)
Fixes #9166

This is intended to reduce errors related to lock timeouts by making
the following changes:

* Improves error reporting when acquiring a lock fails (addressing
  #9166) - there is no longer an attempt to release the lock if an
  acquire fails
* By default locks taken on individual packages no longer have a
  timeout. This allows multiple spack instances to install overlapping
  dependency DAGs. For debugging purposes, a timeout can be added by
  setting 'package_lock_timeout' in config.yaml
* Reduces the polling frequency when trying to acquire a lock, to
  reduce impact in the case where NFS is overtaxed. A simple
  adaptive strategy is implemented, which starts with a polling
  interval of .1 seconds and quickly increases to .5 seconds
  (originally it would poll up to 10^5 times per second).
  A test is added to check the polling interval generation logic.
* The timeout for Spack's whole-database lock (e.g. for managing
  information about installed packages) is increased from 60s to
  120s
* Users can configure the whole-database lock timeout using the
  'db_lock_timout' setting in config.yaml

Generally, Spack locks (those created using spack.llnl.util.lock.Lock)
now have no timeout by default

This does not address implementations of NFS that do not support file
locking, or detect cases where services that may be required
(nfslock/statd) aren't running.

Users may want to be able to more-aggressively release locks when
they know they are the only one using their Spack instance, and they
encounter lock errors after a crash (e.g. a remote terminal disconnect
mentioned in #8915).
2018-09-25 18:58:51 -07:00
Adam J. Stewart
7f419d34ac Set DYLD_LIBRARY_PATH on macOS (#9073) 2018-08-23 15:39:44 +02:00
Geoffrey Malcolm Oxberry
e380b08bab spack general packages.yaml defaults: add unwind
This commit adds 'libunwind' as the default 'unwind' provider in
spack. This preference should be overridden by the
darwin/packages.yaml file on macOS. Continues the changes discussed in
2018-08-06 22:01:50 -07:00
Geoffrey Malcolm Oxberry
ec0050a1bb darwin packages.yaml defaults: add unwind provider
This commit adds default unwind providers to the default packages.yaml
for darwin. Compiler versions are supplied with the apple-unwind
package so that apple-unwind is only used with Apple's clang fork, and
not with LLVM's clang.
2018-08-06 22:01:50 -07:00
Christoph Junghans
8bc3f7d726 Add ccache support (#3761)
If the user sets "ccache: true" in spack's config.yaml, Spack will use an available
ccache executable when compiling c/c++ code. This feature is disabled by default
(i.e. "ccache: false") and the documentation is updated with how to enable
ccache support
2018-07-09 14:06:10 -07:00
Todd Gamblin
c4c1d37fcd refactor: move templates from root to share/spack
- This complies with the unix directory hierarchy standard (which Spack
  attempts to follow)
- Also unclutters the repo root directory.
2018-06-24 16:38:36 -07:00
Todd Gamblin
54201e3c02
locks: add configuration and command-line options to enable/disable locks (#7692)
- spack.util.lock behaves the same as llnl.util.lock, but Lock._lock and
  Lock._unlock do nothing.

- can be disabled with a control variable.

- configuration options can enable/disable locking:
  - `locks` option in spack configuration controls whether Spack will use filesystem locks or not.
  - `-l` and `-L` command-line options can force-disable or force-enable locking.

- Spack will check for group- and world-writability before disabling
  locks, and it will not allow a group- or world-writable instance to
  have locks disabled.

- update documentation
2018-05-18 14:41:03 -07:00
Adam J. Stewart
8d681925b2
Change default elf provider on macOS (#7939) 2018-05-05 17:27:13 -05:00
Adam J. Stewart
5af4d4f0ac
Set default provider for every virtual package (#7784)
* Set default provider for every virtual package

* Add unit test to ensure that every virtual package has a default provider explicitly set
2018-05-05 17:26:47 -05:00
Veselin Dobrev
7f46ec49e7 Specify default providers for the glu virtual package. (#7733) 2018-04-12 13:30:49 -05:00
healther
9f07300982 fix outdated comments in config.yaml (#7594)
* `spack purge` now is `spack clean`
2018-03-25 08:49:55 -05:00
Mario Melara
2970a50ef6 Add the ability to separate architecture in path scheme (#4329)
* Add format to separate target and os for path

spec format can now handle separations of target and os for setting
up the path.

* Added ${PLATFORM} et al to spec.format()

${PLATFORM}, ${OS}, ${TARGET}

* Update tests

Updated tests and got rid of unnecessary code.

* Also update documentation to reflect this new ability.

* Add default path scheme to config.yaml

Added default path scheme to config.yaml. Users can overwrite this
section if they want.
2018-03-23 12:05:07 -07:00
Scott Wittenburg
d2c3441527 Update gl/qt options for vtk package (#6551)
This provides options for hardware and software rendering with the
gl API and updates the vtk package to make use of those options:

* Create new "gl" virtual package, provided by mesa
* Add external-only "opengl" package, which represents a system
  install that provides gl
* For vtk: prefer system gl implementation by default to get speed
  up from hardware rendering. When software rendering is specifically
  requested (+osmesa), try to use the llvmpipe approach, as it is
  much faster.
* Make qt dependency optional for vtk
* Add basic support for VTK on osx
* qt: Depend on virtual gl package, rather than mesa impl
2018-01-22 10:46:44 -08:00
Michael Kuhn
5bbbfe9446 Introduce virtual dependency pkgconfig (#5198)
There are two providers, pkgconf and pkg-config, with the former being
the default provider.
2017-11-23 08:05:38 -07:00
Massimiliano Culpo
b1d129e681 Modulefiles generated with a template engine (#3183)
* Module files now are generated using a template engine refers #2902 #3173

jinja2 has been hooked into Spack.

The python module `modules.py` has been splitted into several modules
under the python package `spack/modules`. Unit tests stressing module
file generation have been refactored accordingly.

The module file generator for Lmod has been extended to multi-providers
and deeper hierarchies.

* Improved the support for templates in module files.

Added an entry in `config.yaml` (`template_dirs`) to list all the
directories where Spack could find templates for `jinja2`.

Module file generators have a simple override mechanism to override
template selection ('modules.yaml' beats 'package.py' beats 'default').

* Added jinja2 and MarkupSafe to vendored packages.

* Spec.concretize() sets mutual spec-package references

The correct place to set the mutual references between spec and package
objects at the end of concretization. After a call to concretize we
should now be ensured that spec is the same object as spec.package.spec.

Code in `build_environment.py` that was performing the same operation
has been turned into an assertion to be defensive on the new behavior.

* Improved code and data layout for modules and related tests.

Common fixtures related to module file generation have been extracted
in `conftest.py`. All the mock configurations for module files have been
extracted from python code and have been put into their own yaml file.

Added a `context_property` decorator for the template engine, to make
it easy to define dictionaries out of properties.

The default for `verbose` in `modules.yaml` is now False instead of True.

* Extendable module file contexts + short description from docstring

The contexts that are used in conjunction with `jinja2` templates to
generate module files can now be extended from package.py and
modules.yaml.

Module files generators now infer the short description from package.py
docstring (and as you may expect it's the first paragraph)

* 'module refresh' regenerates all modules by default

`module refresh` without `--module-type` specified tries to
regenerate all known module types. The same holds true for `module rm`

Configure options used at build time are extracted and written into the
module files where possible.

* Fixed python3 compatibility, tests for Lmod and Tcl.

Added test for exceptional paths of execution when generating Lmod
module files.

Fixed a few compatibility issues with python3.

Fixed a bug in Tcl with naming_scheme and autoload + unit tests

* Updated module file tutorial docs. Fixed a few typos in docstrings.

The reference section for module files has been reorganized. The idea is
to have only three topics at the highest level:

  - shell support + spack load/unload use/unuse
  - module file generation (a.k.a. APIs + modules.yaml)
  - module file maintenance (spack module refresh/rm)

Module file generation will cover the entries in modules.yaml

Also:

  - Licenses have been updated to include NOTICE and extended to 2017
  - docstrings have been reformatted according to Google style

* Removed redundant arguments to RPackage and WafPackage.

All the callbacks in `RPackage` and `WafPackage` that are not build
phases have been modified not to accept a `spec` and a `prefix`
argument. This permits to leverage the common `configure_args` signature
to insert by default the configuration arguments into the generated
module files. I think it's preferable to handling those packages
differently than `AutotoolsPackage`. Besides only one package seems
to override one of these methods.

* Fixed broken indentation + improved resiliency of refresh

Fixed broken indentation in `spack module refresh` (probably a rebase
gone silently wrong?). Filter the writers for blacklisted specs before
searching for name clashes. An error with a single writer will not
stop regeneration, but instead will print a warning and continue
the command.
2017-09-19 12:34:20 -07:00
Sergey Kosukhin
a48b5a627c Make jpeg a virtual dependency. (#5190)
* Make jpeg a virtual dependency.

* Make 'libjpeg-turbo' the default implementation of 'jpeg'.
2017-08-24 10:30:35 +02:00
Adam J. Stewart
db657d938d Refactor IntelInstaller into IntelPackage base class (#4300)
* Refactor IntelInstaller into IntelPackage base class

* Move license attributes from __init__ to class-level

* Flake8 fixes: remove unused imports

* Fix logic that writes the silent.cfg file

* More specific version numbers for Intel MPI

* Rework logic that selects components to install

* Final changes necessary to get intel package working

* Various updates to intel-parallel-studio

* Add latest version of every Intel package

* Add environment variables for Intel packages

* Update env vars for intel package

* Finalize components for intel-parallel-studio package

Adds a +tbb variant to intel-parallel-studio.
The tbb package was renamed to intel-tbb.
Now both intel-tbb and intel-parallel-studio+tbb
provide tbb.

* Overhaul environment variables set by intel-parallel-studio

* Point dependent packages to the correct MPI wrappers

* Never default to intel-parallel-studio

* Gather env vars by sourcing setup scripts

* Use mpiicc instead of mpicc when using Intel compiler

* Undo change to ARCH

* Add changes from intel-mpi to intel-parallel-studio

* Add comment explaining mpicc vs mpiicc

* Prepend env vars containing 'PATH' or separators

* Flake8 fix

* Fix bugs in from_sourcing_file

* Indentation fix

* Prepend, not set if contains separator

* Fix license symlinking broken by changes to intel-parallel-studio

* Use comments instead of docstrings to document attributes

* Flake8 fixes

* Use a set instead of a list to prevent duplicate components

* Fix MKL and MPI library linking directories

* Remove +all variant from intel-parallel-studio

* It is not possible to build with MKL, GCC, and OpenMP at this time

* Found a workaround for locating GCC libraries

* Typos and variable names

* Fix initialization of empty LibraryList
2017-08-16 12:21:07 -05:00
snehring
db74f4c3a1 Add a new package for IcedTea (#4804)
* alsa-lib: new package

* cups: new package

* icedtea: new package

* icedtea: adding JAVA_HOME variable

* alsa: removing placeholder

* icedtea: typo fix

* alsa: fixing formattting

* icedtea: formatting fixes and cleanup

* icedtea: additional cleanup

* jdk: adding provides('java') to jdk, setting jdk to default java.

* icedtea: adding jdk as a dependency

* java: changing depends_on('jdk') and similar entries to depends_on('java')

* icedtea: removing unused imports, trying to placate flake8.

* fastqc: fixing flake8 error

* bazel: fixing typo in java dependency

* jdk: changing provides java to a range

* icedtea: cleaning up. Splitting up checksums.

* icedtea: jdk dependency is build only
2017-08-05 14:53:09 -05:00
Sergey Kosukhin
f6c2adc0ad New package libaec: a BSD-licensed replacement for the szip library. (#4437)
* Added package for 'libaec'.

* New version for package 'libaec'.
2017-08-04 10:38:54 -05:00
Mark Olesen
1f2e56e1f3 refactor openfoam packages (#3669)
* Several improvements for the openfoam packages

--

Refactor openfoam packages by adding an OpenfoamArch class

Use separate configure, build, install phases.

Provide FOAM_PROJECT_DIR dependent env for openfoam packages
- easier way to locate

Eliminate intermediate installation directories
- unneeded clutter.
- makes it less than easy to find the etc/bashrc file

Add versioning for all openfoam patches
- no certainty which parts (if any) will be needed in future versions,
  especially if we strive to ensure that the upstream version builds
  well with spack to begin with.

Support build of develop branches
- helps track build regressions for future openfoam releases

STYLE: use common/ and assets/ to provide additional (build) resources ...

* - adjust OpenFOAM provider

Move openfoam-com up front since this is the one being used as a base
for the others
2017-06-21 11:35:31 -05:00
Adam J. Stewart
bd1beedaf5 Allow users to set parallel jobs in config.yaml (#3812)
* Allow users to set parallel jobs in config.yaml

* Undo change from endash to emdash

* Remove parallel config, rename jobs to build_jobs
2017-04-15 08:31:00 -07:00
Adam J. Stewart
c0cfaacbc8 Set default providers for all virtual dependencies (#3634)
* Set default providers for everything

* Add default OpenFOAM provider
2017-04-02 14:48:27 -05:00
Adam J. Stewart
4866b4592f Add missing dependencies to gawk (#3481)
* Add missing dependencies to gawk

* Fix make check

* Add mawk packages and awk virtual dependency

* Use mawk instead of gawk to build ncurses

* Rearrange order of patches and dependencies
2017-03-18 19:13:32 -05:00
Tom Scogland
e0f5b12475 add ACLOCAL_PATH updates for packages that register m4 macros (#2518) 2016-12-07 13:53:57 -08:00
Adam J. Stewart
4e6d535058 Set Clang as the default compiler on macOS (#2225)
* Set OS-specific default compilers

* Fix flake8 warnings
2016-11-04 17:32:25 -07:00
Todd Gamblin
da760a66de source_cache, misc_cache, verify_ssl, checksum, & dirty in config.yaml
- Added new preferences to config.yaml:
  - source_cache
  - misc_cache
  - verify_ssl
  - checksum
  - dirty
2016-10-30 23:55:00 -07:00
Todd Gamblin
22b14e0f23 Add module_roots to config.yaml 2016-10-30 23:55:00 -07:00
Todd Gamblin
0da639298c Move temp directory configuration to config.yaml
- Moved temp finding logic to spack.stage
- Updated stage tests
- Added tests for new path substaitution of $user, $spack, $tempdir
2016-10-30 23:55:00 -07:00
Todd Gamblin
9347f86939 Rename install.yaml -> config.yaml, install_area -> "store"
- Added a schema for config.yaml
- Moved install tree configuration to config.yaml
- Moved etc/spack/install.yaml to etc/spack/defaults/config.yaml
- renamed install_area to "store", to use a term in common with guix/nix.
  - in `config.yaml` file, it's called the `install_tree` to be more
    intuitive to users.
  - `install_tree` might've worked in the code, but `install_tree` is
    already a global function in the spack namespace, from
    llnl.util.filesystem.
2016-10-30 23:55:00 -07:00
Benedikt Hegner
c557e76565 refactor settings for install area and make them config options 2016-10-30 23:55:00 -07:00
Elizabeth Fischer
3352889d73 py-pil: Protect against building with Python3. (#1868)
* py-pil: Does not build with Python3.

* Set py-pillow to be the default pil provider

* Update package.py

* Change to comments requested by adamjstewart

* Remove version constraint from extends(), avoid a Spack bug.
2016-09-28 16:51:41 -04:00
Todd Gamblin
26480f14f9 Add "default" configuration scope.
- Default scope is versioned with spack and can be overridden by site
  or user config.

- Default scope provides sensible default concretization preferences
  for all of Spack.

- per-platform concretization scope can be added later (to force a
  particular MPI on, e.g., Cray systems)
2016-07-19 17:10:17 -07:00
Glenn Johnson
6d36ec3d76 Add share/man to prefix inspections.
The share/man path should be in prefix_inspections for MANPATH as
share/man is a common location for manual pages.
2016-07-11 13:35:22 -05:00
Ben Boeckel
973caa3a07 modules.yaml: use the right envvar for pkg-config
Fixes #1072.
2016-06-20 11:10:29 -04:00
Todd Gamblin
0216fa2a00 Simplify /etc/spack/modules.yaml formatting. 2016-05-11 09:05:24 -07:00
alalazo
f8f71b1c2c modules : prefix_inspections moved to modules.yaml 2016-05-10 13:37:03 +02:00
alalazo
7fc0b1ebd2 module file : reverted to sensible default 2016-04-05 13:33:11 +02:00
alalazo
d546d828d3 module file : added filtering based on environment variable name 2016-04-05 13:20:28 +02:00
alalazo
e4adc675e5 module files configuration : added default configuration 2016-03-24 10:04:02 +01:00
Todd Gamblin
cc349e9a32 Add default repos.yaml 2015-11-28 19:11:51 -08:00