Commit graph

4032 commits

Author SHA1 Message Date
Harmen Stoppels
84eae97f91
aspirin for buildaches (#17437) 2020-07-09 22:00:38 -05:00
Sajid Ali
12099ed55e
clear mpicc and friends before each build (#17450)
* clear mpi env vars
2020-07-09 16:14:49 -05:00
Greg Becker
d0f5b69a19
installation: skip repository metadata for externals (#16954)
When Spack installs a package, it stores repository package.py files
for it and all of its dependencies - any package with a Spack metadata
directory in its installation prefix.

It turns out this was too broad: this ends up including external
packages installed by Spack (e.g. installed by another Spack instance).
Currently Spack doesn't store the namespace properly for such packages,
so even though the package file could be fetched from the external,
Spack is unable to locate it.

This commit avoids the issue by skipping any attempt to locate and copy
from the package repository of externals, regardless of whether they
have a Spack repo directory.
2020-07-09 11:08:51 -07:00
Patrick Gartung
8c41173678
Buildcache: bindist test without invoking spack compiler wrappers. (#15687)
* Buildcache:
   * Try mocking an install of quux, corge and garply using prebuilt binaries
   * Put patchelf install after ccache restore
   * Add script to install patchelf from source so it can be used on Ubuntu:Trusty which does not have a patchelf pat package. The script will skip building on macOS
   * Remove mirror at end of bindist test
   * Add patchelf to Ubuntu build env
   * Revert mock patchelf package to allow other tests to run.
   * Remove depends_on('patchelf', type='build') relying instead on
   * Test fixture to ensure patchelf is available.

* Call g++ command to build libraries directly during test build

* Flake8

* Install patchelf in before_install stage using apt unless on Trusty where a build is done.

* Add some symbolic links between packages

* Flake8

* Flake8:

* Update mock packages to write their own source files

* Create the stage because spec search does not create it any longer

* updates after change of list command arguments

* cleanup after merge

* flake8
2020-07-08 15:05:58 -05:00
Adam J. Stewart
207e496162
spack create: ask how many to download (#17373) 2020-07-08 09:38:42 +02:00
Simon Byrne
f85da868ac
Improve Travis sample in the docs (#17420)
- printf is better than echo for multiline strings
- ** should be &&
- use line continuation
- Use multiline block
2020-07-08 07:25:37 +02:00
Patrick Gartung
7f8e827db8
Relocate rpaths for all binaries, then do text bin replacement if the rpaths still exist after running patchelf/otool (#17418) 2020-07-07 16:46:39 -05:00
Massimiliano Culpo
f4ac3770b4
CudaPackage: maintainers are listed in the docstring (#17409)
fixes #17396

This prevents the class attribute to be inherited and
saves current maintainers from becoming the default
maintainers of every Cuda package.
2020-07-07 20:45:41 +02:00
Todd Gamblin
b0506a722e
releases: document releases/latest tag (#17402)
We got rid of `master` after #17377, but users still want a way to get
the latest stable release without knowing its number.

We've added a `releases/latest` tag to replace what was once `master`.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-07-07 11:44:15 -07:00
Peter Scheibel
650ab563f4
Uninstall: tolerate hook failures when force=true (#16513)
Fixes #16478

This allows an uninstall to proceed even when encountering pre-uninstall
hook failures if the user chooses the --force option for the uninstall.

This also prevents post-uninstall hook failures from raising an exception,
which would terminate a sequence of uninstalls. This isn't likely essential
for #16478, but I think overall it will improve the user experience: if
the post-uninstall hook fails, there isn't much point in terminating a
sequence of spec uninstalls because at the point where the post-uninstall
hook is run, the spec has already been removed from the database (so it
will never have another chance to run).

Notes:

* When doing spack uninstall -a, certain pre/post-uninstall hooks aren't
  important to run, but this isn't easy to track with the current model.
  For example: if you are uninstalling a package and its extension, you
  do not have to do the activation check for the extension.
* This doesn't handle the uninstallation of specs that are not in the DB,
  so it may leave "dangling" specs in the installation prefix
2020-07-07 11:37:36 -07:00
Todd Gamblin
9ec9327f5a docs: document releases and branches in Spack
- [x] Remove references to `master` branch
- [x] Document how release branches are structured
- [x] Document how to make a major release
- [x] Document how to make a point release
- [x] Document how to do work in our release projects
2020-07-06 11:39:19 -07:00
Todd Gamblin
11088df402 Remove references to master from CI
- [x] remove master from github actions
- [x] remove master from .travis.yml
- [x] make `develop` the default branch for `spack ci`
2020-07-06 11:39:19 -07:00
Todd Gamblin
4ea76dc95c change master/child to controller/minion in pty docstrings
PTY support used the concept of 'master' and 'child' processes. 'master'
has been renamed to 'controller' and 'child' to 'minion'.
2020-07-06 11:39:19 -07:00
Michael Kuhn
516c3e659f
autotools bugfix: handle missing config.guess (#17356)
Spack was attempting to calculate abspath on the located config.guess
path even when it was not found (None); this commit skips the abspath
calculation when config.guess is not found.
2020-07-06 10:53:02 -07:00
Shahzeb Siddiqui
04f3000646
Pipelines doc: fixed two broken links (#17355) 2020-07-03 12:29:45 +02:00
Michael Kuhn
10016a34e0
autotools: Fix config.guess detection, take two (#17333)
The previous fix from #17149 contained a thinko that produced errors for
packages that overwrite configure_directory.
2020-07-02 00:45:41 -07:00
Massimiliano Culpo
a5eabfad91
Moved flake8, shell and documentation tests to Github Action (#17328)
* Move flake8 tests on Github Actions

* Move shell test to Github Actions

* Moved documentation build to Github Action

* Don't run coverage on Python 2.6

Since we get connection errors consistently on Travis
when trying to upload coverage results for Python 2.6,
avoid computing coverage entirely to speed-up tests.
2020-07-01 11:58:53 -05:00
Adam J. Stewart
6a77f1ff45
Fix hashlib function capitalization (#17323) 2020-07-01 09:46:20 -05:00
Glenn Johnson
60283775b3
Documentation update for container example (#17321)
This updates the documentation to reflect #17316.
2020-07-01 08:40:36 +02:00
Gregory Becker
49aebb44b1 bump version number 2020-06-30 18:21:32 -05:00
Massimiliano Culpo
486b4671b6
Don't detect "classic" on Cray to avoid a compiler bug (#17314)
* Don't detect "classic" on Cray to avoid a compiler bug

* add tests

Co-authored-by: Gregory Becker <becker33@llnl.gov>
2020-06-30 15:45:29 -05:00
Johannes Blaschke
1d55adfd2b
Add fish shell support (#9279)
* share/spack/setup-env.fish file to setup environment in fish shell

* setup-env.fish testing script

* Update share/spack/setup-env.fish

Co-Authored-By: Elsa Gonsiorowski, PhD <gonsie@me.com>

* Update share/spack/qa/setup-env-test.fish

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* updates completions using `spack commands --update-completion`

* added stderr-nocaret warning

* added fish shell tests to CI system


Co-authored-by: becker33 <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Elsa Gonsiorowski, PhD <gonsie@me.com>
2020-06-30 14:26:27 -05:00
Greg Becker
212299a021
Cray compiler: fix implicit rpaths for classic versions (#17310)
* check link dirs for existence
2020-06-30 12:46:20 -05:00
Greg Becker
b52390dacc
cray compilers: fix bug with verifying cray compilers (#17303)
* fix bug with verifying cray compilers
2020-06-29 20:26:46 -05:00
Greg Becker
d71fdc9719
remove three commands that have been deprecated since v0.13.0 (#17291)
* remove three commands that have been deprecated since v0.13.0
2020-06-29 11:15:56 -05:00
Michael Kuhn
b07d38b3be
autotools: Fix config.guess detection (#17149)
The config.guess detection used a relative path that did not work in
combination with `check_call`. Use an absolute path instead.
2020-06-29 10:08:22 -05:00
Massimiliano Culpo
789d060ff6
Detect FE compilers automatically on Cray (#17267)
* cray: detect frontend compilers automatically

This commit permits to detect frontend compilers
automatically, with the exception of cce.

Co-authored-by: Gregory Becker <becker33.llnl.gov>
2020-06-29 09:53:08 -05:00
George Hartzell
e6bb8360d0
Fix indentation in containerize example (#17228)
[george.hartzell@172-16-193-97 spack-explore-docker]$ spack containerize
Running `spack containerize` with the example `spack.yaml` file fails
with an error that ends like so:

```
[...]
  File "/local_scratch/hartzell/tmp/spack-explore-docker/lib/spack/external/ruamel/yaml/scanner.py", line 165, in need_more_tokens
    self.stale_possible_simple_keys()
  File "/local_scratch/hartzell/tmp/spack-explore-docker/lib/spack/external/ruamel/yaml/scanner.py", line 309, in stale_possible_simple_keys
    "could not find expected ':'", self.get_mark())
ruamel.yaml.scanner.ScannerError: while scanning a simple key
  in "/local_scratch/hartzell/tmp/spack-explore-docker/spack.yaml", line 26, column 1
could not find expected ':'
  in "/local_scratch/hartzell/tmp/spack-explore-docker/spack.yaml", line 28, column 5
```

Indenting the block string fixes the problem for me.

CentOS 7,

```
$ spack --version
0.14.2-1529-ec58f28c2
```
2020-06-29 08:58:51 +02:00
Greg Becker
77351bd287
Restore compilers.yaml default directory (#17283) 2020-06-29 08:03:22 +02:00
Adam J. Stewart
edf776aeb9
Add support for macOS Big Sur (#17236)
* Add support for macOS Big Sur

* Big Sur is actually macOS 11.0
2020-06-27 15:07:21 -05:00
Greg Becker
56b4abbe38
env: no automatic activation (#17258)
* env: no automatic activation

* Ensure ci rebuild jobs activate the environment (no longer automagic)

Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2020-06-26 17:20:15 -05:00
Scott Wittenburg
dfac09eade
Use json for buildcache index (#15002)
* Start moving toward a json buildcache index

* Add spec and database index schemas

* Add a schema for buildcache spec.yaml files

* Provide a mode for database class to generate buildcache index

* Update db and ci tests to validate object w/ new schema

* Remove unused temporary upload-s3 command

* Use database class to generate buildcache index

* Do not generate index with each buildcache creation

* Make buildcache index mode into a couple of constructor args to Database class

* Use keyword args for  _createtarball 

* Parse new json index when we get specs from buildcache

Now that only one index file per mirror needs to be fetched in
order to have all the concrete specs for binaries available on the
mirror, we can just fetch and refresh the cached specs every time
instead of needing to use the '-f' flag to force re-reading.
2020-06-26 17:05:56 -05:00
Greg Becker
c39983eb1a
build_environment: verify compiler executables exist are are accessible (#17260)
* build_environment: verify compiler executables exist and are accessible

* fix existing tests

* test compiler executable verification
2020-06-26 16:12:22 -05:00
Omar Padron
1ed035def6
fix sorting issue with ci workarounds (#17277) 2020-06-26 14:24:08 -05:00
cedricchevalier19
529c659bd0
First fix for SPACK_DEPENDENCIES problem when doing setup (#10715)
* First fix for SPACK_DEPENDENCIES problem when doing setup

* Get rid of transitive include path in setup.

* Export SPACK_INCLUDE_DIRS into spconfig.py
2020-06-26 14:06:43 -05:00
eugeneswalker
d8da1e6dbc
spack ci generate: allow use of --output-file <file> with wo/ preceding path (#17275) 2020-06-26 11:18:29 -07:00
Jeffrey Salmond
1602b7a561
add environment-awareness to buildcache create (#16580)
* add buildcache create test

* add functionality and test to create buildcache from environment

* use env.concretized_user_specs rather than env.roots to get concretized specs, as suggested in review from becker33
2020-06-26 10:01:12 -05:00
Greg Becker
ec108dbebd
Allow spack remove -f and spack uninstall to work on matrices (#17222)
* Allow `spack remove -f` and `spack uninstall` to work on matrices

Allow Environment.remove(force=True) to remove the concrete spec from the environment
even when the user spec cannot be removed because it is in a matrix.
2020-06-26 09:57:22 -05:00
Massimiliano Culpo
c401c63156
Allow detection for "-classic" version of CCE (#17256)
* cce: detect modules based on the classic backend

* cce: tweaked version checks for clang based options

* Added unit test for cce flags
2020-06-25 21:20:09 -05:00
Omar Padron
7c54aa2eb0
add workaround for gitlab ci needs limit (#17219)
* add workaround for gitlab ci needs limit

* fix style/address review comments

* convert filter obj to list

* update command completion

* remove dict comprehension

* add workaround tests

* fix sorting issue between disparate types

* add indeces to format
2020-06-25 14:27:20 -04:00
Greg Becker
096bd69a94
prevent multiple version sigils in the same spec (#17246)
* prevent multiple version sigils in the same spec

* fix packages with malformed versions
2020-06-25 12:34:09 -05:00
Greg Becker
f936e3a1db
environment spec lists: improve ability to query architecture in when clauses (#17056) 2020-06-25 19:13:26 +02:00
Massimiliano Culpo
14599f09be
Separate Apple Clang from LLVM Clang (#17110)
* Separate Apple Clang from LLVM Clang

Apple Clang is a compiler of its own. All places
referring to "-apple" suffix have been updated.

* Hack to use a dash in 'apple-clang'

To be able to use autodoc from Sphinx we need
a valid Python name for the module that contains
Apple's Clang code.

* Updated packages to account for the existence of apple-clang

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* Added unit test for XCode related functions

Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-06-25 11:18:48 -05:00
Peter Scheibel
a31c115d79
Fix global activation check for upstream extendees (#17231)
* short-circuit is_activated check when the extendee is installed upstream

* add test for checking activation status of packages with an extendee installed upstream
2020-06-25 08:27:27 -05:00
Greg Becker
b26e93af3d
spack config: new subcommands add/remove (#13920)
spack config add <value>: add nested value value to the configuration scope specified
spack config remove/rm: remove specified configuration from the relevant scope
2020-06-25 09:38:01 +02:00
Tamara Dahlgren
48d3e8d350
features: Add install failure tracking removal through spack clean (#15314)
* Add ability to force removal of install failure tracking data through spack clean

* Add clean failures option to packaging guide
2020-06-24 20:28:53 -05:00
Sergey Kosukhin
bc53bb9b7c
Unset environment variables that are most commonly used by Autotools packages. (#8623) 2020-06-24 10:39:04 -05:00
Tom Payerle
fd710fc93e
Some minor fixes to set_permissions() in file_permissions.py (#17020)
* Some minor fixes to set_permissions() in file_permissions.py

The set_permissions() routine claims to prevent users from creating
world writable suid binaries.  However, it seems to only be checking
for/preventing group writable suid binaries.

This patch modifies the routine to check for both world and group
writable suid binaries, and complain appropriately.

* permissions.py: Add test to check blocks world writable SUID files

The original test_chmod_rejects_group_writable_suid tested
that the set_permissions() function in
lib/spack/spack/util/file_permissions.py
would raise an exception if changed permission on a file with
both SUID and SGID plus sticky bits is chmod-ed to g+rwx and o+rwx.

I have modified so that more narrowly tests a file with SUID
(and no SGID or sticky bit) set is chmod-ed to g+w.

I have added a second test test_chmod_rejects_world_writable_suid
that checks that exception is raised if an SUID file is chmod-ed
to o+w

* file_permissions.py: Raise exception when try to make sgid file world writable

Updated set_permissions() in file_permissions.py to also raise
an exception if try to make an SGID file world writable.  And
added corresponding unit test as well.

* Remove debugging prints from permissions.py
2020-06-23 19:50:19 -05:00
Tamara Dahlgren
e74c8e71cc
tests: check rpath presence not equality (#17216) 2020-06-23 17:17:35 -05:00
Matthias Wolf
0b57567824
Module index should not be unconditionally overwritten (#14837)
* Module index should not be unconditionally overwritten

Uncovered after we switched our CI to generate modules for packages
one-by-one rather than in bulk. This overwrote a complete module index
with an index with a single entry, and broke our downstream Spack
instances that needed the upstream module index.
2020-06-23 15:38:04 -05:00