Commit graph

4022 commits

Author SHA1 Message Date
Todd Gamblin
9ec9327f5a docs: document releases and branches in Spack
- [x] Remove references to `master` branch
- [x] Document how release branches are structured
- [x] Document how to make a major release
- [x] Document how to make a point release
- [x] Document how to do work in our release projects
2020-07-06 11:39:19 -07:00
Todd Gamblin
11088df402 Remove references to master from CI
- [x] remove master from github actions
- [x] remove master from .travis.yml
- [x] make `develop` the default branch for `spack ci`
2020-07-06 11:39:19 -07:00
Todd Gamblin
4ea76dc95c change master/child to controller/minion in pty docstrings
PTY support used the concept of 'master' and 'child' processes. 'master'
has been renamed to 'controller' and 'child' to 'minion'.
2020-07-06 11:39:19 -07:00
Michael Kuhn
516c3e659f
autotools bugfix: handle missing config.guess (#17356)
Spack was attempting to calculate abspath on the located config.guess
path even when it was not found (None); this commit skips the abspath
calculation when config.guess is not found.
2020-07-06 10:53:02 -07:00
Shahzeb Siddiqui
04f3000646
Pipelines doc: fixed two broken links (#17355) 2020-07-03 12:29:45 +02:00
Michael Kuhn
10016a34e0
autotools: Fix config.guess detection, take two (#17333)
The previous fix from #17149 contained a thinko that produced errors for
packages that overwrite configure_directory.
2020-07-02 00:45:41 -07:00
Massimiliano Culpo
a5eabfad91
Moved flake8, shell and documentation tests to Github Action (#17328)
* Move flake8 tests on Github Actions

* Move shell test to Github Actions

* Moved documentation build to Github Action

* Don't run coverage on Python 2.6

Since we get connection errors consistently on Travis
when trying to upload coverage results for Python 2.6,
avoid computing coverage entirely to speed-up tests.
2020-07-01 11:58:53 -05:00
Adam J. Stewart
6a77f1ff45
Fix hashlib function capitalization (#17323) 2020-07-01 09:46:20 -05:00
Glenn Johnson
60283775b3
Documentation update for container example (#17321)
This updates the documentation to reflect #17316.
2020-07-01 08:40:36 +02:00
Gregory Becker
49aebb44b1 bump version number 2020-06-30 18:21:32 -05:00
Massimiliano Culpo
486b4671b6
Don't detect "classic" on Cray to avoid a compiler bug (#17314)
* Don't detect "classic" on Cray to avoid a compiler bug

* add tests

Co-authored-by: Gregory Becker <becker33@llnl.gov>
2020-06-30 15:45:29 -05:00
Johannes Blaschke
1d55adfd2b
Add fish shell support (#9279)
* share/spack/setup-env.fish file to setup environment in fish shell

* setup-env.fish testing script

* Update share/spack/setup-env.fish

Co-Authored-By: Elsa Gonsiorowski, PhD <gonsie@me.com>

* Update share/spack/qa/setup-env-test.fish

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* updates completions using `spack commands --update-completion`

* added stderr-nocaret warning

* added fish shell tests to CI system


Co-authored-by: becker33 <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Elsa Gonsiorowski, PhD <gonsie@me.com>
2020-06-30 14:26:27 -05:00
Greg Becker
212299a021
Cray compiler: fix implicit rpaths for classic versions (#17310)
* check link dirs for existence
2020-06-30 12:46:20 -05:00
Greg Becker
b52390dacc
cray compilers: fix bug with verifying cray compilers (#17303)
* fix bug with verifying cray compilers
2020-06-29 20:26:46 -05:00
Greg Becker
d71fdc9719
remove three commands that have been deprecated since v0.13.0 (#17291)
* remove three commands that have been deprecated since v0.13.0
2020-06-29 11:15:56 -05:00
Michael Kuhn
b07d38b3be
autotools: Fix config.guess detection (#17149)
The config.guess detection used a relative path that did not work in
combination with `check_call`. Use an absolute path instead.
2020-06-29 10:08:22 -05:00
Massimiliano Culpo
789d060ff6
Detect FE compilers automatically on Cray (#17267)
* cray: detect frontend compilers automatically

This commit permits to detect frontend compilers
automatically, with the exception of cce.

Co-authored-by: Gregory Becker <becker33.llnl.gov>
2020-06-29 09:53:08 -05:00
George Hartzell
e6bb8360d0
Fix indentation in containerize example (#17228)
[george.hartzell@172-16-193-97 spack-explore-docker]$ spack containerize
Running `spack containerize` with the example `spack.yaml` file fails
with an error that ends like so:

```
[...]
  File "/local_scratch/hartzell/tmp/spack-explore-docker/lib/spack/external/ruamel/yaml/scanner.py", line 165, in need_more_tokens
    self.stale_possible_simple_keys()
  File "/local_scratch/hartzell/tmp/spack-explore-docker/lib/spack/external/ruamel/yaml/scanner.py", line 309, in stale_possible_simple_keys
    "could not find expected ':'", self.get_mark())
ruamel.yaml.scanner.ScannerError: while scanning a simple key
  in "/local_scratch/hartzell/tmp/spack-explore-docker/spack.yaml", line 26, column 1
could not find expected ':'
  in "/local_scratch/hartzell/tmp/spack-explore-docker/spack.yaml", line 28, column 5
```

Indenting the block string fixes the problem for me.

CentOS 7,

```
$ spack --version
0.14.2-1529-ec58f28c2
```
2020-06-29 08:58:51 +02:00
Greg Becker
77351bd287
Restore compilers.yaml default directory (#17283) 2020-06-29 08:03:22 +02:00
Adam J. Stewart
edf776aeb9
Add support for macOS Big Sur (#17236)
* Add support for macOS Big Sur

* Big Sur is actually macOS 11.0
2020-06-27 15:07:21 -05:00
Greg Becker
56b4abbe38
env: no automatic activation (#17258)
* env: no automatic activation

* Ensure ci rebuild jobs activate the environment (no longer automagic)

Co-authored-by: Scott Wittenburg <scott.wittenburg@kitware.com>
2020-06-26 17:20:15 -05:00
Scott Wittenburg
dfac09eade
Use json for buildcache index (#15002)
* Start moving toward a json buildcache index

* Add spec and database index schemas

* Add a schema for buildcache spec.yaml files

* Provide a mode for database class to generate buildcache index

* Update db and ci tests to validate object w/ new schema

* Remove unused temporary upload-s3 command

* Use database class to generate buildcache index

* Do not generate index with each buildcache creation

* Make buildcache index mode into a couple of constructor args to Database class

* Use keyword args for  _createtarball 

* Parse new json index when we get specs from buildcache

Now that only one index file per mirror needs to be fetched in
order to have all the concrete specs for binaries available on the
mirror, we can just fetch and refresh the cached specs every time
instead of needing to use the '-f' flag to force re-reading.
2020-06-26 17:05:56 -05:00
Greg Becker
c39983eb1a
build_environment: verify compiler executables exist are are accessible (#17260)
* build_environment: verify compiler executables exist and are accessible

* fix existing tests

* test compiler executable verification
2020-06-26 16:12:22 -05:00
Omar Padron
1ed035def6
fix sorting issue with ci workarounds (#17277) 2020-06-26 14:24:08 -05:00
cedricchevalier19
529c659bd0
First fix for SPACK_DEPENDENCIES problem when doing setup (#10715)
* First fix for SPACK_DEPENDENCIES problem when doing setup

* Get rid of transitive include path in setup.

* Export SPACK_INCLUDE_DIRS into spconfig.py
2020-06-26 14:06:43 -05:00
eugeneswalker
d8da1e6dbc
spack ci generate: allow use of --output-file <file> with wo/ preceding path (#17275) 2020-06-26 11:18:29 -07:00
Jeffrey Salmond
1602b7a561
add environment-awareness to buildcache create (#16580)
* add buildcache create test

* add functionality and test to create buildcache from environment

* use env.concretized_user_specs rather than env.roots to get concretized specs, as suggested in review from becker33
2020-06-26 10:01:12 -05:00
Greg Becker
ec108dbebd
Allow spack remove -f and spack uninstall to work on matrices (#17222)
* Allow `spack remove -f` and `spack uninstall` to work on matrices

Allow Environment.remove(force=True) to remove the concrete spec from the environment
even when the user spec cannot be removed because it is in a matrix.
2020-06-26 09:57:22 -05:00
Massimiliano Culpo
c401c63156
Allow detection for "-classic" version of CCE (#17256)
* cce: detect modules based on the classic backend

* cce: tweaked version checks for clang based options

* Added unit test for cce flags
2020-06-25 21:20:09 -05:00
Omar Padron
7c54aa2eb0
add workaround for gitlab ci needs limit (#17219)
* add workaround for gitlab ci needs limit

* fix style/address review comments

* convert filter obj to list

* update command completion

* remove dict comprehension

* add workaround tests

* fix sorting issue between disparate types

* add indeces to format
2020-06-25 14:27:20 -04:00
Greg Becker
096bd69a94
prevent multiple version sigils in the same spec (#17246)
* prevent multiple version sigils in the same spec

* fix packages with malformed versions
2020-06-25 12:34:09 -05:00
Greg Becker
f936e3a1db
environment spec lists: improve ability to query architecture in when clauses (#17056) 2020-06-25 19:13:26 +02:00
Massimiliano Culpo
14599f09be
Separate Apple Clang from LLVM Clang (#17110)
* Separate Apple Clang from LLVM Clang

Apple Clang is a compiler of its own. All places
referring to "-apple" suffix have been updated.

* Hack to use a dash in 'apple-clang'

To be able to use autodoc from Sphinx we need
a valid Python name for the module that contains
Apple's Clang code.

* Updated packages to account for the existence of apple-clang

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* Added unit test for XCode related functions

Co-authored-by: Gregory Becker <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-06-25 11:18:48 -05:00
Peter Scheibel
a31c115d79
Fix global activation check for upstream extendees (#17231)
* short-circuit is_activated check when the extendee is installed upstream

* add test for checking activation status of packages with an extendee installed upstream
2020-06-25 08:27:27 -05:00
Greg Becker
b26e93af3d
spack config: new subcommands add/remove (#13920)
spack config add <value>: add nested value value to the configuration scope specified
spack config remove/rm: remove specified configuration from the relevant scope
2020-06-25 09:38:01 +02:00
Tamara Dahlgren
48d3e8d350
features: Add install failure tracking removal through spack clean (#15314)
* Add ability to force removal of install failure tracking data through spack clean

* Add clean failures option to packaging guide
2020-06-24 20:28:53 -05:00
Sergey Kosukhin
bc53bb9b7c
Unset environment variables that are most commonly used by Autotools packages. (#8623) 2020-06-24 10:39:04 -05:00
Tom Payerle
fd710fc93e
Some minor fixes to set_permissions() in file_permissions.py (#17020)
* Some minor fixes to set_permissions() in file_permissions.py

The set_permissions() routine claims to prevent users from creating
world writable suid binaries.  However, it seems to only be checking
for/preventing group writable suid binaries.

This patch modifies the routine to check for both world and group
writable suid binaries, and complain appropriately.

* permissions.py: Add test to check blocks world writable SUID files

The original test_chmod_rejects_group_writable_suid tested
that the set_permissions() function in
lib/spack/spack/util/file_permissions.py
would raise an exception if changed permission on a file with
both SUID and SGID plus sticky bits is chmod-ed to g+rwx and o+rwx.

I have modified so that more narrowly tests a file with SUID
(and no SGID or sticky bit) set is chmod-ed to g+w.

I have added a second test test_chmod_rejects_world_writable_suid
that checks that exception is raised if an SUID file is chmod-ed
to o+w

* file_permissions.py: Raise exception when try to make sgid file world writable

Updated set_permissions() in file_permissions.py to also raise
an exception if try to make an SGID file world writable.  And
added corresponding unit test as well.

* Remove debugging prints from permissions.py
2020-06-23 19:50:19 -05:00
Tamara Dahlgren
e74c8e71cc
tests: check rpath presence not equality (#17216) 2020-06-23 17:17:35 -05:00
Matthias Wolf
0b57567824
Module index should not be unconditionally overwritten (#14837)
* Module index should not be unconditionally overwritten

Uncovered after we switched our CI to generate modules for packages
one-by-one rather than in bulk. This overwrote a complete module index
with an index with a single entry, and broke our downstream Spack
instances that needed the upstream module index.
2020-06-23 15:38:04 -05:00
Tamara Dahlgren
96932d65a8 Added support for --fail-fast install option to terminate on first failure 2020-06-23 10:22:41 -07:00
Tamara Dahlgren
f54a8a77b4 Allow a single ctrl-c to terminate an install in progress 2020-06-23 10:22:41 -07:00
Greg Becker
9067378c24
fix compiler environment handling to reset environment after (#17204)
bugfix: fix compiler environment handling to reset environment after
2020-06-23 09:26:15 -07:00
Omar Padron
224dc95159
Pre ci optimization (#16372)
* add initial optimization script

* integrate optimization in spack ci

* make optimization opt-in

* fix import error

* flake8 fixes

* update command completion

* work around vermin errors

* fix sphynx errors
2020-06-22 13:19:47 -04:00
Joseph Ciurej
42f2c168d2
swig: fix older builds on newer platforms by updating config files (#16854)
* swig: revise 'autotools' automated 'config.*' update mechanism to support 'config.sub' and adapt 'swig' accordingly
2020-06-22 10:44:29 -05:00
Axel Huebl
2fd3ab3c9c
Fetching from urls: Error Message (#16434)
* Fetching from urls: Error Message

Fix the error message when fetching from consecutive `urls` of a
package version. Each fail should show the currently failing URL,
not the first url.

Example multi-problem run that occured in real life:
```
==> 5821: Installing util-macros
curl: (28) Connection timed out after 10000 milliseconds
curl: (16) Error in the HTTP2 framing layer
curl: (22) The requested URL returned error: 403 Forbidden
==> Fetching https://www.x.org/archive/individual/util/util-macros-1.19.1.tar.bz2
==> Failed to fetch file from URL: https://www.x.org/archive/individual/util/util-macros-1.19.1.tar.bz2
    Curl failed with error 28
==> Fetching https://mirrors.ircam.fr/pub/x.org/individual/util/util-macros-1.19.1.tar.bz2
==> Failed to fetch file from URL: https://www.x.org/archive/individual/util/util-macros-1.19.1.tar.bz2
    Curl failed with error 16
==> Fetching http://xorg.mirrors.pair.com/individual/util/util-macros-1.19.1.tar.bz2
==> Failed to fetch file from URL: https://www.x.org/archive/individual/util/util-macros-1.19.1.tar.bz2
    URL https://www.x.org/archive/individual/util/util-macros-1.19.1.tar.bz2 was not found!
==> Fetching from https://www.x.org/archive/individual/util/util-macros-1.19.1.tar.bz2 failed.
==> Error: FetchError: All fetchers failed for spack-stage-util-macros-1.19.1-se2a2e74oyusj2r4esgcb7pr3qhh45ef
```

- `urls[0]`: HTTP2 layer error
- `urls[1]`: timeout
- `urls[2]`: missing file on mirror

* x.org: two more mirrors

x.org mirrors are a bit tricky, since many are out-of-sync or off.
A good package to test with is `util-macros`, which had a "recent"
release.
2020-06-18 10:37:32 +02:00
Adam J. Stewart
1f85d6eceb
Docs: fix rst link to spack ci (#17147) 2020-06-18 09:44:03 +02:00
Greg Becker
5b59e883c1
cray module do not work without CRAY_LD_LIBRARY_PATH (#17031)
Co-authored-by: Gregory Becker <becker33.llnl.gov>
2020-06-17 15:53:30 -07:00
Axel Huebl
896a14af82
Doc: E4S Build Cache Link (#16986)
Link to the E4S build cache in the according section of the docs.
2020-06-16 15:53:34 -07:00
psakievich
dc31afe67d
Yaml spec parsing corner case (#17101)
* Catch yaml filename corner case in spec parser

* improved regex and remove redundant error checking

Co-authored-by: Gregory Becker <becker33@llnl.gov>
2020-06-16 13:27:33 -05:00