Commit graph

4206 commits

Author SHA1 Message Date
iarspider
399ca3b671
Add qgraf (#19404)
* Add recipe for qgraf

* Revert "Add recipe for qgraf"

This reverts commit 76783f73867a32b4a96e980e31a433ed3c0037fd.

* Add qgraf

* Update package.py

Changes from review

* Changes from MR

* Fix for URLs containing @ symbol

Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-10-21 19:16:11 -05:00
Massimiliano Culpo
c696518efd
Skip malformed spec strings when searching for externals (#19438)
fixes #19266

fzf search method has also been updated

Co-authored-by: Tom Scogland <tom.scogland@gmail.com>
2020-10-21 21:35:02 +02:00
Omri Mor
1147220b9b
CMakePackage: added 'ipo' variant (#18374)
+ipo sets CMAKE_INTERPROCEDURAL_OPTIMIZATION=ON
The option is not supported for CMake < 3.9
2020-10-21 11:09:45 +02:00
Massimiliano Culpo
bed929e7a9
Make url_fetch test independent of locale settings (#19390) 2020-10-21 09:18:50 +02:00
Greg Becker
a5faf7d27a
Change 'any' to wildcard for variants (#19381) 2020-10-21 08:05:29 +02:00
Massimiliano Culpo
a51702e3a2
py-archspec: added new package at v0.1.1 (#19395) 2020-10-20 18:40:31 +02:00
GaneshPrasadMA
0253f0af29
Adding AOCC compiler to SPACK community (#19345)
* Adding AOCC compiler to SPACK community

The AOCC compiler system offers a high level of advanced optimizations, multi-threading and processor support that includes global optimization, vectorization, inter-procedural analyses, loop transformations, and code generation. AMD also provides highly optimized libraries, which extract the optimal performance from each x86 processor core when utilized. The AOCC Compiler Suite simplifies and accelerates development and tuning for x86 applications.

* Added unit tests for detection and flags for AOCC

* Addressed reviewers comments w.r.t version checks and url,checksum related line lengths

Co-authored-by: Test User <spack@example.com>
2020-10-20 10:50:09 -05:00
Greg Becker
bf47045302
Fix python scripts relying on external python in env (#19241) 2020-10-20 10:09:13 +02:00
Scott Wittenburg
f561d3845b
Fix "buildcache update-index --keys ..." when mirror is S3 (#19141) 2020-10-19 15:24:41 -06:00
Sergey Kosukhin
fd7bfb1a50
filter_compiler_wrappers: a fix for NAG (#17133) 2020-10-18 23:18:18 -05:00
elsagermann
4750d479a0
Add testing option to dev-build command (#17293)
* ADD: testing to dev-build command

* RM: mutally exclusive group for testing in parser

* FIX: test option to subparser and not testing

* ADD: spack-completion.bash

* RM: local devbuildcosmo cmd

* FIX: bad merge --drop-in -b --before options forgotten

* FIX: --test place in spack-completion.bash

* FIX: typo

* FIX: blank line removing

* FIX: trailing white space

Co-authored-by: Elsa Germann <egermann@tsa-ln002.cm.cluster>
2020-10-18 23:17:07 -05:00
Wouter Deconinck
9471e9cb03
[docs] Pkg list: current version, not latest release (#18213)
The package list at https://spack.readthedocs.io/en/latest/package_list.html claims "it is automatically generated based on the packages in the latest Spack release" but it is actually based on the develop branch. This leads to confusion when users find that e.g. herwigpp is included in the list, but it cannot be found when they install the latest release. That latest release has a package list at https://spack.readthedocs.io/en/stable/package_list.html which does indeed not include herwigpp.

Changing the language from "the latest Spack release" to "this Spack version" might make that clearer. Maybe.
2020-10-18 23:11:20 -05:00
Jason Miller
25f817b8ae
Fix for buildcache -o (#19354)
* Fix for buildcache -o

The method has more positional arguments than the caller expects.

* Address length issue.

Fix pylint/flake errors.
2020-10-18 19:40:28 -05:00
Omri Mor
bff0291dac
Spec: fix multiple generator iteration in satisfies_dependencies (#18527) (#18559) 2020-10-17 11:40:31 +02:00
Scott McMillan
a612be1c98
New compiler: nvhpc (NVIDIA HPC SDK) (#19294)
* Add nvhpc compiler definition: "spack compiler add" will now look
  for instances of the NVIDIA HPC SDK compiler executables
  (nvc, nvc++, nvfortran) in supplied paths
* Add the nvhpc package which installs the nvhpc compiler
* Add testing for nvhpc detection and C++-standard/pic flags

Co-authored-by: Scott McMillan <smcmillan@nvidia.com>
2020-10-16 14:04:27 -07:00
iarspider
8c7385096e
spack external find: fix debug output (#19342)
Output was, e.g. `Executables in /bin and /,u,s,r,/,b,i,n are both associated with the same spec xz@5.2.2`, will be `Executables in /bin and /usr/bin are both associated with the same spec xz@5.2.2`.
2020-10-16 16:46:41 +02:00
Toyohisa Kameyama
a481087695
autotools: recursively patch config.guess and config.sub (#18347)
Previously config.guess and config.sub were patched only
in the root of the source path. 

This modification extend the previous behavior to patch every
config.guess or config.sub file even in subfolders, if need be.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-10-16 11:30:06 +02:00
Greg Becker
7a6268593c
Environments: specify packages for developer builds (#15256)
* allow environments to specify dev-build packages

* spack develop and spack undevelop commands

* never pull dev-build packges from bincache

* reinstall dev_specs when code has changed; reinstall dependents too

* preserve dev info paths and versions in concretization as special variant

* move install overwrite transaction into installer

* move dev-build argument handling to package.do_install

now that specs are dev-aware, package.do_install can add
necessary args (keep_stage=True, use_cache=False) to dev
builds. This simplifies driving logic in cmd and env._install

* allow 'any' as wildcard for variants

* spec: allow anonymous dependencies

raise an error when constraining by or normalizing an anonymous dep
refactor concretize_develop to remove dev_build variant
refactor tests to check for ^dev_path=any instead of +dev_build

* fix variant class hierarchy
2020-10-15 17:23:16 -07:00
Tomoki, Karatsu
72f431b67b
autotools.py: fix the list of objects to be removed from libtool (Fujitsu). (#19303) 2020-10-14 19:01:49 +02:00
Massimiliano Culpo
b84812256d
autotools: add attribute to delete libtool archives .la files (#18850)
* autotools: add attribute to delete libtool archives .la files

According to Autotools Mythbuster (https://autotools.io/libtool/lafiles.html)
libtool archive files are mostly vestigial, but they might create issues
when relocating binary packages as shown in #18694.

For GCC specifically, most distributions remove these files with
explicit commands:

https://git.stg.centos.org/rpms/gcc/blob/master/f/gcc.spec#_1303

Considered all of that, this commit adds an easy way for each
AutotoolsPackage to remove every .la file that has been installed.
The default, for the time being, is to maintain them - to be consistent
with what Spack was doing previously.

* autotools: delete libtool archive files by default

Following review this commit changes the default for
libtool archive files deletion and adds test to verify
the behavior.
2020-10-13 09:15:48 -07:00
Massimiliano Culpo
2399c2e78d
autotools: refactor search paths for aclocal in its own method (#19258)
This commit refactors the computation of the search path
for aclocal in its own method, so that it's easier to reuse
for packages that need to have a custom autoreconf phase.

Co-authored-by: Toyohisa Kameyama <kameyama@riken.jp>
2020-10-12 16:35:52 +02:00
Tomoki, Karatsu
db16f3e0d4
autotools.py: removed some options from libtool only for Fujitsu. (#19217) 2020-10-12 14:42:49 +02:00
Adam J. Stewart
372ac4a073
Add testing for Python 3.9 (#19261) 2020-10-11 21:16:00 -07:00
Adam J. Stewart
759e8ee4c2
CUDA: update maintainers (#19262) 2020-10-11 21:11:17 -07:00
谭九鼎
5e9f4dc982
Use https for links (#19244) 2020-10-09 11:24:09 -05:00
Scott Wittenburg
438f80d19e
Revert binary distribution cache manager (#19158)
This reverts #18359 and follow-on PRs intended to address issues with
#18359 because that PR changes the hash of all specs. A future PR will
reintroduce the changes.

* Revert "Fix location in spec.yaml where we look for full_hash (#19132)"
* Revert "Fix fetch of spec.yaml files from buildcache (#19101)"
* Revert "Merge pull request #18359 from scottwittenburg/add-binary-distribution-cache-manager"
2020-10-05 16:02:37 -07:00
Scott Wittenburg
7cfdf61979
Fix location in spec.yaml where we look for full_hash (#19132)
When we attempt to determine whether a remote spec (in a binary mirror)
is up-to-date or needs to be rebuilt, we compare the full_hash stored in
the remote spec.yaml file against the full_hash computed from the local
concrete spec.  Since the full_hash moved into the spec (and is no longer
at the top level of the spec.yaml), we need to look there for it.  This
oversight from #18359 was causing all specs to get rebuilt when the
full_hash wasn't fouhd at the expected location.
2020-10-02 15:37:47 -06:00
Scott Wittenburg
a44135dccf
Update buildcache key index when we update the package index (#19117)
This changes makes sure that when we run the pipeline job that updates
the buildcache package index on the remote mirror, we also update the
key index.  The public keys corresponding to the signing keys used to
sign the package was pushed to the mirror as a part of creating the
buildcache index, so this is just ensuring those keys are reflected
in the key index.

Also, this change makes sure the "spack buildcache update-index"
job runs even when there may have been pipeline failures, since we
would like the index always to reflect the true state of the mirror.
2020-10-02 11:00:42 -06:00
Scott Wittenburg
d3d98075c5
Fix fetch of spec.yaml files from buildcache (#19101)
Since those files currently exist in buildcaches (in S3 buckets) with
potentially different content types, we should be less restrictive in
what content types we accept when attempting to fetch them.  This PR
removes the content type constraint so any file with the matching
name will be found.
2020-10-01 14:32:43 -06:00
Patrick Gartung
a2795519df
Binary caching: avoid duplicate RPATHs, unnecessary updates (#19061)
* Remove duplication of reconstructed RPATHs caused by multiple
  identical entries in prefixes dictionary
* Don't rewrite RPATHs if relative RPATHs are unchanged because the
  directory layout is unchanged
2020-10-01 13:21:02 -07:00
Patrick Gartung
a380ceb139
Buildcache: Need to check the binary is not a Mach-o binary in a linux package or an ELF binary in a macOS package. (#18670)
* Need to check the binary is not a Mach-o binary in a linux package or an ELF binary in a macOS package.

* use sys.platform

* Darwin -> darwin for sys.platform
2020-10-01 12:39:59 -05:00
Scott Wittenburg
075c3e0d92
Merge pull request #18359 from scottwittenburg/add-binary-distribution-cache-manager
Add binary distribution cache manager
2020-09-30 16:37:35 -06:00
Axel Huebl
8edb31e934
CUDA: added v11.1.0 (#19036)
Compiler conflicts have been updated accordingly
2020-09-29 11:38:56 +02:00
David Beckingsale
04771ad9f8
Fixup conflicts for CUDA 11.0.2 and GCC (#19035)
* Fixup conflicts for CUDA 11.0.2 and GCC
* Updates for ppc64le
* Fix missing "or newer"

Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2020-09-28 18:01:20 -07:00
Tomoki, Karatsu
b76189a2e5
autotools: patch 'libtool' recursively in subdirectories (#18620)
Previous version was doing it only in the root build directory.
2020-09-28 10:53:56 +02:00
Adam J. Stewart
c8ac61979b
Fix usage of builtin file as variable name (#19018) 2020-09-28 07:14:39 +02:00
Omar Padron
2d93154119
Streamline key management for build caches (#17792)
* Rework spack.util.web.list_url()

list_url() now accepts an optional recursive argument (default: False)
for controlling whether to only return files within the prefix url or to
return all files whose path starts with the prefix url.  Allows for the
most effecient implementation for the given prefix url scheme.  For
example, only recursive queries are supported for S3 prefixes, so the
returned list is trimmed down if recursive == False, but the native
search is returned as-is when recursive == True.  Suitable
implementations for each case are also used for file system URLs.

* Switch to using an explicit index for public keys

Switches to maintaining a build cache's keys under build_cache/_pgp.
Within this directory is an index.json file listing all the available
keys and a <fingerprint>.pub file for each such key.

 - Adds spack.binary_distribution.generate_key_index()
   - (re)generates a build cache's key index

 - Modifies spack.binary_distribution.build_tarball()
   - if tarball is signed, automatically pushes the key used for signing
     along with the tarball
   - if regenerate_index == True, automatically (re)generates the build
     cache's key index along with the build cache's package index; as in
     spack.binary_distribution.generate_key_index()

 - Modifies spack.binary_distribution.get_keys()
   - a build cache's key index is now used instead of programmatic
     listing

 - Adds spack.binary_distribution.push_keys()
   - publishes keys from Spack's keyring to a given list of mirrors

 - Adds new spack subcommand: spack gpg publish
   - publishes keys from Spack's keyring to a given list of mirrors

 - Modifies spack.util.gpg.Gpg.signing_keys()
   - Accepts optional positional arguments for filtering the set of keys
     returned

 - Adds spack.util.gpg.Gpg.public_keys()
   - As spack.util.gpg.Gpg.signing_keys(), except public keys are
     returned

 - Modifies spack.util.gpg.Gpg.export_keys()
   - Fixes an issue where GnuPG would prompt for user input if trying to
     overwrite an existing file

 - Modifies spack.util.gpg.Gpg.untrust()
   - Fixes an issue where GnuPG would fail for input that were not key
     fingerprints

 - Modifies spack.util.web.url_exists()
   - Fixes an issue where url_exists() would throw instead of returning
     False

* rework gpg module/fix error with very long GNUPGHOME dir

* add a shim for functools.cached_property

* handle permission denied error in gpg util

* fix tests/make gpgconf optional if no socket dir is available
2020-09-25 12:54:24 -04:00
Greg Becker
f616422fd7
refactor install_tree to use projections format (#18341)
* refactor install_tree to use projections format

* Add update method for config.yaml

* add test for config update config
2020-09-25 11:15:49 -05:00
Greg Becker
5565b6494d
typo (#18845) 2020-09-21 11:54:23 -05:00
Massimiliano Culpo
fcb4dfc307
Ensure variant defaults are parsable from CLI. (#18661)
- Add a unit test to check if there are unparsable defaults
- Fix 'rust' and 'nsimd' variants
2020-09-19 07:54:26 +02:00
Greg Becker
7585b37865
do out of source builds in hashed directories (#18574) 2020-09-18 12:21:13 -07:00
Greg Becker
2e4892c111
env view failures: print underlying error message (#18713) 2020-09-18 10:21:14 -07:00
Scott Wittenburg
28ef5b1204 Do not assume we sit in the directory where the env file lives. 2020-09-14 10:37:42 -06:00
Scott Wittenburg
031490f6aa Remove :<name> interpolation, add SPACK_VERSION variables
Also fix issues with documentation to reflect changes
2020-09-14 10:37:42 -06:00
Scott Wittenburg
bf90cdd6c7 Document pipeline keys which can be global but overridden
Update pipelines documentation to describe how 'tags', 'variables',
'image', 'before_script', 'script', and 'after_script' can be
supplied at the top level, to be used by any of the runner mappings,
and also overridden by any of the runner mappings.

Also show an example of capturing the custom spack SHA at pipeline
generation time, so all jobs are sure to run with the same version
of spack, as a means to illustrate the $env:VARIABLE_NAME syntax.
2020-09-14 10:37:42 -06:00
Scott Wittenburg
d9e0718c9d Allow overridable global runner attributes 2020-09-14 10:37:42 -06:00
Scott Wittenburg
e686f1500e Update pipeline documentation to describe user-provided scripts 2020-09-14 10:37:42 -06:00
Scott Wittenburg
e18612a321 Add test for variable interpolation and scripts 2020-09-14 10:37:42 -06:00
Scott Wittenburg
2386f7582a Support variable interpolation at pipeline generation time 2020-09-14 10:37:42 -06:00
Scott Wittenburg
ace52bd476 Provide your own script, before_script, and after_script 2020-09-14 10:37:42 -06:00