Commit graph

3746 commits

Author SHA1 Message Date
Todd Gamblin
b3a5f2e3c3
lock transactions: ensure that nested write transactions write
If a write transaction was nested inside a read transaction, it would not
write properly on release, e.g., in a sequence like this, inside our
`LockTransaction` class:

```
1  with spack.store.db.read_transaction():
2    with spack.store.db.write_transaction():
3      ...
4  with spack.store.db.read_transaction():
   ...
```

The WriteTransaction on line 2 had no way of knowing that its
`__exit__()` call was the last *write* in the nesting, and it would skip
calling its write function.

The `__exit__()` call of the `ReadTransaction` on line 1 wouldn't know
how to write, and the file would never be written.

The DB would be correct in memory, but the `ReadTransaction` on line 4
would re-read the whole DB assuming that other processes may have
modified it.  Since the DB was never written, we got stale data.

- [x] Make `Lock.release_write()` return `True` whenever we release the
      *last write* in a nest.
2019-12-23 23:18:44 -08:00
Todd Gamblin
98577e3af5
lock transactions: fix non-transactional writes
Lock transactions were actually writing *after* the lock was
released. The code was looking at the result of `release_write()` before
writing, then writing based on whether the lock was released.  This is
pretty obviously wrong.

- [x] Refactor `Lock` so that a release function can be passed to the
      `Lock` and called *only* when a lock is really released.

- [x] Refactor `LockTransaction` classes to use the release function
  instead of checking the return value of `release_read()` / `release_write()`
2019-12-23 23:18:44 -08:00
Todd Gamblin
a85b9070cb
performance: avoid repeated DB locking on view generation
`ViewDescriptor.regenerate()` checks repeatedly whether packages are
installed and also does a lot of DB queries.  Put a read transaction
around the whole thing to avoid repeatedly locking and unlocking the DB.
2019-12-23 23:18:44 -08:00
Todd Gamblin
91ea90c253
performance: speed up spack find in environments
`Environment.added_specs()` has a loop around calls to
`Package.installed()`, which can result in repeated DB queries.  Optimize
this with a read transaction in `Environment`.
2019-12-23 23:17:59 -08:00
Todd Gamblin
5bdba98837
performance: spack spec should use a read transacction with -I
`spack spec -I` queries the database for installation status and should
use a read transaction around calls to `Spec.tree()`.
2019-12-23 23:17:59 -08:00
Todd Gamblin
cbf8553406
concretization: improve performance by avoiding database locks
Checks for deprecated specs were repeatedly taking out read locks on the
database, which can be very slow.

- [x] put a read transaction around the deprecation check
2019-12-23 23:17:58 -08:00
Todd Gamblin
48befd67b5
performance: memoize spack.architecture.get_platform()
`get_platform()` is pretty expensive and can be called many times in a
spack invocation.

- [x] memoize `get_platform()`
2019-12-23 23:17:58 -08:00
Sajid Ali
37eac1a226
use sys.executable instead of python in _source_single_file (#14252) 2019-12-23 23:16:30 -08:00
Peter Josef Scheibel
639156130b
Patch fetching: remove unnecessary argument 2019-12-23 23:03:10 -08:00
Peter Josef Scheibel
587c650b88
Mirrors: skip attempts to fetch BundlePackages
BundlePackages use a noop fetch strategy. The mirror logic was assuming
that the fetcher had a resource to cach after performing a fetch. This adds
a special check to skip caching if the stage is associated with a
BundleFetchStrategy. Note that this should allow caching resources
associated with BundlePackages.
2019-12-23 23:03:10 -08:00
Peter Josef Scheibel
d71428622b
Mirrors: avoid re-downloading patches
When updating a mirror, Spack was re-retrieving all patches (since the
fetch logic for patches is separate). This updates the patch logic to
allow the mirror logic to avoid this.
2019-12-23 23:03:10 -08:00
Peter Josef Scheibel
a69b3c85b0
Mirrors: perform checksum of fetched sources
Since cache_mirror does the fetch itself, it also needs to do the
checksum itself if it wants to verify that the source stored in the
mirror is valid. Note that this isn't strictly required because fetching
(including from mirrors) always separately verifies the checksum.
2019-12-23 23:03:09 -08:00
Peter Josef Scheibel
98b498c671
Mirrors: fix cosmetic symlink targets
The targets for the cosmetic paths in mirrrors were being calculated
incorrectly as of fb3a3ba: the symlinks used relative paths as targets,
and the relative path was computed relative to the wrong directory.
2019-12-23 23:03:09 -08:00
Peter Josef Scheibel
64209dda97
Allow repeated invocations of 'mirror create'
When creating a cosmetic symlink for a resource in a mirror, remove
it if it already exists. The symlink is removed in case the logic to
create the symlink has changed.
2019-12-23 23:03:09 -08:00
Paul Ferrell
c15e55c668
mirror bug fixes: symlinks, duplicate patch names, and exception handling (#13789)
* Some packages (e.g. mpfr at the time of this patch) can have patches
  with the same name but different contents (which apply to different
  versions of the package). This appends part of the patch hash to the
  cache file name to avoid conflicts.
* Some exceptions which occur during fetching are not a subclass of
  SpackError and therefore do not have a 'message' attribute. This
  updates the logic for mirroring a single spec (add_single_spec)
  to produce an appropriate error message in that case (where before
  it failed with an AttributeError)
* In various circumstances, a mirror can contain the universal storage
  path but not a cosmetic symlink; in this case it would not generate
  a symlink. Now "spack mirror create" will create a symlink for any
  package that doesn't have one.
2019-12-23 23:03:03 -08:00
Todd Gamblin
d7f2a32887 performance: dont' read spec.yaml files twice in view regeneration
`ViewDescriptor.regenerate()` calls `get_all_specs()`, which reads
`spec.yaml` files, which is slow.  It's fine to do this once, but
`view.remove_specs()` *also* calls it immediately afterwards.

- [x] Pass the result of `get_all_specs()` as an optional parameter to
  `view.remove_specs()` to avoid reading `spec.yaml` files twice.
2019-12-23 18:36:56 -08:00
Todd Gamblin
78b84e4ade performance: don't recompute hashes when regenerating environments
`ViewDescriptor.regenerate()` was copying specs and stripping build
dependencies, which clears `_hash` and other cached fields on concrete
specs, which causes a bunch of YAML hashes to be recomputed.

- [x] Preserve the `_hash` and `_normal` fields on stripped specs, as
  these will be unchanged.
2019-12-23 18:36:56 -08:00
Todd Gamblin
9b90d7e801 performance: reduce system calls required for remove_dead_links
`os.path.exists()` will report False if the target of a symlink doesn't
exist, so we can avoid a costly call to realpath here.
2019-12-23 18:36:56 -08:00
Todd Gamblin
c83e365c59 performance: only regenerate env views once in spack install
`spack install` previously concretized, writes the entire environment
out, regenerated views, then wrote and regenerated views
again. Regenerating views is slow, so ensure that we only do that once.

- [x] add an option to env.write() to skip view regeneration

- [x] add a note on whether regenerate_views() shouldn't just be a
  separate operation -- not clear if we want to keep it as part of write
  to ensure consistency, or take it out to avoid performance issues.
2019-12-23 18:36:56 -08:00
Todd Gamblin
0fb3280011 performance: add read transactions for install_all() and install()
Environments need to read the DB a lot when installing all specs.

- [x] Put a read transaction around `install_all()` and `install()`
  to avoid repeated locking
2019-12-23 18:36:56 -08:00
Todd Gamblin
6c9467e8c6 lock transactions: avoid redundant reading in write transactions
Our `LockTransaction` class was reading overly aggressively.  In cases
like this:

```
1  with spack.store.db.read_transaction():
2    with spack.store.db.write_transaction():
3      ...
```

The `ReadTransaction` on line 1 would read in the DB, but the
WriteTransaction on line 2 would read in the DB *again*, even though we
had a read lock the whole time.  `WriteTransaction`s were only
considering nested writes to decide when to read, but they didn't know
when we already had a read lock.

- [x] `Lock.acquire_write()` return `False` in cases where we already had
       a read lock.
2019-12-23 18:36:56 -08:00
Todd Gamblin
bb517fdb84 lock transactions: ensure that nested write transactions write
If a write transaction was nested inside a read transaction, it would not
write properly on release, e.g., in a sequence like this, inside our
`LockTransaction` class:

```
1  with spack.store.db.read_transaction():
2    with spack.store.db.write_transaction():
3      ...
4  with spack.store.db.read_transaction():
   ...
```

The WriteTransaction on line 2 had no way of knowing that its
`__exit__()` call was the last *write* in the nesting, and it would skip
calling its write function.

The `__exit__()` call of the `ReadTransaction` on line 1 wouldn't know
how to write, and the file would never be written.

The DB would be correct in memory, but the `ReadTransaction` on line 4
would re-read the whole DB assuming that other processes may have
modified it.  Since the DB was never written, we got stale data.

- [x] Make `Lock.release_write()` return `True` whenever we release the
      *last write* in a nest.
2019-12-23 18:36:56 -08:00
Todd Gamblin
eb8fc4f3be lock transactions: fix non-transactional writes
Lock transactions were actually writing *after* the lock was
released. The code was looking at the result of `release_write()` before
writing, then writing based on whether the lock was released.  This is
pretty obviously wrong.

- [x] Refactor `Lock` so that a release function can be passed to the
      `Lock` and called *only* when a lock is really released.

- [x] Refactor `LockTransaction` classes to use the release function
  instead of checking the return value of `release_read()` / `release_write()`
2019-12-23 18:36:56 -08:00
Todd Gamblin
779ac9fe3e performance: avoid repeated DB locking on view generation
`ViewDescriptor.regenerate()` checks repeatedly whether packages are
installed and also does a lot of DB queries.  Put a read transaction
around the whole thing to avoid repeatedly locking and unlocking the DB.
2019-12-23 18:36:56 -08:00
Sajid Ali
96063f9168 use sys.executable instead of python in _source_single_file (#14252) 2019-12-21 00:02:28 -08:00
Massimiliano Culpo
80495d83ed microarchitectures: fix ppc flags for clang (#14196) 2019-12-20 14:40:54 -08:00
Massimiliano Culpo
497fddfcb9 Fetching from URLs falls back to mirrors if they exist (#13881)
Users can now list mirrors of the main url in packages.

- [x] Instead of just a single `url` attribute, users can provide a list (`urls`) in the package, and these will be tried by in order by the fetch strategy.

- [x] To handle one of the most common mirror cases, define a `GNUMirrorPackage` mixin to handle all the standard GNU mirrors.  GNU packages can set `gnu_mirror_path` to define the path within a mirror, and the mixin handles setting up all the requisite GNU mirror URLs.

- [x] update all GNU packages in `builtin` to use the `GNUMirrorPackage` mixin.
2019-12-20 14:32:18 -08:00
Chris Green
dc17d548c8 Add missing __init__.py under test, and correct bad file name from #13889. (#14228) 2019-12-19 17:27:53 -06:00
Todd Gamblin
af249d3cf6 package_sanity: add a test to enforce no nonexisting dependencies in builtin
We shouldn't allow packages to have missing dependencies in the mainline.

- [x] Add a test to enforce this.
2019-12-18 21:10:31 -08:00
Todd Gamblin
531f370e0d possible_dependencies() now reports missing dependencies
- Add an optional argument so that `possible_dependencies()` will report
  missing dependencies.
- Add a test to ensure it works.
- Ignore missing dependencies in `possible_dependencies()` by default.
2019-12-18 21:10:31 -08:00
Todd Gamblin
81b147cc0a package: add spack.package.possible_dependencies method
- this version allows getting possible dependencies of multiple packages
  or specs at once.

- New method handles calling `PackageBase.possible_dependencies` multiple
  times and passing `visited` dict around.
2019-12-18 21:10:31 -08:00
Todd Gamblin
a3799b2c7b performance: speed up spack find in environments
`Environment.added_specs()` has a loop around calls to
`Package.installed()`, which can result in repeated DB queries.  Optimize
this with a read transaction in `Environment`.
2019-12-18 16:07:28 -08:00
Todd Gamblin
0e9c8d236c performance: spack spec should use a read transacction with -I
`spack spec -I` queries the database for installation status and should
use a read transaction around calls to `Spec.tree()`.
2019-12-18 16:07:28 -08:00
Todd Gamblin
f73cdac731 concretization: improve performance by avoiding database locks
Checks for deprecated specs were repeatedly taking out read locks on the
database, which can be very slow.

- [x] put a read transaction around the deprecation check
2019-12-18 16:07:28 -08:00
Todd Gamblin
33335c9d0a performance: memoize spack.architecture.get_platform()
`get_platform()` is pretty expensive and can be called many times in a
spack invocation.

- [x] memoize `get_platform()`
2019-12-18 16:07:28 -08:00
Todd Gamblin
52ebc19b4e bugfix: don't fail if checking for "real" compiler version
doesn't understand a custom, user-defined compiler version.  However, if
the compiler's version check fails, you can't build anything with the
custom compiler.

- [x] Be more lenient: fall back to the custom compiler version and use
  it verbatim if the version check fails.
2019-12-18 11:37:59 -08:00
Todd Gamblin
4eb54b6358 bugfix: pgcc -V returns 2 on power machines
`pgcc -V` was failing on power machines because it returns 2 (despite
correctly printing version information).  On x86_64 machines the same
command returns 0 and doesn't cause an error.

- [x] Ignore return value of 2 for pgcc when doign a version check
2019-12-18 11:37:59 -08:00
Adam J. Stewart
18c2029fef Fix argparse rST parsing of help messages (#14014)
Thanks!
2019-12-17 10:23:22 -08:00
Massimiliano Culpo
5eca4f1470 microarchitectures: readable names for AArch64 vendors (#13825)
Vendors for ARM come out of `/proc/cpuinfo` as hex numbers instead of readable strings.

- Add support for associating vendor names with the hex numbers.
- Also move these mappings from Python code to `microarchitectures.json`
- Move darwin feature name mappings to `microarchitectures.json` as well
2019-12-17 00:47:50 -08:00
Greg Becker
9f1d728646 match bootstrapped compiler to architecture (#14059) 2019-12-14 16:42:46 -08:00
Peter Scheibel
60580f5871 package hash: gracefully handle @when with non-string args (#14153)
* when constructing package hash, default to including a method in the content hash if we can't determine whether it would be included by examining the AST
* add a test for updated content-hash calculations
* refactor content hash tests to eliminate repeated lines
2019-12-14 14:31:39 -08:00
Zack Galbreath
0f5724e908 Split out CDash options to a separate help document (#13704)
Prevent `spack help install` from getting too cluttered with CDash-specific documentation.
2019-12-13 10:15:22 -08:00
Peter Josef Scheibel
8c2305e867 Patch fetching: remove unnecessary argument 2019-12-13 08:38:50 +01:00
Peter Josef Scheibel
8199f22e7c Mirrors: skip attempts to fetch BundlePackages
BundlePackages use a noop fetch strategy. The mirror logic was assuming
that the fetcher had a resource to cach after performing a fetch. This adds
a special check to skip caching if the stage is associated with a
BundleFetchStrategy. Note that this should allow caching resources
associated with BundlePackages.
2019-12-13 08:38:50 +01:00
Peter Josef Scheibel
b62ba7609d Mirrors: avoid re-downloading patches
When updating a mirror, Spack was re-retrieving all patches (since the
fetch logic for patches is separate). This updates the patch logic to
allow the mirror logic to avoid this.
2019-12-13 08:38:50 +01:00
Peter Josef Scheibel
754dd6eb1f Mirrors: perform checksum of fetched sources
Since cache_mirror does the fetch itself, it also needs to do the
checksum itself if it wants to verify that the source stored in the
mirror is valid. Note that this isn't strictly required because fetching
(including from mirrors) always separately verifies the checksum.
2019-12-13 08:38:50 +01:00
Peter Josef Scheibel
b2cc50aa6a Mirrors: fix cosmetic symlink targets
The targets for the cosmetic paths in mirrrors were being calculated
incorrectly as of fb3a3ba: the symlinks used relative paths as targets,
and the relative path was computed relative to the wrong directory.
2019-12-13 08:38:50 +01:00
Peter Josef Scheibel
b64f458102 Allow repeated invocations of 'mirror create'
When creating a cosmetic symlink for a resource in a mirror, remove
it if it already exists. The symlink is removed in case the logic to
create the symlink has changed.
2019-12-13 08:38:50 +01:00
Greg Becker
917224cb3c pytest: add __init__ files for all test subdirs (#13889)
* pytest: add __init__ files for all test subdirs

* add licenses to empty files

* Fix Sphinx warning message about comment within docstring

* Further fixes to Sphinx docstring
2019-12-10 15:19:09 -06:00
Massimiliano Culpo
59d222c172 Better error message when setting unknown variants during concretization (#13128)
fixes #13124
2019-12-10 11:21:45 -08:00
Omar Padron
0592c58030
Follow up/11117 fixes and testing (#13607)
* fix docstring in generate_package_index() refering to "public" keys as "signing" keys

* use explicit kwargs in push_to_url()

* simplify url_util.parse() per tgamblin's suggestion

* replace standardize_header_names() with the much simpler get_header()

* add some basic tests

* update s3_fetch tests

* update S3 list code to strip leading slashes from prefix

* correct minor warning regression introduced in #11117

* add more tests

* flake8 fixes

* add capsys fixture to mirror_crud test

* add get_header() tests

* use get_header() in more places

* incorporate review comments
2019-12-09 17:23:33 -05:00
Greg Becker
da9a562182 environments: allow 'add' command to add virtuals (#13787)
This PR allows virtual packages to be added to the specs list using
the add command.

Virtual packages are already allowed in named lists in spack
environments/stacks, and they are already allowed in the specs list
when added using the yaml directly.
2019-12-09 12:23:03 -08:00
Andras Wacha
b33b8a3e29 Apply URLFetchStrategy to ftp:// and ftps:// url schemes (#13939)
* Apply URLFetchStrategy to ftp:// and ftps:// url schemes

* Corrected trailing whitespace error
2019-12-09 11:18:08 -06:00
George Hartzell
1d06949306 Tuneup docs re setting up sphinx for building docs (#14005)
I have, more than once, tried to install the list of things that need
to build the docs, only to discover that the list doesn't use Spack's
package names.  I'm tired of facepalming....

While I was there I touched up the prose about activating the new
Python packages; activating a python package doesn't add anything to
your PYTHONPATH, it links things into a directory that's *already* on
your PYTHONPATH.  Note that this all presupposes that you're using
that same python....
2019-12-08 16:22:25 -06:00
Axel Huebl
d705e96a63
Spec Header Dirs: Only first include/ (#13991)
* CUDA HeaderList: Unit Test

* Spec Header Dirs: Only first include/

Avoid matching recurringly nested include paths that usually
refer to internally shipped libraries in packages.
Example in CUDA Toolkit, shipping a libc++ fork internally
with libcu++ since 10.2.89:
`<prefix>/include/cuda/some/more/details/include/` or
`<prefix>/include/cuda/std/detail/libcxx/include`

regex: non-greedy first match of include

Co-Authored-By: Massimiliano Culpo <massimiliano.culpo@gmail.com>

* CUDA: Re-Enable 10.2.89 as Default
2019-12-06 23:47:03 -08:00
Tamara Dahlgren
842e9d9375 docs: Alphabetize create cmd template options (#13993)
* Bugfix: Display template options for create command.

* Alphabetize "spack create" template options for readability

* Revert template choices format; alphabetize list

* flake8 fix
2019-12-06 18:07:45 -06:00
Greg Becker
99c9365974 Bugfix: spack find -p fails in environment (#13972)
* force spack find -p to print abstract specs without prefixes
* hashes have the same issue; improve handling of find -L to match find -l
2019-12-06 12:29:57 -08:00
Tamara Dahlgren
b0f59484d3 docs: Minimal BundlePackage build system doc (#13992)
* Minimal BundlePackage build system doc

* Add link to new bundlepackage file

* Fixed link bug and added create command example
2019-12-06 07:26:59 -08:00
Scott Wittenburg
a288449f0b cdash: fix reporting issue with python 3 2019-12-05 11:37:03 -07:00
Peter Scheibel
e9ee9eaf50 patching: do strict version range checking (#13989)
* apply strict constraint checks for patches, otherwise Spack may incorrectly treat a version range constraint as satisfied when mixing x.y and x.y.z versions
* add mixed version checks to version comparison tests
2019-12-04 22:27:08 -08:00
Todd Gamblin
3fee65cb92
version bump: 0.13.2 2019-12-04 21:36:42 -08:00
Todd Gamblin
46b68263d8
version bump: 0.13.2 2019-12-04 21:20:09 -08:00
Peter Scheibel
03a5771b9d
Bugfix: allow missing modules if they are blacklisted (#13540)
`spack module loads` and `spack module find` previously failed if any upstream modules were missing.  This prevented it from being used with upstreams (or, really, any spack instance) that blacklisted modules.

This PR makes module finding is now more lenient (especially for blacklisted modules).

- `spack module find` now does not report an error if the spec is blacklisted
  - instead, it prints a single warning if any modules will be omitted from the loads file
  - It comments the missing modules out of the loads file so the user can see what's missing
  - Debug messages are also printed so users can check this with `spack -d...`

- also added tests for new functionality
2019-12-04 20:55:50 -08:00
Peter Scheibel
e5f04f9abc Bugfix: allow missing modules if they are blacklisted (#13540)
`spack module loads` and `spack module find` previously failed if any upstream modules were missing.  This prevented it from being used with upstreams (or, really, any spack instance) that blacklisted modules.

This PR makes module finding is now more lenient (especially for blacklisted modules).

- `spack module find` now does not report an error if the spec is blacklisted
  - instead, it prints a single warning if any modules will be omitted from the loads file
  - It comments the missing modules out of the loads file so the user can see what's missing
  - Debug messages are also printed so users can check this with `spack -d...`

- also added tests for new functionality
2019-12-04 19:17:40 -08:00
Adam J. Stewart
ba7ac53b75
Fix spack test when SPACK_ROOT isn't set (#13980) 2019-12-04 18:05:08 -06:00
Massimiliano Culpo
e9f027210f Fixed x86-64 optimization flags for clang (#13913)
* Fixed x86-64 optimization flags for clang
* Fixed expected results in unit tests

Before the flags used where the one for llc, the underlying compiler from LLVM IR to machine assembly. It turns out that the semantic of `-march`, `-mtune` and `-mcpu` changes from clang front-end to llc.

I found no definitive reference for the flags submitted in this PR, but I checked the assembly on a vectorizable function using Godbolt's web-site.
2019-12-04 09:11:34 -08:00
Massimiliano Culpo
a93a613668
Speedup environment activation (#13557)
* Add a transaction around repeated calls to `spec.prefix` in the activation process
* cache the computation of home in the python package to speed up setting deps
* ensure that module-scope variables are only set *once* per module
2019-12-02 14:05:35 -08:00
Massimiliano Culpo
bca59f8d83 Speedup environment activation (#13557)
* Add a transaction around repeated calls to `spec.prefix` in the activation process
* cache the computation of home in the python package to speed up setting deps
* ensure that module-scope variables are only set *once* per module
2019-12-02 14:05:02 -08:00
Adam J. Stewart
71eaffa46b Version bump: 1.13.1 (#13942) 2019-12-02 06:58:30 -08:00
Todd Gamblin
c36d9f297f
bugfix: mirror path works for unknown versions (#13626)
`mirror_archive_path` was failing to account for the case where the fetched version isn't known to Spack.

- [x] don't require the fetched version to be in `Package.versions`
- [x] add regression test for mirror paths when package does not have a version
2019-12-01 20:23:06 -08:00
Adam J. Stewart
b44df29316
environments: don't try to modify run-env if a spec is not installed (#13589)
Fixes #13529
Fixes #13509
2019-12-01 20:22:31 -08:00
Greg Becker
fba963f1d1
use semicolons instead of newlines in module/python command (#13904) 2019-12-01 20:21:34 -08:00
Daryl W. Grunau
9c5b583508
verify.py: os.path.exists exception handling (#13656) 2019-12-01 20:21:08 -08:00
Chris Green
c5da94eb58 Armor test script against shebang character limit if appropriate. (#12447) 2019-11-30 13:00:08 -06:00
Adam J. Stewart
ab98ef1765
Mass conversion of setup_(dependent_)?environment (#13924)
* Mass conversion of setup_(dependent_)?environment

* prefix -> self.prefix
2019-11-29 15:00:44 -06:00
Greg Becker
1450c30656 use semicolons instead of newlines in module/python command (#13904) 2019-11-26 17:09:24 -08:00
Adam J. Stewart
ad1ad83664
IntelPackage: setup_env -> setup_build_env (#13888) 2019-11-26 16:48:32 -06:00
Massimiliano Culpo
f6781d65f0 buildcache list: do not display duplicate specs (#13758)
fixes #13757
2019-11-26 13:24:15 -07:00
Massimiliano Culpo
77c206185b Remove warning from the "Custom Extensions" section of the docs (#13804)
Extensions have been available for a while and the overall design
seems solid enough to be feasible for extensions without losing
backward compatibility.
2019-11-26 09:30:49 -08:00
René Widera
9c10b80591 add package cuda@10.2.89 (#13819)
* add package cuda@10.2.89

add CUDA 10.2 support

* CudaPackage: Add 10.2 Conflicts

* CUDA: Modernize Run Environment

See #11115
2019-11-25 13:23:48 -06:00
Greg Becker
6c55a7c85f
cmd/install: remove unused install_status option (#13751)
* cmd/install: remove unused install_status option

* update bash completions for spack install
2019-11-22 11:17:37 -07:00
Massimiliano Culpo
0684a58d16 Fixed detection for cascadelake microarchitecture (#13820)
fixes #13803
2019-11-21 13:09:48 -07:00
Paul Ferrell
fb3a3ba95e mirror bug fixes: symlinks, duplicate patch names, and exception handling (#13789)
* Some packages (e.g. mpfr at the time of this patch) can have patches
  with the same name but different contents (which apply to different
  versions of the package). This appends part of the patch hash to the
  cache file name to avoid conflicts.
* Some exceptions which occur during fetching are not a subclass of
  SpackError and therefore do not have a 'message' attribute. This
  updates the logic for mirroring a single spec (add_single_spec)
  to produce an appropriate error message in that case (where before
  it failed with an AttributeError)
* In various circumstances, a mirror can contain the universal storage
  path but not a cosmetic symlink; in this case it would not generate
  a symlink. Now "spack mirror create" will create a symlink for any
  package that doesn't have one.
2019-11-20 17:00:44 -07:00
t-karatsu
513fe55fc3 Features/expand microarch for aarch64 (#13780)
* Add process to determine aarch64 microarchitecture

* add microarchitectures for thunderx2 and a64fx

* Add optimize flags for gcc on aarch64 family processors thunderx2 and a64fx.

* Add optimize flags for clang on aarch64 family processors thunderx2 and a64fx

* Add testing for thunderx2 and a64fx microarchitectures
2019-11-20 00:01:12 -07:00
Greg Becker
99dfff4475
binary distribution: relocate text files properly in relative binaries (#13578)
* Make relative binaries relocate text files properly
* rb strings aren't valid in python 2
* move perl to new interface for setup_environment family methods
2019-11-18 14:13:42 -07:00
Adam J. Stewart
eb22abc521 Document use of the maintainers field (#13748) 2019-11-17 15:30:08 -07:00
Greg Becker
aa83c483f7
Bugfix/config caching 13754 (#13759)
* remove reference to `spack.store` in method definition

Referencing `spack.store` in method definition will cache the `spack.config.config` singleton variable too early, before we have a chance to add command line and environment scopes.
2019-11-15 13:55:51 -08:00
Tim Fuller
3dbafb5ee7 Fix for issue #13754 (#13755)
* remove reference to `spack.store` in method definition

Referencing `spack.store` in method definition will cache the `spack.config.config` singleton variable too early, before we have a chance to add command line and environment scopes.
2019-11-15 13:32:58 -08:00
Adam J. Stewart
5510bba1fd
Document use of the maintainers field (#12270)
* Document use of the maintainers field

* Use fake GitHub usernames

* GitHub action is not automatic yet
2019-11-15 13:58:24 -06:00
Greg Becker
74e04b7e20 Config option to allow gpg warning suppression (#13744)
Add a configuration option to suppress gpg warnings during binary
package verification. This only suppresses warnings: a gpg failure
will still fail the install. This allows users who have already
explicitly trusted the gpg key they are using to avoid seeing
repeated warnings that it is self-signed.
2019-11-14 16:22:19 -08:00
Greg Becker
c587c76537 Config option to allow gpg warning suppression (#13743)
Add a configuration option to suppress gpg warnings during binary
package verification. This only suppresses warnings: a gpg failure
will still fail the install. This allows users who have already
explicitly trusted the gpg key they are using to avoid seeing
repeated warnings that it is self-signed.
2019-11-14 16:22:01 -08:00
Peter Scheibel
28163cb34f determine target relative to the link directory rather than the full link path (which includes the file name) (#13727) 2019-11-14 12:15:47 -08:00
Peter Scheibel
3f861e18b0 symlink relativization: determine target relative to the link directory (#13710)
when making a package relative, relocate links relative to link directory

rather than the full link path (which includes the file name) because `os.path.relpath` expects a directory.
2019-11-13 17:51:57 -08:00
Greg Becker
43b0356e14 Allow binary relocation of strings in relative binaries (#13725)
Binaries with relative RPATHS currently do not relocate strings
hard-coded in binaries

This PR extends the best-effort relocation of strings hard-coded
in binaries to those whose RPATHs have been relativized.
2019-11-13 16:03:35 -08:00
Greg Becker
b280034380 Allow binary relocation of strings in relative binaries (#13724)
Binaries with relative RPATHS currently do not relocate strings
hard-coded in binaries

This PR extends the best-effort relocation of strings hard-coded
in binaries to those whose RPATHs have been relativized.
2019-11-13 16:03:16 -08:00
William F Godoy
d6274e1a1d Docs update for deprecated spack sha256 (#13701)
* Docs update for deprecated `spack sha256`

* Added macOS shasum

* Update lib/spack/docs/packaging_guide.rst

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
2019-11-13 15:21:27 -06:00
Todd Gamblin
da386b560a docs: use nicer looking code blocks in docs (#13708)
- [x] Use higher contrast terminal output font
- [x] Use higher contrast code block background color than default
- [x] Use a noticeable prompt character

See also https://github.com/spack/spack-tutorial/pull/10.
2019-11-12 18:18:41 -08:00
Daryl W. Grunau
5a9389a528 verify.py: os.path.exists exception handling (#13656) 2019-11-11 17:47:47 -06:00
Adam J. Stewart
bd9907809e environments: don't try to modify run-env if a spec is not installed (#13589)
Fixes #13529
Fixes #13509
2019-11-11 13:02:36 +01:00
Adam J. Stewart
60070759a0
Improve spack checksum formatting (#13666)
* Improve spack checksum formatting

* Add colon

* Fix Python 2 bug
2019-11-10 16:47:56 -06:00
Todd Gamblin
c9c267518f
bugfix: mirror path works for unknown versions (#13626)
`mirror_archive_path` was failing to account for the case where the fetched version isn't known to Spack.

- [x] don't require the fetched version to be in `Package.versions`
- [x] add regression test for mirror paths when package does not have a version
2019-11-07 17:13:51 -07:00
Andreas Baumbach
7a0bed7893 add depends_on(python) to PythonPackage stub (#13595) 2019-11-05 13:18:17 -06:00
Todd Gamblin
a6ccb5b2fa
bugfix: uninstall should find concrete specs by DAG hash (#13598)
This fixes a regression introduced in #10792.  `spack uninstall` in an
environment would not match concrete query specs properly after the index
hash of enviroments changed.

- [x] Search by DAG hash for specs to remove instead of by build hash
2019-11-05 06:12:12 -07:00
Todd Gamblin
b77974e986
spack find now displays variants and other spec constraints (#13596)
If you do this in a spack environment:

    spack add hdf5+hl

hdf5+hl will be the root added to the `spack.yaml` file, and you should
really expect `hdf5+hl` to display as a root in the environment.

- [x] Add decoration to roots so that you can see the details about what
      is required to build.
- [x] Add a test.
2019-11-05 05:39:56 -07:00
Todd Gamblin
b4501c4586
spack find now displays variants and other spec constraints
If you do this in a spack environment:

    spack add hdf5+hl

hdf5+hl will be the root added to the `spack.yaml` file, and you should
really expect `hdf5+hl` to display as a root in the environment.

- [x] Add decoration to roots so that you can see the details about what
      is required to build.
- [x] Add a test.
2019-11-05 02:49:17 -07:00
Todd Gamblin
ecfe84def7
bugfix: uninstall should find concrete specs by DAG hash
This fixes a regression introduced in #10792.  `spack uninstall` in an
environment would not match concrete query specs properly after the index
hash of enviroments changed.

- [x] Search by DAG hash for specs to remove instead of by build hash
2019-11-05 02:47:27 -07:00
Greg Becker
fbb5cf944b
environments: make shell modifications partially unconditional (#13523)
* environments: make shell modifications partially unconditional

* flake

* missing module name

* add regression test

* flake
2019-11-04 23:35:23 -07:00
Greg Becker
385e41d70b binary distribution: relocate text files properly in relative binaries (#13578)
* Make relative binaries relocate text files properly
* rb strings aren't valid in python 2
* move perl to new interface for setup_environment family methods
2019-11-04 22:03:35 -07:00
Omar Padron
edf9548310
bugfix: fetch prefers to fetch local mirrors over remote resources (#13545)
- [x] insert at beginning of list so fetch grabs local mirrors before remote resources
- [x] update the S3FetchStrategy so that it throws a SpackError if the fetch fails.  
      Before, it was throwing URLError, which was not being caught in stage.py.
- [x] move error handling out of S3FetchStrategy and into web_util.read_from_url()
- [x] pass string instead of URLError to SpackWebError
2019-11-03 17:13:02 -08:00
Omar Padron
0784ec1a6d bugfix: fetch prefers to fetch local mirrors over remote resources (#13545)
- [x] insert at beginning of list so fetch grabs local mirrors before remote resources
- [x] update the S3FetchStrategy so that it throws a SpackError if the fetch fails.  
      Before, it was throwing URLError, which was not being caught in stage.py.
- [x] move error handling out of S3FetchStrategy and into web_util.read_from_url()
- [x] pass string instead of URLError to SpackWebError
2019-11-03 17:11:30 -08:00
Greg Becker
7cdb241f80
environments: only write when necessary (#13546)
This changes Spack environments so that the YAML file associated with the environment is *only* written when necessary (i.e., if it is changed *by spack*).  The lockfile is still written out as before.

There is a larger question here of which part of Spack should be responsible for setting defaults in config files, and how we can get rid of empty lists and data structures currently cluttering files like `compilers.yaml`.  But that probably requires a rework of the default-setting validator in `spack.config`, as well as the code that uses `spack.config`.  This will at least help for `spack.yaml`.
2019-11-03 15:47:19 -08:00
Greg Becker
aa7c229c8f environments: only write when necessary (#13546)
This changes Spack environments so that the YAML file associated with the environment is *only* written when necessary (i.e., if it is changed *by spack*).  The lockfile is still written out as before.

There is a larger question here of which part of Spack should be responsible for setting defaults in config files, and how we can get rid of empty lists and data structures currently cluttering files like `compilers.yaml`.  But that probably requires a rework of the default-setting validator in `spack.config`, as well as the code that uses `spack.config`.  This will at least help for `spack.yaml`.
2019-11-03 15:46:41 -08:00
Adam J. Stewart
a8228e1aec
spack create: http -> https (#13547) 2019-11-01 17:40:49 -05:00
Peter Scheibel
2e029fc2e5
Bugfix: respect order of mirrors in mirrors.yaml (#13544)
Commands like "spack mirror list" were displaying mirrors in a
different order than what was listed in the corresponding mirrors.yaml
file.

This restores commands to iterate over mirrors in the order that
they appear in the config file.
2019-11-01 14:02:08 -07:00
Greg Becker
24ec9f0ce3
environments: make shell modifications partially unconditional (#13523)
* environments: make shell modifications partially unconditional

* flake

* missing module name

* add regression test

* flake
2019-11-01 12:00:16 -05:00
Marc Mengel
2cea0633fa allow bootstrap buildcache install of patchelf (#13430)
* allow bootstrap buildcache install of patchelf

* file not path_name on one

* style

* add test for relocating patchelf

* blank lines..
2019-11-01 10:54:55 -05:00
Todd Gamblin
d670765b97
version bump: 0.13.1 2019-11-01 03:52:04 -07:00
Omar Padron
01a0d554f5
bugfix: spack.util.url.join() now handles absolute paths correctly (#13488)
* fix issue where spack.util.url.join() failed to correctly handle absolute path components
* add url util tests
2019-11-01 03:50:47 -07:00
Gregory Lee
2a9d6b9fbf
sbang: use utf-8 for encoding when patching (#13490)
This fixes a UnicodeDecodeError in the sbang patching function.
2019-11-01 03:50:38 -07:00
Massimiliano Culpo
eb286bb80f
Specs with quoted flags containing spaces are parsed correctly (#13521) 2019-11-01 03:50:26 -07:00
Massimiliano Culpo
1cc69e1ce0
targets: print a warning message before downgrading (#13513)
* Make package preferences a soft failure for targets, instead of a hard failure.
* Added unit tests for preferences expressed via packages.yaml
2019-11-01 03:50:17 -07:00
Adam J. Stewart
338a532e07
Travis CI: Test Python 3.8 (#13347)
* Travis CI: Test Python 3.8

* Fix use of deprecated cgi.escape method

* Fix version comparison

* Fix flake8 F811 change in Python 3.8

* Make flake8 happy

* Use Python 3.8 for all test categories
2019-11-01 03:50:02 -07:00
Greg Becker
30c9609c4e
Documentation: Database.query methods share docstrings (#13515)
Currently, query arguments in the Spack core are documented on the
Database._query method, where the functionality is defined.

For users of the spack python command, this makes the python builtin
method help less than ideally useful, as help(spack.store.db.query)
and help(spack.store.db.query_local) do not show relevant information.

This PR updates the doc attributes for the Database.query and
Database.query_local arguments to mirror everything after the first
line of the Database._query docstring.
2019-11-01 03:49:47 -07:00
Massimiliano Culpo
944d7b3d62
cuda: fix conflict statements for x86-64 targets (#13472)
* cuda: fix conflict statements for x86-64 targets

fixes #13462

This build system mixin was not updated after the support for specific
targets has been merged.

* Updated the version range of cuda that conflicts with gcc@8:

* Updated the version range of cuda that conflicts with gcc@8: for ppc64le

* Relaxed conflicts for version > 10.1

* Updated versions in conflicts

Co-Authored-By: Axel Huebl <axel.huebl@plasma.ninja>
2019-11-01 03:49:35 -07:00
Greg Becker
b727f922a4
cpu: fix clang flags for generic x86_64 (#13491)
* cpu: differentiate flags used for pristine LLVM vs. Apple's version
2019-11-01 03:49:16 -07:00
Peter Scheibel
835df4b2e4
syaml_int type should use int.__repr__ rather than str.__repr__ (#13487) 2019-11-01 03:49:00 -07:00
Omar Padron
0f816561db bugfix: spack.util.url.join() now handles absolute paths correctly (#13488)
* fix issue where spack.util.url.join() failed to correctly handle absolute path components
* add url util tests
2019-11-01 03:42:43 -07:00
Gregory Lee
3bdab6f686 sbang: use utf-8 for encoding when patching (#13490)
This fixes a UnicodeDecodeError in the sbang patching function.
2019-11-01 03:39:45 -07:00
Massimiliano Culpo
390ffb80e7 Specs with quoted flags containing spaces are parsed correctly (#13521) 2019-11-01 03:38:11 -07:00
Massimiliano Culpo
42b8355269 targets: print a warning message before downgrading (#13513)
* Make package preferences a soft failure for targets, instead of a hard failure.
* Added unit tests for preferences expressed via packages.yaml
2019-11-01 03:36:28 -07:00
Peter Scheibel
6ec39b6c81
Bugfix: BundlePackage staging/installation (#13524)
4af4487 added a mirror_id function to most FetchStrategy
implementations that is used to calculate resource locations in
mirrors. It left out BundleFetchStrategy which broke all packages
making use of BundlePackage (e.g. xsdk). This adds a noop
implementation of mirror_id to BundleFetchStrategy so that the
download/installation of BundlePackages can proceed as normal.
2019-10-31 12:56:38 -07:00
Adam J. Stewart
4b56933643
Docs: unlock requirement versions (#13384) 2019-10-31 14:21:05 -05:00
Adam J. Stewart
536486f0e5
Travis CI: Test Python 3.8 (#13347)
* Travis CI: Test Python 3.8

* Fix use of deprecated cgi.escape method

* Fix version comparison

* Fix flake8 F811 change in Python 3.8

* Make flake8 happy

* Use Python 3.8 for all test categories
2019-10-31 14:20:46 -05:00
Greg Becker
a97faeb3c7 Documentation: Database.query methods share docstrings (#13515)
Currently, query arguments in the Spack core are documented on the
Database._query method, where the functionality is defined.

For users of the spack python command, this makes the python builtin
method help less than ideally useful, as help(spack.store.db.query)
and help(spack.store.db.query_local) do not show relevant information.

This PR updates the doc attributes for the Database.query and
Database.query_local arguments to mirror everything after the first
line of the Database._query docstring.
2019-10-30 16:47:48 -07:00
Greg Becker
230c6aa326
cpu: fix clang flags for generic x86_64 (#13491)
* cpu: differentiate flags used for pristine LLVM vs. Apple's version
2019-10-30 17:16:13 -05:00
Massimiliano Culpo
cf1c38162e cuda: fix conflict statements for x86-64 targets (#13472)
* cuda: fix conflict statements for x86-64 targets

fixes #13462

This build system mixin was not updated after the support for specific
targets has been merged.

* Updated the version range of cuda that conflicts with gcc@8:

* Updated the version range of cuda that conflicts with gcc@8: for ppc64le

* Relaxed conflicts for version > 10.1

* Updated versions in conflicts

Co-Authored-By: Axel Huebl <axel.huebl@plasma.ninja>
2019-10-30 09:25:46 -07:00
Peter Scheibel
c99d367a11 syaml_int type should use int.__repr__ rather than str.__repr__ (#13487) 2019-10-29 13:51:48 -07:00
Tamara Dahlgren
3cf21e6edc bugfix: make spack -d test test_changed_files work (#13229)
The `test_changed_files` in `test/cmd/flake8.py` was failing because it calls
`ArgumentParser.parse_args()` without arguments.  Normally that would just
parse `sys.argv` but it seems to fail because of something in either `spack test`
or `pytest`.  Call it with an empty array so that it doesn't try to touch`sys.argv`
at all.

- [x] allow `-d` spack option for `test_changed_files`
2019-10-29 00:25:59 -07:00
Adam J. Stewart
2264e30d99
Update URL parsing regexes and tests (#13411)
* Update URL parsing regexes and tests

* Get rid of no longer used README

* Merge py-udunits and py-cf-units

* netcdf -> netcdf-c

* setup_environment -> setup_*_environment

* Fix doc tests

* Few last minute fixes

* Simplify prefix removal copypasta
2019-10-28 20:27:54 -05:00
Adam J. Stewart
b4e027e918
Output directory Spack is looking in for MKL libs (#13465) 2019-10-28 11:46:59 -05:00
Todd Gamblin
8eeb64096c
version bump: 0.13.0 2019-10-25 21:53:52 -07:00
Todd Gamblin
847703c13d
tutorial: move tutorial to standalone site (#13450)
* docs: add a spack environment for building the docs

* docs: remove tutorial and link to spack-tutorial.readthedocs.io

The tutorial now has its own standalone website, versioned by instances
of the tutorial.  Link to that instead of versioning it directly with Spack.
2019-10-25 21:49:27 -07:00
Peter Scheibel
4af448724f mirror create --all can mirror everything (#12940)
Support mirroring all packages with `spack mirror create --all`.

In this mode there is no concretization:

* Spack pulls every version of every package into the created mirror.
* It also makes multiple attempts for each package/version combination
  (if there is a temporary connection failure).
* Continues if all attempts fail. i.e., this makes its best effort to
  fetch evrerything, even if all attempts to fetch one package fail.

This also changes mirroring logic to prefer storing sources by their hash
or by a unique name derived from the source.  For example:

* Archives with checksums are named by the sha256 sum, i.e.,
  `archive/f6/f6cf3bd233f9ea6147b21c7c02cac24e5363570ce4fd6be11dab9f499ed6a7d8.tar.gz`
  vs the previous `<package-name>-package-version>.tar.gz`
* VCS repositories are stored by a path derived from their URL,
  e.g. `git/google/leveldb.git/master.tar.gz`.

The new mirror layout allows different packages to refer to the same
resource or source without duplicating that download in the
mirror/cache. This change is not essential to mirroring everything but is
expected to save space when mirroring packages that all use the same
resource.

The new structure of the mirror is:

```
<base directory>/
  _source-cache/   <-- the _source-cache directory is new
    archive/       <-- archives/resources/patches stored by hash
      00/          <-- 2-letter sha256 prefix
        002748bdd0319d5ab82606cf92dc210fc1c05d0607a2e1d5538f60512b029056.tar.gz
      01/
        0154c25c45b5506b6d618ca8e18d0ef093dac47946ac0df464fb21e77b504118.tar.gz
        0173a74a515211997a3117a47e7b9ea43594a04b865b69da5a71c0886fa829ea.tar.gz
        ...
    git/
      OpenFAST/
        openfast.git/
          master.tar.gz     <-- repo by branch name
      PHASTA/
        phasta.git/
          11f431f2d1a53a529dab4b0f079ab8aab7ca1109.tar.gz  <-- repo by commit
      ...
    svn/      <-- each fetch strategy has its own subdirectory
      ...
  openmpi/   <-- the remaining package directories have the old format
    openmpi-1.10.1.tar.gz  <-- human-readable name is symlink to _source-cache
```

In addition to the archive names as described above, `mirror create` now
also creates symlinks with the old format to help users understand which
package each mirrored archive is associated with, and to allow mirrors to
work with old spack versions. The symlinks are relative so the mirror
directory can still itself be archived.

Other improvements:

* `spack mirror create` will not re-download resources that have already
  been placed in it.

* When creating a mirror, the resources downloaded to the mirror will not
  be cached (things are not stored twice).
2019-10-25 21:47:28 -07:00
Peter Scheibel
cfbac14cba bugfix: restore upstream lock safety; update tests
Restore upstream lock safety; avoid calling methods directly on upstream DB in test.
2019-10-25 16:01:45 -07:00
Andreas Baumbach
ab55890ba8 Make error msg of test_api_for_build_and_run_environment more informative (#13435) 2019-10-25 19:24:41 +02:00
Todd Gamblin
ebeb1ed62f
bugfix: reindexing is not necessary for DB v0.9.3 to v5 upgrade (#13434)
reindexing takes a significant amount of time, and there's no reason to
do it from DB version 0.9.3 to version 5. The only difference is that v5
can contain "deprecated_for" fields.

- [x] Add a `_skip_reindex` list at the start of `database.py`

- [x] Skip the reindex for upgrades in this list. The new version will
  just be written to the file the first time we actually have to write
  the DB out (e.g., after an install), and reads will still work fine.
2019-10-25 07:10:03 -07:00
Todd Gamblin
dbee91f7f1
bugfix: allow fetching no-code packages (#13429)
Previously, spack would error out if we tried to fetch something with no
code, but that would prevent fetching dependencies. In particular, this
would fail:

    spack fetch --dependencies xsdk

- [x] Instead of raising an error, just print a message that there is nothing
      to be fetched for packages like xsdk that do not have code.

- [x] Make BundleFetchStrategy a bit more quiet about doing nothing.
2019-10-24 16:02:28 -07:00
Todd Gamblin
757387dc2a
commands: Add --json argument to spack spec (#13431)
We've had `spack spec --yaml` for a while, and we've had methods for JSON
for a while as well.  We just haven't has a `--json` argument for `spack spec`.

- [x] Add a `--json` argument to `spack spec`, just like `--yaml`
2019-10-24 16:01:31 -07:00
Chris Green
77af4684aa Improvements to detection of AMD architectures. (#13407)
New entry for K10 microarchitecture.

Reorder Zen* microarchitectures to avoid triggering as k10.

Remove some desktop-specific flags that were preventing Opteron Bulldozer/Piledriver/Steamroller/Excavator CPUs from being recognized as such.

Remove one or two flags which weren't produced in /proc/cpuinfo on older OS (RHEL6 and friends).
2019-10-24 15:48:54 -07:00
Chris Green
0913328812 Correctly identify Skylake CPUs on Darwin. (#13377)
* Correctly identify Skylake CPUs on Darwin.

* Add a test for haswell on Mojave.
2019-10-24 12:44:58 -05:00