Commit graph

3345 commits

Author SHA1 Message Date
Javier Cervantes
a948954886 Print environment name when removing active env (#12198) 2019-07-31 14:40:11 +02:00
Adam J. Stewart
d973fca00e Fix Sphinx deprecation warning when building docs (#12205)
* Fix Sphinx deprecation warning when building docs

* Set minimum required version of Sphinx
2019-07-31 10:23:56 +02:00
Adam J. Stewart
b34da4e108
Add SIPPackage base class (#12157) 2019-07-30 17:12:48 -05:00
Adam J. Stewart
23420a6524 Typo fixes in Environments Tutorial (#12107) 2019-07-29 17:13:56 -05:00
Greg Becker
f78fc30032 intel: Add mpicc wrapper bindir to dependent env (#12132) 2019-07-29 12:04:15 -07:00
Gregory Lee
e8a71089a6 use Excecutable instead of exec for editing licenses (#11968)
* fix defunct editor exit in #11691
2019-07-25 11:48:32 -07:00
Todd Gamblin
b0abbfecb8 new command: spack maintainers queries package maintainers
- We don't currently make enough use of the maintainers field on
  packages, though we could use it to assign reviews.

- add a command that allows maintainers to be queried
  - can ask who is maintaining a package or packages
  - can ask what packages users are maintaining
  - can list all maintained or unmaintained packages

- add tests for the command
2019-07-24 14:10:08 -07:00
Tamara Dahlgren
7411347a29 tests: Test install of unconcretized spec (#12099) 2019-07-24 13:30:27 -05:00
Massimiliano Culpo
6fac0ae687 Fixed a name clash in the 'from_environment_diff' function (#12116)
* Added a unit test reproducing the failure in 12085

* Fixed name clash in the 'from_environment_diff' function

The bug reported in #12085 stemmed from a name clash among variables,
introduced during the refactor in #10753 and not caught by unit tests
and reviews.
2019-07-24 10:25:24 -05:00
Adam J. Stewart
58427a7102
Typo fixes in Spack Environments docs (#12100) 2019-07-23 17:42:16 -05:00
Greg Becker
daac376862
error messages: include whole spec for "no externals found" message (#12070)
* Provide better error message when matching externals are found for package marked not buildable
2019-07-23 13:58:07 -05:00
Gregory Becker
a086dda82f hashing: fix caching of dependency hashes in to_node_dict 2019-07-23 08:58:49 -07:00
Todd Gamblin
3cb40e1c02 commands: add tests for all subcommands of spack pkg 2019-07-22 23:01:45 -07:00
Todd Gamblin
0b17bccdce commands: add spack pkg changed subcommand, better test support
- Add `spack pkg changed` to show changed and added packages for a
  commit.

- Make `spack pkg` support tests better.
2019-07-22 23:01:45 -07:00
Denis Davydov
8cef81363c gcc: support without command line tools (#11040) 2019-07-22 16:53:50 -05:00
Adam J. Stewart
2d144316a8 Fix version scraping for CRAN packages (#12021)
* Fix version scraping for CRAN packages
* Remove set literals
2019-07-22 13:05:55 -07:00
Todd Gamblin
cc4094bb9b bugfix: env.write() should stringify Spec lists.
- Setting specs from lockfiles was not correctly stringifying concretized
  user specs.

- Fix `_set_user_specs_from_lockfile`

- Add some validation code to `SpecList` constructor
2019-07-22 13:45:34 -05:00
Peter Josef Scheibel
7ec1d320a8 hashes: consolidate and make hashing logic more consistent
Spack has evolved to have three types of hash functions, and it's
becoming hard to tell when each one is called. Whlie we aren't yet ready
to get rid of them, we can refactor them so that the code is clearer and
easier to track.

- Add a `hash_types` module with concise descriptors for hashes.

- Consolidate hashing logic in a private `Spec._spec_hash()` function.

- `dag_hash()`, `build_hash()`, and `full_hash()` all call `_spec_hash()`

- `to_node_dict()`, `to_dict()`, `to_yaml()` and `to_json()` now take a
  `hash` parameter consistent with the one that `_spec_hash()` requires.

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2019-07-22 13:45:34 -05:00
Peter Josef Scheibel
0715b512a1 env: environments index specs by full DAG w/build deps
- ensure that Spec._build_hash attr is defined
- add logic to compute _build_hash when it is not already computed (e.g. for specs prior to this PR)
- add test to ensure that different instance of a build dep are preserved
- test conversion of old env lockfile format to new format
- tests: avoid view creation, DAG display in tests using MockPackage
- add regression test for more-general bug also fixed by this PR
- update lockfile version since the way we are maintaining hashes has changed
- write out backup for version-1 lockfiles and test that
2019-07-22 13:45:34 -05:00
Adam J. Stewart
7d954720ff Add OpenMP flag for Apple Clang (#12049) 2019-07-20 13:40:36 -07:00
Massimiliano Culpo
a2cb26f520 tests: speed up tests that rely on the database fixture (#12031)
The database and mutable_database fixtures were installing and uninstalling the same specs multiple times to ensure the database for tests has the correct state.

This commit optimizes the procedure by caching the state in an external directory, and copying it in instead of going through the installation or uninstallation again.

The database fixture is meant not to be modified by tests. This commit enforces this invariant by making the database read-only before starting the test.

* Added missing db markers to tests
* Added test for uninstall_by_spec
* `database` fixture now returns a read-only database
* Tests that modify the DB now use `mutable_database` fixture
2019-07-20 10:18:50 -07:00
Greg Becker
67cd524e93 environment views: refactor stripping build deps (#12069) 2019-07-20 09:45:20 -07:00
Greg Becker
8ec098716b compilers: don't raise errors for duplicate compiler definitions (#11910)
Summary:
- Allow multiple definitions of compiler in compilers.yaml (use first instance)
- Still print debug messages when there are duplicates, to assist users in finding this issue.

Merging configs from different scopes can result in multiple compiler being present in the same configuration list.  Instead of raising when there are duplicates, take the one with highest precedence.

Print a debug message instead of raising, so that we can still diagnose this.  We don't have a good way of warning the user about inconsistent configuration *in the same file* -- we'd need to dig into YAML file/line info for that.
2019-07-20 00:42:12 -07:00
Todd Gamblin
993ee7f199 environments: add activate/deactivate tests, work wtih set -u
- [x] Add shell tests to ensure that `spack env activate`, `spack env
  deactivate`, and `despacktivate` continue to work.

- [x] Also ensure that activate and deactivate both work with `set -u`
2019-07-20 00:36:56 -07:00
Greg Becker
4f9131fdc2 Package inheritance: Find patch files defined in parent classes using MRO (#12051)
Fixes #8908, 11844

Use Python MRO to find patch files from parent classes.
2019-07-19 18:08:05 -07:00
albestro
3a026f1412 Fix #11240 (#11995)
* extends mkdirs with permissions for intermediate folders

Does not use os.makedirs mode parameter because its behavior is changed
with Python 3.7 (it ignores it for intermediate dirs), and moreover it
was not possible to set different modes for newly-created folders
and leaf folder.

reference:
- https://bugs.python.org/issue19930
- https://docs.python.org/3.7/library/os.html#os.makedirs

* comment mkdirp step easing code understanding

* revert mkdir to default for package metapath

since metapath is nested in package folder, there is no need
to specify permissions for intermediate folders because the prefix
already exists.

* comment create_install_directory package modes
2019-07-19 10:13:29 -05:00
Gregory Becker
ba0cd4d912 concretization: fix transient hang in python 3.5
Bug relates to the interplay between:
1. random dict orders in python 3.5
2. bugfix in initial implementation of stacks for `_concretize_dependencies`
   when `self._dependencies` is empty
3. bug in coconcretization algorithm computation of split specs

Result was transient hang in coconcretization.
Fixed #3 (bug in coconcretization) to resolve.
2019-07-18 19:28:50 -07:00
Gregory Becker
ec6e5b0fd3 stacks: add option to link only roots or all specs, default to all 2019-07-18 19:28:50 -07:00
Gregory Becker
7ec89aa036 stacks: refactor view descriptors into a separate object 2019-07-18 19:28:50 -07:00
Gregory Becker
6661cb1926 refactor: clean up Environment class
- remove redundant code in Environment.__init__
- use socket.gethostname() instead of which('hostname')
- refactor updating SpecList references
- refactor 'specs' literals to a single variable for default list name
- use six.string_types for python 2/3 compatibility
2019-07-18 19:28:50 -07:00
Gregory Becker
88295d927e bugfix: spec lists can now include multiple references 2019-07-18 19:28:50 -07:00
Gregory Becker
9577ade3ce stacks: use new format string syntax in environments code
- Change old spec expressions to use Spack's new spec formatting sytnax.
2019-07-18 19:28:50 -07:00
Gregory Becker
2f6af64157 docs: Add documentation for views in environments 2019-07-18 19:28:50 -07:00
Gregory Becker
cebf1fd668 stacks: update view management for multiple/combinatorial views
This adds notion of a default view, other views in environments
2019-07-18 19:28:50 -07:00
Gregory Becker
d0bfe0d6a8 stacks: environment add/remove respect multiply-defined lists 2019-07-18 19:28:50 -07:00
Gregory Becker
f9e6de5474 stacks: allow force-removing from a speclist used in a matrix 2019-07-18 19:28:50 -07:00
Gregory Becker
533a8d7c82 stacks: update environment add/remove algorithms 2019-07-18 19:28:50 -07:00
Gregory Becker
6cfbfcc967 concretization: regression test for constrain() with deps
- ensure that `Spec('foo').constrain('foo ^bar')` works
- prior to stacks implementation, this constraint would have done nothing.
2019-07-18 19:28:50 -07:00
Gregory Becker
12c8996294 docs: initial documentation for environments 2019-07-18 19:28:50 -07:00
Gregory Becker
d450a2fce2 stacks: initial implementation of stacks on environments
- stack syntax in env schema
- switch environment specs over to SpecList object
- add stack functionality to environments
- handle definition extensions through stack.yaml and SpecList
- implement conditional definitions
- tests
2019-07-18 19:28:50 -07:00
Adam J. Stewart
7f3048c8af
Add support for .txz and .tbz2 file extensions (#12048) 2019-07-17 16:23:23 -05:00
Tamara Dahlgren
b856e24444 tests: uninstall_by_spec error and rpath_args tests (#11971)
* tests: Add uninstall_by_spec error tests.

These tests were originally included in PR #11797.
2019-07-17 15:08:08 -05:00
Greg Becker
5cf8878185 feature: Allow developers to use Spack for partial builds (#12006)
Added new diy option.
2019-07-17 11:46:56 -07:00
Massimiliano Culpo
2fe1ecbaa2 Ignore Modules v4 environment variables in from_sourcing_file (#10753)
* from_sourcing_file: fixed a bug + added a few ignored variables

closes #7536

Credits for this change goes to mgsternberg (original author of #7536)

The new variables being ignored are specific to Modules v4.

* Use Spack Executable in 'EnvironmentModifications.from_sourcing_file'

Using this class avoids duplicating lower level logic to decode
stdout and handle non-zero return codes

* Extracted a function that returns the environment after sourcing files

The logic in `EnvironmentModifications.from_sourcing_file` has been
simplified by extracting a function that returns a dictionary with the
environment one would have after sourcing the files passed as argument.

* Further refactoring of EnvironmentModifications.from_sourcing_file

Extracted a function that sanitizes a dictionary removing keys that are
blacklisted, but keeping those that are whitelisted. Blacklisting and
whitelisting can be done on literals or regex.

Extracted a new factory that creates an instance of
EnvironmentModifications from a diff of two environments.

* Added unit tests

* PS1 is blacklisted + more readable names for some variables
2019-07-16 10:54:33 -05:00
Michael Kuhn
a7e915d38b Fix build_jobs exceeding number of cores
All documentation mentions that `build_jobs` is limited by the number of
cores available in the system. This is also enforced when setting it via
`--jobs`. However, when setting it via `config.yaml`, it can exceed the
number of cores available, making builds run out of memory.
2019-07-15 11:40:49 -07:00
Glenn Johnson
3f83a2a7d8 Add auto-dispatch specification to Intel packages (#11697)
This PR adds the ability to specify the auto-dispatch targets that can
be used by the Intel compilers. The `-ax` flag will be written to the
respective compiler configuration files. This ability is very handy when
wanting to build optimized builds for various architectures. This PR
does not set any optimization flags, however.
2019-07-15 11:37:54 -07:00
Massimiliano Culpo
5acbe449e5 spack uninstall can uninstall specs with multiple roots (#11977)
Fixes #3690
Fixes #5637

Uninstalling dependents of a spec was relying on a traversal of the
parents done by inspecting spec._dependents. This is in turn a
DependencyMap that maps a package name to a single DependencySpec object
(an edge in the DAG) and cannot thus model the case where a spec has
multiple configurations of the same parent package installed (for
example if different versions of the same Python library depend on
the same Python installation).

This commit works around this issue by constructing the list of specs to
be uninstalled in an alternative way, and adds tests to verify the
behavior. The core issue with DependencyMap is not resolved here.
2019-07-15 10:30:01 -07:00
Tim Fuller
5bc15b2d9a find_libraries searches lib and lib64 before prefix (#11958)
The default library search for a package checks the lib/ and lib64/
directories for libraries before the root prefix, in order to save
time when searching for libraries provided by externals (which e.g.
may have '/usr/' as their root).

This moves that logic into the "find_libraries" utility method so
packages implementing their own custom library search logic can
benefit from it.

This also updates packages which appear to be replicating this logic
exactly, replacing it with a single call to "find_libraries".
2019-07-12 17:46:47 -07:00
Oliver Breitwieser
d1af7ae5ab Copy --{use,no}-cache from install to bootstrap (#11786) 2019-07-12 16:30:12 -07:00
Carson Woods
76f1ee5f32 'spack compiler add' resolves relative path to absolute path (#11792)
Fixes #11782

Spack was not properly resolving relative paths to absolute paths
when a relative path was passed to "spack compiler add [PATH]".
Now, if provided a relative path, the absolute path is written to
compilers.yaml rather than the relative path.
2019-07-12 16:06:26 -07:00