Commit graph

3382 commits

Author SHA1 Message Date
Tamara Dahlgren
b42324c23b bugfix: template create constructor args (#12536)
c9e214f updated template creation by passing **kwargs to package
template classes but the template classes were not updated to accept
them; this adds **kwargs to package template initializers where they
are needed.
2019-08-23 10:45:17 -07:00
Seth R. Johnson
c86006e948 Skip invisible non-packages in package directory. (#12467)
Having a non-directory invisible file causes `spack find` to die. This
fixes the logic to ignore invalid module names but only warn if they're
visible.
```
NotADirectoryError: [Errno 20] Not a directory: '/spack/var/spack/repos/builtin/packages/.DS_Store/package.py'
```
2019-08-23 06:42:17 +02:00
Tamara Dahlgren
c9e214f6d3 Spack BundlePackage: a group of other packages (#11981)
This adds a special package type to Spack which is used to aggregate
a set of packages that a user might commonly install together; it
does not include any source code itself and does not require a
download URL like other Spack packages. It may include an 'install'
method to generate scripts, and Spack will run post-install hooks
(including module generation).

* Add new BundlePackage type
* Update the Xsdk package to be a BundlePackage and remove the
  'install' method (previously it had a noop install method)
* "spack create --template" now takes "bundle" as an option
* Rename cmd_create_repo fixture to "mock_test_repo" and relocate it
  to shared pytest fixtures
* Add unit tests for BundlePackage behavior
2019-08-22 11:08:23 -07:00
Greg Becker
5af2378ece compiler install: fix bug in construction of compiler spec (#12507) 2019-08-22 09:57:53 -07:00
Peter Scheibel
661b1da5d0
'spack spec --yaml' command now writes build deps to yaml (#12317)
This allows "spack spec --yaml" to generate a spec YAML file that can
be used with "spack install -f". Before, this would fail in cases
where the spec had build dependencies.
2019-08-21 18:32:31 -07:00
Chris Green
2499c047f0 Emacs directives for fill-column setting. 2019-08-21 18:21:05 -07:00
Chris Green
ad4309b782 Fetch strategies: new global option no_cache, new git option get_full_repo.
* All fetch strategies now accept the Boolean version keyword option `no_cache` in order to allow per-version control of cache-ability.
* New git-specific version keyword option `get_full_repo` (Boolean). When true, disables the default `--depth 1` and `--single-branch` optimizations that are applied if supported by the git version and (in the former case) transport protocol.
* The try / catch blog attempting `--depth 1` and retrying on failure has been removed in favor of more accurately ascertaining when the `--depth` option should work based on git version and protocol choice. Any failure is now treated as a real problem, and the clone is only attempted once.
* Test improvements:
  * `mock_git_repository.checks[type_of_test].args['git']` is now specified as the URL (with leading `file://`) in order to avoid complaints when using `--depth`.
  * New type_of_test `tag-branch`.
  * mock_git_repository now provides `git_exe`.
  * Improved the action of the `git_version` fixture, which was previously hard-wired.
  * New tests of `--single-branch` and `--depth 1` behavior.
* Add documentation of new options to the packaging guide.
2019-08-21 18:21:05 -07:00
Chris Green
ccb688f575 Fix to option text for help command. (#12444) 2019-08-21 17:11:24 -07:00
Peter Scheibel
5127d95b2e properly format message with all paths that could not be used. also report canonicalized paths vs. Spack config variables like 'tempdir' (#12492) 2019-08-21 15:46:12 -07:00
Chris Green
2441c0bad7 pkg command uses a working directory for git invocations rather than -C. (#12491) 2019-08-21 09:31:27 -07:00
Greg Becker
3b115fffb1 permissions: fix file permissions on intermediate install directories (#12399)
- mkdirp now takes arguments to allow it to properly set permissions on created directories.
- Two arguments (group and mode) set permissions for the leaf directory.
- Intermediate directories can inherit permissions from either the topmost existing directory (the parent) or the leaf.
2019-08-20 23:08:02 -07:00
Todd Gamblin
368bf2d69c
stage: avoid using group name in _first_accessible_path()
On machines where $TMP is owned by a gid with no name, this avoids the
following error when the default spack stage does not exist:

    (spackbook):spack$ spack clean
    ==> Removing all temporary build stages
    ==> Error: 'getgrgid(): gid not found: 57095'

Spack needs to deal with gids directly unless users pass them in.
2019-08-20 14:52:53 -07:00
Tamara Dahlgren
aed204e409
cleanup: remove extraneous prints (#12474)
Removes messages accidentally introduced in #12072 .
2019-08-19 14:49:09 -07:00
Peter Scheibel
31ff791180 features: Update compiler caching (#7675)
Compiler caching was using the `id()` function to refer to configuration dictionary objects. If these objects are garbage-collected, this can produce incorrect results (false positive cache hits). This change replaces `id()` with an object that keeps a reference to the config dictionary so that it is not garbage-collected.
2019-08-19 11:24:05 -07:00
Tamara Dahlgren
0ea6e0f817
features: Remove stage symlinks (#12072)
Fixes #11163

The goal of this work is to simplify stage directory structures by eliminating use of symbolic links. This means, among other things, that` $spack/var/spack/stage` will no longer be the core staging directory. Instead, the first accessible `config:build_stage` path will be used.

Spack will no longer automatically append `spack-stage` (or the like) to configured build stage directories so the onus of distinguishing the directory from other work -- so the other work is not automatically removed with a `spack clean` operation -- falls on the user.
2019-08-19 10:31:24 -07:00
Zack Galbreath
cef1e4e0b4 Do not report on packages installed from the cache (#12336)
Skip generating reports for any packages that were found in the binary cache.
2019-08-16 10:19:11 -07:00
Greg Becker
0307b5a3dc mirrors: make spack mirror environment aware (#12345)
Allow "spack mirror" with no spec arguments to create a mirror for the
active environment (download all concretized specs in the environment)
2019-08-15 18:45:17 -07:00
Peter Scheibel
1f6befabb1
Fix "spack loads" when retrieving upstream modules (#12203)
Fixes #12062

406c791 addressed "spack module load" for upstream modules but not
the "spack module loads" command. This applies the same fixes from
406c791 to "spack module loads".
2019-08-13 11:09:40 -07:00
Massimiliano Culpo
ab4b5deb97 bugfix: Python 2.6 parsing error (#11867)
Apparently shlex.split can't deal with unicode encoded characters in
Python2.6. The solution is to convert to str before calling the
function.
2019-08-11 13:00:36 -07:00
Greg Becker
be6ac6ce6d bugfix: nested matrices in spec lists (#12320)
* stack concretization: fix handling of variant names with dashes
* spec_list: bugfix for handling nested matrices
2019-08-08 20:53:31 -07:00
Adam J. Stewart
44f4ad7bbf Allow Spack to uninstall external extensions without permissions (#11991)
* Allow Spack to uninstall external extensions without permissions

Raise when self.projections is not empty
2019-08-08 17:46:14 -07:00
Pariksheet Nanda
72bb7646fe docs: fix compiler config (#12330)
It's no longer possible to set compiler flags under as an entry under
"paths" in compilers.yaml; instead the user must list these under the
"flags" section. This updates the docs accordingly.
2019-08-08 16:47:24 -07:00
Adam J. Stewart
b4862be875 Fix typo in yaml config docs (#12312) 2019-08-08 10:56:11 +02:00
Greg Becker
36fbd6cf4c bugfix: spack stacks mixed dependency and non-dep constraints (#12315)
Spack stacks drop invalid dependencies applied to packages by a
spec_list matrix operation

Without this fix, Spack would raise an error if orthogonal dependency
constraints and non-dependency constraints were applied to the same
package by a matrix and the dependency constraint was invalid for
that package. This is an error, fixed by this PR.

An example failing configuration:

spack:
  definitions:
  - packages: [libelf, hdf5+mpi]
  - compilers: ['%gcc']
  - mpis: [^openmpi]

  specs:
  - matrix:
    - $packages
    - $compilers
    - $mpis
2019-08-07 18:34:41 -07:00
Peter Scheibel
92ec732f9a compiler bootstrapping: enable both single-node and multi-node DAGS(#12310)
5f74f22 enabled installing compilers for dependencies but not for the root package (and in particular not for DAGs which consist of one package)

this enables bootstrapping compilers for both types of DAGs
2019-08-07 14:43:25 -07:00
Tamara Dahlgren
951d42596b tests: explain and test dependency flattening routines (#11993)
- Add comments to explain that `install_dependency_symlinks` and `flatten_dependencies` are actually used.
- Add a test that exercises the routines.
2019-08-06 17:11:23 -07:00
Greg Becker
15884a679b
mirrors: mirror config should use spack variable expansions (#9027)
- ensure that `$spack` and other variables are substituted into mirror
  paths
2019-08-06 13:54:50 -07:00
Greg Becker
3b4d6ddc03 stacks: fix reference handling in env.write() (#12096)
* stacks: Fix env.write to properly write references
* stacks: regression test for 12095
2019-08-06 12:53:41 -07:00
Pariksheet Nanda
f68cdd1ad0 Fix typo in config (#12267)
Using "compilers" with the "s" is an invalid config section and throws an error.

Traceback (most recent call last):
  File "spack/bin/spack", line 48, in <module>
    sys.exit(spack.main.main())
  File "/home/omsai/src/libkmap/spack/lib/spack/spack/main.py", line 633, in main
    env = ev.find_environment(args)
  File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 263, in find_environment
    return Environment(env)
  File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 534, in __init__
    self._read_manifest(f)
  File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 561, in _read_manifest
    self.yaml = _read_yaml(f)
  File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 402, in _read_yaml
    validate(data, filename)
  File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 395, in validate
    e, data, filename, e.instance.lc.line + 1)
spack.config.ConfigFormatError: /home/omsai/src/libkmap/spack.yaml:15: Additional properties are not allowed ('compilers' was unexpected)
2019-08-03 20:18:44 -05:00
Adam J. Stewart
98605bba0d Fix CNL version detection (#12207) 2019-08-03 12:00:30 -07:00
Massimiliano Culpo
90756d0428 Refactor UI logic out of Environment.concretize (#12213)
Environment.concretize returns newly-concretized specs rather than
printing them; as a result, the _display argument is removed from
Environment.concretize (originally only used to avoid printing specs
during unit testing). Command logic which invokes
Environment.concretize prints these explicitly.
2019-08-02 17:27:51 -07:00
Seth R. Johnson
46027bea13 Fix QT4 build for mac (#10944)
This updates the Spack QT package to enable building qt version 4 on
MacOS. 

This includes the following changes to the qt package:

* add version 4.8.7
* add option to build with or without shared libs
* add options to disable tools, ssl, sql, and freetype support
* add qt4-tools patch when building qt@4+tools
* add option to build as a framework (only available on MacOS)
* replace qt4-el-capitan patch with qt4-mac patch (which includes the
  edits from qt4-el-capitan)
* apply qt4-pcre-include-conflict.patch only for version 4.8.6
  (rather than all 4.x versions)
* apply qt4-gcc-and-webkit.patch for 4.x versions before 4.8.7 and
  create a separate qt4-gcc-and-webkit-487.patch for version 4.8.7
* update patch function for qt@4 on MacOS to update configure
  variables relevant to Spack (e.g. PREFIX)
* add option to build freetype with Spack, as a vendored dependency
  of QT, or not at all (default is to build with Spack)

This includes the following edits outside of the qt package:

* Update MacOS version utility function to return all parts of the
  Mac version (rather than just the first two)
* gettext package: implement "libs"
* python package: add gettext as a dependency
2019-08-01 10:22:35 -07:00
Patrick Gartung
4ff14bd0b2
Buildcache: skip binary string replacement with padding when the new install path is longer than the old install path. (#12227)
* Raise an exception and exit with a meaningful message when binary path substitution fails.

* Skip binary text replacement with padding and issue a warning when the new install path is longer than the old install path.
2019-08-01 12:07:04 -05:00
Levi Baber
8ec9ebc92f should not need bin prefix on spack command (#12215) 2019-07-31 12:58:01 -05:00
Todd Gamblin
33701cc355
docs: update tutorial slides for PEARC19 2019-07-31 12:55:59 -05:00
Adam J. Stewart
f7d1df4d3a
Update Package Creation Tutorial for PEARC19 (#12209)
* Update Package Creation Tutorial for PEARC19

* A few remaining fixes

* Fix section link
2019-07-31 12:54:09 -05:00
Christoph Junghans
4ea5e177cd
CMake: make ninja verbose as well (#12200) 2019-07-31 08:41:58 -06:00
Javier Cervantes
a948954886 Print environment name when removing active env (#12198) 2019-07-31 14:40:11 +02:00
Adam J. Stewart
d973fca00e Fix Sphinx deprecation warning when building docs (#12205)
* Fix Sphinx deprecation warning when building docs

* Set minimum required version of Sphinx
2019-07-31 10:23:56 +02:00
Adam J. Stewart
b34da4e108
Add SIPPackage base class (#12157) 2019-07-30 17:12:48 -05:00
Adam J. Stewart
23420a6524 Typo fixes in Environments Tutorial (#12107) 2019-07-29 17:13:56 -05:00
Greg Becker
f78fc30032 intel: Add mpicc wrapper bindir to dependent env (#12132) 2019-07-29 12:04:15 -07:00
Gregory Lee
e8a71089a6 use Excecutable instead of exec for editing licenses (#11968)
* fix defunct editor exit in #11691
2019-07-25 11:48:32 -07:00
Todd Gamblin
b0abbfecb8 new command: spack maintainers queries package maintainers
- We don't currently make enough use of the maintainers field on
  packages, though we could use it to assign reviews.

- add a command that allows maintainers to be queried
  - can ask who is maintaining a package or packages
  - can ask what packages users are maintaining
  - can list all maintained or unmaintained packages

- add tests for the command
2019-07-24 14:10:08 -07:00
Tamara Dahlgren
7411347a29 tests: Test install of unconcretized spec (#12099) 2019-07-24 13:30:27 -05:00
Massimiliano Culpo
6fac0ae687 Fixed a name clash in the 'from_environment_diff' function (#12116)
* Added a unit test reproducing the failure in 12085

* Fixed name clash in the 'from_environment_diff' function

The bug reported in #12085 stemmed from a name clash among variables,
introduced during the refactor in #10753 and not caught by unit tests
and reviews.
2019-07-24 10:25:24 -05:00
Adam J. Stewart
58427a7102
Typo fixes in Spack Environments docs (#12100) 2019-07-23 17:42:16 -05:00
Greg Becker
daac376862
error messages: include whole spec for "no externals found" message (#12070)
* Provide better error message when matching externals are found for package marked not buildable
2019-07-23 13:58:07 -05:00
Gregory Becker
a086dda82f hashing: fix caching of dependency hashes in to_node_dict 2019-07-23 08:58:49 -07:00
Todd Gamblin
3cb40e1c02 commands: add tests for all subcommands of spack pkg 2019-07-22 23:01:45 -07:00