Commit graph

4001 commits

Author SHA1 Message Date
Massimiliano Culpo
2a4ca34a61
Recognize system installed gcc-10 as compilers (#16884)
Now that the version number of GCC reached double digits, an update
to the regex is needed to recognize gcc-10 as an executable to be
inspected when searching for compilers.
2020-06-01 13:59:14 -07:00
Sergey Kosukhin
9d1f71e89f
repo: fix construction of UnknownPackageError (#16865)
Also improve the error messages
2020-05-29 20:07:44 +02:00
Robert Rosca
edd075a5e1
Mention that packages can be uninstalled by hash (#16863) 2020-05-29 13:23:02 +02:00
Greg Becker
bd95b7df86
spack module_cmd: set LD_LIBRARY_PATH for python in subshell. (#16827)
* Ensure python runs with proper ld_library_path within Spack regardless of environment
2020-05-29 11:57:21 +02:00
Jane Herriman
6af3272e1d
Update error message when UnavailableCompilerVersionError is triggered (#16838)
Co-authored-by: Herriman <herriman1@panorama.llnl.gov>
2020-05-29 10:53:43 +02:00
Greg Becker
a21fab6fe4
compilers: add opt_flags and debug_flags properties (#16710) 2020-05-29 10:52:31 +02:00
Greg Becker
38ac78489a
bugfix: use flags when computing implicit rpaths (#16634)
* make verbose_flag a property

* tests
2020-05-28 17:11:28 -07:00
Massimiliano Culpo
8f208b17e2 spack containerize: allow 0.14.3 tag for next release 2020-05-27 10:21:14 -07:00
Massimiliano Culpo
187cb51b49 spack containerize: allow 0.14.1 and 0.14.2 tags
fixes #16727
2020-05-27 10:21:14 -07:00
Sergey Kosukhin
e73c39a5db
concretize: fix UnboundLocalError due to import within a function (#16809) 2020-05-27 11:45:14 +02:00
Greg Becker
e9dcab9464
backwards compatibility for naming scheme (#16812)
* backwards compatibility for naming scheme
2020-05-26 14:06:25 -07:00
Massimiliano Culpo
c01433f60a
spack.relocate: further coverage for ELF related functions (#16585)
* make_link_relative: added docstring

* make_elf_binaries_relative: added docstring, unit tests

* raise_if_not_relocatable: added docstring, added unit test for exceptional case

* relocate_links: removed unused arguments, added docstring and comments

Also fixed a possible bug that was issuing spurious
warning when a file was relocated successfully

* relocate_text: added docstring and comments, renamed arguments

* relocate_text_bin: added docstring and comments, renamed arguments, unit tests
2020-05-26 11:47:31 -07:00
Stephen Herbein
c4031f0455
lua: fix compilation on Cray (#16713)
Problem: when calling `static_to_shared_library` on the `cray` arch, it
produces a non-sensical compiler command with no input files.  For
example, when installing lua@5.2.4, it produced:
'gcc -lm -ldl -o /big-long-spack-path/liblua.so.5.2.4'

Solution: do the same thing on `cray` that is done for `linux`
2020-05-25 15:18:47 +02:00
Peter Scheibel
768fa6bc53
bugfix: schema errors without line numbers (#16765)
* account for schema validation errors where the associated instance doesn't have a line number

* fix unrelated flake error (but it must be fixed because this PR touches this file and the flake rules have been updated since the last edit to this file)
2020-05-25 15:08:47 +02:00
Tomoki, Karatsu
a9b64bcaf1
autotools: delete args from postdep objects when %fj (#16274) 2020-05-21 20:50:57 +02:00
Greg Becker
cfb6f21236
externals: allow package prefs to configure default not buildable (#16735)
Allows `all` to be configured non-buildable in packages.yaml.

The following config would only allow zlib to be built by Spack, all other packages would have to be found as externals.

```
packages:
  all:
    buildable: False
  zlib:
    buildable: True
```
2020-05-20 17:09:07 -07:00
Tamara Dahlgren
92989fbbfd
tests: check presence not equality for rpath changes (#16637)
Fixes #16636
2020-05-20 18:11:26 +02:00
Scott Wittenburg
b1f1f5dddb Pipelines: Ensure consistent spack version for entire pipeline
Also retry jobs always, as script failure is not confined to only our
spack ci rebuild command exit code.
2020-05-19 09:40:02 -06:00
Greg Becker
84889ef3a3
move source dir to last cmake arg (#16709) 2020-05-18 16:39:13 -07:00
TZ
2ebadbc267
intel-mpi: add libfabrics directory to SPACK_COMPILER_EXTRA_RPATHS (#15214)
workaround for not finding an appropriate libfabrics.so with Intel
MPI 2019+. Fixes #11140 and #12493
2020-05-16 10:40:40 -05:00
Greg Becker
32a9adcf60
modules: use projections format for naming schemes (#16629)
* update tcl naming_scheme to use projections

* add projections to lmod modules
2020-05-15 11:12:52 -07:00
Scott Wittenburg
e0572a7d96 Pipelines: Support DAG scheduling and dynamic child pipelines
This change also adds a code path through the spack ci pipelines
infrastructure which supports PR testing on the Spack repository.
Gitlab pipelines run as a result of a PR (either creation or pushing
to a PR branch) will only verify that the packages in the environment
build without error.  When the PR branch is merged to develop,
another pipeline will run which results in the generated binaries
getting pushed to the binary mirror.
2020-05-14 21:11:07 -07:00
Scott Wittenburg
47b3dda1aa Support os-specific $padding in config:install_tree
Providing only $padding or ${padding} results in an attempt to
substitute a padding of maximum system path length, while leaving
room for the parts of the install path spack generates.  Providing
$padding-<len> or ${padding-<len>} simply substitutes padding of
the specified length.
2020-05-14 21:11:07 -07:00
Greg Becker
445cae5c81
Feature: Allow lmod configuration to set core specs (#16517)
Packages built with lmod core_compiler are placed in `Core`.

Other packages may belong in `Core`. For example, python may be built with a proprietary compiler for performance, but belong on the `Core` directory.

With this PR, lmod config can include a `core_specs` list. Any package that satisfies a spec in that list is placed in `Core`, regardless of its compiler or dependencies.
2020-05-14 11:27:37 -07:00
Greg Becker
7411fe2d43
cmake build system: filter system paths from rpaths (#16612) 2020-05-13 14:31:33 -07:00
Peter Scheibel
701fc1fdb1
Update docs on "spack external find" (#16482)
This improves the documentation for `spack external find` in several ways:

* Provide a code example of implementing `determine_spec_details` for a package
* Explain how to define executables to look for (and also e.g. that they are treated as regular expressions and so can pull in unexpected files).
* Add the "why" for a couple of constraints (i.e. explain that this logic only works for build/run deps because it examines `PATH` for executables)
* Spread the docs between build customization and packaging sections
* Add cross-references
* Add a label so that `spack external find` is linked from the command reference.
2020-05-12 17:08:08 -07:00
Greg Becker
0307111dd8
bugfix: reorder variants in Spec strings (#16462)
* change print order for variants to avoid zsh parsing bugs

* change tests for new variant parse order
2020-05-12 10:23:42 -07:00
Sajid Ali
b526a90220
Charm++: fix build prefix; add ucx/pmi support (#15666)
* Add pmi support (required by ucx, ofi, and gni backends)
* Add support for ucx backend
* Add dependency on MPI for pmi=simplepmi, slurmpmi, or slurmpmi2
* Remove charmpp as an MPI provider since the changes in this PR can
  add MPI as a dependency (mentioned previously)
* Install into transport_protocol-OS-arch subdirectory to match
  default charmpp installation behavior (which helps dependents find it)
2020-05-11 16:27:38 -07:00
Massimiliano Culpo
43c9ad3421
Remove 'spack bootstrap' and associated docs (#15179)
fixes #15145

This commit removes the outdated `spack bootstrap`
command and any reference to it in the documentation
and unit tests.
2020-05-11 10:55:18 -07:00
Massimiliano Culpo
b06bc31029
Increase coverage of spack.relocate (#16475)
- add docstrings and make parameter names consistent in `relocate.py`
- Make `replace_prefix_*` and other functions private (as they are implementation details)
- remove unused function _replace_prefix_nullterm()

- Add unit tests for `relocate.py` functions
  - add patchelf to Travis and use it during tests
  - add hello_world fixture with a compiled binary, so we can test relocation
2020-05-09 10:41:50 -07:00
Todd Gamblin
7b8e5c8999 bugfix: don't use sys.stdout as a default arg value (#16541)
After migrating to `travis-ci.com`, we saw I/O issues in our tests --
tests that relied on `capfd` and `capsys` were failing. We've also seen
this in GitHub actions, and it's kept us from switching to them so far.

Turns out that the issue is that using streams like `sys.stdout` as
default arguments doesn't play well with `pytest` and output redirection,
as `pytest` changes the values of `sys.stdout` and `sys.stderr`. if these
values are evaluated before output redirection (as they are when used as
default arg values), output won't be captured properly later.

- [x] replace all stream default arg values with `None`, and only assign stream
      values inside functions.
- [x] fix tests we didn't notice were relying on this erroneous behavior
2020-05-09 00:56:18 -07:00
Axel Huebl
f5844d3d81
all_urls: add urls[0] for versions (#16435)
This adds the `url` alternative `urls` to `package.all_urls`. With
this addition, one can find again new versions with
`spack versions <package>` for packages that are populated with
from mixin mirror `urls`.

Example: `util-macros` from x.org mixin.
2020-05-08 10:12:45 +02:00
Erik Schnetter
4ebff5cef9
Documentation: typo in option description (#16518) 2020-05-07 22:23:16 -07:00
iarspider
e2d42672b7
Config option to disable setting S_ISGID bit when creating installation directory (#14479)
* Add config option to disable setting S_ISGID bit when creating installation directory. 

Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch>
2020-05-07 17:21:53 -07:00
iarspider
08f449ae9a
"spack checksum" QoL (#14311)
* Non-interactive mode for spack checksum; allow passing 'package@version' to spack checksum

* Flake8 fixes

* Update checksum.py

Fix typo

* Update spack-completion script

* Automatically set non-interactive mode if more than one version passed

* Update lib/spack/spack/cmd/checksum.py

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* Add documentation and update spack-completion

* Flake8

* Rename option

* Update spack-completion

* Update lib/spack/spack/cmd/checksum.py

Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>

* Update checksum.py

* Update stage.py

* Update create.py

Use batch mode when adding a new package

Co-authored-by: Ivan Razumov <ivan.razumov@cern.ch>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-05-07 18:34:36 -05:00
Peter Scheibel
1ed564a1e6
Testing: fix unintended interactions between tests, part 2 (#16429)
This fixes some errors with setting up test configuration. These
errors do not cause current Spack tests to fail but do create
red herring issues elsewhere (see #15666). Fixing these errors
leads to more errors in tests that depended on the original
misconfigured state, so those are also addressed here.

This is an update to #16003 which accounts for some unit tests with
conflicting config/mutable_config fixtures. These conflicts were
not exposed until the mutable_config fixture was fixed. Details are
included below. The change which builds on #16003 is prefixed with
"(new)".

* For tests that use the real Spack package repository, the config
  needs to avoid using MPI providers that are not intended to be
  installed by Spack. Without this, it is possible that Spack tests
  which concretize the MPI virtual will end up trying to use an
  implementation that it shouldn't (e.g. one that is always
  provided externally). See #15666 for an example.
* The mutable_config test fixture was not initializing the scope
  roots to the right directories (so the resulting config was empty).
* The current_host fixture in the concretize.py tests was using the
  config fixture rather than mutable_config, and was polluting the
  config cache for other tests.
* One test in concretize.py was clearing a nonexistent cache
  (PackagePrefs._packages_config_cache). This reference has been
  removed.
* The test 'test_preferred_compilers' was was depending on cross
  test config pollution to succeed. The initial spec before
  concretization has been updated to updated to be explicit about
  the desired result.
* (new) For tests that use install_mockery and mutable_config,
  replace install_mockery with a separate install_mockery_mutable_config
  fixture that is exactly the same as install_mockery but uses the
  mutable_config fixture to avoid conflicts.
2020-05-07 14:01:58 -07:00
Todd Gamblin
b196e4396a
bugfix: spack shouldn't fail in an incomplete environment (#16473)
Fixed #15884.

Spack asks every package linked into an environment to tell us how
environment variables should be modified when a spack environment is
activated. As part of this, specs in an environment are symlinked into
the environment's view (see #13249), and the package calculates
environment modifications with *the default view as the prefix*.

All of this works nicely for pointing the user's environment at the view
*if* every package is successfully linked. Unfortunately, right now we
only track what specs "should" be in a view, not which specs actually
are. So we end up calculating environment modifications on things that
aren't linked into thee view, and the exception isn't caught, so lots of
spack commands end up failing.

This fixes the issue by ignoring and warning about specs where
calculating environment modifications fails. So we can still keep using
Spack even if the current environment is incomplete.

We should probably also just avoid computing env modifications *entirely*
for unlinked packages, but right now that is a slow operation (requires a
lot of YAML parsing). We should revisit that when we have some better
state management for views, but the fix adopted here will still be
necessary, as we want spack commands to be resilient to other types of
bugs in `setup_run_environment()` and friends. That code is in packages
and we have to assume it could be buggy when we call it outside of builds
(as it might fail more than just the build).
2020-05-07 02:30:09 -07:00
Greg Becker
d0220a990c
fix underscore/dash problems for mic_knl and skylake_avx512 (#16491) 2020-05-06 12:10:08 -07:00
Peter Scheibel
b030a81a5f
Automatically find externals (#15158)
Add a `spack external find` command that tries to populate
`packages.yaml` with external packages from the user's `$PATH`. This
focuses on finding build dependencies. Currently, support has only been
added for `cmake`.

For a package to be discoverable with `spack external find`, it must define:
  * an `executables` class attribute containing a list of
    regular expressions that match executable names.
  * a `determine_spec_details(prefix, specs_in_prefix)` method

Spack will call `determine_spec_details()` once for each prefix where
executables are found, passing in the path to the prefix and the path to
all found executables. The package is responsible for invoking the
executables and figuring out what type of installation(s) are in the
prefix, and returning one or more specs (each with version, variants or
whatever else the user decides to include in the spec).

The found specs and prefixes will be added to the user's `packages.yaml`
file. Providing the `--not-buildable` option will mark all generated
entries in `packages.yaml` as `buildable: False`
2020-05-05 17:37:34 -07:00
Greg Becker
dd3762d0f9
cray platform: support cray Cluster and XC type machines (#12989)
Cray has two machine types. "XC" machines are the larger
machines more common in HPC, but "Cluster" machines are
also cropping up at some HPC sites. Cluster machines run
a slightly different form of the CrayPE programming environment,
and often come without default modules loaded. Cluster
machines also run different versions of some software, and run
a linux distro on the backend nodes instead of running Compute 
Node Linux (CNL).

Below are the changes made to support "Cluster" machines in
Spack. Some of these changes are semi-related general upkeep
of the cray platform.

* cray platform: detect properly after module purge

* cray platform: support machines running OSs other than CNL

Make Cray backend OS delegate to LinuxDistro when no cle_release file
favor backend over frontend OS when name clashes

* cray platform: target detection uses multiple strategies

This commit improves the robustness of target
detection on Cray by trying multiple strategies.

The first one that produces results wins. If
nothing is found only the generic family of the
frontend host is used as a target.

* cray-libsci: add package from NERSC

* build_env: unload cray-libsci module when not explicitly needed

cray-libsci is a package in Spack. The cray PrgEnv
modules load it implicitly when we set up the compiler.
We now unload it after setting up the compiler and
only reload it when requested via external package.

* util/module_cmd: more robust module parsing

Cray modules have documentation inside the module
that is visible to the `module show` command.
Spack module parsing is now robust to documentation 
inside modules.

* cce compiler: uses clang flags for versions >= 9.0

* build_env: push CRAY_LD_LIBRARY_PATH into everything

Some Cray modules add paths to CRAY_LD_LIBRARY_PATH
instead of LD_LIBRARY_PATH. This has performance benefits
at load time, but leads to Spack builds not finding their
dependencies from external modules.
Spack now prepends CRAY_LD_LIBRARY_PATH to
LD_LIBRARY_PATH before beginning the build.

* mvapich2: setup cray compilers when on cray

previously, mpich was the only mpi implementation to support
cray systems (because it is the MPI on Cray XC systems). 
Cray cluster systems use mvapich2, which now supports cray
compiler wrappers.

* build_env: clean pkgconf from environment

Cray modules silently add pkgconf to the user environment
This can break builds that do not user pkgconf.
Now we remove it frmo the environment and add it again if it
is in the spec. 

* cray platform: cheat modules for rome/zen2 module on naples/zen node

Cray modules for naples/zen architecture currently specify
rome/zen2. For now, we detect this and return zen for modules
named `craype-x86-rome`.

* compiler: compiler default versions

When detecting compiler default versions for target/compiler
compatibility checks, Spack previously ran the compiler without
setting up its environment. Now we setup a temporary environment
to run the compiler with its modules to detect its version.

* compilers/cce: improve logic to determine C/C++ std flags

* tests: fix existing tests to play nicely with new cray support

* tests: test new functionality

Some new functionality can only be tested on a cray system.
Add tests for what can be tested on a linux system.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-05-05 13:58:46 -07:00
Massimiliano Culpo
aa9c98b6a2
Make usage example consistent with current MV variants (#16468)
Since #9481 Python's None is not permitted as a value for
MV variants. The string 'none' is used instead.

Add the same fix for the amgx and lammps packages
2020-05-05 20:22:32 +02:00
Greg Becker
3ae76d8c1e
bugfix: config:install_hash_length ignored (#15919) 2020-05-04 23:10:25 -07:00
Greg Becker
8e2f5ba861
environments: allow sigils to apply to entire reference (#15245)
* environments: allow sigils to apply to entire reference
2020-05-04 15:11:10 -07:00
Oliver Breitwieser
1cff717ca8 Fix git-related commands not working in worktrees
If spack is checked out in a git worktree (see [1]), all git-related
commands fail because the `spack_is_git_repo()`-check is not thorough
enough.

When developing in a feature-branch in a seperate worktree, this is
annoying as all unittests regarding git-related spack commands fail,
cluttering the test results with false-positives.

[1]: https://git-scm.com/docs/git-worktree

Change-Id: I94b573a2c0e058e9ccc169e7ee6561626fbb06fd
2020-05-04 14:39:25 -07:00
Peter Scheibel
40df44e021
Revert "Testing: fix unintended interactions between tests (#16003)" (#16420)
This was breaking tests on develop.
2020-05-01 13:39:43 -07:00
Peter Scheibel
31d12d380f
Testing: fix unintended interactions between tests (#16003)
* For tests that use the real Spack package repository, the config
  needs to avoid using MPI providers that are not intended to be
  installed by Spack. Without this, it is possible that Spack tests
  which concretize the MPI virtual will end up trying to use an
  implementation that it shouldn't (e.g. one that is always
  provided externally). See #15666 for an example.
* The mutable_config test fixture was not initializing the scope
  roots to the right directories (so the resulting config was empty).
* The current_host fixture in the concretize.py tests was using the
  config fixture rather than mutable_config, and was polluting the
  config cache for other tests.
* One test in concretize.py was clearing a nonexistent cache
  (PackagePrefs._packages_config_cache). This reference has been
  removed.
* The test 'test_preferred_compilers' was was depending on cross
  test config pollution to succeed. The initial spec before
  concretization has been updated to updated to be explicit about
  the desired result.
2020-05-01 10:47:46 -07:00
Axel Huebl
d0dfa1ea4d
dev-build: --drop-in <shell> (#14887)
* dev-build: --drop-in <shell>

Add a `--drop-in <shell>` option to `spack dev-build`.
This option will automatically run a
`spack build-env <spec> -- <shell>` at the end of a `dev-build`, e.g.
to quickly drop-and-devel into a build phase of a package.

Example usage:
```
spack dev-build --before cmake --drop-in bash openpmd-api@develop
```

* build_env: drop in unit test

Co-authored-by: Greg Becker <becker33@llnl.gov>
2020-05-01 09:37:21 -07:00
Adam J. Stewart
022c287e47
Fix typos in new AutotoolsPackage function (#16406) 2020-04-30 15:13:04 -05:00
Chris Green
9698907e74
Spack command extensions: error-handling (#13635)
Generally speaking, errors that are encountered when attempting to load
command extensions now terminate the running Spack instance.

* Added new exceptions `spack.cmd.PythonNameError` and
  `spack.cmd.CommandNameError`.
* New functions `spack.cmd.require_python_name(pname)` and
  `spack.cmd.require_cmd_name(cname)` check that `pname` and `cname`
  respectively meet requirements, throwing the appropriate error if not.
* `spack.cmd.get_module()` uses `require_cmd_name()` and passes through
  exceptions from module load attempts.
* `spack.cmd.get_command()` uses `require_cmd_name()` and invokes
  `get_module()` with the correct command-name form rather than the
  previous (incorrect) Python name.
* Added New exceptions `spack.extensions.CommandNotFoundError` and
  `spack.extensions.ExtensionNamingError`.
* `_extension_regexp` has a new leading underscore to indicate expected
  privacy.
* `spack.extensions.extension_name()` raises an `ExtensionNamingError`
  rather than using `tty.warn()`.
* `spack.extensions.load_command_extension()` checks command source
  existence early and bails out if missing. Also, exceptions raised by
  `load_module_from_file()` are passed through.
* `spack.extensions.get_module()` raises `CommandNotFoundError` as
  appropriate.
* Spack `main()` allows `parser.add_command()` exceptions to cause
  program end.

Tests:

* More common boilerplate has been pulled out into fixtures including
  `sys.modules` dictionary cleanup and resource-managed creation of a
  simple command extension with specified contents in the source file
  for a single named command.
* "Hello, World!" test now uses a command named `hello-world` instead of
  `hello` in order to verify correct handling of commands with hyphens.
* New tests for:
  * Missing (or misnamed) command.
  * Badly-named extension.
  * Verification that errors encountered during import of a command are
    propagated upward.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2020-04-29 18:18:48 -07:00
Nathan
b6b44948b8
prefix gets removed when adding additional configure options (#16335)
* prefix gets removed when adding additional configure options

* Update waf documentation to reflect the new change
2020-04-29 10:52:23 -05:00
Axel Huebl
00d83cd79d
dev-build: stop before phase (#14699)
Add `-b,--before` option to dev-build command to stop before the phase in question.
2020-04-28 09:55:57 -07:00
Axel Huebl
7670ae468f
Docs: Limit Microarch on Travis CI (#16200)
Update the travis CI docs: limiting the microarch is essential
for caching.
2020-04-28 11:54:02 -05:00
Massimiliano Culpo
a540e36922
relocate: remove an assertion in _make_relative (#16351)
This was an oversight in #15654 since `os.path.isfile`
checks that the something is a file in the current
filesystem.
2020-04-28 14:53:22 +02:00
Massimiliano Culpo
6b648c6e89
Improve the coverage of spack.relocate (#15654)
This PR introduces trivial refactoring in:
- `get_existing_elf_rpaths`
- `get_relative_elf_rpaths`
- `get_normalized_elf_rpaths`
- `set_placeholder`

mainly to be more consistent with practices used in other 
parts of the code and to simplify functions locally. It also 
adds or reworks unit tests for these functions and extends 
their docstrings.

Co-authored-by: Patrick Gartung <gartung@fnal.gov>
Co-authored-by: Peter J. Scheibel <scheibel1@llnl.gov>
2020-04-27 12:17:32 +02:00
Axel Huebl
5828a8f7e6
spack env: activate note (#16279)
print a note on how to activate a newly created environment
2020-04-24 13:24:52 -07:00
Todd Gamblin
c6ada206af
tests: each mock package now has its own class (#16157)
Packages in Spack are classes, and we need to be able to execute class
methods on mock packages.  The previous design used instances of a single
MockPackage class; this version gives each package its own class that can
spider depenencies.  This allows us to implement class methods like
`possible_dependencies()` on mock packages.

This design change moves mock package creation into the
`MockPackageMultiRepo`, and mock packages now *must* be created from a
repo.  This is required for us to mock `possible_dependencies()`, which
needs to be able to get dependency packages from the package repo.

Changes include:

* `MockPackage` is now `MockPackageBase`
* `MockPackageBase` instances must now be created with
  `MockPackageMultiRepo.add_package()`
* add `possible_dependencies()` method to `MockPackageBase`
* refactor tests to use new code structure
* move package mocking infrastructure into `spack.util.mock_package`,
  as it's becoming a more sophisticated class and it gets lots in `conftest.py`
2020-04-23 18:21:49 -07:00
Todd Gamblin
0c0f11caf6
spack info: allow variants section to be as wide as the terminal (#16254)
The variants table in `spack info` is cramped, as the *widest* it can be
is 80 columns.  And that's actually only sort of true -- the padding
calculation is off, so it still wraps on terminals of size 80 because it
comes out *slightly* wider.

This change looks at the terminal size and calculates the width of the
description column based on it.  On larger terminals, the output looks
much nicer, and on small terminals, the output no longer wraps.

Here's an example  for `spack info qmcpack` with 110 columns.

Before:
    Name [Default]          Allowed values          Description
    ====================    ====================    ==============================

    afqmc [off]             on, off                 Install with AFQMC support.
                                                    NOTE that if used in
                                                    combination with CUDA, only
                                                    AFQMC will have CUDA.
    build_type [Release]    Debug, Release,         The build type to build
                            RelWithDebInfo
    complex [off]           on, off                 Build the complex (general
                                                    twist/k-point) version
    cuda [off]              on, off                 Build with CUDA

After:
    Name [Default]          Allowed values          Description
    ====================    ====================    ========================================================

    afqmc [off]             on, off                 Install with AFQMC support. NOTE that if used in
                                                    combination with CUDA, only AFQMC will have CUDA.
    build_type [Release]    Debug, Release,         The build type to build
                            RelWithDebInfo
    complex [off]           on, off                 Build the complex (general twist/k-point) version
    cuda [off]              on, off                 Build with CUDA
2020-04-23 12:14:40 -07:00
Massimiliano Culpo
6f9373dd66
spack info: replace "True, False" with "on, off" (#16235)
fixes #16184
2020-04-22 17:48:52 -07:00
Greg Becker
ec23e4ffe5
update compiler config with bootstrapped compiler when already installed (#16221)
Update compiler config with bootstrapped compiler when it was already installed and added config defaults to code so mutable_config test fixture works.
2020-04-22 14:26:29 -07:00
George Hartzell
63ff121980
spack environments documentation: -E -> -e (#16242)
To specify an environment for a comment, the user can specify
"spack -e <env>". The documentation incorrectly specified "-E" (which
is actually used to ignore any implicit use of environments).
2020-04-22 13:04:17 -07:00
Axel Huebl
ca4c0b97ec
Doc: Conflict Msg Pkg Guide (#16201)
Document the nice `msg=` argument in conflicts. Super useful
for less generic conflict guidance.
2020-04-21 16:22:30 -05:00
Andrew W Elble
f1050c4be2
compiler wrappers: don't override -isystem with -I (#16077)
If the Spack compiler wrapper encounters any "-isystem" option, then
when adding include directories for Spack dependencies, Spack will
use "-isystem" instead of "-I". This prevents Spack-generated "-I"
options from overriding the "-isystem" options generated by the build
system. To ensure that build-system "-isystem" directories are
searched first, Spack places all of its inserted "-isystem"
directories after.

The new ordering of -isystem includes is:

* -isystem from build system (not system directories)
* -isystem from Spack
* -isystem from build system (for directories like /usr/include)

The prior order of "-I" arguments is preserved (although as of this
commit Spack no longer generates -I if -isystem is detected):

* -I from build system (not system directories)
* -I from Spack (only if there are no "-isystem" options)
* -I from build system (for directories like /usr/include)
2020-04-21 13:56:50 -07:00
Todd Gamblin
de13ea2284
autotools: remove FORCE_UNSAFE_CONFIGURE from docs, commands, code (#16145)
Since #16132, we've consolidated the setting of FORCE_UNSAFE_CONFIGURE to
`autotools.py`, so we don't need to use it in packages like `coreutils`,
in our commands, or in our container recipes.

- [x] Remove FORCE_UNSAFE_CONFIGURE from packages
- [x] Remove FORCE_UNSAFE_CONFIGURE from container recipes
- [x] Remove FORCE_UNSAFE_CONFIGURE from `spack ci` command
2020-04-20 08:32:03 -07:00
George Hartzell
c62f18304d
Typo: vew -> view (#16155) 2020-04-19 20:13:44 -05:00
Diego Magdaleno
d640bf899a
autotools: set FORCE_UNSAFE_CONFIGURE to 1 in autotools.py (#16132)
This commit sets the `FORCE_UNSAFE_CONFIGURE` environment variable to 1 in autotools builds.

We see a lot of builds popping up and complaining about `FORCE_UNSAFE_CONFIGURE`.  This behavior is not actually part of `autoconf` per se.  It comes from this patch to `mknod.m4`, which is used by a lot of autoconf builds:

    * https://lists.gnu.org/archive/html/bug-gnulib/2010-07/msg00282.html

Which originated from this problem that someone had on AIX:

    * https://lists.gnu.org/archive/html/bug-gnulib/2010-07/msg00279.html

The gist of the problem seems to be that they want to check whether `mknod` can do something as root, but instead of checking whether they're running as root and using `su` or something to test this, they just made it harder to run `configure` as root.

This seems very ad hoc and this is one of many checks that are run as root in `configure`.  Many of them run before this check, so it's not clear that the `FORCE_UNSAFE_CONFIGURE` thing is even preventing bad things from happening.

So:
1. This only happens in `autotools` builds, so we should go ahead and put it into `autotools.py` instead of in the global build environment, and

2. The variable does too little and provides a false sense of security in the first place, so we'll just disable it and avoid the nuisance.  If we really feel strongly about this we can put some warnings in Spack about running as root, but at the top level, not in the middle of an already running script like `configure`.
2020-04-18 14:25:19 -07:00
Todd Gamblin
eaccc58156
remove files accidentally committed with 0.14.0 (#16138) 2020-04-17 22:28:33 -07:00
Adam J. Stewart
ea8793f8f5
AutotoolsPackage: add aclocal for all build deps (#15784) 2020-04-17 16:30:21 -05:00
Adam J. Stewart
284e450c91
Language-specific PIC flags (#15474)
* Language-specific PIC flags

* Add tests for every compiler flag

* Fix bad rebase

* pic_flag -> cxx_pic_flag
2020-04-17 16:28:51 -05:00
Axel Huebl
abbc47823d
SourceForge: Mirror Mixin (#16112)
* SourceForge: Mirror Mixin

Add a mixing class for direct `CNAME`s to sourceforge mirrors.
Since the main gateway servers are often down, this could reduce
timeouts and fetch errors for sourceforge.net hosted software.

* SourceForge: unspectacular mirror replacement

add mirrors to all sourceforge packages with trivial
download logic.

tested fetch of latest version of each of these packages
with various mirrors before committing.

* SourceForge: xz

the author homepage is chronocially overrun and this is the offical
upload with many mirrors.
2020-04-16 21:35:30 -07:00
Todd Gamblin
f6d26db939 macos: use DYLD_FALLBACK_LIBRARY_PATH instead of DYLD_LIBRARY_PATH
`DYLD_LIBRARY_PATH` can frequently break builtin macOS software when
pointed at Spack libraries.  This is because it takes *higher* precedence
than the default library search paths, which are used by system software.

`DYLD_FALLBACK_LIBRARY_PATH`, on the other hand, takes lower precedence.
At first glance, this might seem bad, because the software installed by
Spack in an environment needs to find *its* libraries, and it should not
use the defaults.  However, Spack's isntallations are always `RPATH`'d,
so they do not have this problem.

`DYLD_FALLBACK_LIBRARY_PATH` is thus useful for things built in an
environment that need to use Spack's libraries, that don't set *their*
RPATHs correctly for whatever reason. We now prefer it to
`DYLD_LIBRARY_PATH` in modules and in environments because it helps a
little bit, and it is much less intrusive.
2020-04-16 17:23:33 -07:00
Marc Allen
a704922c29
Bugfix: For spack buildcache check, Handle specs where only the hash is (#15663)
provided (#15662).

Prior to this fix, the checked Spec object would not be populated, and
concretization would fail.

Co-authored-by: Marc Allen <mrcall@amazon.com>
2020-04-16 14:06:13 -07:00
Todd Gamblin
bd3f24afe1 Merge branch 'releases/v0.14' into develop 2020-04-15 15:27:00 -07:00
Michael Kuhn
277350ceed Make tags case insensitive
Currently, tags are case sensitive, which is unintuitive:
```console
$ spack list -t hpc
==> 2 packages.
nek5000  nektools
$ spack list -t HPC
==> 1 packages.
mfem
$ spack list -t Hpc
==> 0 packages.
```

This change makes them case insensitive:
```console
$ spack list -t hpc
==> 3 packages.
mfem  nek5000  nektools
$ spack list -t HPC
==> 3 packages.
mfem  nek5000  nektools
$ spack list -t Hpc
==> 3 packages.
mfem  nek5000  nektools
```
2020-04-15 14:56:25 -07:00
Todd Gamblin
a3bcd88f8d version bump: 0.14.2 2020-04-15 14:30:58 -07:00
Todd Gamblin
740f8fe1a9 bugfix: spack test should not output [+] for mock installs (#15609)
`spack test` has a spurious '[+] ' in the output:

```
lib/spack/spack/test/install.py .........[+] ......
```

Output is properly suppressed:

```
lib/spack/spack/test/install.py ...............
```
2020-04-15 14:30:58 -07:00
Todd Gamblin
430ca7c7cf spack install terminal output handling in foreground/background (#15723)
Makes the following changes:

* (Fixes #15620) tty configuration was failing when stdout was 
  redirected. The implementation now creates a pseudo terminal for
  stdin and checks stdout properly, so redirections of stdin/out/err
  should be handled now.
* Handles terminal configuration when the Spack process moves between
  the foreground and background (possibly multiple times) during a
  build.
* Spack adjusts terminal settings to allow users to to enable/disable
  build process output to the terminal using a "v" toggle, abnormal
  exit cases (like CTRL-C) could leave the terminal in an unusable
  state. This is addressed here with a special-case handler which
  restores terminal settings.

Significantly extend testing of process output logger:

* New PseudoShell object for setting up a master and child process
  and configuring file descriptor inheritance between the two
* Tests for "v" verbosity toggle making use of the added PseudoShell
  object
* Added `uniq` function which takes a list of elements and replaces
  any consecutive sequence of duplicate elements with a single
  instance (e.g. "112211" -> "121")

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-04-15 12:47:41 -07:00
Massimiliano Culpo
55f5afaf3c database: maintain in-memory consistency on remove (#15777)
The performance improvements done in #14693 where leaving the DB in an inconsistent state when specs were removed from it. This PR updates the DB internal state whenever the DB is written to a file.

Note that we still cannot properly enumerate installed dependents, so there is a TODO in this code. Fixing that will require the dependents dictionaries in specs to be re-keyed (either by hash, or not keyed at all -- a list would do).  See #11983 for details.
2020-04-15 12:47:16 -07:00
Andrew W Elble
6b559912c1 performance: add a verification file to the database (#14693)
Reading the database repeatedly can be quite slow.  We need a way to speed
up Spack when it reads the DB multiple times, but the DB has not been
modified between reads (which is nearly all the time).

- [x] Add a file containing a unique uuid that is regenerated at database
    write time. Use this uuid to suppress re-parsing the database
    contents if we know a previous uuid and the uuid has not changed.

- [x] Fix mutable_database fixture so that it resets the last seen
    verifier when it resets.

- [x] Enable not rereading the database immediately after a write. Make
    the tests reset the last seen verifier in between tests that use the
    database fixture.

- [x] make presence of uuid module optional
2020-04-15 12:47:00 -07:00
Peter Scheibel
9b5805a5cd Remove DB conversion of old index.yaml (#15298)
Removed the code that was converting the old index.yaml format into
index.json. Since the change happened in #2189 it should be
considered safe to drop this (untested) code.
2020-04-15 12:45:57 -07:00
Peter Scheibel
b9688a8c35 Environments/views: only override spec prefix for non-external packages (#15475)
* only override spec prefix for non-external packages

* add test that environment shell modifications respect explicitly-specified prefixes for external packages

* add clarifying comment
2020-04-15 12:37:37 -07:00
Jonathon Anderson
ed2781973c Source devnull in environment_after_sourcing_files (closes #15775) (#15791)
spack.util.environment_after_sourcing_files compares the local
environment against a shell environment after having sourced a
file; but this ends up including the default shell profile and
rc, which might differ from the local environment.

To change this, compare against the default shell environment,
expressed here as 'source /dev/null'.
2020-04-15 12:37:16 -07:00
Todd Gamblin
99bb88aead bugfix: TERM may not be in the environment on Cray (#15630) 2020-04-15 12:37:03 -07:00
Massimiliano Culpo
a85cce05a1 Blacklist Lmod variable modifications when sourcing files (#15778)
fixes #15775

Add all the variables listed here:

https://lmod.readthedocs.io/en/latest/090_configuring_lmod.html

to the list of those blacklisted when constructing environment
modifications by sourcing files.
2020-04-15 12:36:53 -07:00
Axel Huebl
e1e804168a Tests: Mirror Mixin Classes 2020-04-15 11:44:14 -07:00
Axel Huebl
7e5257e44a x.org: mirror mixin
According to my nightly CI/CD tests, x.org is another large provider
of software in common build chains that is often down.

Added a hand-selected amount of mirrors that is well up-to-sync.
Tested with `util-macros` that has a quite "recent" patch release.

Other packages to follow in an individual PR.
2020-04-15 11:44:14 -07:00
Todd Gamblin
a563884af3
spack install terminal output handling in foreground/background (#15723)
Makes the following changes:

* (Fixes #15620) tty configuration was failing when stdout was 
  redirected. The implementation now creates a pseudo terminal for
  stdin and checks stdout properly, so redirections of stdin/out/err
  should be handled now.
* Handles terminal configuration when the Spack process moves between
  the foreground and background (possibly multiple times) during a
  build.
* Spack adjusts terminal settings to allow users to to enable/disable
  build process output to the terminal using a "v" toggle, abnormal
  exit cases (like CTRL-C) could leave the terminal in an unusable
  state. This is addressed here with a special-case handler which
  restores terminal settings.

Significantly extend testing of process output logger:

* New PseudoShell object for setting up a master and child process
  and configuring file descriptor inheritance between the two
* Tests for "v" verbosity toggle making use of the added PseudoShell
  object
* Added `uniq` function which takes a list of elements and replaces
  any consecutive sequence of duplicate elements with a single
  instance (e.g. "112211" -> "121")

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2020-04-15 11:05:41 -07:00
Massimiliano Culpo
a11de1d29d
Package extensions: fixed a link in docs (#16040)
* Moved link to the right place in the docs

* Fixed a few minor issues in extensions docs

Fixed a typo, added a subsubsection for better
navigation, reworded "modules in Python" as
"Python packages"
2020-04-14 12:54:53 -05:00
Axel Huebl
5acea35e82
sourceware.org: mirror urls (#15992)
sourceware.org is often quite overrun and times out or results in
certificate errors.

Since libffi, bzip2, elfutils, etc. are quite fundamental in
build chains, lets add some official mirrors.

libffi, bzip2, elfutils, lvm2, valgrind: add mirrors
2020-04-14 11:09:30 -05:00
Massimiliano Culpo
7e46da73dd
database: maintain in-memory consistency on remove (#15777)
The performance improvements done in #14693 where leaving the DB in an inconsistent state when specs were removed from it. This PR updates the DB internal state whenever the DB is written to a file.

Note that we still cannot properly enumerate installed dependents, so there is a TODO in this code. Fixing that will require the dependents dictionaries in specs to be re-keyed (either by hash, or not keyed at all -- a list would do).  See #11983 for details.
2020-04-12 13:14:59 -07:00
Tomoki, Karatsu
08a491b062
fujitsu compiler: Delete custom environment function. (#15988) 2020-04-10 09:38:02 -05:00
Peter Scheibel
75640f0ed9
NAG compiler fix: skip implicit RPATH detection (#15902)
* Skip collection of compiler link paths if compiler does not define a verbose flag

* modules config bug: allow user to configure a compiler without an explicit entry for loaded modules
2020-04-09 10:48:29 +02:00
Scott Wittenburg
de95a9f5f3 pipelines: Document dynamic generation limitations 2020-04-07 15:20:01 -06:00
LPettey-Arm
da9ba2b254
Add build number to Arm compiler version identification (#15809)
* Add capability for detecting build number for Arm compilers

* Fixing fleck8 errors and updating test_arm_version_detection function for more detailed Arm compielr version detection

* Ran flake8 locally and corrected errors

* Altering Arm compielr version check to remove else clause and be more consistent with other compielr version checks. Added test case so both the 'if' and 'else' conditionals of the Arm compiler version check have a test case

Co-authored-by: EC2 Default User <ec2-user@ip-172-31-7-135.us-east-2.compute.internal>
2020-04-07 10:58:54 -05:00
Adam J. Stewart
fab1f68399
Docs: fix Spack Environments command (#15891) 2020-04-06 11:54:22 -05:00
Massimiliano Culpo
06f3381dfd
Blacklist Lmod variable modifications when sourcing files (#15778)
fixes #15775

Add all the variables listed here:

https://lmod.readthedocs.io/en/latest/090_configuring_lmod.html

to the list of those blacklisted when constructing environment
modifications by sourcing files.
2020-04-06 10:02:03 +02:00
Jonathon Anderson
359d7adf7e
Source devnull in environment_after_sourcing_files (closes #15775) (#15791)
spack.util.environment_after_sourcing_files compares the local
environment against a shell environment after having sourced a
file; but this ends up including the default shell profile and
rc, which might differ from the local environment.

To change this, compare against the default shell environment,
expressed here as 'source /dev/null'.
2020-04-06 10:00:42 +02:00
Stephen McDowell
51fbe518b8
verify other.exe attr exists before use in __eq__ (#15882) 2020-04-05 14:08:13 -05:00
G-Ragghianti
bae4f91bfe
Add option "--first" for "spack load" (#15622)
* Implemented --first option for "spack load"

* added test for "spack load --first"

Co-authored-by: gragghia <gragghia@localhost.localdomain>
2020-04-03 13:33:20 -07:00
Peter Scheibel
0b99cc2261
Environments/views: only override spec prefix for non-external packages (#15475)
* only override spec prefix for non-external packages

* add test that environment shell modifications respect explicitly-specified prefixes for external packages

* add clarifying comment
2020-04-03 13:26:33 -07:00
Adam J. Stewart
75a2f8046a
Add commands to facilitate Spack/Python/OS reporting (#15834)
* Add --version arg to spack python command
* Add `spack debug report` command
2020-04-02 23:12:03 -07:00