Commit graph

3401 commits

Author SHA1 Message Date
Michael Kuhn
a7e915d38b Fix build_jobs exceeding number of cores
All documentation mentions that `build_jobs` is limited by the number of
cores available in the system. This is also enforced when setting it via
`--jobs`. However, when setting it via `config.yaml`, it can exceed the
number of cores available, making builds run out of memory.
2019-07-15 11:40:49 -07:00
Glenn Johnson
3f83a2a7d8 Add auto-dispatch specification to Intel packages (#11697)
This PR adds the ability to specify the auto-dispatch targets that can
be used by the Intel compilers. The `-ax` flag will be written to the
respective compiler configuration files. This ability is very handy when
wanting to build optimized builds for various architectures. This PR
does not set any optimization flags, however.
2019-07-15 11:37:54 -07:00
Massimiliano Culpo
5acbe449e5 spack uninstall can uninstall specs with multiple roots (#11977)
Fixes #3690
Fixes #5637

Uninstalling dependents of a spec was relying on a traversal of the
parents done by inspecting spec._dependents. This is in turn a
DependencyMap that maps a package name to a single DependencySpec object
(an edge in the DAG) and cannot thus model the case where a spec has
multiple configurations of the same parent package installed (for
example if different versions of the same Python library depend on
the same Python installation).

This commit works around this issue by constructing the list of specs to
be uninstalled in an alternative way, and adds tests to verify the
behavior. The core issue with DependencyMap is not resolved here.
2019-07-15 10:30:01 -07:00
Tim Fuller
5bc15b2d9a find_libraries searches lib and lib64 before prefix (#11958)
The default library search for a package checks the lib/ and lib64/
directories for libraries before the root prefix, in order to save
time when searching for libraries provided by externals (which e.g.
may have '/usr/' as their root).

This moves that logic into the "find_libraries" utility method so
packages implementing their own custom library search logic can
benefit from it.

This also updates packages which appear to be replicating this logic
exactly, replacing it with a single call to "find_libraries".
2019-07-12 17:46:47 -07:00
Oliver Breitwieser
d1af7ae5ab Copy --{use,no}-cache from install to bootstrap (#11786) 2019-07-12 16:30:12 -07:00
Carson Woods
76f1ee5f32 'spack compiler add' resolves relative path to absolute path (#11792)
Fixes #11782

Spack was not properly resolving relative paths to absolute paths
when a relative path was passed to "spack compiler add [PATH]".
Now, if provided a relative path, the absolute path is written to
compilers.yaml rather than the relative path.
2019-07-12 16:06:26 -07:00
Tamara Dahlgren
d615d0a1ec Add template creation test, --skip-editor option (#11970)
* Add template creation test
* Added --skip-editor option to "spack create": normally
  "spack create" opens an editor for the user after generating a
  package file; when the --skip-editor option is used, "spack create"
  only generates the package file and does not open an editor
* Added --skip-editor option to bash completion
2019-07-12 11:22:09 -07:00
t-karatsu
adbb9e2d8e Fix pic_flag and test contents about Fujitsu compiler. (#11996) 2019-07-12 10:34:08 -05:00
Todd Gamblin
91e35541ee
bugfix: show dot after namespaces in spack find -N (#11945)
- Namepsaces were shown without dots after the new format strings were
  added.

- Add a test for `spack find` to ensure that find -N shows the right
  output.
2019-07-12 08:32:47 -07:00
Tamara Dahlgren
05b6a5e53d test: Extra possible-dependencies coverage check (#11988) 2019-07-11 17:15:21 -07:00
Tamara Dahlgren
e3299e6923 Rename build logs and make names consistent (#11806)
Fixes #11781

* Rename build log to spack-build-log.txt
* Rename environment variables file to spack-build-env.txt
* The name of the log and env files is now the same during the build
  and after the build completes
* Update packages which referred to the build log/env files
* For packages installed before this commit using older names for the
  build and env files, search for the older names
2019-07-11 13:32:06 -07:00
Todd Gamblin
c2de2558b6
tutorial: correct image link on tutorial page 2019-07-10 00:56:25 -07:00
Todd Gamblin
0cd64d7b6e
tutorial: updates slides for DOE/NSF workshop. 2019-07-10 00:50:59 -07:00
Todd Gamblin
51b58f1478
bugfix: conflicts should print out the spec that has the conflict (#11947)
- Fix a bug introdcued by removing parse_anonymous_spec()

- Conflicts' when specs are now *actually* anonymous, and the name of the
  package is implicit, so we need to remember to add it back to error
  messages.
2019-07-08 08:00:43 -07:00
Todd Gamblin
d2e5b8474a
bugfix: always generate a stack trace when spack is run with --debug (#11940)
- We weren't previously printing stack traces on SystemExit or
  KeyboardInterrupts.

- Either raise or print the stacktrace in these cases.
2019-07-05 19:57:00 -07:00
Todd Gamblin
515b4045e9 specs: remove parse_anonymous_spec(); use Spec() instead
- `parse_anonymous_spec()` is a vestige of the days when Spack didn't
  support nameless specs.  We don't need it anymore because now we can
  write Spec() for a spec that will match anything, and satisfies()
  semantics work properly for anonymous specs.

- Delete `parse_anonymous_spec()` and replace its uses with simple calls
  to the Spec() constructor.

- make then handling of when='...' specs in directives more consistent.

- clean up Spec.__contains__()

- refactor directives and tests slightly to accommodate the change.
2019-07-01 18:45:44 -07:00
Todd Gamblin
12b9fad7b6 cray: use the cle-release file to determine CNL version
- CNL OS previously used the *Cray PE* version to determine the OS
  version.  Cray does not synchronize PE and CLE releases; you can run
  CLE7 with PrgEnv 6 (and NERSC currently does).

- Fix Spack's OS detection by using the cle-release file to detect the OS
  version.  This file is updated with every CLE OS release.

- Add some tests for our parsing logic
2019-07-01 18:36:02 -07:00
Patrick Gartung
9bf9c0cec3
buildcache: add replace_prefix_bin function to deal with hard coded paths in libraries and executables. (#11882) 2019-07-01 10:16:23 -05:00
Todd Gamblin
4189640ee8 style: remove unnecessary NOQA exceptions 2019-06-30 16:01:23 -07:00
Peter Scheibel
9c16b4a7f6 Allow uninstalling missing packages (#11874)
Remove package access from directory_layout; add regression test to ensure
that specs can be uninstalled without a package being known
2019-06-29 16:04:15 -07:00
Mark W. Krentel
8aa8b50f91 docs: add example for an external module in packages.yaml
Add an example of a 'modules:' entry for an external package in
packages.yaml.  The 'External Packages' section of 'Build
Customization' mentions 'paths:' and 'modules:' and gives an
example of paths, but not modules.
2019-06-28 11:59:10 -07:00
Toyohisa Kameyama
b185f87c55 Add --frontend and --backend option to spack arch command (#11746) 2019-06-26 08:19:46 -07:00
Patrick Gartung
30ce818fc4
Fix for #10063 : install from source if buildcache with different DAG exist (#11834)
* Fix for #10063 : install from source if buildcache with different DAG exist

* Flake8
2019-06-24 21:09:24 -05:00
Peter Scheibel
1ae03b327c
Use Stage.archive_file to access non-expanded download (#11817)
Fixes #11816

Allow packages to refer to non-expanded downloads (e.g. a single
script) using Stage.archive_file. This addresses a regression from
#11688 and adds a unit test for it.
2019-06-21 16:56:38 -07:00
Adam J. Stewart
9b8215ce9d Ignore pgcc-llvm and friends, default to pgcc (#11703)
This change reverts to the previous behavior of only looking for pgcc
and friends, not pgcc-llvm and friends.

The llvm variant doesn't support all the same features as the
traditional variant of the pgi code generator; this change avoids
treating the llvm variant as a default pgi compiler.

This retains the changes in #10704 which accept the "LLVM" suffix of
the version string of the PGI compiler, which allows users to
explicitly add the llvm-pgi compiler if desired.
2019-06-20 19:07:29 -07:00
Peter Scheibel
284ae9d1cc
Resources: use expanded archive name by default (#11688)
For resources, it is desirable to use the expanded archive name of
the resource as the name of the directory when adding it to the root
staging area.

#11528 established 'spack-src' as the universal directory where
source files are placed, which also affected the behavior of
resources managed with Stages.

This adds a new property ('srcdir') to Stage to remember the name of
the expanded source directory, and uses this as the default name when
placing a resource directory in the root staging area.

This also:

* Ensures that downloaded sources are archived using the expanded
  archive name (otherwise Spack will not be able to determine the
  original directory name when using a cached archive).
* Updates working_dir context manager to guarantee restoration of
  original working directory when an exception occurs
* Adds a "temp_cwd" context manager which creates a temporary
  directory and sets it as the working directory
2019-06-20 11:09:31 -07:00
Gregory Lee
8706ac1479 fixed spec dependence attribute writing. Fixes #11686 (#11776) 2019-06-19 04:13:12 +02:00
Greg Becker
9a4917644a Remove test dependency on /usr/bin/gcc (#11769)
The regression test for #11678 fails on at least some Mac OS systems
because they have a /usr/bin/gcc that is secretly clang.

This PR replaces the dependency on a system gcc executable with a
test-generated script that generates the expected output for the
compiler logic.
2019-06-18 18:23:02 -07:00
Elizabeth Fischer
f886c7e59d Rename build.out to build.txt (#11749)
Rename build.out to build.txt; makes it easier when people post log files to the Spack mailing list.
2019-06-18 10:32:00 -07:00
Tamara Dahlgren
6e067a05c7 bugfix: use config fixture for stage tests (#11701)
Some tests introduced in #11528 temporarily set the user's `config:build_stage`, which affected (or created) a config.yaml file in the user's `$HOME/.spack` directory that could leave entries behind if the tests fail.

This change ensures only temporary configuration files are used/affected by these tests.
2019-06-17 11:38:34 +02:00
Tamara Dahlgren
a551e4329e docs: updates for stage.source_path (#11702) 2019-06-17 11:36:22 +02:00
Todd Gamblin
45c65a255a
docs: update tutorial page for ISC19 tutorial (#11742) 2019-06-16 12:35:12 +02:00
Michael Kuhn
f3671244e4 docs: fix config tutorial (#11715)
This fixes a few typos, adds a hint to `spack config blame` and brings
the variant specification in line with other docs sections.
2019-06-15 21:34:32 +02:00
Tamara Dahlgren
1b8f641016
Ensure mock configuration directory is cleaned up on session exit. (#11700) 2019-06-13 16:41:00 -07:00
Tamara Dahlgren
25b21c093a Add tests for "spack location" command (#11661)
The "spack location" command was previously untested. This also adds
a check to ensure that composite Stages can report whether they were
expanded (this property was previously only recorded in Stage but not
in CompositeStage).
2019-06-13 12:20:13 -07:00
Tamara Dahlgren
5cf62e69ce Always treat DIYStage as expanded (#11663)
DIYStage, used to treat a user-managed directory as a staging area,
should always be considered expanded (i.e. the source has been
decompressed if it was stored in an archive).

This also:

* Adds checks to ensure that the path used to instantiate a
  DIYStage refers to an existing directory.
* Adds tests to check the behavior of DIYStage (including behavior
  added here, but it was generally untested before).
2019-06-13 11:14:35 -07:00
Tamara Dahlgren
16530f84be Update remaining packages to use Stage.source_path (#11662)
#11528 updated Stage to always store a Package's source in a fixed
directory accessible via `Stage.source_path` This left behind a
number of packages which were expecting to access the source code
via `Stage.path`. This Updates those packages to use
`Stage.source_path` instead.

This also updates the name of the fixed directory: The original name
of the fixed directory was "src", so if an expanded archive created a
"src" directory, then users inspecting the directory structure could
see paths like "src/src" (which wasn't wrong but could be confusing).
Therefore this also updates the name of the fixed directory to
"spack-src".
2019-06-12 17:07:47 -07:00
Massimiliano Culpo
3ce90741a3 Make "spack compiler find" check PATH by default (#11683)
Fixes #11678

`spack compiler find` was not searching `PATH` when provided with no
arguments. ea7910a updated the API for the search function and the
command logic did not update how it called this function. This also
adds a test to ensure that `spack compiler find` will collect
compilers from `PATH`.
2019-06-12 15:28:16 -07:00
Adam J. Stewart
4e812090c0
Add additional common C++ and Fortran header file extensions (#11600)
* Add additional common C++ and Fortran header file extensions

* Add .hxx extension

* Add .txx and .tcc extensions

* Add .icc extension
2019-06-11 20:13:55 -04:00
Chuck Atkins
91205545f0 Add extra logic for C std flags on PGI, XL, and Cray (#11635) 2019-06-11 14:15:55 +02:00
Peter Scheibel
406c791b88
Fix recursive module find for upstream dependencies (#11304)
"spack module tcl find -r <spec>" (and equivalents for other module
systems) was failing when a dependency was installed in an upstream
Spack instance. This updates the module index to handle locating
module files for upstream Spack installations (encapsulating the
logic in a new class called UpstreamModuleIndex); the updated index
handles the case where a Spack installation has multiple upstream
instances.

Note that if a module is not available locally but we are using the
local package, then we shouldn't use a module (i.e. if the package is
also installed upstream, and there is a module file for it, Spack
should not use that module). Likewise, if we are instance X using
upstreams Y and Z like X->Y->Z, and if we are using a package from
instance Y, then we should only use a module from instance Y. This
commit includes tests to check that this is handled properly.
2019-06-10 16:56:11 -07:00
Peter Scheibel
f31711b84e concretization: don't apply build-dep constraints for installed packages (#11594)
Spack currently tries to unify everything in the DAG, but this is too strict for build dependencies, where it is fine to build a dependency with a tool that conflicts with a version fo that tool for a dependent's build.

To enable a workaround for conflicts among build dependencies, so that users can install in multiple steps to avoid these conflicts, make the following changes:

* Dont apply package dependency constraints for build deps of installed packages
* Avoid applying constraints for installed packages vs. concrete packages
* Mark all dependencies of installed packages as visited in normalization method
* don't remove dependency links for concrete specs in flat_dependencies

Also add tests:
* Update test to ensure that link dependencies of installed packages have constraints applied
* Add test to check for proper handling of transitive dependencies (which is currently not the case)
2019-06-08 11:43:26 -07:00
Axel Huebl
61333dc606 spack list: latest version (JSON) (#11652)
List the latest version of each package in JSON encoding.
Preparation for consumption for a "spack badge" service.
2019-06-07 14:14:08 -07:00
Massimiliano Culpo
6d56d45454 Compiler search uses a pool of workers (#10190)
- spack.compilers.find_compilers now uses a multiprocess.pool.ThreadPool to execute
  system commands for the detection of compiler versions.

- A few memoized functions have been introduced to avoid poking the filesystem multiple
  times for the same results.

- Performance is much improved, and Spack no longer fork-bombs the system when doing a `compiler find`
2019-06-07 09:57:26 -07:00
Todd Gamblin
06cc799fd3
commands: remove unused spack list --format=rst (#11651)
- We use `spack list --foramt=html` now, as it is much faster and doesn't
  make the docs build take forever.

- Remove `spack list --format=rst` as it is no longer used.
2019-06-06 18:26:43 -07:00
Todd Gamblin
0c13c3f91f
bugfix: StageComposite must include the expanded property` (#11647) 2019-06-06 14:09:19 -07:00
Jennifer Herting
541578a456 tests: package_sanity now checks packages in 3rd-party repos (#11637) 2019-06-06 09:18:43 -07:00
Tamara Dahlgren
b76fc827ec tests: add extra coverage for fetch strategy tests 2019-06-05 22:41:28 -07:00
Tamara Dahlgren
1842873f85 stage: make source_path available before stage is built
- `stage.source_path` was previously overloaded; it returned `None` if it
  didn't exist and this was used by client code
  - we want to be able to know the `source_path` before it's created

- make stage.source_path available before it exists.
  - use a well-known stage source path name, `$stage_path/src` that is
    available when `Stage` is instantiated but does not exist until it's
    "expanded"
  - client code can now use the variable before the stage is created.
  - client code can test whether the tarball is expanded by using the new
    `stage.expanded` property instead of testing whether `source_path` is
    `None`

- add tests for the new source_path semantics
2019-06-05 22:41:28 -07:00
Tamara Dahlgren
eb584d895b refactor: remove unused spack.stage._get_mirrors() function 2019-06-05 22:41:28 -07:00
Tamara Dahlgren
8e3fd3f7c2 tty: make tty.* print exception types
- make tty.msg, tty.info, etc. print the exception type and stringified
  message if the message argument is an exception.

- simplify parts of the code that call tty.debug(str(e))

- add extra tty.debug statements in places where exceptions were
  previously ignored
2019-06-05 22:41:28 -07:00
Justin S
f4e7786786 clang: fix error messages in c11_flag, cxx17_flag (#11632) 2019-06-05 23:51:38 +02:00
Todd Gamblin
dc8af3023e graph: refactor static graphs
- `spack graph --static` (and `spack.graph.dot_graph`) now do the "right
  thing" and print the possible dependency graph of provided packages.

- `spack graph --static` no longer concretizes specs, as it only relies
  on class level metadata

- Previously the behavior was not consistent -- `spack graph --static`
  would graph possible dependencies of concrete specs, but would only
  include some of them.  The new code properly pursues all possible
  dependencies, and allows traversing by different dependency types.
2019-06-05 06:11:18 -07:00
Todd Gamblin
2e22fc1090 spack dependencies: support --deptype argument
- `spack dependencies` can now take a --deptype argument to only traverse
  particular deptypes

- add a new "common" argument for deptype in spack.cmd.common.arguments

- Database.installed_relatives() can now also take a deptype argument
  - this is used by `spack dependencies --installed`
2019-06-05 06:11:18 -07:00
Todd Gamblin
3dac78fc19 package: make possible_dependencies consider deptypes
- `PackageBase.possible_dependencies` now:
  - accepts a deptype param that controls dependency types traversed
  - returns a dict mapping possible depnames to their immediate possible
    dependencies (this lets you build a graph easily)

- Add tests for PackageBaes
2019-06-05 06:11:18 -07:00
Todd Gamblin
087a511da7 refactor: remove vestigial ALLOWED_URL_SCHEMES from package.py 2019-06-05 06:11:18 -07:00
Todd Gamblin
87e6cb9f72 refactor: make Package.name consistent with other class attributes
- The 'name' attribute for packages was being set in DirectiveMeta, which
  wasn't consistent with other class properties (like fullname, etc.)

- Move it to be a class property of `PackageMeta`, and add the
  corresponding property method wrapper on `PackageBase`
2019-06-05 06:11:18 -07:00
Todd Gamblin
3f5141d629 refactor: make PackageBase.possible_dependencies() a class method. 2019-06-05 06:11:18 -07:00
Justin S
4ac64e6cd8 add C standard flags to compiler classes (#11618)
* add c99_flag, c11_flag to compiler class

* implement c99_flag, c11_flag for gcc

* implement c99_flag, c11_flag for arm

* implement c99_flag for cce

* implement c99_flag, c11_flag for clang

* implement c99_flag, c11_flag for intel

* implement c99_flag, c11_flag for xl
2019-06-05 11:43:11 +02:00
Greg Becker
0990f12dd9 modules: set permissions based on package configuration (#11337)
Previously, module files were not set with the same permissions as the package installation.  For world-readable packages, this would not cause a problem.  For group readable packages, it does:

```
packages:
  mypackage:
    permissions:
      group: mygroup
      read: group
      write: group
```

In this case, the modulefile is unreadable by members of the group other than the one who installed it.  Add logic to the modulefile writers to set the permissions based on the configuration in `packages.yaml`
2019-06-04 19:15:47 -04:00
Patrick Gartung
964a1d5997
Buildcache relocate.py error fix (#11616)
* Add a trailing / if file --mime does not return a mimetype with a / in it

* Flake8
2019-06-04 14:39:04 -05:00
Patrick Gartung
88473a8da3
Build cache: relocate path to spack/bin/sbang in text files. (#11592)
* Build cache: relocate path to spack/bin/sbang in text files.

* Found in testing.

* update packaging test

* Make sbang replacement including #!/bin/bash. Add an additional spack prefix replacement to fix stage directory references.

* flake8

* Use buildinfo.get() so old buildcaches without buildinfo['spackprefix'] can be read.
2019-05-31 11:33:29 -05:00
Massimiliano Culpo
01ece824e1 Cap the maximum number of build jobs (#11373)
* config:build_jobs now controls the number of parallel jobs to spawn during
builds, but cannot ever exceed the number of cores on the machine.

* The default is set to 16 or the number of available cores, whatever
is lowest.

* Updated docs to reflect the changes done to limit parallel builds
2019-05-28 06:42:04 -07:00
Todd Gamblin
3ae5262182
docs: set gettext_uuid to False to reduce churn (#11567)
- `gettext_uuid=True` makes every commit update every .pot file in spack/localized-docs,
  and speeds up the internationalized doc build slightly.

- Optimize for less repository churn, and use `python-levenshtein` to accelerate 
  the build instead.
2019-05-27 13:42:27 -07:00
Axel Huebl
6d54212b94 mirror create: non-zero return code (#8585)
`mirror create` should return a non-zero return code if errors
occured.
2019-05-27 13:01:01 +02:00
Todd Gamblin
d6f2ff1426 link_tree: add option to merge link trees with relative targets
- previous version of link trees would only do absolute symlinks

- this version can do relative links using merge(relative=True)
2019-05-26 18:23:44 -07:00
Todd Gamblin
f32843528e docs: make docs build relocatable for localized builds
- make all Spack paths relative to a `_spack_root` symlink, so that we
  can easily relocate the docs build *outside* lib/spack/docs

- set some useful defaults for gettext translation variables in conf.py

- update `relativeinclude` and other references to the spack root in the
  RST files to use _spack_root
2019-05-26 18:23:44 -07:00
Todd Gamblin
8bf1bd4637 docs: remove legacy import for Sphinx 1.7 2019-05-26 18:23:44 -07:00
Todd Gamblin
e1c5d1378b docs: match ReadTheDocs; use -W (warnings as errors) with Sphinx 2019-05-26 18:23:44 -07:00
Todd Gamblin
3340d586c4 commands: add --update option to spack list
- Add a `--update FILE` option to `spack list`
- Output is written to the file only if any package is newer than the file
- Simplify the code in docs/conf.py using this new option
2019-05-26 18:23:44 -07:00
Todd Gamblin
6380f1917a commands: Add --header and --update options to spack commands
The Spack documentation currently hard-codes some functionality in
`conf.py`, which makes the doc build less "pluggable" for things like
localized doc builds.

In particular, we unconditionally generate an index of commands and a
package list as part of the docs, but those should really only be done if
things are not up to date.

This commit does the following:

- Add `--header` option to `spack commands` so that it can do the work of
  prepending text to its output.

- Add `--update FILE` option to `spack commands` that makes it generate a
  new command index *only* if FILE is out of date w.r.t. commands in the
  Spack source.

- Simplify code in `conf.py` to use these options and only update the
  command index when needed.
2019-05-26 18:23:44 -07:00
Todd Gamblin
43aaf8c404 docs: Use HDF5 as the example package for spack graph 2019-05-26 18:23:44 -07:00
Massimiliano Culpo
c291866b9a build env: simplify handling of parallel jobs (#11524)
This PR implements several refactors requested in #11373, specifically:

- Config scopes are used to handle builtin defaults, command line overrides 
  and package overrides (`parallel=False`)
- `Package.make_jobs` attribute has been removed; `make_jobs` remains
  as a module-scope variable in the build environment.
- The use of the argument `-j` has been rationalized across commands
  - move '-j'/'--jobs' argument into `spack.cmd.common.arguments`
- Add unit tests to check that setting parallel jobs works as expected
  - add new test to ensure that build job setting is isolated to each build
- Fix packages that used `Package.make_jobs` (i.e. `bazel`)
2019-05-24 11:45:22 -07:00
Todd Gamblin
ab21b3d194 docs: truncate spack list output in basic usage
`spack list` output is very long and takes up a lot of space in the docs.
Truncate it to just 10 lines and link to the package list page.
2019-05-23 12:40:01 -07:00
t-karatsu
1e9bb8c512 Add Fujitsu compiler to Spack. (#11287)
* Add Fujitsu compiler to Spack.

* Fixes for flake8

* Chenges location of FCC to subdirectory called case-insensitive

* Add compiler tests for Fujitsu compiler

* Modify the logic of taking compiler version for new version of Fujitsu compiler
2019-05-21 20:31:14 -05:00
Adam J. Stewart
e2065fad72 Make flake8 check spack script as well as .py files (#11513)
* Make flake8 check bin/spack
2019-05-20 13:39:25 -07:00
Mario Melara
4d71117080 Get Cray OS version from PrgEnv-cray (#10774)
The regex used for finding the Cray OS version from the PrgEnv-cray
module was not exact and was at times pulling the version from other
PrgEnv modules. This updates the regular expression to be more exact.
2019-05-20 11:06:02 -07:00
George Hartzell
e592262da8 Typos: funciton, woudl, hm,... (#11511) 2019-05-18 11:39:23 -05:00
Mario Melara
9957093e53 Execute modulecmd in bash shell (#11510)
Adds executable=/bin/bash into Popen. We discovered this bug while
working in a csh/tsch environment. By executing with /bin/bash we ensure
that the module command works.
2019-05-18 01:49:19 +02:00
Massimiliano Culpo
c03be0d65a Command extensions can import code from modules in root or cmd folder (#11209)
#8612 added command extensions to Spack: a command implemented in a
separate directory. This improves the implementation by allowing
the command to import additional utility code stored within the
established directory structure for commands.

This also:

* Adds tests for command extensions
* Documents command extensions (including the expected directory
  layout)
2019-05-16 17:27:42 -07:00
Chris Green
b9370bf20b Allow command access to dump/pickle_environment from #8476 (#11434)
* Allow command access to dump/pickle_environment from #8476
2019-05-17 09:15:32 +09:00
Denis Davydov
9d34326992 fix suite-sparse built with tbb from intel-parallel-studio (#11134)
* fix suite-sparse built with tbb from intel-parallel-studio

* intel: add tbb_headers, strip newline from cxx_lib

* use property
2019-05-15 11:03:19 -07:00
Todd Gamblin
1dc8f952a5
Use svn info --xml instead of svn info to get svn revisions (#11466)
- `svn info` prints different results depending on the system locale
  - in particular, Japanese output doesn't contain "Revision:"

- Change Spack code to use XML output instead of using the human output
2019-05-15 13:37:02 +02:00
Greg Becker
b5316c31af Remove vestigial print statement in module logic (#11438)
Remove a vestigial print statement introduced in #8570
2019-05-14 09:03:34 +09:00
Greg Becker
f67a59fabc permissions: preserve suid and sgid bits (#10727)
* Don't overwrite suid/sgid bits when setting permissions
* add tests for permission setting
2019-05-10 00:38:13 -07:00
Glenn Johnson
c752af098d Intel packages: multiple installs and optional scalapack libs (#11384)
Add fixes to support multiple installs and dependents using a subset
of IntelPackage functionality.

* Update IntelPackage to only return scalapack libraries if the root
  spec depends on MPI: scalapack requires MPI to be mentioned as a
  dependency in the DAG. Package builds using intel-mkl for its
  blas/lapack implementations but not for scalapack were failing to
  build.

  Ideally it would be possible to ask if any of the packages in the
  DAG are actually requesting the scalapack functionality provided by
  the IntelPackage and only return scalapack libs in that case, but
  that is not easily done at this time.

  Fixes #11314
  Fixes #11289

* set HOME when the intel silent installer is run. This prevents the
  installer from using the ~/intel directory (which can cause
  conflicts for multiple installs of the same IntelPackage)

  Fixes #9713
2019-05-10 10:54:05 +09:00
Greg Becker
3d3cea1c87 modules: use new module function instead of get_module_cmd (#8570)
Use new `module` function instead of `get_module_cmd`

Previously, Spack relied on either examining the bash `module()` function or using the `which` command to find the underlying executable for modules. More complicated module systems do not allow for the sort of simple analysis we were doing (see #6451).

Spack now uses the `module` function directly and copies environment changes from the resulting subprocess back into Spack. This should provide a future-proof implementation for changes to the logic underlying the module system on various HPC systems.
2019-05-09 15:04:24 -07:00
Peter Josef Scheibel
53ec16c9e5 make license check more-strict
The license text is now expected to match almost exactly (not
accounting for formatting in different file types (e.g. rst vs.
bash script vs. python)
2019-05-10 04:37:56 +09:00
Seth R Johnson
6cdbc33c90 Hide some variables in spack-build.env file
Don't arbitrarily reset PWD and OLDPWD when sourcing, as well as other
session-specific IDs
2019-05-08 06:36:52 +09:00
Chris Green
01eaca607f Environment path sanitization and sourcing (#8476)
Add two functions to the EnvironmentModifications object to help
users sanitize environment variables in their package definitions:

* deprioritize_system_paths: this keeps system paths in the
  environment variable but moves them to the end.
* prune_duplicate_paths: remove any duplicate paths from the
  variable

This includes testing for the new functions as well as for
(previously-untested) old convenience functions for environment
variable manipulation.

This also adds special handling for bash functions so they
will be defined when the exported environment file is sourced.
2019-05-07 11:29:18 +09:00
Patrick Gartung
9a85a7a5aa Binary caches on MacOS - allow expanded RPATHs (#11345)
Fixes #11335

Update the Spack compiler wrappers to add the headerpad_max_install_names
linker flag on MacOS. This allows the install_name_tool to rewrite
the RPATH entry of the binary to be longer if needed. This is
primarily useful for creating and distributing binary caches of
packages (i.e. using the "spack buildcache" command); binary caches
created on MacOS before this commit may not successfully relocate
(if the target root path is larger).
2019-05-04 06:41:57 +09:00
Massimiliano Culpo
5ffb270714 Added a function that concretizes specs together (#11158)
* Added a function that concretizes specs together

* Specs concretized together are copied instead of being referenced

This makes the specs different objects and removes any reference to the
fake root package that is needed currently for concretization.

* Factored creating a repository for concretization into its own function

* Added a test on overlapping dependencies
2019-05-04 03:04:38 +09:00
Denis Davydov
5b82bf47af extend Version class so that 2.0 > 1.develop > 1.1 and develop > master > head > trunk > 9999 (#1983)
* extend Version class so that 2.0 > 1.develop > 1.1

* add concretization tests, with preferences and preferred version.

* add master, head, trunk as develop-like versions, develop > master > head > trunk

* update documentation on version comparison
2019-05-03 03:32:40 +09:00
Levi Baber
9e72fc48c8
fix error when adding package to env in container (#11321) 2019-05-02 11:04:21 -05:00
Patrick Gartung
309122c329
Don't use buildcache to install patchelf (#11343) 2019-05-02 10:02:25 -05:00
Todd Gamblin
8e5e36dc0b Fix spack flake8 to use Travis's target as base when needed 2019-05-01 04:44:02 +09:00
Daryl W. Grunau
edca0cde1c _valid_tokens list is missing intended comma (#11271)
* _valid_tokens list is missing a needed comma
2019-04-24 08:55:20 +09:00
Todd Gamblin
552c9c57fd
Update tutorial page with RIKEN tutorial slides. (#11254)
- Add link to container image, as well as a description of VMs
- Update slide link to point to latest RIKEN tutorial
2019-04-23 01:53:47 +09:00
Greg Becker
5aa4edb939 Fix transitions between tutorial sections (#11251) 2019-04-22 17:14:26 +09:00
Todd Gamblin
0aed3bcea6 spack edit: use execv instead of Executable (#11245)
- `spack edit` previously used `spack.util.executable` `Executable` objects,
  and didn't `exec` the editor like you'd expect it to

- This meant that Spack was still running while your editor was, and
 stdout/stdin were being set up in weird ways

- e.g. on macOS, if you call `spack edit` with `EDITOR` set to the
  builtin `emacs` command, then type `Ctrl-g`, the whole thing dies with
  a `==> Error: Keyboard interrupt`

- Fix all this by changing spack.util.editor to use `os.execv` instead of
  Spack's `Executable` object
2019-04-20 20:51:45 -07:00
Adam J. Stewart
7255d5ee3c
Fix detection of LLVM-enabled PGI compilers (#10704)
* Fix detection of LLVM-enabled PGI compilers

* Add unit tests for LLVM-enabled PGI compiler version detection
2019-04-20 17:52:55 -05:00
Denis Davydov
95fafb4e44 Move NoLibrariesError/NoHeadersError into error.py (#10997)
Also add constructor to NoLibrariesError which can either take an
error message (like other SpackErrors) or a name and prefix (in
which case the error message is constructed).
2019-04-20 14:08:05 -07:00
Justin S
6f1fe3904c Fix outdated R packages failing to fetch (#11039)
PR #10758 made a slight change to find_versions_of_archive() which included
archive_url in the search process. While this fixed `spack create` and
`spack checksum` missing command-line arguments, it caused `spack
install` to prefer those URLs over those it found in the scrape process.

As a result, the package url was treated as a list_url causing all R
packages to stop fetching once the package was updated on CRAN.

This patch is more selective about including the archive_url in the
remote versions, explicitly overriding it with matching versions found
by the scraper.
2019-04-19 17:39:13 -07:00
Greg Becker
141e8b8de3 Fix backwards compatibility for module naming (#11236)
f242f5f8 changed the format strings but maintained backwards
compatibility in all cases except one: The list of valid tokens for
the module naming schemes was not updated properly to contain both
the new and old styles for compilers and package names.

This PR re-adds the old tokens into the list of valid tokens.
2019-04-19 13:20:41 -07:00
Peter Scheibel
8a23154755
Spack chain docs: config should go in upstreams.yaml (#11225)
#11152 added documentation for #8772 but some details were based on
an earlier implementation that had changed by the time #8772 was
merged. In particular, #11152 mentioned that upstream Spack instances
were configured in config.yaml, when in fact they should be placed in
a separate upstreams.yaml config file; this PR updates the
documentation accordingly.
2019-04-18 18:57:10 -07:00
Massimiliano Culpo
9cf650d2ab Moved cleanup before raising (the code couldn't be reached before) 2019-04-18 18:41:22 -07:00
Massimiliano Culpo
400aa5fe51 Cleaned get methods of Repo and RepoPath 2019-04-18 18:41:22 -07:00
Massimiliano Culpo
76b5af6bf3 Used functools.wrap for the decorator + reordered imports 2019-04-18 18:41:22 -07:00
Massimiliano Culpo
bbb5d61662 Removed 'namespace' argument from Repo and RepoPath
fixes #11159

The 'namespace' argument to both Repo and RepoPath were used to set the
"super namespace". Currently it seems to be vestigial as the only
"super namespace" allowed for packages is 'spack.pkg' since 39c9bbf
2019-04-18 18:41:22 -07:00
Zack Galbreath
7febb88c2a improvements to our CDash reporter (#11168)
* Make a separate CDash report for each package installed

Previously, we generated a single CDash report ("build") for the complete results
of running a `spack install` command. Now we create a separate CDash build for
each package that was installed.

This commit also changes some of the tests related to CDash reporting.
Now only one of the tests exercises the code path of uploading to a
(nonexistent) CDash server. The rest of the related tests write their reports
to disk without trying to upload them.

* Don't report errors to CDash for successful packages

Convert errors detected by our log scraper into warnings when the package
being installed reports that it was successful.

* Report a maximum of 50 errors/warnings to CDash

This is in line with what CTest does. The idea is that if you have more than
50 errors/warnings you probably aren't going to read through them all anyway.
This change reduces the amount of data that we need to transfer and store.
2019-04-18 09:39:35 -07:00
Greg Becker
f242f5f8a9 Features: Improve Spec format strings (#10556)
* Update spec format to simpler syntax, maintain backwards compatibility
* Switch to new spec.format method throughout internals
* update package files for new format strings
* documentation and minor code cleanup. removed nonsensical variant sigils
2019-04-17 18:21:40 -07:00
Satish Balay
7280f21397 Add ftn to lib/spack/env (#11180)
Fixes #11070 #11010

Spack attempts to intercede on behalf of all compiler invocations for
a build. This involves adding its wrappers to PATH. Cray systems
include a "ftn" executable and Spack was only redirecting this call
when the Spec was built with cce. This updates the compiler wrappers
to add "ftn" in all cases.
2019-04-17 13:12:25 -07:00
Peter Scheibel
ef8b5c73ce
don't record 'view: True' in environment config (#11182)
The default (implied) behavior for all environments, as of ea1de6b,
is that an environment will maintain a view in a location of its
choosing. ea1de6b explicitly recorded all three possible states of
maintaining a view:

 1. Maintain a view, and let the environment decide where to put it
    (default)
 2. Maintain a view, and let the user decide
 3. Don't maintain a view

This commit updates the config writer so that for case [1], nothing
will be written to the config.yaml. This will not change any existing
behavior, it just serves to keep the config more compact.
2019-04-16 19:08:10 -07:00
Adam J. Stewart
dbfa928cdb Add unit tests for Cray compiler detection (#11191) 2019-04-16 10:01:42 +02:00
Greg Becker
5f74f22dc6 Bugfix: Install missing compilers for dependency packages (#11175)
Compilers are treated separately from other dependencies in Spack.
#10761 added the option to automatically install compilers when a
package specifies using a compiler that is not available in Spack.
However, this did not work correctly for dependency packages (it
would only build a compiler for the root of an install DAG). This
commit enables the building of compilers for dependency packages.
2019-04-15 17:41:15 -07:00
Massimiliano Culpo
263d8a818a Updated Sphinx configuration (#11165) 2019-04-11 14:38:52 -07:00
Denis Davydov
177b7f111a intel: optionally take gcc executable from cflags (#11136) 2019-04-11 12:33:59 -07:00
Peter Scheibel
ea1de6b941 Maintain a view for an environment (#10017)
Environments are nowm by default, created with views.  When activated, if an environment includes a view, this view will be added to `PATH`, `CPATH`, and other shell variables to expose the Spack environment in the user's shell.

Example:

```
spack env create e1 #by default this will maintain a view in the directory Spack maintains for the env
spack env create e1 --with-view=/abs/path/to/anywhere
spack env create e1 --without-view
```

The `spack.yaml` manifest file now looks like this:

```
spack:
  specs:
  - python
  view: true #or false, or a string
```

These commands can be used to control the view configuration for the active environment, without hand-editing the `spack.yaml` file:

```
spack env view enable
spack env view envable /abs/path/to/anywhere
spack env view disable
```

Views are automatically updated when specs are installed to an environment. A view only maintains one copy of any package. An environment may refer to a package multiple times, in particular if it appears as a dependency. This PR establishes a prioritization for which environment specs are added to views: a spec has higher priority if it was concretized first. This does not necessarily exactly match the order in which specs were added, for example, given `X->Z` and `Y->Z'`:

```
spack env activate e1
spack add X
spack install Y # immediately concretizes and installs Y and Z'
spack install # concretizes X and Z
```

In this case `Z'` will be favored over `Z`. 

Specs in the environment must be concrete and installed to be added to the view, so there is another minor ordering effect: by default the view maintained for the environment ignores file conflicts between packages. If packages are not installed in order, and there are file conflicts, then the version chosen depends on the order.

Both ordering issues are avoided if `spack install`/`spack add` and `spack install <spec>` are not mixed.
2019-04-10 16:00:12 -07:00
Mario Melara
a1db22ba1a Unload altd and darshan (#11162) 2019-04-10 13:19:41 -07:00
Peter Scheibel
c1f8fdd5df
Add documentation for chaining Spack instances (#11152)
Add documentation for the Spack chain feature added in #8772
2019-04-10 11:54:13 -07:00
George Hartzell
0256766ab9 Typo: Unkown -> Unknown (#11150) 2019-04-09 18:36:45 -05:00
Denis Davydov
1f1ea2c859 intel: consolidate libs() in the base class (#11143)
* intel: consolidate libs() in the base class

* white space

* flake8
2019-04-09 13:54:21 -07:00
Scott Wittenburg
3616617804 Allow spack install to take either cdash stamp or track (#11106)
When providing a track, the cdash reporter will format the stamp
itself, as it has always done, and register the build during the
package installation process.  When providing a stamp, it should
first be formatted as cdash expects, and then cdash will be sure
to report results to same build id which was registered manually
elsewhere.
2019-04-06 15:30:34 -07:00
Greg Becker
8a17282001 Fix reading externals from old databases (#11118)
* Update Spec.prefix to have special case for 'None' in database path; regression test
* Update in database reader rather than spec
* Change assertion to conditional + raise
* Added test for concrete check in Spec.prefix
2019-04-05 14:58:57 -07:00
Michael Kuhn
320e00276f Fix module_parsing test (#11087)
The module_parsing test checks whether the module function is available
by looking for the string 'not found'. If the user has set a different
locale, the test can assume that the module function is available when
it actually is not.
2019-04-05 11:26:40 -07:00
Tim Fuller
a9ae507152 Fix directory layout using namespace (#11076)
Directory layouts using ${NAMESPACE} were broken. This addresses that
and adds a test to prevent regression.
2019-04-02 17:47:57 -07:00
Tristan Carel
4793242158 spack repo add: fix error message when `packages' directory is missing (#11031) 2019-04-02 17:05:57 -07:00
Massimiliano Culpo
a1255b61c0 Added tests on compiler's version detection (#10931)
* Split get_compiler_version into two functions:
  get_compiler_version_output runs the compiler with the relevant
  option to print the version; extract_version_from_output determines
  the version by examining this output. This makes it easier to test
  the customized version detection for each compiler. Users can
  customize this by overriding the following:
  * version_argument: this is the argument that tells the compiler to
    print its version. It assumes that the compiler will report its
	version if invoked with a single option (like "--version")
  * version_regex: the regular expression used to extract the version
    from the compiler argument. This assumes that a regular
	expression is sufficient to extract the version, and that the
	version can be extracted from a single capture group (Spack uses
	the first capture group)
  * default_version: allows you to completely override all version
    detection logic
  * get_compiler_version_output: if getting the compiler to report
    its version is more complex than invoking it with a single arg
  * extract_version_from_output: if it is difficult to define a regex
    that can be used to extract the version from the output
* Added tests for version detection of most compilers
* Removed redundant code from xl_r compiler class (by inheriting
  from xl compiler definition)
2019-04-02 12:52:47 -07:00
Massimiliano Culpo
e3f00750e8 Update llnl.util.lang.memoized so that Sphinx can extract signature (#11055)
Replace the original implementation of the "memoized" decorator with
an implementation that exposes the docstring and arguments of the
wrapped function. This is achieved using functools.wraps.
2019-03-29 17:11:44 -07:00
Denis Davydov
def5b23763 add libs property to IntelMkl and IntelParallelStudio (#10993)
* add libs property to IntelMkl and IntelParallelStudio
* fix scalapack_libs when MPI is provided by intel-parallel-studio
2019-03-29 13:01:43 -07:00
Massimiliano Culpo
0a006351c8 Spack can be extended with external commands (#8612)
This provides a mechanism to implement a new Spack command in a
separate directory, and with a small configuration change point Spack
to the new command.

To register the command, the directory must be added to the
"extensions" section of config.yaml. The command directory name must
have the prefix "spack-", and have the following layout:

  spack-X/
    pytest.ini #optional, for testing
    X/
	  cmd/
	    name-of-command1.py
	    name-of-command2.py
	    ...
    tests/ #optional
      conftest.py
	  test_name-of-command1.py
    templates/ #optional jinja templates, if needed

And in config.yaml:

  config:
    extensions:
      - /path/to/spack-X

If the extension includes tests, you can run them via spack by adding
the --extension option, like "spack test --extension=X"
2019-03-28 16:56:36 -07:00
Peter Scheibel
a6511fbafc Consistent patch ordering (#10879)
* preserve the order in which patches are applied by packages (in spite of grouping them by 'when')

* add tests confirming patch order
2019-03-28 11:25:44 -07:00
Peter Scheibel
99f35c3338 spack chain (#8772)
* initial work to make use of an 'upstream' spack installation: this uses the DB of the upstream installation to check if a package is installed

* need to query upstream dbs when adding new record to local db

* prevent reindexing upstream DBs

* set prefix on specs read from DB based on path stored in install record

* check that Spack does not install packages that are recorded as installed in an upstream db

* externals do not add their path to install records - need to use 'external_path' to get path of upstream externals

* views need to check for upstream installations when linking metadata

* package and spec now calculate upstream installation properties on-demand themselves rather than depending on concretization to set these properties up-front. The added tests for upstream installations don't work with this new strategy so they need to be updated

* only refresh modules for local specs (not those in upstream packages); optionally generate local module files for packages installed upstream

* when a user tries to locate a module file for a package installed upstream, tell them to use the upstream spack instance to locate it

* support recursive upstream databases (allow upstream databases to use their own upstream databases)

* separate upstream config into separate file with its own schema; each entry now also includes a name

* metadata_dir is no longer customizable on a per-instance basis for YamlDirectoryLayout

* treat metadata_dir as an instance variable but dont set it from kwargs; this follows several other hardcoded variables which must be consistent between upstream and downstream DBs. Also update DirectoryLayout.metadata_path to work entirely with Spec.prefix, since Spec.prefix is set from the DB when available (so metadata_path was duplicating that logic)
2019-03-27 13:06:46 -07:00
Mario Melara
298a55b28f Replace previous bash command to new one (#7005)
Prevents infinite recursion caused by sourcing `setup-env.sh` in bashrc
2019-03-27 10:26:52 -07:00
Chris Green
e88c1d585c Move CMakePackage build directory to base stage directory (#8431)
Change the location of the CMake build area from the staged source
directory to the stage base directory.

This change allows CMake packages to refer to the build directory in
setup_environment (e.g. if tests need to have a directory in PATH):
Staging happens after the call to setup_environment(), and if the
stage area does not exist, then spec.stage.source_path returns None.

To accommodate this change, archived files (like config.log for
Autotools packages) are archived relative to the stage base directory
rather than the expanded source directory.

Other packages (those not using CMake) will still use the staged
source directory as the default working directory for builds (and
will still be unable to reference this directory in
setup_environment())
2019-03-26 17:40:49 -05:00
ajw1980
e5b86c5527 Retrieve environment-modules prefix based on architecture (#10975)
When multiple instances of environment-modules were installed with
different architectures, Spack was not retrieving the installation
appropriate for the current architecture when finding the module
prefix.
2019-03-26 15:23:17 -05:00
Seth R. Johnson
844ca31894 Use 'shlex' to split default Executable arguments (#10929)
This allows shell commands for `spack edit` to be executed correctly if
they have quoted arguments.
2019-03-26 14:02:32 -05:00
Massimiliano Culpo
89b9880719 Add regression marker to pytest.ini (#11011) 2019-03-26 12:04:36 -05:00
Nichols A. Romero
66172f80f3 Fixed some issues with CUDA-Intel compiler conflicts. (#10924)
* Fixed some issues with CUDA-Intel compiler conflicts.

* Comment about expressing CUDA-compiler conflicts.

* More precise conflicts and also add support for Intel 19.0
2019-03-26 15:31:56 +01:00
Shahzeb Siddiqui
724acb6421 typo in documentation (#10976) 2019-03-21 14:29:40 -05:00
George Hartzell
aa1e70aab1 Minor commentary correction, TCL -> lmod (#10951) 2019-03-21 11:17:41 +01:00
Stephen McDowell
70e436e932 prevent UnboundLocalError when sourcing files (#10950)
Patch extracted from #7536 courtesy of @mgsternberg
2019-03-21 11:08:29 +01:00
Chris Green
9ad02685d9 Fix help text for --no-checksum. (#10932) 2019-03-19 15:02:06 -05:00
Adam J. Stewart
4e6285a19b
Ensure that every package has a description (#10896) 2019-03-14 20:49:07 -05:00
Chris Green
9b51fb09f1 Support VISUAL environment variable for editing. (#10898)
If the user has set the environment variable VISUAL, it will be used
in preference to EDITOR for all Spack editing activities. If VISUAL
is not set or fails (perhaps due to a lack of graphical editing
capabilities),EDITOR will be used instead. We fall back to one of
several common editors if neither bears fruit.

This feature has been tailored to:

* Provide identical behavior to the previous implementation in the
  case that VISUAL is not set.
* Not require any change to code utilizing the editor feature.
* Follow usual UNIX behavior concerning VISUAL and EDITOR.
2019-03-14 16:16:26 -05:00
Tristan Carel
c3662492de Do not use string module to be compatible with python 3 (#248) (#10667)
`string.find` is not part of Python 3 anymore.
2019-03-12 20:34:47 -05:00
Justin Stanley
2bea940e43 find_versions_of_archive: expand link search (#10758) 2019-03-12 20:19:46 -05:00
Nick Forrington
6bda37f542 Fix clearing EnvironmentModifications with python2 (#10791)
* Fix clearing EnvironmentModifications with python2

* Add EnvironmentModifications::clear unit test

Use re-assignment rather than del to clear array

* Fix flake issues
2019-03-12 20:12:51 -05:00
Shahzeb Siddiqui
d20b5ce2ec format change and typo in doc (#10848) 2019-03-11 13:19:13 -05:00
Massimiliano Culpo
a42fd7f276 Improved detection of Clang versions (#10316)
Fixes #10191

* Add more regular expressions to detect clang versions that were
  not being picked up
* Add a test for parsing versions from the output of Clang (this
  does not run Clang, but rather uses example outputs from Clang)
* Separate Clang version parsing into its own method (to make it
  easier to test)
2019-03-11 13:15:34 -05:00