Commit graph

3288 commits

Author SHA1 Message Date
Todd Gamblin
3ae5262182
docs: set gettext_uuid to False to reduce churn (#11567)
- `gettext_uuid=True` makes every commit update every .pot file in spack/localized-docs,
  and speeds up the internationalized doc build slightly.

- Optimize for less repository churn, and use `python-levenshtein` to accelerate 
  the build instead.
2019-05-27 13:42:27 -07:00
Axel Huebl
6d54212b94 mirror create: non-zero return code (#8585)
`mirror create` should return a non-zero return code if errors
occured.
2019-05-27 13:01:01 +02:00
Todd Gamblin
d6f2ff1426 link_tree: add option to merge link trees with relative targets
- previous version of link trees would only do absolute symlinks

- this version can do relative links using merge(relative=True)
2019-05-26 18:23:44 -07:00
Todd Gamblin
f32843528e docs: make docs build relocatable for localized builds
- make all Spack paths relative to a `_spack_root` symlink, so that we
  can easily relocate the docs build *outside* lib/spack/docs

- set some useful defaults for gettext translation variables in conf.py

- update `relativeinclude` and other references to the spack root in the
  RST files to use _spack_root
2019-05-26 18:23:44 -07:00
Todd Gamblin
8bf1bd4637 docs: remove legacy import for Sphinx 1.7 2019-05-26 18:23:44 -07:00
Todd Gamblin
e1c5d1378b docs: match ReadTheDocs; use -W (warnings as errors) with Sphinx 2019-05-26 18:23:44 -07:00
Todd Gamblin
3340d586c4 commands: add --update option to spack list
- Add a `--update FILE` option to `spack list`
- Output is written to the file only if any package is newer than the file
- Simplify the code in docs/conf.py using this new option
2019-05-26 18:23:44 -07:00
Todd Gamblin
6380f1917a commands: Add --header and --update options to spack commands
The Spack documentation currently hard-codes some functionality in
`conf.py`, which makes the doc build less "pluggable" for things like
localized doc builds.

In particular, we unconditionally generate an index of commands and a
package list as part of the docs, but those should really only be done if
things are not up to date.

This commit does the following:

- Add `--header` option to `spack commands` so that it can do the work of
  prepending text to its output.

- Add `--update FILE` option to `spack commands` that makes it generate a
  new command index *only* if FILE is out of date w.r.t. commands in the
  Spack source.

- Simplify code in `conf.py` to use these options and only update the
  command index when needed.
2019-05-26 18:23:44 -07:00
Todd Gamblin
43aaf8c404 docs: Use HDF5 as the example package for spack graph 2019-05-26 18:23:44 -07:00
Massimiliano Culpo
c291866b9a build env: simplify handling of parallel jobs (#11524)
This PR implements several refactors requested in #11373, specifically:

- Config scopes are used to handle builtin defaults, command line overrides 
  and package overrides (`parallel=False`)
- `Package.make_jobs` attribute has been removed; `make_jobs` remains
  as a module-scope variable in the build environment.
- The use of the argument `-j` has been rationalized across commands
  - move '-j'/'--jobs' argument into `spack.cmd.common.arguments`
- Add unit tests to check that setting parallel jobs works as expected
  - add new test to ensure that build job setting is isolated to each build
- Fix packages that used `Package.make_jobs` (i.e. `bazel`)
2019-05-24 11:45:22 -07:00
Todd Gamblin
ab21b3d194 docs: truncate spack list output in basic usage
`spack list` output is very long and takes up a lot of space in the docs.
Truncate it to just 10 lines and link to the package list page.
2019-05-23 12:40:01 -07:00
t-karatsu
1e9bb8c512 Add Fujitsu compiler to Spack. (#11287)
* Add Fujitsu compiler to Spack.

* Fixes for flake8

* Chenges location of FCC to subdirectory called case-insensitive

* Add compiler tests for Fujitsu compiler

* Modify the logic of taking compiler version for new version of Fujitsu compiler
2019-05-21 20:31:14 -05:00
Adam J. Stewart
e2065fad72 Make flake8 check spack script as well as .py files (#11513)
* Make flake8 check bin/spack
2019-05-20 13:39:25 -07:00
Mario Melara
4d71117080 Get Cray OS version from PrgEnv-cray (#10774)
The regex used for finding the Cray OS version from the PrgEnv-cray
module was not exact and was at times pulling the version from other
PrgEnv modules. This updates the regular expression to be more exact.
2019-05-20 11:06:02 -07:00
George Hartzell
e592262da8 Typos: funciton, woudl, hm,... (#11511) 2019-05-18 11:39:23 -05:00
Mario Melara
9957093e53 Execute modulecmd in bash shell (#11510)
Adds executable=/bin/bash into Popen. We discovered this bug while
working in a csh/tsch environment. By executing with /bin/bash we ensure
that the module command works.
2019-05-18 01:49:19 +02:00
Massimiliano Culpo
c03be0d65a Command extensions can import code from modules in root or cmd folder (#11209)
#8612 added command extensions to Spack: a command implemented in a
separate directory. This improves the implementation by allowing
the command to import additional utility code stored within the
established directory structure for commands.

This also:

* Adds tests for command extensions
* Documents command extensions (including the expected directory
  layout)
2019-05-16 17:27:42 -07:00
Chris Green
b9370bf20b Allow command access to dump/pickle_environment from #8476 (#11434)
* Allow command access to dump/pickle_environment from #8476
2019-05-17 09:15:32 +09:00
Denis Davydov
9d34326992 fix suite-sparse built with tbb from intel-parallel-studio (#11134)
* fix suite-sparse built with tbb from intel-parallel-studio

* intel: add tbb_headers, strip newline from cxx_lib

* use property
2019-05-15 11:03:19 -07:00
Todd Gamblin
1dc8f952a5
Use svn info --xml instead of svn info to get svn revisions (#11466)
- `svn info` prints different results depending on the system locale
  - in particular, Japanese output doesn't contain "Revision:"

- Change Spack code to use XML output instead of using the human output
2019-05-15 13:37:02 +02:00
Greg Becker
b5316c31af Remove vestigial print statement in module logic (#11438)
Remove a vestigial print statement introduced in #8570
2019-05-14 09:03:34 +09:00
Greg Becker
f67a59fabc permissions: preserve suid and sgid bits (#10727)
* Don't overwrite suid/sgid bits when setting permissions
* add tests for permission setting
2019-05-10 00:38:13 -07:00
Glenn Johnson
c752af098d Intel packages: multiple installs and optional scalapack libs (#11384)
Add fixes to support multiple installs and dependents using a subset
of IntelPackage functionality.

* Update IntelPackage to only return scalapack libraries if the root
  spec depends on MPI: scalapack requires MPI to be mentioned as a
  dependency in the DAG. Package builds using intel-mkl for its
  blas/lapack implementations but not for scalapack were failing to
  build.

  Ideally it would be possible to ask if any of the packages in the
  DAG are actually requesting the scalapack functionality provided by
  the IntelPackage and only return scalapack libs in that case, but
  that is not easily done at this time.

  Fixes #11314
  Fixes #11289

* set HOME when the intel silent installer is run. This prevents the
  installer from using the ~/intel directory (which can cause
  conflicts for multiple installs of the same IntelPackage)

  Fixes #9713
2019-05-10 10:54:05 +09:00
Greg Becker
3d3cea1c87 modules: use new module function instead of get_module_cmd (#8570)
Use new `module` function instead of `get_module_cmd`

Previously, Spack relied on either examining the bash `module()` function or using the `which` command to find the underlying executable for modules. More complicated module systems do not allow for the sort of simple analysis we were doing (see #6451).

Spack now uses the `module` function directly and copies environment changes from the resulting subprocess back into Spack. This should provide a future-proof implementation for changes to the logic underlying the module system on various HPC systems.
2019-05-09 15:04:24 -07:00
Peter Josef Scheibel
53ec16c9e5 make license check more-strict
The license text is now expected to match almost exactly (not
accounting for formatting in different file types (e.g. rst vs.
bash script vs. python)
2019-05-10 04:37:56 +09:00
Seth R Johnson
6cdbc33c90 Hide some variables in spack-build.env file
Don't arbitrarily reset PWD and OLDPWD when sourcing, as well as other
session-specific IDs
2019-05-08 06:36:52 +09:00
Chris Green
01eaca607f Environment path sanitization and sourcing (#8476)
Add two functions to the EnvironmentModifications object to help
users sanitize environment variables in their package definitions:

* deprioritize_system_paths: this keeps system paths in the
  environment variable but moves them to the end.
* prune_duplicate_paths: remove any duplicate paths from the
  variable

This includes testing for the new functions as well as for
(previously-untested) old convenience functions for environment
variable manipulation.

This also adds special handling for bash functions so they
will be defined when the exported environment file is sourced.
2019-05-07 11:29:18 +09:00
Patrick Gartung
9a85a7a5aa Binary caches on MacOS - allow expanded RPATHs (#11345)
Fixes #11335

Update the Spack compiler wrappers to add the headerpad_max_install_names
linker flag on MacOS. This allows the install_name_tool to rewrite
the RPATH entry of the binary to be longer if needed. This is
primarily useful for creating and distributing binary caches of
packages (i.e. using the "spack buildcache" command); binary caches
created on MacOS before this commit may not successfully relocate
(if the target root path is larger).
2019-05-04 06:41:57 +09:00
Massimiliano Culpo
5ffb270714 Added a function that concretizes specs together (#11158)
* Added a function that concretizes specs together

* Specs concretized together are copied instead of being referenced

This makes the specs different objects and removes any reference to the
fake root package that is needed currently for concretization.

* Factored creating a repository for concretization into its own function

* Added a test on overlapping dependencies
2019-05-04 03:04:38 +09:00
Denis Davydov
5b82bf47af extend Version class so that 2.0 > 1.develop > 1.1 and develop > master > head > trunk > 9999 (#1983)
* extend Version class so that 2.0 > 1.develop > 1.1

* add concretization tests, with preferences and preferred version.

* add master, head, trunk as develop-like versions, develop > master > head > trunk

* update documentation on version comparison
2019-05-03 03:32:40 +09:00
Levi Baber
9e72fc48c8
fix error when adding package to env in container (#11321) 2019-05-02 11:04:21 -05:00
Patrick Gartung
309122c329
Don't use buildcache to install patchelf (#11343) 2019-05-02 10:02:25 -05:00
Todd Gamblin
8e5e36dc0b Fix spack flake8 to use Travis's target as base when needed 2019-05-01 04:44:02 +09:00
Daryl W. Grunau
edca0cde1c _valid_tokens list is missing intended comma (#11271)
* _valid_tokens list is missing a needed comma
2019-04-24 08:55:20 +09:00
Todd Gamblin
552c9c57fd
Update tutorial page with RIKEN tutorial slides. (#11254)
- Add link to container image, as well as a description of VMs
- Update slide link to point to latest RIKEN tutorial
2019-04-23 01:53:47 +09:00
Greg Becker
5aa4edb939 Fix transitions between tutorial sections (#11251) 2019-04-22 17:14:26 +09:00
Todd Gamblin
0aed3bcea6 spack edit: use execv instead of Executable (#11245)
- `spack edit` previously used `spack.util.executable` `Executable` objects,
  and didn't `exec` the editor like you'd expect it to

- This meant that Spack was still running while your editor was, and
 stdout/stdin were being set up in weird ways

- e.g. on macOS, if you call `spack edit` with `EDITOR` set to the
  builtin `emacs` command, then type `Ctrl-g`, the whole thing dies with
  a `==> Error: Keyboard interrupt`

- Fix all this by changing spack.util.editor to use `os.execv` instead of
  Spack's `Executable` object
2019-04-20 20:51:45 -07:00
Adam J. Stewart
7255d5ee3c
Fix detection of LLVM-enabled PGI compilers (#10704)
* Fix detection of LLVM-enabled PGI compilers

* Add unit tests for LLVM-enabled PGI compiler version detection
2019-04-20 17:52:55 -05:00
Denis Davydov
95fafb4e44 Move NoLibrariesError/NoHeadersError into error.py (#10997)
Also add constructor to NoLibrariesError which can either take an
error message (like other SpackErrors) or a name and prefix (in
which case the error message is constructed).
2019-04-20 14:08:05 -07:00
Justin S
6f1fe3904c Fix outdated R packages failing to fetch (#11039)
PR #10758 made a slight change to find_versions_of_archive() which included
archive_url in the search process. While this fixed `spack create` and
`spack checksum` missing command-line arguments, it caused `spack
install` to prefer those URLs over those it found in the scrape process.

As a result, the package url was treated as a list_url causing all R
packages to stop fetching once the package was updated on CRAN.

This patch is more selective about including the archive_url in the
remote versions, explicitly overriding it with matching versions found
by the scraper.
2019-04-19 17:39:13 -07:00
Greg Becker
141e8b8de3 Fix backwards compatibility for module naming (#11236)
f242f5f8 changed the format strings but maintained backwards
compatibility in all cases except one: The list of valid tokens for
the module naming schemes was not updated properly to contain both
the new and old styles for compilers and package names.

This PR re-adds the old tokens into the list of valid tokens.
2019-04-19 13:20:41 -07:00
Peter Scheibel
8a23154755
Spack chain docs: config should go in upstreams.yaml (#11225)
#11152 added documentation for #8772 but some details were based on
an earlier implementation that had changed by the time #8772 was
merged. In particular, #11152 mentioned that upstream Spack instances
were configured in config.yaml, when in fact they should be placed in
a separate upstreams.yaml config file; this PR updates the
documentation accordingly.
2019-04-18 18:57:10 -07:00
Massimiliano Culpo
9cf650d2ab Moved cleanup before raising (the code couldn't be reached before) 2019-04-18 18:41:22 -07:00
Massimiliano Culpo
400aa5fe51 Cleaned get methods of Repo and RepoPath 2019-04-18 18:41:22 -07:00
Massimiliano Culpo
76b5af6bf3 Used functools.wrap for the decorator + reordered imports 2019-04-18 18:41:22 -07:00
Massimiliano Culpo
bbb5d61662 Removed 'namespace' argument from Repo and RepoPath
fixes #11159

The 'namespace' argument to both Repo and RepoPath were used to set the
"super namespace". Currently it seems to be vestigial as the only
"super namespace" allowed for packages is 'spack.pkg' since 39c9bbf
2019-04-18 18:41:22 -07:00
Zack Galbreath
7febb88c2a improvements to our CDash reporter (#11168)
* Make a separate CDash report for each package installed

Previously, we generated a single CDash report ("build") for the complete results
of running a `spack install` command. Now we create a separate CDash build for
each package that was installed.

This commit also changes some of the tests related to CDash reporting.
Now only one of the tests exercises the code path of uploading to a
(nonexistent) CDash server. The rest of the related tests write their reports
to disk without trying to upload them.

* Don't report errors to CDash for successful packages

Convert errors detected by our log scraper into warnings when the package
being installed reports that it was successful.

* Report a maximum of 50 errors/warnings to CDash

This is in line with what CTest does. The idea is that if you have more than
50 errors/warnings you probably aren't going to read through them all anyway.
This change reduces the amount of data that we need to transfer and store.
2019-04-18 09:39:35 -07:00
Greg Becker
f242f5f8a9 Features: Improve Spec format strings (#10556)
* Update spec format to simpler syntax, maintain backwards compatibility
* Switch to new spec.format method throughout internals
* update package files for new format strings
* documentation and minor code cleanup. removed nonsensical variant sigils
2019-04-17 18:21:40 -07:00
Satish Balay
7280f21397 Add ftn to lib/spack/env (#11180)
Fixes #11070 #11010

Spack attempts to intercede on behalf of all compiler invocations for
a build. This involves adding its wrappers to PATH. Cray systems
include a "ftn" executable and Spack was only redirecting this call
when the Spec was built with cce. This updates the compiler wrappers
to add "ftn" in all cases.
2019-04-17 13:12:25 -07:00
Peter Scheibel
ef8b5c73ce
don't record 'view: True' in environment config (#11182)
The default (implied) behavior for all environments, as of ea1de6b,
is that an environment will maintain a view in a location of its
choosing. ea1de6b explicitly recorded all three possible states of
maintaining a view:

 1. Maintain a view, and let the environment decide where to put it
    (default)
 2. Maintain a view, and let the user decide
 3. Don't maintain a view

This commit updates the config writer so that for case [1], nothing
will be written to the config.yaml. This will not change any existing
behavior, it just serves to keep the config more compact.
2019-04-16 19:08:10 -07:00
Adam J. Stewart
dbfa928cdb Add unit tests for Cray compiler detection (#11191) 2019-04-16 10:01:42 +02:00
Greg Becker
5f74f22dc6 Bugfix: Install missing compilers for dependency packages (#11175)
Compilers are treated separately from other dependencies in Spack.
#10761 added the option to automatically install compilers when a
package specifies using a compiler that is not available in Spack.
However, this did not work correctly for dependency packages (it
would only build a compiler for the root of an install DAG). This
commit enables the building of compilers for dependency packages.
2019-04-15 17:41:15 -07:00
Massimiliano Culpo
263d8a818a Updated Sphinx configuration (#11165) 2019-04-11 14:38:52 -07:00
Denis Davydov
177b7f111a intel: optionally take gcc executable from cflags (#11136) 2019-04-11 12:33:59 -07:00
Peter Scheibel
ea1de6b941 Maintain a view for an environment (#10017)
Environments are nowm by default, created with views.  When activated, if an environment includes a view, this view will be added to `PATH`, `CPATH`, and other shell variables to expose the Spack environment in the user's shell.

Example:

```
spack env create e1 #by default this will maintain a view in the directory Spack maintains for the env
spack env create e1 --with-view=/abs/path/to/anywhere
spack env create e1 --without-view
```

The `spack.yaml` manifest file now looks like this:

```
spack:
  specs:
  - python
  view: true #or false, or a string
```

These commands can be used to control the view configuration for the active environment, without hand-editing the `spack.yaml` file:

```
spack env view enable
spack env view envable /abs/path/to/anywhere
spack env view disable
```

Views are automatically updated when specs are installed to an environment. A view only maintains one copy of any package. An environment may refer to a package multiple times, in particular if it appears as a dependency. This PR establishes a prioritization for which environment specs are added to views: a spec has higher priority if it was concretized first. This does not necessarily exactly match the order in which specs were added, for example, given `X->Z` and `Y->Z'`:

```
spack env activate e1
spack add X
spack install Y # immediately concretizes and installs Y and Z'
spack install # concretizes X and Z
```

In this case `Z'` will be favored over `Z`. 

Specs in the environment must be concrete and installed to be added to the view, so there is another minor ordering effect: by default the view maintained for the environment ignores file conflicts between packages. If packages are not installed in order, and there are file conflicts, then the version chosen depends on the order.

Both ordering issues are avoided if `spack install`/`spack add` and `spack install <spec>` are not mixed.
2019-04-10 16:00:12 -07:00
Mario Melara
a1db22ba1a Unload altd and darshan (#11162) 2019-04-10 13:19:41 -07:00
Peter Scheibel
c1f8fdd5df
Add documentation for chaining Spack instances (#11152)
Add documentation for the Spack chain feature added in #8772
2019-04-10 11:54:13 -07:00
George Hartzell
0256766ab9 Typo: Unkown -> Unknown (#11150) 2019-04-09 18:36:45 -05:00
Denis Davydov
1f1ea2c859 intel: consolidate libs() in the base class (#11143)
* intel: consolidate libs() in the base class

* white space

* flake8
2019-04-09 13:54:21 -07:00
Scott Wittenburg
3616617804 Allow spack install to take either cdash stamp or track (#11106)
When providing a track, the cdash reporter will format the stamp
itself, as it has always done, and register the build during the
package installation process.  When providing a stamp, it should
first be formatted as cdash expects, and then cdash will be sure
to report results to same build id which was registered manually
elsewhere.
2019-04-06 15:30:34 -07:00
Greg Becker
8a17282001 Fix reading externals from old databases (#11118)
* Update Spec.prefix to have special case for 'None' in database path; regression test
* Update in database reader rather than spec
* Change assertion to conditional + raise
* Added test for concrete check in Spec.prefix
2019-04-05 14:58:57 -07:00
Michael Kuhn
320e00276f Fix module_parsing test (#11087)
The module_parsing test checks whether the module function is available
by looking for the string 'not found'. If the user has set a different
locale, the test can assume that the module function is available when
it actually is not.
2019-04-05 11:26:40 -07:00
Tim Fuller
a9ae507152 Fix directory layout using namespace (#11076)
Directory layouts using ${NAMESPACE} were broken. This addresses that
and adds a test to prevent regression.
2019-04-02 17:47:57 -07:00
Tristan Carel
4793242158 spack repo add: fix error message when `packages' directory is missing (#11031) 2019-04-02 17:05:57 -07:00
Massimiliano Culpo
a1255b61c0 Added tests on compiler's version detection (#10931)
* Split get_compiler_version into two functions:
  get_compiler_version_output runs the compiler with the relevant
  option to print the version; extract_version_from_output determines
  the version by examining this output. This makes it easier to test
  the customized version detection for each compiler. Users can
  customize this by overriding the following:
  * version_argument: this is the argument that tells the compiler to
    print its version. It assumes that the compiler will report its
	version if invoked with a single option (like "--version")
  * version_regex: the regular expression used to extract the version
    from the compiler argument. This assumes that a regular
	expression is sufficient to extract the version, and that the
	version can be extracted from a single capture group (Spack uses
	the first capture group)
  * default_version: allows you to completely override all version
    detection logic
  * get_compiler_version_output: if getting the compiler to report
    its version is more complex than invoking it with a single arg
  * extract_version_from_output: if it is difficult to define a regex
    that can be used to extract the version from the output
* Added tests for version detection of most compilers
* Removed redundant code from xl_r compiler class (by inheriting
  from xl compiler definition)
2019-04-02 12:52:47 -07:00
Massimiliano Culpo
e3f00750e8 Update llnl.util.lang.memoized so that Sphinx can extract signature (#11055)
Replace the original implementation of the "memoized" decorator with
an implementation that exposes the docstring and arguments of the
wrapped function. This is achieved using functools.wraps.
2019-03-29 17:11:44 -07:00
Denis Davydov
def5b23763 add libs property to IntelMkl and IntelParallelStudio (#10993)
* add libs property to IntelMkl and IntelParallelStudio
* fix scalapack_libs when MPI is provided by intel-parallel-studio
2019-03-29 13:01:43 -07:00
Massimiliano Culpo
0a006351c8 Spack can be extended with external commands (#8612)
This provides a mechanism to implement a new Spack command in a
separate directory, and with a small configuration change point Spack
to the new command.

To register the command, the directory must be added to the
"extensions" section of config.yaml. The command directory name must
have the prefix "spack-", and have the following layout:

  spack-X/
    pytest.ini #optional, for testing
    X/
	  cmd/
	    name-of-command1.py
	    name-of-command2.py
	    ...
    tests/ #optional
      conftest.py
	  test_name-of-command1.py
    templates/ #optional jinja templates, if needed

And in config.yaml:

  config:
    extensions:
      - /path/to/spack-X

If the extension includes tests, you can run them via spack by adding
the --extension option, like "spack test --extension=X"
2019-03-28 16:56:36 -07:00
Peter Scheibel
a6511fbafc Consistent patch ordering (#10879)
* preserve the order in which patches are applied by packages (in spite of grouping them by 'when')

* add tests confirming patch order
2019-03-28 11:25:44 -07:00
Peter Scheibel
99f35c3338 spack chain (#8772)
* initial work to make use of an 'upstream' spack installation: this uses the DB of the upstream installation to check if a package is installed

* need to query upstream dbs when adding new record to local db

* prevent reindexing upstream DBs

* set prefix on specs read from DB based on path stored in install record

* check that Spack does not install packages that are recorded as installed in an upstream db

* externals do not add their path to install records - need to use 'external_path' to get path of upstream externals

* views need to check for upstream installations when linking metadata

* package and spec now calculate upstream installation properties on-demand themselves rather than depending on concretization to set these properties up-front. The added tests for upstream installations don't work with this new strategy so they need to be updated

* only refresh modules for local specs (not those in upstream packages); optionally generate local module files for packages installed upstream

* when a user tries to locate a module file for a package installed upstream, tell them to use the upstream spack instance to locate it

* support recursive upstream databases (allow upstream databases to use their own upstream databases)

* separate upstream config into separate file with its own schema; each entry now also includes a name

* metadata_dir is no longer customizable on a per-instance basis for YamlDirectoryLayout

* treat metadata_dir as an instance variable but dont set it from kwargs; this follows several other hardcoded variables which must be consistent between upstream and downstream DBs. Also update DirectoryLayout.metadata_path to work entirely with Spec.prefix, since Spec.prefix is set from the DB when available (so metadata_path was duplicating that logic)
2019-03-27 13:06:46 -07:00
Mario Melara
298a55b28f Replace previous bash command to new one (#7005)
Prevents infinite recursion caused by sourcing `setup-env.sh` in bashrc
2019-03-27 10:26:52 -07:00
Chris Green
e88c1d585c Move CMakePackage build directory to base stage directory (#8431)
Change the location of the CMake build area from the staged source
directory to the stage base directory.

This change allows CMake packages to refer to the build directory in
setup_environment (e.g. if tests need to have a directory in PATH):
Staging happens after the call to setup_environment(), and if the
stage area does not exist, then spec.stage.source_path returns None.

To accommodate this change, archived files (like config.log for
Autotools packages) are archived relative to the stage base directory
rather than the expanded source directory.

Other packages (those not using CMake) will still use the staged
source directory as the default working directory for builds (and
will still be unable to reference this directory in
setup_environment())
2019-03-26 17:40:49 -05:00
ajw1980
e5b86c5527 Retrieve environment-modules prefix based on architecture (#10975)
When multiple instances of environment-modules were installed with
different architectures, Spack was not retrieving the installation
appropriate for the current architecture when finding the module
prefix.
2019-03-26 15:23:17 -05:00
Seth R. Johnson
844ca31894 Use 'shlex' to split default Executable arguments (#10929)
This allows shell commands for `spack edit` to be executed correctly if
they have quoted arguments.
2019-03-26 14:02:32 -05:00
Massimiliano Culpo
89b9880719 Add regression marker to pytest.ini (#11011) 2019-03-26 12:04:36 -05:00
Nichols A. Romero
66172f80f3 Fixed some issues with CUDA-Intel compiler conflicts. (#10924)
* Fixed some issues with CUDA-Intel compiler conflicts.

* Comment about expressing CUDA-compiler conflicts.

* More precise conflicts and also add support for Intel 19.0
2019-03-26 15:31:56 +01:00
Shahzeb Siddiqui
724acb6421 typo in documentation (#10976) 2019-03-21 14:29:40 -05:00
George Hartzell
aa1e70aab1 Minor commentary correction, TCL -> lmod (#10951) 2019-03-21 11:17:41 +01:00
Stephen McDowell
70e436e932 prevent UnboundLocalError when sourcing files (#10950)
Patch extracted from #7536 courtesy of @mgsternberg
2019-03-21 11:08:29 +01:00
Chris Green
9ad02685d9 Fix help text for --no-checksum. (#10932) 2019-03-19 15:02:06 -05:00
Adam J. Stewart
4e6285a19b
Ensure that every package has a description (#10896) 2019-03-14 20:49:07 -05:00
Chris Green
9b51fb09f1 Support VISUAL environment variable for editing. (#10898)
If the user has set the environment variable VISUAL, it will be used
in preference to EDITOR for all Spack editing activities. If VISUAL
is not set or fails (perhaps due to a lack of graphical editing
capabilities),EDITOR will be used instead. We fall back to one of
several common editors if neither bears fruit.

This feature has been tailored to:

* Provide identical behavior to the previous implementation in the
  case that VISUAL is not set.
* Not require any change to code utilizing the editor feature.
* Follow usual UNIX behavior concerning VISUAL and EDITOR.
2019-03-14 16:16:26 -05:00
Tristan Carel
c3662492de Do not use string module to be compatible with python 3 (#248) (#10667)
`string.find` is not part of Python 3 anymore.
2019-03-12 20:34:47 -05:00
Justin Stanley
2bea940e43 find_versions_of_archive: expand link search (#10758) 2019-03-12 20:19:46 -05:00
Nick Forrington
6bda37f542 Fix clearing EnvironmentModifications with python2 (#10791)
* Fix clearing EnvironmentModifications with python2

* Add EnvironmentModifications::clear unit test

Use re-assignment rather than del to clear array

* Fix flake issues
2019-03-12 20:12:51 -05:00
Shahzeb Siddiqui
d20b5ce2ec format change and typo in doc (#10848) 2019-03-11 13:19:13 -05:00
Massimiliano Culpo
a42fd7f276 Improved detection of Clang versions (#10316)
Fixes #10191

* Add more regular expressions to detect clang versions that were
  not being picked up
* Add a test for parsing versions from the output of Clang (this
  does not run Clang, but rather uses example outputs from Clang)
* Separate Clang version parsing into its own method (to make it
  easier to test)
2019-03-11 13:15:34 -05:00
Michael Kuhn
a1c91f3c07 Fix find_headers to also look for C++ headers and Fortran modules (#10798)
Currently, only C headers are considered, causing build failures for
packages depending on, e.g., netcdf-fortran and xerces-c. Additionally,
the regex used to look for the include path component did not consider
word boundaries, causing false matches.
2019-03-08 21:06:22 -06:00
Greg Becker
f4d4322a41
Create option to build compilers as needed (#10761)
* Create option to build missing compilers and add them to config before installing packages that use them
* Clean up kwarg passing for do_install, put compiler bootstrapping in separate method
2019-03-07 17:30:48 -08:00
Adam J. Stewart
f7223e54ed Make is a required dependency of Spack (#10386)
Update documentation on getting started to mention that a "make"
executable is required to build packages with Spack.
2019-03-06 19:46:54 -06:00
Patrick Gartung
1d4289afdd
This fixes a problem where the placeholder path was not in the first rpath entry.
* Rework of buildcache creation and install prefix checking using the functions introduced in
https://github.com/spack/spack/pull/9199

Instead of replacing rpaths with placeholder and then checking strings, make use of the functions
relocate.is_recocatable and relocate.is_file_relocatable to decide if a package needs the allow-root option.

This fixes a problem where the placeholder path was not in the first rpath entry. This was seen in c++ libraries and binaries because the compiler was outside the spack install base path and always appears first in the rpath.

Instead of checking the first rpath entry, all rpaths have the placeholder path and the old install path (if it exists) replaced with the new install path.

* flake8
2019-03-01 07:47:26 -06:00
Massimiliano Culpo
e3af8ed454 Added a sub-command to show if packages are relocatable (#9199)
* Added the `spack buildcache preview` sub-command

This is similar to `spack spec -I` but highlights which nodes in a DAG
are relocatable and which are not.

spec.tree has been generalized a little to accept a status function,
instead of always showing the install status

The current implementation works only for ELF, and needs to be
generalized to other platforms.

* Added a test to check if an executable is relocatable or not

This test requires a few commands to be present in the environment.
Currently it will run only under python 3.7 (which uses Xenial instead
of Trusty).

* Added tests for the 'buildcache preview' command.

* Fixed codebase after rebase

* Fixed the list of apt addons for Python 3.7 in travis.yaml

* Only check ELF executables and shared libraries. Skip checking virtual or external packages. (#229)

* Fixed flake8 issues

* Add handling for macOS mach binaries (#231)
2019-02-28 15:36:47 -06:00
Michael Kuhn
617c1a3706 Fix shell integration with environment-modules@4 (#10736) 2019-02-28 13:22:44 +01:00
Adam J. Stewart
a25edb51a9 Fix 'make test' detection when LANG is not in English (#10499) 2019-02-27 14:01:50 -06:00
Massimiliano Culpo
7eec038690
Update environment-modules package (#10717)
The environment modules package has been updated to include 
versions up to 4.0.0. The url of the package and the homepage
have been updated accordingly.

The `spack bootstrap` command now builds version 3.2.10 of
the environment-modules package, and will do until #10708 
is fixed.
2019-02-27 14:00:46 +01:00
Massimiliano Culpo
42386dbe94 Use Package.headers for -I options (#10623)
This restores the use of Package.headers when computing -I options
for building a package that was added in #8136 and reverted in
#10604. #8136 used utility logic that located all header files in
an installation prefix, and calculated the -I options as the
immediate roots containing those header files.

In some cases, for a package containing a directory structure like

  prefix/
    include/
	  ex1.h
	  subdir/
	    ex2.h

dependents may expect to include ex2.h relative to 'include', and
adding 'prefix/include/subdir' as a -I was causing errors,
in particular if ex2.h has the same name as a system header.

This updates header utility logic to by default return the base
"include" directory when it exists, rather than subdirectories.
It also makes it possible for package implementers to override
Package.headers to return the subdirectory when it is required
(for example with libxml2).
2019-02-26 12:42:30 -06:00
Daniel Topa
c832479c65 Add libhio v1.4.1.3, 1.4.1.1 (#10699)
- Switch all libhio tarball listings to sha256 checksums
- Correct typo in alert message in /lib/spack/spack/util/module_cmd.py: fucntion -> function

https://github.com/hpc/libhio/releases
Released 2019-02-01

Verification builds on LANL Darwin:

**Intel Xeon**
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                40
On-line CPU(s) list:   0-39
Thread(s) per core:    2
Core(s) per socket:    10
Socket(s):             2
NUMA node(s):          2
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 63
Model name:            Intel(R) Xeon(R) CPU E5-2660 v3 @ 2.60GHz
Stepping:              2
CPU MHz:               1198.779
CPU max MHz:           3300.0000
CPU min MHz:           1200.0000
BogoMIPS:              5193.70
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              25600K
NUMA node0 CPU(s):     0-9,20-29
NUMA node1 CPU(s):     10-19,30-39
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand lahf_lm abm cpuid_fault epb invpcid_single pti intel_ppin tpr_shadow vnmi flexpriority ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm xsaveopt cqm_llc cqm_occup_llc dtherm ida arat pln pts
`
[+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-centos7-x86_64/gcc-4.8.5/libhio-1.4.1.3-s4fnmesfp65trhks5qi3it5p73ssfpsp
[+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-centos7-x86_64/gcc-4.8.5/libhio-1.4.1.2-fkgh5vqpijvwqywffmokgmsglqxwfrtl
`

**Arm**
Architecture:          aarch64
Byte Order:            Little Endian
CPU(s):                256
On-line CPU(s) list:   0-255
Thread(s) per core:    4
Core(s) per socket:    32
Socket(s):             2
NUMA node(s):          2
Model:                 0
BogoMIPS:              400.00
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              32768K
NUMA node0 CPU(s):     0-127
NUMA node1 CPU(s):     128-255
Flags:                 fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics cpuid asimdrdm
`
[+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/libhio-1.4.1.3-q6nnwiy6bi7ktnghdsngwamom23zpmgy
[+] /scratch/users/dantopa/new-spack/strawman.pr.libhio/opt/spack/linux-rhel7-aarch64/gcc-4.8.5/libhio-1.4.1.2-y6nwovff3qbdy242zc4x2toloz6xpcvm
`

2019-02-25

Signed-off-by: Daniel Topa <dantopa@lanl.gov>
2019-02-26 09:56:20 +01:00
Adam J. Stewart
a75a6ee1b7 Add /hash to spack help --spec (#10691)
* Add /hash to spack help --spec
* Make /hash gray, add ^/hash
2019-02-25 16:20:28 -08:00
Peter Scheibel
98f1c9a291
CPATH warning: downgrade to debug message (#10675)
Spack warns users when a dependency package updates CPATH. This
warning message is generating bug reports and alarm in cases where
there is no problem. For now this downgrades the warning message to
the debug level, so it only shows up if something goes wrong for the
user and they ask for more information from Spack.
2019-02-21 17:26:38 -06:00
Scott Wittenburg
5600c9f0d9 release workflow: Add build scripts for jobs and means to upload pkgs 2019-02-21 15:37:35 -06:00