Commit graph

3560 commits

Author SHA1 Message Date
Peter Scheibel
c99d367a11 syaml_int type should use int.__repr__ rather than str.__repr__ (#13487) 2019-10-29 13:51:48 -07:00
Tamara Dahlgren
3cf21e6edc bugfix: make spack -d test test_changed_files work (#13229)
The `test_changed_files` in `test/cmd/flake8.py` was failing because it calls
`ArgumentParser.parse_args()` without arguments.  Normally that would just
parse `sys.argv` but it seems to fail because of something in either `spack test`
or `pytest`.  Call it with an empty array so that it doesn't try to touch`sys.argv`
at all.

- [x] allow `-d` spack option for `test_changed_files`
2019-10-29 00:25:59 -07:00
Adam J. Stewart
2264e30d99
Update URL parsing regexes and tests (#13411)
* Update URL parsing regexes and tests

* Get rid of no longer used README

* Merge py-udunits and py-cf-units

* netcdf -> netcdf-c

* setup_environment -> setup_*_environment

* Fix doc tests

* Few last minute fixes

* Simplify prefix removal copypasta
2019-10-28 20:27:54 -05:00
Adam J. Stewart
b4e027e918
Output directory Spack is looking in for MKL libs (#13465) 2019-10-28 11:46:59 -05:00
Todd Gamblin
8eeb64096c
version bump: 0.13.0 2019-10-25 21:53:52 -07:00
Todd Gamblin
847703c13d
tutorial: move tutorial to standalone site (#13450)
* docs: add a spack environment for building the docs

* docs: remove tutorial and link to spack-tutorial.readthedocs.io

The tutorial now has its own standalone website, versioned by instances
of the tutorial.  Link to that instead of versioning it directly with Spack.
2019-10-25 21:49:27 -07:00
Peter Scheibel
4af448724f mirror create --all can mirror everything (#12940)
Support mirroring all packages with `spack mirror create --all`.

In this mode there is no concretization:

* Spack pulls every version of every package into the created mirror.
* It also makes multiple attempts for each package/version combination
  (if there is a temporary connection failure).
* Continues if all attempts fail. i.e., this makes its best effort to
  fetch evrerything, even if all attempts to fetch one package fail.

This also changes mirroring logic to prefer storing sources by their hash
or by a unique name derived from the source.  For example:

* Archives with checksums are named by the sha256 sum, i.e.,
  `archive/f6/f6cf3bd233f9ea6147b21c7c02cac24e5363570ce4fd6be11dab9f499ed6a7d8.tar.gz`
  vs the previous `<package-name>-package-version>.tar.gz`
* VCS repositories are stored by a path derived from their URL,
  e.g. `git/google/leveldb.git/master.tar.gz`.

The new mirror layout allows different packages to refer to the same
resource or source without duplicating that download in the
mirror/cache. This change is not essential to mirroring everything but is
expected to save space when mirroring packages that all use the same
resource.

The new structure of the mirror is:

```
<base directory>/
  _source-cache/   <-- the _source-cache directory is new
    archive/       <-- archives/resources/patches stored by hash
      00/          <-- 2-letter sha256 prefix
        002748bdd0319d5ab82606cf92dc210fc1c05d0607a2e1d5538f60512b029056.tar.gz
      01/
        0154c25c45b5506b6d618ca8e18d0ef093dac47946ac0df464fb21e77b504118.tar.gz
        0173a74a515211997a3117a47e7b9ea43594a04b865b69da5a71c0886fa829ea.tar.gz
        ...
    git/
      OpenFAST/
        openfast.git/
          master.tar.gz     <-- repo by branch name
      PHASTA/
        phasta.git/
          11f431f2d1a53a529dab4b0f079ab8aab7ca1109.tar.gz  <-- repo by commit
      ...
    svn/      <-- each fetch strategy has its own subdirectory
      ...
  openmpi/   <-- the remaining package directories have the old format
    openmpi-1.10.1.tar.gz  <-- human-readable name is symlink to _source-cache
```

In addition to the archive names as described above, `mirror create` now
also creates symlinks with the old format to help users understand which
package each mirrored archive is associated with, and to allow mirrors to
work with old spack versions. The symlinks are relative so the mirror
directory can still itself be archived.

Other improvements:

* `spack mirror create` will not re-download resources that have already
  been placed in it.

* When creating a mirror, the resources downloaded to the mirror will not
  be cached (things are not stored twice).
2019-10-25 21:47:28 -07:00
Peter Scheibel
cfbac14cba bugfix: restore upstream lock safety; update tests
Restore upstream lock safety; avoid calling methods directly on upstream DB in test.
2019-10-25 16:01:45 -07:00
Andreas Baumbach
ab55890ba8 Make error msg of test_api_for_build_and_run_environment more informative (#13435) 2019-10-25 19:24:41 +02:00
Todd Gamblin
ebeb1ed62f
bugfix: reindexing is not necessary for DB v0.9.3 to v5 upgrade (#13434)
reindexing takes a significant amount of time, and there's no reason to
do it from DB version 0.9.3 to version 5. The only difference is that v5
can contain "deprecated_for" fields.

- [x] Add a `_skip_reindex` list at the start of `database.py`

- [x] Skip the reindex for upgrades in this list. The new version will
  just be written to the file the first time we actually have to write
  the DB out (e.g., after an install), and reads will still work fine.
2019-10-25 07:10:03 -07:00
Todd Gamblin
dbee91f7f1
bugfix: allow fetching no-code packages (#13429)
Previously, spack would error out if we tried to fetch something with no
code, but that would prevent fetching dependencies. In particular, this
would fail:

    spack fetch --dependencies xsdk

- [x] Instead of raising an error, just print a message that there is nothing
      to be fetched for packages like xsdk that do not have code.

- [x] Make BundleFetchStrategy a bit more quiet about doing nothing.
2019-10-24 16:02:28 -07:00
Todd Gamblin
757387dc2a
commands: Add --json argument to spack spec (#13431)
We've had `spack spec --yaml` for a while, and we've had methods for JSON
for a while as well.  We just haven't has a `--json` argument for `spack spec`.

- [x] Add a `--json` argument to `spack spec`, just like `--yaml`
2019-10-24 16:01:31 -07:00
Chris Green
77af4684aa Improvements to detection of AMD architectures. (#13407)
New entry for K10 microarchitecture.

Reorder Zen* microarchitectures to avoid triggering as k10.

Remove some desktop-specific flags that were preventing Opteron Bulldozer/Piledriver/Steamroller/Excavator CPUs from being recognized as such.

Remove one or two flags which weren't produced in /proc/cpuinfo on older OS (RHEL6 and friends).
2019-10-24 15:48:54 -07:00
Chris Green
0913328812 Correctly identify Skylake CPUs on Darwin. (#13377)
* Correctly identify Skylake CPUs on Darwin.

* Add a test for haswell on Mojave.
2019-10-24 12:44:58 -05:00
Greg Becker
f65f2450df database: update DB version and change to integer database versions (#13410)
Update on version format: change to an integer database format version for simplicity, instead of tracking the Spack version.
2019-10-24 10:43:49 -07:00
Nick Forrington
845df79ac6 Patch libtool when using the Arm compiler (#12004)
* Patch libtool when using the arm, clang, and fujitsu compilers

If libtool does not have values for linker/pic flags, patch them in
2019-10-24 09:51:00 -05:00
Greg Becker
8c7a3e55dd add spack dev-build command; deprecate spack diy (#13374)
Rename the `spack diy` command to `spack dev-build` to make the use case clearer.

The `spack diy` command has some useful functionality for developers using Spack to build their dependencies and configure/build/install the code they are developing.  Developers do not notice it, partly because of the obscure name.

The `spack dev-build` command has a `-u/--until PHASE` option to stop after a given phase of the build. This can be used to configure your project, run cmake on your project, or similarly stop after any stage of the build the user wants. These options are analogous to the existing `spack configure` and `spack build` commands, but for developer builds.

To unify the syntax, we have deprecated the `spack configure` and `spack build` commands, and added a `-u/--until PHASE` option to the `spack install` command as well.

The functionality in `spack dev-build` (specifically `spack dev-build -u cmake`) may be able to supersede the `spack setup` command, but this PR does not deprecate that command as that will require slightly more thought.
2019-10-23 17:08:15 -07:00
Peter Scheibel
f2ddffb840 bugfix: web.push_to_url should not format the local path that is passed to it (#13408)
fd58c98 formats the `Stage`'s `archive_path` in `Stage.archive` (as part of `web.push_to_url`). This is not needed and if the formatted differs from the original path (for example if the archive file name contains a URL query suffix), then the copy fails.

This removes the formatting that occurs in `web.push_to_url`.

We should figure out a way to handle bad cases like this *and* to have nicer filenames for downloaded files.  One option that would work in this particular case would be to also pass `-J` / `--remote-header-name` to `curl`.  We'll need to do follow-up work to determine if we can use `-J` everywhere.

See also: https://github.com/spack/spack/pull/11117#discussion_r338301058
2019-10-23 16:50:51 -07:00
Peter Scheibel
685946b1f1 syaml.dump now allows representing all syaml_ objects (originally this was only available for syaml.dump_config) (#13403) 2019-10-23 14:24:14 -07:00
Massimiliano Culpo
b29eb4212e Users can configure use of RPATH or RUNPATH (#9168)
Add a new entry in `config.yaml`:

    config:
        shared_linking: 'rpath'

If this variable is set to `rpath` (the default) Spack will set RPATH in ELF binaries. If set to `runpath` it will set RUNPATH.

Details:
* Spack cc wrapper explicitly adds `--disable-new-dtags` when linking
* cc wrapper also strips `--enable-new-dtags` from the compile line
    when disabling (and vice versa)
* We specifically do *not* add any dtags flags on macOS, which uses
    Mach-O binaries, not ELF, so there's no RUNPATH)
2019-10-23 13:22:24 -07:00
Greg Becker
cd185c3d28 commands: add spack deprecate command (#12933)
`spack deprecate` allows for the removal of insecure packages with minimal impact to their dependents. It allows one package to be symlinked into the prefix of another to provide seamless transition for rpath'd and hard-coded applications using the old version.

Example usage:

    spack deprecate /hash-of-old-openssl /hash-of-new-openssl

The spack deprecate command is designed for use only in extroardinary circumstances.  The spack deprecate command makes no promises about binary compatibility. It is up to the user to ensure the replacement is suitable for the deprecated package.
2019-10-23 13:11:35 -07:00
Todd Gamblin
26ec644fc8 url summary: show right and wrong parse counts for each regex
Previously this command only showed total counts for each regular
expression.  This doesn't give you a sense of which regexes are working
well and which ones are not.  We now display the number of right, wrong,
and total URL parses per regex.

It's easier to see where we might improve the URL parsing with this
change.
2019-10-23 09:10:32 -07:00
Todd Gamblin
af65146ef6 Preserve comments for Spack YAML objects (#11602)
This updates the configuration loading/dumping logic (now called
load_config/dump_config) in spack_yaml to preserve comments (by using
ruamel.yaml's RoundTripLoader). This has two effects:

* environment spack.yaml files expect to retain comments, which
  load_config now supports. By using load_config, users can now use the
  ':' override syntax that was previously unavailable for environment
  configs (but was available for other config files).

* config files now retain user comments by default (although in cases
  where Spack updates/overwrites config, the comments can still be
  removed).

Details:

* Subclasses `RoundTripLoader`/`RoundTripDumper` to parse yaml into
  ruamel's `CommentedMap` and analogous data structures

* Applies filename info directly to ruamel objects in cases where the
  updated loader returns those

* Copies management of sections in `SingleFileScope` from #10651 to allow
  overrides to occur

* Updates the loader/dumper to handle the processing of overrides by
  specifically checking for the `:` character
  * Possibly the most controversial aspect, but without that, the parsed
    objects have to be reconstructed (i.e. as was done in
    `mark_overrides`). It is possible that `mark_overrides` could remain
    and a deep copy will not cause problems, but IMO that's generally
    worth avoiding.
  * This is also possibly controversial because Spack YAML strings can
    include `:`. My reckoning is that this only occurs for version
    specifications, so it is safe to check for `endswith(':') and not
    ('@' in string)`
  * As a consequence, this PR ends up reserving spack yaml functions
    load_config/dump_config exclusively for the purpose of storing spack
    config
2019-10-23 06:29:49 -07:00
Todd Gamblin
b3f5084b96
tests: avoid extra output in env status test (#13344)
`test_envoronment_status()` was printing extra output during tests.

- [x] disable output only for `env('status')` calls instead of disabling
  it for the whole test.
2019-10-22 23:59:28 -07:00
Greg Becker
95a48b27ec env activation: use package defined env setup methods (#13249)
This PR ensures that environment activation sets all environment variables set by the equivalent `module load` operations, except that the spec prefixes are "rebased" to the view associated with the environment.

Currently, Spack blindly adds paths relative to the environment view root to the user environment on activation. Issue #12731 points out ways in which this behavior is insufficient.

This PR changes that behavior to use the `setup_run_environment` logic for each package to augment the prefix inspections (as in Spack's modulefile generation logic) to ensure that all necessary variables are set to make use of the packages in the environment.

See #12731 for details on the previous problems in behavior.

This PR also updates the `ViewDescriptor` object in `spack.environment` to have a `__contains__` method. This allows for checks like `if spec in self.default_view`. The `__contains__` operator for `ViewDescriptor` objects checks whether the spec satisfies the filters of the View descriptor, not whether the spec is already linked into the underlying `FilesystemView` object.
2019-10-22 23:27:40 -07:00
Massimiliano Culpo
b14f18acda microarchitectures: look in /sbin and /usr/sbin for sysctl (#13365)
This PR ensures that on Darwin we always append /sbin and /usr/sbin to PATH, if they are not already present, when looking for sysctl.

* Make sure we look into /sbin and /usr/sbin for sysctl
* Refactor sysctl for better readability
* Remove marker to make test pass
2019-10-22 21:42:38 -07:00
Massimiliano Culpo
8808207ddf Fixed optimization flags support for old GCC versions (#13362)
These changes update our gcc microarchitecture descriptions based on manuals found here https://gcc.gnu.org/onlinedocs/ and assuming that new architectures are not added during patch releases.
2019-10-22 21:40:45 -07:00
Massimiliano Culpo
cfbdd2179e microarchitectures: add optimization flags for Intel compilers (#13345)
* Added optimization flags for Intel compilers with Intel CPUs
* Added optimization flags for Intel compilers with AMD CPUs
2019-10-22 00:33:59 -07:00
Omar Padron
fd58c98b0e fetching: S3 upload and download (#11117)
This extends Spack functionality so that it can fetch sources and binaries from-, push sources and binaries to-, and index the contents of- mirrors hosted on an S3 bucket.

High level to-do list:

- [x] Extend mirrors configuration to add support for `file://`, and `s3://` URLs.
- [x] Ensure all fetching, pushing, and indexing operations work for `file://` URLs.
- [x] Implement S3 source fetching
- [x] Implement S3 binary mirror indexing
- [x] Implement S3 binary package fetching
- [x] Implement S3 source pushing
- [x] Implement S3 binary package pushing

Important details:

* refactor URL handling to handle S3 URLs and mirror URLs more gracefully.
  - updated parse() to accept already-parsed URL objects.  an equivalent object
    is returned with any extra s3-related attributes intact.  Objects created with
    urllib can also be passed, and the additional s3 handling logic will still be applied.

* update mirror schema/parsing (mirror can have separate fetch/push URLs)
* implement s3_fetch_strategy/several utility changes
* provide more feature-complete S3 fetching
* update buildcache create command to support S3

* Move the core logic for reading data from S3 out of the s3 fetch strategy and into
  the s3 URL handler.  The s3 fetch strategy now calls into `read_from_url()` Since
  read_from_url can now handle S3 URLs, the S3 fetch strategy is redundant.  It's
  not clear whether the ideal design is to have S3 fetching functionality in a fetch
  strategy, directly implemented in read_from_url, or both.

* expanded what can be passed to `spack buildcache` via the -d flag: In addition
  to a directory on the local filesystem, the name of a configured mirror can be
  passed, or a push URL can be passed directly.
2019-10-22 00:32:04 -07:00
Massimiliano Culpo
3d77ecd92e Bootstrap environment modules optimizing for generic architectures (#13105)
fixes #13073

Since #3206 was merged bootstrapping environment-modules was using the architecture of the current host or the best match supported by the default compiler. The former case is an issue since shell integration was looking for a spec targeted at the host microarchitecture.

1. Bootstrap an env modules targeted at generic architectures
2. Look for generic targets in shell integration scripts
3. Add a new entry in Travis to test shell integration
2019-10-21 11:20:05 -07:00
Massimiliano Culpo
498f448ef3 microarchitectures: fix custom compiler versions (#13222)
Custom string versions for compilers were raising a ValueError on
conversion to int. This commit fixes the behavior by trying to detect
the underlying compiler version when in presence of a custom string
version.

* Refactor code that deals with custom versions for better readability
* Partition version components with a regex
* Fix semantic of custom compiler versions with a suffix
* clang@x.y-apple has been special-cased
* Add unit tests
2019-10-21 10:24:57 -07:00
Adam J. Stewart
596f3f1d61
spack create: http -> https (#13269) 2019-10-20 11:05:54 -05:00
Todd Gamblin
20bb6fd733
shebangs: don't warn when patching long shebangs (#13266)
We've been doing this for quite a while now, and it does not seem to
cause issues.

- [x] Switch the noisy warning to a debug to make Spack a bit quieter
  while building.
2019-10-19 14:09:25 -07:00
Massimiliano Culpo
41fb0395a6 Microarchitecture specific optimizations for LLVM (#13250)
* Added architecture specific optimization flags for Clang / LLVM
* Disallow compiler optimizations for mixed toolchains
    * We emit a warning when building for a mixed toolchain
* Fixed issues with suffixed versions of compilers; Apple's Clang will, 
    for the time being, fall back on x86-64 for every compilation.
2019-10-19 13:19:29 -07:00
Peter Scheibel
fd88f8fefa satisfaction checks on dependencies should be strict when the parent check is strict (#13243) 2019-10-17 11:02:14 -07:00
Massimiliano Culpo
9ddc98e46a Separate setting build environment and run environment in packages (#11115)
* Methods setting the environment now do it separately for build and run

Before this commit the `*_environment` methods were setting
modifications to both the build-time and run-time environment
simultaneously. This might cause issues as the two environments
inherently rely on different preconditions:

1. The build-time environment is set before building a package, thus
the package prefix doesn't exist and can't be inspected

2. The run-time environment instead is set assuming the target package
has been already installed

Here we split each of these functions into two: one setting the
build-time environment, one the run-time.

We also adopt a fallback strategy that inspects for old methods and
executes them as before, but prints a deprecation warning to tty. This
permits to port packages to use the new methods in a distributed way,
rather than having to modify all the packages at once.

* Added a test that fails if any package uses the old API

Marked the test xfail for now as we have a lot of packages in that
state.

* Added a test to check that a package modified by a PR is up to date

This test can be used any time we deprecate a method call to ensure
that during the first modification of the package we update also
the deprecated calls.

* Updated documentation
2019-10-17 10:17:21 -07:00
Todd Gamblin
cf9de058aa
multimethods: avoid calling caller_locals() in Python 3 (#13238)
Python 3 metaclasses have a `__prepare__` method that lets us save the
class's dictionary before it is constructed.  In Python 2 we had to walk
up the stack using our `caller_locals()` method to get at this.  Using
`__prepare__` is much faster as it doesn't require us to use `inspect`.

This makes multimethods use the faster `__prepare__` method in Python3,
while still using `caller_locals()` in Python 2.  We try to reduce the
use of caller locals using caching to speed up Python 2 a little bit.
2019-10-17 06:40:23 -07:00
Todd Gamblin
e65b7f8ebf
importing: make importlib_importer recognize .pyc cache (#13239)
Our importer was always parsing from source (which is considerably
slower) because the source size recorded in the .pyc file differed from
the size of the input file.

Override path_stats in the prepending importer to fool it into thinking
that the source size is the size *with* the prepended code.
2019-10-16 17:07:18 -07:00
Michael Kuhn
ffe87ed49f filter_file: fix multiple invocations on the same file (#13234)
Since the backup file is only created on the first invocation, it will
contain the original file without any modifications. Further invocations
will then read the backup file, effectively reverting prior invocations.

This can be reproduced easily by trying to install likwid, which will
try to install into /usr/local. Work around this by creating a temporary
file to read from.
2019-10-16 15:15:24 -07:00
Tamara Dahlgren
1ef71376f2 Bugfix: stage directory permissions and cleaning (#12733)
* This updates stage names to use "spack-stage-" as a prefix.
  This avoids removing non-Spack directories in "spack clean" as
  c141e99 did (in this case so long as they don't contain the
  prefix "spack-stage-"), and also addresses a follow-up issue
  where Spack stage directories were not removed.
* Spack now does more-stringent checking of expected permissions for
  staging directories. For a given stage root that includes a user
  component, all directories before the user component that are
  created by Spack are expected to match the permissions of their
  parent; the user component and all deeper directories are expected
  to be accessible to the user (read/write/execute).
2019-10-16 14:55:37 -07:00
Greg Becker
94e80933f0 Feature: installed file verification (#12841)
This feature generates a verification manifest for each installed
package and provides a command, "spack verify", which can be used to
compare the current file checksums/permissions with those calculated
at installed time.

Verification includes

* Checksums of files
* File permissions
* Modification time
* File size

Packages installed before this PR will be skipped during verification.
To verify such a package you must reinstall it.

The spack verify command has three modes.

* With the -a,--all option it will check every installed package.
* With the -f,--files option, it will check some specific files,
  determine which package they belong to, and confirm that they have
  not been changed.
* With the -s,--specs option or by default, it will check some
  specific packages that no files havae changed.
2019-10-15 14:24:52 -07:00
Massimiliano Culpo
d33b0ffc50 lmod: module files are written in a root folder named by target family (#13121)
fixes #13005

This commit fixes an issue with the name of the root directory for
module file hierarchies. Since #3206 the root folder was named after
the microarchitecture used for the spec, which is too specific and
not backward compatible for lmod hierarchies. Here we compute the
root folder name using the target family instead of the target name
itself and we add target information in the 'whatis' portion of the
module file.
2019-10-15 11:20:49 -07:00
Massimiliano Culpo
5cd28847e8 filter_file uses "surrogateescape" error handling (#12765)
From Python docs:
--
'surrogateescape' will represent any incorrect bytes as code points in
the Unicode Private Use Area ranging from U+DC80 to U+DCFF. These
private code points will then be turned back into the same bytes when
the surrogateescape error handler is used when writing data. This is
useful for processing files in an unknown encoding.
--

This will allow us to process files with unknown encodings.

To accommodate the case of self-extracting bash scripts, filter_file
can now stop filtering text input if a certain marker is found. The
marker must be passed at call time via the "stop_at" function argument.
At that point the file will be reopened in binary mode and copied
verbatim.

* use "surrogateescape" error handling to ignore unknown chars
* permit to stop filtering if a marker is found
* add unit tests for non-ASCII and mixed text/binary files
2019-10-14 20:35:14 -07:00
Greg Becker
3f46f03c83
bugfix: install --only dependencies works in env (#13090)
* bugfix: install --only dependents works in env

includes regression testing
2019-10-14 17:50:38 -07:00
Todd Gamblin
385d2bf3fa
commands: add resource stats to spack url stats (#13205)
`spack url stats` now also looks at packages' resources when outputting
statistics.

Example:

```
$ spack url stats
==> URL stats for 3531 packages:
--------------------------------------------------------------
stat                    versions        %   resources        %
--------------------------------------------------------------
url                         8335    89.3%         339    89.0%
    schemes
        https               6489    69.5%          93    24.4%
        ftp                   32     0.3%           8     2.1%
        http                1763    18.9%         237    62.2%
        file                  51     0.5%           1     0.3%
    checksums
        md5                   26     0.3%           0     0.0%
        sha256              8306    89.0%         336    88.2%
        no checksum            3     0.0%           3     0.8%
--------------------------------------------------------------
go                             1     0.0%           0     0.0%
--------------------------------------------------------------
hg                             7     0.1%           0     0.0%
--------------------------------------------------------------
no code                        4     0.0%           0     0.0%
--------------------------------------------------------------
svn                            4     0.0%          16     4.2%
--------------------------------------------------------------
git                          981    10.5%          26     6.8%
    branch                   442     4.7%           4     1.0%
    commit                   362     3.9%          14     3.7%
    no ref                    36     0.4%           2     0.5%
    tag                      141     1.5%           6     1.6%
--------------------------------------------------------------
```
2019-10-14 08:58:01 -07:00
Adam J. Stewart
e1cf35a82b Fix svn tests, skip if svnadmin not found (#13137) 2019-10-14 10:40:20 +02:00
Greg Becker
1808ba3c68 install: add --cache-only option (#12729)
* add `--cache-only` option to install
* testing for `--cache-only`
* remove extraneous stage creation at stage destroy time
2019-10-12 11:43:37 -07:00
Todd Gamblin
64bdc3251f checksums: enforce that all mainline packages use sha256 checksums
- Add a test that verifies checksums on all packages

- Also add an attribute to packages that indicates whether they need a
  manual download or not, and add an exception in the tests for these
  packages until we can verify them.
2019-10-12 07:19:43 -07:00
Massimiliano Culpo
f8a4a3802c Autotools build system to patch config.guess based on a range check 2019-10-11 21:21:57 -07:00
Massimiliano Culpo
8dd95c1705 Fixed options to compile generic code on ppc64 and ppc64le 2019-10-11 21:20:28 -07:00
Patrick Gartung
d248b0e9d5
Fix python3 errors from string and byte concatenation (#13141) 2019-10-11 03:21:45 -05:00
Tamara Dahlgren
93a44c822c tests: cleanup config:build_stage handling (fixes #12651, #12798) 2019-10-10 16:50:05 -07:00
Tamara Dahlgren
87cdfa2c25 Add support for nested "overrides" scopes. 2019-10-10 16:50:05 -07:00
Massimiliano Culpo
b07460ab5f Added NEON to the list of features required for the aarch64 family
Both floating-point and NEON are required in all standard ARMv8
implementations. Theoretically though specialized markets can support
no NEON or floating-point at all. Source:

https://developer.arm.com/docs/den0024/latest/aarch64-floating-point-and-neon

On the other hand the base procedure call standard for Aarch64
"assumes the availability of the vector registers for passing
floating-point and SIMD arguments". Further "the Arm 64-bit
architecture defines two mandatory register banks: a general-purpose
register bank which can be used for scalar integer processing and
pointer arithmetic; and a SIMD and Floating-Point register bank".
Source:

https://developer.arm.com/docs/ihi0055/latest/procedure-call-standard-for-the-arm-64-bit-architecture

This makes customization of Aarch64 with no NEON instruction set
available so unlikely that we can consider them a feature of the
generic family.
2019-10-10 16:24:36 -07:00
Massimiliano Culpo
77444dff10 ArchSpec: fix constraint satisfaction for targets
fixes #13111

Due to a missing case we were treating a single target that was not
equal to the one we were comparing to as a range open on the right.
2019-10-10 16:22:38 -07:00
Patrick Gartung
5b879459c7
Buildcache: pass string.encode('utf-8') for old_dir and new_dir to replace_prefix_bin. (#13114)
This should fix a Python3 error from concatenating strings and bytes.
2019-10-09 20:36:47 -05:00
Massimiliano Culpo
acbb2af528 "No Spack mirror configured": demoted the warning to a debug message (#13082)
fixes #12010
2019-10-09 07:13:37 -05:00
Adam J. Stewart
72d8febc4e Add macOS Catalina support (#13070) 2019-10-07 23:59:33 -07:00
Massimiliano Culpo
9faee51e22 Spack environments can concretize specs together (#11372)
This PR adds a 'concretize' entry to an environment's spec.yaml file
which controls how user specs are concretized. By default it is
set to 'separately' which means that each spec added by the user is
concretized separately (the behavior of environments before this PR).
If set to 'together', the environment will concretize all of the
added user specs together; this means that all specs and their
dependencies will be consistent with each other (for example, a
user could develop code linked against the set of libraries in the
environment without conflicts).

If the environment was previously concretized, this will re-concretize
all specs, in which case previously-installed specs may no longer be
used by the environment (in this sense, adding a new spec to an
environment with 'concretize: together' can be significantly more
expensive).

The 'concretize: together' setting is not compatible with Spec
matrices; this PR adds a check to look for multiple instances of the
same package added to the environment and fails early when
'concretize: together' is set (to avoid confusing messages about
conflicts later on).
2019-10-07 09:53:23 -07:00
Pariksheet Nanda
19e6f55647 doc: fix #12245 non-functional libdwarf dependency (#12515)
Applying accepted fix from spack/spack.io#4
2019-10-05 21:07:18 -05:00
Michael Kuhn
bdb92f2d91 Consistently support pkg-config files in share subdirectory (#12838)
While the build environment already takes share/pkgconfig into account,
the generated module files etc. only consider lib/pkgconfig and
lib64/pkgconfig.
2019-10-05 21:03:35 -05:00
Massimiliano Culpo
6a1021b81c bugfix: issue with custom dotkit root in config.yaml (#13046)
When removing support for dotkit in #11986 the code trying to set the
paths of the various module files was not updated to skip it. This
results in a failure because of a key error after the deprecation
warning is displayed to user.

This commit fixes the issue and adds a unit test for regression.
Note that code for Spack chains has been updated accordingly but
no unit test has been added for that case.
2019-10-04 23:42:21 -07:00
Adam J. Stewart
24f8d1042b
Update compilers.yaml location in Getting Started docs (#13029) 2019-10-03 10:31:32 -05:00
Massimiliano Culpo
78577c0a90
Generic x86_64 code compiled with GCC uses non deprecated mtune flags (#13022)
fixes #12928
2019-10-03 10:31:03 +02:00
Massimiliano Culpo
76b9c56110 Remove support for generating dotkit files (#11986)
Dotkit is being used only at a few sites and has been deprecated on new
machines. This commit removes all the code that provide support for the
generation of dotkit module files.

A new validator named "deprecatedProperties" has been added to the
jsonschema validators. It permits to prompt a warning message or exit
with an error if a property that has been marked as deprecated is
encountered.

* Removed references to dotkit in the docs
* Removed references to dotkit in setup-env-test.sh
* Added a unit test for the 'deprecatedProperties' schema validator
2019-10-02 22:15:01 -07:00
t-karatsu
a3a0e1cd2e fujitsu compiler: Add 'required_libs'. (#13014) 2019-10-02 11:55:08 -05:00
Patrick Gartung
2a739ed03e
Replace expensive store.reindex() call with db.add() call. (#13021) 2019-10-02 11:30:28 -05:00
Patrick Gartung
9dfec4a57c
'spack buildcache list' should show all buildaches available. (#13002)
* binary cache: show all packages for compatible differing targets

* Don't restrict spack buildcache list to arch or os

* Fix from merge conflict
2019-10-01 22:54:22 -05:00
Massimiliano Culpo
1b18ec90ab Add all compatible system types directory to module paths
fixes #12915
closes #12916

Since Spack has support for specific targets it might happen that
software is built for targets that are not exactly the host because
it was either an explicit user request or the compiler being used is
too old to support the host.

Modules for different targets are written into different directories
and by default Spack was adding to MODULEPATH only the directory
corresponding to the current host. This PR modifies this behavior to
add all the directories that are **compatible** with the current host.
2019-10-01 19:18:27 -07:00
Greg Becker
b92977492d binary cache: show all packages for compatible differing targets (#12943) 2019-10-01 17:03:28 -05:00
Jeffrey Salmond
76178f31ca When removing a file from a view, don't fail if it doesn't exist (#12960)
Sometimes when remove_file is called on a link, that link is missing
(perhaps ctrl-C happened halfway through a previous action). As
removing a non-existent file is no problem, this patch changes the
behavior so Spack continues rather than stopping with an error.

Currently you would see

  ValueError: /path/to/dir is not a link tree!

and now it continues with a warning.
2019-10-01 10:29:10 -07:00
Todd Gamblin
a12c032509 make license check slightly more lenient
bin/spack now needs to have a "-*- python -*-" line after the shebang, so
that emacs will interpret it as a python file instead of as a shell
script.  Add one line to the license check limit to accommodate this.
2019-09-29 09:32:04 -07:00
Massimiliano Culpo
9117dfd118 Add all the 'generic' architectures that are mentioned in recipes (#12958)
LLVM, mesa and other packages check for these generic
microarchitectures. One solution is to let Spack know they exist.
2019-09-28 21:47:05 -07:00
Glenn P Johnson
08a6577861 Fix perl build when using Build.PL
This fixes #12852 where perl builds that use Build.PL will fail when the
shebang of the Build script produced from the configure step is too
long.
2019-09-26 13:38:13 -07:00
Patrick Gartung
7dae058f91
Relocate mach-o binaries using macholib on linux. (#12946)
Changes deps and rpaths for bins and libs, changes id for libs.
2019-09-26 13:04:58 -05:00
eugeneswalker
950338aa92 add --no-deps opt to buildcache-create (#12956) 2019-09-26 12:23:59 -05:00
Patrick Gartung
321e956fa9
External: add macholib and altgraph needed to relocate Mach-o binaries on Linux (#12909) 2019-09-26 11:48:22 -05:00
Adam J. Stewart
065cbe89fe Fix "specific target" detection in Python 3 (#12906)
The output of subprocess.check_output is a byte string in Python 3. This causes dictionary lookup to fail later on.

A try-except around this function prevented this error from being noticed. Removed this so that more errors can propagate out.
2019-09-24 09:47:54 -07:00
Patrick Gartung
6dd9dbc071
Change get_patchelf to immediately return patchelf path if found (#12925) 2019-09-24 08:38:19 -05:00
t-karatsu
9aca0a17f4 Fujitsu compilers: added 'verbose_flag' method (#12922) 2019-09-24 11:17:29 +02:00
Todd Gamblin
18d63a239f bugfix: use string keys to set preferred targets (#12921)
Preferred targets were failing because we were looking them up by
Microarchitecture object, not by string.

- [x] Add a call to `str()` to fix target lookup.
- [x] Add a test to exercise this part of concretization.
- [x] Add documentation for setting `target` in `packages.yaml`
2019-09-24 10:18:48 +02:00
Massimiliano Culpo
2468ccee58 AMD: fix architecture hierarchy (zen) (#12913)
* microarchitectures: zen starts from x86_64, not from excavator
* Unit tests: fixed a test that is wrong with the new modeling
* microarchitectures: fixed features and inheritance for 15h family

bulldozer doesn't inherit from barcelona (10h) + added xop, lwp and tbm
instruction sets to the 15h family (it distinguish the family from 17h)
2019-09-23 21:54:13 -07:00
Adam J. Stewart
d6555f64b0
Fix detection of Apple Clang 11.0.0 (#12912) 2019-09-23 17:25:33 -05:00
Tamara Dahlgren
08e77e1b41
tests: more template creation tests (#12882)
Addresses #12804 

This PR adds the creation of the remaining (16) templates to ensure we can create them with expected content. The goal is to facilitate catching during testing.
2019-09-23 10:18:50 -07:00
Todd Gamblin
b4e148b562 externals: add note to jsonschema about modifications (#12895) 2019-09-22 09:43:57 -05:00
Todd Gamblin
7a1dd517b8 externals: avoid importing requests in jsonschema
Spack doesn't need `requests`, and neither does `jsonschema`, but
`jsonschema` tries to import it, and it'll succeed if `requests` is on
your machine (which is likely, given how popular it is).  This commit
removes the import to improve Spack's startup time a bit.

On a mac with SSD, the import of requests is ~28% of Spack's startup time
when run as `spack --print-shell-vars sh,modules` (.069 / .25 seconds),
which is what `setup-env.sh` runs.

On a Linux cluster where Python is mounted from NFS, this reduces
`setup-env.sh` source time from ~1s to .75s.

Note: This issue will be eliminated if we upgrade to a newer `jsonschema`
(we'd need to drop Python 2.6 for that).  See
https://github.com/Julian/jsonschema/pull/388.
2019-09-21 17:57:36 -07:00
Scott Wittenburg
db4d52d923 Fix how 'gpg --list-secret-keys ...' output is parsed 2019-09-20 11:33:58 -06:00
Gregory Becker
c43f105359 targets: add mic_knl target to microarchitectures.json
- This is needed to support Cray machines -- we need an architecture
  mic_knl > x86_64

- We used Cray's naming scheme for this target to make it work seamlessly
  with the module-based detection sccheme on Cray.  mic_knl is pretty
  much dead, so this will be the last succh target.  We will need to work
  wtih Cray and other vendors in the future.
2019-09-20 00:51:37 -07:00
Massimiliano Culpo
7daf860991 targets: adjust packages to use new specific targets semantics
Seamless translation from 'target=<generic>' to either
- target.family == <generic> (in methods)
- 'target=<generic>:' (in directives)

Also updated docs to show ranges in directives.
2019-09-20 00:51:37 -07:00
Massimiliano Culpo
3c4322bf1a targets: Spack targets can now be fine-grained microarchitectures
Spack can now:

- label ppc64, ppc64le, x86_64, etc. builds with specific
  microarchitecture-specific names, like 'haswell', 'skylake' or
  'icelake'.

- detect the host architecture of a machine from /proc/cpuinfo or similar
  tools.

- Understand which microarchitectures are compatible with which (for
  binary reuse)

- Understand which compiler flags are needed (for GCC, so far) to build
  binaries for particular microarchitectures.

All of this is managed through a JSON file (microarchitectures.json) that
contains detailed auto-detection, compiler flag, and compatibility
information for specific microarchitecture targets.  The `llnl.util.cpu`
module implements a library that allows detection and comparison of
microarchitectures based on the data in this file.

The `target` part of Spack specs is now essentially a Microarchitecture
object, and Specs' targets can be compared for compatibility as well.
This allows us to label optimized binary packages at a granularity that
enables them to be reused on compatible machines.  Previously, we only
knew that a package was built for x86_64, NOT which x86_64 machines it
was usable on.

Currently this feature supports Intel, Power, and AMD chips. Support for
ARM is forthcoming.

Specifics:

- Add microarchitectures.json with descriptions of architectures

- Relaxed semantic of compiler's "target" attribute.  Before this change
  the semantic to check if a compiler could be viable for a given target
  was exact match. This made sense as the finest granularity of targets
  was architecture families.  As now we can target micro-architectures,
  this commit changes the semantic by interpreting as the architecture
  family what is stored in the compiler's "target" attribute. A compiler
  is then a viable choice if the target being concretized belongs to the
  same family. Similarly when a new compiler is detected the architecture
  family is stored in the "target" attribute.

- Make Spack's `cc` compiler wrapper inject target-specific flags on the
  command line

- Architecture concretization updated to use the same algorithm as
  compiler concretization

- Micro-architecture features, vendor, generation etc. are included in
  the package hash.  Generic architectures, such as x86_64 or ppc64, are
  still dumped using the name only.

- If the compiler for a target is not supported exit with an intelligible
  error message. If the compiler support is unknown don't try to use
  optimization flags.

- Support and define feature aliases (e.g., sse3 -> ssse3) in
  microarchitectures.json and on Microarchitecture objects. Feature
  aliases are defined in targets.json and map a name (the "alias") to a
  list of rules that must be met for the test to be successful. The rules
  that are available can be extended later using a decorator.

- Implement subset semantics for comparing microarchitectures (treat
  microarchitectures as a partial order, i.e. (a < b), (a == b) and (b <
  a) can all be false.

- Implement logic to automatically demote the default target if the
  compiler being used is too old to optimize for it. Updated docs to make
  this behavior explicit.  This avoids surprising the user if the default
  compiler is older than the host architecture.

This commit adds unit tests to verify the semantics of target ranges and
target lists in constraints. The implementation to allow target ranges
and lists is minimal and doesn't add any new type.  A more careful
refactor that takes into account the type system might be due later.

Co-authored-by: Gregory Becker <becker33.llnl.gov>
2019-09-20 00:51:37 -07:00
Gregory Becker
dfabf5d6b1 targets: first pass at target detection for linux
Add llnl.util.cpu_name, with initial support for detecting different
microarchitectures on Linux.  This also adds preliminary changes for
compiler support and variants to control the optimizatoin levels by
target.

This does not yet include translations of targets to particular
compilers; that is left to another PR.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2019-09-20 00:51:37 -07:00
Patrick Gartung
a940ff34d7
Put back the use of otool and install_name_tool when running on macOS. Only use machotools on linux. (#12867)
Move verbose messages to debug level

get_patchelf should return None for test platform as well because create_buildinfo invokes patchelf to get rpaths.
2019-09-19 17:16:26 -05:00
Patrick Gartung
7c0c31361a
Update buildcache creation and installation to allow mach-o binary relocation using py-machotools on linux or macos. (#12858)
Update py-machotools dependencies and versions.
2019-09-18 07:24:45 -05:00
Scott Wittenburg
56894b8805 Support yaml paths anywhere specs are handled on CLI (#12561)
Update command-line (CLI) parsing to understand references to yaml
files that store Spack specs. Where a file reference is encountered,
the full Spec in the file will be read in. A file reference may
appear anywhere that a spec could appear before. For example, if you
write "spack spec -y openmpi > openmpi.yaml" you may then install the
spec using the yaml file by running "spack install ./openmpi.yaml";
you can also refer to dependencies in this way (e.g.
"spack install foo^./openmpi.yaml").

There are two requirements for file references:

* A file path entered on the CLI must include a "/" even if the file
  exists in your current working directory. For example, if you
  create an openmpi.yaml file as above and run
  "spack install openmpi.yaml" from the same directory, it will
  report an error.
* A file path entered on the CLI must end with ".yaml"

This commit adds error messages to clearly inform the user of both
violations.
2019-09-17 19:45:37 -07:00
Peter Scheibel
141a1648e6 implicit rpaths filtering (#12789)
* implicit_rpaths are now removed from compilers.yaml config and are always instantiated dynamically, this occurs one time in the build_environment module

* per-compiler list required libraries (e.g. libstdc++, libgfortran) and whitelist directories from rpaths including those libraries. Remove non-whitelisted implicit rpaths. Some libraries default for all compilers.

* reintroduce 'implicit_rpaths' as a config variable that can be used to disable Spack insertion of compiler RPATHs generated at build time.
2019-09-17 17:45:21 -05:00
Adam J. Stewart
21e4b1752e Fix generic body during package creation (#12804)
Fixes incomplete change in #11981 

Use the proper variable (`body_def`) during package creation for package subclasses.
2019-09-17 09:37:46 -07:00
Adam J. Stewart
dce45aa299 Allow build to proceed with failed compiler lib detection (#12756)
Fixes #12732
Fixes #12767

c22a145 added automatic detection and RPATHing of compiler libraries
to Spack builds. However, in cases where the parsing/detection logic
fails this was terminating the build. This makes the compiler library
detection "best-effort" and reports an issue when the detection fails
rather than terminating the build.
2019-09-16 19:09:28 -07:00
Adam J. Stewart
5eac8130b3 Quote shell variable values (#12759)
This updates logic which sets shell variables to quote the values,
which is necessary when the value contains a space (e.g. PATH).
2019-09-16 11:10:36 -07:00
Scott Wittenburg
1050fa5546 Fixes identified in ecp facilities hackathon fixes:
- improve mirror git repo management
- minio s3 implementation needs endpoint_url
- Remove image key from rebuild-index job
- Remove image, rely on tags instead
2019-09-13 22:57:15 -07:00
Scott Wittenburg
6d745a56fd Implement an optional compiler bootstrapping phase 2019-09-13 22:57:15 -07:00