Commit graph

5690 commits

Author SHA1 Message Date
Jonathon Anderson
117c7cc3db
Only hack botocore when needed (#31610)
Newer versions of botocore (>=1.23.47) support the full IOBase
interface, so the hacks added to supplement the missing attributes are
no longer needed. Conditionally disable the hacks if they appear to be
unnecessary based on the class hierarchy found at runtime.
2022-07-18 08:40:13 +02:00
Harmen Stoppels
6f8d7390f1
Use lexists instead of exists during fetch (#31509) 2022-07-15 18:43:14 +02:00
Joseph Snyder
64b41b012c
Bug/fix credentials s3 buildcache update (#31391)
* Add connection information to buildcache update command

Ensure that the s3 connection made when attempting to update the content
of a buildcache attempts to use the extra connection information
from the mirror creation.

* Add unique help for endpoint URL argument

Fix copy/paste error for endpoint URL help which was the same as
the access token

* Re-work URL checking for S3 mirrors

Due to the fact that nested bucket URLs would never match the string used
for checking that the mirror is the same, switch the check used.

Sort all mirror URLs by length to have the most specific cases first
and see if the desired URL "starts with" the mirror URL.

* Long line style fixes

Add execptions for long lines and fix other style errors

* Use format() function to rebuild URL

Use the format command to rebuild the url instead of crafing a
formatted string out of known values

* Add early exit for URL checking

When a valid mirror is found, break from the loop
2022-07-14 20:49:46 +00:00
Todd Gamblin
3d0347ddd3
Deprecate blacklist/whitelist in favor of include/exclude (#31569)
For a long time the module configuration has had a few settings that use
`blacklist`/`whitelist` terminology. We've been asked by some of our users to replace
this with more inclusive language. In addition to being non-inclusive, `blacklist` and
`whitelist` are inconsistent with the rest of Spack, which uses `include` and `exclude`
for the same concepts.

- [x] Deprecate `blacklist`, `whitelist`, `blacklist_implicits` and `environment_blacklist`
      in favor of `exclude`, `include`, `exclude_implicits` and `exclude_env_vars` in module
      configuration, to be removed in Spack v0.20.
- [x] Print deprecation warnings if any of the deprecated names are in module config.
- [x] Update tests to test old and new names.
- [x] Update docs.
- [x] Update `spack config update` to fix this automatically, and include a note in the error
      that you can use this command.
2022-07-14 20:42:33 +00:00
Peter Scheibel
7741bfb7d1
Decompression: use tar exe vs. built-in Python tarfile support (#31563)
Python's built-in tarfile support doesn't address some general
cases of malformed tarfiles that are already handled by the system
'tar' utility; until these can be addressed, use that exclusively.
2022-07-13 14:12:21 -07:00
Massimiliano Culpo
7f2b5e8e57
spack.repo.get() can only be called on concrete specs (#31411)
The goal of this PR is to make clearer where we need a package object in Spack as opposed to a package class.

We currently instantiate a lot of package objects when we could make do with a class.  We should use the class
when we only need metadata, and we should only instantiate and us an instance of `PackageBase` at build time.

Modifications:
- [x] Remove the `spack.repo.get` convenience function (which was used in many places, and not really needed)
- [x] Use `spack.repo.path.get_pkg_class` wherever possible
- [x] Try to route most of the need for `spack.repo.path.get` through `Spec.package`
- [x] Introduce a non-data descriptor, that can be used as a decorator, to have "class level properties"
- [x] Refactor unit tests that had to be modified to reduce code duplication
- [x] `Spec.package` and `Repo.get` now require a concrete spec as input
- [x] Remove `RepoPath.all_packages` and `Repo.all_packages`
2022-07-12 19:45:24 -04:00
Harmen Stoppels
ea91e8f4ca
file_cache.py: idempotent remove without races (#31477)
There's a race condition in `remove()` as the lockfile is removed after
releasing the lock, which is a problem when another process acquires a
write lock during deletion.

Also simplify life a bit in multiprocessing when a file is possibly
removed multiple times, which currently is an error on the second
deletion, so the proposed fix is to make remove(...) idempotent and not
error when deleting non-existing cache entries.

Don't tests for existence of lockfile, cause windows/linux behavior is different
2022-07-12 15:03:48 +00:00
Harmen Stoppels
74e2625dcf
compression.py: buffered copy (#31533) 2022-07-12 16:49:59 +02:00
Harmen Stoppels
b2c519470b
environment.py: only acquire write lock when necessary (#31493) 2022-07-12 16:12:43 +02:00
Harmen Stoppels
d904a789d5
env depfile: add missing touch (#31494)
Oversight in #31433, the non-phony `env` target was missing a file being
created for it, which can cause make to infinitely loop when including
multiple generated makefiles.
2022-07-12 14:16:31 +02:00
Massimiliano Culpo
89b8d33d05
Remove os.getcwd from function signature (#31480)
fixes #29730

This is also wrong since it binds the directory at the time
of module import, rather than the one at the time of function
call.
2022-07-12 10:08:37 +02:00
Jonathon Anderson
25f198aa91
Sanitize ownership when extracting tarfiles (#31524) 2022-07-12 09:28:24 +02:00
Adam J. Stewart
5bd1074afb
py-hatchling: add new packages, document new build backend (#31512)
* py-hatchling: add new package, document new build backend

* Minor doc changes

Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>

Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2022-07-11 16:23:28 -07:00
Cory Bloor
bb92ea59a2
Fix spack edit message when no editor installed (#31412)
When no default editor is installed and no environment variable is set,
which_string would return None and this would be passed to os.execv
resulting in a TypeError. The message presented to the user would be:

    Error: execv: path should be string, bytes or os.PathLike,
    not NoneType

This change checks that which_string has returned successfully before
attempting to execute the result, resulting in a new error message:

    Error: No text editor found! Please set the VISUAL and/or EDITOR
    environment variable(s) to your preferred text editor.

It's not strictly necessary, but I've also changed try_exec to catch
all errors rather than just OSErrors. This would have provided slightly
more context for the original error message.
2022-07-07 17:08:15 +02:00
Tamara Dahlgren
1b919d306f
Documentation: Add SourceforgePackage to the build systems docs (#31473) 2022-07-07 14:30:22 +02:00
Vanessasaurus
6b1e86aecc
removing feature bloat: monitor and analyzers (#31130)
Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2022-07-07 00:49:40 -06:00
Tamara Dahlgren
e68ed3268b
Testing Bugfix: refactor clean --python-cache to support all (#31449)
There were two choices: 1) remove '-p' from '-a' or 2) allow monkeypatching
the cleaning of the python cache since clean's test_function_calls isn't
supposed to be actually cleaning anything.

This commit supports the latter and adds a test case for `-p`.
2022-07-06 11:44:41 -07:00
Harmen Stoppels
dac31ef3c4
Remove fetch from depfile (#31433) 2022-07-05 14:48:32 +02:00
Alberto Invernizzi
bb3a663392
ROCmPackage: unify amdgpu_targets (#30582)
* unify amdgpu_targets for rocsolver and rocblas

* add more archs

* adapt all roc packages to unified amdgpu_targets
2022-07-05 14:33:30 +02:00
Scott Wittenburg
87b8321fa9
gitlab: Fix pipeline generation for non-develop protected pipelines (#31378)
Release branches and tags run protected pipelines, and we noticed
that those pipelines were generating all jobs in the stack, even
when the mirror contained all the built specs and an up to date
index.  The problem was caused because the override mirror was
not present in spacks mirror configuration at the time when the
binary_distribution.update() method was called.  This fixes that
by always adding the mirror override, if present, to the list of
configured mirrors.
2022-07-01 11:56:36 -07:00
Peter Scheibel
abeee9e1fd
Cray manifest: compiler duplicates (#31173)
* remove unhelpful comment
* Filter compiler duplicates while reading manifest
* more-specific version matching edited to use module-specific version (to avoid an issue where a user might add a compiler with the same version to the initial test configuration
2022-06-29 17:05:29 -07:00
Adam J. Stewart
93c16f1ee3
PythonPackage: add default libs/headers attributes (#28527)
* PythonPackage: add default libs/headers attributes

* Style fix

* libs and headers should be properties

* Check both platlib and include

* Fix variable reference
2022-06-29 11:55:24 -07:00
Massimiliano Culpo
557abe04ec
Update containerize templates to account for view indirection (#31321)
fixes #30965
2022-06-29 15:39:18 +02:00
Wouter Deconinck
8f7f96ae2b
oneAPI supports cxx17_flag and cxx20_flag (#31274) 2022-06-28 10:33:19 +02:00
Greg Becker
df44045fdb
Feature: use git branches/tags as versions (#31200)
Building on #24639, this allows versions to be prefixed by `git.`. If a version begins `git.`, it is treated as a git ref, and handled as git commits are starting in the referenced PR.

An exception is made for versions that are `git.develop`, `git.main`, `git.master`, `git.head`, or `git.trunk`. Those are assumed to be greater than all other versions, as those prefixed strings are in other contexts.
2022-06-28 01:54:41 +00:00
Sergey Kosukhin
7fc68240fe
oneapi-* packages: improve use with modules (#30981)
This commit adds some changes which improve use of Spack-installed
oneAPI packages with Spack-generated modules, but not within Spack
(e.g. if you install some of these packages with Spack, then load
their associated modules to build other packages outside of Spack).

The majority of the PR diff is refactoring. The functional changes
are:

* intel-oneapi-mkl:
  * setup_run_environment: update Intel compiler flags to RPATH the
    mkl libs
* intel-oneapi-compilers: update the compiler configuration to RPATH
  libraries installed by this package (note that Spack already handled
  this when installing dependent packages with Spack, but this is
  specifically to use the intel-oneapi-compilers package outside
  of Spack). Specifically:
  * inject_rpaths: this modifies the binaries installed as part of
    the intel-oneapi-compilers package to RPATH libraries provided
    by this package
  * extend_config_flags: this instructs the compiler executables
    provided by this package to RPATH those same libraries

Refactoring includes:

* intel-oneapi-compilers: in addition to the functional changes,
  a large portion of this diff is reorganizing the creation of
  resources/archives downloaded for the install
* The base oneAPI package renamed component_path, so several packages
  changed as a result:
  * intel-oneapi-dpl
  * intel-oneapi-dnn
  * extrae
* intel-oneapi-mpi:
  * Perform file filtering in one pass rather than two
2022-06-27 16:34:15 -07:00
Peter Scheibel
f7ef064bdc
spack external find: handle manifest with bad permissions (#31201)
Allow `spack external find` (with no extra args) to proceed if the manifest file exists but
without sufficient permissions; in that case, print a warning. Also add a test for that behavior.

TODOs:

- [x] continue past any exception raised during manifest parsing as part of `spack external find`, 
      except for CTRL-C (and other errors that indicate immediate program termination)
- [x] Semi-unrelated but came up when discussing this with the user who reported this issue to
      me: the manifest parser now accepts older schemas 

See: https://github.com/spack/spack/issues/31191
2022-06-27 19:26:12 +00:00
Massimiliano Culpo
11d71ca85e
Rename os_facts.lp to os_compatibility.lp (#31206)
Follow-up from #31170
2022-06-27 09:15:25 -07:00
Cody Balos
147f39d7aa
Fix typo in documentation note about concretizer:unify (#31246) 2022-06-24 11:31:43 +02:00
Massimiliano Culpo
70650cacbd
ASP-based solver: rescale target weights so that 0 is always the best score (#31226)
fixes #30997

Instead of giving a penalty of 30 to all nodes when preferences
are not package specific, give a penalty of 100 to all targets
of a node where we have package specific preferences, if the target
is not explicitly preferred.
2022-06-23 23:33:36 -07:00
eugeneswalker
7ef52acfce
relocation: x-pie-executable needs relocation (#31248) 2022-06-22 23:55:28 +00:00
Adam J. Stewart
ef278c601c
spack create: fix for no URL (#31239) 2022-06-22 18:45:03 +00:00
Massimiliano Culpo
55cac3b098
Remove update functions used to ease the transition in v0.16 (#31216) 2022-06-22 11:15:57 +02:00
Massimiliano Culpo
649760dc1a
Canonicalize positional argument to spack bootstrap mirror (#31180)
fixes #31139
2022-06-21 19:25:01 +02:00
Massimiliano Culpo
aebb601b70
Stricter compatibility rules for OS and compiler when reusing specs (#31170)
* Stricter compatibility rules for OS and compiler when reusing specs
* Add unit test
2022-06-20 17:32:13 -07:00
Christian Goll
0fe57962b3
OpenSUSE Tumbleweed: use GLIBC version as distro version (#19895)
Tumbleweed is a rolling release that would have used a date 
as a version instead.
2022-06-20 22:20:31 +02:00
Sam Broderick
760a12c440
Fix request for bzip2, since bzip was pulled due to patent issues (#31198) 2022-06-20 08:13:20 +00:00
Brian Van Essen
15c35a3cff
Bugfix external find --all for libraries (#31186)
* Fixed a bug in the 'external find --all' command where the call failed
to find packages by both executable and library. The bug was that the
call `path.all_packages()` incorrectly turned the variable
`packages_to_check` into a generator rather than keeping it a list.
Thus the second call to `detection.by_library` had no work to do.

* Fixed the help message for the find external and compiler commands as
well as others that used the `scopes_metavar` field to define where
the results should be stored in configuration space.  Specifically,
the fact that configuration could be added to the environment was not
mentioned in the help message.
2022-06-17 21:52:06 +00:00
Massimiliano Culpo
13b609b4b6
docs: quote string to show valid YAML (#31178)
fixes #31167
2022-06-17 18:25:52 +02:00
Chuck Atkins
85dc20cb55
Spec: Add a new virtual-customizable home attribute (#30917)
* Spec: Add a new virtual-customizable home attribute

* java: Use the new builtin home attribute

* python: Use the new builtin home attribute
2022-06-17 10:29:08 -04:00
Robert Cohn
466572dc14
Update Intel package descriptions (#31150) 2022-06-17 09:23:40 +02:00
Massimiliano Culpo
267358a799
ASP-based solver: fix rules on version weights selection (#31153)
* ASP: sort and deduplicate version weights from installed specs

* Pick version weights according to provenance

* Add unit test
2022-06-16 14:17:40 -07:00
Peter Scheibel
392b548312
Manifest parsing: avoid irrelevant files (#31144)
* Manifest directory may not contain manifest files: exclude non-manifest files
* Manifest files use different name for rocmcc: add translation for it
2022-06-16 15:16:48 -05:00
Massimiliano Culpo
895ceeda38
concretize.lp: impose a lower bound on the number of version facts if a solution exists (#31142)
* concretize.lp: impose a lower bound on the number of version facts if a valid version exists

fixes #30864

* Add a unit test
2022-06-16 09:25:56 -07:00
Tom Scogland
b261b2a5ff
Allow bootstrapping to work when partial or incompatible modules exist in the module path (#31035) 2022-06-15 19:09:07 +02:00
G-Ragghianti
bf6220821b
Make the IntelPackage fail successfully (#30773)
Co-authored-by: Robert Cohn <rscohn2@gmail.com>
2022-06-14 15:06:06 +02:00
Jordan Galby
e50d08ce48
Fix spack style arbitrary git rev as base (#31019)
Allow `spack style -b @` or `spack style -b origin/develop` to work as
expected.

Regression since spack 0.17 #25085
2022-06-13 15:22:35 +00:00
Danny McClanahan
0c7fd9bd8c
fix doubly shell quoting args to spack spec (#29282)
* add test to verify fix works
* fix spec cflags/variants parsing test (breaking change)
* fix `spack spec` arg quoting issue
* add error report for deprecated cflags coalescing
* use .group(n) vs subscript regex group extraction for 3.5 compat
* add random test for untested functionality to pass codecov
* fix new test failure since rebase
2022-06-11 12:35:46 -07:00
Todd Gamblin
bf2b30a5f5
bugfix: preserve dict order for Spec.dag_hash() in Python 2 (#31092)
Fix a bug introduced in #21720. `spack_json.dump()` calls `_strify()` on dictionaries to
convert `unicode` to `str`, but it constructs `dict` objects instead of
`collections.OrderedDict` objects, so in Python 2 (or earlier versions of 3) it can
scramble dictionary order.

This can cause hashes to differ between Python 2 and Python 3, or between Python 3.7
and earlier Python 3's.

- [x] use `OrderedDict` in `_strify`
- [x] add a regression test
2022-06-10 20:32:35 -07:00
Sergey Kosukhin
a2afd5b82f
clingo: fix string formatting in error messages (#31084) 2022-06-10 16:24:27 +00:00
Tim Fuller
01f8236bf5
Allow more fine-grained control over what submodules are updated (#27293)
The "submodules" argument of the "version" directive can now accept
a callable that returns a list of submodules, in addition to the usual
Boolean values
2022-06-09 07:45:49 +02:00
Massimiliano Culpo
d245c46487
bootstrap: account for disabled sources (#31042)
* bootstrap: account for disabled sources

Fix a bug introduced in #30192, which effectively skips
any prescription on disabled bootstrapping sources.

* Add unit test to avoid regression
2022-06-08 07:17:37 -06:00
Peter Scheibel
c724e26ba9
Staging: determine extensions from nonstandard URLs (#31027)
Fixes #31021

With #25185, we no longer default to using tar when we can't
determine the extension type, opting to fail instead.

This broke fetching for the pcre package, where we couldn't determine
the extension. To determine the extension, we were attempting to
extract it from the destination filename; however, this file name
may omit details of the origin URL that are required to determine the
extension, so instead we examine the URL directly.

This also updates the decompressor_for method not to set ext=None
by default: it must now always be set by the caller.
2022-06-07 14:12:08 -07:00
Massimiliano Culpo
6b4b1dacd9
docs: update the list of Docker images with Spack preinstalled (#31003)
Also, update the image in the docs and use ghcr.io
2022-06-07 16:43:04 +02:00
Jordan Galby
e74d85a524
Fix empty install prefix post-install sanity check (#30983) 2022-06-07 16:17:33 +02:00
Massimiliano Culpo
4a0ac87d07
archspec: bump to v0.1.4 (#30856)
Fixes compiler flags for oneapi and dpcpp
2022-06-07 08:51:34 +00:00
John W. Parent
5b45df5269
Update decompression support on Windows (#25185)
Most package installations include compressed source files. This
adds support for common archive types on Windows:

* Add support for using system 7zip functionality to decompress .Z
  files when available (and on Windows, use 7zip for .xz archives)
* Default to using built-in Python support for tar/bz2 decompression
  (note that Python tar documentation mentions preservation of file
  permissions)
* Add tests for decompression support
* Extract logic for handling exploding archives (i.e. compressed
  archives that expand to more than one base file) into an
  exploding_archive_catch context manager in the filesystem module
2022-06-06 18:14:43 -07:00
Chuck Atkins
9d7cc43673
Package: Don't warn for missing source on bundle packages without code (#30913) 2022-06-06 15:33:03 -07:00
John W. Parent
13b0e73a4e
Sanitize filepath from URL (#30625)
Spack's staging logic constructs a file path based on a URL. The URL
may contain characters which are not allowed in valid file paths on
the system (e.g. Windows prohibits ':' and '?' among others). This
commit adds a function to remove such offending characters (but
otherwise preserves the URL string when constructing a file path).
2022-06-04 15:06:43 -07:00
Adam J. Stewart
5728ba0122
Use stable URLs for patch-diff GitHub patches (#30953) 2022-06-02 12:52:05 -04:00
Robert Cohn
f3af38ba9b
Fix module support for oneapi compilers (#28901)
Updates to improve Spack-generated modules for Intel oneAPI compilers:

* intel-oneapi-compilers set CC etc.
* Add a new package intel-oneapi-compilers-classic which can be used to
  generate a module which sets CC etc. to older compilers (e.g. icc)
* lmod module logic now updated to treat the intel-oneapi-compilers*
  packages as compilers
2022-05-31 15:02:25 -07:00
Adam J. Stewart
055c9d125d
CUDA: make cuda_arch sticky (#30910) 2022-05-30 12:53:15 -07:00
Todd Gamblin
e6929b9ff9
0.18.0.dev0 -> 0.19.0.dev0 (#30907) 2022-05-28 17:23:01 +00:00
Tom Scogland
18c2f1a57a
refactor: packages import spack.package explicitly (#30404)
Explicitly import package utilities in all packages, and corresponding fallout.

This includes:

* rename `spack.package` to `spack.package_base`
* rename `spack.pkgkit` to `spack.package`
* update all packages in builtin, builtin_mock and tutorials to include `from spack.package import *`
* update spack style
  * ensure packages include the import
  * automatically add the new import and remove any/all imports of `spack` and `spack.pkgkit`
    from packages when using `--fix`
  * add support for type-checking packages with mypy when SPACK_MYPY_CHECK_PACKAGES
    is set in the environment
* fix all type checking errors in packages in spack upstream
* update spack create to include the new imports
* update spack repo to inject the new import, injection persists to allow for a deprecation period

Original message below:
 
As requested @adamjstewart, update all packages to use pkgkit.  I ended up using isort to do this,
so repro is easy:

```console
$ isort -a 'from spack.pkgkit import *' --rm 'spack' ./var/spack/repos/builtin/packages/*/package.py
$ spack style --fix
```

There were several line spacing fixups caused either by space manipulation in isort or by packages
that haven't been touched since we added requirements, but there are no functional changes in here.

* [x] add config to isort to make sure this is maintained going forward
2022-05-28 12:55:44 -04:00
Greg Becker
19087c9d35
target optimization: re-norm optimization scale so that 0 is best. (#29926)
referred targets are currently the only minimization criteria for Spack for which we allow
negative values. That means Spack may be incentivized to add nodes to the DAG if they
match the preferred target.

This PR re-norms the minimization criteria so that preferred targets are weighted from 0,
and default target weights are offset by the number of preferred targets per-package to
calculate node_target_weight.

Also fixes a bug in the test for preferred targets that was making the test easier to pass
than it should be.
2022-05-27 22:49:41 -07:00
Greg Becker
4116b04368
update tutorial command for v0.18.0 and new gpg key (#30904) 2022-05-28 02:36:20 +00:00
Scott Wittenburg
3d43ebec72
Revert "strip -Werror: all specific or none (#30284)" (#30878)
This reverts commit 330832c22c.
2022-05-26 14:17:01 -07:00
Massimiliano Culpo
0636fdbfef
Remove the warning that Spack prints at each spec (#30872)
Add instead a warning box in the documentation
2022-05-26 14:35:20 +00:00
Scott Wittenburg
85e13260cf
ci: Support secure binary signing on protected pipelines (#30753)
This PR supports the creation of securely signed binaries built from spack
develop as well as release branches and tags. Specifically:

- remove internal pr mirror url generation logic in favor of buildcache destination
on command line
    - with a single mirror url specified in the spack.yaml, this makes it clearer where 
    binaries from various pipelines are pushed
- designate some tags as reserved: ['public', 'protected', 'notary']
    - these tags are stripped from all jobs by default and provisioned internally
    based on pipeline type
- update gitlab ci yaml to include pipelines on more protected branches than just
develop (so include releases and tags)
    - binaries from all protected pipelines are pushed into mirrors including the
    branch name so releases, tags, and develop binaries are kept separate
- update rebuild jobs running on protected pipelines to run on special runners
provisioned with an intermediate signing key
    - protected rebuild jobs no longer use "SPACK_SIGNING_KEY" env var to
    obtain signing key (in fact, final signing key is nowhere available to rebuild jobs)
    - these intermediate signatures are verified at the end of each pipeline by a new
    signing job to ensure binaries were produced by a protected pipeline
- optionallly schedule a signing/notary job at the end of the pipeline to sign all
packges in the mirror
    - add signing-job-attributes to gitlab-ci section of spack environment to allow
    configuration
    - signing job runs on special runner (separate from protected rebuild runners)
    provisioned with public intermediate key and secret signing key
2022-05-26 08:31:22 -06:00
Todd Gamblin
d51f949768
bugfix: do not compute package_hash for old concrete specs (#30861)
Old concrete specs were slipping through in `_assign_hash`, and `package_hash` was
attempting to recompute a package hash when we could not know the package a time
of concretization.

Part of this was that the logic for `_assign_hash` was hard to understand -- it was
called twice from `_finalize_concretization` and had special cases for both args it
was called with. It's much easier to understand the logic here if we just inline it.

- [x] Get rid of `_assign_hash` and just integrate it with `_finalize_concretization`
- [x] Don't call `_package_hash` at all for already-concrete specs.
- [x] Add regression test.
2022-05-26 03:12:24 +00:00
Scott Wittenburg
70824e4a5e
buildcache: Update layout and signing (#30750)
This PR introduces a new build cache layout and package format, with improvements for
both efficiency and security.

## Old Format
Currently a binary package consists of a `spec.json` file at the root and a `.spack` file,
which is a `tar` archive containing a copy of the `spec.json` format, possibly a detached
signature (`.asc`) file, and a tar-gzip compressed archive containing the install tree.

```
build_cache/
  # metadata (for indexing)
  <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json
  <arch>/
    <compiler>/
      <name>-<ver>/
        # tar archive
        <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spack
          # tar archive contents:
          # metadata (contains sha256 of internal .tar.gz)
          <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json
          # signature
          <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json.asc
          # tar.gz-compressed prefix
          <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.tar.gz
```

After this change, the nesting has been removed so that the `.spack` file is the
compressed archive of the install tree.  Now signed binary packages, will take the
form of a clearsigned `spec.json` file (a `spec.json.sig`) at the root, while unsigned
binary packages will contain a `spec.json` at the root.

## New Format

```
build_cache/
  # metadata (for indexing, contains sha256 of .spack file)
  <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json
  # clearsigned spec.json metadata
  <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spec.json.sig
  <arch>/
    <compiler>/
      <name>-<ver>/
        # tar.gz-compressed prefix (may support more compression formats later)
        <arch>-<compiler>-<name>-<ver>-24zvipcqgg2wyjpvdq2ajy5jnm564hen.spack
```

## Benefits
The major benefit of this change is that the signatures on binary packages can be
verified without:

1. Having to download the tarball, or
2. having to extract an unknown tarball.

(1) is an improvement in efficiency; (2) is a security fix: we now ensure that we trust the
binary before we try to run it through `tar`, which avoids potential attacks.

## Backward compatibility
Also after this change, spack should still be able to handle the previous buildcache
structure and binary mirrors with mixed layouts.
2022-05-24 17:39:20 -04:00
Massimiliano Culpo
ba907defca
Add a command to generate a local mirror for bootstrapping (#28556)
This PR builds on #28392 by adding a convenience command to create a local mirror that can be used to bootstrap Spack. This is to overcome the inconvenience in setting up this mirror manually, which has been reported when trying to setup Spack on air-gapped systems.

Using this PR the user can create a bootstrapping mirror, on a machine with internet access, by:

% spack bootstrap mirror --binary-packages /opt/bootstrap
==> Adding "clingo-bootstrap@spack+python %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "gnupg@2.3: %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding "patchelf@0.13.1:0.13.99 %apple-clang target=x86_64" and dependencies to the mirror at /opt/bootstrap/local-mirror
==> Adding binary packages from "https://github.com/alalazo/spack-bootstrap-mirrors/releases/download/v0.1-rc.2/bootstrap-buildcache.tar.gz" to the mirror at /opt/bootstrap/local-mirror

To register the mirror on the platform where it's supposed to be used run the following command(s):
  % spack bootstrap add --trust local-sources /opt/bootstrap/metadata/sources
  % spack bootstrap add --trust local-binaries /opt/bootstrap/metadata/binaries
The mirror has to be moved over to the air-gapped system, and registered using the commands shown at prompt. The command has options to:

1. Add pre-built binaries downloaded from Github (default is not to add them)
2. Add development dependencies for Spack (currently the Python packages needed to use spack style)

* bootstrap: refactor bootstrap.yaml to move sources metadata out

* bootstrap: allow adding/removing custom bootstrapping sources

This operation can be performed from the command line since
new subcommands have been added to `spack bootstrap`

* Add --trust argument to spack bootstrap add

* Add a command to generate a local mirror for bootstrapping

* Add a unit test for mirror creation
2022-05-24 21:33:52 +00:00
Massimiliano Culpo
f2a81af70e
Best effort co-concretization (iterative algorithm) (#28941)
Currently, environments can either be concretized fully together or fully separately. This works well for users who create environments for interoperable software and can use `concretizer:unify:true`. It does not allow environments with conflicting software to be concretized for maximal interoperability.

The primary use-case for this is facilities providing system software. Facilities provide multiple MPI implementations, but all software built against a given MPI ought to be interoperable.

This PR adds a concretization option `concretizer:unify:when_possible`. When this option is used, Spack will concretize specs in the environment separately, but will optimize for minimal differences in overlapping packages.

* Add a level of indirection to root specs

This commit introduce the "literal" atom, which comes with
a few different "arities". The unary "literal" contains an
integer that id the ID of a spec literal. Other "literals"
contain information on the requests made by literal ID. For
instance zlib@1.2.11 generates the following facts:

literal(0,"root","zlib").
literal(0,"node","zlib").
literal(0,"node_version_satisfies","zlib","1.2.11").

This should help with solving large environments "together
where possible" since later literals can be now solved
together in batches.

* Add a mechanism to relax the number of literals being solved

* Modify spack solve to display the new criteria

Since the new criteria is above all the build criteria,
we need to modify the way we display the output.

Originally done by Greg in #27964 and cherry-picked
to this branch by the co-author of the commit.

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>

* Inject reusable specs into the solve

Instead of coupling the PyclingoDriver() object with
spack.config, inject the concrete specs that can be
reused.

A method level function takes care of reading from
the store and the buildcache.

* spack solve: show output of multi-rounds

* add tests for best-effort coconcretization

* Enforce having at least a literal being solved

Co-authored-by: Greg Becker <becker33@llnl.gov>
2022-05-24 12:13:28 -07:00
Seth R. Johnson
6a57aede57
environments: fail gracefully on missing keys (#26378) 2022-05-24 08:52:40 -07:00
edwardsp
ba701a7cf8
Update regex to correctly identify quoted args (#23494)
Previously the regex was only checking for presence of quotes as a beginning
or end character and not a matching set.  This erroneously identified the
following *single* argument as being quoted:

    source bashenvfile &> /dev/null && python3 -c "import os, json; print(json.dumps(dict(os.environ)))"
2022-05-24 08:26:07 -07:00
Greg Becker
817ee81eaa
compiler flags: imposed hashes impose the lack of additional compiler flags (#30797) 2022-05-24 01:22:29 -04:00
Tom Scogland
330832c22c
strip -Werror: all specific or none (#30284)
Add a config option to strip `-Werror*` or `-Werror=*` from compile lines everywhere.

```yaml
config:
    keep_werror: false
```

By default, we strip all `-Werror` arguments out of compile lines, to avoid unwanted
failures when upgrading compilers.  You can re-enable `-Werror` in your builds if
you really want to, with either:

```yaml
config:
    keep_werror: all
```

or to keep *just* specific `-Werror=XXX` args:

```yaml
config:
    keep_werror: specific
```

This should make swapping in newer versions of compilers much smoother when
maintainers have decided to enable `-Werror` by default.
2022-05-24 00:57:09 -04:00
Todd Gamblin
306bed48d7
specs: emit better parsing errors for specs. (#24860)
Parse error information is kept for specs, but it doesn't seem like we propagate it
to the user when we encounter an error.  This fixes that.

e.g., for this error in a package:

```python
    depends_on("python@:3.8", when="0.900:")
```

Before, with no context and no clue that it's even from a particular spec:

```
==> Error: Unexpected token: ':'
```

With this PR:

```
==> Error: Unexpected token: ':'
  Encountered when parsing spec:
    0.900:
         ^
```
2022-05-24 03:33:43 +00:00
Greg Becker
8616ba04db
Documentation and new method for CachedCMakePackage build system (#22706)
Co-authored-by: Axel Huebl <axel.huebl@plasma.ninja>
2022-05-23 22:48:12 +00:00
Massimiliano Culpo
7c4cc1c71c
archspec: add oneapi and dpcpp flag support (#30783) 2022-05-23 13:28:54 -07:00
Harmen Stoppels
f7258e246f
Deprecate spack:concretization over concretizer:unify (#30038)
* Introduce concretizer:unify option to replace spack:concretization

* Deprecate concretization

* Make spack:concretization overrule concretize:unify for now

* Add environment update logic to move from spack:concretization to spack:concretizer:reuse

* Migrate spack:concretization to spack:concretize:unify in all locations

* For new environments make concretizer:unify explicit, so that defaults can be changed in 0.19
2022-05-23 13:20:34 -07:00
Glenn Johnson
d688a699fa
Fix toolchain detection for oneapi/dpcpp compilers (#30775)
The oneapi and dpcpp compilers are essentially the same except for which
binary is used foc CXX. Spack will detect them as "mixed toolchain" and
not inject compiler optimization flags. This will be needed once
archspec has entries for the oneapi and dpcpp compilers. This PR detects
when dpcpp and oneapi are in the toolchains list and explicitly sets
`is_mixed_toolchain` to `False`.
2022-05-21 08:38:01 +02:00
Greg Becker
ee04a1ab0b
errors: model error messages as an optimization problem (#30669)
Error messages for the clingo concretizer have proven challenging. The current messages are incredibly vague and often don't help users at all. Unsat cores in clingo are not guaranteed to be minimal, and lead to cores that are either not useful or need to be post-processed for hours to reach a minimal core.

Following up on an idea from a slack conversation with kwryankrattiger on slack, this PR takes a new approach. We eliminate most integrity constraints and minima/maxima on choice rules in clingo, and instead force invalid states to imply an error predicate. The error predicate can include context on the cause of the error (Package, Version, etc). These error predicates are then heavily optimized against, to ensure that we do not include error facts in the solution when a solution with no error facts could be generated. When post-processing the clingo solution to construct specs, any error facts cause the program to raise an exception. This leads to much more legible error messages. Each error predicate includes a priority and an error message. The error message is formatted by the remaining arguments to produce the error message. The priority is used to ensure that when clingo has a choice of which rules to violate, it chooses the one which will be most informative to the user.

Performance:

"fresh" concretizations appear to suffer a ~20% performance penalty under this branch, while "reuse" concretizations see a speedup of around 33%. 

Possible optimizations if users still see unhelpful messages:

There are currently 3 levels of priority of the error messages. Additional priorities are possible, and can allow us finer granularity to ensure more informative error messages are provided in lieu of less informative ones.

Future work:

Improve tests to ensure that every possible rule implying an error message is exercised
2022-05-20 08:27:07 -07:00
Jordan Galby
262c3f07bf
Non-existent upstream is not fatal (#30746)
A non-existent upstream should not be fatal: it could only mean it is
not deployed yet. In the meantime, it should not block the user to
rebuild anything it needs.

A warning is still emitted, to let the user decide if this is ok or not.
2022-05-19 18:27:43 +00:00
Jordan Galby
c2fd98ccd2
Fix spack install chgrp on symlinks (#30743)
Fixes missing chgrp on symlinks in package installations, and errors on
symlinks referencing non-existent or non-writable locations.

Note: `os.chown(.., follow_symlinks=False)` is python3 only, but
`os.lchown` exists in both versions.
2022-05-19 08:50:24 -07:00
Jordan Galby
8fe39be3df
Don't try to mkdir upstream directory when nonexistent (#30744)
When an upstream is specified but the directory does not exist, don't
create the directory for it, it might not be yours.
2022-05-19 14:45:18 +00:00
robgics
1f6b880fff
Add license dir to config (#30135)
* Change license dir from hard-coded to a configurable item

* Change config item to be a string not an array

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-05-18 18:26:42 -07:00
Todd Gamblin
8ff2b4b747
bugfix: handle new dag_hash() on old concrete specs gracefully. (#30678)
Trying to compute `dag_hash()` or `package_hash()` on a concrete spec that doesn't have
a `_package_hash` attribute would attempt to recompute the package hash.

This most commonly manifests as a failed lookup of a namespace if you attempt to uninstall
or compute the hashes of packages in exsternal repositories that aren't registered, e.g.:

```console
> spack spec --json c/htno
==> Error: Unknown namespace: myrepo
```

While it wouldn't change the already-assigned `dag_hash` value, this behavior is
incorrect, since the package file for a previously concrete spec:
  1. might have changed since concretization,
  2. might not exist anymore, or
  3. might just not be findable by Spack.

This PR ensures that the package hash can't be computed on older concrete specs. Instead
of calling `package_hash()` from within `to_node_dict()`, we now check for the `_package_hash`
attribute and only add the package_hash to the spec record if it's there.

This PR also handles the tricky semantics of computing `package_hash()` at concretization
time. We have to compute it *before* marking the spec concrete so that `to_node_dict` can
use it. But this means that the logic for `package_hash()` can't rely on `spec.concrete`,
as it is called *during* concretization. Instead of checking for concreteness, `package_hash()`
now checks `_patches_assigned()` to determine whether it should add them to the package
hash.

- [x] Add an assert to `package_hash()` so it can't be called on specs for which it
      would be wrong.
- [x] Add an `_assign_hash()` method to handle tricky semantics of `package_hash`
      and `dag_hash`.
- [x] Rework concretization to call `_assign_hash()` before and after marking specs
      concrete.
- [x] Rework content hash part of package hash to check for `_patches_assigned()`
      instead of `spec.concrete`.
- [x] regression test
2022-05-18 22:21:22 +00:00
Massimiliano Culpo
c775c322ec
vendored externals: update archspec (#30683)
- Better support for 164fx
- Better support for Apple M1(pro)
2022-05-18 11:31:20 +02:00
Harmen Stoppels
1185eb9199
Compiler wrapper: fix globbing and debug out.log bell chars (#30699)
* Disable globbing

* Split on bell char when dumping cmd to out.log
2022-05-18 09:06:54 +02:00
Massimiliano Culpo
f454a683b5
Mark test_repo_last_mtime xfail on Python < 3.5 (#30696) 2022-05-17 12:45:52 +02:00
andymwood
97ec8f1d19
Avoid calling a method on a NoneType object (#30637) 2022-05-16 21:59:08 +00:00
Todd Gamblin
0fdc3bf420
bugfix: use deterministic edge order for spack graph (#30681)
Previously we sorted by hash values for `spack graph`, but changing hashes can make the
test brittle and the node order seem nondeterministic to users.

- [x] Sort nodes in `spack graph` by the default edge order, which takes into account
      parent and child names as well as dependency types.
- [x] Update ASCII test output for new order.
2022-05-16 11:36:41 +02:00
Danny McClanahan
a681fd7b42
Introduce GroupedExceptionHandler and use it to simplify bootstrap error handling (#30192) 2022-05-15 10:59:11 +00:00
Alberto Invernizzi
f40f1b5c7c
Fix for spack stage command not extracting packages in custom paths (#30448) 2022-05-15 12:13:42 +02:00
Michael Kuhn
ff03e2ef4c
uninstall: fix dependency check (#30674)
The dependency check currently checks whether there are only build
dependencies left for a particular package. However, the database also
contains uninstalled packages, which can cause the check to fail.

For instance, with `bison` and `flex` having already been uninstalled,
`m4` will have the following dependents:
```
bison ('build', 'run')--> m4
flex ('build',)--> m4
libnl ('build',)--> m4
```
`bison` and `flex` should be ignored in this case because they are not
installed anymore.

Fixes #30673
2022-05-14 18:01:29 -07:00
John W. Parent
e24e71be6a
Preserve Permissions on .zip extraction (#30407)
#24556 merged in support for Python's .zip file support via ZipFile.
However as per #30200 ZipFile does not preserve file permissions of
the extracted contents. This PR returns to using the `unzip`
executable on non-Windows systems (as was the case before #24556)
and now uses `tar` on Windows to extract .zip files.
2022-05-13 13:38:05 -07:00
Todd Gamblin
5cb40cbcd2 directory_layout: remove outdated checks for old DAG hash
We previously had checks in `directory_layout` to check for build-dependency
conflicts when we weren't storing build dependencies.  We don't need
those anymore; we can just rely on the DAG hash now that it includes everything
we know about each spec.

- [x] Remove vestigial code for checking installed spec against concrete spec
      in `ensure_installed()`
- [x] Remove `SpecHashCollisionError` -- if specs have the same hash now, they're
      the same as far as `DirectoryLayout` should be concerned.
- [x] Convert spec comparison to `dag_hash()` comparison when adding extensions.
2022-05-13 10:45:12 -07:00
Todd Gamblin
c93e465134 full hash: fix uninstall and gc with full hash DB
The database now stores full hashes, so we need to adjust the criteria we use to
determine if something can be uninstalled. Specifically, it's ok to uninstall thing that
have remaining build-only dependents.
2022-05-13 10:45:12 -07:00
Todd Gamblin
521c206030 concretizer: enable hash reuse with full hash
With the original DAG hash, we did not store build dependencies in the database, but
with the full DAG hash, we do. Previously, we'd never tell the concretizer about build
dependencies of things used by hash, because we never had them. Now, we have to avoid
telling the concretizer about them, or they'll unnecessarily constrain build
dependencies for new concretizations.

- [x] Make database track all dependencies included in the `dag_hash`
- [x] Modify spec_clauses so that build dependency information is optional
      and off by default.
- [x] `spack diff` asks `spec_clauses` for build dependencies for completeness
- [x] Modify `concretize.lp` so that reuse optimization doesn't affect fresh
      installations.
- [x] Modify concretizer setup so that it does *not* prioritize installed versions
      over package versions. We don't need this with reuse, so they're low priority.
- [x] Fix `test_installed_deps` for full hash and new concretizer (does not work
      for old concretizer with full hash -- leave this for later if we need it)
- [x] Move `test_installed_deps` mock packages to `builtin.mock` for easier debugging
      with `spack -m`.
- [x] Fix `test_reuse_installed_packages_when_package_def_changes` for full hash
2022-05-13 10:45:12 -07:00
Todd Gamblin
15eb98368d bugfix: tests trying to ignore package changes should use build_hash
- [x] update test to use `build_hash` instead of `dag_hash`, as we're testing for
      graph structure, and specifically NOT testing for package changes.
- [x] make hash descriptors callable on specs to simplify syntax for invoking them
- [x] make `Spec.spec_hash()` public
2022-05-13 10:45:12 -07:00
Todd Gamblin
7c1d566959 Remove all uses of runtime_hash; document lockfile formats and fix tests
This removes all but one usage of runtime hash. The runtime hash was being used to write
historical lockfiles for tests, but we don't need it for that; we can just save those
lockfiles.

- [x] add legacy lockfiles for v1, v2, v3
- [x] fix bugs with v1 lockfile tests (the dummy lockfile we were writing was not actually
      a v1 lockfile because it used the new spec file format).
- [x] remove all but one runtime_hash usage -- that one needs a small rework of the
      concretizer to really fix, as it's about separate concretization of build
      dependencies.
- [x] Document the history of the lockfile format in `environment/__init__.py`
2022-05-13 10:45:12 -07:00
Todd Gamblin
7ab46e26b5 content_hash(): make it work on abstract specs
Some test cases had to be modified in a kludgy way so that abstract specs made
concrete would have versions on them. We shouldn't *need* to do this, as the
only reason we care is because the content hash has to be able to get an archive
for a version.

This modifies the content hash so that it can be called on abstract specs,
including only relevant content.

This does NOT add a partial content hash to the DAG hash, as we do not really
want that -- we don't need in-memory spec hashes to need to load package files.
It just makes `Package.content_hash()` less prickly and tests easier to
understand.
2022-05-13 10:45:12 -07:00
Todd Gamblin
6db215dd89 spec: fix serialization, avoid double call to node_dict_with_hashes 2022-05-13 10:45:12 -07:00
Todd Gamblin
72b38851eb hashes: revert spack monitor hash changes to preserve original protocol
`spack monitor` expects a field called `spec_full_hash`, so we shouldn't change that.
Instead, we can pass a `dag_hash` (which is now the full hash) but not change the field
name.
2022-05-13 10:45:12 -07:00
Todd Gamblin
9d9e970367 fix full hash calls in spack graph 2022-05-13 10:45:12 -07:00
Todd Gamblin
283a4e6068 remove no longer needed full hash check 2022-05-13 10:45:12 -07:00
Todd Gamblin
d20cc7b124 spec: remove hashes_final as it's no longer needed.
`hashes_final` was used to indicate when a spec was concrete but possibly lacked
`full_hash` or `build_hash` fields. This was only necessary because older Spacks
didn't generate them, and we want to avoid recomputing them, as we likely do not
have the same package files as existed at concretization time.

Now, we don't need to do that -- there is only the DAG hash and specs are either
concrete and have a `dag_hash`, or not concrete and have no `dag_hash`. There's
no middle ground.
2022-05-13 10:45:12 -07:00
Scott Wittenburg
c202953528 gitlab ci: Docstring methods or make them private 2022-05-13 10:45:12 -07:00
Scott Wittenburg
be0e3f4458 binary_distribution: Refactor index generation into smaller methods 2022-05-13 10:45:12 -07:00
Scott Wittenburg
fd3bb5177b env: Use order of roots to resolve DAG hash conflicts in legacy lockfiles 2022-05-13 10:45:12 -07:00
Scott Wittenburg
9de61c0197 env: enforce predictable ordering when reading lockfile
Without some enforcement of spec ordering, python 2 produced
different results in the affected test than did python 3.  This
change makes the arbitrary but reproducible decision to sort
the specs by their lockfile key alphabetically.
2022-05-13 10:45:12 -07:00
Scott Wittenburg
84cfb3b7fe spec: fix infinite recursion when computing package hash
Issue described in the following PR comment:

https://github.com/spack/spack/pull/28504#issuecomment-1051835568

Solution described in subsequent comment:

https://github.com/spack/spack/pull/28504#issuecomment-1053986132
2022-05-13 10:45:12 -07:00
Scott Wittenburg
cb0d12b9d5 Fix how environments are read from lockfile 2022-05-13 10:45:12 -07:00
Scott Wittenburg
f6e7c0b740 hashes: remove full_hash and build_hash from spack 2022-05-13 10:45:12 -07:00
Scott Wittenburg
512645ff2e environment: key by dag_hash instead of build_hash 2022-05-13 10:45:12 -07:00
Scott Wittenburg
32a2c22b2b tests: fix failing test_hash_change
The full hash appears twice in the spec dict now, replacing just
the value replaces it under "hash" and "full_hash".  Only replace
the one that appears after "full_hash".

I'm actually not sure what purpose this test served, so maybe it
could be removed, as it may be testing some distinction between
full and dag hash which no longer exists.
2022-05-13 10:45:12 -07:00
Todd Gamblin
e02020c80a Include all deps and package content in the dag_hash()
For a long time, Spack has used a coarser hash to identify packages
than it likely should. Packages are identified by `dag_hash()`, which
includes only link and run dependencies. Build dependencies are
stripped before hashing, and we have notincluded hashes of build
artifacts or the `package.py` files used to build.  This means the
DAG hash actually doesn't represent all the things Spack can build,
and it reduces reproducibility.

We did this because, in the early days, users were (rightly) annoyed
when a new version of CMake, autotools, or some other build dependency
would necessitate a rebuild of their entire stack. Coarsening the hash
avoided this issue and enabled a modicum of stability when only reusing
packages by hash match.

Now that we have `--reuse`, we don't need to be so careful. Users can
avoid unnecessary rebuilds much more easily, and we can add more
provenance to the spec without worrying that frequent hash changes
will cause too many rebuilds.

This commit starts the refactor with the following major change:

- [x] Make `Spec.dag_hash()` include build, run, and link
      dependencides and the package hash (it is now equivalent to
      `full_hash()`).

It also adds a couple of bugfixes for problems discovered during
the switch:

- [x] Don't add a `package_hash()` in `to_node_dict()` unless
      the spec is concrete (fixes breaks on abstract specs)

- [x] Don't add source ids to the package hash for packages without
      a known fetch strategy (may mock packages are like this)

- [x] Change how `Spec.patches` is memoized. Using
      `llnl.util.lang.memoized` on `Spec` objects causes specs to
      be stored in a `dict`, which means they need a hash.  But,
      `dag_hash()` now includes patch `sha256`'s via the package
      hash, which can lead to infinite recursion
2022-05-13 10:45:12 -07:00
Massimiliano Culpo
d900ac2003
Reuse concretization by default (#30396)
* Enable reuse by default in Spack
* Update documentation to match new default
* Configure pipelines not to reuse software
2022-05-13 09:11:10 -07:00
Massimiliano Culpo
745c191d73
Gitlab pipelines: add a small legend in the logs to interpret "[x]" (#30643) 2022-05-12 21:46:35 +02:00
Harmen Stoppels
3033abb5bd
Add cuda 11.7 compat bounds for gcc/clang (#30639) 2022-05-12 10:41:30 +00:00
Tamara Dahlgren
1b254d19c4
Allow read-only access to file cache (when needed) (#29693)
* Allow read-only access to file cache (when needed)
* Tweaked and added unit tests
* Skip test_cache_init_entry_fails for windows
2022-05-11 16:25:06 -07:00
Thomas Dickerson
72e594fb10
Fix default buildcache location (#30230)
Resolve path/URL parsing issues introduced by #27021
2022-05-11 17:50:04 +02:00
Robert Cohn
c47c5d75e4
oneapi: add v2022.2 (#30531) 2022-05-10 07:25:06 +02:00
Todd Gamblin
a0d4630448
bugfix: spack pkg list should be more picky about what's a package (#30577)
`spack pkg list` tests were broken by #29593 for cases when your `builtin.mock` repo
still has stale backup files (or, really, stale directories) sitting around. This
happens if you switch branches a lot. In this case, things like this were causing
erroneous packages in the mock listing:

```
var/spack/repos/builtin.mock/packages/
    foo/
        package.py~
```

- [x] make `list_packages` consider only directories with one-deep `package.py` files.
2022-05-10 04:57:58 +00:00
Tom Scogland
7f1659786b
Add a Lua build-system (#28854)
Reworking lua to allow easier substitution of the base lua implementation.

Also adding in a maintained version of luajit and re-factoring the entire stack 
to use a custom build-system to centralize functionality like environment 
variable management and luarocks installation.

The `lua-lang` virtual is now versioned so that a package that requires 
Lua 5.1 semantics can get any lua, but one that requires 5.2 will only 
get upstream lua.

The luaposix package requires lua-bit32, but only when built with a 
lua conforming to version 5.1.  This adds the package, and the 
dependencies, but exposed a problem with luarocks dependency 
detection.  Since we're  installing each package in its own "tree" and 
there's no environment  variable to list extra trees, spack now 
generates a luarocks config  file that lists all the trees of all the 
dependencies, and references  it by setting `LUAROCKS_CONFIG` 
in the build environment of every LuaPackage.  This allows luarocks 
to find the spack installed  dependencies correctly rather than 
trying (and failing) to download them.

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Tom Scogland <tscogland@llnl.gov>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-05-10 06:54:38 +02:00
Todd Gamblin
7997dfcf80
tests: fix references to hard-coded master branch in git tests (#30572)
Some of our `git` tests still fail when `init.defaultBranch` is set to something other
than `master`.

- [x] get rid of all hard-coded `master` refs
- [x] Use `'default'` to key tests that use the default branch
2022-05-09 16:42:25 -07:00
Dom Heinzeller
c49508648a
Get timeout for web requests with urllib from spack config, same as for curl (#30468) 2022-05-09 10:35:17 -07:00
John W. Parent
9bcf496f21
Windows permissions: uninstalling and cleaning stages (#29714)
When running on Windows, Spack may generate files in the stage/install
prefixes that do not have write permissions, which prevents the
removal of those directories (e.g. when cleaning stages or uninstalling).
There should be a refactoring to avoid this in the first place, but that
is assumed to be longer term, so the temporary fix is to make such files
writable if they are not. This PR:

* Automatically handles these permissions errors when uninstalling
  packages from the Spack root (makes then writable)
* Updates similar already-existing logic when removing Spack-managed
  stage directories (the error-handling was assuming all errors were
  permissions errors and was therefore handling other errors
  inappropriately)

Note: these permissions issues only appear on Windows so this logic is
only applied there (permissions are not modified for this purpose on
Linux etc.).

This also adds special handling for a case where calling `isdir`
on an `os.DirEntry` object would fail for improperly-created symlinks
(e.g. on Windows, using `os.symlink` without `target_is_directory=True`).
Note this specific issue only came up when enabling link_tree tests
(specifically `source_merge_visitor_cant_be_cyclical`).
2022-05-09 10:28:14 -07:00
Peter Scheibel
0858c281e4
Cray manifest file: accept "nvidia" as "nvhpc" (#30428)
* create function for translating compiler names on specs/compiler entries in manifest

* add tests for translating compiler names on spec/compiler entries

* use higher-level function in test and add comment to prefer testing via higher-level function

* opensuse clingo check should not fail on account of this pr, but I cannot get it to pass by restarting via CI UI
2022-05-08 17:51:27 -07:00
Robert Pavel
5ab526185a
Force GCC to always provide a C++14 flag (#29781)
* Force GCC to always provide a C++14 flag

Updated gnu logic so that the c++14 flag for g++ is always propagated.
This fixes issues with build systems that error out if passed an empty
string for a flag.

Engaging in the best kind of software engineering by updating the unit
test to pass with the value it is now passed. This should better match
the expected flag for g++ compiling with the C++14 standard
2022-05-07 13:09:20 -04:00
Greg Becker
27462bc982
Fix improper type for InvalidDependencyError argument (#30504) 2022-05-06 16:23:12 +02:00
Harmen Stoppels
2f14695882
docs: jobserver & generated makefiles (#30526) 2022-05-06 14:04:48 +02:00
Tom Scogland
d3a0ac1c0a
Preserve jobserver file descriptors into build environment (#30302)
This ensures that multiple spack instances called from `make` will respect the maximum number of jobs in the POSIX jobserver across packages.

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-05-06 04:09:58 +00:00
Tom Scogland
6898b7c2f6
harden unit tests, enable basic parallelism (#29593)
* use the init.defaultBranch name, not master

* make tcl and modules/common independent

Both used to use not just the same directory, but the same *file* for
their outputs.  In parallel this can cause problems, but it can also
accidentally allow expected failures to pass if the file is left around
by mistake.

* use a non-global misc_cache in tests

* make pkg tests resilient to gitignore

* make source cache and module directories non-global
2022-05-05 11:48:16 -07:00
Harmen Stoppels
2836648904
Makefile generator for parallel spack install of environments (#30254)
`make` solves a lot of headaches that would otherwise have to be implemented in Spack:

1. Parallelism over packages through multiple `spack install` processes
2. Orderly output of parallel package installs thanks to `make --sync-output=recurse` or `make -Orecurse` (works well in GNU Make 4.3; macOS is unfortunately on a 16 years old 3.x version, but it's one `spack install gmake` away...)
3. Shared jobserver across packages, which means a single `-j` to rule them all, instead of manually finding a balance between `#spack install processes` & `#jobs per package` (See #30302).

This pr adds the `spack env depfile` command that generates a Makefile with dag hashes as
targets, and dag hashes of dependencies as prerequisites, and a command
along the lines of `spack install --only=packages /hash` to just install
a single package.

It exposes two convenient phony targets: `all`, `fetch-all`. The former installs the environment, the latter just fetches all sources. So one can either use `make all -j16` directly or run `make fetch-all -j16` on a login node and `make all -j16` on a compute node. 

Example:

```yaml
spack:
  specs: [perl]
  view: false
```

running

```
$ spack -e . env depfile --make-target-prefix env | tee Makefile
```
generates

```Makefile
SPACK ?= spack

.PHONY: env/all env/fetch-all env/clean

env/all: env/env

env/fetch-all: env/fetch

env/env: env/.install/cdqldivylyxocqymwnfzmzc5sx2zwvww
	@touch $@

env/fetch: env/.fetch/cdqldivylyxocqymwnfzmzc5sx2zwvww env/.fetch/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.fetch/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.fetch/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.fetch/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.fetch/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.fetch/73t7ndb5w72hrat5hsax4caox2sgumzu env/.fetch/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.fetch/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/.fetch/c4go4gxlcznh5p5nklpjm644epuh3pzc
	@touch $@

env/dirs:
	@mkdir -p env/.fetch env/.install

env/.fetch/%: | env/dirs
	$(info Fetching $(SPEC))
	$(SPACK) -e '/tmp/tmp.7PHPSIRACv' fetch $(SPACK_FETCH_FLAGS) /$(notdir $@) && touch $@

env/.install/%: env/.fetch/%
	$(info Installing $(SPEC))
	+$(SPACK) -e '/tmp/tmp.7PHPSIRACv' install $(SPACK_INSTALL_FLAGS) --only-concrete --only=package --no-add /$(notdir $@) && touch $@

# Set the human-readable spec for each target
env/%/cdqldivylyxocqymwnfzmzc5sx2zwvww: SPEC = perl@5.34.1%gcc@10.3.0+cpanm+shared+threads arch=linux-ubuntu20.04-zen2
env/%/gv5kin2xnn33uxyfte6k4a3bynhmtxze: SPEC = berkeley-db@18.1.40%gcc@10.3.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu20.04-zen2
env/%/cuymc7e5gupwyu7vza5d4vrbuslk277p: SPEC = bzip2@1.0.8%gcc@10.3.0~debug~pic+shared arch=linux-ubuntu20.04-zen2
env/%/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk: SPEC = diffutils@3.8%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
env/%/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws: SPEC = libiconv@1.16%gcc@10.3.0 libs=shared,static arch=linux-ubuntu20.04-zen2
env/%/yfz2agazed7ohevqvnrmm7jfkmsgwjao: SPEC = gdbm@1.19%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
env/%/73t7ndb5w72hrat5hsax4caox2sgumzu: SPEC = readline@8.1%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
env/%/trvdyncxzfozxofpm3cwgq4vecpxixzs: SPEC = ncurses@6.2%gcc@10.3.0~symlinks+termlib abi=none arch=linux-ubuntu20.04-zen2
env/%/sbzszb7v557ohyd6c2ekirx2t3ctxfxp: SPEC = pkgconf@1.8.0%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
env/%/c4go4gxlcznh5p5nklpjm644epuh3pzc: SPEC = zlib@1.2.12%gcc@10.3.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu20.04-zen2

# Install dependencies
env/.install/cdqldivylyxocqymwnfzmzc5sx2zwvww: env/.install/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.install/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.install/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.install/c4go4gxlcznh5p5nklpjm644epuh3pzc
env/.install/cuymc7e5gupwyu7vza5d4vrbuslk277p: env/.install/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk
env/.install/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk: env/.install/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws
env/.install/yfz2agazed7ohevqvnrmm7jfkmsgwjao: env/.install/73t7ndb5w72hrat5hsax4caox2sgumzu
env/.install/73t7ndb5w72hrat5hsax4caox2sgumzu: env/.install/trvdyncxzfozxofpm3cwgq4vecpxixzs
env/.install/trvdyncxzfozxofpm3cwgq4vecpxixzs: env/.install/sbzszb7v557ohyd6c2ekirx2t3ctxfxp

env/clean:
	rm -f -- env/env env/fetch env/.fetch/cdqldivylyxocqymwnfzmzc5sx2zwvww env/.fetch/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.fetch/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.fetch/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.fetch/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.fetch/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.fetch/73t7ndb5w72hrat5hsax4caox2sgumzu env/.fetch/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.fetch/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/.fetch/c4go4gxlcznh5p5nklpjm644epuh3pzc env/.install/cdqldivylyxocqymwnfzmzc5sx2zwvww env/.install/gv5kin2xnn33uxyfte6k4a3bynhmtxze env/.install/cuymc7e5gupwyu7vza5d4vrbuslk277p env/.install/7vangk4jvsdgw6u6oe6ob63pyjl5cbgk env/.install/hyb7ehxxyqqp2hiw56bzm5ampkw6cxws env/.install/yfz2agazed7ohevqvnrmm7jfkmsgwjao env/.install/73t7ndb5w72hrat5hsax4caox2sgumzu env/.install/trvdyncxzfozxofpm3cwgq4vecpxixzs env/.install/sbzszb7v557ohyd6c2ekirx2t3ctxfxp env/.install/c4go4gxlcznh5p5nklpjm644epuh3pzc
```

Then with `make -O` you get very nice orderly output when packages are built in parallel:
```console
$ make -Orecurse -j16
spack -e . install --only-concrete --only=package /c4go4gxlcznh5p5nklpjm644epuh3pzc && touch c4go4gxlcznh5p5nklpjm644epuh3pzc
==> Installing zlib-1.2.12-c4go4gxlcznh5p5nklpjm644epuh3pzc
...
  Fetch: 0.00s.  Build: 0.88s.  Total: 0.88s.
[+] /tmp/tmp.b1eTyAOe85/store/linux-ubuntu20.04-zen2/gcc-10.3.0/zlib-1.2.12-c4go4gxlcznh5p5nklpjm644epuh3pzc
spack -e . install --only-concrete --only=package /sbzszb7v557ohyd6c2ekirx2t3ctxfxp && touch sbzszb7v557ohyd6c2ekirx2t3ctxfxp
==> Installing pkgconf-1.8.0-sbzszb7v557ohyd6c2ekirx2t3ctxfxp
...
  Fetch: 0.00s.  Build: 3.96s.  Total: 3.96s.
[+] /tmp/tmp.b1eTyAOe85/store/linux-ubuntu20.04-zen2/gcc-10.3.0/pkgconf-1.8.0-sbzszb7v557ohyd6c2ekirx2t3ctxfxp
```

For Perl, at least for me, using `make -j16` versus `spack -e . install -j16` speeds up the builds from 3m32.623s to 2m22.775s, as some configure scripts run in parallel.

Another nice feature is you can do Makefile "metaprogramming" and depend on packages built by Spack. This example fetches all sources (in parallel) first, print a message, and only then build packages (in parallel).

```Makefile
SPACK ?= spack

.PHONY: env

all: env

spack.lock: spack.yaml
	$(SPACK) -e . concretize -f

env.mk: spack.lock
	$(SPACK) -e . env depfile -o $@ --make-target-prefix spack

fetch: spack/fetch
	@echo Fetched all packages && touch $@

env: fetch spack/env
	@echo This executes after the environment has been installed

clean:
	rm -rf spack/ env.mk spack.lock

ifeq (,$(filter clean,$(MAKECMDGOALS)))
include env.mk
endif
```
2022-05-05 10:45:21 -07:00
Tamara Dahlgren
011a491b16
package audit: ensure stand-alone test method not include in build-phase testing (#30352) 2022-05-05 18:04:16 +02:00
Greg Becker
e6346eb033
spack external find: add search path customization (#30479) 2022-05-05 08:59:44 +02:00
Sergey Kosukhin
8b0f6187e0
bugfix: fix filter_compiler_wrappers for Cray compiler (#29786) 2022-05-04 10:13:12 -07:00
Massimiliano Culpo
5c7d6c6e10
Remove deprecated "--run-tests" option of "spack install" (#30461) 2022-05-04 07:43:29 +02:00
Massimiliano Culpo
a5e92893d3
Simplify fixture used to test spack info (#30456) 2022-05-03 09:39:00 -07:00
Brian Van Essen
4576fbe648
OpenCV and OpenBLAS: add external find support (#30240)
Added support for finding the OpenCV package via the find external
command. Included support for identifying variants based on available
shared libraries.

Added support to finding the OpenBLAS package via the find external
command.

Enabled packages to show that they can be discovered via the find
external command in the info message.

Updated the OpenCV and OpenBLAS packages to use the extensible search
mechanism for library extensions on multiple OS platforms.

Corrected how find externals works on Darwin for OpenCV and OpenBLAS
to accommodate that the version numbers are placed before the file
extension instead of after it, as on Linux.

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-05-03 09:04:50 +02:00
Ken Raffenetti
e2c72e583f
spack ci: Fix typo in error message (#30438) 2022-05-02 20:33:54 +00:00
Todd Gamblin
9a028e3b15
bugfix: don't calculate spack tutorial version from version info
A recent switch to the way we do `develop` versioning broke this.  We
should hard-code the latest tutorial version.
2022-05-01 23:13:50 -07:00
Massimiliano Culpo
c06f69d0bf
spack.yaml: add concretizer.yaml to the schema (#30376) 2022-04-29 01:06:36 -06:00
Peter Scheibel
bb43308c44
Add command for reading JSON-based DB description (now with more tests) (#29652)
This is an amended version of https://github.com/spack/spack/pull/24894 (reverted in https://github.com/spack/spack/pull/29603). https://github.com/spack/spack/pull/24894
broke all instances of `spack external find` (namely when it is invoked without arguments/options)
because it was mandating the presence of a file which most systems would not have.
This allows `spack external find` to proceed if that file is not present and adds tests for this.

- [x] Add a test which confirms that `spack external find` successfully reads a manifest file
      if present in the default manifest path

--- Original commit message ---

Adds `spack external read-cray-manifest`, which reads a json file that describes a
set of package DAGs. The parsed results are stored directly in the database. A user
can see these installed specs with `spack find` (like any installed spec). The easiest
way to use them right now as dependencies is to run
`spack spec ... ^/hash-of-external-package`.

Changes include:

* `spack external read-cray-manifest --file <path/to/file>` will add all specs described
  in the file to Spack's installation DB and will also install described compilers to the
  compilers configuration (the expected format of the file is described in this PR as well including examples of the file)
* Database records now may include an "origin" (the command added in this PR
  registers the origin as "external-db"). In the future, it is assumed users may want
  to be able to treat installs registered with this command differently (e.g. they may
  want to uninstall all specs added with this command)
* Hash properties are now always preserved when copying specs if the source spec
  is concrete
  * I don't think the hashes of installed-and-concrete specs should change and this
    was the easiest way to handle that
  * also specs that are concrete preserve their `.normal` property when copied
    (external specs may mention compilers that are not registered, and without this
    change they would fail in `normalize` when calling `validate_or_raise`)
  * it might be this should only be the case if the spec was installed

- [x] Improve testing
- [x] Specifically mark DB records added with this command (so that users can do
      something like "uninstall all packages added with `spack read-external-db`)
  * This is now possible with `spack uninstall --all --origin=external-db` (this will
    remove all specs added from manifest files)
- [x] Strip variants that are listed in json entries but don't actually exist for the package
2022-04-28 10:56:26 -07:00
Massimiliano Culpo
6a9df34abd
ASP-based solver: discard unknown packages from reuse (#30357)
* ASP-based solver: discard unknown packages from reuse

This is an add-on to #28259 that cover for the case of
a single package.py being removed from a repository,
rather than an entire custom repository being removed.

* Add unit test
2022-04-28 10:40:28 -07:00
Frédéric Simonis
1006dd54de
Add BUILD_TESTING to standard CMake arguments (#30374)
CTest determines whether to enable tests using the BUILD_TESTING variable.
This should be used by projects to conditionally enable the compilation of tests.
Spack knowns which packages have to run tests and can thus automatically define this variable.
2022-04-28 17:03:13 +00:00
Greg Becker
3e863848f8
build_env/test_env: add concretizer args (#30289) 2022-04-28 11:37:15 +02:00
Harmen Stoppels
e7a0b952ab
install --overwrite: use rename instead of tmpdir (#29746)
I tried to use --overwrite on nvhpc, but nvhpc's install size is 16GB. Seems
better to do os.rename in the same directory than moving the directory to
`/tmp`.

- [x] install --overwrite: use rename instead of tmpdir
- [x] use tempfile
2022-04-28 01:42:42 -06:00
Massimiliano Culpo
d5fc859f46
ASP-based solver: handle installed specs from unknown namespaces (#30092)
fixes #28259

This commit discard specs from unknown namespaces from the
ones that can be "reused" during concretization. Previously
Spack would just error out when encountering them.
2022-04-27 02:10:46 -06:00
Tamara Dahlgren
0c31ab87c9
Feature: Allow re-use of run_test() in install_time_test_callbacks (#26594)
Allow re-use of run_test() in install_time_test_callbacks

Co-authored-by: Greg Becker <becker33@llnl.gov>
2022-04-26 17:40:05 -07:00
Ken Raffenetti
a24070d532
docs: Fix ROCmPackage example syntax (#30168) 2022-04-26 16:00:40 -07:00
Greg Becker
f99614be02
old concretizer: update deprecation warning (#30227) 2022-04-26 14:17:17 -07:00
John W. Parent
83b91246b1
Windows: fix termination of process output redirection (#29923)
The parent thread in the process stdout redirection logic on Windows
was closing a file that was being read in child thread, which lead to
error-based termination of the reader thread. This updates the
interaction to avoid the error.
2022-04-26 12:56:13 -07:00
lorddavidiii
3a0aba0835
spack spec: add '--format' argument (#27908) 2022-04-26 09:08:56 -07:00
Mark W. Krentel
c81affa551
setup_package: don't unload modules on Cray when run with --dirty (#30261)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-04-26 16:53:10 +02:00
Massimiliano Culpo
b9d6a5103d
ASP-based solver: allow configuring target selection (#29835)
* ASP-based solver: allow configuring target selection

This commit adds a new "concretizer:targets" configuration
section, and two options under it.

 - "concretizer:targets:granularity" allows switching from
considering only generic targets to consider all possible
microarchitectures.

 - "concretizer:targets:host_compatible" instead controls
whether we can concretize for microarchitectures that
are incompatible with the current host.

* Add documentation

* Add unit-tests
2022-04-25 17:19:51 -07:00
iarspider
834f8e04ca
Environments: add flag to skip printing concretized specs (#30272)
With an active environment, you can now run "spack concretize --quiet"
and it will suppress printing the concretized specs.
2022-04-25 15:54:54 -07:00
Massimiliano Culpo
268c671dc8
ASP-based solver: always consider version of installed packages (#29933)
* ASP-based solver: always consider version of installed packages

fixes #29201

Explicitly add facts for versions of installed software when
using the --reuse option, so that we could consider versions
that are not declared in package.py
2022-04-25 10:09:49 -07:00
Harmen Stoppels
3dd4999fd7
spec.py: make parser select from environment (#30276)
The parser is already committing a crime of querying the database for
specs when it encounters a `/hash`. It's helpful, but unfortunately not
helpful when trying to install a specific spec in an environment by
hash. Therefore, consider the environment first, then the database.

This allows the following:

```console
$ spack -e . concretize
==> Starting concretization
==> Environment concretized in 0.27 seconds.
==> Concretized diffutils
 -   7vangk4  diffutils@3.8%gcc@10.3.0 arch=linux-ubuntu20.04-zen2
 -   hyb7ehx      ^libiconv@1.16%gcc@10.3.0 libs=shared,static arch=linux-ubuntu20.04-zen2

$ spack -e . install /hyb7ehx
==> Installing libiconv-1.16-hyb7ehxxyqqp2hiw56bzm5ampkw6cxws
...
==> libiconv: Successfully installed libiconv-1.16-hyb7ehxxyqqp2hiw56bzm5ampkw6cxws
  Fetch: 0.01s.  Build: 17.54s.  Total: 17.55s.
[+] /tmp/tmp.VpvYApofVm/store/linux-ubuntu20.04-zen2/gcc-10.3.0/libiconv-1.16-hyb7ehxxyqqp2hiw56bzm5ampkw6cxws
```
2022-04-25 08:41:48 -07:00
Todd Gamblin
d729b4e72b
bugfix: installed and installed_upstream should not assert (#30271)
Fix bug introduced in #30191. `Spec.installed` and `Spec.installed_upstream` should just return
`False` for abstract specs, as they can be called in that context.

- [x] `Spec.installed` returns `False` now instead of asserting that the `Spec`
      is concrete.
- [x] `Spec.installed_upstream` returns `False` now instead of asserting that the `Spec`
      is concrete.
- [x] `Spec.installed_upstream` no longer caches its result, as install status seems
      like a bad thing to cache -- it can easily be invalidated. Calling code should
      use transactions if there are peformance issues, as with other places in Spack.
- [x] add tests for `Spec.installed` and `Spec.installed_upstream`
2022-04-25 07:46:21 +00:00
Massimiliano Culpo
a9fbc0175d
Move "installed" and "installed_upstream" from PackageBase to Spec (#30191)
This PR moves the `installed` and `installed_upstream` properties from `PackageBase` to `Spec` and is a step towards being able to reuse specs for which we don't have a `package.py` available. It _should_ be sufficient to complete the concretization step and see the spec in the concretized DAG.

To fully reuse a spec without a package.py though we need a way to serialize enough data to reconstruct the results of calls to:
- `Spec.libs`, `Spec.headers` and `Spec.ommand`
- `Package.setup_dependent_*_environment` and `Package.setup_run_environment`

- [x] Add stub methods to packages with warnings
- [x] Add a missing "root=False" in cmd/fetch.py
- [x] Assert that a spec is concrete before checking installation status
2022-04-22 16:46:45 -07:00
Peter Scheibel
267da78559
Package-level submodule attribute: support explicit versions (#30085) 2022-04-22 11:02:17 -07:00
Massimiliano Culpo
f961a11187
Update Dockerfiles and images for Spack v0.18.0 (#30216)
This PR updates the list of images we build nightly, deprecating 
Ubuntu 16.04 and CentOS 8 and adding Ubuntu 20.04, Ubuntu 22.04
and CentOS Stream. It also removes a lot of duplication by generating
the Dockerfiles during the CI workflow and uploading them as artifacts
for later inspection or reuse.
2022-04-22 08:51:26 +02:00
Harmen Stoppels
b7f8899d45
docs: add (config.yaml) to sections for faster lookup by config file (#30157) 2022-04-20 15:00:43 +02:00
Adam J. Stewart
83533fc31f
PythonPackage docs: Spack now supports Windows (#30177) 2022-04-20 09:11:19 +02:00
Scott Wittenburg
a710a2425a
ci: Make test independent from any changes in spack repo (#30150)
Fix test_ci_generate_prune_untouched(), which would fail if run when
the latest commit changed the .gitlab-ci.yml.  This change mocks the
get_stack_changed() method in that test to disregard the state of
the current spack repo in favor of a mock repo under test control.
2022-04-19 16:27:24 -06:00
Harmen Stoppels
abc162cf3b
environment.py: write lockfile last (#30039)
This makes it easier to write `spack.lock: spack.yaml` type of rules in `Makefiles`.
2022-04-19 12:32:59 -07:00
John W. Parent
0f80a5a9d5
Remove assert in favor of error message (#29797) 2022-04-15 22:44:44 +00:00
John W. Parent
b1de8a5680
Windows platform: don't allow MakefilePackage (#29904) 2022-04-14 11:41:12 -07:00
Zack Galbreath
dec3e31e60
spack ci: remove relate-CDash-builds functionality (#29950)
gitlab ci: Remove code for relating CDash builds

Relating CDash builds to their dependencies was a seldom used feature.  Removing
it will make it easier for us to reorganize our CDash projects & build groups in the 
future by eliminating the needs to keep track of CDash build ids in our binary mirrors.
2022-04-14 10:42:30 -06:00
Peter Scheibel
89f6db21f1
Ad-hoc Git commit versions: support submodules (#30037)
* Allow packages to add a 'submodules' property that determines when ad-hoc Git-commit-based versions should initialize submodules

* add support for ad-hoc git-commit-based versions to instantiate submodules if the associated package has a 'submodules' property and it indicates this should happen for the associated spec

* allow Package-level submodule request to influence all explicitly-defined version() in the Package

* skip test on windows which fails because of long paths
2022-04-13 20:05:14 -07:00
Massimiliano Culpo
c846b5149d
Add support for Python 3.10 (#29581)
* Add support for Python 3.10

* Update unit-tests to use 3.10

* Update Getting started section of the docs

* Update bootstrap action
2022-04-13 14:32:23 -07:00
Nathan Hanford
a3520d14bd
Splice differing virtual packages (#27919)
Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-12 13:31:39 -07:00
psakievich
7d534f38d6
Don't allow replacement of root develop specs with --reuse (#28605)
* Fix to concretize.lp

do not allow dev specs to be reused

Co-authored-by: Gregory Becker <becker33@llnl.gov>
2022-04-12 10:37:24 -07:00
Peter Scheibel
2aec5b65f3
Git commit versions bugfix: Environments and Concretization (#29717)
Spack added support in #24639 for ad-hoc Git-commit-hash-based
versions: A user can install a package x@hash, where X is a package
that stores its source code in a Git repository, and the hash refers
to a commit in that repository which is not recorded as an explicit
version in the package.py file for X.

A couple issues were found relating to this:

* If an environment defines an alternative package repo (i.e. with
  repos.yaml), and spack.yaml contains user Specs with ad-hoc
  Git-commit-hash-based versions for packages in that repo,
  then as part of retrieving the data needed for version comparisons
  it will attempt to retrieve the package before the environment's
  configuration is instantiated.
* The bookkeeping information added to compare ad-hoc git versions was
  being stripped from Specs during concretization (such that user
  Specs which succeeded before concretizing would then fail after)

This addresses the issues:

* The first issue is resolved by deferring access to the associated
  Package until the versions are actually compared to one another.
* The second issue is resolved by ensuring that the Git bookkeeping
  information is explicitly applied to Specs after they are concretized.

This also:

* Resolves an ambiguity in the mock_git_version_info fixture used to
  create a tree of Git commits and provide a list where each index
  maps to a known commit.
* Isolates the cache used for Git repositories in tests using the
  mock_git_version_info fixture
* Adds a TODO which points out that if the remote Git repository
  overwrites tags, that Spack will then fail when using
  ad-hoc Git-commit-hash-based versions
2022-04-12 09:58:26 -07:00
Doug Jacobsen
a9b4f33f23
Update gpg publish to work with mirror arguments (#28740)
This commit updates the `gpg publish` command to work with the mirror
arguments, when trying to push keys to a mirror.

- [x] update `gpg publish command
- [x] add test for publishing GPG keys and rebuilding the key index within a mirror
2022-04-11 14:48:08 -07:00
Sergey Kosukhin
6e5cba7b82
cray platform: unload cray-mpich (#29898) 2022-04-11 14:34:09 -07:00
Seth R. Johnson
1ea05cd456
macos: fewer calls to sw_vers (#29997)
In a typical call to spack, the OperatingSystem gets instantiated
multiple times. For macOS, each one requires a call to `sw_vers`, which
is done through the Executable helper class. Memoizing
reduces the call count from "spac spec" from three to one.
2022-04-11 09:31:24 +02:00
Massimiliano Culpo
23e85f4086
Environments: unify the spec objects on first concretization (#29948)
Currently environments are indexed by build hashes. When looking into this bug I noticed there is a disconnect between environments that are concretized in memory for the first time and environments that are read from a `spack.lock`. The issue is that specs read from a `spack.lock` don't have a full hash, since they are indexed by a build hash which is strictly coarser. They are also marked "final" as they are read from a file, so we can't compute additional hashes. 

This bugfix PR makes "first concretization" equivalent to re-reading the specs from a corresponding `spack.lock`, and doing so unveiled a few tests were we were making wrong assumptions and relying on the fact that a `spack.lock` file was not there already.

* Add unit test
* Modify mpich to trigger jobs in pipelines
* Fix two failing unit tests
* Fix another full_hash vs. build_hash mismatch in tests
2022-04-08 15:26:17 -06:00
Harmen Stoppels
99083f1706
Deprecate top-level module config (#28659)
* Ignore top-level module config; add auto-update

In Spack 0.17 we got module sets (modules:[name]:[prop]), and for
backwards compat modules:[prop] was short for modules:default:[prop].

But this makes it awkward to define default config for the "default"
module set.

Since 0.17 is branched off, we can now deprecate top-level module config
(that is, just ignore it with a warning).

This PR does that, and it implements `spack config update modules` to
make upgrading easy (we should have added that to 0.17 already...)

It also removes references to  `dotkit` stuff which was already
deprecated in 0.13 and could have been removed in 0.14.

Prefix inspections are the only exception, since the top-level prefix inspections
used for `spack load` and `spack env activate`.
2022-04-08 19:00:35 +00:00
Greg Becker
79ba0c50c1
concretize.lp: enforce target compatibility through DAG (#29694)
Spack currently allows dependencies to be concretized for an 
architecture incompatible with the root. This commit adds rules
to make this situation impossible by design.
2022-04-08 11:01:04 +02:00
Massimiliano Culpo
ff04d1bfc1
Use the non-deprecated MetaPathFinder interface (#29745)
* Extract the MetaPathFinder and Loaders for packages in their own classes

https://peps.python.org/pep-0451/

Currently, RepoPath and Repo implement the (deprecated) interface of
MetaPathFinder (find_module) and of Loader (load_module). This commit
extracts both of them and places the code in their own classes.

The MetaPathFinder interface is updated to contain both the deprecated
"find_module" (for Python 2.7 support) and the recommended "find_spec".
Update of the Loader interface is deferred at a subsequent commit.

* Move the lines to be prepended inside "RepoLoader"

Also adjust the naming of a few variables too

* Remove spack.util.imp, since code is only used in spack.repo

* Remove support from loading Python modules Python > 3 but < 3.5

* Remove `Repo._create_namespace`

This function was interacting badly with the MetaPathFinder
and causing issues with "normal" imports. Removing the
function allows to do things like:
```python
import spack.pkg.builtin.mpich
cls = spack.pkg.builtin.mpich.Mpich
```

* Remove code needed to trigger the Singleton evaluation

The finder is coded in a way to trigger the Singleton,
so we don't need external code now that we register it
at module level into `sys.meta_path`.

* Add unit tests
2022-04-07 15:58:20 -07:00
Scott Wittenburg
b2eda32e55
ci: clean up env between tests with working_dir (#29807) 2022-04-07 17:11:58 +02:00
Scott Wittenburg
ff33978b0d
ci: do not prune specs when stacks are touched (#29825) 2022-04-07 08:55:40 -06:00
Harmen Stoppels
43577beb9c
autotools.py: pic flags for %nvhpc (#29920) 2022-04-07 14:52:47 +02:00
Harmen Stoppels
beff697cc0
web.py: set User-Agent (#29919)
Some servers require `User-Agent` to be set, and otherwise error with
access denied. One such example is mpich.

To fix this, set `User-Agent: Spackbot/[version]` as a header.
Apparently by convention, it should include the word `bot`.
2022-04-06 15:59:20 +02:00
Harmen Stoppels
558f7f007e
link_tree.py: format conflict error message (#29870)
Show each `[src a] and [src b] both project to [dst]` on a separate
line.
2022-04-06 07:27:02 +02:00
Erik Schnetter
69b3a88fa3
Bugfix: CVS fetching (#29793)
#27021 broke fetching for CVS-based packages because:

- The mirror logic was using URL parsing to extract a path from the
  CVS repository location
- #27021 added sanity checks to enforce that strings passed to the
  URL parser were actually URLs

This replaces the call to "url_util.parse" with logic that is
customized for CVS. This implies that VCSFetchStrategy should
rename the "url_attr" attribute to something more generic, but
that should be handled separately.
2022-04-05 18:45:45 -07:00
Massimiliano Culpo
f2fc4ee9af
Allow conditional possible values in variants (#29530)
Allow declaring possible values for variants with an associated condition. If the variant takes one of those values, the condition is imposed as a further constraint.

The idea of this PR is to implement part of the mechanisms needed for modeling [packages with multiple build-systems]( https://github.com/spack/seps/pull/3). After this PR the build-system directive can be implemented as:
```python
variant(
    'build-system',
    default='cmake',
    values=(
        'autotools',
        conditional('cmake', when='@X.Y:')
    ), 
    description='...',
)
```

Modifications:
- [x] Allow conditional possible values in variants
- [x] Add a unit-test for the feature
- [x] Add documentation
2022-04-04 17:37:57 -07:00
Nathan Hanford
88d8ca9b65
rewiring of spliced specs (#26873)
* tests for rewiring pure specs to spliced specs

* relocate text, binaries, and links

* using llnl.util.symlink for windows compat.

Note: This does not include CLI hooks for relocation.

Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-04 14:45:35 -07:00
Thomas Dickerson
ee505e6c69
Add support for racket packages (#27564)
- Add variants for various common build flags, including support for both versions of the Racket VM environment.
- Prevent `-j` flags to `make`, which has been known to cause problems with Racket builds.
- Prefer the minimal release to improve install times. Bells and whistles carry their own runtime dependencies and should be installed via `raco`. An enterprising user may even create a `RacketPackage` class to make spack aware of `raco` installed packages.
- Match the official version numbering scheme.
2022-04-04 10:32:25 +02:00
Brian Van Essen
29da99427e
"spack external find": also find library-only packages (#28005)
Update "spack external find --all" to also find library-only packages.
A Package can add a ".libraries" attribute, which is a list of regular
expressions to use to find libraries associated with the Package.
"spack external find --all" will search LD_LIBRARY_PATH for potential
libraries.

This PR adds examples for NCCL, RCCL, and hipblas packages. These
examples specify the suffix ".so" for the regular expressions used
to find libraries, so generally are only useful for detecting library
packages on Linux.
2022-04-01 13:30:10 -07:00
Peter Scheibel
a58fa289b9
Allow "spack install foo@git-hash" without terminal prompt (#29827)
Do not prompt user with checksum warning when using git commit hashes
as versions. Spack was incorrectly reporting this as a potential
problem: it would display a prompt asking the user whether they
want to proceed if Spack was running in a terminal, or it would
terminate the running instance of Spack if running as part of a
script.
2022-04-01 10:27:03 -07:00
John W. Parent
88b1bf751d
Windows Support: Fixup Perl build (#29711)
* Add pl2bat to PATH: Windows on Perl requires the script pl2bat.bat
  and Perl to be available to the installer via the PATH. The build
  and dependent environments of Perl on Windows have the install
  prefix bin added to the PATH.
* symlink with win32file module instead of using Executable to
  call mklink (mklink is a shell function and so is not accessible
  in this manner).
2022-03-31 11:47:11 -07:00
Ivo Steinbrecher
44263b7226
Allow use of "git:"-based URLs (#29765) 2022-03-31 10:14:18 -07:00
Scott Wittenburg
685e3d7ae9
spack ci: filter untouched pkgs from PR pipelines (#29697)
We've previously generated CI pipelines for PRs, and they rebuild any packages that don't have
a binary in an existing build cache.  The assumption we were making was that ALL prior merged
builds would be in cache, but due to the way we do security in the pipeline, they aren't. `develop`
pipelines can take a while to catch up with the latest PRs, and while it does that, there may be a
bunch of redundant builds on PRs that duplicate things being rebuilt on `develop`.  Until we can
do better caching of PR builds, we'll have this problem.

We can do better in PRs, though, by *only* rebuilding things in the CI environment that are actually
touched by the PR.  This change computes exactly what packages are changed by a PR branch and
*only* includes those packages' dependents and dependencies in the generated pipeline.  Other
as-yet unbuilt packages are pruned from CI for the PR.

For `develop` pipelines, we still want to build everything to ensure that the stack works, and to ensure
that `develop` catches up with PRs. This is especially true since we do not do rebuilds for *every* commit
on `develop` -- just the most recent one after each `develop` pipeline finishes.  Since we skip around,
we may end up missing builds unless we ensure that we rebuild everything.

We differentiate between `develop` and PR pipelines in `.gitlab-ci.yml` by setting 
`SPACK_PRUNE_UNTOUCHED` for PRs. `develop` will still have the old behavior.

- [x] Add `SPACK_PRUNE_UNTOUCHED` variable to `spack ci`
- [x] Refactor `spack pkg` command by moving historical package checking logic to `spack.repo`
- [x] Implement pruning logic in `spack ci` to remove untouched packages
- [x] add tests
2022-03-30 17:17:29 -07:00
Adam J. Stewart
f97be99f2a
Mixed compilers: drop warning message to debug level (#29748) 2022-03-30 15:18:37 -06:00
Jörg Behrmann
4571f4c994
Add completion of locals to spack python (#29702) 2022-03-30 14:38:39 -06:00