Commit graph

5473 commits

Author SHA1 Message Date
Harmen Stoppels
80195bd1ed
sbang.py: single lstat (#29670) 2022-03-22 23:12:06 +01:00
Greg Becker
dba7a03daa
config add: fix parsing of validator error to infer type from oneOf (#29475) 2022-03-21 16:55:02 +01:00
Harmen Stoppels
8f5b9a89fb
major.minor.micro.dev0 Spack version (#25267)
When you install Spack from a tarball, it will always show an exact
version for Spack itself, even when you don't download a tagged commit:

```
$ wget -q https://github.com/spack/spack/archive/refs/heads/develop.tar.gz
$ tar -xf develop.tar.gz
$ ./spack-develop/bin/spack --version
0.16.2
```

This PR sets the Spack version to `0.18.0.dev0` on develop, following [PEP440](https://github.com/spack/spack/pull/25267#issuecomment-896340234) as
suggested by Adam Stewart.

```
spack (fix/set-dev-version)$ spack --version
0.18.0.dev0 (git 0.17.1-1526-e270464ae0)
spack (fix/set-dev-version)$ mv .git .git_
spack $ spack --version
0.18.0.dev0
```

- [x] Update the release guide
- [x] Add __version__ to spack's __init__.py
- [x] Use PEP 440 canonical version strings
- [x] Make spack --version output [actual version] (git version)

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-20 22:37:55 -06:00
Doug Jacobsen
9b0d5cbabf
Enable GCS urls as valid (#29592)
* Add tests to ensure google cloud storage urls work as mirrors

This commit adds two tests to track that GCS buckets can work as
mirrors, and can be parsed as valid URLs.

Currently, gs:// format URLs are not correctly parsed.

* Fix URL parsing for GCS buckets

This commit adds GCS bucket URLs as valid URLs.
2022-03-20 01:13:00 -07:00
Peter Scheibel
26552533be
We were doing a check in a metaclass to ensure that Package classes weren't defining a function themselves; this check failed (erroneously) if that Package class subclassed another package class (because the check examined all superclasses and thought the definition we automatically add was an offender) (#29569) 2022-03-19 21:37:40 -07:00
Nils Vu
bfb6873ce3
Revert "Add command for reading a json-based DB description (#24894)" (#29603)
This reverts commit 531b1c5c3d.
2022-03-19 16:30:46 -06:00
Tom Scogland
3ffe9a27b2
allow ci reproduce to work when spack is cloned to a directory not named spack (#29518) 2022-03-18 20:22:28 -06:00
Tom Scogland
9e01e17dc6
R versions (#29258)
* lower priority of package-provided urls

This change favors urls found in a scraped page over those provided by
the package from `url_for_version`.  In most cases this doesn't matter,
but R specifically returns known bad URLs in some cases, and the
fallback path for a failed fetch uses `fetch_remote_versions` to find a
substitute.  This fixes that problem.

fixes #29204

* consider what links actually exist in all cases

Checksum was only actually scraping when called with no versions.  It
now always scrapes and then selects URLs from the set of URLs known to
exist whenever possible.

fixes #25831

* bow to the wrath of flake8

* test-fetch urls from package, prefer if successful

* Update lib/spack/spack/package.py

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>

* reword as suggested

* re-enable mypy specific ignore and ignore pyflakes

* remove flake8 ignore from .flake8

* address review comments

* address comments

* add sneaky missing substitute

I missed this one because we call substitute on a URL that doesn't
contain a version component.  I'm not sure how that's supposed to work,
but apparently it's required by at least one mock package, so back in it
goes.

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
2022-03-18 18:42:17 -06:00
Peter Scheibel
531b1c5c3d
Add command for reading a json-based DB description (#24894)
Adds `spack external read-cray-manifest`, which reads a json file that describes a set of package DAGs. The parsed results are stored directly in the database. A user can see these installed specs with `spack find` (like any installed spec). The easiest way to use them right now as dependencies is to run `spack spec ... ^/hash-of-external-package`.

Changes include:

* `spack external read-cray-manifest --file <path/to/file>` will add all specs described in the file to Spack's installation DB and will also install described compilers to the compilers configuration (the expected format of the file is described in this PR as well including examples of the file)
* Database records now may include an "origin" (the command added in this PR registers the origin as "external-db"). In the future, it is assumed users may want to be able to treat installs registered with this command differently (e.g. they may want to uninstall all specs added with this command)
* Hash properties are now always preserved when copying specs if the source spec is concrete
  * I don't think the hashes of installed-and-concrete specs should change and this was the easiest way to handle that
  * also specs that are concrete preserve their `.normal` property when copied (external specs may mention compilers that are not registered, and without this change they would fail in `normalize` when calling `validate_or_raise`)
  * it might be this should only be the case if the spec was installed

- [x] Improve testing
- [x] Specifically mark DB records added with this command (so that users can do something like "uninstall all packages added with `spack read-external-db`)
  * This is now possible with `spack uninstall --all --origin=external-db` (this will remove all specs added from manifest files)
- [x] Strip variants that are listed in json entries but don't actually exist for the package

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-03-18 17:07:22 -07:00
Harmen Stoppels
62fcf407e2
environment: Don't print 'updating view at' when no specs (#29536) 2022-03-18 08:49:47 -04:00
Massimiliano Culpo
2fa495154e
Split the workflow section and remove outdated advices (#29344)
This PR removes a few outdated sections from the "Basics" part of the 
documentation. It also makes a few topic under the environment section
more prominent by removing an unneeded spack.yaml subsection and 
promoting everything under it.
2022-03-18 10:41:27 +01:00
Harmen Stoppels
ea13122f3c
Add deprecation warnings for activate/deactivate/view (#29430) 2022-03-17 14:52:56 -07:00
Zack Galbreath
5a72204d38
Remove references to features/windows-support branch (#29565) 2022-03-17 10:29:29 -07:00
John Parent
4aee27816e Windows Support: Testing Suite integration
Broaden support for execution of the test suite
on Windows.
General bug and review fixups
2022-03-17 09:01:01 -07:00
Peter Josef Scheibel
e63b4f752a Failed _write now track that the DB is inconsistent
Consistency is restored on next transaction
2022-03-17 09:01:01 -07:00
John Parent
cf1349ba35 "spack commands --update-completion" 2022-03-17 09:01:01 -07:00
John W. Parent
e4d4a5193f Path handling (#28402)
Consolidate Spack's internal filepath logic to a select
few places and refactor to consistent internal useage of
os.path utilities. Creates a prefix, and a series of utilities
in the path utility module that facilitate handling paths
in a platform agnostic manner.

Convert Windows paths to posix paths internally

Prefer posixpath.join instead of os.path.join

Updated util/ directory to account for Windows integration

Co-authored-by: Stephen Crowell <stephen.crowell@khq.kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>

Module template format for windows (#23041)
2022-03-17 09:01:01 -07:00
John Parent
df4129d395 Expand external find for Windows (#27588)
* Incorporate new search location

* Add external user option

* proper doc string

* Explicit commands in getting started

* raise during chgrp on Win

recover installer changes

Notate admin privleges

Windows phase install hooks

Find external python and install ninja (#23496)

Allow external find python to find windows python and spack install ninja

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
06aef626cb Update tests support for Windows
Fixup common tests

    * Remove requirement for Python 2.6
    * Skip new failing test

Windows: Update url util to handle Windows paths (#27959)

    * update url util to handle windows paths

    * Update tests to handle fixed url handling

    * canonicalize path only when the path type matches the host platform

    * Skip some url tests on Windows

Co-authored-by: Omar Padron <omar.padron@kitware.com>

Use threading.TIMEOUT_MAX when available (#24246)

This value was introduced in Python 3.2. Specifying a timeout greater than
this value will raise an OverflowError.

Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
John Parent
31daf0f2b6 Add Windows to platform and target changes
Add compiler hint to the root spec for Windows

Reporters on Windows (#26038)

Reporters use Jinja2 as the templating engine, and Jinja2 indexes
templates by Unix separators, even on Windows, so search using Unix paths
on all systems.

Support patching on win via git (#25871)

Handle GRP on windows
2022-03-17 09:01:01 -07:00
John Parent
3a994032f8 Spack on Windows package ports
CMake - Windows Bootstrap (#25825)

Remove hardcoded cmake compiler (#26410)

Revert breaking cmake changes
Ensure no autotools on Windows

Perl on Windows (#26612)

Python source build windows (#26313)

Reconfigure sysconf for Windows

Python2.6 compatibility

Fxixup new sbang tests for windows

Ruby support (#28287)

Add NASM support (#28319)

Add mock Ninja package for testing
2022-03-17 09:01:01 -07:00
John W. Parent
e65d3d14b4 Relocate spack_cmd and scripts from installer to root bin (#24651)
Relocate spack_cmd and scripts from installer
to root bin

Refactor documentation, installer, and launcher
to facilitate that change
2022-03-17 09:01:01 -07:00
Betsy McPhail
4a73bfc3b9 Use Python's zipfile, if available (#24556)
* Style fixes

* Use Python's zipfile, if available

The compression libs are optional in Python. Rely on python as a
first attempt then fall back to `unzip`
2022-03-17 09:01:01 -07:00
Betsy McPhail
f8782c46d7 Windows: VS and CMake support
Fix: Building packages with CMake is broken (#24241)

vsInstallPaths needs to be renamed vs_install_paths (#24297)
2022-03-17 09:01:01 -07:00
Jared Popelar
15ef85e161 Packaging of netlib-lapack for windows (#24993)
MSVC's internal CMake and Ninja now detected by spack external find and added to packages.yaml

Saving progress on packaging zlib for Windows

Fixing the shared CMake flag

* Loading Intel's ifx Fortran compiler into MSVC; if there are multiple
versions of MSVC installed and detected, ifx will only be placed into
the first block written in compilers.yaml. The version number of ifx can
be detected using MSVC's version flag (instead of /QV) by using
ignore_version_errors. This commit also provides support for detection
of Intel compilers in their own compiler block by adding ifx.exe to the
fc/f77_name blocks inside intel.py

* Giving CMake a Fortran compiler argument

* Adding patch file for removing duplicated mangling header for versions 3.9.1 and older; static and shared now successfully building on Windows

* Have netlib-lapack depend  on ninja@1.10

Co-authored-by: John R. Cary <cary@txcorp.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>

Making a default config.yaml for Windows

Small path length for build_stage

Provide more prerequisite details, mention default config.yaml

Killing an unnecessary setvars call

Replacing some lost changes, proofreading, updating windows-supported package list

Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
lou.lawrence@kitware.com
012758c179 Windows: Create installer and environment
* Add 'make-installer' command for Windows

* Add '--bat' arg to env activate, env deactivate and unload commands

* An equivalent script to setup-env on linux: spack_cmd.bat. This script
has a wrapper to evaluate cd, load/unload, env activate/deactivate.(#21734)

* Add spacktivate and config editor (#22049)

* spack_cmd: will find python and spack on its own. It preferentially
tries to use python on your PATH (#22414)

* Ignore Windows python installer if found (#23134)

* Bundle git in windows installer (#23597)

* Add Windows section to Getting Started document
(#23131), (#23295), (#24240)

Co-authored-by: Stephen Crowell <stephen.crowell@kitware.com>
Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>
Co-authored-by: Ben Cowan <benc@txcorp.com>

Update Installer CI

Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
a7101db39d MSVC Compiler Find and vcvars env integration (#22856)
Made the vcvars batch script location a member variable of the msvc compiler subclass, initialized from the compiler executable path.  Added a setup_custom_environment() method to the msvc subclass that sources the vcvars script, dumps the environment, and copies the relevant environment variables to the Spack environment.  Added class variables to the Windows OS and MSVC compiler subclasses to enable finding the compiler executables and determining their versions.
2022-03-17 09:01:01 -07:00
loulawrence
f587a9ce68 use pytest stdout/err capture (#22584)
* On windows, write to StringIO without redirection in test cases to avoid conflicting with logger
2022-03-17 09:01:01 -07:00
Ben Cowan
a0164793cb Features/windows install (#22204)
* Fixed path and uid issues.

* Added needed import statement; kluged .exe extension.

* Got package to build.  Some manual intervention necessary, including sourcing the MSVC setup script and having certain configuration parameters.

* Removed CMake executable suffix hack.
2022-03-17 09:01:01 -07:00
Betsy McPhail
d4d101f57e Allow 'spack external find' to find executables on the system path (#22091)
Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
fb0e91c534 Windows: Symlink support
To provide Windows-compatible functionality, spack code should use
llnl.util.symlink instead of os.symlink. On non-Windows platforms
and on Windows where supported, os.symlink will still be used.

Use junctions when symlinks aren't supported on Windows (#22583)

Support islink for junctions (#24182)

Windows: Update llnl/util/filesystem

* Use '/' as path separator on Windows.
* Recognizing that Windows paths start with '<Letter>:/' instead of '/'

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
a7de2fa380 Create rename utility function
os.rename() fails on Windows if file already exists.

Create getuid utility function (#21736)

On Windows, replace os.getuid with ctypes.windll.shell32.IsUserAnAdmin().

Tests: Use getuid util function

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
b60a0eea01 Workarounds for install errors on Windows (#21890)
1. Forwarding sys.stdin, e.g. use input_multiprocess_fd,
gives an error on Windows. Skipping for now

3.  subprocess_context needs to serialize for Windows, like it does
for Mac.

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Ben Cowan
81bc00d61f Adding basic Windows features (#21259)
* Snapshot of some MSVC infrastructure added during experiments a while ago.  Rebasing from spack/develop.

* Added platform and OS definitions for Windows.

* Updated Windows platform file to conform to new archspec use.

* Added Windows as a platform; introduced some debugging code.

* Added type annotations.

* Fixed copyright.

* Removed print statements.

* Ensure `spack arch` returns correctly on Windows (#21428)

* Correctly identify windows as 'windows-Windows10-AMD64'
2022-03-17 09:01:01 -07:00
Harmen Stoppels
55544950e2
PackageViewMixin: fix symlinks conflict issue (#29515)
`stat`'ing a file in the dst dir is the wrong thing to do, you should
`lstat` to capture broken symlinks.
2022-03-17 08:18:28 -06:00
Tom Scogland
94794d061a
optimize instantiation and comparison of versions (#29429)
Re-work the checks and comparisons around commit versions, when no
commit version is involved the overhead is now in the noise, where one
is the overhead is now constant rather than linear.
2022-03-15 18:53:28 +00:00
Sergey Kosukhin
f1bdbefd46
mpich: fix the detection of the package (#29284) 2022-03-14 16:33:56 +01:00
Harmen Stoppels
474493713b
compiler.py: early return in compiler_environment when no modules (#29441) 2022-03-14 11:27:49 +00:00
Massimiliano Culpo
a6eed4a7c7
Remove "setup_environment" and "setup_dependent_environment" (#29463)
fixes #29446

The new setup_*_environment functions have been falling back
to calling the old functions and warn the user since #11115.

This commit removes the fallback behavior and any use of:
- setup_environment
- setup_dependent_environment
in the codebase
2022-03-13 11:51:55 -04:00
Harmen Stoppels
8adc6b7e8e
module_cmd.py: use posix awk; fix stderr redirection (#29440)
Emulates `env -0` in a posix compliant way, avoiding a slow python process, speeds up setting up the build env when modules should load.
2022-03-11 18:29:11 +01:00
Harmen Stoppels
88fbba3e1e
intel-mkl: load compiler modules when querying compiler (#29439) 2022-03-10 14:14:17 -08:00
Massimiliano Culpo
2cd5c00923
Allow for multiple dependencies/dependents from the same package (#28673)
Change the internal representation of `Spec` to allow for multiple dependencies or
dependents stemming from the same package. This change permits to represent cases
which are frequent in cross compiled environments or to bootstrap compilers.

Modifications:
- [x] Substitute `DependencyMap` with `_EdgeMap`. The main differences are that the
      latter does not support direct item assignment and can be modified only through its
      API. It also provides a `select_by` method to query items.
- [x] Reworked a few public APIs of `Spec` to get list of dependencies or related edges.
- [x] Added unit tests to prevent regression on #11983 and prove the synthetic construction
      of specs with multiple deps from the same package. 

Since #22845 went in first, this PR reuses that format and thus it should not change hashes.
The same package may be present multiple times in the list of dependencies with different
associated specs (each with its own hash).
2022-03-10 11:53:45 -08:00
Harmen Stoppels
dc78f4c58a
environment.py: allow link:run (#29336)
* environment.py: allow link:run

Some users want minimal views, excluding run-type dependencies, since
those type of dependencies are covered by rpaths and the symlinked
libraries in the view aren't used anyways.

With this change, an environment like this:

```
spack:
  specs: ['py-flake8']
  view:
    default:
      root: view
      link: run
```

includes python packages and python, but no link type deps of python.
2022-03-09 12:35:26 -08:00
百地 希留耶
3270aa106b
Hide package repository count when redirecting stdout (#29402) 2022-03-09 17:07:07 +00:00
百地 希留耶
3370d3f57e
Fix tab completion erroring with spack unit-test (#29405) 2022-03-09 16:09:57 +01:00
Harmen Stoppels
3df90e3e33
ci.py: fix utf-8 issue with git log output (#29419) 2022-03-09 13:13:37 +00:00
Harmen Stoppels
a39a6c4ea7
version.py: set is_commit in constructor (#29369)
Speeds up comparison on `Version` by ~2.5x, e.g.

```python
In [1]: v = spack.version.Version('1.0.0'); w = spack.version.Version('1.0.2')

In [2]: %timeit v < w

1.47 µs ± 5.59 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

535 ns ± 1.75 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```
2022-03-08 10:50:48 +01:00
Todd Gamblin
d424d0e54e
find: deprecate spack find -bootstrap for spack -bootstrap find (#29152)
Since adding the `spack --bootstrap` argument we don't need `spack find --bootstrap`
anymore.  Deprecate it.
2022-03-05 21:41:18 -08:00
Harmen Stoppels
80a6eab2db Add a test 2022-03-04 18:13:34 +01:00
Harmen Stoppels
9199800bbd VersionRange.satisfies should test non-empty intersection 2022-03-04 18:13:34 +01:00
Massimiliano Culpo
b48bdc9e19
Fix importing Spack packages as Python modules (#29221)
fixes #29203

This PR fixes a subtle bug we have when importing
Spack packages as Python modules that can lead to
multiple module objects being created for the same
package.

It also fixes all the places in unit-tests where
"relying" on the old bug was crucial to have a new
"clean" state of the package class.
2022-03-04 08:42:27 +01:00
Doug Jacobsen
06fd0f892e
Revert GCS fetch strategy, to remove s3 interface (#28736)
This commit reverts the GCS fetch strategy to before commit:
d759612523

The previous commit added some s3 syntax to handle connections, but
added them into the GCS fetch strategy in a way that prevents GCS from
working anymore.
2022-03-03 16:34:33 -08:00
Greg Becker
e2b87ade06
rocmcc compiler: initial commit based on aocc and clang (#28575)
* rocmcc compiler: initial commit based on aocc and clang

Co-authored-by: luker <luke.roskop@hpe.com>
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
2022-03-03 14:34:22 -07:00
Michael Kuhn
49069e4f58
installer: Fix cosmetic problem with terminal title (#29070)
The status displayed in the terminal title could be wrong when doing
distributed builds. For instance, doing `spack install glib` in two
different terminals could lead to the current package being reported as
`40/29` due to the way Spack handles retrying locks.

Work around this by keeping track of the package IDs that were already
encountered to avoid counting packages twice.
2022-03-03 14:21:15 +01:00
Scott Wittenburg
c72735229f
test/installer.py: remove commented code and inaccurate docstring (#29305) 2022-03-03 10:19:57 +01:00
Danny McClanahan
2c331a1d7f
make @llnl.util.lang.memoized support kwargs (#21722)
* make memoized() support kwargs

* add testing for @memoized
2022-03-02 11:12:15 -08:00
Massimiliano Culpo
8d118104c7
Fix typos when forwarding arguments to traverse_edges (#29261)
A few calls use `deptypes=...` instead of `deptype=...`
2022-03-02 08:43:26 +01:00
Tamara Dahlgren
b20df12d09
test_env_install_two_specs_same_dep: properly check installed specs (#29222) 2022-03-01 10:35:14 +01:00
Scott Wittenburg
b082c33c85
commands: Propgate command return value as exit code (#29223) 2022-02-25 10:49:56 -08:00
Tamara Dahlgren
0b4f40ab79
Testing: Summarize test results and add verbose output (#28700) 2022-02-23 18:36:21 -08:00
Massimiliano Culpo
1ddad522a4
Move early exit for setup only argument (#29041)
See https://github.com/spack/spack/pull/28468/files#r809156986

If we exit before generating the:

 error("Dependencies must have compatible OS's with their dependents").
 ...

facts we'll output a problem that is effectively
different by the one solved by clingo.
2022-02-23 01:46:52 -08:00
Tom Scogland
a9ba40164a
Checksum match (#28989)
* cmd/checksum: prefer url matching url_from_version

This is a minimal change toward getting the right archive from places
like github.  The heuristic is:

* if an archive url exists, take its version
* generate a url from the package with pkg.url_from_version
* if they match
  * stop considering other URLs for this version
  * otherwise, continue replacing the url for the version

I doubt this will always work, but it should address a variety of
versions of this bug.  A good test right now is `spack checksum gh`,
which checksums macos binaries without this, and the correct source
packages with it.

fixes #15985
related to #14129
related to #13940

* add heuristics to help create as well

Since create can't rely on an existing package, this commit adds another
pair of heuristics:
1. if the current version is a specifically listed archive, don't
   replace it
2. if the current url matches the result of applying
   `spack.url.substitute_version(a, ver)` for any a in archive_urls,
   prefer it and don't replace it

fixes #13940

* clean up style and a lingering debug import

* ok flake8, you got me

* document reference_package argument

* Update lib/spack/spack/util/web.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* try to appease sphinx

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2022-02-23 00:55:59 +00:00
Todd Gamblin
36b0730fac
Add spack --bootstrap option for accessing bootstrap store (#25601)
We can see what is in the bootstrap store with `spack find -b`, and you can clean it with `spack
clean -b`, but we can't do much else with it, and if there are bootstrap issues they can be hard to
debug.

We already have `spack --mock`, which allows you to swap in the mock packages from the command
line. This PR introduces `spack -b` / `spack --bootstrap`, which runs all of spack with
`ensure_bootstrap_configuration()` set. This means that you can run `spack -b find`, `spack -b
install`, `spack -b spec`, etc. to see what *would* happen with bootstrap configuration, to remove
specific bootstrap packages, etc. This will hopefully make developers' lives easier as they deal
with bootstrap packages.

This PR also uses a `nullcontext` context manager. `nullcontext` has been implemented in several
other places in Spack, and this PR consolidates them to `llnl.util.lang`, with a note that we can
delete the function if we ever reqyire a new enough Python.

- [x] introduce `spack --bootstrap` option
- [x] consolidated all `nullcontext` usages to `llnl.util.lang`
2022-02-22 12:35:34 -07:00
Todd Gamblin
7912a8e90b
bugfix: Not all concrete versions on the CLI should be considered real (#28620)
Some "concrete" versions on the command line, e.g. `qt@5` are really
meant to satisfy some actual concrete version from a package. We should
only assume the user is introducing a new, unknown version on the CLI
if we, well, don't know of any version that satisfies the user's
request.  So, if we know about `5.11.1` and `5.11.3` and they ask for
`5.11.2`, we'd ask the solver to consider `5.11.2` as a solution.  If
they just ask for `5`, though, `5.11.1` or `5.11.3` are fine solutions,
as they satisfy `@5`, so use them.

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-02-21 11:46:37 -08:00
Alberto Invernizzi
b8d042273a
Bring back cuda@11.4.0 conflicts for GCC and clang; add 11.4.3:11.4.4 (#29076)
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-02-21 20:28:56 +01:00
Massimiliano Culpo
7fd94fc4bc
spack external find: change default behavior (#29031)
See https://github.com/spack/spack/issues/25353#issuecomment-1041868116

This commit changes the default behavior of
```
$ spack external find
```
from searching all the possible packages Spack knows about to
search only for the ones tagged as being a "build-tool".

It also introduces a `--all` option to restore the old behavior.
2022-02-18 11:51:01 -07:00
Seth R. Johnson
6c61c2695a
darwin: robust macos version detection (#28991)
Prefer `sw_vers` to `platform.mac_ver`. In anaconda3 installation, for example, the latter reports 10.16 on Monterey -- I think this is affected by how and where the python instance was built.

Use MACOSX_DEPLOYMENT_TARGET if present to override the operating system choice.
2022-02-17 20:50:41 -07:00
Scott Wittenburg
38643dcd7e
gitlab: Propagate stack name to downstream build jobs (#29019)
It will be useful for metrics gathering and possibly debugging to
have this environment variable available in the runner pods that
do the actual rebuilds.
2022-02-17 15:36:48 -05:00
Tamara Dahlgren
fefe65a35b
Testing: optionally run tests on externally installed packages (#28701)
Since Spack does not install external packages, this commit skips them by
default when running stand-alone tests. The assumption is that such packages
have likely undergone an acceptance test process. 

However, the tests can be run against installed externals using 
```
% spack test run --externals ...
```
2022-02-17 19:47:42 +01:00
Harmen Stoppels
d93f9b82ac
Reduce verbosity of patches=... variant (#29015)
* reduce verbosity of patches=... variant

* Special-case prefix-matches for satisfies of patches variant
2022-02-17 11:06:32 +01:00
Massimiliano Culpo
fa132614e0
ASP-based solver: don't sort when defining variant possible values (#29013)
fixes #28260

Since we iterate over different variants from many packages, the variant 
values may have types which are not comparable, which causes errors 
at runtime. This is not a real issue though, since we don't need the facts
to be ordered. Thus, to avoid needless sorting, the sorted function has 
been removed and a comment has been added to tip any developer that
might need to inspect these clauses for debugging to add back sorting 
on the first two items only.

It's kind of difficult to add a test for this, since the error depends on 
whether Python sorting algorithm ever needs to compare the third 
value of a tuple being ordered.
2022-02-17 08:50:50 +01:00
Tom Scogland
8f5fcc6e95
extensions: allow multiple "extends" directives (#28853)
* extensions: allow multiple "extends" directives

This will allow multiple extends directives in a package as long as only one of
them is selected as a dependency in the concrete spec.

* document the option to have multiple extends
2022-02-16 21:23:12 +00:00
Todd Gamblin
b1ff9c05bc concretizer: refactor argument passing for reuse
Reuse previously was a very invasive change that required parameters to be added to all
the methods that called `concretize()` on a `Spec` object. With the addition of
concretizer configuration, we can use the config system to simplify this argument
passing and keep the code cleaner.

We decided that concretizer config options should be read at `Solver` instantiation
time, and if config changes between instnatiation of a particular solver and
`solve()` invocation, the `Solver` should use the settings from `__init__()`.

- [x] remove `reuse` keyword argument from most concretize functions
- [x] refactor usages to use `spack.config.override("concretizer:reuse", True)`
- [x] rework argument passing in `Solver` so that parameters are set from config
      at instantiation time
2022-02-16 10:17:18 -08:00
Todd Gamblin
d33973df6c docs: add section on concretizer configuration
* Document `concretizer.yaml`, `--reuse`, and `--fresh`.
2022-02-16 10:17:18 -08:00
Todd Gamblin
a2b8e0c3e9 commands: refactor --reuse handling to use config
`--reuse` was previously handled individually by each command that
needed it. We are growing more concretization options, and they'll
need their own section for commands that support them.

Now there are two concretization options:

* `--reuse`: Attempt to reuse packages from installs and buildcaches.
* `--fresh`: Opposite of reuse -- traditional spack install.

To handle thes, this PR adds a `ConfigSetAction` for `argparse`, so
that you can write argparse code like this:

```
     subgroup.add_argument(
        '--reuse', action=ConfigSetAction, dest="concretizer:reuse",
        const=True, default=None,
        help='reuse installed dependencies/buildcaches when possible'
     )
```

With this, you don't need to add logic to pull the argument out and
handle it; the `ConfigSetAction` just does it for you. This can probably
be used to clean up some other commands later, as well.

Code that was previously passing `reuse=True` around everywhere has
been refactored to use config, and config is set from the CLI using
a new `add_concretizer_args()` function in `spack.cmd.common.arguments`.

- [x] Add `ConfigSetAction` to simplify concretizer config on the CLI
- [x] Refactor code so that it does not pass `reuse=True` to every function.
- [x] Refactor commands to use `add_concretizer_args()` and to pass
      concretizer config using the config system.
2022-02-16 10:17:18 -08:00
Todd Gamblin
f155de7462 tests: consolidate mock scope creation logic in conftest.py
Config scopes were different for `config` and `mutable_config`,
and `mutable_config` did not have a command line scope.

- [x] Fix by consolidating the creation logic for the two fixtures.
2022-02-16 10:17:18 -08:00
Todd Gamblin
800ed16e7a config: add a new concretizer config section
The concretizer is going to grow to have many more configuration,
and we really need some structured config for that.

* We have the `config:concretizer` option that chooses the solver,
  but extending that is awkward (we'd need to replace a string with
  a `dict`) and the solver choice will be deprecated eventually.

* We have the `concretization` option in environments, but it's
  not a top-level config section -- it's just for environments,
  and it also only admits a string right now.

To avoid overlapping with either of these and to allow the most
extensibility in the future, this adds a new `concretizer` config
section that can be used in and outside of environments. There
is only one option right now: `reuse`.  This can expand to include
other options later.

Likely, we will soon deprecate `config:concretizer` and warn when
the user doesn't use `clingo`, and we will eventually (sometime later)
move the `together` / `separately` options from `concretization` into
the top-level `concretizer` section.

This commit just adds the new section and schema. Fully wiring it
up is TBD.
2022-02-16 10:17:18 -08:00
Todd Gamblin
1903e45eec refactor: convert spack.solver.asp.solve() to a class
The solver has a lot of configuration associated with it. Rather
than adding arguments to everything, we should encapsulate that
in a class. This is the start of that work; it replaces `solve()`
and its kwargs with a class and properties.
2022-02-16 10:17:18 -08:00
Mark W. Krentel
87a3b72ef0
Add 'stable' to the list of infinity version names. (#28772)
* Add 'stable' to the list of infinity version names.
Rename libunwind 1.5-head to 1.5-stable.

* Add stable to the infinite version list in packaging_guide.rst.
2022-02-16 09:08:51 -08:00
Adam J. Stewart
3c1b2c0fc9
find_libraries: search for both .so and .dylib on macOS (#28924) 2022-02-16 14:07:44 +01:00
Stephen Sachs
79f22423b8
intel compiler: fix link time error with LLVMgold.so (#28731)
The Intel compiler will, at link time, call `ld -plugin LLVMgold.so`, which
expects libraries like `libimfo.so` to be found either in the `LD_LIBRARY_PATH` or
in `LLVMgold.so`s RPATH.

As `LLVMgold.so` already uses RUNPATH, I used that to extend this to the
necessary library locations.

This PR should fix issues:
https://github.com/spack/spack/issues/10308
https://github.com/spack/spack/issues/18606
https://github.com/spack/spack/issues/17100
https://github.com/spack/spack/issues/21237
https://github.com/spack/spack/issues/4261

Co-authored-by: Stephen Sachs <stesachs@amazon.com>
2022-02-15 17:47:29 +00:00
Harmen Stoppels
cebe4fdf1d
Make spack -e [env] spec show environment root specs (#25941) 2022-02-15 09:42:05 -08:00
Harmen Stoppels
55996d3ad4
Unalias despacktivate only when alias exists (#28939) 2022-02-15 16:21:19 +01:00
Seth R. Johnson
08cad7d0ee
darwin: make sure MACOSX_DEPLOYMENT_TARGET has a minor component (#28926) 2022-02-15 05:50:22 +00:00
Danny McClanahan
e8838109d8
move typing_extensions.py back into typing.py =\ (#28549) 2022-02-11 09:52:01 -08:00
Seth R. Johnson
2fa6cd6d23
macOS: always set MACOSX_DEPLOYMENT_TARGET (#28797)
* core: Make platform environment an instance not class method

In preparation for accessing data constructed in __init__.

* macos: set consistent macosx deployment target

This should silence numerous warnings from mixed gcc/macos toolchains.

* perl: prevent too-new deployment target version

```
*** Unexpected MACOSX_DEPLOYMENT_TARGET=11
***
*** Please either set it to a valid macOS version number (e.g., 10.15) or to empty.
```

* Stylin'

* Add deployment target overrides to failing autoconf packages

* Move configure workaround to base autoconf package

This reverts commit 3c119eaf8b4fb37c943d503beacf5ad2aa513d4c.

* Stylin'

* macos: add utility functions for SDK

These aren't yet used but should probably be added to spack debug
report.
2022-02-10 23:22:30 +00:00
Massimiliano Culpo
e6e109cbc5
ASP-based solver: reduce input facts and add heuristic (#28848)
* Remove node_target_satisfies/3 in favor of target_satisfies/2

When emitting input facts we don't need to couple target with
packages, but we can emit fewer facts independently and let
the grounder combine them.

* Remove compiler_version_satisfies/4 in favor of compiler_version_satisfies/3

When emitting input facts we don't need to couple compilers with
packages, but we can emit fewer facts independently and let
the grounder combine them.

* Introduce heuristic in the ASP-program

With heuristic we can drive clingo to make better
initial guesses, which lead to fewer choices and
conflicts in the overall solve
2022-02-10 11:37:10 -08:00
Seth R. Johnson
92b26257f4
Fix CMakePackage.define for libs/headers (#28838)
The 'libs' property returned by a spec is not a list nor tuple.

Closes #28836.
2022-02-10 13:43:22 +00:00
Greg Becker
130354b867
spack audit: fix spurious failures for target/platform conflicts (#28860) 2022-02-10 09:10:23 +01:00
Tamara Dahlgren
36ef59bc67
Tests: move has_test_method to spack.package (#28813) 2022-02-09 22:27:48 -08:00
Massimiliano Culpo
549c785227
Detecting "Cray" as "linux" during bootstrap (#28726)
* bootstrap: avoid detecting "Cray" and treat the platform as "linux"

* bootstrap: create a proper context manager to disable cray
2022-02-09 17:41:11 -05:00
Stephen Sachs
14902a5821
intel-mkl: BLACS with intel-oneapi-mpi (#28476)
Identify the correct BLACS libaries when `intel-oneapi-mpi` is used.
2022-02-07 13:26:20 +01:00
Jordan Galby
37ae4c0fdb
Support config variables in config.yaml extensions paths (#17772) 2022-02-07 11:40:52 +01:00
Harmen Stoppels
73077f3a67
database: fix reindex with uninstalled deps (#28764)
* Fix reindex with uninstalled deps

When a prefix of a dep is removed, and the db is reindexed, it is added
through the dependent, but until now it incorrectly listed the spec as
'installed'.

There was also some questionable behavior in the db when the same spec
was added multiple times, it would always be marked installed.

* Always reserve path

* Only add installed spec's prefixes to install prefixes set

* Improve warning, and ensure ensure only ensures

* test: reindex with every file system remnant removed except for the old index; it should give a database with nothing installed, including records with installed==False,external==False,ref_count==0,explicit=True, and these should be removable from the database
2022-02-04 19:31:39 +00:00
Massimiliano Culpo
5881a03408
Use Spec.constrain to construct spec lists for stacks (#28783)
* stacks: add regression tests for matrix expansion

* Use constrain semantics to construct spec lists for stacks

* Fix semantics for constraining an anonymous spec. Add tests
2022-02-04 19:17:23 +00:00
Massimiliano Culpo
cd04109e17
Add a "sticky" property to variants (#28630)
* Add sticky variants

* Add unit tests for sticky variants

* Add documentation for sticky variants

* Revert "Revert 19736 because conflicts are avoided by clingo by default (#26721)"

This reverts commit 33ef7d57c1.

* Add stickiness to "allow-unsupported-compiler"
2022-02-02 10:05:24 -08:00
Harmen Stoppels
b93b64ca67
TermStatusLine: fix python 2.7 and add test (#28715)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-02-01 21:59:29 +01:00
Massimiliano Culpo
bc06c1206d
macholib, altgraph: update vendored dependency (#28664) 2022-01-28 10:55:12 -08:00
Massimiliano Culpo
3cf5df7e3b
Ensure "spack unit-test" can bootstrap clingo (#28572) 2022-01-26 14:19:15 +01:00
Harmen Stoppels
e3d62b2f7b
Print 'Waiting for another process to install x, y, z' in distributed builds (#28535)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2022-01-26 10:42:08 +01:00
Adam J. Stewart
947c270446
Resource stage: no space before colon (#28560) 2022-01-24 10:01:28 +01:00
Jordan Galby
b526eafa45
Fix spack -C command_line_scope with other flags (#28418)
The option `spack -C` was broken if another flag that touched config was
also set (`spack -C ... -d`, `-c config:...` etc...).
2022-01-23 11:02:13 +01:00
Danny McClanahan
0c2de252f1
introduce llnl.util.compat to remove sys.version_info checks (#21720)
- also split typing.py into typing_extensions and add py2 shims
2022-01-21 12:32:52 -08:00
Bram Veenboer
bd43467cbf
CUDA: add v11.6.0 (#28439) 2022-01-21 11:08:20 +00:00
Danny McClanahan
645b40b249
add six.raise_from() to preserve exception traceback (#28532)
* add six.raise_from() to preserve exception traceback

* add tests for code coverage
2022-01-21 00:24:12 -08:00
Adam J. Stewart
22426f17d1
SIPPackage: fix build system (#28447) 2022-01-19 12:35:46 +01:00
Logan Harbour
9723d918bb
spack compiler info exit 1 when no compilers are found (#28493)
fixes #28492
2022-01-19 11:50:15 +01:00
Harmen Stoppels
e72f87ec64
Switch lmod default all:autoload from none to direct (#28357)
* Switch lmod module all autoload default from none to direct

* Fix the docs
2022-01-18 09:06:41 -08:00
Tamara Dahlgren
f238835b65
is_system_path: return False if path is None (#28403) 2022-01-17 08:44:10 -07:00
Todd Gamblin
93377942d1 Update copyright year to 2022 2022-01-14 22:50:21 -08:00
Todd Gamblin
e3527983ac spack license update-copyright-year now updates the LICENSE-MIT file
`spack license update-copyright-year` was updating license headers but not the MIT
license file. Make it do that and add a test.

Also simplify the way we bump the latest copyright year so that we only need to
update it in one place.
2022-01-14 22:50:21 -08:00
Adam J. Stewart
3540f8200a
PythonPackage: install packages with pip (#27798)
* Use pip to bootstrap pip

* Bootstrap wheel from source

* Update PythonPackage to install using pip

* Update several packages

* Add wheel as base class dep

* Build phase no longer exists

* Add py-poetry package, fix py-flit-core bootstrapping

* Fix isort build

* Clean up many more packages

* Remove unused import

* Fix unit tests

* Don't directly run setup.py

* Typo fix

* Remove unused imports

* Fix issues caught by CI

* Remove custom setup.py file handling

* Use PythonPackage for installing wheels

* Remove custom phases in PythonPackages

* Remove <phase>_args methods

* Remove unused import

* Fix various packages

* Try to test Python packages directly in CI

* Actually run the pipeline

* Fix more packages

* Fix mappings, fix packages

* Fix dep version

* Work around bug in concretizer

* Various concretization fixes

* Fix gitlab yaml, packages

* Fix typo in gitlab yaml

* Skip more packages that fail to concretize

* Fix? jupyter ecosystem concretization issues

* Solve Jupyter concretization issues

* Prevent duplicate entries in PYTHONPATH

* Skip fenics-dolfinx

* Build fewer Python packages

* Fix missing npm dep

* Specify image

* More package fixes

* Add backends for every from-source package

* Fix version arg

* Remove GitLab CI stuff, add py-installer package

* Remove test deps, re-add install_options

* Function declaration syntax fix

* More build fixes

* Update spack create template

* Update PythonPackage documentation

* Fix documentation build

* Fix unit tests

* Remove pip flag added only in newer pip

* flux: add explicit dependency on jsonschema

* Update packages that have been added since this was branched off of develop

* Move Python 2 deprecation to a separate PR

* py-neurolab: add build dep on py-setuptools

* Use wheels for pip/wheel

* Allow use of pre-installed pip for external Python

* pip -> python -m pip

* Use python -m pip for all packages

* Fix py-wrapt

* Add both platlib and purelib to PYTHONPATH

* py-pyyaml: setuptools is needed for all versions

* py-pyyaml: link flags aren't needed

* Appease spack audit packages

* Some build backend is required for all versions, distutils -> setuptools

* Correctly handle different setup.py filename

* Use wheels for py-tomli to avoid circular dep on py-flit-core

* Fix busco installation procedure

* Clarify things in spack create template

* Test other Python build backends

* Undo changes to busco

* Various fixes

* Don't test other backends
2022-01-14 12:37:57 -06:00
Adam J. Stewart
e0f044561e
Python: improve site_packages_dir handling (#28346)
* Python: improve site_packages_dir handling

* Replace all site_packages_dir with purelib/platlib
2022-01-13 20:11:16 -06:00
Harmen Stoppels
d74396ad21
Do not initialize config on spack compiler list (#28042)
When `spack compiler list` is run without being restricted to a
particular scope, and no compilers are found, say that none are 
available, and hint that the use should run spack compiler find to 
auto detect compilers.

* Improve docs
* Check if stdin is a tty
* add a test
2022-01-12 16:26:28 +00:00
Massimiliano Culpo
91fc4cf28f
bootstrap: fix bootstrapping GnuPG from different macOS versions (#28350) 2022-01-12 08:18:16 -08:00
Todd Gamblin
54d741ba54 unparser: handle package-level loops, if statements, and with blocks
Many packages implement logic at the class level to handle complex dependencies and
conflicts. Others have started using `with when("@1.0"):` blocks since we added that
capability. The loops and other control logic can cause some pure directive logic not to
be removed by our package hashing logic -- and in many cases that's a lot of code that
will cause unnecessary rebuilds.

This commit changes the unparser so that it will descend into these blocks. Specifically:

  1. Descend into loops, if statements, and with blocks at the class level.
  2. Don't look inside function definitions (in or outside a class).
  3. Don't look at nested class definitions (they don't have directives)
  4. Add logic to *remove* empty loops/with blocks/if statements if all directives
     in them were removed.

This allows our package hash to ignore a lot of pure metadata that it was not ignoring
before, and makes it less sensitive.

In addition, we add `maintainers` and `tags` to the list of metadata attributes that
Spack should remove from packages when constructing canonoical source for a package
hash.

- [x] Make unparser handle if/for/while/with at class level.
- [x] Add tests for control logic removal.
- [x] Add a test to ensure that all packages are not only unparseable, but also
      that their canonical source is still compilable. This is a test for
      our control logic removal.
- [x] Add another unparse test package that has complex logic.
2022-01-12 06:14:18 -08:00
Todd Gamblin
101f080138 unparser: add unparser unit tests
These are the unit tests from astunparse, converted to pytest, with a few backports from
upstream cpython. These should hopefully keep `unparser.py` well covered as we change it.
2022-01-12 06:14:18 -08:00
Todd Gamblin
4d7226832d unparser: rename t to node to mirror upstream
These refactors have happened in upstream `ast.unparse()`
2022-01-12 06:14:18 -08:00
Todd Gamblin
0370324f1f unparser: rename _Class() methods to visit_Class() to mirror upstream
These are refactors that have happened in upstream `ast.unparse()`
2022-01-12 06:14:18 -08:00
Todd Gamblin
ec16c2d7c2 unparser: do a better job of roundtripping strings
Handle complex f-strings.  Backport of:

    a993e901eb#
2022-01-12 06:14:18 -08:00
Todd Gamblin
e9612696fd unparser: treat print(a, b, c) and print((a, b, c)) the same
We can't tell `print(a, b, c)` and `print((a, b, c))` apart -- both of these expressions
generate different ASTs in Python 2 and Python 3.  However, we can decide that we don't
care.  This commit treats both of them the same when `py_ver_consistent` is set with
`unparse()`.

This means that the package hash won't notice changes from printing a tuple to printing
multiple values, but we don't care, because this is extremely unlikely to affect the build.
More than likely this is just an error message for the user of the package.

- [x] treat `print(a, b, c)` and `print((a, b, c))` the same in py2 and py3
- [x] add another package parsing test -- legion -- that exercises this feature
2022-01-12 06:14:18 -08:00
Todd Gamblin
a18a0e7a47 commands: add spack pkg source and spack pkg hash
To make it easier to see how package hashes change and how they are computed, add two
commands:

* `spack pkg source <spec>`: dumps source code for a package to the terminal

* `spack pkg source --canonical <spec>`: dumps canonicalized source code for a
   package to the terminal. It strips comments, directives, and known-unused
   multimethods from the package. It is used to generate package hashes.

* `spack pkg hash <spec>`: This gives the package hash for a particular spec.
  It is generated from the canonical source code for the spec.

- [x] `add spack pkg source` and `spack pkg hash`
- [x] add tests
- [x] fix bug in multimethod resolution with boolean `@when` values

Co-authored-by: Greg Becker <becker33@llnl.gov>
2022-01-12 06:14:18 -08:00
Todd Gamblin
106ae7abe6 package_hash: switch to using canonical source instead of AST repr
We are planning to switch to using full hashes for Spack specs, which means that the
package hash will be included in the deployment descriptor. This means we need a more
robust package hash than simply dumping the `repr` of the AST.

The AST repr that we previously used for package content is unreliable because it can
vary between python versions (Python's AST actually changes fairly frequently).

- [x] change `package_hash`, `package_ast`, and `canonical_source` to accept a string for
      alternate source instead of a filename.
- [x] consolidate package hash tests in `test/util/package_hash.py`.
- [x] remove old `package_content` method.
- [x] make `package_hash` do what `canonical_source_hash` was doing before.
- [x] modify `content_hash` in `package.py` to use the new `package_hash` function.

Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-01-12 06:14:18 -08:00
Todd Gamblin
39afe946a0 unparser: Don't omit parenthesis when unparsing a slice
Backport of
  c102a14825

Includes support for Python 2.
2022-01-12 06:14:18 -08:00
Todd Gamblin
0776c3b4d6 unparser: Don't put unnecessary parentheses on class declarations
Backport of
* 25160cdc47
2022-01-12 06:14:18 -08:00
Todd Gamblin
ff5e73d6eb package_hash: add test to ensure that every package in Spack can be unparsed
- [x] add option to canonical source to *not* filter multimethods
- [x] add test to unparse every package in builtin
2022-01-12 06:14:18 -08:00
Todd Gamblin
b6dde510bd package_hash: add test to ensure consistency across Python versions
Our package hash is supposed to be consistent from python version to python version.
Test this by adding some known unparse inputs and ensuring that they always have the
same canonical hash.  This test relies on the fact that we run Spack's unit tests
across many python versions.  We can't compute for several python versions within the
same test run so we precompute the hashes and check them in CI.
2022-01-12 06:14:18 -08:00
Todd Gamblin
800229a448 package_hash: fix handling of multimethods and add tests
Package hashing was not properly handling multimethods. In particular, it was removing
any functions that had decorators from the output, so we'd miss things like
`@run_after("install")`, etc.

There were also problems with handling multiple `@when`'s in a single file, and with
handling `@when` functions that *had* to be evaluated dynamically.

- [x] Rework static `@when` resolution for package hash
- [x] Ensure that functions with decorators are not removed from output
- [x] Add tests for many different @when scenarios (multiple @when's,
      combining with other decorators, default/no default, etc.)

Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-01-12 06:14:18 -08:00
Todd Gamblin
93a6c51d88 package_hash: rework RemoveDirectives and add a test
Previously we used `directives.__all__` to get directive names, but it wasn't
quite right -- it included `DirectiveMeta`, etc.  It's not wrong, but it's also
not the clearest way to do this.

- [x] Refactor `@directive` to track names in `directive_names` global
- [x] Rename `_directive_names` to `_directive_dict_names` in `DirectiveMeta`
- [x] Add a test for `RemoveDirectives`

Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-01-12 06:14:18 -08:00
Todd Gamblin
8880a00862 package_hash: remove all unassigned strings, not just docstrings
Some packages use top-level unassigned strings instead of comments, either just after a
docstring on in the body somewhere else. Ignore those strings becasue they have no
effect on package behavior.

- [x] adjust RemoveDocstrings to remove all free-standing strings.
- [x] move tests for util/package_hash.py to test/util/package_hash.py

Co-authored-by: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
2022-01-12 06:14:18 -08:00
Todd Gamblin
572fbf4f49 unparser: handle unicode string literals consistently across Python versions
Python 2 and 3 represent string literals differently in the AST. Python 2 requires '\x'
literals, and Python 3 source is always unicode, and allows unicode to be written
directly.  These also unparse differently by default.

- [x] modify unparser to write both out the way `repr` would in Python 2 when
  `py_ver_consistent` is provided.
2022-01-12 06:14:18 -08:00
Todd Gamblin
396c37d82f unparser: implement operator precedence algorithm for unparser
Backport operator precedence algorithm from here:
    397b96f6d7

This eliminates unnecessary parentheses from our unparsed output and makes Spack's unparser
consistent with the one in upstream Python 3.9+, with one exception.

Our parser normalizes argument order when `py_ver_consistent` is set, so that star arguments
in function calls come last.  We have to do this because Python 2's AST doesn't have information
about their actual order.

If we ever support only Python 3.9 and higher, we can easily switch over to `ast.unparse`, as
the unparsing is consistent except for this detail (modulo future changes to `ast.unparse`)
2022-01-12 06:14:18 -08:00
Todd Gamblin
afb358313a unparser: refactor delimiting with context managers in ast.unparse
Backport of 4b3b1226e8
2022-01-12 06:14:18 -08:00
Todd Gamblin
5847eb1e65 unparser: add block() context manager for indentation
This is a backport of a refactor from cpython 3.9
2022-01-12 06:14:18 -08:00
Todd Gamblin
2badd6500e unparse: Make unparsing consistent for 2.7 and 3.5-3.10
Previously, there were differences in the unparsed code for Python 2.7 and for 3.5-3.10.
This makes unparsed code the same across these Python versions by:

    1. Ensuring there are no spaces between unary operators and
       their operands.
    2. Ensuring that *args and **kwargs are always the last arguments,
       regardless of the python version.
    3. Always unparsing print as a function.
    4. Not putting an extra comma after Python 2 class definitions.

Without these changes, the same source can generate different code for different
Python versions, depending on subtle AST differences.

One place where single source will generate an inconsistent AST is with
multi-argument print statements, e.g.:

```
    print("foo", "bar", "baz")
```

In Python 2, this prints a tuple; in Python 3, it is the print function with
multiple arguments.  Use `from __future__ import print_function` to avoid
this inconsistency.
2022-01-12 06:14:18 -08:00
Todd Gamblin
b324fe5d95 externals: add astunparse
Add `astunparse` as `spack_astunparse`. This library unparses Python ASTs and we're
adding it under our own name so that we can make modifications to it.

Ultimately this will be used to make `package_hash` consistent across Python versions.
2022-01-12 06:14:18 -08:00
Harmen Stoppels
c8e01752a1
Use depends_on over load in lmod module files generated by Spack (#28352) 2022-01-12 13:29:22 +00:00
Peter Scheibel
9f7fb6d01a
stage.steal_source: preserve symlinks
This avoids dangling symlink errors. ignore_dangling_symlinks option would be more-targeted but is only available for Python >= 3.2 (#28318)
2022-01-10 10:10:49 -08:00
Adam J. Stewart
cc32b08205
Python: set default config_vars (#28290)
* Python: set default config_vars

* Add missing commas

* dso_suffix not present for some reason

* Remove use of default_site_packages_dir

* Use config_vars during bootstrapping too

* Catch more errors

* Fix unit tests

* Catch more errors

* Update docstring
2022-01-10 12:00:06 -06:00
Harmen Stoppels
f8d4b4746c
Fix double space in prompt after 'spack env activate -p' (#28279) 2022-01-06 04:59:07 -07:00
Andrew W Elble
a4f0fbafbb
Add function to determine Linux kernel version (#27855)
This reports the kernel version (vs. the distro version) on Linux and
returns a valid Version (stripping characters like '+' which may be
present for custom-built kernels).
2022-01-05 10:22:43 -08:00
Massimiliano Culpo
c2e1a12cdf
Ensure some version known to Spack can satisfy constraints in depends_on (#28131)
Add a new check to `spack audit` to scan and verify that version constraints may be satisfied

Modifications:
 - [x] Add a new check to `spack audit` to scan and verify that version constraints may be satisfied by some version declared in the built-in repository
- [x] Fix issues found by CI

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2021-12-29 12:16:07 -08:00
Martin Diehl
3cd599d6f6
Fixed typos: 'wtih' instead of 'with' (#28166) 2021-12-28 09:45:43 +01:00
Massimiliano Culpo
60a5b2a85e Merge tag 'v0.17.1' into develop 2021-12-23 19:48:36 +01:00
Massimiliano Culpo
4381cb5957
New subcommand: spack bootstrap status (#28004)
This command pokes the environment, Python interpreter
and bootstrap store to check if dependencies needed by
Spack are available.

If any are missing, it shows a comprehensible message.
2021-12-23 10:34:04 -08:00
Massimiliano Culpo
69b8cddb1b Bump version and update CHANGELOG.md 2021-12-23 16:02:09 +01:00
Tom Scogland
8e659f512e locks: allow locks to work under high contention (#27846)
* locks: allow locks to work under high contention

This is a bug found by Harshitha Menon.  

The `lock=None` line shouldn't be a release but should be 
```
return (lock_type, None)
``` 
to inform the caller it couldn't get the lock type requested without
disturbing the existing lock object in the database.  There were also a
couple of bugs due to taking write locks at the beginning without any
checking or release, and not releasing read locks before requeueing.
This version no longer gives me read upgrade to write errors, even
running 200 instances on one box.

* Change lock in check_deps_status to read, release if not installed,
  not sure why this was ever write, but read definitely is more
  appropriate here, and the read lock is only held out of the scope if
  the package is installed.

* Release read lock before requeueing to reduce chance of livelock, the
  timeout that caused the original issue now happens in roughly 3 of 200
  workers instead of 199 on average.
2021-12-23 16:02:09 +01:00
Harmen Stoppels
5daf023aec Regenerate views when specs already installed (#28113)
With this commit:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
==> Updating view at /tmp/spack-faiirgmt/.spack-env/view
$ spack install zlib
==> All of the packages are already installed
```

Before this PR:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
$ spack install zlib
==> All of the packages are already installed
```

No view was generated
2021-12-23 16:02:09 +01:00
Paul Spencer
e1cc28a30a sbang: respect package permissive package permissions for sbang (#25764)
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2021-12-23 16:02:09 +01:00
victorusu
17edf1ae90 Add setdefault option to tcl module (#14686)
This commit introduces the command

spack module tcl setdefault <package>

similar to the one already available for lmod

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-12-23 16:02:09 +01:00
Harmen Stoppels
79fd1c5114 Set backup=False by default in filter_file (#28036) 2021-12-23 16:02:09 +01:00
Harmen Stoppels
13e36c5457 Fix table formatting (#28037) 2021-12-23 16:02:09 +01:00
Harmen Stoppels
b2694013d4 Revert "patches: make re-applied patches idempotent (#26784)" (#27625)
This reverts commit c5ca0db27f.
2021-12-23 16:02:09 +01:00
Andrew W Elble
8f3b025b55 MANPATH needs a trailing ':' to utilize system defaults (#21682)
otherwise spack breaks using system man pages by default.

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-12-23 16:02:09 +01:00
Christian Goll
37fbe30c4a Added opensuse/leap:15 to spack containerize (#27837)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-12-23 16:02:09 +01:00
Tamara Dahlgren
314867e635 Provide meaningful message for empty environment installs (#28031)
* Provide a meaningful failure message for installation of an empty environment

* Allow regenerating view per offline discussion
2021-12-23 16:02:09 +01:00
Greg Becker
9345bf81b9 Add option to minimize full debug cores. include warning message about performance (#27970)
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-12-23 16:02:09 +01:00
Harmen Stoppels
20ad47f9e1 Install dir creation message demoted to "debug" level (#27911) 2021-12-23 16:02:09 +01:00
Greg Becker
7e5de95a30 Improve debug info from concretizer (#27707) 2021-12-23 16:02:09 +01:00
Massimiliano Culpo
e9f7fb03c9 spack audit: fix API calls (#27713)
This broke in #24858
2021-12-23 16:02:09 +01:00
Paul Ferrell
9d4291e590 Handle byte sequences which are not encoded as UTF8 while logging. (#21447)
Fix builds which produce a lines with non-UTF8 output while logging
The alternative is to read in binary mode, and then decode while
ignoring errors.
2021-12-23 16:02:09 +01:00
Harmen Stoppels
654f6839eb ci: run style unit tests only if we target develop (#27472)
Some tests assume the base branch is develop, but this branch may not
have been checked out.
2021-12-23 16:02:09 +01:00
Harmen Stoppels
c8daa7218d Turn some verbose messages into debug messages (#27408) 2021-12-23 16:02:09 +01:00
Harmen Stoppels
d862507bcf Fix overloaded argparse keys (#27379)
Commands should not reuse option names defined in main.
2021-12-23 16:02:09 +01:00
Jordan Galby
7c6b253d89 Fix log-format reporter ignoring install errors (#25961)
When running `spack install --log-format junit|cdash ...`, install
errors were ignored. This made spack continue building dependents of
failed install, ignoring `--fail-fast`, and exit 0 at the end.
2021-12-23 16:02:09 +01:00
Dylan Simon
544826c825 make --enable-locks actually enable locks (#24675) 2021-12-23 16:02:09 +01:00
Tom Scogland
b7b6542804
locks: allow locks to work under high contention (#27846)
* locks: allow locks to work under high contention

This is a bug found by Harshitha Menon.  

The `lock=None` line shouldn't be a release but should be 
```
return (lock_type, None)
``` 
to inform the caller it couldn't get the lock type requested without
disturbing the existing lock object in the database.  There were also a
couple of bugs due to taking write locks at the beginning without any
checking or release, and not releasing read locks before requeueing.
This version no longer gives me read upgrade to write errors, even
running 200 instances on one box.

* Change lock in check_deps_status to read, release if not installed,
  not sure why this was ever write, but read definitely is more
  appropriate here, and the read lock is only held out of the scope if
  the package is installed.

* Release read lock before requeueing to reduce chance of livelock, the
  timeout that caused the original issue now happens in roughly 3 of 200
  workers instead of 199 on average.
2021-12-22 16:25:05 +01:00
Joseph Snyder
34873f5fe7
Use consistent method of checking for presence of info in connection (#27694)
Fixes #27652

Ensure that mirror's to_dict function returns a syaml_dict object for all code
paths.

Switch to using the .get function for accessing the potential information from
the S3 mirror objects.  If the key is not there, it will gracefully return
None instead of failing with a KeyError

Additionally, check that the connection object is a dictionary before trying
to "get" from it.

Add a test for the capturing of the new S3 information.
2021-12-22 16:15:49 +01:00
Harmen Stoppels
db69a291d4
Regenerate views when specs already installed (#28113)
With this commit:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
==> Updating view at /tmp/spack-faiirgmt/.spack-env/view
$ spack install zlib
==> All of the packages are already installed
```

Before this PR:
```
$ spack env activate --temp
$ spack install zlib
==> All of the packages are already installed
$ spack install zlib
==> All of the packages are already installed
```

No view was generated
2021-12-21 18:41:12 +01:00
Robert Cohn
d45280369f
Intel oneAPI packages: add 2022.1.1 release (#28096)
Co-authored-by: Egorov, Andrey <andrey.egorov@intel.com>
2021-12-20 14:06:32 -08:00
Vanessasaurus
a94b4eef79
Fixing spacing of libabigail to : (#28085) 2021-12-20 03:56:03 -07:00
Vanessasaurus
da9e152ed1
Fix bugs in spack monitor (#27511)
Updates to installer.py did not account for spack monitor, so as currently implemented
there are three cases of failure that spack monitor will not account for. To fix this we add additional
hooks, including an on cancel and also do a custom action on concretization fail.

Signed-off-by: vsoch <vsoch@users.noreply.github.com>

Co-authored-by: vsoch <vsoch@users.noreply.github.com>
2021-12-20 06:54:41 +01:00
Todd Gamblin
bfc69f0e4b jsonschema: use more specific schema identifiers
The latest version of `jsonschema` fails if we're not specific about which schema draft
specification we're using.  Update all of them to use the latest one (draft-07).
2021-12-19 12:55:42 -08:00
Todd Gamblin
7703043195 externals: Upgrade jsonschema to v3.2.0
Our `jsonschema` external won't support Python 3.10, so we need to upgrade it.
It currently generates this warning:

    lib/spack/external/jsonschema/compat.py:6: DeprecationWarning: Using or importing the ABCs
        from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and
        in 3.10 it will stop working

This upgrades `jsonschema` to 3.2.0, the latest version with support for Python 2.7.  The next
version after this (4.0.0) drops support for 2.7 and 3.6, so we'll have to wait to upgrade to it.

Dependencies have been added in prior commits.
2021-12-19 12:55:42 -08:00
Todd Gamblin
f83e0fb81a externals: add attrs for new jsonschema
Updating `jsonschema` to 3.2.0 requires `attrs`. Add it to externals.
2021-12-19 12:55:42 -08:00
Todd Gamblin
90592b3cbe externals: add pyrsistent for new jsonschema
Updating `jsonschema` to 3.2.0 requires `pyrsistent`. Adding just the pieces of it
that are needed for `jsonschema`.
2021-12-19 12:55:42 -08:00
Todd Gamblin
04536db387 externals: add functools32 for new jsonschema
Updating `jsonschema` to 3.2.0 requires `functools32`, just for Python 2.
2021-12-19 12:55:42 -08:00
Tamara Dahlgren
e470131a77
Tests: remove test_get_stage_root_in_spack since it writes to the spack directory (#28056) 2021-12-19 11:31:38 -08:00
Paul Spencer
30244c2c40
sbang: respect package permissive package permissions for sbang (#25764)
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2021-12-18 22:07:20 -08:00
Vanessasaurus
d9c4b91af3
Remove ability to run spack monitor without auth (#27888)
spack monitor now requires authentication as each build must be associated
with a user, so it does not make sense to allow the --monitor-no-auth flag
and this commit will remove it
2021-12-17 18:00:43 +01:00
victorusu
18615b1485
Add setdefault option to tcl module (#14686)
This commit introduces the command

spack module tcl setdefault <package>

similar to the one already available for lmod

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-12-17 10:05:32 +01:00
Harmen Stoppels
fb93979b94
Set backup=False by default in filter_file (#28036) 2021-12-16 13:50:20 +01:00
Harmen Stoppels
6357de4e61
Fix table formatting (#28037) 2021-12-16 12:13:12 +00:00
Andrew W Elble
96535cc4f9
MANPATH needs a trailing ':' to utilize system defaults (#21682)
otherwise spack breaks using system man pages by default.

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-12-16 10:54:35 +00:00
Christian Goll
bd0ffa8a3c
Added opensuse/leap:15 to spack containerize (#27837)
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
2021-12-16 09:47:15 +00:00
Tamara Dahlgren
ac49ce8b3b
Provide meaningful message for empty environment installs (#28031)
* Provide a meaningful failure message for installation of an empty environment

* Allow regenerating view per offline discussion
2021-12-15 22:56:54 -08:00
Tamara Dahlgren
9240614928
Bugfix: Simplify preferred-test versions; set checksum defaults (#28026)
* Simplify preferred-test versions; set checksum defaults

* Fix test_preferred failure
2021-12-16 06:55:28 +01:00
Harmen Stoppels
72ca7d6ee5
Revert "patches: make re-applied patches idempotent (#26784)" (#27625)
This reverts commit c5ca0db27f.
2021-12-15 08:56:03 -08:00
Greg Becker
5319b6e3b1
Add option to minimize full debug cores. include warning message about performance (#27970)
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2021-12-15 08:52:53 +01:00
Zack Galbreath
3139894794
ci: Catch Exceptions when attempting to register a CDash build (#27883)
This PR also slightly changes the behavior in ci_rebuild().
We now still attempt to submit `spack install` results to CDash
even if the initial registration failed due to connection issues.

This commit follows in the spirit of #24299. We do not want `spack install`
to exit with a non-zero status when something goes wrong while attempting to
report results to CDash.
2021-12-14 14:18:08 -07:00
Harmen Stoppels
f6802b733a
Install dir creation message demoted to "debug" level (#27911) 2021-12-13 11:45:31 +01:00
Adam J. Stewart
0960c0810c
Extends: support spec, not just package name (#27754) 2021-12-10 13:30:21 +01:00
Greg Becker
dc87157e80
Improve debug info from concretizer (#27707) 2021-12-10 10:49:33 +01:00
Massimiliano Culpo
d17511a806
Refactor "spack buildcache" command (#27776)
This PR is meant to move code with "business logic" from `spack.cmd.buildcache` to appropriate core modules[^1]. 

Modifications:
- [x] Add `spack.binary_distribution.push` to create a binary package from a spec and push it to a mirror
- [x] Add `spack.binary_distribution.install_root_node` to install only the root node of a concrete spec from a buildcache (may check the sha256 sum if it is passed in as input)
- [x] Add `spack.binary_distribution.install_single_spec` to install a single concrete spec from a buildcache
- [x] Add `spack.binary_distribution.download_single_spec` to download a single concrete spec from a buildcache to a local destination
- [x] Add `Spec.from_specfile` that construct a spec given the path of a JSON or YAML spec file
- [x] Removed logic from `spack.cmd.buildcache`
- [x] Removed calls to `spack.cmd.buildcache` in `spack.bootstrap`
- [x] Deprecate `spack buildcache copy` with a message that says it will be removed in v0.19.0

[^1]: The rationale is that commands should be lightweight wrappers of the core API, since that helps with both testing and scripting (easier mocking and no need to invoke `SpackCommand`s in a script).
2021-12-10 10:23:14 +01:00
Massimiliano Culpo
01d077d4bc
Make external detection more resilient to individual package errors (#27854)
After this PR an error in a single package while detecting
external software won't abort the entire procedure.

The error is reported to screen as a warning.
2021-12-08 13:58:21 -08:00
Tamara Dahlgren
d458e82286
rocm: add gfx90a (#27823) 2021-12-07 17:19:48 -08:00
Harmen Stoppels
3d1b9e4dbc
"spack buildcache install": don't catch exception (#27674)
Remove a try/catch for an error with no handling. If the affected
code doesn't execute successfully, then the associated variable
is undefined and another (more-obscure) error occurs shortly after.
2021-12-07 13:17:17 -08:00
Massimiliano Culpo
f81d84dfc6
Release procedure: add a step to update docs (#27734) 2021-12-07 11:30:14 +00:00
Peter Scheibel
edb971a10e
Support packages which need to explicitly refer to dpcpp by name (#27168)
* Hack to support packages which need to explicitly refer to dpcpp by name
* cc script needs to know about dpcpp
2021-12-02 15:49:20 -08:00
Massimiliano Culpo
645a7dc14c
spack audit: fix API calls (#27713)
This broke in #24858
2021-11-30 14:59:55 +01:00