Commit graph

5288 commits

Author SHA1 Message Date
Tom Scogland
94794d061a
optimize instantiation and comparison of versions (#29429)
Re-work the checks and comparisons around commit versions, when no
commit version is involved the overhead is now in the noise, where one
is the overhead is now constant rather than linear.
2022-03-15 18:53:28 +00:00
Sergey Kosukhin
f1bdbefd46
mpich: fix the detection of the package (#29284) 2022-03-14 16:33:56 +01:00
Harmen Stoppels
474493713b
compiler.py: early return in compiler_environment when no modules (#29441) 2022-03-14 11:27:49 +00:00
Massimiliano Culpo
a6eed4a7c7
Remove "setup_environment" and "setup_dependent_environment" (#29463)
fixes #29446

The new setup_*_environment functions have been falling back
to calling the old functions and warn the user since #11115.

This commit removes the fallback behavior and any use of:
- setup_environment
- setup_dependent_environment
in the codebase
2022-03-13 11:51:55 -04:00
Harmen Stoppels
8adc6b7e8e
module_cmd.py: use posix awk; fix stderr redirection (#29440)
Emulates `env -0` in a posix compliant way, avoiding a slow python process, speeds up setting up the build env when modules should load.
2022-03-11 18:29:11 +01:00
Harmen Stoppels
88fbba3e1e
intel-mkl: load compiler modules when querying compiler (#29439) 2022-03-10 14:14:17 -08:00
Massimiliano Culpo
2cd5c00923
Allow for multiple dependencies/dependents from the same package (#28673)
Change the internal representation of `Spec` to allow for multiple dependencies or
dependents stemming from the same package. This change permits to represent cases
which are frequent in cross compiled environments or to bootstrap compilers.

Modifications:
- [x] Substitute `DependencyMap` with `_EdgeMap`. The main differences are that the
      latter does not support direct item assignment and can be modified only through its
      API. It also provides a `select_by` method to query items.
- [x] Reworked a few public APIs of `Spec` to get list of dependencies or related edges.
- [x] Added unit tests to prevent regression on #11983 and prove the synthetic construction
      of specs with multiple deps from the same package. 

Since #22845 went in first, this PR reuses that format and thus it should not change hashes.
The same package may be present multiple times in the list of dependencies with different
associated specs (each with its own hash).
2022-03-10 11:53:45 -08:00
Harmen Stoppels
dc78f4c58a
environment.py: allow link:run (#29336)
* environment.py: allow link:run

Some users want minimal views, excluding run-type dependencies, since
those type of dependencies are covered by rpaths and the symlinked
libraries in the view aren't used anyways.

With this change, an environment like this:

```
spack:
  specs: ['py-flake8']
  view:
    default:
      root: view
      link: run
```

includes python packages and python, but no link type deps of python.
2022-03-09 12:35:26 -08:00
百地 希留耶
3270aa106b
Hide package repository count when redirecting stdout (#29402) 2022-03-09 17:07:07 +00:00
百地 希留耶
3370d3f57e
Fix tab completion erroring with spack unit-test (#29405) 2022-03-09 16:09:57 +01:00
Harmen Stoppels
3df90e3e33
ci.py: fix utf-8 issue with git log output (#29419) 2022-03-09 13:13:37 +00:00
Harmen Stoppels
a39a6c4ea7
version.py: set is_commit in constructor (#29369)
Speeds up comparison on `Version` by ~2.5x, e.g.

```python
In [1]: v = spack.version.Version('1.0.0'); w = spack.version.Version('1.0.2')

In [2]: %timeit v < w

1.47 µs ± 5.59 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

535 ns ± 1.75 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```
2022-03-08 10:50:48 +01:00
Todd Gamblin
d424d0e54e
find: deprecate spack find -bootstrap for spack -bootstrap find (#29152)
Since adding the `spack --bootstrap` argument we don't need `spack find --bootstrap`
anymore.  Deprecate it.
2022-03-05 21:41:18 -08:00
Harmen Stoppels
80a6eab2db Add a test 2022-03-04 18:13:34 +01:00
Harmen Stoppels
9199800bbd VersionRange.satisfies should test non-empty intersection 2022-03-04 18:13:34 +01:00
Massimiliano Culpo
b48bdc9e19
Fix importing Spack packages as Python modules (#29221)
fixes #29203

This PR fixes a subtle bug we have when importing
Spack packages as Python modules that can lead to
multiple module objects being created for the same
package.

It also fixes all the places in unit-tests where
"relying" on the old bug was crucial to have a new
"clean" state of the package class.
2022-03-04 08:42:27 +01:00
Doug Jacobsen
06fd0f892e
Revert GCS fetch strategy, to remove s3 interface (#28736)
This commit reverts the GCS fetch strategy to before commit:
d759612523

The previous commit added some s3 syntax to handle connections, but
added them into the GCS fetch strategy in a way that prevents GCS from
working anymore.
2022-03-03 16:34:33 -08:00
Greg Becker
e2b87ade06
rocmcc compiler: initial commit based on aocc and clang (#28575)
* rocmcc compiler: initial commit based on aocc and clang

Co-authored-by: luker <luke.roskop@hpe.com>
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
2022-03-03 14:34:22 -07:00
Michael Kuhn
49069e4f58
installer: Fix cosmetic problem with terminal title (#29070)
The status displayed in the terminal title could be wrong when doing
distributed builds. For instance, doing `spack install glib` in two
different terminals could lead to the current package being reported as
`40/29` due to the way Spack handles retrying locks.

Work around this by keeping track of the package IDs that were already
encountered to avoid counting packages twice.
2022-03-03 14:21:15 +01:00
Scott Wittenburg
c72735229f
test/installer.py: remove commented code and inaccurate docstring (#29305) 2022-03-03 10:19:57 +01:00
Danny McClanahan
2c331a1d7f
make @llnl.util.lang.memoized support kwargs (#21722)
* make memoized() support kwargs

* add testing for @memoized
2022-03-02 11:12:15 -08:00
Massimiliano Culpo
8d118104c7
Fix typos when forwarding arguments to traverse_edges (#29261)
A few calls use `deptypes=...` instead of `deptype=...`
2022-03-02 08:43:26 +01:00
Tamara Dahlgren
b20df12d09
test_env_install_two_specs_same_dep: properly check installed specs (#29222) 2022-03-01 10:35:14 +01:00
Scott Wittenburg
b082c33c85
commands: Propgate command return value as exit code (#29223) 2022-02-25 10:49:56 -08:00
Tamara Dahlgren
0b4f40ab79
Testing: Summarize test results and add verbose output (#28700) 2022-02-23 18:36:21 -08:00
Massimiliano Culpo
1ddad522a4
Move early exit for setup only argument (#29041)
See https://github.com/spack/spack/pull/28468/files#r809156986

If we exit before generating the:

 error("Dependencies must have compatible OS's with their dependents").
 ...

facts we'll output a problem that is effectively
different by the one solved by clingo.
2022-02-23 01:46:52 -08:00
Tom Scogland
a9ba40164a
Checksum match (#28989)
* cmd/checksum: prefer url matching url_from_version

This is a minimal change toward getting the right archive from places
like github.  The heuristic is:

* if an archive url exists, take its version
* generate a url from the package with pkg.url_from_version
* if they match
  * stop considering other URLs for this version
  * otherwise, continue replacing the url for the version

I doubt this will always work, but it should address a variety of
versions of this bug.  A good test right now is `spack checksum gh`,
which checksums macos binaries without this, and the correct source
packages with it.

fixes #15985
related to #14129
related to #13940

* add heuristics to help create as well

Since create can't rely on an existing package, this commit adds another
pair of heuristics:
1. if the current version is a specifically listed archive, don't
   replace it
2. if the current url matches the result of applying
   `spack.url.substitute_version(a, ver)` for any a in archive_urls,
   prefer it and don't replace it

fixes #13940

* clean up style and a lingering debug import

* ok flake8, you got me

* document reference_package argument

* Update lib/spack/spack/util/web.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* try to appease sphinx

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2022-02-23 00:55:59 +00:00
Todd Gamblin
36b0730fac
Add spack --bootstrap option for accessing bootstrap store (#25601)
We can see what is in the bootstrap store with `spack find -b`, and you can clean it with `spack
clean -b`, but we can't do much else with it, and if there are bootstrap issues they can be hard to
debug.

We already have `spack --mock`, which allows you to swap in the mock packages from the command
line. This PR introduces `spack -b` / `spack --bootstrap`, which runs all of spack with
`ensure_bootstrap_configuration()` set. This means that you can run `spack -b find`, `spack -b
install`, `spack -b spec`, etc. to see what *would* happen with bootstrap configuration, to remove
specific bootstrap packages, etc. This will hopefully make developers' lives easier as they deal
with bootstrap packages.

This PR also uses a `nullcontext` context manager. `nullcontext` has been implemented in several
other places in Spack, and this PR consolidates them to `llnl.util.lang`, with a note that we can
delete the function if we ever reqyire a new enough Python.

- [x] introduce `spack --bootstrap` option
- [x] consolidated all `nullcontext` usages to `llnl.util.lang`
2022-02-22 12:35:34 -07:00
Todd Gamblin
7912a8e90b
bugfix: Not all concrete versions on the CLI should be considered real (#28620)
Some "concrete" versions on the command line, e.g. `qt@5` are really
meant to satisfy some actual concrete version from a package. We should
only assume the user is introducing a new, unknown version on the CLI
if we, well, don't know of any version that satisfies the user's
request.  So, if we know about `5.11.1` and `5.11.3` and they ask for
`5.11.2`, we'd ask the solver to consider `5.11.2` as a solution.  If
they just ask for `5`, though, `5.11.1` or `5.11.3` are fine solutions,
as they satisfy `@5`, so use them.

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-02-21 11:46:37 -08:00
Alberto Invernizzi
b8d042273a
Bring back cuda@11.4.0 conflicts for GCC and clang; add 11.4.3:11.4.4 (#29076)
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-02-21 20:28:56 +01:00
Massimiliano Culpo
7fd94fc4bc
spack external find: change default behavior (#29031)
See https://github.com/spack/spack/issues/25353#issuecomment-1041868116

This commit changes the default behavior of
```
$ spack external find
```
from searching all the possible packages Spack knows about to
search only for the ones tagged as being a "build-tool".

It also introduces a `--all` option to restore the old behavior.
2022-02-18 11:51:01 -07:00
Seth R. Johnson
6c61c2695a
darwin: robust macos version detection (#28991)
Prefer `sw_vers` to `platform.mac_ver`. In anaconda3 installation, for example, the latter reports 10.16 on Monterey -- I think this is affected by how and where the python instance was built.

Use MACOSX_DEPLOYMENT_TARGET if present to override the operating system choice.
2022-02-17 20:50:41 -07:00
Scott Wittenburg
38643dcd7e
gitlab: Propagate stack name to downstream build jobs (#29019)
It will be useful for metrics gathering and possibly debugging to
have this environment variable available in the runner pods that
do the actual rebuilds.
2022-02-17 15:36:48 -05:00
Tamara Dahlgren
fefe65a35b
Testing: optionally run tests on externally installed packages (#28701)
Since Spack does not install external packages, this commit skips them by
default when running stand-alone tests. The assumption is that such packages
have likely undergone an acceptance test process. 

However, the tests can be run against installed externals using 
```
% spack test run --externals ...
```
2022-02-17 19:47:42 +01:00
Harmen Stoppels
d93f9b82ac
Reduce verbosity of patches=... variant (#29015)
* reduce verbosity of patches=... variant

* Special-case prefix-matches for satisfies of patches variant
2022-02-17 11:06:32 +01:00
Massimiliano Culpo
fa132614e0
ASP-based solver: don't sort when defining variant possible values (#29013)
fixes #28260

Since we iterate over different variants from many packages, the variant 
values may have types which are not comparable, which causes errors 
at runtime. This is not a real issue though, since we don't need the facts
to be ordered. Thus, to avoid needless sorting, the sorted function has 
been removed and a comment has been added to tip any developer that
might need to inspect these clauses for debugging to add back sorting 
on the first two items only.

It's kind of difficult to add a test for this, since the error depends on 
whether Python sorting algorithm ever needs to compare the third 
value of a tuple being ordered.
2022-02-17 08:50:50 +01:00
Tom Scogland
8f5fcc6e95
extensions: allow multiple "extends" directives (#28853)
* extensions: allow multiple "extends" directives

This will allow multiple extends directives in a package as long as only one of
them is selected as a dependency in the concrete spec.

* document the option to have multiple extends
2022-02-16 21:23:12 +00:00
Todd Gamblin
b1ff9c05bc concretizer: refactor argument passing for reuse
Reuse previously was a very invasive change that required parameters to be added to all
the methods that called `concretize()` on a `Spec` object. With the addition of
concretizer configuration, we can use the config system to simplify this argument
passing and keep the code cleaner.

We decided that concretizer config options should be read at `Solver` instantiation
time, and if config changes between instnatiation of a particular solver and
`solve()` invocation, the `Solver` should use the settings from `__init__()`.

- [x] remove `reuse` keyword argument from most concretize functions
- [x] refactor usages to use `spack.config.override("concretizer:reuse", True)`
- [x] rework argument passing in `Solver` so that parameters are set from config
      at instantiation time
2022-02-16 10:17:18 -08:00
Todd Gamblin
d33973df6c docs: add section on concretizer configuration
* Document `concretizer.yaml`, `--reuse`, and `--fresh`.
2022-02-16 10:17:18 -08:00
Todd Gamblin
a2b8e0c3e9 commands: refactor --reuse handling to use config
`--reuse` was previously handled individually by each command that
needed it. We are growing more concretization options, and they'll
need their own section for commands that support them.

Now there are two concretization options:

* `--reuse`: Attempt to reuse packages from installs and buildcaches.
* `--fresh`: Opposite of reuse -- traditional spack install.

To handle thes, this PR adds a `ConfigSetAction` for `argparse`, so
that you can write argparse code like this:

```
     subgroup.add_argument(
        '--reuse', action=ConfigSetAction, dest="concretizer:reuse",
        const=True, default=None,
        help='reuse installed dependencies/buildcaches when possible'
     )
```

With this, you don't need to add logic to pull the argument out and
handle it; the `ConfigSetAction` just does it for you. This can probably
be used to clean up some other commands later, as well.

Code that was previously passing `reuse=True` around everywhere has
been refactored to use config, and config is set from the CLI using
a new `add_concretizer_args()` function in `spack.cmd.common.arguments`.

- [x] Add `ConfigSetAction` to simplify concretizer config on the CLI
- [x] Refactor code so that it does not pass `reuse=True` to every function.
- [x] Refactor commands to use `add_concretizer_args()` and to pass
      concretizer config using the config system.
2022-02-16 10:17:18 -08:00
Todd Gamblin
f155de7462 tests: consolidate mock scope creation logic in conftest.py
Config scopes were different for `config` and `mutable_config`,
and `mutable_config` did not have a command line scope.

- [x] Fix by consolidating the creation logic for the two fixtures.
2022-02-16 10:17:18 -08:00
Todd Gamblin
800ed16e7a config: add a new concretizer config section
The concretizer is going to grow to have many more configuration,
and we really need some structured config for that.

* We have the `config:concretizer` option that chooses the solver,
  but extending that is awkward (we'd need to replace a string with
  a `dict`) and the solver choice will be deprecated eventually.

* We have the `concretization` option in environments, but it's
  not a top-level config section -- it's just for environments,
  and it also only admits a string right now.

To avoid overlapping with either of these and to allow the most
extensibility in the future, this adds a new `concretizer` config
section that can be used in and outside of environments. There
is only one option right now: `reuse`.  This can expand to include
other options later.

Likely, we will soon deprecate `config:concretizer` and warn when
the user doesn't use `clingo`, and we will eventually (sometime later)
move the `together` / `separately` options from `concretization` into
the top-level `concretizer` section.

This commit just adds the new section and schema. Fully wiring it
up is TBD.
2022-02-16 10:17:18 -08:00
Todd Gamblin
1903e45eec refactor: convert spack.solver.asp.solve() to a class
The solver has a lot of configuration associated with it. Rather
than adding arguments to everything, we should encapsulate that
in a class. This is the start of that work; it replaces `solve()`
and its kwargs with a class and properties.
2022-02-16 10:17:18 -08:00
Mark W. Krentel
87a3b72ef0
Add 'stable' to the list of infinity version names. (#28772)
* Add 'stable' to the list of infinity version names.
Rename libunwind 1.5-head to 1.5-stable.

* Add stable to the infinite version list in packaging_guide.rst.
2022-02-16 09:08:51 -08:00
Adam J. Stewart
3c1b2c0fc9
find_libraries: search for both .so and .dylib on macOS (#28924) 2022-02-16 14:07:44 +01:00
Stephen Sachs
79f22423b8
intel compiler: fix link time error with LLVMgold.so (#28731)
The Intel compiler will, at link time, call `ld -plugin LLVMgold.so`, which
expects libraries like `libimfo.so` to be found either in the `LD_LIBRARY_PATH` or
in `LLVMgold.so`s RPATH.

As `LLVMgold.so` already uses RUNPATH, I used that to extend this to the
necessary library locations.

This PR should fix issues:
https://github.com/spack/spack/issues/10308
https://github.com/spack/spack/issues/18606
https://github.com/spack/spack/issues/17100
https://github.com/spack/spack/issues/21237
https://github.com/spack/spack/issues/4261

Co-authored-by: Stephen Sachs <stesachs@amazon.com>
2022-02-15 17:47:29 +00:00
Harmen Stoppels
cebe4fdf1d
Make spack -e [env] spec show environment root specs (#25941) 2022-02-15 09:42:05 -08:00
Harmen Stoppels
55996d3ad4
Unalias despacktivate only when alias exists (#28939) 2022-02-15 16:21:19 +01:00
Seth R. Johnson
08cad7d0ee
darwin: make sure MACOSX_DEPLOYMENT_TARGET has a minor component (#28926) 2022-02-15 05:50:22 +00:00
Danny McClanahan
e8838109d8
move typing_extensions.py back into typing.py =\ (#28549) 2022-02-11 09:52:01 -08:00