Commit graph

6632 commits

Author SHA1 Message Date
John W. Parent
7d072cc16f
Windows: detect all available SDK versions (#39823)
Currently, Windows SDK detection will only pick up SDK versions
related to the current version of Windows Spack is running on.
However, in some circumstances, we want to detect other version
of the SDK, for example, for compiling on Windows 11 for Windows
10 to ensure an API is compatible with Win10.
2023-09-29 20:17:10 +00:00
Peter Scheibel
3969653f1b
Cray manifest: compiler handling fixes (#40061)
* Make use of `prefix` in the Cray manifest schema (prepend it to
  the relative CC etc.) - this was a Spack error.
* Warn people when wrong-looking compilers are found in the manifest
  (i.e. non-existent CC path).
* Bypass compilers that we fail to add (don't allow a single bad
  compiler to terminate the entire read-cray-manifest action).
* Refactor Cray manifest tests: module-level variables have been
  replaced with fixtures, specifically using the `test_platform`
  fixture, which allows the unit tests to run with the new
  concretizer.
* Add unit test to check case where adding a compiler raises an
  exception (check that this doesn't prevent processing the
  rest of the manifest).
2023-09-29 09:47:30 -07:00
Jordan Galby
db37672abf
Print error when missing git (#40254)
Like a missing curl.
2023-09-29 15:14:39 +00:00
Harmen Stoppels
605835fe42
Don't drop build deps on overwrite install (#40252)
If you `spack install x ^y` where `y` is a pure build dep of `x`, and
then uninstall `y`, and then `spack install --overwrite x ^y`, the build
fails because `y` is not re-installed.

Same can happen when you install a develop spec, run `spack gc`,
modify sources, and install again; develop specs rely on overwrite
install to work correctly.
2023-09-29 16:40:21 +02:00
Massimiliano Culpo
210d221357
Test package detection in a systematic way (#18175)
This PR adds a new audit sub-command to check that detection of relevant packages
is performed correctly in a few scenarios mocking real use-cases. The data for each 
package being tested is in a YAML file called detection_test.yaml alongside the 
corresponding package.py file.

This is to allow encoding detection tests for compilers and other widely used tools, 
in preparation for compilers as dependencies.
2023-09-29 10:24:42 +02:00
Massimiliano Culpo
a236fce31f
Partial removal of circular dependencies between spack and llnl (#40090)
Modifications:
- [x] Move `spack.util.string` to `llnl.string`
- [x] Remove dependency of `llnl` on `spack.error`
- [x] Move path of `spack.util.path` to `llnl.path`
- [x] Move `spack.util.environment.get_host_*` to `spack.spec`
2023-09-28 16:21:52 +00:00
Harmen Stoppels
6d55066b94
Use st_nlink in hardlink tracking (#39328)
Only add potential hardlinks to a set/dict, instead of each file. This
should be much cheaper, since hardlinks are very rare.
2023-09-28 13:24:56 +00:00
Brian Vanderwende
fba47e87d7
Support optional vars script arguments (#40064) 2023-09-28 06:52:22 -04:00
Harmen Stoppels
5c25437c9f
spack --profile: dump to stderr (#40209) 2023-09-26 16:18:47 +02:00
dependabot[bot]
37c48fc82c
build(deps): bump urllib3 from 2.0.4 to 2.0.5 in /lib/spack/docs (#40119)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.4 to 2.0.5.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.0.4...v2.0.5)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-24 17:35:48 +02:00
Harmen Stoppels
716196930a
Remove distutils dependency in Spack (#40153)
* msvc.py: don't import distutils

Introduced in #27021, makes Spack forward incompatible with Python.

The module was already deprecated at the time of the PR.

* update spack package
2023-09-22 13:04:58 +02:00
Adam J. Stewart
419878f035
Add support for macOS Sonoma (#40115) 2023-09-22 04:21:26 -05:00
John W. Parent
aeaca77630
Harden compiler detection against errors (#39736)
Fixes #39622

Add a timeout to compiler detection and allow Spack to proceed when
this timeout occurs.

In all cases, the timeout is 120 seconds: it is assumed any compiler
invocation we do for the purposes of verifying it would resolve in
that amount of time.

Also refine executables that are tested as being possible MSVC
instances, and limit where we try to detect MSVC. In more detail:

* Compiler detection should timeout after a certain period of time.
  Because compiler detection executes arbitrary executables on the
  system, we could encounter a program that just hangs, or even a
  compiler that hangs on a license key or similar. A timeout
  prevents this from hanging Spack.
* Prevents things like cl-.* from being detected as potential MSVC
  installs. cl is always just cl in all cases that Spack supports.
  Change the MSVC class to indicate this.
* Prevent compilers unsupported on certain platforms from being
  detected there (i.e. don't look for MSVC on systems other than
  Windows).

The first point alone is sufficient to address #39622, but the
next two reduce the likelihood of timeouts (which is useful since
those slow down the user even if they are survivable).
2023-09-21 10:42:12 -07:00
Massimiliano Culpo
abad16c198
Restore virtuals normalization on edge construction (#40130)
Put back normalization of the "virtuals" input as a sorted tuple. 

Without this we might get edges that differ just for the order of 
virtuals, or that have lists, which are not hashable.

Add unit-tests to prevent regressions.
2023-09-21 17:02:34 +02:00
Massimiliano Culpo
e963d02a07
Fix a leftover typo from depflag rework (#40101) 2023-09-20 14:16:53 +00:00
Marc Mengel
e1b22325ea
Add details on error messages from requirements (#40092) 2023-09-20 06:16:29 +02:00
Massimiliano Culpo
3b4ca0374e
Use process pool executors for web-crawling and retrieving archives (#39888)
Fix a race condition when searching urls, and updating a shared
set '_visited'.
2023-09-19 15:32:59 +02:00
Massimiliano Culpo
4f14db19c4
ASP-based solver: declare deprecated versions iff config:deprecated:true (#40011)
By default, do not let deprecated versions enter the solve.

Previously you could concretize to something deprecated, only to get errors on install.

With this commit, we get errors on concretization, so the issue is caught earlier.
2023-09-19 09:44:49 +00:00
Adam J. Stewart
dbd520f851
Better detection of Python libs/headers (#39308) 2023-09-18 15:46:33 -07:00
Michael Kuhn
be3f7b5da3
checksum: use FIXME instead of FIX ME when adding versions (#40050)
The former is highlighted by editors, while the latter is not.
2023-09-18 17:31:11 +02:00
Caetano Melone
76816d722a
CI: add spec to job vars (#39905)
* CI: add details about built spec to ci job variables

Co-authored-by: Alec Scott <alec@bcs.sh>
Co-authored-by: Alec Scott <hi@alecbcs.com>
2023-09-15 14:40:00 -06:00
John W. Parent
060bc01273
Windows RPATHing: fix symlink error (#39933)
With 349ba83, you cannot symlink() if the link already exists.
Update the simulated RPATHing logic on Windows to account for that.
2023-09-15 12:55:18 -07:00
Massimiliano Culpo
fb9e5fcc4f
Group primitive url/path handling functions together (#40028) 2023-09-15 15:43:23 +02:00
dependabot[bot]
74a6c48d96
build(deps): bump sphinx from 7.2.5 to 7.2.6 in /lib/spack/docs (#40029)
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.2.5 to 7.2.6.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES.rst)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.5...v7.2.6)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-15 09:51:51 +02:00
Massimiliano Culpo
34402beeb7
Ensure PythonExtension is consistent when finding externals (#40012)
PythonExtension is a base class for PythonPackage, and
is meant to be used for any package that is a Python
extension but is not built using "python_pip".

The "update_external_dependency" method in the base
class calls another method that is defined in the derived
class.

Push "get_external_python_for_prefix" up in the hierarchy
to make method calls consistent.
2023-09-14 18:26:40 +02:00
Harmen Stoppels
6838ee6bb8
Add efficient deptype flag and spack.deptypes module (#39472)
This commit replaces the internal representation of deptypes with `int`, which is more compact
and faster to operate with.

Double loops like:
```
any(x in ys for x in xs)
```
are replaced by constant operations bool(xs & ys), where xs and ys are dependency types. 

Global constants are exposed for convenience in `spack.deptypes`
2023-09-14 12:25:24 +02:00
Harmen Stoppels
d7b5a27d1d
Revert "ASP-based solver: don't declare deprecated versions unless required (#38181)" (#40010)
This reverts commit babd29da50.
2023-09-13 23:33:55 +02:00
Massimiliano Culpo
babd29da50
ASP-based solver: don't declare deprecated versions unless required (#38181)
Currently, the concretizer emits facts for all versions known to Spack, including deprecated versions, and has a specific optimization objective to minimize their use.

This commit simplifies how deprecated versions are handled by considering possible versions for a spec only if they appear in a spec literal, or if the `config:deprecated:true` is set directly or through the `--deprecated` flag. The optimization objective has also been removed, in favor of just ordering versions and having deprecated ones last.

This results in:

a) no delayed errors on install, but concretization errors when deprecated versions would be the only option. This is in particular relevant for CI where it's better to get errors early
b) a slight concretization speed-up due to fewer facts
c) a simplification of the logic program.

Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-09-13 18:15:57 +02:00
Massimiliano Culpo
c4e2d24ca9
Spec.tree: add type hints (#39948) 2023-09-13 11:53:14 +02:00
Massimiliano Culpo
60a5f70b80
Fix typo in metaclasses (#39947) 2023-09-12 22:53:21 +02:00
Harmen Stoppels
a009a1a62a
asp.py: fix deprecation warning (#39943) 2023-09-12 09:27:03 +00:00
dependabot[bot]
28a3be3eca
build(deps): bump black from 23.7.0 to 23.9.1 in /lib/spack/docs (#39937)
Bumps [black](https://github.com/psf/black) from 23.7.0 to 23.9.1.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/23.7.0...23.9.1)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-12 07:44:19 +02:00
Massimiliano Culpo
3302b176fd
Update archspec to latest commit (#39920)
- [x] Intel flags for old architectures
- [x] Support for Sapphire Rapids
- [x] Cache the "ancestors" computation
2023-09-11 10:03:35 -07:00
John W. Parent
fc391d5332
NMake Builder: change property name (#39824)
NMake makefiles are still called makefiles. The corresponding builder
variable was called "nmakefile", which is a bit unintuitive and lead
to a few easy-to-make, hard-to-notice mistakes when creating packages.
This commit renames the builder property to be "makefile"
2023-09-08 11:23:13 -07:00
Dan Lipsa
9e7fe04a77
Windows decompression: fix removal of intermediate file (#38958)
Extensionless archives requiring two-stage decompression and extraction
require intermediate archives to be renamed after decompression/extraction
to prevent collision. Prior behavior attempted to cleanup the intermediate
archive with the original name, this PR ensures the renamed folder is
cleaned instead.

Co-authored-by: Dan Lipsa <dan.lipsa@khq.kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
2023-09-08 10:27:43 -07:00
Massimiliano Culpo
39b9f214a8
Speed-up spack external find execution (#39843)
* Perform external spec detection with multiple workers

The logic to perform external spec detection has been refactored
into classes. These classes use the GoF "template" pattern to account
for the small differences between searching for "executables" and
for "libraries", while unifying the larger part of the algorithm.

A ProcessPoolExecutor is used to parallelize the work.

* Speed-up external find by tagging detectable packages automatically

Querying packages by tag is much faster than inspecting the repository,
since tags are cached. This commit adds a "detectable" tag to every
package that implements the detection protocol, and external detection
uses it to search for packages.

* Pass package names instead of package classes to workers

The slowest part of the search is importing the Python modules
associated with candidate packages. The import is done serially
before we distribute the work to the pool of executors.

This commit pushes the import of the Python module to the job
performed by the workers, and passes just the name of the packages
to the executors.

In this way imports can be done in parallel.

* Rework unit-tests for Windows

Some unit tests were doing a full e2e run of a command
just to check a input handling. Make the test more
focused by just stressing a specific function.

Mark as xfailed 2 tests on Windows, that will be fixed
by a PR in the queue. The tests are failing because we
monkeypatch internals in the parent process, but the
monkeypatching is not done in the "spawned" child
process.
2023-09-08 09:25:50 +02:00
dependabot[bot]
a4d2f8332f
build(deps): bump pytest from 7.4.1 to 7.4.2 in /lib/spack/docs (#39883)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.1 to 7.4.2.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/7.4.1...7.4.2)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-08 08:16:19 +02:00
Tamara Dahlgren
007f02e06a
Docs/Packaging guide: Add BundlePackage (#39691)
* Docs/Packaging guide: Add BundlePackage
* Adjusted version ordering note to reflect convention.
2023-09-07 19:45:51 -07:00
kwryankrattiger
8ec1657136
CI Timing Statistics (#38598)
* Write timing information for installs from cache

* CI: aggregate and upload install_times.json to artifacts

* CI: Don't change root directory for artifact generation

* Flat event based timer variation

Event based timer allows for easily starting and stopping timers without
wiping sub-timer data. It also requires less branching logic when
tracking time.

The json output is non-hierarchical in this version and hierarchy is
less rigidly enforced between starting and stopping.

* Add and write timers for top level install

* Update completion

* remove unused subtimer api

* Fix unit tests

* Suppress timing summary option

* Save timers summaries to user_data artifacts

* Remove completion from fish

* Move spack python to script section

* Write timer correctly for non-cache installs

* Re-add hash to timer file

* Fish completion updates

* Fix null timer yield value

* fix type hints

* Remove timer-summary-file option

* Add "." in front of non-package timer name

---------

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-09-07 15:41:31 -05:00
Harmen Stoppels
7bd95f6ad3
move determine_number_of_jobs into spack.util.cpus, use it in concretize (#37620) 2023-09-07 13:16:51 +02:00
Rémi Lacroix
b81aa42179
CUDA 10.1/10.2: Fix conflicts with PGI 19+. (#39747)
The condition probably did not get updated when the behavior of the version specifier changed.
2023-09-07 00:58:19 -04:00
Dominic Hofer
3b1b261cd8
Handles case of empty craype_dir (#39371)
The craype_dir folder can be empty. It happens to be the case on Arolla and Tsa at CSCS.
2023-09-06 11:54:42 -07:00
John W. Parent
40c4c81c19
"spack config add": support values with ":" (#39831)
This is a fixed version of b72a268

* That commit would discard the final key component (so if you set
  "config:install_tree:root", it would discard "root" and just set
  install tree).
* When setting key:"value", with the quotes, that commit would
  discard the quotes, which would confuse the system if adding a
  value like "{example}" (the "{" character indicates a dictionary).
  This commit retains the quotes.
2023-09-06 18:44:00 +00:00
Adam J. Stewart
8b5b4ade0e
CMakePackage: remove -DBUILD_TESTING flag (#37967) 2023-09-05 18:29:34 -05:00
Harmen Stoppels
f8676db7f4
Revert ""spack config add": allow values with a ":" (#39279)" (#39825)
This reverts commit b72a268bc5.
2023-09-05 20:07:16 +02:00
Wouter Deconinck
cf031e83f0
compilers/gcc.py: support cxx{20,23}_flag (#39777) 2023-09-05 17:09:22 +02:00
Massimiliano Culpo
818c9aeb5a
Add type-hints to the spack.detection package (#39803) 2023-09-05 14:19:57 +02:00
dependabot[bot]
189cd59d13
build(deps): bump pytest from 7.4.0 to 7.4.1 in /lib/spack/docs (#39790)
Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.4.0 to 7.4.1.
- [Release notes](https://github.com/pytest-dev/pytest/releases)
- [Changelog](https://github.com/pytest-dev/pytest/blob/main/CHANGELOG.rst)
- [Commits](https://github.com/pytest-dev/pytest/compare/7.4.0...7.4.1)

---
updated-dependencies:
- dependency-name: pytest
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-09-05 09:51:41 +02:00
Harmen Stoppels
29aa7117f4
glibc: add package (#39695) 2023-09-04 07:55:57 +02:00
John W. Parent
ca872f9c34
Windows: fix pwsh env activate/deactivate; load/unload (#39118)
These commands are currently broken on powershell (Windows) due to
improper use of the InvokeCommand commandlet and a lack of direct
support for the `--pwsh` argument in `spack load`, `spack unload`,
and `spack env deactivate`.
2023-09-01 11:36:27 -07:00
John W. Parent
b72a268bc5
"spack config add": allow values with a ":" (#39279)
If you wanted to set a configuration option like
`config:install_tree:root` to "C:/path/to/config.yaml", Spack  had
trouble parsing this because of the ":" in the value. This adds
logic to allow using quotes to enclose the value, so you can add
`config:install_tree:root:"C:/path/to/config.yaml"`.

Configuration keys should never contain a quote character, so the
presence of any quote is taken to mean that the rest of the string
is specifying the value.
2023-09-01 11:05:02 -07:00
Massimiliano Culpo
86216cc36e
environment: improve spack remove matching (#39390)
search for equivalent specs, not for equal strings when selecting a spec to remove.
2023-08-31 09:28:52 +00:00
dependabot[bot]
ecb7ad493f
build(deps): bump sphinx from 7.2.4 to 7.2.5 in /lib/spack/docs (#39716)
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.2.4 to 7.2.5.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.4...v7.2.5)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-31 10:40:33 +02:00
Harmen Stoppels
fb1e81657c
Remove a few local imports in tests (#39719) 2023-08-31 10:40:02 +02:00
Harmen Stoppels
acb02326aa
ASP-based solver: add hidden mode to ignore versions that are moving targets, use that in CI (#39611)
Setting the undocumented variable SPACK_CONCRETIZER_REQUIRE_CHECKSUM
now causes the solver to avoid accounting for versions that are not checksummed.

This feature is used in CI to avoid spurious concretization against e.g. develop branches.
2023-08-31 08:09:37 +00:00
John W. Parent
1ee7c735ec
Windows: oneapi/msvc consistency (#39180)
Currently, OneAPI's setvars scripts effectively disregard any arguments
we're passing to the MSVC vcvars env setup script, and additionally,
completely ignore the requested version of OneAPI, defaulting to whatever
the latest installed on the system is.

This leads to a scenario where we have improperly constructed Windows
native development environments, with potentially multiple versions of
MSVC and OneAPI being loaded or called in the same env. Obviously this is
far from ideal and leads to some fairly inscrutable errors such as
overlapping header files between MSVC and OneAPI and a different version
of OneAPI being called than the env was setup for.

This PR solves this issue by creating a structured invocation of each
relevant script in an order that ensures the correct values are set in
the resultant build env.

The order needs to be:

1. MSVC vcvarsall
2. The compiler specific env.bat script for the relevant version of
   the oneapi compiler we're looking for. The root setvars scripts seems
   to respect this as well, although it is less explicit
3. The root oneapi setvars script, which sets up everything else the
   oneapi env needs and seems to respect previous env invocations.
2023-08-30 23:19:38 +00:00
Todd Gamblin
396f219011 completion: add alias handling
Bash completion is now smarter about handling aliases. In particular, if all completions
for some input command are aliased to the same thing, we'll just complete with that thing.

If you've already *typed* the full alias for a command, we'll complete the alias.

So, for example, here there's more than one real command involved, so all aliases are
shown:

```console
$ spack con
concretise    concretize    config        containerise  containerize
```

Here, there are two possibilities: `concretise` and `concretize`, but both map to
`concretize` so we just complete that:

```console
$ spack conc
concretize
```

And here, the user has already typed `concretis`, so we just go with it as there is only
one option:

```console
 spack concretis
concretise
```
2023-08-30 12:42:31 -07:00
Todd Gamblin
a3ecd7efed Add concretise and containerise aliases for our UK users
From a user:

> Aargh.
> ```
> ==> Error: concretise is not a recognized Spack command or extension command; check with `spack commands`.
> ```

To make things easier for our friends in the UK, this adds `concretise` and
`containerise` aliases for the `spack concretize` and `spack containerize` commands.

- [x] add aliases
- [x] update completions
2023-08-30 12:42:31 -07:00
Harmen Stoppels
5138c71d34
Revert "Add style tool to fix fish file formatting (#39155)" (#39680)
This reverts commit 70c71e8f93.
2023-08-29 18:12:19 +02:00
dependabot[bot]
ffddaabaa0
build(deps): bump sphinx from 7.2.3 to 7.2.4 in /lib/spack/docs (#39668)
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.2.3 to 7.2.4.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.3...v7.2.4)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-29 14:45:06 +02:00
Adam J. Stewart
70c71e8f93
Add style tool to fix fish file formatting (#39155) 2023-08-28 22:16:44 -05:00
Jonathon Anderson
d9d1eb24f9
modules: copy matched config to prevent bleed (#39421) 2023-08-28 15:03:29 -07:00
Peter Scheibel
95f78440f1
External ROCm: add example configuration (#39602)
* add an example of an external rocm configuration
* include more info
* generalize section to all GPU support
2023-08-26 15:46:25 -07:00
James Smillie
349ba83bc6
Windows symlinking support (#38599)
This reapplies 66f7540, which adds supports for hardlinks/junctions on
Windows systems where developer mode is not enabled.

The commit was reverted on account of multiple issues:

* Checks added to prevent dangling symlinks were interfering with
  existing CI builds on Linux (i.e. builds that otherwise succeed were
  failing for creating dangling symlinks).
* The logic also updated symlinking to perform redirection of relative
  paths, which lead to malformed symlinks.

This commit fixes these issues.
2023-08-25 12:18:19 -07:00
Harmen Stoppels
8914d26867
rebuild-index: fix race condition by avoiding parallelism (#39594) 2023-08-24 08:26:56 +02:00
Massimiliano Culpo
fdea5e7624
Remove leftover attributes from parser (#39574)
#35042 introduced lazy hash parsing, but didn't remove a
few attributes from the parser that were needed only for
concrete specs

This commit removes them, since they are effectively
dead code.
2023-08-24 08:04:43 +02:00
dependabot[bot]
ca1e4d54b5
build(deps): bump sphinx from 7.2.2 to 7.2.3 in /lib/spack/docs (#39603)
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 7.2.2 to 7.2.3.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v7.2.2...v7.2.3)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-24 07:55:14 +02:00
Massimiliano Culpo
806b8aa966
Uppercase global constants in spec.py (#39573)
* Uppercase global constants in spec.py

Co-authored-by: Bernhard Kaindl <bernhardkaindl7@gmail.com>
2023-08-23 21:26:30 +00:00
Greg Becker
9e5ca525f7
do not warn for invalid package name on repo.yaml if subdirectory: '' (#39439) 2023-08-23 23:23:35 +02:00
Massimiliano Culpo
e0059ef961
ASP-based solver: split heuristic for duplicate nodes (#39593)
The heuristic for duplicate nodes contains a few typos, and
apparently slows down the solve for specs that have a lot of
sub-optimal choices to be taken.

This is likely because with a lot of sub-optimal choices, the
low priority, flawed heuristic is being used by clingo.

Here I split the heuristic, so complex rules that matter only
if we allow multiple nodes from the same package are used
only in that case.
2023-08-23 19:02:20 +00:00
Dom Heinzeller
31d5f56913
Add --fail-fast option for generating build caches (#38496)
Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-08-23 11:20:33 +02:00
Tamara Dahlgren
3f5f4cfe26
docs: API, f-string, and path updates to test section (#39584) 2023-08-23 09:22:25 +02:00
Kamil Iskra
c21e00f504
spack.caches: make fetch_cache_location lowercase (#39575)
fetch_cache_location was erroneously renamed to FETCH_cache_location
as part of #39428, breaking "spack module create".
2023-08-22 16:18:51 -04:00
Massimiliano Culpo
9ae1317e79
ASP-based solver: use edge properties for reused specs (#39508)
Since #34821 we are annotating virtual dependencies on
DAG edges, and reconstructing virtuals in memory when
we read a concrete spec from previous formats.

Therefore, we can remove a TODO in asp.py, and rely on
"virtual_on_edge" facts to be imposed.
2023-08-22 12:07:51 -07:00
dependabot[bot]
afebc11742
Bump sphinx from 6.2.1 to 7.2.2 in /lib/spack/docs (#39502)
Bumps [sphinx](https://github.com/sphinx-doc/sphinx) from 6.2.1 to 7.2.2.
- [Release notes](https://github.com/sphinx-doc/sphinx/releases)
- [Changelog](https://github.com/sphinx-doc/sphinx/blob/master/CHANGES)
- [Commits](https://github.com/sphinx-doc/sphinx/compare/v6.2.1...v7.2.2)

---
updated-dependencies:
- dependency-name: sphinx
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-22 12:02:58 +02:00
dependabot[bot]
1d8ff7f742
Bump sphinx-rtd-theme from 1.2.2 to 1.3.0 in /lib/spack/docs (#39562)
Bumps [sphinx-rtd-theme](https://github.com/readthedocs/sphinx_rtd_theme) from 1.2.2 to 1.3.0.
- [Changelog](https://github.com/readthedocs/sphinx_rtd_theme/blob/master/docs/changelog.rst)
- [Commits](https://github.com/readthedocs/sphinx_rtd_theme/compare/1.2.2...1.3.0)

---
updated-dependencies:
- dependency-name: sphinx-rtd-theme
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-22 09:32:39 +02:00
Jordan Galby
f8e0c8caed
Fix Spack freeze on install child process unexpected exit (#39015)
* Fix spack frozen on child process defunct

* Rename parent/child pipe to read/write to emphasize non-duplex mode
2023-08-18 09:41:02 +02:00
Massimiliano Culpo
a022e45866
ASP-based solver: optimize key to intermediate dicts (#39471)
Computing str(spec) is faster than computing hash(spec), and
since all the abstract specs we deal with come from user configuration
they cannot cover DAG structures that are not captured by str() but
are captured by hash()
2023-08-17 14:11:49 +02:00
dependabot[bot]
b19691d503
Bump mypy from 1.5.0 to 1.5.1 in /lib/spack/docs (#39478)
Bumps [mypy](https://github.com/python/mypy) from 1.5.0 to 1.5.1.
- [Commits](https://github.com/python/mypy/compare/v1.5.0...v1.5.1)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-17 12:35:49 +02:00
Peter Scheibel
a89f1b1bf4
Add debugging statements to file search (#39121)
Co-authored-by: Scheibel <scheibel1@ml-9983616.the-lab.llnl.gov>
2023-08-17 08:31:06 +02:00
Harmen Stoppels
190a1bf523
Delay abstract hashes lookup (#39251)
Delay lookup for abstract hashes until concretization time, instead of
until Spec comparison. This has a few advantages:

1. `satisfies` / `intersects` etc don't always know where to resolve the
   abstract hash (in some cases it's wrong to look in the current env,
   db, buildcache, ...). Better to let the call site dictate it.
2. Allows search by abstract hash without triggering a database lookup,
   causing quadratic complexity issues (accidental nested loop during
   search)
3. Simplifies queries against the buildcache, they can now use Spec
   instances instead of strings.

The rules are straightforward:

1. a satisfies b when b's hash is prefix of a's hash
2. a intersects b when either a's or b's hash is a prefix of b's or a's
   hash respectively
2023-08-17 08:08:50 +02:00
Massimiliano Culpo
f41d7a89f3 Extract Package from PackageNode for error messages 2023-08-16 06:20:57 -07:00
Harmen Stoppels
4f07205c63
Avoid sort on singleton list during edge insertion (#39458)
The median length of this list of 1. For reasons I don't know, `.sort()`
still like to call the key function.

This saves ~9% of total database read time, and the number of calls
goes from 5305 -> 1715.
2023-08-16 14:33:03 +02:00
Massimiliano Culpo
08f9c7670e
Do not impose provider conditions, if the node is not a provider (#39456)
* Do not impose provider conditions, if the node is not a provider

fixes #39455

When a node can be a provider of a spec, but is not selected as
a provider, we should not be imposing provider conditions on the
virtual.

* Adjust the integrity constraint, by using the correct atom
2023-08-16 05:15:13 -07:00
Harmen Stoppels
b451791336
json: minify by default (#39457) 2023-08-16 11:26:40 +00:00
Massimiliano Culpo
47f176d635
Add new custom markers to unit tests (#33862)
* Add "only_clingo", "only_original" and "not_on_windows" markers

* Modify tests to use the "not_on_windows" marker

* Mark tests that run only with clingo

* Mark tests that run only with the original concretizer
2023-08-16 09:04:10 +02:00
Massimiliano Culpo
9bb5cffc73 Change semantic for providers
If a possible provider is not used to satisfy a vdep,
then it's not a provider of that vdep.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
135b44ca59 Change "effect_rules" for symmetry with trigger rules
This even though right now we don't have cases where
the effect is on another package.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
d3aca68e8f Rework conflicts so that "vendors" is not needed anymore 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
fb83f8ef31 Add a description at the top of lp files 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
f69c18a922 Remove commented out code in lp files 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
b95a9d2e47 Reduce line length in lp file 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
def4d19980 Demote warning to debug message 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
1db91e0ccd Rename "main_node" -> "make_node" 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
34ebe7f53c Rename ""*_node" -> "*_dupe" 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
d07d5410f3 Rename "stringify", improve docs 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
1db73eb1f2 Add vendors directive
For the time being this directive prevents the vendored package
to be in the same DAG as the one vendoring it.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
2da34de519 Add "^" automatically for named conflicts that don't refer to 'this' package
See https://github.com/spack/spack/pull/38447#discussion_r1285291520
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
d237430f47 Inline a few functions that are not needed anymore 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
3f0adae9ef Remove the need for "node_regex" 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
3b4d7bf119 Rename method: "root_node" -> "main_node" 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
b3087b32c6 Rename const: "root_node_id" -> "main_node_id" 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
ad9c90cb2e Rename atom: "special_case" -> "multiple_nodes_attribute" 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
1b0e113a9d Rename atom: "facts" -> "pkg_fact" 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
6df5738482 Simplify "node_has_variant" internal atom. 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
927d831612 Removed leftover TODOs 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
1de5117ef1 Improve handling of cases with cycles
To avoid paying the cost of setup and of a full grounding again,
move cycle detection into a separate program and check first if
the solution has cycles.

If it has, ground only the integrity constraint preventing cycles
and solve again.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
006e69265e Optimize grounding of "can_inherit_flags" 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
eaec3062a1 Fix computation of max nodes 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
d5eb5106b0 Add unit-tests for use cases requiring separate concretization of build deps 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
9f8edbf6bf Add a new configuration option to select among different concretization modes
The "concretizer" section has been extended with a "duplicates:strategy"
attribute, that can take three values:

- "none": only 1 node per package
- "minimal": allow multiple nodes opf specific packages
- "full": allow full duplication for a build tool
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
a4301badef Fix a few bugs in the encoding when imposing constraints on build deps only
These bugs would show up when we try to split nodes by
imposing different targets or different compilers to all
build dependencies.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
4565811556 Construct unification sets on demand, improve heuristic 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
b94d54e4d9 Reduce the number of unification sets to only two 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
a410b22098 Make cycle detection optional, to speed-up grounding and solving 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
c1a73878ea Deduplicate trigger and effect conditions in packages
This refactor introduces extra indices for triggers and
effect of a condition, so that the corresponding clauses
are evaluated once for every condition they apply to.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
ae553051c8 Extract a function to emit variant rules 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
b94e22b284 ASP-based solver: do not optimize on known dimensions
All the solution modes we use imply that we have to solve for all
the literals, except for "when possible".

Here we remove a minimization on the number of literals not
solved, and emit directly a fact when a literal *has* to be
solved.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
e25dcf73cd Tweak a unit test by allowing a different type of exception to be raised 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
b7cc4bd247 Reduce the dependency types in a solve
Count the maximum number of nodes based on dependency types
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
22c95923e3 Parametrize all the logic program for multiple nodes
Introduce the concept of "condition sets", i.e. the set of packages on which
a package can require / impose conditions. This currently maps to the link/run
sub-dag of each package + its direct build dependencies.

Parametrize the "condition" and "requirement" logic to multiple nodes.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
c050b99a06 Introduce unification sets
Unification sets are possibly overlapping sets of nodes that
might contain at most a single configuration for any package.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
60f82685ae Allow clingo to generate edges 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
27ab53b68a Rework the encoding to introduce node(ID, Package) nested facts
So far the encoding has a single ID per package, i.e. all the
facts will be node(0, Package). This will prepare the stage for
extending this logic and having multiple nodes from the same
package in a DAG.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
907a80ca71 Remove unneeded #defined directives 2023-08-15 15:54:37 -07:00
Massimiliano Culpo
a53cc93016 Remove useless rule
The version_equivalent fact was deleted in #36347,
but the corresponding rule was not removed.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
6ad0dc3722 Transform many package related facts to use a nested function
Each fact that is deduced from package rules, and start with
a bare package atom, is transformed into a "facts" atom containing
a nested function.

For instance we transformed

  version_declared(Package, ...) -> facts(Package, version_declared(...))

This allows us to clearly mark facts that represent a rule on the package,
and will be of help later when we'll have to distinguish the cases where
the atom "Package" is being used referred to package rules and not to a
node in the DAG.
2023-08-15 15:54:37 -07:00
Massimiliano Culpo
29e2997bd5
spack.caches: uppercase global variables (#39428) 2023-08-15 09:59:02 +02:00
Harmen Stoppels
0976ad3184
spack.config: use all caps for globals (#39424) 2023-08-15 08:19:36 +02:00
markus-ferrell
c202a045e6
Windows: executable/path handling (#37762)
Windows executable paths can have spaces in them, which was leading to
errors when constructing Executable objects: the parser was intended
to handle cases where users could provide an executable along with one
or more space-delimited arguments.

* Executable now assumes that it is constructed with a string argument
  that represents the path to the executable, but no additional arguments.
* Invocations of Executable.__init__ that depended on this have been
  updated (this includes the core, tests, and one instance of builtin
  repository package).
* The error handling for failed invocations of Executable.__call__ now
  includes a check for whether the executable name/path contains a
  space, to help users debug cases where they (now incorrectly)
  concatenate the path and the arguments.
2023-08-14 23:29:12 +00:00
markus-ferrell
d823037c40
Windows: enable "spack install" tests (#34696)
* The module-level skip for tests in `cmd.install` on Windows is removed.
  A few classes of errors still persist:

  * Cdash tests are not working on Windows
  * Tests for failed installs are also not working (this will require
    investigating bugs in output redirection)
  * Environments are not yet supported on Windows

  overall though, this enables testing of most basic uses of "spack install"
* Git repositories cached for version lookups were using a layout that
  mimicked the URL as much as possible. This was useful for listing the
  cache directory and understanding what was present at a glance, but
  the paths were overly long on Windows. On all systems, the layout is
  now a single directory based on a hash of the Git URL and is shortened
  (which ensures a consistent and acceptable length, and also avoids
  special characters).
  * In particular, this removes util.url.parse_git_url and its associated
    test, which were used exclusively for generating the git cache layout
* Bootstrapping is now enabled for unit tests on Windows
2023-08-14 13:15:40 -07:00
Massimiliano Culpo
6e31676b29
Fix style issues with latest versions of tools (#39422) 2023-08-14 12:38:59 -04:00
Harmen Stoppels
a2a52dfb21
Fix containerize view symlink issue (#39419) 2023-08-14 16:02:48 +00:00
Harmen Stoppels
da46b63a34
Fix broken semver regex (#39414) 2023-08-14 15:38:06 +02:00
Harmen Stoppels
c882214273
spack bootstrap dev: detect git as an external (#39417)
#36770 added git as a dependency to `setuptools-scm`. This in turn makes `git` a
transitive dependency for our bootstrapping process. 

Since `git` may take a long time to build, and is found on most systems, try to 
detect it as an external.
2023-08-14 10:29:15 +00:00
Sergey Kosukhin
237a56a305
autotools: set 'ldlibs' as 'LIBS' (#17254) 2023-08-13 00:38:10 -07:00
Scott Wittenburg
1b1ed1b1fa
ci: continue to support SPACK_SIGNING_KEY (#39170) 2023-08-12 08:52:46 -07:00
Adam J. Stewart
f7918fd8ab
Python: remove maintainer (#39384) 2023-08-12 09:08:25 +02:00
Todd Gamblin
ea082539e4
Revert "package import: remove magic import line (#39183)" (#39380)
This reverts commit 8e7c53a8ba.
2023-08-11 09:04:40 +02:00
Massimiliano Culpo
143146f4f3
spack.repo: uppercase the global PATH variable (#39372)
This makes the name of the global variable representing
the repository currently in use uppercase. Doing so is advised
by pylint rules, and helps to identify where the global is used.
2023-08-11 09:04:16 +02:00
dependabot[bot]
c764f9b1ab
build(deps): bump mypy from 1.4.1 to 1.5.0 in /lib/spack/docs (#39383)
Bumps [mypy](https://github.com/python/mypy) from 1.4.1 to 1.5.0.
- [Commits](https://github.com/python/mypy/compare/v1.4.1...v1.5.0)

---
updated-dependencies:
- dependency-name: mypy
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-11 08:01:34 +02:00
markus-ferrell
f3c080e546
Windows build systems: use ninja and enable tests (#33589)
* Set default CMake generator is ninja on Windows
* Enable build systems tests (except for autotools/make)
2023-08-10 17:16:23 +00:00
dependabot[bot]
37634f8b08
build(deps): bump pygments from 2.15.1 to 2.16.1 in /lib/spack/docs (#39365)
Bumps [pygments](https://github.com/pygments/pygments) from 2.15.1 to 2.16.1.
- [Release notes](https://github.com/pygments/pygments/releases)
- [Changelog](https://github.com/pygments/pygments/blob/master/CHANGES)
- [Commits](https://github.com/pygments/pygments/compare/2.15.1...2.16.1)

---
updated-dependencies:
- dependency-name: pygments
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-10 18:10:32 +02:00
dependabot[bot]
3f83ef6566
build(deps): bump flake8 from 6.0.0 to 6.1.0 in /lib/spack/docs (#39366)
Bumps [flake8](https://github.com/pycqa/flake8) from 6.0.0 to 6.1.0.
- [Commits](https://github.com/pycqa/flake8/compare/6.0.0...6.1.0)

---
updated-dependencies:
- dependency-name: flake8
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-10 05:34:33 -04:00
dependabot[bot]
554ce7f063
build(deps): bump sphinx-design from 0.4.1 to 0.5.0 in /lib/spack/docs (#39367)
Bumps [sphinx-design](https://github.com/executablebooks/sphinx-design) from 0.4.1 to 0.5.0.
- [Release notes](https://github.com/executablebooks/sphinx-design/releases)
- [Changelog](https://github.com/executablebooks/sphinx-design/blob/main/CHANGELOG.md)
- [Commits](https://github.com/executablebooks/sphinx-design/compare/v0.4.1...v0.5.0)

---
updated-dependencies:
- dependency-name: sphinx-design
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-10 05:29:52 -04:00
Aiden Grossman
23963779f4
Prefix conflict messages with package name (#39106)
* Prefix conflict messages with package name

This patch prefixes all conflict messages with the package name to
alleviate what was otherwise a very manual process. Note that this patch
is a one line change but has a fairly outsized impact.

* same for requires directive

---------

Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-08-10 08:09:00 +00:00
Ricard Zarco Badia
45c5af10c3
Fixed broken calls to _if_ninja_target_execute (#38992)
Co-authored-by: Tamara Dahlgren <35777542+tldahlgren@users.noreply.github.com>
2023-08-10 09:34:02 +02:00
Harmen Stoppels
532a37e7ba
Revert "Spec versions: allow git. references for branches with / (#38239)" (#39354)
This reverts commit 3453259c98.
2023-08-10 08:56:39 +02:00
dependabot[bot]
a3c7ad7669
build(deps): bump urllib3 from 2.0.3 to 2.0.4 in /lib/spack/docs (#39002)
Bumps [urllib3](https://github.com/urllib3/urllib3) from 2.0.3 to 2.0.4.
- [Release notes](https://github.com/urllib3/urllib3/releases)
- [Changelog](https://github.com/urllib3/urllib3/blob/main/CHANGES.rst)
- [Commits](https://github.com/urllib3/urllib3/compare/2.0.3...2.0.4)

---
updated-dependencies:
- dependency-name: urllib3
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-10 05:04:58 +00:00
markus-ferrell
01b7cc5106
Windows: enable more stage tests (#36834)
Enable some already-supported tests (no changes to staging logic).
2023-08-09 14:24:30 -07:00
Andrey Parfenov
b1e33ae37b
add info about devito and qe for intel env (#39357)
Signed-off-by: Andrey Parfenov <andrey.parfenov@intel.com>
2023-08-09 12:47:35 -04:00
Jordan Galby
c36617f9da
Fix package.py error handling bug (#39017) 2023-08-09 17:47:43 +02:00
dependabot[bot]
deadb64206
build(deps): bump black from 23.1.0 to 23.7.0 in /lib/spack/docs (#38982)
Bumps [black](https://github.com/psf/black) from 23.1.0 to 23.7.0.
- [Release notes](https://github.com/psf/black/releases)
- [Changelog](https://github.com/psf/black/blob/main/CHANGES.md)
- [Commits](https://github.com/psf/black/compare/23.1.0...23.7.0)

---
updated-dependencies:
- dependency-name: black
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-09 17:19:49 +02:00
Harmen Stoppels
8e7c53a8ba
package import: remove magic import line (#39183) 2023-08-09 14:28:06 +00:00
Adam J. Stewart
39d4c402d5
Fix suffix of tab completion scripts (#39154) 2023-08-09 08:28:55 -05:00
Harmen Stoppels
e51748ee8f
zlib-api: new virtual with zlib/zlib-ng as providers (#37372)
Introduces a new virtual zlib-api, which replaces zlib in most packages.

This allows users to switch to zlib-ng by default for better performance.
2023-08-09 09:22:58 -04:00
Jonathon Anderson
9aa3b4619b
containerize: ensure bootstrap images contain all system dependencies (#36818)
This also makes `spack bootstrap status` exit 1 if some dependency is missing
2023-08-09 09:46:59 +02:00
Peter Scheibel
3453259c98
Spec versions: allow git. references for branches with / (#38239) 2023-08-09 09:07:04 +02:00
Massimiliano Culpo
97d632a161
Push conflict between rocm and blt down to packages that actually use blt (#39339) 2023-08-09 07:53:17 +02:00
Adam J. Stewart
361632fc4b
Ensure that all variants have a description (#39025)
* Ensure that all variants have a description

* Update mock packages too

* Fix test invocations

* Black fix

* mgard: update variant descriptions

* flake8 fix

* black fix

* Add to audit tests

* Relax type hints

* Older Python support

* Undo all changes to mock packages

* Flake8 fix
2023-08-08 09:29:49 -05:00
simonLeary42
6576655137
Name clash error message include hash of clashing packages (#39234)
Co-authored-by: Simon <simonleary@umass.edu>
2023-08-08 14:36:49 +02:00
Harmen Stoppels
86f9d3865b
Fix broken inode assertion (#39188) 2023-08-08 09:21:23 +02:00
Harmen Stoppels
7bd5d1fd3c
use {%compiler.name}{@compiler.version} in matching_specs (#39247) 2023-08-07 18:54:10 +02:00
Massimiliano Culpo
ba1d295023
Extract prefix locks and failure markers from Database (#39024)
This PR extracts two responsibilities from the `Database` class:
1. Managing locks for prefixes during an installation
2. Marking installation failures

and pushes them into their own class (`SpecLocker` and `FailureMarker`). These responsibilities are also pushed up into the `Store`, leaving to `Database` only the duty to manage `index.json` files.

`SpecLocker` classes no longer share a global list of locks, but locks are per instance. Their identifier is simply `(dag hash, package name)`, and not the spec prefix path, to avoid circular dependencies across Store / Database / Spec.
2023-08-07 06:47:52 -04:00
Harmen Stoppels
27f04b3544
Picklable HTTPError (#39285) 2023-08-07 10:48:35 +02:00
Harmen Stoppels
ef544a3b6d
Add a more detailed HTTPError (#39187) 2023-08-05 11:16:51 +02:00
Adam J. Stewart
4eed832653
py-pyqt6: add new package (#32696) 2023-08-04 18:49:54 -05:00
Harmen Stoppels
6e933ac7df
repo cache: use -inf default instead of 0 (#39214)
FastPackageChecker.modified_since should use a default number < 0

When the repo cache does not exist, Spack uses mtime 0. This causes the repo
cache not to be generated when the repo has mtime 0.

Some popular package managers such as spack use 0 mtime normalization for
reproducible tarballs. So when installing spack with spack from a buildcache, the
repo cache doesn't generate

Also add some typehints
2023-08-03 14:13:13 +02:00
Harmen Stoppels
2069a42ba3
Buildcache commands cleanup, again... (#39203)
* Inform mypy that tty.die is noreturn

* avoid temporary allocation in env

* update spack buildcache save-specfile

* fix spack buildcache check/download/get-buildcache-name

- ensure that required args and mutually exclusive ones are marked as
  such in argparse for better error messages
- deprecate --spec-file everywhere
- use disambiguate for better error messages
2023-08-03 10:44:02 +02:00
Adam J. Stewart
edbf12cfa8
Add qmake virtual provider (#38848) 2023-08-02 13:50:37 -05:00
kwryankrattiger
0b4631a774
CI: Refactor ci reproducer (#37088)
* CI: Refactor ci reproducer

* Autostart container
* Reproducer paths match CI paths
* Generate start scripts for docker and reproducer

* CI: Add interactive and gpg options to reproduce-build

* Interactive will determine if the docker container persists
  after running reproduction.
* GPG path/url allow downloading GPG keys needed for binary
  cache download validation. This is important for running
  reproducer for protected CI jobs.

* Add exit_on_failure option to CI scripts

* CI: Add runtime option for reproducer
2023-08-02 09:51:12 -07:00
Harmen Stoppels
e7fa6d99bf
version: move to module, avoid circular imports (#39077) 2023-08-02 17:47:08 +02:00
Harmen Stoppels
03c0d74139
buildcache extractall: extract directly into spec.prefix (#37441)
- Run `mkdirp` on `spec.prefix`
- Extract directly into `spec.prefix`
  1. No need for `$store/tmp.xxx` where we extract the tarball directly, pray that it has one subdir `<name>-<version>-<hash>`, and then `rm -rf` the package prefix followed by `mv`.
  2. No need to clean up this temp dir in `spack clean`.
  3. Instead figure out package directory prefix from the tarball contents, and strip the tarinfo entries accordingly (kinda like tar --strip-components but more strict)
- Set package dir permissions
- Don't error during error handling when files cannot removed
- No need to "enrich" spec.json with this tarball-toplevel-path

After this PR, we can in fact tarball packages relative to `/` instead of `spec.prefix/..`, which makes it possible to use Spack tarballs as container layers, where relocation is impossible, and rootfs tarballs are expected.
2023-08-02 17:06:13 +02:00
Wouter Deconinck
a14f4b5a02
feat: move -N/--namespace(s) to common args, allow in buildcache list (#36719)
`spack buildcache list` did not have a way to display the namespace of
packages in the buildcache. This PR adds that functionality.

For consistency's sake, it moves the `-N/--namespace` arg definition to
the `common/arguments.py` and modifies `find`, `solve`, `spec` to use
the common definition.

Previously, `find` was using `--namespace` (singular) to control whether
to display the namespace (it doesn't restrict the search to that
namespace). The other commands were using `--namespaces` (plural). For
backwards compatibility and for consistency with `--deps`, `--tags`,
etc, the plural `--namespaces` was chosen. The argument parser ensures
that `find --namespace` will continue to behave as before.

Co-authored-by: Harmen Stoppels <me@harmenstoppels.nl>
2023-08-02 11:16:14 +00:00
Chris White
8d2a32f66d
remove CMAKE_GENERATOR from the host-config because it cannot be overwritten on the command line (#39044) 2023-08-01 08:57:56 -07:00
Alec Scott
d4f41b51f4
Add spack checksum --verify, fix --add (#38458)
* Add rewrite of spack checksum to include --verify and better add versions to package.py files
* Fix formatting and remove unused import
* Update checksum unit-tests to correctly test multiple versions and add to package
* Remove references to latest in stage.py
* Update bash-completion scripts to fix unit tests failures
* Fix docs generation
* Remove unused url_dict argument from methods
* Reduce chance of redundant remote_versions work
* Add print() before tty.die() to increase error readablity
* Update version regular expression to allow for multi-line versions
* Add a few unit tests to improve test coverage
* Update command completion
* Add type hints to added functions and fix a few py-lint suggestions
* Add @no_type_check to prevent mypy from failing on pkg.versions
* Add type hints to format.py and fix unit test
* Black format lib/spack/spack/package_base.py
* Attempt ignoring type errors
* Add optional dict type hint and declare versions in PackageBase
* Refactor util/format.py to allow for url_dict as an optional parameter
* Directly reference PackageBase class instead of using TypeVar
* Fix comment typo

---------

Co-authored-by: Tamara Dahlgren <dahlgren1@llnl.gov>
2023-07-31 21:49:43 +00:00
Adam J. Stewart
9394fa403e
Remove Xcode mock-up (#39020)
* Remove Xcode mock-up

* Remove unused imports
2023-07-31 10:24:04 -07:00
Scott Wittenburg
2c74b433aa
ci: Make signing requirement explicit (#38995)
Instead of inferring whether to sign binaries, make it explicit,
and fail rebuild jobs early if signing is required but cannot be
accomplished.
2023-07-26 09:16:15 -06:00
Jonathon Anderson
3ad65bbfc1
Always set workflow:rules in spack ci pipelines (#38921) 2023-07-24 14:31:39 -05:00
Ashwin Kumar Karnad
399271832b
Fix typo: binary_caches.rst (#39064) 2023-07-24 15:09:34 +02:00
Harmen Stoppels
4bcceddba9
Don't add .spack/binary_distribution twice to the tarball when re-distributing (#39042)
Previously, spack would list the ./spack/binary_distribution file twice when pushing
a package that was installed from a binary tarball itself.
2023-07-24 14:39:37 +02:00
Harmen Stoppels
0fff219aa4
Fix broken sanitize_file_path (#38926)
The sanitization function is completely bogus as it tries to replace /
on unix after ... splitting on it. The way it's implemented is very
questionable: the input is a file name, not a path. It doesn't make
sense to interpret the input as a path and then make the components
valid -- you'll interpret / in a filename as a dir separator.

It also fails to deal with path components that contain just unsupported
characters (resulting in empty component).

The correct way to deal with this is to have a function that takes a
potential file name and replaces unsupported characters.

I'm not going to fix the other issues on Windows, such as reserved file
names, but left a note, and hope that @johnwparent can fix that
separately.

(Obviously we wouldn't have this problem at all if we just fixed the
filename in a safe way instead of trying to derive something from
the url; we could use the content digest when available for example)
2023-07-24 12:30:47 +02:00
百地 希留耶
90ac0ef66e
Implement fish completion (#29549)
* commands: provide more information to Command

* fish: Add script to generate fish completion

* fish: auto prepend `spack` command to avoid duplication

* fish: impove completion generation code readability

* commands: replace match-case with if-else

* fish: fix optspec variable name prefix

* fish: fix return value in get_optspecs

* fish: fix return value in get_optspecs

* format: split long line and trim trailing space

* bugfix: replace f-string with interpolation

* fish: compete more specs and some fixes

* fish: complete hash spec starts with /

* fish: improve compatibility

* style: trim trailing whitespace

* commands: add fish to update args and update tests

* commands: add fish completion file

* style: merge imports

* fish: source completion in setup-env

* fish: caret only completes dependencies

* fish: make sure we always get same order of output

* fish: spack activate
only show installed packages that have extensions

* fish: update completion file

* fish: make dict keys sorted

* Blacken code

* Fix bad merge

* Undo style changes to setup-env.fish

* Fix unit tests

* Style fix

* Compatible with fish_indent

* Use list for stability of order

* Sort one more place

* Sort more things

* Sorting unneeded

* Unsort

* Print difference

* Style fix

* Help messages need quotes

* Arguments to -a must be quoted

* Update types

* Update types

* Update types

* Add type hints

* Change order of positionals

* Always expand help

* Remove shared base class

* Fix type hints

* Remove platform-specific choices

* First line of help only

* Remove unused maps

* Remove suppress

* Remove debugging comments

* Better quoting

* Fish completions have no double dash

* Remove test for deleted class

* Fix grammar in header file

* Use single quotes in most places

* Better support for remainder nargs

* No magic strings

* * and + can also complete multiple

* lower case, no period

---------

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2023-07-22 08:55:12 -05:00
Massimiliano Culpo
c0b42151c3
Remove spack.repo.IndexError (#39029)
This exception is never used and
overrides a built-in.
2023-07-21 15:33:30 +02:00
Ashwin Kumar Karnad
405f563909
binary_caches.rst: fix typo (#39030) 2023-07-21 10:39:53 +02:00
Sergey Kosukhin
d2566e3d62
nag: update the versioning scheme (#35457)
* nag: append build number to the version
* nag: add version 7.1.7125
* nag: deprecate unavailable versions
2023-07-20 12:03:26 -07:00
Massimiliano Culpo
50b90e430d
spack.util.lock: add type-hints, remove **kwargs in method signatures (#39011) 2023-07-20 09:41:23 +02:00
Massimiliano Culpo
3a565c66e9
Respect custom user store when bootstrapping (#39001)
The user store is lazily evaluated. The change
in #38975 made it such that the first evaluation
was happening in the middle of swapping to user
configuration.

Ensure we construct the user store before that.
2023-07-19 19:53:33 -04:00
Xavier Delaruelle
d9fbdfbee9
modules: use curly braces to enclose value in Tcl modulefile (#38375)
Use curly braces instead of quotes to enclose value or text in Tcl
modulefile. Within curly braces Tcl special characters like [, ] or $
are treated verbatim whereas they are evaluated within quotes.

Curly braces is Tcl recommended way to enclose verbatim content [1].

Note: if curly braces charaters are used within content, they must be
balanced. This point has been checked against current repository and no
unbalanced curly braces has been spotted.

Fixes #24243

[1] https://wiki.tcl-lang.org/page/Tcl+Minimal+Escaping+Style
2023-07-19 17:57:37 +02:00
Harmen Stoppels
185bccb70f
Fetch & patch: actually acquire stage lock, and many more issues (#38903)
* Fetching patches wouldn't result in acquiring a stage lock during install
* The installer would acquire a stage lock *after* fetching instead of
   before, leading to races
* The name of the stage for patches was random, so on build failure
   (where stage dirs are not removed), these directories would continue
   to exist after a second successful install.
* There was this redundant "composite fetch" object -- there's already
   a composite stage. Remove this.
* For some reason we do *double* shasum validation of patches, before
   and after compression -- that's just too much? I removed it.
2023-07-19 15:06:56 +02:00
Harmen Stoppels
886946395d
drop redundant rpaths post install (#38976)
Spack heuristically adds `<install prefix>/lib` and `<install prefix>/lib64` as rpath entries, as it doesn't know what the install dir is going to be ahead of the build. This PR cleans up non-existing, absolute paths[^1], which

1. avoids redundant stat calls at runtime
2. drops redundant rpaths in `patchelf`, making it relocatable -- you don't need patchelf recursively then.

[^1]: It also removes relative paths not starting with `$` (so, `$ORIGIN/../lib` is retained -- we _could_ interpolate `$ORIGIN`, but that's hard to get right when symlinks have to be taken into account). Relative paths _are_ supported in glibc, but are relative to _the current working directory_, which is madness, and it would be better to drop those paths.
2023-07-19 09:48:31 +00:00
Massimiliano Culpo
f34c93c5f8
llnl.util.lock: add type-hints (#38977)
Also uppercase global variables in the module
2023-07-19 11:23:08 +02:00
Massimiliano Culpo
a7f2abf924
Remove LazyReference from code (#38944)
A LazyReference object is a reference to an attribute of a 
lazily evaluated singleton. Its only purpose is to let developers
use shorter names to refer to such attribute.

This class does more harm than good, as it obfuscates the fact
that we are using the attribute of a global object. Also, it can easily
go out of sync with the singleton it refers to if, for instance, the
singleton is updated but the references are not.

This commit removes the LazyReference class entirely, and access
the attributes explicitly passing through the global value to which
they are attached.
2023-07-19 11:08:51 +02:00
markus-ferrell
416edfa229
Windows testing: enable tests for installer components (#36970)
These tests now work without any changes to core. Furthermore, it is
surprising that they had to be disabled (at least, as long as the
installer.py tests are run on Windows: these tests are more-basic
and their functionality would have been exercised automatically).
2023-07-18 16:19:14 -07:00
Massimiliano Culpo
14f3297cca
Ensure the bootstrap store has a padding length of zero (#38975)
Without this PR, padded length was propagating from user configuration to
bootstrap configuration, and was causing the issue reported in #38963
2023-07-18 23:49:22 +02:00
markus-ferrell
f24f98a1e2
Windows testing: enable bootstrap test (#36972) 2023-07-18 14:42:26 -07:00
markus-ferrell
ffc283ab8b
test_clear_failures_success: run on Windows too (#36792) 2023-07-18 22:42:57 +02:00
Harmen Stoppels
3fef586cfb
binary cache docs: remove redundant flag and comment (#38960) 2023-07-18 22:38:04 +02:00
markus-ferrell
b710778bda
Windows testing: enable architecture test (#36973)
Works out of the box: remove skip.
2023-07-18 12:33:52 -07:00
markus-ferrell
a965fe9354
Windows testing: enable "spack clean" tests (#36840)
They work out out of the box on windows. Simply removing skips.
2023-07-18 12:25:32 -07:00