Commit graph

5366 commits

Author SHA1 Message Date
Nathan Hanford
a3520d14bd
Splice differing virtual packages (#27919)
Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-12 13:31:39 -07:00
psakievich
7d534f38d6
Don't allow replacement of root develop specs with --reuse (#28605)
* Fix to concretize.lp

do not allow dev specs to be reused

Co-authored-by: Gregory Becker <becker33@llnl.gov>
2022-04-12 10:37:24 -07:00
Peter Scheibel
2aec5b65f3
Git commit versions bugfix: Environments and Concretization (#29717)
Spack added support in #24639 for ad-hoc Git-commit-hash-based
versions: A user can install a package x@hash, where X is a package
that stores its source code in a Git repository, and the hash refers
to a commit in that repository which is not recorded as an explicit
version in the package.py file for X.

A couple issues were found relating to this:

* If an environment defines an alternative package repo (i.e. with
  repos.yaml), and spack.yaml contains user Specs with ad-hoc
  Git-commit-hash-based versions for packages in that repo,
  then as part of retrieving the data needed for version comparisons
  it will attempt to retrieve the package before the environment's
  configuration is instantiated.
* The bookkeeping information added to compare ad-hoc git versions was
  being stripped from Specs during concretization (such that user
  Specs which succeeded before concretizing would then fail after)

This addresses the issues:

* The first issue is resolved by deferring access to the associated
  Package until the versions are actually compared to one another.
* The second issue is resolved by ensuring that the Git bookkeeping
  information is explicitly applied to Specs after they are concretized.

This also:

* Resolves an ambiguity in the mock_git_version_info fixture used to
  create a tree of Git commits and provide a list where each index
  maps to a known commit.
* Isolates the cache used for Git repositories in tests using the
  mock_git_version_info fixture
* Adds a TODO which points out that if the remote Git repository
  overwrites tags, that Spack will then fail when using
  ad-hoc Git-commit-hash-based versions
2022-04-12 09:58:26 -07:00
Doug Jacobsen
a9b4f33f23
Update gpg publish to work with mirror arguments (#28740)
This commit updates the `gpg publish` command to work with the mirror
arguments, when trying to push keys to a mirror.

- [x] update `gpg publish command
- [x] add test for publishing GPG keys and rebuilding the key index within a mirror
2022-04-11 14:48:08 -07:00
Sergey Kosukhin
6e5cba7b82
cray platform: unload cray-mpich (#29898) 2022-04-11 14:34:09 -07:00
Seth R. Johnson
1ea05cd456
macos: fewer calls to sw_vers (#29997)
In a typical call to spack, the OperatingSystem gets instantiated
multiple times. For macOS, each one requires a call to `sw_vers`, which
is done through the Executable helper class. Memoizing
reduces the call count from "spac spec" from three to one.
2022-04-11 09:31:24 +02:00
Massimiliano Culpo
23e85f4086
Environments: unify the spec objects on first concretization (#29948)
Currently environments are indexed by build hashes. When looking into this bug I noticed there is a disconnect between environments that are concretized in memory for the first time and environments that are read from a `spack.lock`. The issue is that specs read from a `spack.lock` don't have a full hash, since they are indexed by a build hash which is strictly coarser. They are also marked "final" as they are read from a file, so we can't compute additional hashes. 

This bugfix PR makes "first concretization" equivalent to re-reading the specs from a corresponding `spack.lock`, and doing so unveiled a few tests were we were making wrong assumptions and relying on the fact that a `spack.lock` file was not there already.

* Add unit test
* Modify mpich to trigger jobs in pipelines
* Fix two failing unit tests
* Fix another full_hash vs. build_hash mismatch in tests
2022-04-08 15:26:17 -06:00
Harmen Stoppels
99083f1706
Deprecate top-level module config (#28659)
* Ignore top-level module config; add auto-update

In Spack 0.17 we got module sets (modules:[name]:[prop]), and for
backwards compat modules:[prop] was short for modules:default:[prop].

But this makes it awkward to define default config for the "default"
module set.

Since 0.17 is branched off, we can now deprecate top-level module config
(that is, just ignore it with a warning).

This PR does that, and it implements `spack config update modules` to
make upgrading easy (we should have added that to 0.17 already...)

It also removes references to  `dotkit` stuff which was already
deprecated in 0.13 and could have been removed in 0.14.

Prefix inspections are the only exception, since the top-level prefix inspections
used for `spack load` and `spack env activate`.
2022-04-08 19:00:35 +00:00
Greg Becker
79ba0c50c1
concretize.lp: enforce target compatibility through DAG (#29694)
Spack currently allows dependencies to be concretized for an 
architecture incompatible with the root. This commit adds rules
to make this situation impossible by design.
2022-04-08 11:01:04 +02:00
Massimiliano Culpo
ff04d1bfc1
Use the non-deprecated MetaPathFinder interface (#29745)
* Extract the MetaPathFinder and Loaders for packages in their own classes

https://peps.python.org/pep-0451/

Currently, RepoPath and Repo implement the (deprecated) interface of
MetaPathFinder (find_module) and of Loader (load_module). This commit
extracts both of them and places the code in their own classes.

The MetaPathFinder interface is updated to contain both the deprecated
"find_module" (for Python 2.7 support) and the recommended "find_spec".
Update of the Loader interface is deferred at a subsequent commit.

* Move the lines to be prepended inside "RepoLoader"

Also adjust the naming of a few variables too

* Remove spack.util.imp, since code is only used in spack.repo

* Remove support from loading Python modules Python > 3 but < 3.5

* Remove `Repo._create_namespace`

This function was interacting badly with the MetaPathFinder
and causing issues with "normal" imports. Removing the
function allows to do things like:
```python
import spack.pkg.builtin.mpich
cls = spack.pkg.builtin.mpich.Mpich
```

* Remove code needed to trigger the Singleton evaluation

The finder is coded in a way to trigger the Singleton,
so we don't need external code now that we register it
at module level into `sys.meta_path`.

* Add unit tests
2022-04-07 15:58:20 -07:00
Scott Wittenburg
b2eda32e55
ci: clean up env between tests with working_dir (#29807) 2022-04-07 17:11:58 +02:00
Scott Wittenburg
ff33978b0d
ci: do not prune specs when stacks are touched (#29825) 2022-04-07 08:55:40 -06:00
Harmen Stoppels
43577beb9c
autotools.py: pic flags for %nvhpc (#29920) 2022-04-07 14:52:47 +02:00
Harmen Stoppels
beff697cc0
web.py: set User-Agent (#29919)
Some servers require `User-Agent` to be set, and otherwise error with
access denied. One such example is mpich.

To fix this, set `User-Agent: Spackbot/[version]` as a header.
Apparently by convention, it should include the word `bot`.
2022-04-06 15:59:20 +02:00
Harmen Stoppels
558f7f007e
link_tree.py: format conflict error message (#29870)
Show each `[src a] and [src b] both project to [dst]` on a separate
line.
2022-04-06 07:27:02 +02:00
Erik Schnetter
69b3a88fa3
Bugfix: CVS fetching (#29793)
#27021 broke fetching for CVS-based packages because:

- The mirror logic was using URL parsing to extract a path from the
  CVS repository location
- #27021 added sanity checks to enforce that strings passed to the
  URL parser were actually URLs

This replaces the call to "url_util.parse" with logic that is
customized for CVS. This implies that VCSFetchStrategy should
rename the "url_attr" attribute to something more generic, but
that should be handled separately.
2022-04-05 18:45:45 -07:00
Massimiliano Culpo
f2fc4ee9af
Allow conditional possible values in variants (#29530)
Allow declaring possible values for variants with an associated condition. If the variant takes one of those values, the condition is imposed as a further constraint.

The idea of this PR is to implement part of the mechanisms needed for modeling [packages with multiple build-systems]( https://github.com/spack/seps/pull/3). After this PR the build-system directive can be implemented as:
```python
variant(
    'build-system',
    default='cmake',
    values=(
        'autotools',
        conditional('cmake', when='@X.Y:')
    ), 
    description='...',
)
```

Modifications:
- [x] Allow conditional possible values in variants
- [x] Add a unit-test for the feature
- [x] Add documentation
2022-04-04 17:37:57 -07:00
Nathan Hanford
88d8ca9b65
rewiring of spliced specs (#26873)
* tests for rewiring pure specs to spliced specs

* relocate text, binaries, and links

* using llnl.util.symlink for windows compat.

Note: This does not include CLI hooks for relocation.

Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-04 14:45:35 -07:00
Thomas Dickerson
ee505e6c69
Add support for racket packages (#27564)
- Add variants for various common build flags, including support for both versions of the Racket VM environment.
- Prevent `-j` flags to `make`, which has been known to cause problems with Racket builds.
- Prefer the minimal release to improve install times. Bells and whistles carry their own runtime dependencies and should be installed via `raco`. An enterprising user may even create a `RacketPackage` class to make spack aware of `raco` installed packages.
- Match the official version numbering scheme.
2022-04-04 10:32:25 +02:00
Brian Van Essen
29da99427e
"spack external find": also find library-only packages (#28005)
Update "spack external find --all" to also find library-only packages.
A Package can add a ".libraries" attribute, which is a list of regular
expressions to use to find libraries associated with the Package.
"spack external find --all" will search LD_LIBRARY_PATH for potential
libraries.

This PR adds examples for NCCL, RCCL, and hipblas packages. These
examples specify the suffix ".so" for the regular expressions used
to find libraries, so generally are only useful for detecting library
packages on Linux.
2022-04-01 13:30:10 -07:00
Peter Scheibel
a58fa289b9
Allow "spack install foo@git-hash" without terminal prompt (#29827)
Do not prompt user with checksum warning when using git commit hashes
as versions. Spack was incorrectly reporting this as a potential
problem: it would display a prompt asking the user whether they
want to proceed if Spack was running in a terminal, or it would
terminate the running instance of Spack if running as part of a
script.
2022-04-01 10:27:03 -07:00
John W. Parent
88b1bf751d
Windows Support: Fixup Perl build (#29711)
* Add pl2bat to PATH: Windows on Perl requires the script pl2bat.bat
  and Perl to be available to the installer via the PATH. The build
  and dependent environments of Perl on Windows have the install
  prefix bin added to the PATH.
* symlink with win32file module instead of using Executable to
  call mklink (mklink is a shell function and so is not accessible
  in this manner).
2022-03-31 11:47:11 -07:00
Ivo Steinbrecher
44263b7226
Allow use of "git:"-based URLs (#29765) 2022-03-31 10:14:18 -07:00
Scott Wittenburg
685e3d7ae9
spack ci: filter untouched pkgs from PR pipelines (#29697)
We've previously generated CI pipelines for PRs, and they rebuild any packages that don't have
a binary in an existing build cache.  The assumption we were making was that ALL prior merged
builds would be in cache, but due to the way we do security in the pipeline, they aren't. `develop`
pipelines can take a while to catch up with the latest PRs, and while it does that, there may be a
bunch of redundant builds on PRs that duplicate things being rebuilt on `develop`.  Until we can
do better caching of PR builds, we'll have this problem.

We can do better in PRs, though, by *only* rebuilding things in the CI environment that are actually
touched by the PR.  This change computes exactly what packages are changed by a PR branch and
*only* includes those packages' dependents and dependencies in the generated pipeline.  Other
as-yet unbuilt packages are pruned from CI for the PR.

For `develop` pipelines, we still want to build everything to ensure that the stack works, and to ensure
that `develop` catches up with PRs. This is especially true since we do not do rebuilds for *every* commit
on `develop` -- just the most recent one after each `develop` pipeline finishes.  Since we skip around,
we may end up missing builds unless we ensure that we rebuild everything.

We differentiate between `develop` and PR pipelines in `.gitlab-ci.yml` by setting 
`SPACK_PRUNE_UNTOUCHED` for PRs. `develop` will still have the old behavior.

- [x] Add `SPACK_PRUNE_UNTOUCHED` variable to `spack ci`
- [x] Refactor `spack pkg` command by moving historical package checking logic to `spack.repo`
- [x] Implement pruning logic in `spack ci` to remove untouched packages
- [x] add tests
2022-03-30 17:17:29 -07:00
Adam J. Stewart
f97be99f2a
Mixed compilers: drop warning message to debug level (#29748) 2022-03-30 15:18:37 -06:00
Jörg Behrmann
4571f4c994
Add completion of locals to spack python (#29702) 2022-03-30 14:38:39 -06:00
Greg Becker
58a32b04d9
patch cache: fix bug finding inherited packages (#29574) 2022-03-30 14:19:52 +02:00
Harmen Stoppels
9516fa9447
cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH (#29703)
* cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH

Spack has a heuristic to add rpaths for packages it knows are required,
but it's really a heuristic, and it does not work when the dependencies
put their libraries in a different folder than `<prefix>/lib{64,}`.

CMake patches binaries after install with the "install rpaths", which by
default are provided by Spack and its heuristic through
`CMAKE_INSTALL_RPATH`.

CMake however knows better what libraries are effectively being linked
to, and has an option to include those in the install rpath too, through
`CMAKE_INSTALL_RPATH_USE_LINK_PATH`.

These two CMake options are complementary, repeated rpaths seem to be
filtered, and the "use link path" paths are appended to Spack's
heuristic "install rpath".

So, it seems like a good idea to enable "use link path" by default, so
that:
- `dlopen` by library name uses Spack's heuristic search paths
- linked libraries in non-standard locations within a prefix get an
rpath thanks to CMake.

* docs
2022-03-29 12:24:10 -04:00
Tamara Dahlgren
fd055d4678
spack info: make sections optional; add build/stand-alone test information (#22097)
Add output of build- and install-time tests to info command

Enable dependencies, variants, and versions by default (i.e., provide --no* 
options; add gcc to test_info_fields to increase coverage for c_names->v_names
2022-03-28 22:15:38 +00:00
John W. Parent
1c2fff2cb7
Add ssh to filter of valid urls (#29749) 2022-03-28 14:13:55 -07:00
Massimiliano Culpo
2856b29740
Removed unneeded conflicts in CudaPackage and RocmPackage (#29699)
Since the variant is conditional there's no need to add
conflicts too.
2022-03-28 10:58:56 -06:00
百地 希留耶
cd00eba9d6
autotools doc: add missing right quote (#29734) 2022-03-27 22:42:29 +00:00
Massimiliano Culpo
048a0de35b
Use the appropriate function to remove files in the stage directory (#29690)
We shouldn't be using "remove_linked_tree" to remove the lock file,
since that function expects to receive a directory path as an
argument.

Also, as a further measure to avoid regression, this commit restores
the "ignore_errors=True" argument on linux and adds a unit test
checking that "remove_linked_tree" doesn't change file permissions
as a side effect of a failure to remove.
2022-03-25 08:39:09 +01:00
Harmen Stoppels
e22fbdb6b8
environment.py: ensure view dir does not exist (#29641) 2022-03-24 15:42:33 -06:00
Harmen Stoppels
1b55057f36
add_files_to_view: flip incorrect default (#29700) 2022-03-24 11:59:03 -06:00
Harmen Stoppels
f8224f284c
environment.py: concrete root specs to view (#29634)
Slight simplification of the code: just pass the list of concrete root
specs to the view
2022-03-24 14:59:24 +01:00
Harmen Stoppels
1a728c98ff
fix --reuse w/ upstreams in env (#29680) 2022-03-24 13:25:00 +00:00
Harmen Stoppels
59e522e815
environment views: single pass view generation (#29443)
Reduces the number of stat calls to a bare minimum:
- Single pass over src prefixes
- Handle projection clashes in memory

Symlinked directories in the src prefixes are now conditionally
transformed into directories with symlinks in the dst dir. Notably
`intel-mkl`, `cuda` and `qt` has top-level symlinked directories that
previously resulted in empty directories in the view. We now avoid
cycles and possible exponential blowup by only expanding symlinks that:
- point to dirs deeper in the folder structure;
- are a fixed depth of 2.
2022-03-24 03:54:33 -06:00
Harmen Stoppels
011a8b3f3e
environment.py: clean up broken views on failure (#29643)
When view creation fails for some reason, remove it, so that the next
time around it can start from scratch.
2022-03-24 09:04:42 +01:00
Harmen Stoppels
c300b92047
environment: be more defensive when deleting roots for old views (#29636)
Currently `old_root` is computed by reading the symlink at `self.root`.
We should be more defensive in removing it by checking that it is in the
same directory as the new root. Otherwise, in the worst case, when
someone runs `spack env create --with-view=./view -d .` and `view`
already exists and is a symlink to `/`, Spack effectively runs `rm -rf /`.
2022-03-23 08:54:55 -06:00
Harmen Stoppels
773da7ceba
python: drop dependency on file for script check (#29513)
`file` was used to detect Python scripts with shebangs, so that the interpreter could be changed from <python prefix> to <view path>. With this change, we detect shebangs using Python instead, so that `file` is no longer required.
2022-03-23 10:31:23 +01:00
Adam J. Stewart
5df10c04cd
Use stable URLs and ?full_index=1 for all github patches (#29239)
The number of commit characters in patch files fetched from GitHub can change,
so we should use `full_index=1` to enforce full commit hashes (and a stable
patch `sha256`).

Similarly, URLs for branches like `master` don't give us stable patch files,
because branches are moving targets. Use specific tags or commits for those.

- [x] update all github patch URLs to use `full_index=1`
- [x] don't use `master` or other branches for patches
- [x] add an audit check and a test for `?full_index=1`

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-23 08:50:00 +01:00
Massimiliano Culpo
8f89932aad
Remove known issues from documentation (#29664)
Known issues reports only 2 issues, among the bugs reported on GitHub.
One of the two is also outdated, since the issue has been solved
with the new concretizer. Thus, this commit removes the section.
2022-03-23 08:06:49 +01:00
Harmen Stoppels
80195bd1ed
sbang.py: single lstat (#29670) 2022-03-22 23:12:06 +01:00
Greg Becker
dba7a03daa
config add: fix parsing of validator error to infer type from oneOf (#29475) 2022-03-21 16:55:02 +01:00
Harmen Stoppels
8f5b9a89fb
major.minor.micro.dev0 Spack version (#25267)
When you install Spack from a tarball, it will always show an exact
version for Spack itself, even when you don't download a tagged commit:

```
$ wget -q https://github.com/spack/spack/archive/refs/heads/develop.tar.gz
$ tar -xf develop.tar.gz
$ ./spack-develop/bin/spack --version
0.16.2
```

This PR sets the Spack version to `0.18.0.dev0` on develop, following [PEP440](https://github.com/spack/spack/pull/25267#issuecomment-896340234) as
suggested by Adam Stewart.

```
spack (fix/set-dev-version)$ spack --version
0.18.0.dev0 (git 0.17.1-1526-e270464ae0)
spack (fix/set-dev-version)$ mv .git .git_
spack $ spack --version
0.18.0.dev0
```

- [x] Update the release guide
- [x] Add __version__ to spack's __init__.py
- [x] Use PEP 440 canonical version strings
- [x] Make spack --version output [actual version] (git version)

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-20 22:37:55 -06:00
Doug Jacobsen
9b0d5cbabf
Enable GCS urls as valid (#29592)
* Add tests to ensure google cloud storage urls work as mirrors

This commit adds two tests to track that GCS buckets can work as
mirrors, and can be parsed as valid URLs.

Currently, gs:// format URLs are not correctly parsed.

* Fix URL parsing for GCS buckets

This commit adds GCS bucket URLs as valid URLs.
2022-03-20 01:13:00 -07:00
Peter Scheibel
26552533be
We were doing a check in a metaclass to ensure that Package classes weren't defining a function themselves; this check failed (erroneously) if that Package class subclassed another package class (because the check examined all superclasses and thought the definition we automatically add was an offender) (#29569) 2022-03-19 21:37:40 -07:00
Nils Vu
bfb6873ce3
Revert "Add command for reading a json-based DB description (#24894)" (#29603)
This reverts commit 531b1c5c3d.
2022-03-19 16:30:46 -06:00
Tom Scogland
3ffe9a27b2
allow ci reproduce to work when spack is cloned to a directory not named spack (#29518) 2022-03-18 20:22:28 -06:00