Commit graph

5419 commits

Author SHA1 Message Date
Zack Galbreath
dec3e31e60
spack ci: remove relate-CDash-builds functionality (#29950)
gitlab ci: Remove code for relating CDash builds

Relating CDash builds to their dependencies was a seldom used feature.  Removing
it will make it easier for us to reorganize our CDash projects & build groups in the 
future by eliminating the needs to keep track of CDash build ids in our binary mirrors.
2022-04-14 10:42:30 -06:00
Peter Scheibel
89f6db21f1
Ad-hoc Git commit versions: support submodules (#30037)
* Allow packages to add a 'submodules' property that determines when ad-hoc Git-commit-based versions should initialize submodules

* add support for ad-hoc git-commit-based versions to instantiate submodules if the associated package has a 'submodules' property and it indicates this should happen for the associated spec

* allow Package-level submodule request to influence all explicitly-defined version() in the Package

* skip test on windows which fails because of long paths
2022-04-13 20:05:14 -07:00
Massimiliano Culpo
c846b5149d
Add support for Python 3.10 (#29581)
* Add support for Python 3.10

* Update unit-tests to use 3.10

* Update Getting started section of the docs

* Update bootstrap action
2022-04-13 14:32:23 -07:00
Nathan Hanford
a3520d14bd
Splice differing virtual packages (#27919)
Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-12 13:31:39 -07:00
psakievich
7d534f38d6
Don't allow replacement of root develop specs with --reuse (#28605)
* Fix to concretize.lp

do not allow dev specs to be reused

Co-authored-by: Gregory Becker <becker33@llnl.gov>
2022-04-12 10:37:24 -07:00
Peter Scheibel
2aec5b65f3
Git commit versions bugfix: Environments and Concretization (#29717)
Spack added support in #24639 for ad-hoc Git-commit-hash-based
versions: A user can install a package x@hash, where X is a package
that stores its source code in a Git repository, and the hash refers
to a commit in that repository which is not recorded as an explicit
version in the package.py file for X.

A couple issues were found relating to this:

* If an environment defines an alternative package repo (i.e. with
  repos.yaml), and spack.yaml contains user Specs with ad-hoc
  Git-commit-hash-based versions for packages in that repo,
  then as part of retrieving the data needed for version comparisons
  it will attempt to retrieve the package before the environment's
  configuration is instantiated.
* The bookkeeping information added to compare ad-hoc git versions was
  being stripped from Specs during concretization (such that user
  Specs which succeeded before concretizing would then fail after)

This addresses the issues:

* The first issue is resolved by deferring access to the associated
  Package until the versions are actually compared to one another.
* The second issue is resolved by ensuring that the Git bookkeeping
  information is explicitly applied to Specs after they are concretized.

This also:

* Resolves an ambiguity in the mock_git_version_info fixture used to
  create a tree of Git commits and provide a list where each index
  maps to a known commit.
* Isolates the cache used for Git repositories in tests using the
  mock_git_version_info fixture
* Adds a TODO which points out that if the remote Git repository
  overwrites tags, that Spack will then fail when using
  ad-hoc Git-commit-hash-based versions
2022-04-12 09:58:26 -07:00
Doug Jacobsen
a9b4f33f23
Update gpg publish to work with mirror arguments (#28740)
This commit updates the `gpg publish` command to work with the mirror
arguments, when trying to push keys to a mirror.

- [x] update `gpg publish command
- [x] add test for publishing GPG keys and rebuilding the key index within a mirror
2022-04-11 14:48:08 -07:00
Sergey Kosukhin
6e5cba7b82
cray platform: unload cray-mpich (#29898) 2022-04-11 14:34:09 -07:00
Seth R. Johnson
1ea05cd456
macos: fewer calls to sw_vers (#29997)
In a typical call to spack, the OperatingSystem gets instantiated
multiple times. For macOS, each one requires a call to `sw_vers`, which
is done through the Executable helper class. Memoizing
reduces the call count from "spac spec" from three to one.
2022-04-11 09:31:24 +02:00
Massimiliano Culpo
23e85f4086
Environments: unify the spec objects on first concretization (#29948)
Currently environments are indexed by build hashes. When looking into this bug I noticed there is a disconnect between environments that are concretized in memory for the first time and environments that are read from a `spack.lock`. The issue is that specs read from a `spack.lock` don't have a full hash, since they are indexed by a build hash which is strictly coarser. They are also marked "final" as they are read from a file, so we can't compute additional hashes. 

This bugfix PR makes "first concretization" equivalent to re-reading the specs from a corresponding `spack.lock`, and doing so unveiled a few tests were we were making wrong assumptions and relying on the fact that a `spack.lock` file was not there already.

* Add unit test
* Modify mpich to trigger jobs in pipelines
* Fix two failing unit tests
* Fix another full_hash vs. build_hash mismatch in tests
2022-04-08 15:26:17 -06:00
Harmen Stoppels
99083f1706
Deprecate top-level module config (#28659)
* Ignore top-level module config; add auto-update

In Spack 0.17 we got module sets (modules:[name]:[prop]), and for
backwards compat modules:[prop] was short for modules:default:[prop].

But this makes it awkward to define default config for the "default"
module set.

Since 0.17 is branched off, we can now deprecate top-level module config
(that is, just ignore it with a warning).

This PR does that, and it implements `spack config update modules` to
make upgrading easy (we should have added that to 0.17 already...)

It also removes references to  `dotkit` stuff which was already
deprecated in 0.13 and could have been removed in 0.14.

Prefix inspections are the only exception, since the top-level prefix inspections
used for `spack load` and `spack env activate`.
2022-04-08 19:00:35 +00:00
Greg Becker
79ba0c50c1
concretize.lp: enforce target compatibility through DAG (#29694)
Spack currently allows dependencies to be concretized for an 
architecture incompatible with the root. This commit adds rules
to make this situation impossible by design.
2022-04-08 11:01:04 +02:00
Massimiliano Culpo
ff04d1bfc1
Use the non-deprecated MetaPathFinder interface (#29745)
* Extract the MetaPathFinder and Loaders for packages in their own classes

https://peps.python.org/pep-0451/

Currently, RepoPath and Repo implement the (deprecated) interface of
MetaPathFinder (find_module) and of Loader (load_module). This commit
extracts both of them and places the code in their own classes.

The MetaPathFinder interface is updated to contain both the deprecated
"find_module" (for Python 2.7 support) and the recommended "find_spec".
Update of the Loader interface is deferred at a subsequent commit.

* Move the lines to be prepended inside "RepoLoader"

Also adjust the naming of a few variables too

* Remove spack.util.imp, since code is only used in spack.repo

* Remove support from loading Python modules Python > 3 but < 3.5

* Remove `Repo._create_namespace`

This function was interacting badly with the MetaPathFinder
and causing issues with "normal" imports. Removing the
function allows to do things like:
```python
import spack.pkg.builtin.mpich
cls = spack.pkg.builtin.mpich.Mpich
```

* Remove code needed to trigger the Singleton evaluation

The finder is coded in a way to trigger the Singleton,
so we don't need external code now that we register it
at module level into `sys.meta_path`.

* Add unit tests
2022-04-07 15:58:20 -07:00
Scott Wittenburg
b2eda32e55
ci: clean up env between tests with working_dir (#29807) 2022-04-07 17:11:58 +02:00
Scott Wittenburg
ff33978b0d
ci: do not prune specs when stacks are touched (#29825) 2022-04-07 08:55:40 -06:00
Harmen Stoppels
43577beb9c
autotools.py: pic flags for %nvhpc (#29920) 2022-04-07 14:52:47 +02:00
Harmen Stoppels
beff697cc0
web.py: set User-Agent (#29919)
Some servers require `User-Agent` to be set, and otherwise error with
access denied. One such example is mpich.

To fix this, set `User-Agent: Spackbot/[version]` as a header.
Apparently by convention, it should include the word `bot`.
2022-04-06 15:59:20 +02:00
Harmen Stoppels
558f7f007e
link_tree.py: format conflict error message (#29870)
Show each `[src a] and [src b] both project to [dst]` on a separate
line.
2022-04-06 07:27:02 +02:00
Erik Schnetter
69b3a88fa3
Bugfix: CVS fetching (#29793)
#27021 broke fetching for CVS-based packages because:

- The mirror logic was using URL parsing to extract a path from the
  CVS repository location
- #27021 added sanity checks to enforce that strings passed to the
  URL parser were actually URLs

This replaces the call to "url_util.parse" with logic that is
customized for CVS. This implies that VCSFetchStrategy should
rename the "url_attr" attribute to something more generic, but
that should be handled separately.
2022-04-05 18:45:45 -07:00
Massimiliano Culpo
f2fc4ee9af
Allow conditional possible values in variants (#29530)
Allow declaring possible values for variants with an associated condition. If the variant takes one of those values, the condition is imposed as a further constraint.

The idea of this PR is to implement part of the mechanisms needed for modeling [packages with multiple build-systems]( https://github.com/spack/seps/pull/3). After this PR the build-system directive can be implemented as:
```python
variant(
    'build-system',
    default='cmake',
    values=(
        'autotools',
        conditional('cmake', when='@X.Y:')
    ), 
    description='...',
)
```

Modifications:
- [x] Allow conditional possible values in variants
- [x] Add a unit-test for the feature
- [x] Add documentation
2022-04-04 17:37:57 -07:00
Nathan Hanford
88d8ca9b65
rewiring of spliced specs (#26873)
* tests for rewiring pure specs to spliced specs

* relocate text, binaries, and links

* using llnl.util.symlink for windows compat.

Note: This does not include CLI hooks for relocation.

Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-04 14:45:35 -07:00
Thomas Dickerson
ee505e6c69
Add support for racket packages (#27564)
- Add variants for various common build flags, including support for both versions of the Racket VM environment.
- Prevent `-j` flags to `make`, which has been known to cause problems with Racket builds.
- Prefer the minimal release to improve install times. Bells and whistles carry their own runtime dependencies and should be installed via `raco`. An enterprising user may even create a `RacketPackage` class to make spack aware of `raco` installed packages.
- Match the official version numbering scheme.
2022-04-04 10:32:25 +02:00
Brian Van Essen
29da99427e
"spack external find": also find library-only packages (#28005)
Update "spack external find --all" to also find library-only packages.
A Package can add a ".libraries" attribute, which is a list of regular
expressions to use to find libraries associated with the Package.
"spack external find --all" will search LD_LIBRARY_PATH for potential
libraries.

This PR adds examples for NCCL, RCCL, and hipblas packages. These
examples specify the suffix ".so" for the regular expressions used
to find libraries, so generally are only useful for detecting library
packages on Linux.
2022-04-01 13:30:10 -07:00
Peter Scheibel
a58fa289b9
Allow "spack install foo@git-hash" without terminal prompt (#29827)
Do not prompt user with checksum warning when using git commit hashes
as versions. Spack was incorrectly reporting this as a potential
problem: it would display a prompt asking the user whether they
want to proceed if Spack was running in a terminal, or it would
terminate the running instance of Spack if running as part of a
script.
2022-04-01 10:27:03 -07:00
John W. Parent
88b1bf751d
Windows Support: Fixup Perl build (#29711)
* Add pl2bat to PATH: Windows on Perl requires the script pl2bat.bat
  and Perl to be available to the installer via the PATH. The build
  and dependent environments of Perl on Windows have the install
  prefix bin added to the PATH.
* symlink with win32file module instead of using Executable to
  call mklink (mklink is a shell function and so is not accessible
  in this manner).
2022-03-31 11:47:11 -07:00
Ivo Steinbrecher
44263b7226
Allow use of "git:"-based URLs (#29765) 2022-03-31 10:14:18 -07:00
Scott Wittenburg
685e3d7ae9
spack ci: filter untouched pkgs from PR pipelines (#29697)
We've previously generated CI pipelines for PRs, and they rebuild any packages that don't have
a binary in an existing build cache.  The assumption we were making was that ALL prior merged
builds would be in cache, but due to the way we do security in the pipeline, they aren't. `develop`
pipelines can take a while to catch up with the latest PRs, and while it does that, there may be a
bunch of redundant builds on PRs that duplicate things being rebuilt on `develop`.  Until we can
do better caching of PR builds, we'll have this problem.

We can do better in PRs, though, by *only* rebuilding things in the CI environment that are actually
touched by the PR.  This change computes exactly what packages are changed by a PR branch and
*only* includes those packages' dependents and dependencies in the generated pipeline.  Other
as-yet unbuilt packages are pruned from CI for the PR.

For `develop` pipelines, we still want to build everything to ensure that the stack works, and to ensure
that `develop` catches up with PRs. This is especially true since we do not do rebuilds for *every* commit
on `develop` -- just the most recent one after each `develop` pipeline finishes.  Since we skip around,
we may end up missing builds unless we ensure that we rebuild everything.

We differentiate between `develop` and PR pipelines in `.gitlab-ci.yml` by setting 
`SPACK_PRUNE_UNTOUCHED` for PRs. `develop` will still have the old behavior.

- [x] Add `SPACK_PRUNE_UNTOUCHED` variable to `spack ci`
- [x] Refactor `spack pkg` command by moving historical package checking logic to `spack.repo`
- [x] Implement pruning logic in `spack ci` to remove untouched packages
- [x] add tests
2022-03-30 17:17:29 -07:00
Adam J. Stewart
f97be99f2a
Mixed compilers: drop warning message to debug level (#29748) 2022-03-30 15:18:37 -06:00
Jörg Behrmann
4571f4c994
Add completion of locals to spack python (#29702) 2022-03-30 14:38:39 -06:00
Greg Becker
58a32b04d9
patch cache: fix bug finding inherited packages (#29574) 2022-03-30 14:19:52 +02:00
Harmen Stoppels
9516fa9447
cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH (#29703)
* cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH

Spack has a heuristic to add rpaths for packages it knows are required,
but it's really a heuristic, and it does not work when the dependencies
put their libraries in a different folder than `<prefix>/lib{64,}`.

CMake patches binaries after install with the "install rpaths", which by
default are provided by Spack and its heuristic through
`CMAKE_INSTALL_RPATH`.

CMake however knows better what libraries are effectively being linked
to, and has an option to include those in the install rpath too, through
`CMAKE_INSTALL_RPATH_USE_LINK_PATH`.

These two CMake options are complementary, repeated rpaths seem to be
filtered, and the "use link path" paths are appended to Spack's
heuristic "install rpath".

So, it seems like a good idea to enable "use link path" by default, so
that:
- `dlopen` by library name uses Spack's heuristic search paths
- linked libraries in non-standard locations within a prefix get an
rpath thanks to CMake.

* docs
2022-03-29 12:24:10 -04:00
Tamara Dahlgren
fd055d4678
spack info: make sections optional; add build/stand-alone test information (#22097)
Add output of build- and install-time tests to info command

Enable dependencies, variants, and versions by default (i.e., provide --no* 
options; add gcc to test_info_fields to increase coverage for c_names->v_names
2022-03-28 22:15:38 +00:00
John W. Parent
1c2fff2cb7
Add ssh to filter of valid urls (#29749) 2022-03-28 14:13:55 -07:00
Massimiliano Culpo
2856b29740
Removed unneeded conflicts in CudaPackage and RocmPackage (#29699)
Since the variant is conditional there's no need to add
conflicts too.
2022-03-28 10:58:56 -06:00
百地 希留耶
cd00eba9d6
autotools doc: add missing right quote (#29734) 2022-03-27 22:42:29 +00:00
Massimiliano Culpo
048a0de35b
Use the appropriate function to remove files in the stage directory (#29690)
We shouldn't be using "remove_linked_tree" to remove the lock file,
since that function expects to receive a directory path as an
argument.

Also, as a further measure to avoid regression, this commit restores
the "ignore_errors=True" argument on linux and adds a unit test
checking that "remove_linked_tree" doesn't change file permissions
as a side effect of a failure to remove.
2022-03-25 08:39:09 +01:00
Harmen Stoppels
e22fbdb6b8
environment.py: ensure view dir does not exist (#29641) 2022-03-24 15:42:33 -06:00
Harmen Stoppels
1b55057f36
add_files_to_view: flip incorrect default (#29700) 2022-03-24 11:59:03 -06:00
Harmen Stoppels
f8224f284c
environment.py: concrete root specs to view (#29634)
Slight simplification of the code: just pass the list of concrete root
specs to the view
2022-03-24 14:59:24 +01:00
Harmen Stoppels
1a728c98ff
fix --reuse w/ upstreams in env (#29680) 2022-03-24 13:25:00 +00:00
Harmen Stoppels
59e522e815
environment views: single pass view generation (#29443)
Reduces the number of stat calls to a bare minimum:
- Single pass over src prefixes
- Handle projection clashes in memory

Symlinked directories in the src prefixes are now conditionally
transformed into directories with symlinks in the dst dir. Notably
`intel-mkl`, `cuda` and `qt` has top-level symlinked directories that
previously resulted in empty directories in the view. We now avoid
cycles and possible exponential blowup by only expanding symlinks that:
- point to dirs deeper in the folder structure;
- are a fixed depth of 2.
2022-03-24 03:54:33 -06:00
Harmen Stoppels
011a8b3f3e
environment.py: clean up broken views on failure (#29643)
When view creation fails for some reason, remove it, so that the next
time around it can start from scratch.
2022-03-24 09:04:42 +01:00
Harmen Stoppels
c300b92047
environment: be more defensive when deleting roots for old views (#29636)
Currently `old_root` is computed by reading the symlink at `self.root`.
We should be more defensive in removing it by checking that it is in the
same directory as the new root. Otherwise, in the worst case, when
someone runs `spack env create --with-view=./view -d .` and `view`
already exists and is a symlink to `/`, Spack effectively runs `rm -rf /`.
2022-03-23 08:54:55 -06:00
Harmen Stoppels
773da7ceba
python: drop dependency on file for script check (#29513)
`file` was used to detect Python scripts with shebangs, so that the interpreter could be changed from <python prefix> to <view path>. With this change, we detect shebangs using Python instead, so that `file` is no longer required.
2022-03-23 10:31:23 +01:00
Adam J. Stewart
5df10c04cd
Use stable URLs and ?full_index=1 for all github patches (#29239)
The number of commit characters in patch files fetched from GitHub can change,
so we should use `full_index=1` to enforce full commit hashes (and a stable
patch `sha256`).

Similarly, URLs for branches like `master` don't give us stable patch files,
because branches are moving targets. Use specific tags or commits for those.

- [x] update all github patch URLs to use `full_index=1`
- [x] don't use `master` or other branches for patches
- [x] add an audit check and a test for `?full_index=1`

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-23 08:50:00 +01:00
Massimiliano Culpo
8f89932aad
Remove known issues from documentation (#29664)
Known issues reports only 2 issues, among the bugs reported on GitHub.
One of the two is also outdated, since the issue has been solved
with the new concretizer. Thus, this commit removes the section.
2022-03-23 08:06:49 +01:00
Harmen Stoppels
80195bd1ed
sbang.py: single lstat (#29670) 2022-03-22 23:12:06 +01:00
Greg Becker
dba7a03daa
config add: fix parsing of validator error to infer type from oneOf (#29475) 2022-03-21 16:55:02 +01:00
Harmen Stoppels
8f5b9a89fb
major.minor.micro.dev0 Spack version (#25267)
When you install Spack from a tarball, it will always show an exact
version for Spack itself, even when you don't download a tagged commit:

```
$ wget -q https://github.com/spack/spack/archive/refs/heads/develop.tar.gz
$ tar -xf develop.tar.gz
$ ./spack-develop/bin/spack --version
0.16.2
```

This PR sets the Spack version to `0.18.0.dev0` on develop, following [PEP440](https://github.com/spack/spack/pull/25267#issuecomment-896340234) as
suggested by Adam Stewart.

```
spack (fix/set-dev-version)$ spack --version
0.18.0.dev0 (git 0.17.1-1526-e270464ae0)
spack (fix/set-dev-version)$ mv .git .git_
spack $ spack --version
0.18.0.dev0
```

- [x] Update the release guide
- [x] Add __version__ to spack's __init__.py
- [x] Use PEP 440 canonical version strings
- [x] Make spack --version output [actual version] (git version)

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-20 22:37:55 -06:00
Doug Jacobsen
9b0d5cbabf
Enable GCS urls as valid (#29592)
* Add tests to ensure google cloud storage urls work as mirrors

This commit adds two tests to track that GCS buckets can work as
mirrors, and can be parsed as valid URLs.

Currently, gs:// format URLs are not correctly parsed.

* Fix URL parsing for GCS buckets

This commit adds GCS bucket URLs as valid URLs.
2022-03-20 01:13:00 -07:00
Peter Scheibel
26552533be
We were doing a check in a metaclass to ensure that Package classes weren't defining a function themselves; this check failed (erroneously) if that Package class subclassed another package class (because the check examined all superclasses and thought the definition we automatically add was an offender) (#29569) 2022-03-19 21:37:40 -07:00
Nils Vu
bfb6873ce3
Revert "Add command for reading a json-based DB description (#24894)" (#29603)
This reverts commit 531b1c5c3d.
2022-03-19 16:30:46 -06:00
Tom Scogland
3ffe9a27b2
allow ci reproduce to work when spack is cloned to a directory not named spack (#29518) 2022-03-18 20:22:28 -06:00
Tom Scogland
9e01e17dc6
R versions (#29258)
* lower priority of package-provided urls

This change favors urls found in a scraped page over those provided by
the package from `url_for_version`.  In most cases this doesn't matter,
but R specifically returns known bad URLs in some cases, and the
fallback path for a failed fetch uses `fetch_remote_versions` to find a
substitute.  This fixes that problem.

fixes #29204

* consider what links actually exist in all cases

Checksum was only actually scraping when called with no versions.  It
now always scrapes and then selects URLs from the set of URLs known to
exist whenever possible.

fixes #25831

* bow to the wrath of flake8

* test-fetch urls from package, prefer if successful

* Update lib/spack/spack/package.py

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>

* reword as suggested

* re-enable mypy specific ignore and ignore pyflakes

* remove flake8 ignore from .flake8

* address review comments

* address comments

* add sneaky missing substitute

I missed this one because we call substitute on a URL that doesn't
contain a version component.  I'm not sure how that's supposed to work,
but apparently it's required by at least one mock package, so back in it
goes.

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
2022-03-18 18:42:17 -06:00
Peter Scheibel
531b1c5c3d
Add command for reading a json-based DB description (#24894)
Adds `spack external read-cray-manifest`, which reads a json file that describes a set of package DAGs. The parsed results are stored directly in the database. A user can see these installed specs with `spack find` (like any installed spec). The easiest way to use them right now as dependencies is to run `spack spec ... ^/hash-of-external-package`.

Changes include:

* `spack external read-cray-manifest --file <path/to/file>` will add all specs described in the file to Spack's installation DB and will also install described compilers to the compilers configuration (the expected format of the file is described in this PR as well including examples of the file)
* Database records now may include an "origin" (the command added in this PR registers the origin as "external-db"). In the future, it is assumed users may want to be able to treat installs registered with this command differently (e.g. they may want to uninstall all specs added with this command)
* Hash properties are now always preserved when copying specs if the source spec is concrete
  * I don't think the hashes of installed-and-concrete specs should change and this was the easiest way to handle that
  * also specs that are concrete preserve their `.normal` property when copied (external specs may mention compilers that are not registered, and without this change they would fail in `normalize` when calling `validate_or_raise`)
  * it might be this should only be the case if the spec was installed

- [x] Improve testing
- [x] Specifically mark DB records added with this command (so that users can do something like "uninstall all packages added with `spack read-external-db`)
  * This is now possible with `spack uninstall --all --origin=external-db` (this will remove all specs added from manifest files)
- [x] Strip variants that are listed in json entries but don't actually exist for the package

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-03-18 17:07:22 -07:00
Harmen Stoppels
62fcf407e2
environment: Don't print 'updating view at' when no specs (#29536) 2022-03-18 08:49:47 -04:00
Massimiliano Culpo
2fa495154e
Split the workflow section and remove outdated advices (#29344)
This PR removes a few outdated sections from the "Basics" part of the 
documentation. It also makes a few topic under the environment section
more prominent by removing an unneeded spack.yaml subsection and 
promoting everything under it.
2022-03-18 10:41:27 +01:00
Harmen Stoppels
ea13122f3c
Add deprecation warnings for activate/deactivate/view (#29430) 2022-03-17 14:52:56 -07:00
Zack Galbreath
5a72204d38
Remove references to features/windows-support branch (#29565) 2022-03-17 10:29:29 -07:00
John Parent
4aee27816e Windows Support: Testing Suite integration
Broaden support for execution of the test suite
on Windows.
General bug and review fixups
2022-03-17 09:01:01 -07:00
Peter Josef Scheibel
e63b4f752a Failed _write now track that the DB is inconsistent
Consistency is restored on next transaction
2022-03-17 09:01:01 -07:00
John Parent
cf1349ba35 "spack commands --update-completion" 2022-03-17 09:01:01 -07:00
John W. Parent
e4d4a5193f Path handling (#28402)
Consolidate Spack's internal filepath logic to a select
few places and refactor to consistent internal useage of
os.path utilities. Creates a prefix, and a series of utilities
in the path utility module that facilitate handling paths
in a platform agnostic manner.

Convert Windows paths to posix paths internally

Prefer posixpath.join instead of os.path.join

Updated util/ directory to account for Windows integration

Co-authored-by: Stephen Crowell <stephen.crowell@khq.kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>

Module template format for windows (#23041)
2022-03-17 09:01:01 -07:00
John Parent
df4129d395 Expand external find for Windows (#27588)
* Incorporate new search location

* Add external user option

* proper doc string

* Explicit commands in getting started

* raise during chgrp on Win

recover installer changes

Notate admin privleges

Windows phase install hooks

Find external python and install ninja (#23496)

Allow external find python to find windows python and spack install ninja

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
06aef626cb Update tests support for Windows
Fixup common tests

    * Remove requirement for Python 2.6
    * Skip new failing test

Windows: Update url util to handle Windows paths (#27959)

    * update url util to handle windows paths

    * Update tests to handle fixed url handling

    * canonicalize path only when the path type matches the host platform

    * Skip some url tests on Windows

Co-authored-by: Omar Padron <omar.padron@kitware.com>

Use threading.TIMEOUT_MAX when available (#24246)

This value was introduced in Python 3.2. Specifying a timeout greater than
this value will raise an OverflowError.

Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
John Parent
31daf0f2b6 Add Windows to platform and target changes
Add compiler hint to the root spec for Windows

Reporters on Windows (#26038)

Reporters use Jinja2 as the templating engine, and Jinja2 indexes
templates by Unix separators, even on Windows, so search using Unix paths
on all systems.

Support patching on win via git (#25871)

Handle GRP on windows
2022-03-17 09:01:01 -07:00
John Parent
3a994032f8 Spack on Windows package ports
CMake - Windows Bootstrap (#25825)

Remove hardcoded cmake compiler (#26410)

Revert breaking cmake changes
Ensure no autotools on Windows

Perl on Windows (#26612)

Python source build windows (#26313)

Reconfigure sysconf for Windows

Python2.6 compatibility

Fxixup new sbang tests for windows

Ruby support (#28287)

Add NASM support (#28319)

Add mock Ninja package for testing
2022-03-17 09:01:01 -07:00
John W. Parent
e65d3d14b4 Relocate spack_cmd and scripts from installer to root bin (#24651)
Relocate spack_cmd and scripts from installer
to root bin

Refactor documentation, installer, and launcher
to facilitate that change
2022-03-17 09:01:01 -07:00
Betsy McPhail
4a73bfc3b9 Use Python's zipfile, if available (#24556)
* Style fixes

* Use Python's zipfile, if available

The compression libs are optional in Python. Rely on python as a
first attempt then fall back to `unzip`
2022-03-17 09:01:01 -07:00
Betsy McPhail
f8782c46d7 Windows: VS and CMake support
Fix: Building packages with CMake is broken (#24241)

vsInstallPaths needs to be renamed vs_install_paths (#24297)
2022-03-17 09:01:01 -07:00
Jared Popelar
15ef85e161 Packaging of netlib-lapack for windows (#24993)
MSVC's internal CMake and Ninja now detected by spack external find and added to packages.yaml

Saving progress on packaging zlib for Windows

Fixing the shared CMake flag

* Loading Intel's ifx Fortran compiler into MSVC; if there are multiple
versions of MSVC installed and detected, ifx will only be placed into
the first block written in compilers.yaml. The version number of ifx can
be detected using MSVC's version flag (instead of /QV) by using
ignore_version_errors. This commit also provides support for detection
of Intel compilers in their own compiler block by adding ifx.exe to the
fc/f77_name blocks inside intel.py

* Giving CMake a Fortran compiler argument

* Adding patch file for removing duplicated mangling header for versions 3.9.1 and older; static and shared now successfully building on Windows

* Have netlib-lapack depend  on ninja@1.10

Co-authored-by: John R. Cary <cary@txcorp.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>

Making a default config.yaml for Windows

Small path length for build_stage

Provide more prerequisite details, mention default config.yaml

Killing an unnecessary setvars call

Replacing some lost changes, proofreading, updating windows-supported package list

Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
lou.lawrence@kitware.com
012758c179 Windows: Create installer and environment
* Add 'make-installer' command for Windows

* Add '--bat' arg to env activate, env deactivate and unload commands

* An equivalent script to setup-env on linux: spack_cmd.bat. This script
has a wrapper to evaluate cd, load/unload, env activate/deactivate.(#21734)

* Add spacktivate and config editor (#22049)

* spack_cmd: will find python and spack on its own. It preferentially
tries to use python on your PATH (#22414)

* Ignore Windows python installer if found (#23134)

* Bundle git in windows installer (#23597)

* Add Windows section to Getting Started document
(#23131), (#23295), (#24240)

Co-authored-by: Stephen Crowell <stephen.crowell@kitware.com>
Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>
Co-authored-by: Ben Cowan <benc@txcorp.com>

Update Installer CI

Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
a7101db39d MSVC Compiler Find and vcvars env integration (#22856)
Made the vcvars batch script location a member variable of the msvc compiler subclass, initialized from the compiler executable path.  Added a setup_custom_environment() method to the msvc subclass that sources the vcvars script, dumps the environment, and copies the relevant environment variables to the Spack environment.  Added class variables to the Windows OS and MSVC compiler subclasses to enable finding the compiler executables and determining their versions.
2022-03-17 09:01:01 -07:00
loulawrence
f587a9ce68 use pytest stdout/err capture (#22584)
* On windows, write to StringIO without redirection in test cases to avoid conflicting with logger
2022-03-17 09:01:01 -07:00
Ben Cowan
a0164793cb Features/windows install (#22204)
* Fixed path and uid issues.

* Added needed import statement; kluged .exe extension.

* Got package to build.  Some manual intervention necessary, including sourcing the MSVC setup script and having certain configuration parameters.

* Removed CMake executable suffix hack.
2022-03-17 09:01:01 -07:00
Betsy McPhail
d4d101f57e Allow 'spack external find' to find executables on the system path (#22091)
Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
fb0e91c534 Windows: Symlink support
To provide Windows-compatible functionality, spack code should use
llnl.util.symlink instead of os.symlink. On non-Windows platforms
and on Windows where supported, os.symlink will still be used.

Use junctions when symlinks aren't supported on Windows (#22583)

Support islink for junctions (#24182)

Windows: Update llnl/util/filesystem

* Use '/' as path separator on Windows.
* Recognizing that Windows paths start with '<Letter>:/' instead of '/'

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
a7de2fa380 Create rename utility function
os.rename() fails on Windows if file already exists.

Create getuid utility function (#21736)

On Windows, replace os.getuid with ctypes.windll.shell32.IsUserAnAdmin().

Tests: Use getuid util function

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
b60a0eea01 Workarounds for install errors on Windows (#21890)
1. Forwarding sys.stdin, e.g. use input_multiprocess_fd,
gives an error on Windows. Skipping for now

3.  subprocess_context needs to serialize for Windows, like it does
for Mac.

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Ben Cowan
81bc00d61f Adding basic Windows features (#21259)
* Snapshot of some MSVC infrastructure added during experiments a while ago.  Rebasing from spack/develop.

* Added platform and OS definitions for Windows.

* Updated Windows platform file to conform to new archspec use.

* Added Windows as a platform; introduced some debugging code.

* Added type annotations.

* Fixed copyright.

* Removed print statements.

* Ensure `spack arch` returns correctly on Windows (#21428)

* Correctly identify windows as 'windows-Windows10-AMD64'
2022-03-17 09:01:01 -07:00
Harmen Stoppels
55544950e2
PackageViewMixin: fix symlinks conflict issue (#29515)
`stat`'ing a file in the dst dir is the wrong thing to do, you should
`lstat` to capture broken symlinks.
2022-03-17 08:18:28 -06:00
Tom Scogland
94794d061a
optimize instantiation and comparison of versions (#29429)
Re-work the checks and comparisons around commit versions, when no
commit version is involved the overhead is now in the noise, where one
is the overhead is now constant rather than linear.
2022-03-15 18:53:28 +00:00
Sergey Kosukhin
f1bdbefd46
mpich: fix the detection of the package (#29284) 2022-03-14 16:33:56 +01:00
Harmen Stoppels
474493713b
compiler.py: early return in compiler_environment when no modules (#29441) 2022-03-14 11:27:49 +00:00
Massimiliano Culpo
a6eed4a7c7
Remove "setup_environment" and "setup_dependent_environment" (#29463)
fixes #29446

The new setup_*_environment functions have been falling back
to calling the old functions and warn the user since #11115.

This commit removes the fallback behavior and any use of:
- setup_environment
- setup_dependent_environment
in the codebase
2022-03-13 11:51:55 -04:00
Harmen Stoppels
8adc6b7e8e
module_cmd.py: use posix awk; fix stderr redirection (#29440)
Emulates `env -0` in a posix compliant way, avoiding a slow python process, speeds up setting up the build env when modules should load.
2022-03-11 18:29:11 +01:00
Harmen Stoppels
88fbba3e1e
intel-mkl: load compiler modules when querying compiler (#29439) 2022-03-10 14:14:17 -08:00
Massimiliano Culpo
2cd5c00923
Allow for multiple dependencies/dependents from the same package (#28673)
Change the internal representation of `Spec` to allow for multiple dependencies or
dependents stemming from the same package. This change permits to represent cases
which are frequent in cross compiled environments or to bootstrap compilers.

Modifications:
- [x] Substitute `DependencyMap` with `_EdgeMap`. The main differences are that the
      latter does not support direct item assignment and can be modified only through its
      API. It also provides a `select_by` method to query items.
- [x] Reworked a few public APIs of `Spec` to get list of dependencies or related edges.
- [x] Added unit tests to prevent regression on #11983 and prove the synthetic construction
      of specs with multiple deps from the same package. 

Since #22845 went in first, this PR reuses that format and thus it should not change hashes.
The same package may be present multiple times in the list of dependencies with different
associated specs (each with its own hash).
2022-03-10 11:53:45 -08:00
Harmen Stoppels
dc78f4c58a
environment.py: allow link:run (#29336)
* environment.py: allow link:run

Some users want minimal views, excluding run-type dependencies, since
those type of dependencies are covered by rpaths and the symlinked
libraries in the view aren't used anyways.

With this change, an environment like this:

```
spack:
  specs: ['py-flake8']
  view:
    default:
      root: view
      link: run
```

includes python packages and python, but no link type deps of python.
2022-03-09 12:35:26 -08:00
百地 希留耶
3270aa106b
Hide package repository count when redirecting stdout (#29402) 2022-03-09 17:07:07 +00:00
百地 希留耶
3370d3f57e
Fix tab completion erroring with spack unit-test (#29405) 2022-03-09 16:09:57 +01:00
Harmen Stoppels
3df90e3e33
ci.py: fix utf-8 issue with git log output (#29419) 2022-03-09 13:13:37 +00:00
Harmen Stoppels
a39a6c4ea7
version.py: set is_commit in constructor (#29369)
Speeds up comparison on `Version` by ~2.5x, e.g.

```python
In [1]: v = spack.version.Version('1.0.0'); w = spack.version.Version('1.0.2')

In [2]: %timeit v < w

1.47 µs ± 5.59 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

535 ns ± 1.75 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```
2022-03-08 10:50:48 +01:00
Todd Gamblin
d424d0e54e
find: deprecate spack find -bootstrap for spack -bootstrap find (#29152)
Since adding the `spack --bootstrap` argument we don't need `spack find --bootstrap`
anymore.  Deprecate it.
2022-03-05 21:41:18 -08:00
Harmen Stoppels
80a6eab2db Add a test 2022-03-04 18:13:34 +01:00
Harmen Stoppels
9199800bbd VersionRange.satisfies should test non-empty intersection 2022-03-04 18:13:34 +01:00
Massimiliano Culpo
b48bdc9e19
Fix importing Spack packages as Python modules (#29221)
fixes #29203

This PR fixes a subtle bug we have when importing
Spack packages as Python modules that can lead to
multiple module objects being created for the same
package.

It also fixes all the places in unit-tests where
"relying" on the old bug was crucial to have a new
"clean" state of the package class.
2022-03-04 08:42:27 +01:00
Doug Jacobsen
06fd0f892e
Revert GCS fetch strategy, to remove s3 interface (#28736)
This commit reverts the GCS fetch strategy to before commit:
d759612523

The previous commit added some s3 syntax to handle connections, but
added them into the GCS fetch strategy in a way that prevents GCS from
working anymore.
2022-03-03 16:34:33 -08:00
Greg Becker
e2b87ade06
rocmcc compiler: initial commit based on aocc and clang (#28575)
* rocmcc compiler: initial commit based on aocc and clang

Co-authored-by: luker <luke.roskop@hpe.com>
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
2022-03-03 14:34:22 -07:00
Michael Kuhn
49069e4f58
installer: Fix cosmetic problem with terminal title (#29070)
The status displayed in the terminal title could be wrong when doing
distributed builds. For instance, doing `spack install glib` in two
different terminals could lead to the current package being reported as
`40/29` due to the way Spack handles retrying locks.

Work around this by keeping track of the package IDs that were already
encountered to avoid counting packages twice.
2022-03-03 14:21:15 +01:00