Commit graph

5400 commits

Author SHA1 Message Date
Massimiliano Culpo
f2fc4ee9af
Allow conditional possible values in variants (#29530)
Allow declaring possible values for variants with an associated condition. If the variant takes one of those values, the condition is imposed as a further constraint.

The idea of this PR is to implement part of the mechanisms needed for modeling [packages with multiple build-systems]( https://github.com/spack/seps/pull/3). After this PR the build-system directive can be implemented as:
```python
variant(
    'build-system',
    default='cmake',
    values=(
        'autotools',
        conditional('cmake', when='@X.Y:')
    ), 
    description='...',
)
```

Modifications:
- [x] Allow conditional possible values in variants
- [x] Add a unit-test for the feature
- [x] Add documentation
2022-04-04 17:37:57 -07:00
Nathan Hanford
88d8ca9b65
rewiring of spliced specs (#26873)
* tests for rewiring pure specs to spliced specs

* relocate text, binaries, and links

* using llnl.util.symlink for windows compat.

Note: This does not include CLI hooks for relocation.

Co-authored-by: Nathan Hanford <hanford1@llnl.gov>
2022-04-04 14:45:35 -07:00
Thomas Dickerson
ee505e6c69
Add support for racket packages (#27564)
- Add variants for various common build flags, including support for both versions of the Racket VM environment.
- Prevent `-j` flags to `make`, which has been known to cause problems with Racket builds.
- Prefer the minimal release to improve install times. Bells and whistles carry their own runtime dependencies and should be installed via `raco`. An enterprising user may even create a `RacketPackage` class to make spack aware of `raco` installed packages.
- Match the official version numbering scheme.
2022-04-04 10:32:25 +02:00
Brian Van Essen
29da99427e
"spack external find": also find library-only packages (#28005)
Update "spack external find --all" to also find library-only packages.
A Package can add a ".libraries" attribute, which is a list of regular
expressions to use to find libraries associated with the Package.
"spack external find --all" will search LD_LIBRARY_PATH for potential
libraries.

This PR adds examples for NCCL, RCCL, and hipblas packages. These
examples specify the suffix ".so" for the regular expressions used
to find libraries, so generally are only useful for detecting library
packages on Linux.
2022-04-01 13:30:10 -07:00
Peter Scheibel
a58fa289b9
Allow "spack install foo@git-hash" without terminal prompt (#29827)
Do not prompt user with checksum warning when using git commit hashes
as versions. Spack was incorrectly reporting this as a potential
problem: it would display a prompt asking the user whether they
want to proceed if Spack was running in a terminal, or it would
terminate the running instance of Spack if running as part of a
script.
2022-04-01 10:27:03 -07:00
John W. Parent
88b1bf751d
Windows Support: Fixup Perl build (#29711)
* Add pl2bat to PATH: Windows on Perl requires the script pl2bat.bat
  and Perl to be available to the installer via the PATH. The build
  and dependent environments of Perl on Windows have the install
  prefix bin added to the PATH.
* symlink with win32file module instead of using Executable to
  call mklink (mklink is a shell function and so is not accessible
  in this manner).
2022-03-31 11:47:11 -07:00
Ivo Steinbrecher
44263b7226
Allow use of "git:"-based URLs (#29765) 2022-03-31 10:14:18 -07:00
Scott Wittenburg
685e3d7ae9
spack ci: filter untouched pkgs from PR pipelines (#29697)
We've previously generated CI pipelines for PRs, and they rebuild any packages that don't have
a binary in an existing build cache.  The assumption we were making was that ALL prior merged
builds would be in cache, but due to the way we do security in the pipeline, they aren't. `develop`
pipelines can take a while to catch up with the latest PRs, and while it does that, there may be a
bunch of redundant builds on PRs that duplicate things being rebuilt on `develop`.  Until we can
do better caching of PR builds, we'll have this problem.

We can do better in PRs, though, by *only* rebuilding things in the CI environment that are actually
touched by the PR.  This change computes exactly what packages are changed by a PR branch and
*only* includes those packages' dependents and dependencies in the generated pipeline.  Other
as-yet unbuilt packages are pruned from CI for the PR.

For `develop` pipelines, we still want to build everything to ensure that the stack works, and to ensure
that `develop` catches up with PRs. This is especially true since we do not do rebuilds for *every* commit
on `develop` -- just the most recent one after each `develop` pipeline finishes.  Since we skip around,
we may end up missing builds unless we ensure that we rebuild everything.

We differentiate between `develop` and PR pipelines in `.gitlab-ci.yml` by setting 
`SPACK_PRUNE_UNTOUCHED` for PRs. `develop` will still have the old behavior.

- [x] Add `SPACK_PRUNE_UNTOUCHED` variable to `spack ci`
- [x] Refactor `spack pkg` command by moving historical package checking logic to `spack.repo`
- [x] Implement pruning logic in `spack ci` to remove untouched packages
- [x] add tests
2022-03-30 17:17:29 -07:00
Adam J. Stewart
f97be99f2a
Mixed compilers: drop warning message to debug level (#29748) 2022-03-30 15:18:37 -06:00
Jörg Behrmann
4571f4c994
Add completion of locals to spack python (#29702) 2022-03-30 14:38:39 -06:00
Greg Becker
58a32b04d9
patch cache: fix bug finding inherited packages (#29574) 2022-03-30 14:19:52 +02:00
Harmen Stoppels
9516fa9447
cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH (#29703)
* cmake: use CMAKE_INSTALL_RPATH_USE_LINK_PATH

Spack has a heuristic to add rpaths for packages it knows are required,
but it's really a heuristic, and it does not work when the dependencies
put their libraries in a different folder than `<prefix>/lib{64,}`.

CMake patches binaries after install with the "install rpaths", which by
default are provided by Spack and its heuristic through
`CMAKE_INSTALL_RPATH`.

CMake however knows better what libraries are effectively being linked
to, and has an option to include those in the install rpath too, through
`CMAKE_INSTALL_RPATH_USE_LINK_PATH`.

These two CMake options are complementary, repeated rpaths seem to be
filtered, and the "use link path" paths are appended to Spack's
heuristic "install rpath".

So, it seems like a good idea to enable "use link path" by default, so
that:
- `dlopen` by library name uses Spack's heuristic search paths
- linked libraries in non-standard locations within a prefix get an
rpath thanks to CMake.

* docs
2022-03-29 12:24:10 -04:00
Tamara Dahlgren
fd055d4678
spack info: make sections optional; add build/stand-alone test information (#22097)
Add output of build- and install-time tests to info command

Enable dependencies, variants, and versions by default (i.e., provide --no* 
options; add gcc to test_info_fields to increase coverage for c_names->v_names
2022-03-28 22:15:38 +00:00
John W. Parent
1c2fff2cb7
Add ssh to filter of valid urls (#29749) 2022-03-28 14:13:55 -07:00
Massimiliano Culpo
2856b29740
Removed unneeded conflicts in CudaPackage and RocmPackage (#29699)
Since the variant is conditional there's no need to add
conflicts too.
2022-03-28 10:58:56 -06:00
百地 希留耶
cd00eba9d6
autotools doc: add missing right quote (#29734) 2022-03-27 22:42:29 +00:00
Massimiliano Culpo
048a0de35b
Use the appropriate function to remove files in the stage directory (#29690)
We shouldn't be using "remove_linked_tree" to remove the lock file,
since that function expects to receive a directory path as an
argument.

Also, as a further measure to avoid regression, this commit restores
the "ignore_errors=True" argument on linux and adds a unit test
checking that "remove_linked_tree" doesn't change file permissions
as a side effect of a failure to remove.
2022-03-25 08:39:09 +01:00
Harmen Stoppels
e22fbdb6b8
environment.py: ensure view dir does not exist (#29641) 2022-03-24 15:42:33 -06:00
Harmen Stoppels
1b55057f36
add_files_to_view: flip incorrect default (#29700) 2022-03-24 11:59:03 -06:00
Harmen Stoppels
f8224f284c
environment.py: concrete root specs to view (#29634)
Slight simplification of the code: just pass the list of concrete root
specs to the view
2022-03-24 14:59:24 +01:00
Harmen Stoppels
1a728c98ff
fix --reuse w/ upstreams in env (#29680) 2022-03-24 13:25:00 +00:00
Harmen Stoppels
59e522e815
environment views: single pass view generation (#29443)
Reduces the number of stat calls to a bare minimum:
- Single pass over src prefixes
- Handle projection clashes in memory

Symlinked directories in the src prefixes are now conditionally
transformed into directories with symlinks in the dst dir. Notably
`intel-mkl`, `cuda` and `qt` has top-level symlinked directories that
previously resulted in empty directories in the view. We now avoid
cycles and possible exponential blowup by only expanding symlinks that:
- point to dirs deeper in the folder structure;
- are a fixed depth of 2.
2022-03-24 03:54:33 -06:00
Harmen Stoppels
011a8b3f3e
environment.py: clean up broken views on failure (#29643)
When view creation fails for some reason, remove it, so that the next
time around it can start from scratch.
2022-03-24 09:04:42 +01:00
Harmen Stoppels
c300b92047
environment: be more defensive when deleting roots for old views (#29636)
Currently `old_root` is computed by reading the symlink at `self.root`.
We should be more defensive in removing it by checking that it is in the
same directory as the new root. Otherwise, in the worst case, when
someone runs `spack env create --with-view=./view -d .` and `view`
already exists and is a symlink to `/`, Spack effectively runs `rm -rf /`.
2022-03-23 08:54:55 -06:00
Harmen Stoppels
773da7ceba
python: drop dependency on file for script check (#29513)
`file` was used to detect Python scripts with shebangs, so that the interpreter could be changed from <python prefix> to <view path>. With this change, we detect shebangs using Python instead, so that `file` is no longer required.
2022-03-23 10:31:23 +01:00
Adam J. Stewart
5df10c04cd
Use stable URLs and ?full_index=1 for all github patches (#29239)
The number of commit characters in patch files fetched from GitHub can change,
so we should use `full_index=1` to enforce full commit hashes (and a stable
patch `sha256`).

Similarly, URLs for branches like `master` don't give us stable patch files,
because branches are moving targets. Use specific tags or commits for those.

- [x] update all github patch URLs to use `full_index=1`
- [x] don't use `master` or other branches for patches
- [x] add an audit check and a test for `?full_index=1`

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-23 08:50:00 +01:00
Massimiliano Culpo
8f89932aad
Remove known issues from documentation (#29664)
Known issues reports only 2 issues, among the bugs reported on GitHub.
One of the two is also outdated, since the issue has been solved
with the new concretizer. Thus, this commit removes the section.
2022-03-23 08:06:49 +01:00
Harmen Stoppels
80195bd1ed
sbang.py: single lstat (#29670) 2022-03-22 23:12:06 +01:00
Greg Becker
dba7a03daa
config add: fix parsing of validator error to infer type from oneOf (#29475) 2022-03-21 16:55:02 +01:00
Harmen Stoppels
8f5b9a89fb
major.minor.micro.dev0 Spack version (#25267)
When you install Spack from a tarball, it will always show an exact
version for Spack itself, even when you don't download a tagged commit:

```
$ wget -q https://github.com/spack/spack/archive/refs/heads/develop.tar.gz
$ tar -xf develop.tar.gz
$ ./spack-develop/bin/spack --version
0.16.2
```

This PR sets the Spack version to `0.18.0.dev0` on develop, following [PEP440](https://github.com/spack/spack/pull/25267#issuecomment-896340234) as
suggested by Adam Stewart.

```
spack (fix/set-dev-version)$ spack --version
0.18.0.dev0 (git 0.17.1-1526-e270464ae0)
spack (fix/set-dev-version)$ mv .git .git_
spack $ spack --version
0.18.0.dev0
```

- [x] Update the release guide
- [x] Add __version__ to spack's __init__.py
- [x] Use PEP 440 canonical version strings
- [x] Make spack --version output [actual version] (git version)

Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
2022-03-20 22:37:55 -06:00
Doug Jacobsen
9b0d5cbabf
Enable GCS urls as valid (#29592)
* Add tests to ensure google cloud storage urls work as mirrors

This commit adds two tests to track that GCS buckets can work as
mirrors, and can be parsed as valid URLs.

Currently, gs:// format URLs are not correctly parsed.

* Fix URL parsing for GCS buckets

This commit adds GCS bucket URLs as valid URLs.
2022-03-20 01:13:00 -07:00
Peter Scheibel
26552533be
We were doing a check in a metaclass to ensure that Package classes weren't defining a function themselves; this check failed (erroneously) if that Package class subclassed another package class (because the check examined all superclasses and thought the definition we automatically add was an offender) (#29569) 2022-03-19 21:37:40 -07:00
Nils Vu
bfb6873ce3
Revert "Add command for reading a json-based DB description (#24894)" (#29603)
This reverts commit 531b1c5c3d.
2022-03-19 16:30:46 -06:00
Tom Scogland
3ffe9a27b2
allow ci reproduce to work when spack is cloned to a directory not named spack (#29518) 2022-03-18 20:22:28 -06:00
Tom Scogland
9e01e17dc6
R versions (#29258)
* lower priority of package-provided urls

This change favors urls found in a scraped page over those provided by
the package from `url_for_version`.  In most cases this doesn't matter,
but R specifically returns known bad URLs in some cases, and the
fallback path for a failed fetch uses `fetch_remote_versions` to find a
substitute.  This fixes that problem.

fixes #29204

* consider what links actually exist in all cases

Checksum was only actually scraping when called with no versions.  It
now always scrapes and then selects URLs from the set of URLs known to
exist whenever possible.

fixes #25831

* bow to the wrath of flake8

* test-fetch urls from package, prefer if successful

* Update lib/spack/spack/package.py

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>

* reword as suggested

* re-enable mypy specific ignore and ignore pyflakes

* remove flake8 ignore from .flake8

* address review comments

* address comments

* add sneaky missing substitute

I missed this one because we call substitute on a URL that doesn't
contain a version component.  I'm not sure how that's supposed to work,
but apparently it's required by at least one mock package, so back in it
goes.

Co-authored-by: Seth R. Johnson <johnsonsr@ornl.gov>
2022-03-18 18:42:17 -06:00
Peter Scheibel
531b1c5c3d
Add command for reading a json-based DB description (#24894)
Adds `spack external read-cray-manifest`, which reads a json file that describes a set of package DAGs. The parsed results are stored directly in the database. A user can see these installed specs with `spack find` (like any installed spec). The easiest way to use them right now as dependencies is to run `spack spec ... ^/hash-of-external-package`.

Changes include:

* `spack external read-cray-manifest --file <path/to/file>` will add all specs described in the file to Spack's installation DB and will also install described compilers to the compilers configuration (the expected format of the file is described in this PR as well including examples of the file)
* Database records now may include an "origin" (the command added in this PR registers the origin as "external-db"). In the future, it is assumed users may want to be able to treat installs registered with this command differently (e.g. they may want to uninstall all specs added with this command)
* Hash properties are now always preserved when copying specs if the source spec is concrete
  * I don't think the hashes of installed-and-concrete specs should change and this was the easiest way to handle that
  * also specs that are concrete preserve their `.normal` property when copied (external specs may mention compilers that are not registered, and without this change they would fail in `normalize` when calling `validate_or_raise`)
  * it might be this should only be the case if the spec was installed

- [x] Improve testing
- [x] Specifically mark DB records added with this command (so that users can do something like "uninstall all packages added with `spack read-external-db`)
  * This is now possible with `spack uninstall --all --origin=external-db` (this will remove all specs added from manifest files)
- [x] Strip variants that are listed in json entries but don't actually exist for the package

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-03-18 17:07:22 -07:00
Harmen Stoppels
62fcf407e2
environment: Don't print 'updating view at' when no specs (#29536) 2022-03-18 08:49:47 -04:00
Massimiliano Culpo
2fa495154e
Split the workflow section and remove outdated advices (#29344)
This PR removes a few outdated sections from the "Basics" part of the 
documentation. It also makes a few topic under the environment section
more prominent by removing an unneeded spack.yaml subsection and 
promoting everything under it.
2022-03-18 10:41:27 +01:00
Harmen Stoppels
ea13122f3c
Add deprecation warnings for activate/deactivate/view (#29430) 2022-03-17 14:52:56 -07:00
Zack Galbreath
5a72204d38
Remove references to features/windows-support branch (#29565) 2022-03-17 10:29:29 -07:00
John Parent
4aee27816e Windows Support: Testing Suite integration
Broaden support for execution of the test suite
on Windows.
General bug and review fixups
2022-03-17 09:01:01 -07:00
Peter Josef Scheibel
e63b4f752a Failed _write now track that the DB is inconsistent
Consistency is restored on next transaction
2022-03-17 09:01:01 -07:00
John Parent
cf1349ba35 "spack commands --update-completion" 2022-03-17 09:01:01 -07:00
John W. Parent
e4d4a5193f Path handling (#28402)
Consolidate Spack's internal filepath logic to a select
few places and refactor to consistent internal useage of
os.path utilities. Creates a prefix, and a series of utilities
in the path utility module that facilitate handling paths
in a platform agnostic manner.

Convert Windows paths to posix paths internally

Prefer posixpath.join instead of os.path.join

Updated util/ directory to account for Windows integration

Co-authored-by: Stephen Crowell <stephen.crowell@khq.kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>

Module template format for windows (#23041)
2022-03-17 09:01:01 -07:00
John Parent
df4129d395 Expand external find for Windows (#27588)
* Incorporate new search location

* Add external user option

* proper doc string

* Explicit commands in getting started

* raise during chgrp on Win

recover installer changes

Notate admin privleges

Windows phase install hooks

Find external python and install ninja (#23496)

Allow external find python to find windows python and spack install ninja

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
06aef626cb Update tests support for Windows
Fixup common tests

    * Remove requirement for Python 2.6
    * Skip new failing test

Windows: Update url util to handle Windows paths (#27959)

    * update url util to handle windows paths

    * Update tests to handle fixed url handling

    * canonicalize path only when the path type matches the host platform

    * Skip some url tests on Windows

Co-authored-by: Omar Padron <omar.padron@kitware.com>

Use threading.TIMEOUT_MAX when available (#24246)

This value was introduced in Python 3.2. Specifying a timeout greater than
this value will raise an OverflowError.

Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
John Parent
31daf0f2b6 Add Windows to platform and target changes
Add compiler hint to the root spec for Windows

Reporters on Windows (#26038)

Reporters use Jinja2 as the templating engine, and Jinja2 indexes
templates by Unix separators, even on Windows, so search using Unix paths
on all systems.

Support patching on win via git (#25871)

Handle GRP on windows
2022-03-17 09:01:01 -07:00
John Parent
3a994032f8 Spack on Windows package ports
CMake - Windows Bootstrap (#25825)

Remove hardcoded cmake compiler (#26410)

Revert breaking cmake changes
Ensure no autotools on Windows

Perl on Windows (#26612)

Python source build windows (#26313)

Reconfigure sysconf for Windows

Python2.6 compatibility

Fxixup new sbang tests for windows

Ruby support (#28287)

Add NASM support (#28319)

Add mock Ninja package for testing
2022-03-17 09:01:01 -07:00
John W. Parent
e65d3d14b4 Relocate spack_cmd and scripts from installer to root bin (#24651)
Relocate spack_cmd and scripts from installer
to root bin

Refactor documentation, installer, and launcher
to facilitate that change
2022-03-17 09:01:01 -07:00
Betsy McPhail
4a73bfc3b9 Use Python's zipfile, if available (#24556)
* Style fixes

* Use Python's zipfile, if available

The compression libs are optional in Python. Rely on python as a
first attempt then fall back to `unzip`
2022-03-17 09:01:01 -07:00
Betsy McPhail
f8782c46d7 Windows: VS and CMake support
Fix: Building packages with CMake is broken (#24241)

vsInstallPaths needs to be renamed vs_install_paths (#24297)
2022-03-17 09:01:01 -07:00
Jared Popelar
15ef85e161 Packaging of netlib-lapack for windows (#24993)
MSVC's internal CMake and Ninja now detected by spack external find and added to packages.yaml

Saving progress on packaging zlib for Windows

Fixing the shared CMake flag

* Loading Intel's ifx Fortran compiler into MSVC; if there are multiple
versions of MSVC installed and detected, ifx will only be placed into
the first block written in compilers.yaml. The version number of ifx can
be detected using MSVC's version flag (instead of /QV) by using
ignore_version_errors. This commit also provides support for detection
of Intel compilers in their own compiler block by adding ifx.exe to the
fc/f77_name blocks inside intel.py

* Giving CMake a Fortran compiler argument

* Adding patch file for removing duplicated mangling header for versions 3.9.1 and older; static and shared now successfully building on Windows

* Have netlib-lapack depend  on ninja@1.10

Co-authored-by: John R. Cary <cary@txcorp.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>

Making a default config.yaml for Windows

Small path length for build_stage

Provide more prerequisite details, mention default config.yaml

Killing an unnecessary setvars call

Replacing some lost changes, proofreading, updating windows-supported package list

Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
lou.lawrence@kitware.com
012758c179 Windows: Create installer and environment
* Add 'make-installer' command for Windows

* Add '--bat' arg to env activate, env deactivate and unload commands

* An equivalent script to setup-env on linux: spack_cmd.bat. This script
has a wrapper to evaluate cd, load/unload, env activate/deactivate.(#21734)

* Add spacktivate and config editor (#22049)

* spack_cmd: will find python and spack on its own. It preferentially
tries to use python on your PATH (#22414)

* Ignore Windows python installer if found (#23134)

* Bundle git in windows installer (#23597)

* Add Windows section to Getting Started document
(#23131), (#23295), (#24240)

Co-authored-by: Stephen Crowell <stephen.crowell@kitware.com>
Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
Co-authored-by: Jared Popelar <jpopelar@txcorp.com>
Co-authored-by: Ben Cowan <benc@txcorp.com>

Update Installer CI

Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
a7101db39d MSVC Compiler Find and vcvars env integration (#22856)
Made the vcvars batch script location a member variable of the msvc compiler subclass, initialized from the compiler executable path.  Added a setup_custom_environment() method to the msvc subclass that sources the vcvars script, dumps the environment, and copies the relevant environment variables to the Spack environment.  Added class variables to the Windows OS and MSVC compiler subclasses to enable finding the compiler executables and determining their versions.
2022-03-17 09:01:01 -07:00
loulawrence
f587a9ce68 use pytest stdout/err capture (#22584)
* On windows, write to StringIO without redirection in test cases to avoid conflicting with logger
2022-03-17 09:01:01 -07:00
Ben Cowan
a0164793cb Features/windows install (#22204)
* Fixed path and uid issues.

* Added needed import statement; kluged .exe extension.

* Got package to build.  Some manual intervention necessary, including sourcing the MSVC setup script and having certain configuration parameters.

* Removed CMake executable suffix hack.
2022-03-17 09:01:01 -07:00
Betsy McPhail
d4d101f57e Allow 'spack external find' to find executables on the system path (#22091)
Co-authored-by: Lou Lawrence <lou.lawrence@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
fb0e91c534 Windows: Symlink support
To provide Windows-compatible functionality, spack code should use
llnl.util.symlink instead of os.symlink. On non-Windows platforms
and on Windows where supported, os.symlink will still be used.

Use junctions when symlinks aren't supported on Windows (#22583)

Support islink for junctions (#24182)

Windows: Update llnl/util/filesystem

* Use '/' as path separator on Windows.
* Recognizing that Windows paths start with '<Letter>:/' instead of '/'

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
a7de2fa380 Create rename utility function
os.rename() fails on Windows if file already exists.

Create getuid utility function (#21736)

On Windows, replace os.getuid with ctypes.windll.shell32.IsUserAnAdmin().

Tests: Use getuid util function

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: Betsy McPhail <betsy.mcphail@kitware.com>
2022-03-17 09:01:01 -07:00
Betsy McPhail
b60a0eea01 Workarounds for install errors on Windows (#21890)
1. Forwarding sys.stdin, e.g. use input_multiprocess_fd,
gives an error on Windows. Skipping for now

3.  subprocess_context needs to serialize for Windows, like it does
for Mac.

Co-authored-by: lou.lawrence@kitware.com <lou.lawrence@kitware.com>
Co-authored-by: John Parent <john.parent@kitware.com>
2022-03-17 09:01:01 -07:00
Ben Cowan
81bc00d61f Adding basic Windows features (#21259)
* Snapshot of some MSVC infrastructure added during experiments a while ago.  Rebasing from spack/develop.

* Added platform and OS definitions for Windows.

* Updated Windows platform file to conform to new archspec use.

* Added Windows as a platform; introduced some debugging code.

* Added type annotations.

* Fixed copyright.

* Removed print statements.

* Ensure `spack arch` returns correctly on Windows (#21428)

* Correctly identify windows as 'windows-Windows10-AMD64'
2022-03-17 09:01:01 -07:00
Harmen Stoppels
55544950e2
PackageViewMixin: fix symlinks conflict issue (#29515)
`stat`'ing a file in the dst dir is the wrong thing to do, you should
`lstat` to capture broken symlinks.
2022-03-17 08:18:28 -06:00
Tom Scogland
94794d061a
optimize instantiation and comparison of versions (#29429)
Re-work the checks and comparisons around commit versions, when no
commit version is involved the overhead is now in the noise, where one
is the overhead is now constant rather than linear.
2022-03-15 18:53:28 +00:00
Sergey Kosukhin
f1bdbefd46
mpich: fix the detection of the package (#29284) 2022-03-14 16:33:56 +01:00
Harmen Stoppels
474493713b
compiler.py: early return in compiler_environment when no modules (#29441) 2022-03-14 11:27:49 +00:00
Massimiliano Culpo
a6eed4a7c7
Remove "setup_environment" and "setup_dependent_environment" (#29463)
fixes #29446

The new setup_*_environment functions have been falling back
to calling the old functions and warn the user since #11115.

This commit removes the fallback behavior and any use of:
- setup_environment
- setup_dependent_environment
in the codebase
2022-03-13 11:51:55 -04:00
Harmen Stoppels
8adc6b7e8e
module_cmd.py: use posix awk; fix stderr redirection (#29440)
Emulates `env -0` in a posix compliant way, avoiding a slow python process, speeds up setting up the build env when modules should load.
2022-03-11 18:29:11 +01:00
Harmen Stoppels
88fbba3e1e
intel-mkl: load compiler modules when querying compiler (#29439) 2022-03-10 14:14:17 -08:00
Massimiliano Culpo
2cd5c00923
Allow for multiple dependencies/dependents from the same package (#28673)
Change the internal representation of `Spec` to allow for multiple dependencies or
dependents stemming from the same package. This change permits to represent cases
which are frequent in cross compiled environments or to bootstrap compilers.

Modifications:
- [x] Substitute `DependencyMap` with `_EdgeMap`. The main differences are that the
      latter does not support direct item assignment and can be modified only through its
      API. It also provides a `select_by` method to query items.
- [x] Reworked a few public APIs of `Spec` to get list of dependencies or related edges.
- [x] Added unit tests to prevent regression on #11983 and prove the synthetic construction
      of specs with multiple deps from the same package. 

Since #22845 went in first, this PR reuses that format and thus it should not change hashes.
The same package may be present multiple times in the list of dependencies with different
associated specs (each with its own hash).
2022-03-10 11:53:45 -08:00
Harmen Stoppels
dc78f4c58a
environment.py: allow link:run (#29336)
* environment.py: allow link:run

Some users want minimal views, excluding run-type dependencies, since
those type of dependencies are covered by rpaths and the symlinked
libraries in the view aren't used anyways.

With this change, an environment like this:

```
spack:
  specs: ['py-flake8']
  view:
    default:
      root: view
      link: run
```

includes python packages and python, but no link type deps of python.
2022-03-09 12:35:26 -08:00
百地 希留耶
3270aa106b
Hide package repository count when redirecting stdout (#29402) 2022-03-09 17:07:07 +00:00
百地 希留耶
3370d3f57e
Fix tab completion erroring with spack unit-test (#29405) 2022-03-09 16:09:57 +01:00
Harmen Stoppels
3df90e3e33
ci.py: fix utf-8 issue with git log output (#29419) 2022-03-09 13:13:37 +00:00
Harmen Stoppels
a39a6c4ea7
version.py: set is_commit in constructor (#29369)
Speeds up comparison on `Version` by ~2.5x, e.g.

```python
In [1]: v = spack.version.Version('1.0.0'); w = spack.version.Version('1.0.2')

In [2]: %timeit v < w

1.47 µs ± 5.59 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)

535 ns ± 1.75 ns per loop (mean ± std. dev. of 7 runs, 1000000 loops each)
```
2022-03-08 10:50:48 +01:00
Todd Gamblin
d424d0e54e
find: deprecate spack find -bootstrap for spack -bootstrap find (#29152)
Since adding the `spack --bootstrap` argument we don't need `spack find --bootstrap`
anymore.  Deprecate it.
2022-03-05 21:41:18 -08:00
Harmen Stoppels
80a6eab2db Add a test 2022-03-04 18:13:34 +01:00
Harmen Stoppels
9199800bbd VersionRange.satisfies should test non-empty intersection 2022-03-04 18:13:34 +01:00
Massimiliano Culpo
b48bdc9e19
Fix importing Spack packages as Python modules (#29221)
fixes #29203

This PR fixes a subtle bug we have when importing
Spack packages as Python modules that can lead to
multiple module objects being created for the same
package.

It also fixes all the places in unit-tests where
"relying" on the old bug was crucial to have a new
"clean" state of the package class.
2022-03-04 08:42:27 +01:00
Doug Jacobsen
06fd0f892e
Revert GCS fetch strategy, to remove s3 interface (#28736)
This commit reverts the GCS fetch strategy to before commit:
d759612523

The previous commit added some s3 syntax to handle connections, but
added them into the GCS fetch strategy in a way that prevents GCS from
working anymore.
2022-03-03 16:34:33 -08:00
Greg Becker
e2b87ade06
rocmcc compiler: initial commit based on aocc and clang (#28575)
* rocmcc compiler: initial commit based on aocc and clang

Co-authored-by: luker <luke.roskop@hpe.com>
Co-authored-by: Tom Scogland <scogland1@llnl.gov>
2022-03-03 14:34:22 -07:00
Michael Kuhn
49069e4f58
installer: Fix cosmetic problem with terminal title (#29070)
The status displayed in the terminal title could be wrong when doing
distributed builds. For instance, doing `spack install glib` in two
different terminals could lead to the current package being reported as
`40/29` due to the way Spack handles retrying locks.

Work around this by keeping track of the package IDs that were already
encountered to avoid counting packages twice.
2022-03-03 14:21:15 +01:00
Scott Wittenburg
c72735229f
test/installer.py: remove commented code and inaccurate docstring (#29305) 2022-03-03 10:19:57 +01:00
Danny McClanahan
2c331a1d7f
make @llnl.util.lang.memoized support kwargs (#21722)
* make memoized() support kwargs

* add testing for @memoized
2022-03-02 11:12:15 -08:00
Massimiliano Culpo
8d118104c7
Fix typos when forwarding arguments to traverse_edges (#29261)
A few calls use `deptypes=...` instead of `deptype=...`
2022-03-02 08:43:26 +01:00
Tamara Dahlgren
b20df12d09
test_env_install_two_specs_same_dep: properly check installed specs (#29222) 2022-03-01 10:35:14 +01:00
Scott Wittenburg
b082c33c85
commands: Propgate command return value as exit code (#29223) 2022-02-25 10:49:56 -08:00
Tamara Dahlgren
0b4f40ab79
Testing: Summarize test results and add verbose output (#28700) 2022-02-23 18:36:21 -08:00
Massimiliano Culpo
1ddad522a4
Move early exit for setup only argument (#29041)
See https://github.com/spack/spack/pull/28468/files#r809156986

If we exit before generating the:

 error("Dependencies must have compatible OS's with their dependents").
 ...

facts we'll output a problem that is effectively
different by the one solved by clingo.
2022-02-23 01:46:52 -08:00
Tom Scogland
a9ba40164a
Checksum match (#28989)
* cmd/checksum: prefer url matching url_from_version

This is a minimal change toward getting the right archive from places
like github.  The heuristic is:

* if an archive url exists, take its version
* generate a url from the package with pkg.url_from_version
* if they match
  * stop considering other URLs for this version
  * otherwise, continue replacing the url for the version

I doubt this will always work, but it should address a variety of
versions of this bug.  A good test right now is `spack checksum gh`,
which checksums macos binaries without this, and the correct source
packages with it.

fixes #15985
related to #14129
related to #13940

* add heuristics to help create as well

Since create can't rely on an existing package, this commit adds another
pair of heuristics:
1. if the current version is a specifically listed archive, don't
   replace it
2. if the current url matches the result of applying
   `spack.url.substitute_version(a, ver)` for any a in archive_urls,
   prefer it and don't replace it

fixes #13940

* clean up style and a lingering debug import

* ok flake8, you got me

* document reference_package argument

* Update lib/spack/spack/util/web.py

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>

* try to appease sphinx

Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
2022-02-23 00:55:59 +00:00
Todd Gamblin
36b0730fac
Add spack --bootstrap option for accessing bootstrap store (#25601)
We can see what is in the bootstrap store with `spack find -b`, and you can clean it with `spack
clean -b`, but we can't do much else with it, and if there are bootstrap issues they can be hard to
debug.

We already have `spack --mock`, which allows you to swap in the mock packages from the command
line. This PR introduces `spack -b` / `spack --bootstrap`, which runs all of spack with
`ensure_bootstrap_configuration()` set. This means that you can run `spack -b find`, `spack -b
install`, `spack -b spec`, etc. to see what *would* happen with bootstrap configuration, to remove
specific bootstrap packages, etc. This will hopefully make developers' lives easier as they deal
with bootstrap packages.

This PR also uses a `nullcontext` context manager. `nullcontext` has been implemented in several
other places in Spack, and this PR consolidates them to `llnl.util.lang`, with a note that we can
delete the function if we ever reqyire a new enough Python.

- [x] introduce `spack --bootstrap` option
- [x] consolidated all `nullcontext` usages to `llnl.util.lang`
2022-02-22 12:35:34 -07:00
Todd Gamblin
7912a8e90b
bugfix: Not all concrete versions on the CLI should be considered real (#28620)
Some "concrete" versions on the command line, e.g. `qt@5` are really
meant to satisfy some actual concrete version from a package. We should
only assume the user is introducing a new, unknown version on the CLI
if we, well, don't know of any version that satisfies the user's
request.  So, if we know about `5.11.1` and `5.11.3` and they ask for
`5.11.2`, we'd ask the solver to consider `5.11.2` as a solution.  If
they just ask for `5`, though, `5.11.1` or `5.11.3` are fine solutions,
as they satisfy `@5`, so use them.

Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-02-21 11:46:37 -08:00
Alberto Invernizzi
b8d042273a
Bring back cuda@11.4.0 conflicts for GCC and clang; add 11.4.3:11.4.4 (#29076)
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
2022-02-21 20:28:56 +01:00
Massimiliano Culpo
7fd94fc4bc
spack external find: change default behavior (#29031)
See https://github.com/spack/spack/issues/25353#issuecomment-1041868116

This commit changes the default behavior of
```
$ spack external find
```
from searching all the possible packages Spack knows about to
search only for the ones tagged as being a "build-tool".

It also introduces a `--all` option to restore the old behavior.
2022-02-18 11:51:01 -07:00
Seth R. Johnson
6c61c2695a
darwin: robust macos version detection (#28991)
Prefer `sw_vers` to `platform.mac_ver`. In anaconda3 installation, for example, the latter reports 10.16 on Monterey -- I think this is affected by how and where the python instance was built.

Use MACOSX_DEPLOYMENT_TARGET if present to override the operating system choice.
2022-02-17 20:50:41 -07:00
Scott Wittenburg
38643dcd7e
gitlab: Propagate stack name to downstream build jobs (#29019)
It will be useful for metrics gathering and possibly debugging to
have this environment variable available in the runner pods that
do the actual rebuilds.
2022-02-17 15:36:48 -05:00
Tamara Dahlgren
fefe65a35b
Testing: optionally run tests on externally installed packages (#28701)
Since Spack does not install external packages, this commit skips them by
default when running stand-alone tests. The assumption is that such packages
have likely undergone an acceptance test process. 

However, the tests can be run against installed externals using 
```
% spack test run --externals ...
```
2022-02-17 19:47:42 +01:00
Harmen Stoppels
d93f9b82ac
Reduce verbosity of patches=... variant (#29015)
* reduce verbosity of patches=... variant

* Special-case prefix-matches for satisfies of patches variant
2022-02-17 11:06:32 +01:00
Massimiliano Culpo
fa132614e0
ASP-based solver: don't sort when defining variant possible values (#29013)
fixes #28260

Since we iterate over different variants from many packages, the variant 
values may have types which are not comparable, which causes errors 
at runtime. This is not a real issue though, since we don't need the facts
to be ordered. Thus, to avoid needless sorting, the sorted function has 
been removed and a comment has been added to tip any developer that
might need to inspect these clauses for debugging to add back sorting 
on the first two items only.

It's kind of difficult to add a test for this, since the error depends on 
whether Python sorting algorithm ever needs to compare the third 
value of a tuple being ordered.
2022-02-17 08:50:50 +01:00
Tom Scogland
8f5fcc6e95
extensions: allow multiple "extends" directives (#28853)
* extensions: allow multiple "extends" directives

This will allow multiple extends directives in a package as long as only one of
them is selected as a dependency in the concrete spec.

* document the option to have multiple extends
2022-02-16 21:23:12 +00:00
Todd Gamblin
b1ff9c05bc concretizer: refactor argument passing for reuse
Reuse previously was a very invasive change that required parameters to be added to all
the methods that called `concretize()` on a `Spec` object. With the addition of
concretizer configuration, we can use the config system to simplify this argument
passing and keep the code cleaner.

We decided that concretizer config options should be read at `Solver` instantiation
time, and if config changes between instnatiation of a particular solver and
`solve()` invocation, the `Solver` should use the settings from `__init__()`.

- [x] remove `reuse` keyword argument from most concretize functions
- [x] refactor usages to use `spack.config.override("concretizer:reuse", True)`
- [x] rework argument passing in `Solver` so that parameters are set from config
      at instantiation time
2022-02-16 10:17:18 -08:00