This reverts commit d06fd26c9a.
The problem is that Bitbucket's API forwards download requests to an S3 bucket using a temporary URL. This URL includes a signature for the request, which embeds the HTTP verb. That means only GET requests are allowed, and HEAD requests would fail verification, leading to 403 erros. The same is observed when using `curl -LI ...`
Using `-Werror` is good practice for development and testing, but causes us a great
deal of heartburn supporting multiple compiler versions, especially as newer compiler
versions add warnings for released packages. This PR adds support for suppressing
`-Werror` through spack's compiler wrappers. There are currently three modes for
the `flags:keep_werror` setting:
* `none`: (default) cancel all `-Werror`, `-Werror=*` and `-Werror-*` flags by
converting them to `-Wno-error[=]*` flags
* `specific`: preserve explicitly selected warnings as errors, such as
`-Werror=format-truncation`, but reverse the blanket `-Werror`
* `all`: keeps all `-Werror` flags
These can be set globally in config.yaml, through the config command-line flags, or
overridden by a particular package (some packages use Werror as a proxy for determining
support for other compiler features). We chose to use this approach because:
1. removing `-Werror` flags entirely broke *many* build systems, especially autoconf
based ones, because of things like checking `-Werror=feature` and making the
assumption that if that did not error other flags related to that feature would also work
2. Attempting to preserve `-Werror` in some phases but not others caused similar issues
3. The per-package setting came about because some packages, even with all these
protections, still use `-Werror` unsafely. Currently there are roughly 3 such packages
known.
For reasons beyond me Python thinks it's a great idea to upgrade HEAD
requests to GET requests when following redirects. So, this PR adds a
better `HTTPRedirectHandler`, and also moves some ad-hoc logic around
for dealing with disabling SSL certs verification.
Also, I'm stumped by the fact that Spack's `url_exists` does not use
HEAD requests at all, so in certain cases Spack awkwardly downloads
something first to see if it can download it, and then downloads it
again because it knows it can download it. So, this PR ensures that both
urllib and botocore use HEAD requests.
Finally, it also removes some things that were there to support currently
unsupported Python versions.
Notice that the HTTP spec [section 10.3.2](https://datatracker.ietf.org/doc/html/rfc2616.html#section-10.3.2) just talks about how to deal
with POST request on redirect (whether to follow or not):
> If the 301 status code is received in response to a request other
> than GET or HEAD, the user agent MUST NOT automatically redirect the
> request unless it can be confirmed by the user, since this might
> change the conditions under which the request was issued.
> Note: When automatically redirecting a POST request after
> receiving a 301 status code, some existing HTTP/1.0 user agents
> will erroneously change it into a GET request.
Python has a comment about this, they choose to go with the "erroneous change".
But they then mess up the HEAD request while following the redirect, probably
because they were too busy discussing how to deal with POST.
See https://github.com/python/cpython/pull/99731
This adds super-lazy maintainer mode to `spack checksum`: Instead of
only printing the new checksums to the terminal, `-a` and
`--add-to-package` will add the new checksums to the `package.py` file
and open it in the editor afterwards for final checks.
* Add a WindowsRegistryView class, which can query for existing
package installations on Windows. This is particularly important
because some Windows packages (including those added here)
do not allow two simultaneous installs, and this can be
queried in order to provide a clear error message.
* Consolidate external path detection logic for Windows into
WindowsKitExternalPaths and WindowsCompilerExternalPaths objects.
* Add external-only packages win-sdk and wgl
* Add win-wdk (including external detection) which depends on
win-sdk
* Replace prior msmpi implementation with a source-based install
(depends on win-wdk). This install can control the install
destination (unlike the binary installation).
* Update MSVC compiler to choose vcvars based on win-sdk dependency
* Provide "msbuild" module-level variable to packages during build
* When creating symlinks on Windows, need to explicitly specify when
a symlink target is a directory
* executables_in_path no-longer defaults to using PATH (this is
now expected to be taken care of by the caller)
Spec traversals can now specify a topological ordering. A topologically-
ordered traversal with input specs X1, X2... will
* include all of X1, X2... and their children
* be ordered such that a given node is guaranteed to appear before any
of its children in the traversal
Other notes:
* Input specs can be children of other input specs (this is useful if
a user specifies a set of specs to uninstall: some of those specs
might be children of others)
* `direction="parents"` will produce a reversed topological order
(children always come before parents).
* `cover="edges"` will generate a list of edges L such that (a) input
edges will always appear before output edges and (b) if you create
a list with the destination of each edge in L the result is
topologically ordered
* test_suite.py: speed up slow test by using mock packages
* Don't resolve the sha during unit-tests
* Skip long-running test that fails, instead of executing it
* uninstall: fix accidental cubic complexity
Currently spack uninstall runs in worst case cubic time complexity
thanks to traversal during traversal during traversal while collecting
the specs to be uninstalled.
Also brings down the number of error messages printed to something
linear in the amount of matching specs instead of quadratic.
* Add a regression test for 33928
* PackageBase should not set `(build|install)_time_test_callbacks`
* Fix audits by preserving the current semantic
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* Enable hdf5 build (including +mpi) on Windows
* This includes updates to hdf5 dependencies openssl (minor edit) and
bzip2 (more-extensive edits)
* Add binary-based installation of msmpi (this is currently the only
supported MPI implementation in Spack for Windows). Note that this
does not install to the Spack-specified prefix. This implementation
will be replaced with a source-based implementation
Co-authored-by: John Parent <john.parent@kitware.com>
#32942 fixed bootstrapping on Windows by having the core Spack
code explicitly add the Clingo package bin/ directory as a
DLL path.
Since then, #33400 has been merged, which ensures that the Python
module installed by the Spack `clingo` package can find the DLLs
in bin/.
Note that this only works for Spack instances which have been
bootstrapped after #33400: for installations bootstrapped before
then, you will need to run `spack clean -b` (this would only
be needed for Spack instances running on Windows).
Revamp the timer so we always have a designated begin and end.
Fix a bug where the phase timer was stopped before the phase started,
resulting in incorrect timing reports in timers.json.
Add spack.ld_so_conf.host_dynamic_linker_search_paths
Retrieve the current host runtime search paths for shared libraries;
for GNU and musl Linux we try to retrieve the dynamic linker from the
current Python interpreter and then find the corresponding config file
(e.g. ld.so.conf or ld-musl-<arch>.path). Similar can be done for
BSD and others, but this is not implemented yet. The default paths
are always returned. We don't check if the listed directories exist.
Use this in spack external find for libraries.
Co-authored-by: Harmen Stoppels <harmenstoppels@gmail.com>
