#22845 revealed a long-standing bug that had never been triggered before, because the
hashing algorithm had been stable for multiple years while the bug was in production. The
bug was that when reading a concretized environment, Spack did not properly read in the
build hashes associated with the specs in the environment. Those hashes were recomputed
(and as long as we didn't change the algorithm, were recomputed identically). Spack's
policy, though, is never to recompute a hash. Once something is installed, we respect its
metadata hash forever -- even if internally Spack changes the hashing method. Put
differently, once something is concretized, it has a concrete hash, and that's it -- forever.
When we changed the hashing algorithm for performance in #22845 we exposed the bug.
This PR fixes the bug at its source, but properly reading in the cached build hash attributes
associated with the specs. I've also renamed some variables in the Environment class
methods to make a mistake of this sort more difficult to make in the future.
* ensure environment build hashes are never recomputed
* add comment clarifying reattachment of env build hashes
* bump lockfile version and include specfile version in env meta
* Fix unit-test for v1 to v2 conversion
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
* Refactor platform etc. to avoid circular dependencies
All the base classes in spack.architecture have been
moved to the corresponding specialized subpackages,
e.g. Platform is now defined within spack.platforms.
This resolves a circular dependency where spack.architecture
was both:
- Defining the base classes for spack.platforms, etc.
- Collecting derived classes from spack.platforms, etc.
Now it dopes only the latter.
* Move a few platform related functions to "spack.platforms"
* Removed spack.architecture.sys_type()
* Fixup for docs
* Rename Python modules according to review
* dvsdk: Turn off variants by default
This allows an install to more easily be explicit about which pieces to
turn on as more variants are added
* dvsdk: effectively disable the broken variants
* Switch http to https where latter exists
* Hopefully restore original permissions
* Add URL updates after include the -L curl option
* Manual corrections to select URL format strings
* Tell gtk-doc where the XML catalog is
The gtk-doc configure script has an option for specifying the path to
the XML catalog. If this is not set the configure script will search
a defined set of directories for a catalog file and will set
`with_xml_catalog` based on that. Only if no system catalog is found will
the XML_CATALOG_FILES be looked at. In order to make sure that the spack
provided catalog is used, pass the `--with-xml-catalog` option.
* Use the property from docbook-xml
Currently as part of installing a package, we lock a prefix, check if
it exists, and create it if not; the logic for creating the prefix
included a check for the existence of that prefix (and raised an
exception if it did), which was redundant.
This also includes removal of tests which were not verifying
anything (they pass with or without the modifications in this PR).
- Parallel install was failing to generate a config file.
- OpenSSH has an extensive test suite, run it if requested.
- 'executables' wrongly had 'rsh', replaced the openssh tools.
There are two ways to build SQLite: With the Autotools setup or the
so-called "amalgamation" which is a single large C file containing the
SQLite implementation. The amalgamation build is controlled by
pre-processor flags and the Spack setup was using an amalgamation
pre-processor flag for a feature that is controlled by an option of the
configure script. As a consequence, until now Spack has always built
SQLite with the rtree feature enabled.
Knowing that spack has patched the code and organized the build is potentially valuable information for GROMACS users and developers troubleshooting their builds.
PLUMED does further patches to GROMACS, so that is expressed directly also.
Modifications:
- Export platforms from spack.platforms directly, so that client modules don't have to import submodules
- Use only plain imports in test/architecture.py
- Parametrized test in test/architecture.py and put most of the setup/teardown in fixtures
This is a major rework of Spack's core core `spec.yaml` metadata format. It moves from `spec.yaml` to `spec.json` for speed, and it changes the format in several ways. Specifically:
1. The spec format now has a `_meta` section with a version (now set to version `2`). This will simplify major changes like this one in the future.
2. The node list in spec dictionaries is no longer keyed by name. Instead, it is a list of records with no required key. The name, hash, etc. are fields in the dictionary records like any other.
3. Dependencies can be keyed by any hash (`hash`, `full_hash`, `build_hash`).
4. `build_spec` provenance from #20262 is included in the spec format. This means that, for spliced specs, we preserve the *full* provenance of how to build, and we can reproduce a spliced spec from the original builds that produced it.
**NOTE**: Because we have switched the spec format, this PR changes Spack's hashing algorithm. This means that after this commit, Spack will think a lot of things need rebuilds.
There are two major benefits this PR provides:
* The switch to JSON format speeds up Spack significantly, as Python's builtin JSON implementation is orders of magnitude faster than YAML.
* The new Spec format will soon allow us to represent DAGs with potentially multiple versions of the same dependency -- e.g., for build dependencies or for compilers-as-dependencies. This PR lays the necessary groundwork for those features.
The old `spec.yaml` format continues to be supported, but is now considered a legacy format, and Spack will opportunistically convert these to the new `spec.json` format.
* Added spackage to build Sina (https://github.com/LLNL/Sina).
* Improvements to sina/package.py
Made numerous simplifications and improvements to sina/package.py
based on PR feedback.
* Added licence info
* Added maintainers
* Changed maintainers to be Github IDs.
Added a dependency for mpip@3.5: when the libunwind is set to true (which is the default)
and '~setjmp' is set to False (which is also the default) to avoid a configure
time error from not finding libunwind.
Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
This modification accounts for:
1. Bootstrapping from sources using system, non-standard Python
2. Using later an ABI compatible standard Python interpreter
* tests: make `spack url [stats|summary]` work on mock packages
Mock packages have historically had mock hashes, but this means they're also invalid
as far as Spack's hash detection is concerned.
- [x] convert all hashes in mock package to md5 or sha256
- [x] ensure that all mock packages have a URL
- [x] ignore some special cases with multiple VCS fetchers
* url stats: add `--show-issues` option
`spack url stats` tells us how many URLs are using what protocol, type of checksum,
etc., but it previously did not tell us which packages and URLs had the issues. This
adds a `--show-issues` option to show URLs with insecure (`http`) URLs or `md5` hashes
(which are now deprecated by NIST).
This allows to fix the compilation of gcc versions less than 11.1.0,
due to the remove of cyclades of libsanitizer as it is described in
the patch:
The Linux kernel has removed the interface to cyclades from the latest
kernel headers due to them being orphaned for the past 13
years. libsanitizer uses this header when compiling against glibc, but
glibcs itself doesn't seem to have any references to cyclades. Further
more it seems that the driver is broken in the kernel and the firmware
doesn't seem to be available anymore. As such since this is breaking
the build of libsanitizer (and so the GCC bootstrap) it is proposed to
remove this.
Co-authored-by: Arcesio Castaneda Medina <arcesio.castaneda.medina@itwm.fraunhofer.de>
