Follow-up to #17110
### Before
```bash
CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC
SPACK_CC=/usr/bin/clang; export SPACK_CC
PATH=...:/Users/Adam/spack/lib/spack/env/apple-clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH
```
### After
```bash
CC=/Users/Adam/spack/lib/spack/env/clang/clang; export CC
SPACK_CC=/usr/bin/clang; export SPACK_CC
PATH=...:/Users/Adam/spack/lib/spack/env/clang:/Users/Adam/spack/lib/spack/env/case-insensitive:/Users/Adam/spack/lib/spack/env:...; export PATH
```
`CC` and `SPACK_CC` were being set correctly, but `PATH` was using the name of the compiler `apple-clang` instead of `clang`. For most packages, since `CC` was set correctly, nothing broke. But for packages using `Makefiles` that set `CC` based on `which clang`, it was using the system compilers instead of the compiler wrappers. Discovered when working on `py-xgboost@0.90`.
An alternative fix would be to copy the symlinks in `env/clang` to `env/apple-clang`. Let me know if you think there's a better way to do this, or to test this.
The dependencies needed a little clean up as several dependencies are
only needed for the +X variant. This PR consolidates all of the
dependencies that actually require +X and explicitly disables them when
~X to prevent accidentally picking up system libraries.
- modified the description of the +X variant
- arranges dependencies to group them
- added missing dependency on xz
- removed unneeded dependencies
- freetype
- glib
- set dependencies when +X
- cairo
- jpeg
- libpng
- libtiff
- tcl/tk
- R uses tcl/tk together, so only tk needs to be depended on, and only
when +X
- moved tcl/tk resources to with/without-x test
- added explicit with/without settings for
- cairo
- jpeglib
- libpng
- libtiff
- tcltk
The fixture was introduced in #19690 maybe accidentally.
It's not used in unit tests, and though it should be
mutable it seems an exact copy of it's immutable version.
Before this change, in pipeline environments where runners do not have access
to persistent shared file-system storage, the only way to pass buildcaches to
dependents in later stages was by using the "enable-artifacts-buildcache" flag
in the gitlab-ci section of the spack.yaml. This change supports a second
mechanism, named "temporary-storage-url-prefix", which can be provided instead
of the "enable-artifacts-buildcache" feature, but the two cannot be used at the
same time. If this prefix is provided (only "file://" and "s3://" urls are
supported), the gitlab "CI_PIPELINE_ID" will be appended to it to create a url
for a mirror where pipeline jobs will write buildcache entries for use by jobs
in subsequent stages. If this prefix is provided, a cleanup job will be
generated to run after all the rebuild jobs have finished that will delete the
contents of the temporary mirror. To support this behavior a new mirror
sub-command has been added: "spack mirror destroy" which can take either a
mirror name or url.
This change also fixes a bug in generation of "needs" list for each job. Each
jobs "needs" list is supposed to only contain direct dependencies for scheduling
purposes, unless "enable-artifacts-buildcache" is specified. Only in that case
are the needs lists supposed to contain all transitive dependencies. This
changes fixes a bug that caused the needs lists to always contain all transitive
dependencies, regardless of whether or not "enable-artifacts-buildcache" was
specified.
* py-typing: new version, avoid issues with newer versions of python
https://pypi.org/project/typing/
"For package maintainers, it is preferred to use
typing;python_version<"3.5" if your package requires it to support
earlier Python versions."
* update conflict version / more message detail
* change the depends_on, leave a comment suggesting correct usage
The actual, documented minimum version of the cfitsio dependency,
v3.181, is now neither available for (easy) download from NASA, nor as
a Spack package. No upper bound on version number exists (at this time).
Pipelines: DAG pruning
During the pipeline generation staging process we check each spec against all configured mirrors to determine whether it is up to date on any of the mirrors. By default, and with the --prune-dag argument to "spack ci generate", any spec already up to date on at least one remote mirror is omitted from the generated pipeline. To generate jobs for up to date specs instead of omitting them, use the --no-prune-dag argument. To speed up the pipeline generation process, pass the --check-index-only argument. This will cause spack to check only remote buildcache indices and avoid directly fetching any spec.yaml files from mirrors. The drawback is that if the remote buildcache index is out of date, spec rebuild jobs may be scheduled unnecessarily.
This change removes the final-stage-rebuild-index block from gitlab-ci section of spack.yaml. Now rebuilding the buildcache index of the mirror specified in the spack.yaml is the default, unless "rebuild-index: False" is set. Spack assigns the generated rebuild-index job runner attributes from an optional new "service-job-attributes" block, which is also used as the source of runner attributes for another generated non-build job, a no-op job, which spack generates to avoid gitlab errors when DAG pruning results in empty pipelines.
Add `manual_download = True` to packages that need to do manual
downloads but do not have the `manual_download attribute set. This
provides a message when installing these packages rather than a generic
fetch error.
Add versions 2020.12 and 2021.01. The viewer and trace viewer are now
integrated into a single program and one tar ball. Now available on
arm/aarch64 and now uses Java 11.
Update some things in hpctoolkit to prepare for a 2021.02.x release:
1. allow binutils to be built with +nls.
2. require libmonitor to be built with +dlopen.
3. allow rocm in more than just develop branch.
4. remove some conflicting setenv's in hpctoolkit module.
The SPACK_PYTHON environment variable can be set to a python interpreter to be
used by the spack command. This allows the spack command itself to use a
consistent and separate interpreter from whatever python might be used for package
building.
