This updates the Spack QT package to enable building qt version 4 on
MacOS.
This includes the following changes to the qt package:
* add version 4.8.7
* add option to build with or without shared libs
* add options to disable tools, ssl, sql, and freetype support
* add qt4-tools patch when building qt@4+tools
* add option to build as a framework (only available on MacOS)
* replace qt4-el-capitan patch with qt4-mac patch (which includes the
edits from qt4-el-capitan)
* apply qt4-pcre-include-conflict.patch only for version 4.8.6
(rather than all 4.x versions)
* apply qt4-gcc-and-webkit.patch for 4.x versions before 4.8.7 and
create a separate qt4-gcc-and-webkit-487.patch for version 4.8.7
* update patch function for qt@4 on MacOS to update configure
variables relevant to Spack (e.g. PREFIX)
* add option to build freetype with Spack, as a vendored dependency
of QT, or not at all (default is to build with Spack)
This includes the following edits outside of the qt package:
* Update MacOS version utility function to return all parts of the
Mac version (rather than just the first two)
* gettext package: implement "libs"
* python package: add gettext as a dependency
* Raise an exception and exit with a meaningful message when binary path substitution fails.
* Skip binary text replacement with padding and issue a warning when the new install path is longer than the old install path.
- We don't currently make enough use of the maintainers field on
packages, though we could use it to assign reviews.
- add a command that allows maintainers to be queried
- can ask who is maintaining a package or packages
- can ask what packages users are maintaining
- can list all maintained or unmaintained packages
- add tests for the command
* Added a unit test reproducing the failure in 12085
* Fixed name clash in the 'from_environment_diff' function
The bug reported in #12085 stemmed from a name clash among variables,
introduced during the refactor in #10753 and not caught by unit tests
and reviews.
- Setting specs from lockfiles was not correctly stringifying concretized
user specs.
- Fix `_set_user_specs_from_lockfile`
- Add some validation code to `SpecList` constructor
Spack has evolved to have three types of hash functions, and it's
becoming hard to tell when each one is called. Whlie we aren't yet ready
to get rid of them, we can refactor them so that the code is clearer and
easier to track.
- Add a `hash_types` module with concise descriptors for hashes.
- Consolidate hashing logic in a private `Spec._spec_hash()` function.
- `dag_hash()`, `build_hash()`, and `full_hash()` all call `_spec_hash()`
- `to_node_dict()`, `to_dict()`, `to_yaml()` and `to_json()` now take a
`hash` parameter consistent with the one that `_spec_hash()` requires.
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
- ensure that Spec._build_hash attr is defined
- add logic to compute _build_hash when it is not already computed (e.g. for specs prior to this PR)
- add test to ensure that different instance of a build dep are preserved
- test conversion of old env lockfile format to new format
- tests: avoid view creation, DAG display in tests using MockPackage
- add regression test for more-general bug also fixed by this PR
- update lockfile version since the way we are maintaining hashes has changed
- write out backup for version-1 lockfiles and test that
The database and mutable_database fixtures were installing and uninstalling the same specs multiple times to ensure the database for tests has the correct state.
This commit optimizes the procedure by caching the state in an external directory, and copying it in instead of going through the installation or uninstallation again.
The database fixture is meant not to be modified by tests. This commit enforces this invariant by making the database read-only before starting the test.
* Added missing db markers to tests
* Added test for uninstall_by_spec
* `database` fixture now returns a read-only database
* Tests that modify the DB now use `mutable_database` fixture
Summary:
- Allow multiple definitions of compiler in compilers.yaml (use first instance)
- Still print debug messages when there are duplicates, to assist users in finding this issue.
Merging configs from different scopes can result in multiple compiler being present in the same configuration list. Instead of raising when there are duplicates, take the one with highest precedence.
Print a debug message instead of raising, so that we can still diagnose this. We don't have a good way of warning the user about inconsistent configuration *in the same file* -- we'd need to dig into YAML file/line info for that.
- [x] Add shell tests to ensure that `spack env activate`, `spack env
deactivate`, and `despacktivate` continue to work.
- [x] Also ensure that activate and deactivate both work with `set -u`
* extends mkdirs with permissions for intermediate folders
Does not use os.makedirs mode parameter because its behavior is changed
with Python 3.7 (it ignores it for intermediate dirs), and moreover it
was not possible to set different modes for newly-created folders
and leaf folder.
reference:
- https://bugs.python.org/issue19930
- https://docs.python.org/3.7/library/os.html#os.makedirs
* comment mkdirp step easing code understanding
* revert mkdir to default for package metapath
since metapath is nested in package folder, there is no need
to specify permissions for intermediate folders because the prefix
already exists.
* comment create_install_directory package modes
Bug relates to the interplay between:
1. random dict orders in python 3.5
2. bugfix in initial implementation of stacks for `_concretize_dependencies`
when `self._dependencies` is empty
3. bug in coconcretization algorithm computation of split specs
Result was transient hang in coconcretization.
Fixed#3 (bug in coconcretization) to resolve.
- remove redundant code in Environment.__init__
- use socket.gethostname() instead of which('hostname')
- refactor updating SpecList references
- refactor 'specs' literals to a single variable for default list name
- use six.string_types for python 2/3 compatibility
* from_sourcing_file: fixed a bug + added a few ignored variables
closes#7536
Credits for this change goes to mgsternberg (original author of #7536)
The new variables being ignored are specific to Modules v4.
* Use Spack Executable in 'EnvironmentModifications.from_sourcing_file'
Using this class avoids duplicating lower level logic to decode
stdout and handle non-zero return codes
* Extracted a function that returns the environment after sourcing files
The logic in `EnvironmentModifications.from_sourcing_file` has been
simplified by extracting a function that returns a dictionary with the
environment one would have after sourcing the files passed as argument.
* Further refactoring of EnvironmentModifications.from_sourcing_file
Extracted a function that sanitizes a dictionary removing keys that are
blacklisted, but keeping those that are whitelisted. Blacklisting and
whitelisting can be done on literals or regex.
Extracted a new factory that creates an instance of
EnvironmentModifications from a diff of two environments.
* Added unit tests
* PS1 is blacklisted + more readable names for some variables
All documentation mentions that `build_jobs` is limited by the number of
cores available in the system. This is also enforced when setting it via
`--jobs`. However, when setting it via `config.yaml`, it can exceed the
number of cores available, making builds run out of memory.
This PR adds the ability to specify the auto-dispatch targets that can
be used by the Intel compilers. The `-ax` flag will be written to the
respective compiler configuration files. This ability is very handy when
wanting to build optimized builds for various architectures. This PR
does not set any optimization flags, however.
Fixes#3690Fixes#5637
Uninstalling dependents of a spec was relying on a traversal of the
parents done by inspecting spec._dependents. This is in turn a
DependencyMap that maps a package name to a single DependencySpec object
(an edge in the DAG) and cannot thus model the case where a spec has
multiple configurations of the same parent package installed (for
example if different versions of the same Python library depend on
the same Python installation).
This commit works around this issue by constructing the list of specs to
be uninstalled in an alternative way, and adds tests to verify the
behavior. The core issue with DependencyMap is not resolved here.
The default library search for a package checks the lib/ and lib64/
directories for libraries before the root prefix, in order to save
time when searching for libraries provided by externals (which e.g.
may have '/usr/' as their root).
This moves that logic into the "find_libraries" utility method so
packages implementing their own custom library search logic can
benefit from it.
This also updates packages which appear to be replicating this logic
exactly, replacing it with a single call to "find_libraries".
Fixes#11782
Spack was not properly resolving relative paths to absolute paths
when a relative path was passed to "spack compiler add [PATH]".
Now, if provided a relative path, the absolute path is written to
compilers.yaml rather than the relative path.
* Add template creation test
* Added --skip-editor option to "spack create": normally
"spack create" opens an editor for the user after generating a
package file; when the --skip-editor option is used, "spack create"
only generates the package file and does not open an editor
* Added --skip-editor option to bash completion
- Namepsaces were shown without dots after the new format strings were
added.
- Add a test for `spack find` to ensure that find -N shows the right
output.
Fixes#11781
* Rename build log to spack-build-log.txt
* Rename environment variables file to spack-build-env.txt
* The name of the log and env files is now the same during the build
and after the build completes
* Update packages which referred to the build log/env files
* For packages installed before this commit using older names for the
build and env files, search for the older names
- Fix a bug introdcued by removing parse_anonymous_spec()
- Conflicts' when specs are now *actually* anonymous, and the name of the
package is implicit, so we need to remember to add it back to error
messages.
- `parse_anonymous_spec()` is a vestige of the days when Spack didn't
support nameless specs. We don't need it anymore because now we can
write Spec() for a spec that will match anything, and satisfies()
semantics work properly for anonymous specs.
- Delete `parse_anonymous_spec()` and replace its uses with simple calls
to the Spec() constructor.
- make then handling of when='...' specs in directives more consistent.
- clean up Spec.__contains__()
- refactor directives and tests slightly to accommodate the change.
- CNL OS previously used the *Cray PE* version to determine the OS
version. Cray does not synchronize PE and CLE releases; you can run
CLE7 with PrgEnv 6 (and NERSC currently does).
- Fix Spack's OS detection by using the cle-release file to detect the OS
version. This file is updated with every CLE OS release.
- Add some tests for our parsing logic
Add an example of a 'modules:' entry for an external package in
packages.yaml. The 'External Packages' section of 'Build
Customization' mentions 'paths:' and 'modules:' and gives an
example of paths, but not modules.
Fixes#11816
Allow packages to refer to non-expanded downloads (e.g. a single
script) using Stage.archive_file. This addresses a regression from
#11688 and adds a unit test for it.
This change reverts to the previous behavior of only looking for pgcc
and friends, not pgcc-llvm and friends.
The llvm variant doesn't support all the same features as the
traditional variant of the pgi code generator; this change avoids
treating the llvm variant as a default pgi compiler.
This retains the changes in #10704 which accept the "LLVM" suffix of
the version string of the PGI compiler, which allows users to
explicitly add the llvm-pgi compiler if desired.
For resources, it is desirable to use the expanded archive name of
the resource as the name of the directory when adding it to the root
staging area.
#11528 established 'spack-src' as the universal directory where
source files are placed, which also affected the behavior of
resources managed with Stages.
This adds a new property ('srcdir') to Stage to remember the name of
the expanded source directory, and uses this as the default name when
placing a resource directory in the root staging area.
This also:
* Ensures that downloaded sources are archived using the expanded
archive name (otherwise Spack will not be able to determine the
original directory name when using a cached archive).
* Updates working_dir context manager to guarantee restoration of
original working directory when an exception occurs
* Adds a "temp_cwd" context manager which creates a temporary
directory and sets it as the working directory
The regression test for #11678 fails on at least some Mac OS systems
because they have a /usr/bin/gcc that is secretly clang.
This PR replaces the dependency on a system gcc executable with a
test-generated script that generates the expected output for the
compiler logic.
Some tests introduced in #11528 temporarily set the user's `config:build_stage`, which affected (or created) a config.yaml file in the user's `$HOME/.spack` directory that could leave entries behind if the tests fail.
This change ensures only temporary configuration files are used/affected by these tests.
The "spack location" command was previously untested. This also adds
a check to ensure that composite Stages can report whether they were
expanded (this property was previously only recorded in Stage but not
in CompositeStage).
DIYStage, used to treat a user-managed directory as a staging area,
should always be considered expanded (i.e. the source has been
decompressed if it was stored in an archive).
This also:
* Adds checks to ensure that the path used to instantiate a
DIYStage refers to an existing directory.
* Adds tests to check the behavior of DIYStage (including behavior
added here, but it was generally untested before).
#11528 updated Stage to always store a Package's source in a fixed
directory accessible via `Stage.source_path` This left behind a
number of packages which were expecting to access the source code
via `Stage.path`. This Updates those packages to use
`Stage.source_path` instead.
This also updates the name of the fixed directory: The original name
of the fixed directory was "src", so if an expanded archive created a
"src" directory, then users inspecting the directory structure could
see paths like "src/src" (which wasn't wrong but could be confusing).
Therefore this also updates the name of the fixed directory to
"spack-src".
Fixes#11678
`spack compiler find` was not searching `PATH` when provided with no
arguments. ea7910a updated the API for the search function and the
command logic did not update how it called this function. This also
adds a test to ensure that `spack compiler find` will collect
compilers from `PATH`.
"spack module tcl find -r <spec>" (and equivalents for other module
systems) was failing when a dependency was installed in an upstream
Spack instance. This updates the module index to handle locating
module files for upstream Spack installations (encapsulating the
logic in a new class called UpstreamModuleIndex); the updated index
handles the case where a Spack installation has multiple upstream
instances.
Note that if a module is not available locally but we are using the
local package, then we shouldn't use a module (i.e. if the package is
also installed upstream, and there is a module file for it, Spack
should not use that module). Likewise, if we are instance X using
upstreams Y and Z like X->Y->Z, and if we are using a package from
instance Y, then we should only use a module from instance Y. This
commit includes tests to check that this is handled properly.
Spack currently tries to unify everything in the DAG, but this is too strict for build dependencies, where it is fine to build a dependency with a tool that conflicts with a version fo that tool for a dependent's build.
To enable a workaround for conflicts among build dependencies, so that users can install in multiple steps to avoid these conflicts, make the following changes:
* Dont apply package dependency constraints for build deps of installed packages
* Avoid applying constraints for installed packages vs. concrete packages
* Mark all dependencies of installed packages as visited in normalization method
* don't remove dependency links for concrete specs in flat_dependencies
Also add tests:
* Update test to ensure that link dependencies of installed packages have constraints applied
* Add test to check for proper handling of transitive dependencies (which is currently not the case)
- spack.compilers.find_compilers now uses a multiprocess.pool.ThreadPool to execute
system commands for the detection of compiler versions.
- A few memoized functions have been introduced to avoid poking the filesystem multiple
times for the same results.
- Performance is much improved, and Spack no longer fork-bombs the system when doing a `compiler find`
- We use `spack list --foramt=html` now, as it is much faster and doesn't
make the docs build take forever.
- Remove `spack list --format=rst` as it is no longer used.
- `stage.source_path` was previously overloaded; it returned `None` if it
didn't exist and this was used by client code
- we want to be able to know the `source_path` before it's created
- make stage.source_path available before it exists.
- use a well-known stage source path name, `$stage_path/src` that is
available when `Stage` is instantiated but does not exist until it's
"expanded"
- client code can now use the variable before the stage is created.
- client code can test whether the tarball is expanded by using the new
`stage.expanded` property instead of testing whether `source_path` is
`None`
- add tests for the new source_path semantics
- make tty.msg, tty.info, etc. print the exception type and stringified
message if the message argument is an exception.
- simplify parts of the code that call tty.debug(str(e))
- add extra tty.debug statements in places where exceptions were
previously ignored
- `spack graph --static` (and `spack.graph.dot_graph`) now do the "right
thing" and print the possible dependency graph of provided packages.
- `spack graph --static` no longer concretizes specs, as it only relies
on class level metadata
- Previously the behavior was not consistent -- `spack graph --static`
would graph possible dependencies of concrete specs, but would only
include some of them. The new code properly pursues all possible
dependencies, and allows traversing by different dependency types.
- `spack dependencies` can now take a --deptype argument to only traverse
particular deptypes
- add a new "common" argument for deptype in spack.cmd.common.arguments
- Database.installed_relatives() can now also take a deptype argument
- this is used by `spack dependencies --installed`
- `PackageBase.possible_dependencies` now:
- accepts a deptype param that controls dependency types traversed
- returns a dict mapping possible depnames to their immediate possible
dependencies (this lets you build a graph easily)
- Add tests for PackageBaes
- The 'name' attribute for packages was being set in DirectiveMeta, which
wasn't consistent with other class properties (like fullname, etc.)
- Move it to be a class property of `PackageMeta`, and add the
corresponding property method wrapper on `PackageBase`
* add c99_flag, c11_flag to compiler class
* implement c99_flag, c11_flag for gcc
* implement c99_flag, c11_flag for arm
* implement c99_flag for cce
* implement c99_flag, c11_flag for clang
* implement c99_flag, c11_flag for intel
* implement c99_flag, c11_flag for xl
Previously, module files were not set with the same permissions as the package installation. For world-readable packages, this would not cause a problem. For group readable packages, it does:
```
packages:
mypackage:
permissions:
group: mygroup
read: group
write: group
```
In this case, the modulefile is unreadable by members of the group other than the one who installed it. Add logic to the modulefile writers to set the permissions based on the configuration in `packages.yaml`
* Build cache: relocate path to spack/bin/sbang in text files.
* Found in testing.
* update packaging test
* Make sbang replacement including #!/bin/bash. Add an additional spack prefix replacement to fix stage directory references.
* flake8
* Use buildinfo.get() so old buildcaches without buildinfo['spackprefix'] can be read.
* config:build_jobs now controls the number of parallel jobs to spawn during
builds, but cannot ever exceed the number of cores on the machine.
* The default is set to 16 or the number of available cores, whatever
is lowest.
* Updated docs to reflect the changes done to limit parallel builds
- `gettext_uuid=True` makes every commit update every .pot file in spack/localized-docs,
and speeds up the internationalized doc build slightly.
- Optimize for less repository churn, and use `python-levenshtein` to accelerate
the build instead.
- make all Spack paths relative to a `_spack_root` symlink, so that we
can easily relocate the docs build *outside* lib/spack/docs
- set some useful defaults for gettext translation variables in conf.py
- update `relativeinclude` and other references to the spack root in the
RST files to use _spack_root
- Add a `--update FILE` option to `spack list`
- Output is written to the file only if any package is newer than the file
- Simplify the code in docs/conf.py using this new option
The Spack documentation currently hard-codes some functionality in
`conf.py`, which makes the doc build less "pluggable" for things like
localized doc builds.
In particular, we unconditionally generate an index of commands and a
package list as part of the docs, but those should really only be done if
things are not up to date.
This commit does the following:
- Add `--header` option to `spack commands` so that it can do the work of
prepending text to its output.
- Add `--update FILE` option to `spack commands` that makes it generate a
new command index *only* if FILE is out of date w.r.t. commands in the
Spack source.
- Simplify code in `conf.py` to use these options and only update the
command index when needed.
This PR implements several refactors requested in #11373, specifically:
- Config scopes are used to handle builtin defaults, command line overrides
and package overrides (`parallel=False`)
- `Package.make_jobs` attribute has been removed; `make_jobs` remains
as a module-scope variable in the build environment.
- The use of the argument `-j` has been rationalized across commands
- move '-j'/'--jobs' argument into `spack.cmd.common.arguments`
- Add unit tests to check that setting parallel jobs works as expected
- add new test to ensure that build job setting is isolated to each build
- Fix packages that used `Package.make_jobs` (i.e. `bazel`)
* Add Fujitsu compiler to Spack.
* Fixes for flake8
* Chenges location of FCC to subdirectory called case-insensitive
* Add compiler tests for Fujitsu compiler
* Modify the logic of taking compiler version for new version of Fujitsu compiler
The regex used for finding the Cray OS version from the PrgEnv-cray
module was not exact and was at times pulling the version from other
PrgEnv modules. This updates the regular expression to be more exact.
Adds executable=/bin/bash into Popen. We discovered this bug while
working in a csh/tsch environment. By executing with /bin/bash we ensure
that the module command works.
#8612 added command extensions to Spack: a command implemented in a
separate directory. This improves the implementation by allowing
the command to import additional utility code stored within the
established directory structure for commands.
This also:
* Adds tests for command extensions
* Documents command extensions (including the expected directory
layout)
- `svn info` prints different results depending on the system locale
- in particular, Japanese output doesn't contain "Revision:"
- Change Spack code to use XML output instead of using the human output
Add fixes to support multiple installs and dependents using a subset
of IntelPackage functionality.
* Update IntelPackage to only return scalapack libraries if the root
spec depends on MPI: scalapack requires MPI to be mentioned as a
dependency in the DAG. Package builds using intel-mkl for its
blas/lapack implementations but not for scalapack were failing to
build.
Ideally it would be possible to ask if any of the packages in the
DAG are actually requesting the scalapack functionality provided by
the IntelPackage and only return scalapack libs in that case, but
that is not easily done at this time.
Fixes#11314Fixes#11289
* set HOME when the intel silent installer is run. This prevents the
installer from using the ~/intel directory (which can cause
conflicts for multiple installs of the same IntelPackage)
Fixes#9713
Use new `module` function instead of `get_module_cmd`
Previously, Spack relied on either examining the bash `module()` function or using the `which` command to find the underlying executable for modules. More complicated module systems do not allow for the sort of simple analysis we were doing (see #6451).
Spack now uses the `module` function directly and copies environment changes from the resulting subprocess back into Spack. This should provide a future-proof implementation for changes to the logic underlying the module system on various HPC systems.
Add two functions to the EnvironmentModifications object to help
users sanitize environment variables in their package definitions:
* deprioritize_system_paths: this keeps system paths in the
environment variable but moves them to the end.
* prune_duplicate_paths: remove any duplicate paths from the
variable
This includes testing for the new functions as well as for
(previously-untested) old convenience functions for environment
variable manipulation.
This also adds special handling for bash functions so they
will be defined when the exported environment file is sourced.
Fixes#11335
Update the Spack compiler wrappers to add the headerpad_max_install_names
linker flag on MacOS. This allows the install_name_tool to rewrite
the RPATH entry of the binary to be longer if needed. This is
primarily useful for creating and distributing binary caches of
packages (i.e. using the "spack buildcache" command); binary caches
created on MacOS before this commit may not successfully relocate
(if the target root path is larger).
* Added a function that concretizes specs together
* Specs concretized together are copied instead of being referenced
This makes the specs different objects and removes any reference to the
fake root package that is needed currently for concretization.
* Factored creating a repository for concretization into its own function
* Added a test on overlapping dependencies
* extend Version class so that 2.0 > 1.develop > 1.1
* add concretization tests, with preferences and preferred version.
* add master, head, trunk as develop-like versions, develop > master > head > trunk
* update documentation on version comparison
- `spack edit` previously used `spack.util.executable` `Executable` objects,
and didn't `exec` the editor like you'd expect it to
- This meant that Spack was still running while your editor was, and
stdout/stdin were being set up in weird ways
- e.g. on macOS, if you call `spack edit` with `EDITOR` set to the
builtin `emacs` command, then type `Ctrl-g`, the whole thing dies with
a `==> Error: Keyboard interrupt`
- Fix all this by changing spack.util.editor to use `os.execv` instead of
Spack's `Executable` object
Also add constructor to NoLibrariesError which can either take an
error message (like other SpackErrors) or a name and prefix (in
which case the error message is constructed).
PR #10758 made a slight change to find_versions_of_archive() which included
archive_url in the search process. While this fixed `spack create` and
`spack checksum` missing command-line arguments, it caused `spack
install` to prefer those URLs over those it found in the scrape process.
As a result, the package url was treated as a list_url causing all R
packages to stop fetching once the package was updated on CRAN.
This patch is more selective about including the archive_url in the
remote versions, explicitly overriding it with matching versions found
by the scraper.
f242f5f8 changed the format strings but maintained backwards
compatibility in all cases except one: The list of valid tokens for
the module naming schemes was not updated properly to contain both
the new and old styles for compilers and package names.
This PR re-adds the old tokens into the list of valid tokens.
#11152 added documentation for #8772 but some details were based on
an earlier implementation that had changed by the time #8772 was
merged. In particular, #11152 mentioned that upstream Spack instances
were configured in config.yaml, when in fact they should be placed in
a separate upstreams.yaml config file; this PR updates the
documentation accordingly.
fixes#11159
The 'namespace' argument to both Repo and RepoPath were used to set the
"super namespace". Currently it seems to be vestigial as the only
"super namespace" allowed for packages is 'spack.pkg' since 39c9bbf
* Make a separate CDash report for each package installed
Previously, we generated a single CDash report ("build") for the complete results
of running a `spack install` command. Now we create a separate CDash build for
each package that was installed.
This commit also changes some of the tests related to CDash reporting.
Now only one of the tests exercises the code path of uploading to a
(nonexistent) CDash server. The rest of the related tests write their reports
to disk without trying to upload them.
* Don't report errors to CDash for successful packages
Convert errors detected by our log scraper into warnings when the package
being installed reports that it was successful.
* Report a maximum of 50 errors/warnings to CDash
This is in line with what CTest does. The idea is that if you have more than
50 errors/warnings you probably aren't going to read through them all anyway.
This change reduces the amount of data that we need to transfer and store.
* Update spec format to simpler syntax, maintain backwards compatibility
* Switch to new spec.format method throughout internals
* update package files for new format strings
* documentation and minor code cleanup. removed nonsensical variant sigils
Fixes#11070#11010
Spack attempts to intercede on behalf of all compiler invocations for
a build. This involves adding its wrappers to PATH. Cray systems
include a "ftn" executable and Spack was only redirecting this call
when the Spec was built with cce. This updates the compiler wrappers
to add "ftn" in all cases.
The default (implied) behavior for all environments, as of ea1de6b,
is that an environment will maintain a view in a location of its
choosing. ea1de6b explicitly recorded all three possible states of
maintaining a view:
1. Maintain a view, and let the environment decide where to put it
(default)
2. Maintain a view, and let the user decide
3. Don't maintain a view
This commit updates the config writer so that for case [1], nothing
will be written to the config.yaml. This will not change any existing
behavior, it just serves to keep the config more compact.
Compilers are treated separately from other dependencies in Spack.
#10761 added the option to automatically install compilers when a
package specifies using a compiler that is not available in Spack.
However, this did not work correctly for dependency packages (it
would only build a compiler for the root of an install DAG). This
commit enables the building of compilers for dependency packages.
Environments are nowm by default, created with views. When activated, if an environment includes a view, this view will be added to `PATH`, `CPATH`, and other shell variables to expose the Spack environment in the user's shell.
Example:
```
spack env create e1 #by default this will maintain a view in the directory Spack maintains for the env
spack env create e1 --with-view=/abs/path/to/anywhere
spack env create e1 --without-view
```
The `spack.yaml` manifest file now looks like this:
```
spack:
specs:
- python
view: true #or false, or a string
```
These commands can be used to control the view configuration for the active environment, without hand-editing the `spack.yaml` file:
```
spack env view enable
spack env view envable /abs/path/to/anywhere
spack env view disable
```
Views are automatically updated when specs are installed to an environment. A view only maintains one copy of any package. An environment may refer to a package multiple times, in particular if it appears as a dependency. This PR establishes a prioritization for which environment specs are added to views: a spec has higher priority if it was concretized first. This does not necessarily exactly match the order in which specs were added, for example, given `X->Z` and `Y->Z'`:
```
spack env activate e1
spack add X
spack install Y # immediately concretizes and installs Y and Z'
spack install # concretizes X and Z
```
In this case `Z'` will be favored over `Z`.
Specs in the environment must be concrete and installed to be added to the view, so there is another minor ordering effect: by default the view maintained for the environment ignores file conflicts between packages. If packages are not installed in order, and there are file conflicts, then the version chosen depends on the order.
Both ordering issues are avoided if `spack install`/`spack add` and `spack install <spec>` are not mixed.
When providing a track, the cdash reporter will format the stamp
itself, as it has always done, and register the build during the
package installation process. When providing a stamp, it should
first be formatted as cdash expects, and then cdash will be sure
to report results to same build id which was registered manually
elsewhere.
* Update Spec.prefix to have special case for 'None' in database path; regression test
* Update in database reader rather than spec
* Change assertion to conditional + raise
* Added test for concrete check in Spec.prefix
The module_parsing test checks whether the module function is available
by looking for the string 'not found'. If the user has set a different
locale, the test can assume that the module function is available when
it actually is not.
* Split get_compiler_version into two functions:
get_compiler_version_output runs the compiler with the relevant
option to print the version; extract_version_from_output determines
the version by examining this output. This makes it easier to test
the customized version detection for each compiler. Users can
customize this by overriding the following:
* version_argument: this is the argument that tells the compiler to
print its version. It assumes that the compiler will report its
version if invoked with a single option (like "--version")
* version_regex: the regular expression used to extract the version
from the compiler argument. This assumes that a regular
expression is sufficient to extract the version, and that the
version can be extracted from a single capture group (Spack uses
the first capture group)
* default_version: allows you to completely override all version
detection logic
* get_compiler_version_output: if getting the compiler to report
its version is more complex than invoking it with a single arg
* extract_version_from_output: if it is difficult to define a regex
that can be used to extract the version from the output
* Added tests for version detection of most compilers
* Removed redundant code from xl_r compiler class (by inheriting
from xl compiler definition)
Replace the original implementation of the "memoized" decorator with
an implementation that exposes the docstring and arguments of the
wrapped function. This is achieved using functools.wraps.
This provides a mechanism to implement a new Spack command in a
separate directory, and with a small configuration change point Spack
to the new command.
To register the command, the directory must be added to the
"extensions" section of config.yaml. The command directory name must
have the prefix "spack-", and have the following layout:
spack-X/
pytest.ini #optional, for testing
X/
cmd/
name-of-command1.py
name-of-command2.py
...
tests/ #optional
conftest.py
test_name-of-command1.py
templates/ #optional jinja templates, if needed
And in config.yaml:
config:
extensions:
- /path/to/spack-X
If the extension includes tests, you can run them via spack by adding
the --extension option, like "spack test --extension=X"
* initial work to make use of an 'upstream' spack installation: this uses the DB of the upstream installation to check if a package is installed
* need to query upstream dbs when adding new record to local db
* prevent reindexing upstream DBs
* set prefix on specs read from DB based on path stored in install record
* check that Spack does not install packages that are recorded as installed in an upstream db
* externals do not add their path to install records - need to use 'external_path' to get path of upstream externals
* views need to check for upstream installations when linking metadata
* package and spec now calculate upstream installation properties on-demand themselves rather than depending on concretization to set these properties up-front. The added tests for upstream installations don't work with this new strategy so they need to be updated
* only refresh modules for local specs (not those in upstream packages); optionally generate local module files for packages installed upstream
* when a user tries to locate a module file for a package installed upstream, tell them to use the upstream spack instance to locate it
* support recursive upstream databases (allow upstream databases to use their own upstream databases)
* separate upstream config into separate file with its own schema; each entry now also includes a name
* metadata_dir is no longer customizable on a per-instance basis for YamlDirectoryLayout
* treat metadata_dir as an instance variable but dont set it from kwargs; this follows several other hardcoded variables which must be consistent between upstream and downstream DBs. Also update DirectoryLayout.metadata_path to work entirely with Spec.prefix, since Spec.prefix is set from the DB when available (so metadata_path was duplicating that logic)
Change the location of the CMake build area from the staged source
directory to the stage base directory.
This change allows CMake packages to refer to the build directory in
setup_environment (e.g. if tests need to have a directory in PATH):
Staging happens after the call to setup_environment(), and if the
stage area does not exist, then spec.stage.source_path returns None.
To accommodate this change, archived files (like config.log for
Autotools packages) are archived relative to the stage base directory
rather than the expanded source directory.
Other packages (those not using CMake) will still use the staged
source directory as the default working directory for builds (and
will still be unable to reference this directory in
setup_environment())
When multiple instances of environment-modules were installed with
different architectures, Spack was not retrieving the installation
appropriate for the current architecture when finding the module
prefix.
* Fixed some issues with CUDA-Intel compiler conflicts.
* Comment about expressing CUDA-compiler conflicts.
* More precise conflicts and also add support for Intel 19.0
If the user has set the environment variable VISUAL, it will be used
in preference to EDITOR for all Spack editing activities. If VISUAL
is not set or fails (perhaps due to a lack of graphical editing
capabilities),EDITOR will be used instead. We fall back to one of
several common editors if neither bears fruit.
This feature has been tailored to:
* Provide identical behavior to the previous implementation in the
case that VISUAL is not set.
* Not require any change to code utilizing the editor feature.
* Follow usual UNIX behavior concerning VISUAL and EDITOR.
* Fix clearing EnvironmentModifications with python2
* Add EnvironmentModifications::clear unit test
Use re-assignment rather than del to clear array
* Fix flake issues
Fixes#10191
* Add more regular expressions to detect clang versions that were
not being picked up
* Add a test for parsing versions from the output of Clang (this
does not run Clang, but rather uses example outputs from Clang)
* Separate Clang version parsing into its own method (to make it
easier to test)
Currently, only C headers are considered, causing build failures for
packages depending on, e.g., netcdf-fortran and xerces-c. Additionally,
the regex used to look for the include path component did not consider
word boundaries, causing false matches.
* Create option to build missing compilers and add them to config before installing packages that use them
* Clean up kwarg passing for do_install, put compiler bootstrapping in separate method
* Rework of buildcache creation and install prefix checking using the functions introduced in
https://github.com/spack/spack/pull/9199
Instead of replacing rpaths with placeholder and then checking strings, make use of the functions
relocate.is_recocatable and relocate.is_file_relocatable to decide if a package needs the allow-root option.
This fixes a problem where the placeholder path was not in the first rpath entry. This was seen in c++ libraries and binaries because the compiler was outside the spack install base path and always appears first in the rpath.
Instead of checking the first rpath entry, all rpaths have the placeholder path and the old install path (if it exists) replaced with the new install path.
* flake8
* Added the `spack buildcache preview` sub-command
This is similar to `spack spec -I` but highlights which nodes in a DAG
are relocatable and which are not.
spec.tree has been generalized a little to accept a status function,
instead of always showing the install status
The current implementation works only for ELF, and needs to be
generalized to other platforms.
* Added a test to check if an executable is relocatable or not
This test requires a few commands to be present in the environment.
Currently it will run only under python 3.7 (which uses Xenial instead
of Trusty).
* Added tests for the 'buildcache preview' command.
* Fixed codebase after rebase
* Fixed the list of apt addons for Python 3.7 in travis.yaml
* Only check ELF executables and shared libraries. Skip checking virtual or external packages. (#229)
* Fixed flake8 issues
* Add handling for macOS mach binaries (#231)
The environment modules package has been updated to include
versions up to 4.0.0. The url of the package and the homepage
have been updated accordingly.
The `spack bootstrap` command now builds version 3.2.10 of
the environment-modules package, and will do until #10708
is fixed.
This restores the use of Package.headers when computing -I options
for building a package that was added in #8136 and reverted in
#10604. #8136 used utility logic that located all header files in
an installation prefix, and calculated the -I options as the
immediate roots containing those header files.
In some cases, for a package containing a directory structure like
prefix/
include/
ex1.h
subdir/
ex2.h
dependents may expect to include ex2.h relative to 'include', and
adding 'prefix/include/subdir' as a -I was causing errors,
in particular if ex2.h has the same name as a system header.
This updates header utility logic to by default return the base
"include" directory when it exists, rather than subdirectories.
It also makes it possible for package implementers to override
Package.headers to return the subdirectory when it is required
(for example with libxml2).
Spack warns users when a dependency package updates CPATH. This
warning message is generating bug reports and alarm in cases where
there is no problem. For now this downgrades the warning message to
the debug level, so it only shows up if something goes wrong for the
user and they ask for more information from Spack.
This spack command adds a new schema for a file which describes the
builder containers available, along with the compilers availabe on
each builder. The release-jobs command then generates the .gitlab-ci.yml
file by first expanding the release spec set, concretizing each spec
(in an appropriate docker container if --this-machine-only argument is
not provided on command line), and then combining and staging all the
concrete specs as jobs to be run by gitlab.
Adds four new sub-commands to the buildcache command:
1. save-yaml: Takes a root spec and a list of dependent spec names,
along with a directory in which to save yaml files, and writes out
the full spec.yaml for each of the dependent specs. This only needs
to concretize the root spec once, then indexes it with the names of
the dependent specs.
2. check: Checks a spec (via either an abstract spec or via a full
spec.yaml) against remote mirror to see if it needs to be rebuilt.
Comparies full_hash stored on remote mirror with full_hash computed
locally to determine whether spec needs to be rebuilt. Can also
generate list of specs to check against remote mirror by expanding
the set of release specs expressed in etc/spack/defaults/release.yaml.
3. get-buildcache-name: Makes it possible to attempt to read directly
the spec.yaml file on a remote or local mirror by providing the path
where the file should live based on concretizing the spec.
4. download: Downloads all buildcache files associated with a spec
on a remote mirror, including any .spack, .spec, and .cdashid files
that might exist. Puts the files into the local path provided on
the command line, and organizes them in the same hierarchy found on
the remote mirror
This commit also refactors lib/spack/spack/util/web.py to expose
functionality allowing other modules to read data from a url.
- add CombinatorialSpecSet in spack.util.spec_set module.
- class is iterable and encaspulated YAML parsing and validation.
- Adjust YAML format to be more generic
- YAML spec-set format now has a `matrix` section, which can contain
multiple lists of specs, generated different ways. Including:
- specs: a raw list of specs.
- packages: a list of package names and versions
- compilers: a list of compiler names and versions
- All of the elements of `matrix` are dimensions for the build matrix;
we take the cartesian product of these lists of specs to generate a
build matrix. This means we can add things like [^mpich, ^openmpi]
to get builds with different MPI versions. It also means we can
multiply the build matrix out with lots of different parameters.
- Add a schema format for spec-sets
Fixes#10617Fixes#10624Closes: #10619#8136 dependended entirely on spec.libs to retrieve library directories
from dependencies. By default this function only retrieves libraries if
their name is something like lib<package> (e.g. "libfoo.so" for a
package called "Foo"). This unconditionally adds lib/lib64 directories
for each dependency as link/rpath directories.
This also filters system paths from link/rpaths/include directories and
removes duplicated paths that #8136 could add.
If the -f <specyamlfile> argument to install is used (rather than
providing package specs on the command line), CDash throws an exception
due to missing the installation command (the packages targeted for
install). This fixes that behavior so CDash reporting succeeds in
either case.
fixes#10601
Due to a bug this attribute is wrong for packages that use directories
as namespaces. For instance it will add "<boost-prefix>/include/boost"
instead of "<boost-prefix>/include" to the include path.
As a minor addition a few loops in the compiler wrappers have been
simplified.
Fixes#7855Closes#8070Closes#2645
When searching for library directories (e.g. to add "-L" arguments to
the compiler wrapper) Spack was only trying the "lib/" and "lib64/"
directories for each dependency install prefix; this missed cases
where packages would install libraries to subdirectories and also was
not customizable. This PR makes use of the ".headers" and ".libs"
properties for more-advanced location of header/library directories.
Since packages can override the default behavior of ".headers" and
".libs", it also allows package writers to customize.
The following environment variables which used to be set by Spack
for a package build have been removed:
* Remove SPACK_PREFIX and SPACK_DEPENDENCIES environment variables as
they are no-longer used
* Remove SPACK_INSTALL environment variable: it was not used before
this PR
* fix permission setter
Fix a typo in islink test when applied to files.
* os.walk explicitly set not to follow links
The algorithm strongly rely on not following links.
* Note that `none` is the default for lmod autoload
Save a bit of confusion by *explicitly* pointing out that `none` is
the default value for autoload in the lmod module file generator.
* Add a tip re building software externally
Add a tip about using `autoload: all` when building packages outside
of the tree that use artifacts (e.g. libraries, includes) within the
tree.
CMake supports the notion of secondary generators which provide extra
information to (e.g.) IDEs over and above that normally provided by
the primary generator. Spack only supports the 'Unix Makefiles' and
'Ninja' primary generators but was not parsing out the primary
generator when a secondary generator was also included (e.g. for
a generator attribute like 'Codeblocks - Ninja'). This adds a regex
for extracting the primary generator for validation.
Since the secondary generator is irrelevant to a Spack build, it is
passed on to CMake without further validation.
* CUDA compiler conflicts for Linux.
* Add Volta and Turing GPUs.
* Add mandatory conflict for Volta and Turing GPUs.
* Revert "CUDA compiler conflicts for Linux."
This reverts commit 7d4ff654ac53aad272c59e9f7f8bb3fbb32bcec4.
* Compiler conflicts introduced from previous commit into CUDA packaged moved and integrated into CUDA build system.
* More conversative with compiler conflicts for cuda 10.0.130, since I don't know what will happen with future cuda 10.x releases.
* Correct off-by-one errors in clang conflicts for x86_64 Linux.
* No restrictions on Apple Clang compiler until we are able to distinguish Xcode clang from github clang more easily. Note to fix this in the future.
* Change comment to clarify that github clang refers to LLVM clang.
* Fix and simplify index range.
* Fix overlapping conflicts for CUDA 10.0.130
* Removed extra ^cuda from conflict.
Debug output now includes the output of modulecmd executions. Only
output module content when a failure occurs; always report when a
module is loaded/unloaded.
"spack install" will install all packages added to the current
environment. When this included external packages, the environment
update would fail because it would attempt to copy log files that
were only generated if Spack handled the install itself. This skips
that step for external packages.
* Allow overwrite nonexistent and multiple packages
initial implementation
give one prompt to users instead of a prompt per spec
testing
* flake
* bugfix: install overwrite check each spec against installed
* python3 compliance for filter/map
* Remove Cray CC compilers causing problems on case-insensitive filesystems
* cray -> cce
* Ensure that compiler-specific directory comes first in build-env
* Point to compiler-specific symlinks
Binary caches of packages with absolute symlinks had broken symlinks.
As a stopgap measure, #9747 addressed this by replacing symlinks with
copies of files when creating binary cached packages.
This reverts #9747 and instead, either relative-izes the symlink or
rewrites the target. If the binary cache is created using '--rel' (as
in "spack buildcache create --rel...") then absolute symlinks will be
replaced with relative symlinks (in addition to making RPATHs relative
as before); otherwise they are rewritten (when the binary cache is
unpacked and installed).
The current output of buildcache list is very verbose and I feel like
some details are getting lost. By making the output similar to find, I
think users will be able to get a better overview of what is stored in
the cache.
* dealii: fix concretization of xsdk package
* tests: add concretization tests for deal.II and xSDK, which are often broken due to limitations in the concretizer
* use pytest.mark.parametrize
Allow customizing views with Spec-formatted directory structure
Allow views to specify projections that are more complicated than
merging every package into a single shared prefix. This will allow
sites to configure a view for the way they want to present packages
to their users; for example this can be used to create a prefix for
each package but omit the DAG hash from the path.
This includes a new YAML format file for specifying the simplified
prefix for a spec in a view. This configuration allows the use of
different prefix formats for different specs (i.e. specs depending
on MPI can include the MPI implementation in the prefix).
Documentation on usage of the view projection configuration is
included.
Depending on the projection configuration, paths are not guaranteed
to be unique and it may not be possible to add multiple installs of
a package to a view.
Fixes#10284#10152 replaced shutil.move with llnl's copy and copy_tree for
resources. This did not copy permissions so led to later failures
if an executable was copied (e.g. a configure script). This uses
install/install_tree instead, which preserve permissions.
* Initial compiler support
* added arm.py
* Changed licence to Arm suggested header
* Changed licence to the same as clang.py
Main author of file is Nick Forrington <Nick.Forrington@arm.com>
Minor changes by Srinath Vadlamani <srinath.vadlamani@arm.com>
* compilers: add arm compiler detection to Spack
- added arm.py with support for detecting `armclang` and `armflang`
Co-authored-by: Srinath Vadlamani <srinath.vadlamani@arm.com>
* Changed to using get get_compiler_version
* linking to general cc for arm compiler
* For arm compiler add CFLAGS to use compiler-rt rtlib.
* Escape for special characters in rexep
* Cleaned up for Flake8 to pass.
* libcompiler-rt should be part of the LDFLAGS not CFLAGS
* fixed m4 when using clang to used LDFLAGS. Fixed comments for arm.py to display compiler --version output with # NOAQ for flakes pass.
* added arm compilers
* proper linked names
This enforces conventions that allow for correct handling of
multi-valued variants where specifying no value is an option,
and adds convenience functionality for specifying multi-valued
variants with conflicting sets of values. This also adds a notion
of "feature values" for variants, which are those that are understood
by the build system (e.g. those that would appear as configure
options). In more detail:
* Add documentation on variants to the packaging guide
* Forbid usage of '' or None as a possible variant value, in
particular as a default. To indicate choosing no value, the user
must explicitly define an option like 'none'. Without this,
multi-valued variants with default set to None were not parsable
from the command line (Fixes#6314)
* Add "disjoint_sets" function to support the declaration of
multi-valued variants with conflicting sets of options. For example
a variant "foo" with possible values "a", "b", and "c" where "c"
is exclusive of the other values ("foo=a,b" and "foo=c" are
valid but "foo=a,c" is not).
* Add "any_combination_of" function to support the declaration of
multi-valued variants where it is valid to choose none of the
values. This automatically defines "none" as an option (exclusive
with all other choices); this value does not appear when iterating
over the variant's values, for example in "with_or_without" (which
constructs autotools option strings from variant values).
* The "disjoint_sets" and "any_combination_of" methods return an
object which tracks the possible values. It is also possible to
indicate that some of these values do not correspond to options
understood by the package's build system, such that methods like
"with_or_without" will not define options for those values (this
occurs automatically for "none")
* Add documentation for usage of new functions for specifying
multi-valued variants
Non-expanded resources were being deleted from the cache on account
of two behaviors:
* ResourceStage was moving files rather than copying them, and uses
"os.path.realpath" to resolve symlinks
* CacheFetchStrategy creates a symlink to a cached resource rather
than copying it
This alters the first behavior: ResourceStage now copies the file
rather than moving it.
"mirror create" was invoking a package's do_patch method in order to
retrieve and archive URL patches. If a package implements a "patch"
method, this is also called as part of do_patch; this failed when the
package-specific implementation referred to environment variables
that are only available at the time the package is built
(e.g. "spack_cc").
This change introduces fetch and clean methods for patches. They are
no-ops for FilePatch but perform the appropriate actions for
UrlPatch. This allows "mirror create" to invoke do_fetch, which does
not call the package's patch method.
