This is similar to #10191. The Ubuntu package for clang 8.0.0 displays
a very unusual version string, and we need this new regex to detect it
as just 8.0.0
Unit test have been complemented by the output that was failing
detection.
- Fix trailing whitespace missed by the bug described in #12755.
- Fix other style issues that have crept in over time (this can happen
when flake8 adds new checks with new versions)
E501 (line too long) exemptions are probably our most common ones -- we
add them for directives, URLs, hashes, etc. in packages. But we
currently add them even when a line *doesn't* need them, which can mask
trailing whitespace errors.
This changes `spack flake8` so that it will only add E501 exemptions if
the line is *actually* too long.
Co-Authored-By: Adam J. Stewart <ajstewart426@gmail.com>
mock_archive can now take multiple extension / tar option pairs (default matches old behavior).
url_fetch.test_fetch tests more archive types.
compression.EXTS split into EXTS and NOTAR_EXTS to avoid unwanted, non-meaningful combinatoric extensions such as .tar.tbz2.
- previously spec parsing didn't allow you to look up missing (but still
known) specs by hash
- This allows you to reference and potentially reinstall
force-uninstalled dependencies
- add testing for force uninstall and for reference by spec
- cmd/install tests now use mutable_database
* When cleaning the stage root, only remove directories that appear
to be used for staging Spack packages. Previously Spack was clearing
all directories in the stage root, which could remove content not
related to Spack if the user chose a staging root which contains
files/directories not managed by Spack.
* The documentation is updated with warnings about choosing a stage
directory that is only managed by Spack (although generally the
check added in this PR for "spack clean" should avoid removing
content that was not created by Spack)
* The default stage directory (in config.yaml) is now
$tempdir/$user/spack-stage and the logic is updated to omit the
$user portion of this path if $tempdir already contains a $user
directory.
* When creating stage root assign user read/write permissions to all
directories in the path under $user. Previously Spack was assigning
the permissions of the first existing parent directory
`spec.prefix` reads from Spack's database, and if you do this with
multiple consecutive read transactions, it can take a long time. Or, at
least, you can see the paths get written out one by one.
This uses an outer read transaction to ensure that actual disk locks are
acquired only once for the whole `spack find` operation, and that each
transaction inside `spec.prefix` is an in-memory operation. This speeds
up `spack find -p` a lot.
Refactor `spack.cmd.display_specs()` and `spack find` so that any options
can be used together with -d. This cleans up the display logic
considerably, as there are no longer multiple "modes".
This is another machine-readable version of `spack find`. Supplying the
`--json` argument causes specs to be written out as json records,
easily filered with tools like jq.
e.g.:
$ spack find --json python | jq -C ".[] | { name, version } "
[
{
"name": "python",
"version": "2.7.16"
},
{
"name": "bzip2",
"version": "1.0.8"
}
]
- spack find --format allows you to supply a format string and have specs
output in a more machine-readable way, without dedcoration
e.g.:
spack find --format "{name}-{version}-{hash}"
autoconf-2.69-icynozk7ti6h4ezzgonqe6jgw5f3ulx4
automake-1.16.1-o5v3tc77kesgonxjbmeqlwfmb5qzj7zy
bzip2-1.0.6-syohzw57v2jfag5du2x4bowziw3m5p67
...
or:
spack find --format "{hash}"
icynozk7ti6h4ezzgonqe6jgw5f3ulx4
o5v3tc77kesgonxjbmeqlwfmb5qzj7zy
syohzw57v2jfag5du2x4bowziw3m5p67
...
This is intended to make it much easier to script with `spack find`
When Spack installs a package it writes the package.py file and
patches to a separate repository (which reflects the state of the
package at the time it was installed). Previously, Spack only wrote
patches that were used at installation time. This updates the
archiving step to include all patch files that are relevant to the
package (in case that repository is used in another context).
This commit removes redundant calls to `libtoolize` and `aclocal`.
Some configurations, such as a Spack user using macOS with a
Homebrew-installed `libtool` added to their `packages.yaml`, have
`autoreconf` and GNU libtoolize installed as `glibtoolize`, but not
`libtoolize`. While Spack installations of `libtool` built from source
would install `glibtoolize` and symlink `libtoolize` to `glibtoolize`,
an external installation of GNU libtoolize as `glibtoolize` will not
have such a symlink, and thus the call `m.libtoolize()` will throw an
error because `libtoolize` does not exist at the path referenced by
`m.libtoolize()` (i.e.,
`self.spec['libtool'].prefix.bin.join('libtoolize')).
However, on these same systems, `autoreconf` runs correctly, and calls
`glibtoolize` instead of `libtoolize`, when appropriate. Thus,
removing the call to `libtoolize` should resolve the error mentioned
above.
The redundant call to `aclocal` is also removed in this commit because
the maintainers of GNU Automake state that "`aclocal` is expected to
disappear" and suggest that downstream users never call `aclocal`
directly -- rather, they suggest calling `autoreconf` instead.
Uses code from CMake to detect implicit link paths from compilers
System paths are filtered out of implicit link paths
Implicit link paths added to compiler config and object under `implicit_rpaths`
Implicit link paths added as rpaths to compile line through env/cc wrapper
Authored by: "Ben Boeckel <ben.boeckel@kitware.com>"
Co-authored by: "Peter Scheibel <scheibel1@llnl.gov>"
Co-authored by: "Gregory Becker <becker33@llnl.gov>"
c9e214f updated template creation by passing **kwargs to package
template classes but the template classes were not updated to accept
them; this adds **kwargs to package template initializers where they
are needed.
Having a non-directory invisible file causes `spack find` to die. This
fixes the logic to ignore invalid module names but only warn if they're
visible.
```
NotADirectoryError: [Errno 20] Not a directory: '/spack/var/spack/repos/builtin/packages/.DS_Store/package.py'
```
This adds a special package type to Spack which is used to aggregate
a set of packages that a user might commonly install together; it
does not include any source code itself and does not require a
download URL like other Spack packages. It may include an 'install'
method to generate scripts, and Spack will run post-install hooks
(including module generation).
* Add new BundlePackage type
* Update the Xsdk package to be a BundlePackage and remove the
'install' method (previously it had a noop install method)
* "spack create --template" now takes "bundle" as an option
* Rename cmd_create_repo fixture to "mock_test_repo" and relocate it
to shared pytest fixtures
* Add unit tests for BundlePackage behavior
This allows "spack spec --yaml" to generate a spec YAML file that can
be used with "spack install -f". Before, this would fail in cases
where the spec had build dependencies.
* All fetch strategies now accept the Boolean version keyword option `no_cache` in order to allow per-version control of cache-ability.
* New git-specific version keyword option `get_full_repo` (Boolean). When true, disables the default `--depth 1` and `--single-branch` optimizations that are applied if supported by the git version and (in the former case) transport protocol.
* The try / catch blog attempting `--depth 1` and retrying on failure has been removed in favor of more accurately ascertaining when the `--depth` option should work based on git version and protocol choice. Any failure is now treated as a real problem, and the clone is only attempted once.
* Test improvements:
* `mock_git_repository.checks[type_of_test].args['git']` is now specified as the URL (with leading `file://`) in order to avoid complaints when using `--depth`.
* New type_of_test `tag-branch`.
* mock_git_repository now provides `git_exe`.
* Improved the action of the `git_version` fixture, which was previously hard-wired.
* New tests of `--single-branch` and `--depth 1` behavior.
* Add documentation of new options to the packaging guide.
- mkdirp now takes arguments to allow it to properly set permissions on created directories.
- Two arguments (group and mode) set permissions for the leaf directory.
- Intermediate directories can inherit permissions from either the topmost existing directory (the parent) or the leaf.
On machines where $TMP is owned by a gid with no name, this avoids the
following error when the default spack stage does not exist:
(spackbook):spack$ spack clean
==> Removing all temporary build stages
==> Error: 'getgrgid(): gid not found: 57095'
Spack needs to deal with gids directly unless users pass them in.
Compiler caching was using the `id()` function to refer to configuration dictionary objects. If these objects are garbage-collected, this can produce incorrect results (false positive cache hits). This change replaces `id()` with an object that keeps a reference to the config dictionary so that it is not garbage-collected.
Fixes#11163
The goal of this work is to simplify stage directory structures by eliminating use of symbolic links. This means, among other things, that` $spack/var/spack/stage` will no longer be the core staging directory. Instead, the first accessible `config:build_stage` path will be used.
Spack will no longer automatically append `spack-stage` (or the like) to configured build stage directories so the onus of distinguishing the directory from other work -- so the other work is not automatically removed with a `spack clean` operation -- falls on the user.
Fixes#12062406c791 addressed "spack module load" for upstream modules but not
the "spack module loads" command. This applies the same fixes from
406c791 to "spack module loads".
It's no longer possible to set compiler flags under as an entry under
"paths" in compilers.yaml; instead the user must list these under the
"flags" section. This updates the docs accordingly.
Spack stacks drop invalid dependencies applied to packages by a
spec_list matrix operation
Without this fix, Spack would raise an error if orthogonal dependency
constraints and non-dependency constraints were applied to the same
package by a matrix and the dependency constraint was invalid for
that package. This is an error, fixed by this PR.
An example failing configuration:
spack:
definitions:
- packages: [libelf, hdf5+mpi]
- compilers: ['%gcc']
- mpis: [^openmpi]
specs:
- matrix:
- $packages
- $compilers
- $mpis
5f74f22 enabled installing compilers for dependencies but not for the root package (and in particular not for DAGs which consist of one package)
this enables bootstrapping compilers for both types of DAGs
Using "compilers" with the "s" is an invalid config section and throws an error.
Traceback (most recent call last):
File "spack/bin/spack", line 48, in <module>
sys.exit(spack.main.main())
File "/home/omsai/src/libkmap/spack/lib/spack/spack/main.py", line 633, in main
env = ev.find_environment(args)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 263, in find_environment
return Environment(env)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 534, in __init__
self._read_manifest(f)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 561, in _read_manifest
self.yaml = _read_yaml(f)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 402, in _read_yaml
validate(data, filename)
File "/home/omsai/src/libkmap/spack/lib/spack/spack/environment.py", line 395, in validate
e, data, filename, e.instance.lc.line + 1)
spack.config.ConfigFormatError: /home/omsai/src/libkmap/spack.yaml:15: Additional properties are not allowed ('compilers' was unexpected)
Environment.concretize returns newly-concretized specs rather than
printing them; as a result, the _display argument is removed from
Environment.concretize (originally only used to avoid printing specs
during unit testing). Command logic which invokes
Environment.concretize prints these explicitly.
This updates the Spack QT package to enable building qt version 4 on
MacOS.
This includes the following changes to the qt package:
* add version 4.8.7
* add option to build with or without shared libs
* add options to disable tools, ssl, sql, and freetype support
* add qt4-tools patch when building qt@4+tools
* add option to build as a framework (only available on MacOS)
* replace qt4-el-capitan patch with qt4-mac patch (which includes the
edits from qt4-el-capitan)
* apply qt4-pcre-include-conflict.patch only for version 4.8.6
(rather than all 4.x versions)
* apply qt4-gcc-and-webkit.patch for 4.x versions before 4.8.7 and
create a separate qt4-gcc-and-webkit-487.patch for version 4.8.7
* update patch function for qt@4 on MacOS to update configure
variables relevant to Spack (e.g. PREFIX)
* add option to build freetype with Spack, as a vendored dependency
of QT, or not at all (default is to build with Spack)
This includes the following edits outside of the qt package:
* Update MacOS version utility function to return all parts of the
Mac version (rather than just the first two)
* gettext package: implement "libs"
* python package: add gettext as a dependency
* Raise an exception and exit with a meaningful message when binary path substitution fails.
* Skip binary text replacement with padding and issue a warning when the new install path is longer than the old install path.
- We don't currently make enough use of the maintainers field on
packages, though we could use it to assign reviews.
- add a command that allows maintainers to be queried
- can ask who is maintaining a package or packages
- can ask what packages users are maintaining
- can list all maintained or unmaintained packages
- add tests for the command
* Added a unit test reproducing the failure in 12085
* Fixed name clash in the 'from_environment_diff' function
The bug reported in #12085 stemmed from a name clash among variables,
introduced during the refactor in #10753 and not caught by unit tests
and reviews.
- Setting specs from lockfiles was not correctly stringifying concretized
user specs.
- Fix `_set_user_specs_from_lockfile`
- Add some validation code to `SpecList` constructor
Spack has evolved to have three types of hash functions, and it's
becoming hard to tell when each one is called. Whlie we aren't yet ready
to get rid of them, we can refactor them so that the code is clearer and
easier to track.
- Add a `hash_types` module with concise descriptors for hashes.
- Consolidate hashing logic in a private `Spec._spec_hash()` function.
- `dag_hash()`, `build_hash()`, and `full_hash()` all call `_spec_hash()`
- `to_node_dict()`, `to_dict()`, `to_yaml()` and `to_json()` now take a
`hash` parameter consistent with the one that `_spec_hash()` requires.
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
- ensure that Spec._build_hash attr is defined
- add logic to compute _build_hash when it is not already computed (e.g. for specs prior to this PR)
- add test to ensure that different instance of a build dep are preserved
- test conversion of old env lockfile format to new format
- tests: avoid view creation, DAG display in tests using MockPackage
- add regression test for more-general bug also fixed by this PR
- update lockfile version since the way we are maintaining hashes has changed
- write out backup for version-1 lockfiles and test that
The database and mutable_database fixtures were installing and uninstalling the same specs multiple times to ensure the database for tests has the correct state.
This commit optimizes the procedure by caching the state in an external directory, and copying it in instead of going through the installation or uninstallation again.
The database fixture is meant not to be modified by tests. This commit enforces this invariant by making the database read-only before starting the test.
* Added missing db markers to tests
* Added test for uninstall_by_spec
* `database` fixture now returns a read-only database
* Tests that modify the DB now use `mutable_database` fixture
Summary:
- Allow multiple definitions of compiler in compilers.yaml (use first instance)
- Still print debug messages when there are duplicates, to assist users in finding this issue.
Merging configs from different scopes can result in multiple compiler being present in the same configuration list. Instead of raising when there are duplicates, take the one with highest precedence.
Print a debug message instead of raising, so that we can still diagnose this. We don't have a good way of warning the user about inconsistent configuration *in the same file* -- we'd need to dig into YAML file/line info for that.
- [x] Add shell tests to ensure that `spack env activate`, `spack env
deactivate`, and `despacktivate` continue to work.
- [x] Also ensure that activate and deactivate both work with `set -u`
* extends mkdirs with permissions for intermediate folders
Does not use os.makedirs mode parameter because its behavior is changed
with Python 3.7 (it ignores it for intermediate dirs), and moreover it
was not possible to set different modes for newly-created folders
and leaf folder.
reference:
- https://bugs.python.org/issue19930
- https://docs.python.org/3.7/library/os.html#os.makedirs
* comment mkdirp step easing code understanding
* revert mkdir to default for package metapath
since metapath is nested in package folder, there is no need
to specify permissions for intermediate folders because the prefix
already exists.
* comment create_install_directory package modes
Bug relates to the interplay between:
1. random dict orders in python 3.5
2. bugfix in initial implementation of stacks for `_concretize_dependencies`
when `self._dependencies` is empty
3. bug in coconcretization algorithm computation of split specs
Result was transient hang in coconcretization.
Fixed#3 (bug in coconcretization) to resolve.
- remove redundant code in Environment.__init__
- use socket.gethostname() instead of which('hostname')
- refactor updating SpecList references
- refactor 'specs' literals to a single variable for default list name
- use six.string_types for python 2/3 compatibility
