Fixes#7855Closes#8070Closes#2645
When searching for library directories (e.g. to add "-L" arguments to
the compiler wrapper) Spack was only trying the "lib/" and "lib64/"
directories for each dependency install prefix; this missed cases
where packages would install libraries to subdirectories and also was
not customizable. This PR makes use of the ".headers" and ".libs"
properties for more-advanced location of header/library directories.
Since packages can override the default behavior of ".headers" and
".libs", it also allows package writers to customize.
The following environment variables which used to be set by Spack
for a package build have been removed:
* Remove SPACK_PREFIX and SPACK_DEPENDENCIES environment variables as
they are no-longer used
* Remove SPACK_INSTALL environment variable: it was not used before
this PR
* fix permission setter
Fix a typo in islink test when applied to files.
* os.walk explicitly set not to follow links
The algorithm strongly rely on not following links.
* Note that `none` is the default for lmod autoload
Save a bit of confusion by *explicitly* pointing out that `none` is
the default value for autoload in the lmod module file generator.
* Add a tip re building software externally
Add a tip about using `autoload: all` when building packages outside
of the tree that use artifacts (e.g. libraries, includes) within the
tree.
CMake supports the notion of secondary generators which provide extra
information to (e.g.) IDEs over and above that normally provided by
the primary generator. Spack only supports the 'Unix Makefiles' and
'Ninja' primary generators but was not parsing out the primary
generator when a secondary generator was also included (e.g. for
a generator attribute like 'Codeblocks - Ninja'). This adds a regex
for extracting the primary generator for validation.
Since the secondary generator is irrelevant to a Spack build, it is
passed on to CMake without further validation.
* CUDA compiler conflicts for Linux.
* Add Volta and Turing GPUs.
* Add mandatory conflict for Volta and Turing GPUs.
* Revert "CUDA compiler conflicts for Linux."
This reverts commit 7d4ff654ac53aad272c59e9f7f8bb3fbb32bcec4.
* Compiler conflicts introduced from previous commit into CUDA packaged moved and integrated into CUDA build system.
* More conversative with compiler conflicts for cuda 10.0.130, since I don't know what will happen with future cuda 10.x releases.
* Correct off-by-one errors in clang conflicts for x86_64 Linux.
* No restrictions on Apple Clang compiler until we are able to distinguish Xcode clang from github clang more easily. Note to fix this in the future.
* Change comment to clarify that github clang refers to LLVM clang.
* Fix and simplify index range.
* Fix overlapping conflicts for CUDA 10.0.130
* Removed extra ^cuda from conflict.
Debug output now includes the output of modulecmd executions. Only
output module content when a failure occurs; always report when a
module is loaded/unloaded.
"spack install" will install all packages added to the current
environment. When this included external packages, the environment
update would fail because it would attempt to copy log files that
were only generated if Spack handled the install itself. This skips
that step for external packages.
* Allow overwrite nonexistent and multiple packages
initial implementation
give one prompt to users instead of a prompt per spec
testing
* flake
* bugfix: install overwrite check each spec against installed
* python3 compliance for filter/map
* Remove Cray CC compilers causing problems on case-insensitive filesystems
* cray -> cce
* Ensure that compiler-specific directory comes first in build-env
* Point to compiler-specific symlinks
Binary caches of packages with absolute symlinks had broken symlinks.
As a stopgap measure, #9747 addressed this by replacing symlinks with
copies of files when creating binary cached packages.
This reverts #9747 and instead, either relative-izes the symlink or
rewrites the target. If the binary cache is created using '--rel' (as
in "spack buildcache create --rel...") then absolute symlinks will be
replaced with relative symlinks (in addition to making RPATHs relative
as before); otherwise they are rewritten (when the binary cache is
unpacked and installed).
The current output of buildcache list is very verbose and I feel like
some details are getting lost. By making the output similar to find, I
think users will be able to get a better overview of what is stored in
the cache.
* dealii: fix concretization of xsdk package
* tests: add concretization tests for deal.II and xSDK, which are often broken due to limitations in the concretizer
* use pytest.mark.parametrize
Allow customizing views with Spec-formatted directory structure
Allow views to specify projections that are more complicated than
merging every package into a single shared prefix. This will allow
sites to configure a view for the way they want to present packages
to their users; for example this can be used to create a prefix for
each package but omit the DAG hash from the path.
This includes a new YAML format file for specifying the simplified
prefix for a spec in a view. This configuration allows the use of
different prefix formats for different specs (i.e. specs depending
on MPI can include the MPI implementation in the prefix).
Documentation on usage of the view projection configuration is
included.
Depending on the projection configuration, paths are not guaranteed
to be unique and it may not be possible to add multiple installs of
a package to a view.
Fixes#10284#10152 replaced shutil.move with llnl's copy and copy_tree for
resources. This did not copy permissions so led to later failures
if an executable was copied (e.g. a configure script). This uses
install/install_tree instead, which preserve permissions.
* Initial compiler support
* added arm.py
* Changed licence to Arm suggested header
* Changed licence to the same as clang.py
Main author of file is Nick Forrington <Nick.Forrington@arm.com>
Minor changes by Srinath Vadlamani <srinath.vadlamani@arm.com>
* compilers: add arm compiler detection to Spack
- added arm.py with support for detecting `armclang` and `armflang`
Co-authored-by: Srinath Vadlamani <srinath.vadlamani@arm.com>
* Changed to using get get_compiler_version
* linking to general cc for arm compiler
* For arm compiler add CFLAGS to use compiler-rt rtlib.
* Escape for special characters in rexep
* Cleaned up for Flake8 to pass.
* libcompiler-rt should be part of the LDFLAGS not CFLAGS
* fixed m4 when using clang to used LDFLAGS. Fixed comments for arm.py to display compiler --version output with # NOAQ for flakes pass.
* added arm compilers
* proper linked names
This enforces conventions that allow for correct handling of
multi-valued variants where specifying no value is an option,
and adds convenience functionality for specifying multi-valued
variants with conflicting sets of values. This also adds a notion
of "feature values" for variants, which are those that are understood
by the build system (e.g. those that would appear as configure
options). In more detail:
* Add documentation on variants to the packaging guide
* Forbid usage of '' or None as a possible variant value, in
particular as a default. To indicate choosing no value, the user
must explicitly define an option like 'none'. Without this,
multi-valued variants with default set to None were not parsable
from the command line (Fixes#6314)
* Add "disjoint_sets" function to support the declaration of
multi-valued variants with conflicting sets of options. For example
a variant "foo" with possible values "a", "b", and "c" where "c"
is exclusive of the other values ("foo=a,b" and "foo=c" are
valid but "foo=a,c" is not).
* Add "any_combination_of" function to support the declaration of
multi-valued variants where it is valid to choose none of the
values. This automatically defines "none" as an option (exclusive
with all other choices); this value does not appear when iterating
over the variant's values, for example in "with_or_without" (which
constructs autotools option strings from variant values).
* The "disjoint_sets" and "any_combination_of" methods return an
object which tracks the possible values. It is also possible to
indicate that some of these values do not correspond to options
understood by the package's build system, such that methods like
"with_or_without" will not define options for those values (this
occurs automatically for "none")
* Add documentation for usage of new functions for specifying
multi-valued variants
Non-expanded resources were being deleted from the cache on account
of two behaviors:
* ResourceStage was moving files rather than copying them, and uses
"os.path.realpath" to resolve symlinks
* CacheFetchStrategy creates a symlink to a cached resource rather
than copying it
This alters the first behavior: ResourceStage now copies the file
rather than moving it.
"mirror create" was invoking a package's do_patch method in order to
retrieve and archive URL patches. If a package implements a "patch"
method, this is also called as part of do_patch; this failed when the
package-specific implementation referred to environment variables
that are only available at the time the package is built
(e.g. "spack_cc").
This change introduces fetch and clean methods for patches. They are
no-ops for FilePatch but perform the appropriate actions for
UrlPatch. This allows "mirror create" to invoke do_fetch, which does
not call the package's patch method.
- in many files, regular strings were used in places where raw strings
should've been used.
- convert these to raw strings and get rid of new flake8 errors
This PR improves the validation of `modules.yaml` by introducing a custom validator that checks if an attribute listed in `properties` or `patternProperties` is a valid spec. This new check applied to the test case in #9857 gives:
```console
$ spack install szip
==> Error: /home/mculpo/.spack/linux/modules.yaml:5: "^python@2.7@" is an invalid spec [Invalid version specifier]
```
Details:
* Moved the set-up of a custom validator class to spack.schema
* In Spack we use `jsonschema` to validate configuration files
against a schema. We also need custom validators to enforce
writing default values within "properties" or "patternProperties"
attributes.
* Currently, validators were customized at the place of use and with the
recent introduction of environments that meant we were setting-up and
using 2 different validator classes in two different modules.
* This commit moves the set-up of a custom validator class in the
`spack.schema` module and refactors the code in `spack.config` and
`spack.environments` to use it.
* Added a custom validator to check if an attribute is a valid spec
* Added a custom validator that can be used on objects, which yields an
error if the attribute is not a valid spec.
* Updated the schema for modules.yaml
* Updated modules.yaml to fix a few inconsistencies:
- a few attributes were not tested properly using 'anyOf'
- suffixes has been updated to also check that the attribute is a spec
- hierarchical_scheme has been updated to hierarchy
* Removed $ref from every schema
* $ref is not composable or particularly legible
* Use python dicts and regular old variables instead.
- The nested directive implementation was broken for python 3
- directive results were not properly removed from the directive list
when it was processed in the DirectiveMeta metaclass.
- the issue was that remove_directives only descended into a list or
tuple, but in Python3, the initial value passed to the function is a
view of dictionary values.
- make it a list to fix things, and add a regression test.
- currently just looks at patches
- allows you to find out which package applied a patch to a spec
- intended to work with tarballs and resources in the future.
- add tab completion for `spack resource` and subcommands
- previously, if a concrete sub-DAG with patched specs was written out
and read back in, its patches would not be found because the dependent
that patched it was no longer in the DAG.
- Add a test to ensure that the PatchCache handles this case.
- Also add tests to ensure that patch objects are properly created from
Specs -- previously we only checked that the patches were on the Spec.
- this fixes a bug where if we save a concretized sug-DAG where a package
had been patched by a dependent, and the dependent was not in the DAG,
we would not read in all patches correctly.
- Rather than looking up patches in the DAG, we look them up globally
from an index created from the entire repository.
- The patch cache is a bit tricky for several reasons:
- we have to cache information from packages, specifically, the patch
level and working directory.
- FilePatches need to know which package owns them, so that they can
figure out where the patch lives. The repo can change locations from
run to run, so we have to store relative paths and restore them when
the cache is reloaded.
- Patch files can change underneath the cache, because repo indexes
only update on package changes. We currently punt on this -- there
are stub methods for needs_update() that will need to check patch
files when packages are loaded. There isn't an easy way to do this
at global indexing time without making the FastPackageChecker a lot
slower. This is TBD for a future commit.
- Currently, the same patch can only be used one way in a package. That
is, if it appears twice with different level/working_dir settings,
bad things will happen. There's no package that current uses the
same patch two different ways, so we've punted on this as well, but
we may need to fix this in the future by moving a lot of the metdata
(level, working dir) to the spec, and *only* caching sha256sums in
the PatchCache. That would require some much more complicated tweaks
to the Spec, so we're holding off on that til later.
- This required patches to be refactored somewhat -- the difference
between a UrlPatch and a FilePatch is still not particularly clean.
- indexes should use json, not YAML, to optimize for speed
- only use YAML in human-editable files
- this makes ProviderIndex consistent with other indexes
- virtual provider cache and tags were previously generated by nearly
identical but separate methods.
- factor out an Indexer interface for updating repository caches, and
provide implementations for each type of index (TagIndex,
ProviderIndex) so that more can be added if needed.
- Among other things, this allows all indexes to be updated at once.
This is an advantage because loading package files is the real
overhead, and building the indexes once the packages are loaded is
trivial. We avoid extra bulk read-ins by generating all package indexes
at once.
- This can be extended for dependents (reverse dependencies) and patches
later.
- cleanup patch.py:
- make patch.py constructors more understandable
- loosen coupling of patch.py with package
- in Package: make package_dir, module, and namespace class properties
- These were previously instance properties and couldn't be called from
directives, e.g. in patch.create()
- make them class properties so that they can be used in class definition
- also add some instance properties to delegate to class properties so
that prior usage on Package objects still works
- When returning string output, use text_type and decode utf-8 in Python
2 instead of using `str`
- This properly handles unicode, whereas before we would pass bad strings
to colify in `spack blame` when reading git output
- add a test that round-trips some unicode through an Executable object
* Remove /nfs/tmp2 from default configuration
* /nfs/tmp2 is going away from LC... and doesn’t exist for the rest of the world.
* update documentation to remove /nfs/tmp2 as well
* Record build output as an array of lines rather than concatenating to a
single large string.
* Use string.find to avoid running re.search on every line of output.
- some commands were missed in the rollout of spack environments
- this makes all commands that need to disambiguate specs restrict the
disambiguation to installed packages in the active environment, as
users would expect
* This fixes a number of bugs:
* Patches were not properly downloaded and added to mirrors.
* Mirror create didn't respect `list_url` in packages
* Update the `spack mirror` command to add all packages in the
concretized DAG (where originally it only added the package specified
by the user). This is required in order to collect patches that are specified
by dependents. Example:
* if X->Y and X requires a patch on Y called Pxy, then Pxy will only
be discovered if you create a mirror with X.
* replace confusing --one-version-per-spec option for `spack mirror create`
with --versions-per-spec; support retrieving multiple versions for
concrete specs
* Implementation details:
* `spack mirror create` now uses regular staging logic to download files
into a mirror, instead of reimplementing it in `add_single_spec`.
* use a separate resource caching object to keep track of new
resources and already-existing resources; also accepts storing
resources retrieved from a cache (unlike the local cache)
* mirror cache object now stores resources that are considered
non-cachable, like (e.g. the tip of a branch);
* the 'create' function of the mirror module no longer traverses
dependencies since this was already handled by the 'mirror' command;
* Change handling of `--no-checksum`:
* now that 'mirror create' uses stages, the mirror tests disable
checksums when creating the mirror
* remove `no_checksum` argument from library functions - this is now
handled at the Spack-command-level (like for 'spack install')
- all multimethod tests are now run for both `multimethod` and
`multimethod-inheritor`
- do this with a parameterized fixture (pkg_name) that runs the same
tests on both
- Since early Spack versions, the SpecParser has (weirdly) been
responsible for initializing Spec fields.
- This refactors initialization to take place in Spec.__init__, as it
probably should have originally.
- This makes the code easier to read, the parser easier to understand,
and removes the use of __new__ in the parser to initialize the Spec.
- This also makes it possible to make a completely empty Spec with
`Spec()` -- this is an abstract Spec that will match anything.
* "spack install" now uses cache by default, update examples accordingly
* Replace some example packages with others
* Packing tutorial reference to "spack env" replaced with "spack build-env"
* Command line prompts in examples are shortened
* Example output (including paths) are updated to be more relevant to training environment
Update all examples that need an MPI provider to build with MPICH; reorganize so that fixing MPICH (as part of environment section) comes first in the tutorial (most examples in the tutorial use an MPI provider).
- previously, uninstall would complain if a spec was needed by an
environment.
- Now, we analyze dependents and dependent environments and simply remove
(not uninstall) specs that are needed by environments
- with no arguments, these commands will now edit or dump the
environment's `spack.yaml` file.
- users may not know where named environments live
- this makes it convenient for users to get to the spack.yaml
configuration file for their named environment.
* Update Makefile to use property methods ("build_targets"/"install_targets")
to demonstrate their usage
* Fix highlighting
* Change cbench example to ESMF:
CBench package file was changed and no longer uses the example shown in
the old docs
Scopes added with -C are now referred to as "custom scopes"
rather than "command line scopes". "command line scope" now refers
to specific config options that are set on the command line (like
"--insecure")
- default is still to use the cache, but we've added back the
`--use-cache` argument so that scripts that used it are still correct.
- `--no-cache` is stil present and is mutually exclusive with `--use-cache`
* Introduce FFTW2 and FFT3 providers for Intel-MKL and FFTW Spack packages.
* make fftw default package for fftw-api virtual package
* virtual package test assertion now provides location of default virtual packages.
* Change name of virtual package to fftw-api and used versioned interface.
- all commands (except `spack find`, through `ConstraintAction`) now go
through get_env() to get the active environment
- ev.active was hard to read -- and the name wasn't descriptive.
- rename it to _active_environment to be more descriptive and to strongly
indicate that spack.environment manages it
- to aovid changing spec hashes drastically, only add this attribute to
differentiated abstract specs.
- othherwise assume that read-in specs are concrete
- spack.yaml files in the current directory were picked up inconsistently
-- make this a sure thing by moving that logic into find_environment()
and moving find_environment() to main()
- simplify arguments to Spack command:
- remove short args for infrequently used commands (--pdb/-D, -P, -s)
- `spack -D` now forces an env with a directory
- The `Spec` class maintains a special `_patches_in_order_of_appearance`
attribute on patch variants, but it is was preserved when specs are
copied.
- This caused issues for some builds
- Add special logic to `Spec` to preserve this variant on copy
- TODO: in the long term we should get rid of the special variant and
make it the responsibility of one of the variant classes.
