- in many files, regular strings were used in places where raw strings
should've been used.
- convert these to raw strings and get rid of new flake8 errors
This PR improves the validation of `modules.yaml` by introducing a custom validator that checks if an attribute listed in `properties` or `patternProperties` is a valid spec. This new check applied to the test case in #9857 gives:
```console
$ spack install szip
==> Error: /home/mculpo/.spack/linux/modules.yaml:5: "^python@2.7@" is an invalid spec [Invalid version specifier]
```
Details:
* Moved the set-up of a custom validator class to spack.schema
* In Spack we use `jsonschema` to validate configuration files
against a schema. We also need custom validators to enforce
writing default values within "properties" or "patternProperties"
attributes.
* Currently, validators were customized at the place of use and with the
recent introduction of environments that meant we were setting-up and
using 2 different validator classes in two different modules.
* This commit moves the set-up of a custom validator class in the
`spack.schema` module and refactors the code in `spack.config` and
`spack.environments` to use it.
* Added a custom validator to check if an attribute is a valid spec
* Added a custom validator that can be used on objects, which yields an
error if the attribute is not a valid spec.
* Updated the schema for modules.yaml
* Updated modules.yaml to fix a few inconsistencies:
- a few attributes were not tested properly using 'anyOf'
- suffixes has been updated to also check that the attribute is a spec
- hierarchical_scheme has been updated to hierarchy
* Removed $ref from every schema
* $ref is not composable or particularly legible
* Use python dicts and regular old variables instead.
- The nested directive implementation was broken for python 3
- directive results were not properly removed from the directive list
when it was processed in the DirectiveMeta metaclass.
- the issue was that remove_directives only descended into a list or
tuple, but in Python3, the initial value passed to the function is a
view of dictionary values.
- make it a list to fix things, and add a regression test.
- currently just looks at patches
- allows you to find out which package applied a patch to a spec
- intended to work with tarballs and resources in the future.
- add tab completion for `spack resource` and subcommands
- previously, if a concrete sub-DAG with patched specs was written out
and read back in, its patches would not be found because the dependent
that patched it was no longer in the DAG.
- Add a test to ensure that the PatchCache handles this case.
- Also add tests to ensure that patch objects are properly created from
Specs -- previously we only checked that the patches were on the Spec.
- this fixes a bug where if we save a concretized sug-DAG where a package
had been patched by a dependent, and the dependent was not in the DAG,
we would not read in all patches correctly.
- Rather than looking up patches in the DAG, we look them up globally
from an index created from the entire repository.
- The patch cache is a bit tricky for several reasons:
- we have to cache information from packages, specifically, the patch
level and working directory.
- FilePatches need to know which package owns them, so that they can
figure out where the patch lives. The repo can change locations from
run to run, so we have to store relative paths and restore them when
the cache is reloaded.
- Patch files can change underneath the cache, because repo indexes
only update on package changes. We currently punt on this -- there
are stub methods for needs_update() that will need to check patch
files when packages are loaded. There isn't an easy way to do this
at global indexing time without making the FastPackageChecker a lot
slower. This is TBD for a future commit.
- Currently, the same patch can only be used one way in a package. That
is, if it appears twice with different level/working_dir settings,
bad things will happen. There's no package that current uses the
same patch two different ways, so we've punted on this as well, but
we may need to fix this in the future by moving a lot of the metdata
(level, working dir) to the spec, and *only* caching sha256sums in
the PatchCache. That would require some much more complicated tweaks
to the Spec, so we're holding off on that til later.
- This required patches to be refactored somewhat -- the difference
between a UrlPatch and a FilePatch is still not particularly clean.
- indexes should use json, not YAML, to optimize for speed
- only use YAML in human-editable files
- this makes ProviderIndex consistent with other indexes
- virtual provider cache and tags were previously generated by nearly
identical but separate methods.
- factor out an Indexer interface for updating repository caches, and
provide implementations for each type of index (TagIndex,
ProviderIndex) so that more can be added if needed.
- Among other things, this allows all indexes to be updated at once.
This is an advantage because loading package files is the real
overhead, and building the indexes once the packages are loaded is
trivial. We avoid extra bulk read-ins by generating all package indexes
at once.
- This can be extended for dependents (reverse dependencies) and patches
later.
- cleanup patch.py:
- make patch.py constructors more understandable
- loosen coupling of patch.py with package
- in Package: make package_dir, module, and namespace class properties
- These were previously instance properties and couldn't be called from
directives, e.g. in patch.create()
- make them class properties so that they can be used in class definition
- also add some instance properties to delegate to class properties so
that prior usage on Package objects still works
- When returning string output, use text_type and decode utf-8 in Python
2 instead of using `str`
- This properly handles unicode, whereas before we would pass bad strings
to colify in `spack blame` when reading git output
- add a test that round-trips some unicode through an Executable object
* Remove /nfs/tmp2 from default configuration
* /nfs/tmp2 is going away from LC... and doesn’t exist for the rest of the world.
* update documentation to remove /nfs/tmp2 as well
* Record build output as an array of lines rather than concatenating to a
single large string.
* Use string.find to avoid running re.search on every line of output.
- some commands were missed in the rollout of spack environments
- this makes all commands that need to disambiguate specs restrict the
disambiguation to installed packages in the active environment, as
users would expect
* This fixes a number of bugs:
* Patches were not properly downloaded and added to mirrors.
* Mirror create didn't respect `list_url` in packages
* Update the `spack mirror` command to add all packages in the
concretized DAG (where originally it only added the package specified
by the user). This is required in order to collect patches that are specified
by dependents. Example:
* if X->Y and X requires a patch on Y called Pxy, then Pxy will only
be discovered if you create a mirror with X.
* replace confusing --one-version-per-spec option for `spack mirror create`
with --versions-per-spec; support retrieving multiple versions for
concrete specs
* Implementation details:
* `spack mirror create` now uses regular staging logic to download files
into a mirror, instead of reimplementing it in `add_single_spec`.
* use a separate resource caching object to keep track of new
resources and already-existing resources; also accepts storing
resources retrieved from a cache (unlike the local cache)
* mirror cache object now stores resources that are considered
non-cachable, like (e.g. the tip of a branch);
* the 'create' function of the mirror module no longer traverses
dependencies since this was already handled by the 'mirror' command;
* Change handling of `--no-checksum`:
* now that 'mirror create' uses stages, the mirror tests disable
checksums when creating the mirror
* remove `no_checksum` argument from library functions - this is now
handled at the Spack-command-level (like for 'spack install')
- all multimethod tests are now run for both `multimethod` and
`multimethod-inheritor`
- do this with a parameterized fixture (pkg_name) that runs the same
tests on both
- Since early Spack versions, the SpecParser has (weirdly) been
responsible for initializing Spec fields.
- This refactors initialization to take place in Spec.__init__, as it
probably should have originally.
- This makes the code easier to read, the parser easier to understand,
and removes the use of __new__ in the parser to initialize the Spec.
- This also makes it possible to make a completely empty Spec with
`Spec()` -- this is an abstract Spec that will match anything.
* "spack install" now uses cache by default, update examples accordingly
* Replace some example packages with others
* Packing tutorial reference to "spack env" replaced with "spack build-env"
* Command line prompts in examples are shortened
* Example output (including paths) are updated to be more relevant to training environment
Update all examples that need an MPI provider to build with MPICH; reorganize so that fixing MPICH (as part of environment section) comes first in the tutorial (most examples in the tutorial use an MPI provider).
- previously, uninstall would complain if a spec was needed by an
environment.
- Now, we analyze dependents and dependent environments and simply remove
(not uninstall) specs that are needed by environments
- with no arguments, these commands will now edit or dump the
environment's `spack.yaml` file.
- users may not know where named environments live
- this makes it convenient for users to get to the spack.yaml
configuration file for their named environment.
* Update Makefile to use property methods ("build_targets"/"install_targets")
to demonstrate their usage
* Fix highlighting
* Change cbench example to ESMF:
CBench package file was changed and no longer uses the example shown in
the old docs
Scopes added with -C are now referred to as "custom scopes"
rather than "command line scopes". "command line scope" now refers
to specific config options that are set on the command line (like
"--insecure")
- default is still to use the cache, but we've added back the
`--use-cache` argument so that scripts that used it are still correct.
- `--no-cache` is stil present and is mutually exclusive with `--use-cache`
* Introduce FFTW2 and FFT3 providers for Intel-MKL and FFTW Spack packages.
* make fftw default package for fftw-api virtual package
* virtual package test assertion now provides location of default virtual packages.
* Change name of virtual package to fftw-api and used versioned interface.
- all commands (except `spack find`, through `ConstraintAction`) now go
through get_env() to get the active environment
- ev.active was hard to read -- and the name wasn't descriptive.
- rename it to _active_environment to be more descriptive and to strongly
indicate that spack.environment manages it
- to aovid changing spec hashes drastically, only add this attribute to
differentiated abstract specs.
- othherwise assume that read-in specs are concrete
- spack.yaml files in the current directory were picked up inconsistently
-- make this a sure thing by moving that logic into find_environment()
and moving find_environment() to main()
- simplify arguments to Spack command:
- remove short args for infrequently used commands (--pdb/-D, -P, -s)
- `spack -D` now forces an env with a directory
- The `Spec` class maintains a special `_patches_in_order_of_appearance`
attribute on patch variants, but it is was preserved when specs are
copied.
- This caused issues for some builds
- Add special logic to `Spec` to preserve this variant on copy
- TODO: in the long term we should get rid of the special variant and
make it the responsibility of one of the variant classes.
- split 'environment' section into 'environments' and 'modules'
- move location to 'query packages' section
- move cd to developer section
- --env-dir no longer has a short optino (was -E)
- -E now means "run without an environment" (no longer same as --env-dir)
- -D now means "run with this directory environment"
- remove short options for may infrequently used top-level commands
- `spack env status` used to show install status; consolidate that into
`spack find`.
- `spack env status` will still print out whether there is an active
environment
- uninstall now:
- restricts its spec search to the current environment
- removes uninstalled specs from the current environment
- reports envs that still need specs you're trying to uninstall
- removed spack env uninstall command
- updated tests
- moved get_env from cmd/env.py to environment.py
- spack install will now install into the active environment when no
arguments are provided. It looks:
1. at the command line
2. for a local spack.yaml file
3. for any currently activated environment
- `spack env create <name>` works as before
- `spack env create <path>` now works as well -- environments can be
created in their own directories outside of Spack.
- `spack install` will look for a `spack.yaml` file in the current
directory, and will install the entire project from the environment
- The Environment class has been refactored so that it does not depend on
the internal Spack environment root; it just takes a path and operates
on an environment in that path (so internal and external envs are
handled the same)
- The named environment interface has been hoisted to the
spack.environment module level.
- env.yaml is now spack.yaml in all places. It was easier to go with one
name for these files than to try to handle logic for both env.yaml and
spack.yaml.
- `spack env activate foo`: sets SPACK_ENV to the current active env name
- `spack env deactivate`: unsets SPACK_ENV, deactivates the environment
- added support to setup_env.sh and setup_env.csh
- other env commands work properly with SPACK_ENV, as with an environment
arguments.
- command-line --env arguments take precedence over the active
environment, if given.
- env.yaml is now meaningful; it contains authoritative user specs
- concretize diffs user specs in env.yaml and env.json to allow user to
add/remove by simply updating env.yaml
- comments are preserved when env.yaml is updated by add/unadd
- env.yaml can contain configuration and include external configuration
either from merged files or from config scopes
- there is only one file format to remember (env.yaml, no separate init
format)
- env.json is now env.lock, and it stores the *last* user specs to be
concretized, along with full provenance.
- internal structure was modified slightly for readability
- env.lock contains a _meta section with metadata, in case needed
- added more tests for environments
- env commands follow Spack conventions; no more `spack env foo install`
- add `SingleFileScope` to configuration, which allows us to pull config
sections from a single file.
- update `env.yaml` and tests to ensure that the env.yaml schema works
when pulling configurtion from the env file.
- Each schema now has a top-level `properties` and `schema` attribute.
- The `properties` is a fragment that can be included in other
jsonschemas, via Python, not via '$ref'
- Th `schema` is a complete `jsonschema` with `title` and `$schema`
properties.
- add a common argument for `-e/--env`
- modify the database to support queries on subsets of hashes
- allow `spack find` to be filtered by hashes in an environment
- logic used in `spack find` was hiding duplicate installations if their
hashes were different
- short hash doesn't work in this scenario, since specs are structurally
identical
- ConstraintAction always works on a DB query, so use the DAG hash to
ensure uniqueness
- `spack.environment` is now the home for most of the infrastructure
around Spack environments
- refactor `cmd/env.py` to use everything from spack.environment
- refactor the cmd/env test to use pytest and fixtures
- `spack.util.environment` is the new home for routines that modify
environment variables.
- This is to make room for `spack.environment` to contain new routines
for dealing with spack environments
- Instead of one method with all parsers, each subcommand gets two
functions: `setup_<cmd>_parser()` and `environment_<cmd>()`
- the `setup_parser()` and `env()` functions now generate the parser
based on these and a list of subcommands.
- it is now easier to associate the arguments with the subcommand.
* modified tutorial packages
* update hint in hdf5 tutorial file (typo for suggested argument)
* add repo.yaml to tutorial repository
* update tutorial docs to refer user to tutorial package repository
* flake edits
* recommend site scope vs. defaults
* you don't specify the repo's name when adding a repo, just the path
* omit symlinks and create file copies when making a binary cache of a package
* unrelated flake edits involving regexes that recent flake is now angry about
* Record stdout for packages without errors
Previously our reporter only stored stdout if something went wrong
while installing a package. This prevented us from properly reporting
on steps where everything went as expected.
* More robustly report all phases to CDash
Previously if a phase generated no output it would not be reported to CDash.
For example, consider the following output:
==> Executing phase: 'configure'
==> Executing phase: 'build'
This would not generate a report for the configure phase. Now it does.
* Add test case for CDash reporting clean builds
* Fix default directory for CDash reports
The default 'cdash_report' directory name was getting overwritten
by 'junit-report'.
* Upload the build phase first to CDash
Older versions of CDash expect Build.xml to be the first file uploaded
for any given build.
* Define cdash_phase before referring to it
fixes#9739
The non-daemonic pool relies heavily on implementation details of the
multiprocessing package. In this commit we provide an implementation
that fits recent python versions.
This allows installing software on a namespace basis by including ${NAMESPACE} in `install_path_scheme`. e.g.,
```
cat ~/.spack/config.yaml
config:
install_path_scheme:
"${ARCHITECTURE}/${NAMESPACE}/${COMPILERNAME}-${COMPILERVER}/${PACKAGE}-${VERSION}-${HASH}"
```
The 'static_to_shared_library' function takes a compiler Executable,
which is intended to be invoked with a list of arguments; the
arguments must be separated from their values in the list, given
the way that 'Executable.__call__' invokes the underlying executable.
'static_to_shared_library' was not doing this, which this commit fixes.
Clang has support for using different fortran compilers with the Clang executable.
Spack includes logic to select a compiler wrapper symlink which refers to the fortran executable (since some build systems depend on the name of the compiler, e.g. 'gfortran' or 'flang').
This selection was previously based on the architecture, and chose incorrectly in some situations (e.g. for clang/gfortran on Linux). This replaces architecture-based wrapper selection with a selection that is based on the name of the Fortran compiler executable.
* Unite Dockerfiles - add build/run/push scripts
* update docker documentation
* update .travis.yml
* switch to using a preprocessor on Dockerfiles
* skip building docker images on pull requests
* update files with copyright info
* tweak when travis builds for docker files are done
fixes#9624
merge_config_rules was using `strict=False` to check if a spec
satisfies a constraint, which loosely translates to "this spec has
no conflict with the constraint, so I can potentially add it to the
spec". We want instead `strict=True` which means "the spec satisfies
the constraint right now".
- #8773 made the default mode 0o777, which is what's documented but
mkdirp actually takes the OS default or umask by default
- revert to the Python default by default, and only set the mode when
asked explicitly.
#9100 added a warning message when a path extracted from a module file
did not appear to be a valid filesystem path. This check was applied
to a variable which could be a list of paths, which would erroneously
trigger the warning. This commit updates the check to run at the
actual point where the path has been extracted.
* Add a build_language config.yaml option which controls the language
of compiler messages
* build_language defaults to "C", in which case the compiler messages
will be in English. This allows Spack log parsing to detect and
highlight error messages (since the regular expressions to find
error messages are in English)
* The user can use the default language in their environment by setting
the build_language config variable to null or ''
- `spack license list-files`: list all files that should have license headers
- `spack license list-lgpl`: list files still under LGPL-2.1
- `spack license verify`: check that license headers are correct
- Added `spack license verify` to style tests
- remove the old LGPL license headers from all files in Spack
- add SPDX headers to all files
- core and most packages are (Apache-2.0 OR MIT)
- a very small number of remaining packages are LGPL-2.1-only
compilers.yaml can track a module that is needed for a compiler, but
Spack does not fill this in automatically. This adds a note to the
documentation informing the user how to do this.
If we do not specify libdir explicitly, Meson chooses something like
lib/x86_64-linux-gnu, which causes problems when trying to find libraries
and pkg-config files.
Spack packages installed using spack buildcache were not running
post-install hooks, which create module files and manage licenses
(if necessary).
This was already occurring for Spack packages installed with
spack install --use-cache
Spack can now be configured to assign permissions to the files installed by a package.
In the `packages.yaml` file under `permissions`, the attributes `read`, `write`, and `group` control the package permissions. These attributes can be set per-package, or for all packages under `all`. If permissions are set under `all` and for a specific package, the package-specific settings take precedence. The `read` and `write` attributes take one of `user`, `group`, and `world`.
packages:
all:
permissions:
write: group
group: spack
my_app:
permissions:
read: group
group: my_team
* Better default CLI arguments for CDash reporting
--log-format=cdash is now implied if you specify the --cdash-upload-url
option to spack install.
We also now default to writing CTest XML files to cdash_report/ when using
the CDash reporter if no --log-file argument was specified.
* Improved documentation on how to use the CDash reporter
* Push default flag handlers into module scope
* Preserve backwards compatibility of builtin flag handler names
Ensure Spack continues to work for packages using the `Package.env_flags` idiom and equivalent.
* update docs and tests to match
* Update packages to match new syntax
Fix two bugs with module file parsing:
* Detection of the CRAY_LD_LIBRARY_PATH variable was broken by #9100.
This fixes it and adds a test for it.
* For module names like "foo-bar/1.0", the associated PACKAGE_DIR
environment variable name would be "FOO_BAR_DIR", but Spack was not
parsing the components and not converting "-" to "_"
Fixes#9166
This is intended to reduce errors related to lock timeouts by making
the following changes:
* Improves error reporting when acquiring a lock fails (addressing
#9166) - there is no longer an attempt to release the lock if an
acquire fails
* By default locks taken on individual packages no longer have a
timeout. This allows multiple spack instances to install overlapping
dependency DAGs. For debugging purposes, a timeout can be added by
setting 'package_lock_timeout' in config.yaml
* Reduces the polling frequency when trying to acquire a lock, to
reduce impact in the case where NFS is overtaxed. A simple
adaptive strategy is implemented, which starts with a polling
interval of .1 seconds and quickly increases to .5 seconds
(originally it would poll up to 10^5 times per second).
A test is added to check the polling interval generation logic.
* The timeout for Spack's whole-database lock (e.g. for managing
information about installed packages) is increased from 60s to
120s
* Users can configure the whole-database lock timeout using the
'db_lock_timout' setting in config.yaml
Generally, Spack locks (those created using spack.llnl.util.lock.Lock)
now have no timeout by default
This does not address implementations of NFS that do not support file
locking, or detect cases where services that may be required
(nfslock/statd) aren't running.
Users may want to be able to more-aggressively release locks when
they know they are the only one using their Spack instance, and they
encounter lock errors after a crash (e.g. a remote terminal disconnect
mentioned in #8915).
When a Spack Executable was configured to capture stderr and the
process failed, the error messages of the process were discarded.
This made it difficult to understand why the process failed. The
exception is now updated to include the stderr of the process when
the Executable captures stderr.
Adds 'code' to the list of suffixes that are excluded from version
parsing of URLs, such that if a URL contains the string
'cistem-1.0.0-beta-source-code', a version X will substitute in to
produce a URL with cistem-X-source-code ('source' was already excluded).
The 'cistem' package version is updated to make use of this (and fix
a fetching bug with the cistem package). A unit test is added to check
this parsing case.
Improve Spack's parsing of module show to eliminate some false
positives (e.g. accepting MODULEPATH when it is in fact looking for
PATH). This makes the following changes:
* Updates the pattern searching for several paths to avoid the case
where they are prefixes of unwanted paths
* Adds a warning message when an extracted path doesn't exist (which
may help catch future module parsing bugs faster)
* Adds a test with the content mentioned in #9083
Spack originally handled environment modifications in the following
order:
1. clear environment variables
(unless Spack was invoked with --dirty)
2. apply spack-specific environment variable updates,
including variables set by Spack core like CC/PKG_CONFIG_PATH
and those set by installed dependencies (e.g. in
setup_dependent_environment)
3. load all external/compiler modules
1 and 2 were done together. This splits 1 into its own function and
imposes the following order for environment modifications:
1. clear environment variables
2. load all external/compiler modules
3. apply spack-specific environment variable updates
As a result, prepend-path actions taken by Spack (or installed Spack
dependencies) take precedence over prepend-path actions from compiler
and external modules. Additionally, when Spack (or a package
dependency) sets/unsets an environment variable, that will override
the actions of external/compiler modules.
* Add 'extra_env' argument to Executable.__call__: this will be added
to the environment but does not affect whether the current
environment is reused. If 'env' is not set, then the current
environment is copied and the variables from 'extra_env' are added
to it.
* MakeExecutable can take a 'jobs_env' parameter that specifies the
name of an environment variable used to set the level of parallelism.
This is added to 'extra_env' (so does not affect whether the current
environment is reused).
* CMake-based Spack packages set 'jobs_env' when executing the 'test'
target for make and ninja (which does not use -j)
Consolidate prefix calculation logic for intel packages into the
IntelPackage class.
Add documentation on installing Intel packages with Spack an
(alternatively) adding them as external packages in Spack.
