closes#5201
Currently, if a user sets an external package to have a prefix that is
one of the system paths (like '/usr') the module files that are
generated will prepend '/usr/bin' to 'PATH', etc. This is particularly
nasty at the time when a module file is unloaded, and e.g. paths like
'/usr/bin' will be discarded from PATH.
This PR solves the issue skipping system paths when a prefix inspection
is made to generate module files.
* Module files now are generated using a template engine refers #2902#3173
jinja2 has been hooked into Spack.
The python module `modules.py` has been splitted into several modules
under the python package `spack/modules`. Unit tests stressing module
file generation have been refactored accordingly.
The module file generator for Lmod has been extended to multi-providers
and deeper hierarchies.
* Improved the support for templates in module files.
Added an entry in `config.yaml` (`template_dirs`) to list all the
directories where Spack could find templates for `jinja2`.
Module file generators have a simple override mechanism to override
template selection ('modules.yaml' beats 'package.py' beats 'default').
* Added jinja2 and MarkupSafe to vendored packages.
* Spec.concretize() sets mutual spec-package references
The correct place to set the mutual references between spec and package
objects at the end of concretization. After a call to concretize we
should now be ensured that spec is the same object as spec.package.spec.
Code in `build_environment.py` that was performing the same operation
has been turned into an assertion to be defensive on the new behavior.
* Improved code and data layout for modules and related tests.
Common fixtures related to module file generation have been extracted
in `conftest.py`. All the mock configurations for module files have been
extracted from python code and have been put into their own yaml file.
Added a `context_property` decorator for the template engine, to make
it easy to define dictionaries out of properties.
The default for `verbose` in `modules.yaml` is now False instead of True.
* Extendable module file contexts + short description from docstring
The contexts that are used in conjunction with `jinja2` templates to
generate module files can now be extended from package.py and
modules.yaml.
Module files generators now infer the short description from package.py
docstring (and as you may expect it's the first paragraph)
* 'module refresh' regenerates all modules by default
`module refresh` without `--module-type` specified tries to
regenerate all known module types. The same holds true for `module rm`
Configure options used at build time are extracted and written into the
module files where possible.
* Fixed python3 compatibility, tests for Lmod and Tcl.
Added test for exceptional paths of execution when generating Lmod
module files.
Fixed a few compatibility issues with python3.
Fixed a bug in Tcl with naming_scheme and autoload + unit tests
* Updated module file tutorial docs. Fixed a few typos in docstrings.
The reference section for module files has been reorganized. The idea is
to have only three topics at the highest level:
- shell support + spack load/unload use/unuse
- module file generation (a.k.a. APIs + modules.yaml)
- module file maintenance (spack module refresh/rm)
Module file generation will cover the entries in modules.yaml
Also:
- Licenses have been updated to include NOTICE and extended to 2017
- docstrings have been reformatted according to Google style
* Removed redundant arguments to RPackage and WafPackage.
All the callbacks in `RPackage` and `WafPackage` that are not build
phases have been modified not to accept a `spec` and a `prefix`
argument. This permits to leverage the common `configure_args` signature
to insert by default the configuration arguments into the generated
module files. I think it's preferable to handling those packages
differently than `AutotoolsPackage`. Besides only one package seems
to override one of these methods.
* Fixed broken indentation + improved resiliency of refresh
Fixed broken indentation in `spack module refresh` (probably a rebase
gone silently wrong?). Filter the writers for blacklisted specs before
searching for name clashes. An error with a single writer will not
stop regeneration, but instead will print a warning and continue
the command.
- converted `log_path` and `env_path` to properties of PackageBase.
- InstallErrors in build_environment are now annotated with the package
that caused them, in the 'pkg' attribute.
- Add `--show-log-on-error` option to `spack install` that catches
InstallErrors and prints the log to stderr if it exists.
Note that adding a reference to the Pakcage allows a lot of stuff
currently handled by do_install() and build_environment to be handled
externally.
- '\b' in regular expression needs to be in a raw string (r'\b')
- Regression test that would've caught this was unintentionally disabled
- This fixes the string and the test
The correct place to set the mutual references between spec and
package objects is at the end of concretization. After a call to
concretize we should now be ensured that spec is the same object
as spec.package.spec.
Code in `build_environment.py` that was performing the same
operation has been turned into an assertion to be defensive on
the new behavior.
- Fixes bugs where concretization would fail due to an erroneously cached
_concrete attribute.
- Ripped out a bunch of code in spec.py that isn't needed/valid anymore:
- The various concrete() methods on different types of Specs would
attempt to statically compute whether the Spec was concrete.
- This dates back to when DAGs were simpler and there were no optional
dependencies. It's actually NOT possible to compute statically
whether a Spec is concrete now. The ONLY way you know is if it goes
through concretization and is marked concrete once that completes.
- This commit removes all simple concreteness checks and relies only on
the _concrete attribute. This should make thinking about
concreteness simpler.
- Fixed a couple places where Specs need to be marked concrete explicitly.
- Specs read from files and Specs that are destructively copied from
concrete Specs now need to be marked concrete explicitly.
- These spots may previously have "worked", but they were brittle and
should be explcitly marked anyway.
- Dependencies in concrete specs did not previously have their cache
fields (_concrete, _normal, etc.) preserved.
- _dup and _dup_deps weren't passing each other enough information to
preserve concreteness properly, so only the root was properly
preserved.
- cached concreteness is now preserved properly for the entire DAG, not
just the root.
- added method docs.
Fixes#4112
This commit extends the support of the AutotoolsPackage methods
`with_or_without` and `enable_or_disable` to bool-valued variants. It
also defines for those functions a convenience short-cut if the
activation parameter is the prefix of a spec (like in
`--with-{pkg}={prefix}`).
This commit also includes:
* Updates to viennarna and adios accordingly: they have been modified to
use `enable_or_disable` and `with_or_without`
* Improved docstrings in `autotools.py`. Raise `KeyError` if name is
not a variant.
Renames the existing bootstrap command to 'clone'. Repurposes
'spack bootstrap' to install packages that are useful to the
operation of Spack (for now this is just environment-modules).
For bash and ksh users running setup-env.sh, if a Spack-installed
instance of environment-modules is detected and environment modules
and dotkit are not externally available, Spack will define the
'module' command in the user's shell to use the environment-modules
built by Spack.
First, quote the environment variable values. Second, export the
variables. sorry, this is bourn-shell syntax. Happy to consider a
shell-independent way to do this, but spack is already using sh-like
"env=value"
* Added support to query packages by tags.
- The querying commands `spack list`, `spack find` and `spack info` have
been modified to support querying by tags. Tests have been added to
check that the feature is working correctly under what should be the
most frequent use cases.
* Refactored Repo class to make insertion of new file caches easier.
- Added the class FastPackageChecker. This class is a Mapping from
package names to stat info, that gets memoized for faster access.
- Extracted the creation of a ProviderIndex to its own factory function.
* Added a cache file for tags.
- Following what was done for providers, a TagIndex class has been added.
This class can serialize and deserialize objects from json. Repo and
RepoPath have a new method 'packages_with_tags', that uses the TagIndex
to compute a list of package names that have all the tags passed as
arguments.
On Ubuntu 14.04 the effect if the cache reduces the time for spack list
from ~3sec. to ~0.3sec. after the cache has been built.
* Fixed colorization of `spack info`
This command broke after #5109. It was using the default value for the
"dirty" argument in `setup_package`. Now it adopts the same logic as
in `spack install`. Changed help for '--clean' and '--dirty'.
Improved coverage of spack env.
The private method `Spec._dup` was missing a line (when setting compiler
flags the parent spec was not set to `self`). This resulted in
an inconsistent state of the duplicated Spec. This problem has been
fixed here. The docstring of `Spec._dup` has been updated.
This change is done to avoid inconsistencies during refactoring. The rationale is that functions at different levels in the call stack all define a default for the 'dirty' argument. This PR removes the default value for all the functions except the top-level one (`PackageBase.do_install`).
In this way not defining 'dirty' will result in an error, instead of the default value being used. This will reduce the risk of having an inconsistent behavior after a refactoring.
* Respect --insecure when fetching list_url.
* Ensure support for Python 2.6, and that urlopen works for python versions prior 2.7.9 and between 3.0 and 3.4.3.
* Simplified Spec.__init__ signature by removing the *dep_like argument.
The `*dep_like` argument of `Spec.__init__` is used only for tests. This
PR removes it from the call signature and introduces an equivalent
fixture to be used in tests.
* Refactored ``spec_from_dict`` to be a static method of ``Spec``
The fixture ``spec_from_dict`` has been refactored to be a static method
of ``Spec``. Test code has been updated accordingly. Added tests for
exceptional paths.
* Renamed argument `unique` to `normal` + added LazySpecCache class
As requested in the review the argument `unique` of `Spec.from_literal`
has been renamed to `normal`. To avoid eager evaluations of
`Spec(spec_like)` expressions a subclass of `collections.defaultdict`
has been introduced.
* Spec object can be keys of the dictionary for a spec literal.
Added back the possibility use a spec directly as a key. This permits
to build DAGs that are partially normalized.
- Update handling of ChildError so that its output is capturable from a
SpackCommand
- Update cmd/install test to make sure Python and build log output is
being displayed properly.
- install and probably other commands were designed to run once, but now
we can to test them from within Spack with SpackCommand
- cmd/install.py assumed that it could modify do_install in PackageBase
and leave it that way; this makes the decorator temporary
- package.py didn't properly initialize its stage if the same package had
been built successfully before (and the stage removed).
- manage stage lifecycle better and remember when Package needs to
re-create the stage
- If a failure comes from an external command and NOT the Python code,
display errors highlighted with some context.
- Add some rudimentary support for parsing errors out of the build log
(not very sophisticated yet).
- Build errors in Python code will still display with Python context.
