- Previously, dependencies and dependency_types were stored as separate
dicts on Package.
- This means a package can only depend on another in one specific way,
which is usually but not always true.
- Prior code unioned dependency types statically across dependencies on
the same package.
- New code stores dependency relationships as their own object, with a
spec constraint and a set of dependency types per relationship.
- Dependency types are now more precise
- There is now room to add more information to dependency relationships.
- New Dependency class lives in dependency.py, along with deptype
definitions that used to live in spack.spec.
Move deptype definitions to spack.dependency
* Add '--test=all' and '--test=root' options to test either the root or the root and all dependencies.
* add a test dependency type that is only used when --test is enabled.
* test dependencies are not added to the spec, but they are provided in the test environment.
closes#5473
Prior to this PR we were not exiting early for external packages, which
caused the `configure_options` property of the contexts to fail with
e.g. a key error because the DAG gets truncated for them. More
importantly Spack configure options don't make any sense for externals.
Now we exit early, and leave a message in the module file clarifying
that this package has been installed outside of Spack.
closes#5201
Currently, if a user sets an external package to have a prefix that is
one of the system paths (like '/usr') the module files that are
generated will prepend '/usr/bin' to 'PATH', etc. This is particularly
nasty at the time when a module file is unloaded, and e.g. paths like
'/usr/bin' will be discarded from PATH.
This PR solves the issue skipping system paths when a prefix inspection
is made to generate module files.
* Module files now are generated using a template engine refers #2902#3173
jinja2 has been hooked into Spack.
The python module `modules.py` has been splitted into several modules
under the python package `spack/modules`. Unit tests stressing module
file generation have been refactored accordingly.
The module file generator for Lmod has been extended to multi-providers
and deeper hierarchies.
* Improved the support for templates in module files.
Added an entry in `config.yaml` (`template_dirs`) to list all the
directories where Spack could find templates for `jinja2`.
Module file generators have a simple override mechanism to override
template selection ('modules.yaml' beats 'package.py' beats 'default').
* Added jinja2 and MarkupSafe to vendored packages.
* Spec.concretize() sets mutual spec-package references
The correct place to set the mutual references between spec and package
objects at the end of concretization. After a call to concretize we
should now be ensured that spec is the same object as spec.package.spec.
Code in `build_environment.py` that was performing the same operation
has been turned into an assertion to be defensive on the new behavior.
* Improved code and data layout for modules and related tests.
Common fixtures related to module file generation have been extracted
in `conftest.py`. All the mock configurations for module files have been
extracted from python code and have been put into their own yaml file.
Added a `context_property` decorator for the template engine, to make
it easy to define dictionaries out of properties.
The default for `verbose` in `modules.yaml` is now False instead of True.
* Extendable module file contexts + short description from docstring
The contexts that are used in conjunction with `jinja2` templates to
generate module files can now be extended from package.py and
modules.yaml.
Module files generators now infer the short description from package.py
docstring (and as you may expect it's the first paragraph)
* 'module refresh' regenerates all modules by default
`module refresh` without `--module-type` specified tries to
regenerate all known module types. The same holds true for `module rm`
Configure options used at build time are extracted and written into the
module files where possible.
* Fixed python3 compatibility, tests for Lmod and Tcl.
Added test for exceptional paths of execution when generating Lmod
module files.
Fixed a few compatibility issues with python3.
Fixed a bug in Tcl with naming_scheme and autoload + unit tests
* Updated module file tutorial docs. Fixed a few typos in docstrings.
The reference section for module files has been reorganized. The idea is
to have only three topics at the highest level:
- shell support + spack load/unload use/unuse
- module file generation (a.k.a. APIs + modules.yaml)
- module file maintenance (spack module refresh/rm)
Module file generation will cover the entries in modules.yaml
Also:
- Licenses have been updated to include NOTICE and extended to 2017
- docstrings have been reformatted according to Google style
* Removed redundant arguments to RPackage and WafPackage.
All the callbacks in `RPackage` and `WafPackage` that are not build
phases have been modified not to accept a `spec` and a `prefix`
argument. This permits to leverage the common `configure_args` signature
to insert by default the configuration arguments into the generated
module files. I think it's preferable to handling those packages
differently than `AutotoolsPackage`. Besides only one package seems
to override one of these methods.
* Fixed broken indentation + improved resiliency of refresh
Fixed broken indentation in `spack module refresh` (probably a rebase
gone silently wrong?). Filter the writers for blacklisted specs before
searching for name clashes. An error with a single writer will not
stop regeneration, but instead will print a warning and continue
the command.
- converted `log_path` and `env_path` to properties of PackageBase.
- InstallErrors in build_environment are now annotated with the package
that caused them, in the 'pkg' attribute.
- Add `--show-log-on-error` option to `spack install` that catches
InstallErrors and prints the log to stderr if it exists.
Note that adding a reference to the Pakcage allows a lot of stuff
currently handled by do_install() and build_environment to be handled
externally.
- '\b' in regular expression needs to be in a raw string (r'\b')
- Regression test that would've caught this was unintentionally disabled
- This fixes the string and the test
The correct place to set the mutual references between spec and
package objects is at the end of concretization. After a call to
concretize we should now be ensured that spec is the same object
as spec.package.spec.
Code in `build_environment.py` that was performing the same
operation has been turned into an assertion to be defensive on
the new behavior.
- Fixes bugs where concretization would fail due to an erroneously cached
_concrete attribute.
- Ripped out a bunch of code in spec.py that isn't needed/valid anymore:
- The various concrete() methods on different types of Specs would
attempt to statically compute whether the Spec was concrete.
- This dates back to when DAGs were simpler and there were no optional
dependencies. It's actually NOT possible to compute statically
whether a Spec is concrete now. The ONLY way you know is if it goes
through concretization and is marked concrete once that completes.
- This commit removes all simple concreteness checks and relies only on
the _concrete attribute. This should make thinking about
concreteness simpler.
- Fixed a couple places where Specs need to be marked concrete explicitly.
- Specs read from files and Specs that are destructively copied from
concrete Specs now need to be marked concrete explicitly.
- These spots may previously have "worked", but they were brittle and
should be explcitly marked anyway.
- Dependencies in concrete specs did not previously have their cache
fields (_concrete, _normal, etc.) preserved.
- _dup and _dup_deps weren't passing each other enough information to
preserve concreteness properly, so only the root was properly
preserved.
- cached concreteness is now preserved properly for the entire DAG, not
just the root.
- added method docs.
Fixes#4112
This commit extends the support of the AutotoolsPackage methods
`with_or_without` and `enable_or_disable` to bool-valued variants. It
also defines for those functions a convenience short-cut if the
activation parameter is the prefix of a spec (like in
`--with-{pkg}={prefix}`).
This commit also includes:
* Updates to viennarna and adios accordingly: they have been modified to
use `enable_or_disable` and `with_or_without`
* Improved docstrings in `autotools.py`. Raise `KeyError` if name is
not a variant.
Renames the existing bootstrap command to 'clone'. Repurposes
'spack bootstrap' to install packages that are useful to the
operation of Spack (for now this is just environment-modules).
For bash and ksh users running setup-env.sh, if a Spack-installed
instance of environment-modules is detected and environment modules
and dotkit are not externally available, Spack will define the
'module' command in the user's shell to use the environment-modules
built by Spack.
First, quote the environment variable values. Second, export the
variables. sorry, this is bourn-shell syntax. Happy to consider a
shell-independent way to do this, but spack is already using sh-like
"env=value"
* Added support to query packages by tags.
- The querying commands `spack list`, `spack find` and `spack info` have
been modified to support querying by tags. Tests have been added to
check that the feature is working correctly under what should be the
most frequent use cases.
* Refactored Repo class to make insertion of new file caches easier.
- Added the class FastPackageChecker. This class is a Mapping from
package names to stat info, that gets memoized for faster access.
- Extracted the creation of a ProviderIndex to its own factory function.
* Added a cache file for tags.
- Following what was done for providers, a TagIndex class has been added.
This class can serialize and deserialize objects from json. Repo and
RepoPath have a new method 'packages_with_tags', that uses the TagIndex
to compute a list of package names that have all the tags passed as
arguments.
On Ubuntu 14.04 the effect if the cache reduces the time for spack list
from ~3sec. to ~0.3sec. after the cache has been built.
* Fixed colorization of `spack info`
This command broke after #5109. It was using the default value for the
"dirty" argument in `setup_package`. Now it adopts the same logic as
in `spack install`. Changed help for '--clean' and '--dirty'.
Improved coverage of spack env.
The private method `Spec._dup` was missing a line (when setting compiler
flags the parent spec was not set to `self`). This resulted in
an inconsistent state of the duplicated Spec. This problem has been
fixed here. The docstring of `Spec._dup` has been updated.
This change is done to avoid inconsistencies during refactoring. The rationale is that functions at different levels in the call stack all define a default for the 'dirty' argument. This PR removes the default value for all the functions except the top-level one (`PackageBase.do_install`).
In this way not defining 'dirty' will result in an error, instead of the default value being used. This will reduce the risk of having an inconsistent behavior after a refactoring.
* Respect --insecure when fetching list_url.
* Ensure support for Python 2.6, and that urlopen works for python versions prior 2.7.9 and between 3.0 and 3.4.3.
* Simplified Spec.__init__ signature by removing the *dep_like argument.
The `*dep_like` argument of `Spec.__init__` is used only for tests. This
PR removes it from the call signature and introduces an equivalent
fixture to be used in tests.
* Refactored ``spec_from_dict`` to be a static method of ``Spec``
The fixture ``spec_from_dict`` has been refactored to be a static method
of ``Spec``. Test code has been updated accordingly. Added tests for
exceptional paths.
* Renamed argument `unique` to `normal` + added LazySpecCache class
As requested in the review the argument `unique` of `Spec.from_literal`
has been renamed to `normal`. To avoid eager evaluations of
`Spec(spec_like)` expressions a subclass of `collections.defaultdict`
has been introduced.
* Spec object can be keys of the dictionary for a spec literal.
Added back the possibility use a spec directly as a key. This permits
to build DAGs that are partially normalized.
- Update handling of ChildError so that its output is capturable from a
SpackCommand
- Update cmd/install test to make sure Python and build log output is
being displayed properly.
- install and probably other commands were designed to run once, but now
we can to test them from within Spack with SpackCommand
- cmd/install.py assumed that it could modify do_install in PackageBase
and leave it that way; this makes the decorator temporary
- package.py didn't properly initialize its stage if the same package had
been built successfully before (and the stage removed).
- manage stage lifecycle better and remember when Package needs to
re-create the stage
- If a failure comes from an external command and NOT the Python code,
display errors highlighted with some context.
- Add some rudimentary support for parsing errors out of the build log
(not very sophisticated yet).
- Build errors in Python code will still display with Python context.
Users can now add an optional custom message to the conflicts directive.
Layout on screen has been changed to improve readability and the long
spec is shown in tree format. Two conflicts in `espresso` have been
modified to showcase the feature.
- Python I/O would not properly interleave (or appear) with output from
subcommands.
- Add a flusing wrapper around sys.stdout and sys.stderr when
redirecting, so that Python output is synchronous with that of
subcommands.
- 'v' toggle was previously only good for the current install.
- subsequent installs needed user to press 'v' again.
- 'v' state is now preserved across dependency installs.
- Previously we would use `os._exit()` in to avoid Spack error handling
in the parent process when build processes failed. This isn't
necessary anymore since build processes propagate their exceptions to
the parent process.
- Use `sys.exit` instead of `os._exit`. This has the advantage of
automatically flushing output streams on quit, so output from child
processes is not lost when Spack exits.
- Simplify interface to log_output. New interface requires only one
context handler instead of two. Before:
with log_output('logfile.txt') as log_redirection:
with log_redirection:
# do things ... output will be logged
After:
with log_output('logfile.txt'):
# do things ... output will be logged
If you also want the output to be echoed to ``stdout``, use the
`echo` parameter::
with log_output('logfile.txt', echo=True):
# do things ... output will be logged and printed out
And, if you just want to echo *some* stuff from the parent, use
``force_echo``:
with log_output('logfile.txt', echo=False) as logger:
# do things ... output will be logged
with logger.force_echo():
# things here will be echoed *and* logged
A key difference between this and the previous implementation is that
*everything* in the context handler is logged. Previously, things like
`Executing phase 'configure'` would not be logged, only output to the
screen, so understanding phases in the build log was difficult.
- The implementation of `log_output()` is different in two major ways:
1. This implementation avoids race cases by using only one pipe (before
we had a multiprocessing pipe and a unix pipe). The logger daemon
stops naturally when the input stream is closed, which avoids a race
in the previous implementation where we'd miss some lines of output
because the parent would shut the daemon down before it was done
with all output.
2. Instead of turning output redirection on and off, which prevented
some things from being logged, this version uses control characters
in the output stream to enable/disable forced echoing. We're using
the time-honored xon and xoff codes, which tell the daemon to echo
anything between them AND write it to the log. This is how
`logger.force_echo()` works.
- Fix places where output could get stuck in buffers by flushing more
aggressively. This makes the output printed to the terminal the same
as that which would be printed through a pipe to `cat` or to a file.
Previously these could be weirdly different, and some output would be
missing when redirecting Spack to a file or pipe.
- Simplify input and color handling in both `build_environment.fork()`
and `llnl.util.tty.log.log_output()`. Neither requires an input_stream
parameter anymore; we assume stdin will be forwarded if possible.
- remove `llnl.util.lang.duplicate_stream()` and remove associated
monkey-patching in tests, as these aren't needed if you just check
whether stdin is a tty and has a fileno attribute.
- Fix issue with color formatting regular expression.
- _separators regex in spec.py could be constructed such that '^' came
first in the character matcher, e.g. '[^@#/]'. This inverts the match
and causes transient KeyErrors.
- Fixed to escape all characters in the constructed regex.
- This bug comes up in Python3 due to its more randomized hash iteration
order, but it could probably also happen in a Python 2 implementation.
- also clean up variable docstrings in spec.py
- Mac OS X Sierra has no /usr/include by default
- Instead of assuming there's an include directory in /usr, mock up a directory that looks like we expect.
This adds sbang hook support for node-js and fixes the sbang filter
for lua (the character class exclusion was swallowing newlines and
reporting a false positive if lua was mentioned anywhere in the
file).
* Docs: Travis-CI Workflow
Add a workflow how to use spack on Travis-CI.
Future Work:
depending if and how we can simplify 5101:
add a multi-compiler, multi-C++-standard, multi-software
build matrix example
* Fix Typos
* Colorize spack info. Adds prominence to preferred version. fixes#2708
This uses 'llnl.util.tty.color' to colorize the output of 'spack info'.
It also displays versions in the order the concretizer would choose
them and shows the preferred in a line on its own and in bold.
* Modified output according to Adam and Denis reviews.
Section titles are not bold + black, but bold + blue. Added a new
section named "Preferred version", which prints the preferred version
in bold characters.
* Further modifications according to Adam and Denis reviews.
After "Homepage:" we now have a single space. Removed newline after each
variant. Preferred version is not in bold fonts anymore. Added a simple
test that just runs the command.
* Improved error message for unsatisfiable specs. fixes#5066
This PR improves the error message for unsatisfiable specs by showing in tree format both the spec that cannot satisfy the constraint and the spec that asked for that constraint. After that follows a readable error message.
This PR allows additional unused properties at the top-level of the config.yaml file. Having these properties permits to use two different versions of Spack, one of which adds a new property, without receiving error messages due to the presence of this new property in a configuration cache stored in the user's home.
This fixes a syntax error in the index.html file generated by the
"spack buildcache" command when creating build caches. This also
fixes support for installing unsigned binaries.
* Refactor IntelInstaller into IntelPackage base class
* Move license attributes from __init__ to class-level
* Flake8 fixes: remove unused imports
* Fix logic that writes the silent.cfg file
* More specific version numbers for Intel MPI
* Rework logic that selects components to install
* Final changes necessary to get intel package working
* Various updates to intel-parallel-studio
* Add latest version of every Intel package
* Add environment variables for Intel packages
* Update env vars for intel package
* Finalize components for intel-parallel-studio package
Adds a +tbb variant to intel-parallel-studio.
The tbb package was renamed to intel-tbb.
Now both intel-tbb and intel-parallel-studio+tbb
provide tbb.
* Overhaul environment variables set by intel-parallel-studio
* Point dependent packages to the correct MPI wrappers
* Never default to intel-parallel-studio
* Gather env vars by sourcing setup scripts
* Use mpiicc instead of mpicc when using Intel compiler
* Undo change to ARCH
* Add changes from intel-mpi to intel-parallel-studio
* Add comment explaining mpicc vs mpiicc
* Prepend env vars containing 'PATH' or separators
* Flake8 fix
* Fix bugs in from_sourcing_file
* Indentation fix
* Prepend, not set if contains separator
* Fix license symlinking broken by changes to intel-parallel-studio
* Use comments instead of docstrings to document attributes
* Flake8 fixes
* Use a set instead of a list to prevent duplicate components
* Fix MKL and MPI library linking directories
* Remove +all variant from intel-parallel-studio
* It is not possible to build with MKL, GCC, and OpenMP at this time
* Found a workaround for locating GCC libraries
* Typos and variable names
* Fix initialization of empty LibraryList
Adds the "buildcache" command to spack. The buildcache command is
used to create gpg signatures for archives of installed spack
packages; the signatures and archives are placed together in a
directory that can be added to a spack mirror. A user can retrieve
the archives from a mirror and verify their integrity using the
buildcache command. It is often the case that the user's Spack
instance is located in a different path compared to the Spack
instance used to generate the package archive and signature, so
this includes logic to relocate the RPATHs generated by Spack.
The action `CleanOrDirtyAction` has been added. It sets the default
value for `dest` to `spack.dirty`, and changes it according to the flags
passed via command line. Added unit tests to check that the arguments
are parsed correctly. Removed lines in `PackageBase` that were setting
the default value of dirty according to what was in the configuration.
Popen.communicate outputs a str object for python2 and a bytes
object for python3. This updates the Executable.__call__ function
to call .decode on the output of Popen.communicate only for python3.
This ensures that Executable.__call__ returns a str for python2 and
python3.
fixes#4236fixes#5002
When a package is defined in more than one repository,
RepoPath.dirname_for_package_name may return the path
to either definition. This sidesteps that ambiguity by
accessing the module associated with the package definition.
* Merged 'purge' command with 'clean'. Deleted 'purge'. fixes#2942
'spack purge' has been merged with 'spack clean'. Documentation has been
updated accordingly. The 'clean' and 'purge' behavior are not mutually
exclusive, and they log brief information to tty while they go.
* Fixed a wrong reference to spack clean in the docs
* Added tests for 'spack clean'. Updated bash completion.
* Typo fixes in docstrings.
* Let OS classes know if the paths they get were explicitly specified by user.
* Fixed regexp for cray compiler version matching.
* Replaced LinuxDistro with CrayFrontend for the Cray platform's frontend.
Fixes#4898
Constraints that were supposed to be conditionally activated for
specified values of a single-valued variant were being activated
unconditionally in the case that the variant was associated with
an implicit dependency. For example if X->Y->Z and Y places a
conditional constraint on Z for a given single-valued variant on
Y, then it would have been applied unconditionally when
concretizing X.
* Add a QMakePackage base class
* Fix sqlite linking bug in qt-creator
* Add latest version of qt-creator
* Add latest version of qwt
* Use raw strings for regular expressions
* Increase minimum required version of qt
* Add comment about specific version of sqlite required
* Fixes for latest version of qwt and qt-creator
* Older versions of Qwt only work with older versions of Qt
* Fix crashes when running spack install under nohup
Fixes#4919
For reasons that I do not entire understand, duplicate_stream() throws
an '[Errno 22] Invalid argument' exception when it tries to
`os.fdopen()` the duplicated file descriptor generated by
`os.dup(original.fileno())`. See spack/llnl/util/lang.py, line
394-ish.
This happens when run under `nohup`, which supposedly has hooked
`stdin` to `/dev/null`.
It seems like opening and using `devnull` on the `input_stream` in
this situation is a reasonable way to handle the problem.
* Be more specific about error being handled.
Only catch the specific error that happens when trying to dup
the stdin that nohup provides.
Catching e as a StandardErorr and then
`type(e).__name__` tells me that it's an OSError.
Printing e.errno tells me that it's 22
Double checking tells me that 22 is EINVAL.
Phew.
- Remove `special_types` dict in spec.py, as only 'all' is still used
- Still allow 'all' to be used as a deptype
- Simplify `canonical_deptype` function
- Clean up args in spack graph
- Add tests
For packages which contain a mix of versions with formats X.Y and
X.Y.Z, if the user entered an X.Y version as a preference in
packages.yaml, Spack would get confused and favor any version A.B.Z
where X=A and Y=B. In the case where there is a mix of these version
types, this commit updates preferences so Spack will favor an exact
match.
* Disable spec colorization when redirecting stdout and add command line flag to re-enable
* Add command line `--color` flag to control output colorization
* Add options to `llnl.util.tty.color` to allow color to be auto/always/never
* Add `Spec.cformat()` function to be used when `format()` should have auto-coloring
* Add universal build_type variant to CMakePackage
* Override build_type in some packages with different possible values
* Remove reference to no longer existent debug variant
* Update CBTF packages with new build_type variant
* Keep note on build size of LLVM
* Change version.up_to() to return Version() object
* Add unit tests for Version.up_to()
* Fix packages that expected up_to() to return a string
* Ensure that up_to() preserves separator characters
* Use version indexing instead of up_to
* Make all Version formatting properties return Version objects
* Update docs
* Tests need to test string representation
Adds SpackCommand class allowing Spack commands to be easily in Python
Example usage:
from spack.main import SpackCommand
info = SpackCommand('info')
out, err = info('mpich')
print(info.returncode)
This allows easier testing of Spack commands.
Also:
* Simplify command tests
* Simplify mocking in command tests.
* Simplify module command test
* Simplify python command test
* Simplify uninstall command test
* Simplify url command test
* SpackCommand uses more compatible output redirection
* Initial work on flag trapping using functions called <flag>_handler and default_flag_handler
* Update packages so they do not obliterate flags
* Added append to EnvironmentModifications class
* changed EnvironmentModifications to have append_flags method
* changed flag_val to be a tuple
* Increased test coverage
* added documentation of flag handling
* Change path to CMakeLists.txt to be relative to root, not pwd
* Changes requested during code review
* Revert back to old naming of root_cmakelists_dir
* Make relative directory more clear in docs
* Revert change causing build_type AttributeError
* Fix forgotten abs_path var
* Update CLHEP with new relative path
* Update more packages with new root_cmakelists_dir syntax
- Lock test can be run either as a node-local test or as an MPI test.
- Lock test is now parametrized by filesystem, so you can test the
locking capabilities of your NFS, Lustre, or GPFS filesystem. See docs
for details.
* mv variants: packages are now needed only during normalization
The relationship among different types of variants have been weakened,
in the sense that now it is permitted to compare MV, SV and BV among
each other. The mechanism that permits this is an implicit conversion
of the variant passed as argument to the type of the variant asking
to execute a constrain, satisfies, etc. operation.
* asbtract variant: added a new type of variant
An abstract variant is like a multi valued variant, but behaves
differently on "satisfies" requests, because it will reply "True"
to requests that **it could** satisfy eventually.
Tests have been modified to reflect the fact that abstract variants
are now what get parsed from expressions like `foo=bar` given by users.
* Removed 'concrete=' and 'normal=' kwargs from Spec.__init__
These two keyword arguments where only used in one test module to force
a Spec to 'appear' concrete. I suspect they are just a leftover from
another refactoring, as now there's the private method '_mark_concrete'
that does essentially the same job. Removed them to reduce a bit the
clutter in Spec.
* Moved yaml related functions from MultiValuedVariant to AbstractVariant.
This is to fix the erros that are occurring in epfl-scitas#73, and that
I can't reproduce locally.
* Parse modules in a way that works for both lmod and tcl
* added test and made method more robust
* refactoring for pythonic clarity
* Improved detection of 'module' shell function + refactored module utilities into spack.util.module_cmd
* Improved regex to reject nested parentheses we are not prepared to handle
* make tests backwards compatible with python 2.6
* Improved regex to account for sh being aliased to bash and used in bash module definition on some systems
* Improve test compatibility with lmod
* Added error for None module_cmd
* Add test for get_module_cmd_from_which()
Add test for get_module_cmd_from_which().
Add -c argument to Popen call to typeset -f module in get_module_cmd_from_bash().
* Increased detection options
Included BASH_FUNC_module() variable outside of typeset as a detection option
This should work on bash even in restricted_shell mode
Kept the typeset detection as an option in case the module function is not exported in bash
Also added try statements to tests, with environment recreation in finally blocks.
* More tests added; some hackiness
* increased test coverage for util/module_cmd
* Code changes to enable system config scope in /etc
Files will go in either /etc/spack or /etc/spack/<platform>
Required minor changes to conftest.
* Updated documentation to match new config scope
- previous code called `which` on $EDITOR, but that doesn't work for
EDITORs like `emacs -nw` or `emacsclient -t -nw`.
- This patch just trusts EDITOR if it is set (same as previous
behavior), and only uses the defaults if it's not.
* issue 4492: added xfailing test, added owner to DependencyMap
* DependencyMap.concrete checks if we have unconditional dependencies
This fixes#4492 and #4107 using some heuristics to avoid an infinite
recursion among Spec.satisfies, Spec.concrete and DependencyMap.concrete
The idea, as suggested by @becker33, is to check just for unconditional
dependencies. This is not covering the whole space and a package with
just conditional dependencies can still fail in the same way. It should
cover though all the **real** packages we have in our repo so far.
* Check for CRAYPE_VERSION instead of path
Architecture tests would fail on Cray since it would not find
the expected path. To make the test correctly work on Cray search
for the CRAYPE version instead.
* Catch SystemExit error in case flake8 not in path
On shared systems having flake8 can involve starting own virtual env.
Skip the test if no flake8 is found to avoid failure reporting.
* Add compatibility to 1.5 svnadmin create
The flag added is needed to correctly create svn repos on NERSC systems.
This could be unnecessary for other sites. I'd like to see others
test before this change gets merged.
- Skip spack flake8 test when flake8 is not installed.
- Fix parsing of dashes in specs broken by new help parser.
- use argparse.REMAINDER instead of narg='?'
- don't interpret parts of specs like -mpi as arguments.
* Initial version of the namd package
* Modified charm to consider compile against intel/intel-mpi
* Correction of namd to compile with intel-mkl and intel compiler
* Adding inclue64 in the prefix
* adding property for the build directory
* removing useless function build
* During install, remove prior unfinished installs
If a user performs an installation which fails, in some cases the
install prefix is still present, and the stage path may also be
present. With this commit, unless the user specifies
'--keep-prefix', installs are guaranteed to begin with a clean
slate. The database is used to decide whether an install finished,
since a database record is not added until the end of the install
process.
* test updates
* repair_partial uses keep_prefix and keep_stage
* use of mock stage object to ensure that stage is destroyed when it should be destroyed (and otherwise not)
* add --restage option to 'install' command; when this option is not set, the default is to reuse a stage if it is found.
- Add a `spack gpg` subcommand in anticipation of signed binaries.
- GPG keys are stored in var/spack/gpg, and the spack gpg command manages them.
- Docs are included on the command.
* Touch up string expansion.
I'm chasing this:
```
$ (module purge; spack install perl %gcc/5.4.0)
==> Error: No installed spec matches the hash: '%s'
```
There's something deeper going on, but the error message isn't helpful.
After this change it tells me this:
```
$ (module purge; spack install perl %gcc/5.4.0)
==> Error: No installed spec matches the hash: '5.4.0'
```
Which is weird because `5.4.0` is not a hash... Whatever is going on here, the error message needs to be fixed.
* Flake8 whitespace
* fix parser
* Removed xfails
* cleaned up debug print statements
* make use of these changes in gcc
* Added comment explaining unreachable line, line left for added protection
* Sphinx no longer supports Python 2.6
* Update vendored sphinxcontrib.programoutput from 0.9.0 to 0.10.0
* Documentation cannot be built in parallel
* Let Travis install programoutput for us
* Remove vendored sphinxcontrib-programoutput
Recent updates to the sphinx package prevent the vendored version
from being found in sys.path. We don't vendor sphinx, so it doesn't
make sense to vendor sphinxcontrib-programoutput either.
PR #3367 inadvertently changed the semantics of _find_recursive and
_find_non_recursive so that the returned list are not ordered as the
input search list. This commit restores the original semantic, and adds
tests to verify it.
Added DFLAGS to the `make.inc` file being written.
These macros are also added to the language specific variables
like CFLAGS, CXXFLAGS and FCFLAGS. Changed `spec.satisfies('foo')`
with `'foo' in spec` in `intel-mkl`, see #4135. Added a basic
build interface to `intel-mpi`.
It seems that parse_anonymous_spec may fail if more than one part
(variant, version range, etc.) is given to the function. Added tests to
code against to fix the problem in #4144.
- Full help is now only generated lazily, when needed.
- Executing specific commands doesn't require loading all of them.
- All commands are only loaded if we need them for help.
- There is now short and long help:
- short help (spack help) shows only basic spack options
- long help (spack help -a) shows all spack options
- Both divide help on commands into high-level sections
- Commands now specify attributes from which help is auto-generated:
- description: used in help to describe the command.
- section: help section
- level: short or long
- Clean up command descriptions
- Add a `spack docs` command to open full documentation
in the browser.
- move `spack doc` command to `spack pydoc` for clarity
- Add a `spack --spec` command to show documentation on
the spec syntax.
* SV variants are evaluated correctly in `when=` statements fixes#4113
The problem here was tricky:
```python
spec.satisfies(other)
```
changes already the MV variants in others into SV variants (where
necessary) if spec is concrete. If it is not concrete it does
nothing because we may be acting at a pure syntactical level.
When evaluating a `when=` keyword spec is for sure not concrete
as it is in the middle of the concretization process. In this case we
have to trigger manually the substitution in other to not end up
comparing a MV variant "foo=bar" to a SV variant "foo=bar" and having
False in return. Which is wrong.
* sv variants: improved error message for typos in "when=" statements
Modifications:
- added support for multi-valued variants
- refactored code related to variants into variant.py
- added new generic features to AutotoolsPackage that leverage multi-valued variants
- modified openmpi to use new features
- added unit tests for the new semantics
This allows people on systems that don't have all the fetchers to still
run Spack tests. Mark tests that require git, subversion, or mercurial to
be skipped if they're not installed.
* Filter all system paths introduced by dependencies from PATH
* Make sure path filtering works *even* for trailing slashes
* Revert some of the changes to `filter_system_paths`
* Yes, `bin64` is a real thing (sigh)
* add tests: /usr, /usr/, /usr/local/../bin, etc.
* Convert from rST to Google-style docstrings
The required hash of a submodule might point to the
non-HEAD commit of the current main branch and hence
would lead to a "no such remote ref" at checkout in
a shallow submodule.
## Motivation
Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically.
I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.
Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.
## Prefix
For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.
Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.
To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.
## Command
In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:
If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.
## Libraries
Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.
The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.
## Headers
In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6
Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
When a compiler was not found a stacktrace was displayed to user because
there were three arguments to be substituted in a string with only two
substitutions to be done.
Fixes#4026#1167 updated Database.reindex to keep old installation records to
support external packages. However, when a user manually removes a
prefix and reindexes this kept the records so the packages were
still installed according to "spack find" etc. This adds a check
for non-external packages to ensure they are properly installed
according to the directory layout.
- add Version.__format__ to support new-style formatting.
- Python3 doesn't handle this well -- it delegates to
object.__format__(), which raises an error for fancy format strings.
- not sure why it doesn't call str(self).__format__ instead, but that's
hwo things are.
* Properly ignore flake8 F811 redefinition errors
* Add unit tests for flake8 command
* Allow spack flake8 to work on systems with older git
* Skip flake8 unit tests for Python 2.6 and 3.3
* treats correctly a change from `explicit=False` to `explicit=True` in an external package DB entry.
* added unit tests
* fixed issues raised by @tgamblin . In particular the PR is no more hash-changing for packages that are not external.
* added a test to check correctness of a spec/yaml round-trip for things that involve an external
* Don't find external module path at each step of concretization
* it's not necessary.. The paths are retrieved at the end of concretizaion
* Don't find replacements for external packages.
* Test root of the DAG if external
* No reason not to test if the root of the DAG is external when external
packages are now first class citizens!
* Create `external` property for Spec (for external_path and external_module)
* Allow users to specify external package paths relative to spack
* Canonicalize external package paths so that users may specify their
locations relative to spack's directory.
* Update tests to use new external_path and external properly.
* skip license hooks on external
- Spack doesn't need eggs -- it manages its own directories
- Simplify install layout and reduce sys.path searches by installing all
packages flat (eggs are deprecated for wheels, and this is also what
wheels do).
- We now supply the --single-version-externally-managed argument to
`setup.py install` for setuptools packages and setuptools.
- modify packages to only use setuptools args if setuptools is an
immediate dependency
- Remove setuptools from packages that do not need it.
- Some packages use setuptools *only* when certain args (likeb
'develop' or 'bdist') are supplied to setup.py, and they specifically
do not use setuptools for installation.
- Spack never calls setup.py this way, so just removing the setuptools
dependency works for these packages.
* fetch git submodules recursively
This is useful if the submodules have submodules themselves. On
the other hand doing a recursive update doesn't hurt if there
is only one level.
* fetch submodules with depth=1 as well (fix#2190)
* use git submodule with depth only for git>=1.8.4
- Spack install would previously fail if it could not load a package for
the thing being uninstalled.
- This reworks uninstall to handle cases where the package is no longer
known, e.g.:
a) the package has been renamed or is no longer in Spack
b) the repository the package came from is no longer registered in
repos.yaml
- gcc on macOS says it's version 4.2.1, but it's really clang, and it's
actually the *same* clang as the system clang.
- It also doesn't respond with a full path when called with
--print-file-name=libstdc++.dylib, which is expected from gcc in abi.py.
Instead, it gives a relative path and _gcc_compiler_compare doesn't
understand what to do with it. This results in errors like:
```
lib/spack/spack/abi.py, line 71, in _gcc_get_libstdcxx_version
libpath = os.readlink(output.strip())
OSError: [Errno 2] No such file or directory: 'libstdc++.dylib'
```
- This commit does two things:
1. Ignore any gcc that's actually clang in abi.py. We can probably do
better than this, but it's not clear there is a need to, since we
should handle the compiler as clang, not gcc.
2. Don't auto-detect any "gcc" that is actually clang anymore. Ignore
it and expect people to use clang (which is the default macOS
compiler anyway).
Users can still add fake gccs to their compilers.yaml if they want, but
it's discouraged.
* Checksum code wasn't opening binary files as binary.
- Fixes Python 3 issue where files are opened as unicode text by default,
and decoding fails for binary blobs.
* Simplify fetch test parametrization.
* - add tests for URL fetching and checksumming.
- fix coverage on interface functions in FetchStrategy superclass
- add some extra crypto tests.
* Package install remove prior unfinished installs
Depending on how spack is terminated in the middle of building a
package it may leave a partially installed package in the install
prefix. Originally Spack treated the package as being installed if
the prefix was present, in which case the user would have to
manually remove the installation prefix before restarting an
install. This commit adds a more thorough check to ensure that a
package is actually installed. If the installation prefix is present
but Spack determines that the install did not complete, it removes
the installation prefix and starts a new install; if the user has
enabled --keep-prefix, then Spack reverts to its old behavior.
* Added test for partial install handling
* Added test for restoring DB
* Style fixes
* Restoring 2.6 compatibility
* Relocated repair logic to separate function
* If --keep-prefix is set, package installs will continue an install from an existing prefix if one is present
* check metadata consistency when continuing partial install
* Added --force option to make spack reinstall a package (and all dependencies) from scratch
* Updated bash completion; removed '-f' shorthand for '--force' for install command
* dont use multiple write modes for completion file
* Add tests to mercurial package
* Add support for --insecure with mercurial fetching
* Install man pages and tab-completion scripts
* Add tests and latest version for all deps
* Flake8 fix
* Use certifi module to find CA certificate
* Flake8 fix
* Unset PYTHONPATH when running hg
* svn_fetch should use to svn-test, not hg-test
* Drop Python 3 support in Mercurial
Python 3 support is a work in progress and isn't currently
recommended:
https://www.mercurial-scm.org/wiki/SupportedPythonVersions
* Test both secure and insecure hg fetching
* Test both secure and insecure git and svn fetching
`set_executable` now checks if a user/group.other had read permission
on a file and if it does then it sets the corresponding executable
bit.
See #1483.
Fixes#2587
The concretizer falls back on using the root architecture (followed
by the default system architecture) to fill in unspecified arch
properties for a spec. It failed to check cases where the root could
be None.
* Remove fake URLs from Spack
* Ignore long lines for URLs that start with ftp:
* Preliminary changes to version regexes
* New redesign of version regexes
* Allow letters in version-only
* Fix detection of versions that end in Final
* Rearrange a few regexes and add examples
* Add tests for common download repositories
* Add test cases for common tarball naming schemes
* Finalize version regexes
* spack url test -> spack url summary
* Clean up comments
* Rearrange suffix checks
* Use query strings for name detection
* Remove no longer necessary url_for_version functions
* Strip off extraneous information after package name
* Add one more test
* Dot in square brackets does not need to be escaped
* Move renaming outside of parse_name_offset
* Fix versions for a couple more packages
* Fix flake8 and doc tests
* Correctly parse Python, Lua, and Bio++ package names
* Use effective URLs for mfem
* Add checksummed version to mitos
* Remove url_for_version from STAR-CCM+ package
* Revert changes to version numbers with underscores and dashes
* Fix name detection for tbb
* Correctly parse Ruby gems
* Reverted mfem back to shortened URLs.
* Updated instructions for better security
* Remove preferred=True from newest version
* Add tests for new `spack url list` flags
* Add tests for strip_name_suffixes
* Add unit tests for version separators
* Fix bugs related to parseable name but in parseable version
* Remove dead code, update docstring
* Ignore 'binary' at end of version string
* Remove platform from version
* Flip libedit version numbers
* Re-support weird NCO alpha/beta versions
* Rebase and remove one new fake URL
* Add / to beginning of regex to avoid picking up similarly named packages
* Ignore weird tar versions
* Fix bug in url parse --spider when no versions found
* Less strict version matching for spack versions
* Don't rename Python packages
* Be a little more selective, version must begin with a digit
* Re-add fake URLs
* Fix up several other packages
* Ignore more file endings
* Add parsing support for Miniconda
* Update tab completion
* XFAILS are now PASSES for 2 web tests
- _spider in web.py was actually failing to spider deeper than a certain
point.
- Fixed multiprocessing pools to not use daemons and to allow recursive
spawning.
- Added detailed tests for spidering and for finding archive versions.
- left some xfail URL finding exercises for the reader.
- Fix noqa annotations for some @when decorators
