* Change version.up_to() to return Version() object
* Add unit tests for Version.up_to()
* Fix packages that expected up_to() to return a string
* Ensure that up_to() preserves separator characters
* Use version indexing instead of up_to
* Make all Version formatting properties return Version objects
* Update docs
* Tests need to test string representation
Adds SpackCommand class allowing Spack commands to be easily in Python
Example usage:
from spack.main import SpackCommand
info = SpackCommand('info')
out, err = info('mpich')
print(info.returncode)
This allows easier testing of Spack commands.
Also:
* Simplify command tests
* Simplify mocking in command tests.
* Simplify module command test
* Simplify python command test
* Simplify uninstall command test
* Simplify url command test
* SpackCommand uses more compatible output redirection
* Initial work on flag trapping using functions called <flag>_handler and default_flag_handler
* Update packages so they do not obliterate flags
* Added append to EnvironmentModifications class
* changed EnvironmentModifications to have append_flags method
* changed flag_val to be a tuple
* Increased test coverage
* added documentation of flag handling
* Change path to CMakeLists.txt to be relative to root, not pwd
* Changes requested during code review
* Revert back to old naming of root_cmakelists_dir
* Make relative directory more clear in docs
* Revert change causing build_type AttributeError
* Fix forgotten abs_path var
* Update CLHEP with new relative path
* Update more packages with new root_cmakelists_dir syntax
- Lock test can be run either as a node-local test or as an MPI test.
- Lock test is now parametrized by filesystem, so you can test the
locking capabilities of your NFS, Lustre, or GPFS filesystem. See docs
for details.
* mv variants: packages are now needed only during normalization
The relationship among different types of variants have been weakened,
in the sense that now it is permitted to compare MV, SV and BV among
each other. The mechanism that permits this is an implicit conversion
of the variant passed as argument to the type of the variant asking
to execute a constrain, satisfies, etc. operation.
* asbtract variant: added a new type of variant
An abstract variant is like a multi valued variant, but behaves
differently on "satisfies" requests, because it will reply "True"
to requests that **it could** satisfy eventually.
Tests have been modified to reflect the fact that abstract variants
are now what get parsed from expressions like `foo=bar` given by users.
* Removed 'concrete=' and 'normal=' kwargs from Spec.__init__
These two keyword arguments where only used in one test module to force
a Spec to 'appear' concrete. I suspect they are just a leftover from
another refactoring, as now there's the private method '_mark_concrete'
that does essentially the same job. Removed them to reduce a bit the
clutter in Spec.
* Moved yaml related functions from MultiValuedVariant to AbstractVariant.
This is to fix the erros that are occurring in epfl-scitas#73, and that
I can't reproduce locally.
* Parse modules in a way that works for both lmod and tcl
* added test and made method more robust
* refactoring for pythonic clarity
* Improved detection of 'module' shell function + refactored module utilities into spack.util.module_cmd
* Improved regex to reject nested parentheses we are not prepared to handle
* make tests backwards compatible with python 2.6
* Improved regex to account for sh being aliased to bash and used in bash module definition on some systems
* Improve test compatibility with lmod
* Added error for None module_cmd
* Add test for get_module_cmd_from_which()
Add test for get_module_cmd_from_which().
Add -c argument to Popen call to typeset -f module in get_module_cmd_from_bash().
* Increased detection options
Included BASH_FUNC_module() variable outside of typeset as a detection option
This should work on bash even in restricted_shell mode
Kept the typeset detection as an option in case the module function is not exported in bash
Also added try statements to tests, with environment recreation in finally blocks.
* More tests added; some hackiness
* increased test coverage for util/module_cmd
* Code changes to enable system config scope in /etc
Files will go in either /etc/spack or /etc/spack/<platform>
Required minor changes to conftest.
* Updated documentation to match new config scope
- previous code called `which` on $EDITOR, but that doesn't work for
EDITORs like `emacs -nw` or `emacsclient -t -nw`.
- This patch just trusts EDITOR if it is set (same as previous
behavior), and only uses the defaults if it's not.
* issue 4492: added xfailing test, added owner to DependencyMap
* DependencyMap.concrete checks if we have unconditional dependencies
This fixes#4492 and #4107 using some heuristics to avoid an infinite
recursion among Spec.satisfies, Spec.concrete and DependencyMap.concrete
The idea, as suggested by @becker33, is to check just for unconditional
dependencies. This is not covering the whole space and a package with
just conditional dependencies can still fail in the same way. It should
cover though all the **real** packages we have in our repo so far.
* Check for CRAYPE_VERSION instead of path
Architecture tests would fail on Cray since it would not find
the expected path. To make the test correctly work on Cray search
for the CRAYPE version instead.
* Catch SystemExit error in case flake8 not in path
On shared systems having flake8 can involve starting own virtual env.
Skip the test if no flake8 is found to avoid failure reporting.
* Add compatibility to 1.5 svnadmin create
The flag added is needed to correctly create svn repos on NERSC systems.
This could be unnecessary for other sites. I'd like to see others
test before this change gets merged.
- Skip spack flake8 test when flake8 is not installed.
- Fix parsing of dashes in specs broken by new help parser.
- use argparse.REMAINDER instead of narg='?'
- don't interpret parts of specs like -mpi as arguments.
* Initial version of the namd package
* Modified charm to consider compile against intel/intel-mpi
* Correction of namd to compile with intel-mkl and intel compiler
* Adding inclue64 in the prefix
* adding property for the build directory
* removing useless function build
* During install, remove prior unfinished installs
If a user performs an installation which fails, in some cases the
install prefix is still present, and the stage path may also be
present. With this commit, unless the user specifies
'--keep-prefix', installs are guaranteed to begin with a clean
slate. The database is used to decide whether an install finished,
since a database record is not added until the end of the install
process.
* test updates
* repair_partial uses keep_prefix and keep_stage
* use of mock stage object to ensure that stage is destroyed when it should be destroyed (and otherwise not)
* add --restage option to 'install' command; when this option is not set, the default is to reuse a stage if it is found.
- Add a `spack gpg` subcommand in anticipation of signed binaries.
- GPG keys are stored in var/spack/gpg, and the spack gpg command manages them.
- Docs are included on the command.
* Touch up string expansion.
I'm chasing this:
```
$ (module purge; spack install perl %gcc/5.4.0)
==> Error: No installed spec matches the hash: '%s'
```
There's something deeper going on, but the error message isn't helpful.
After this change it tells me this:
```
$ (module purge; spack install perl %gcc/5.4.0)
==> Error: No installed spec matches the hash: '5.4.0'
```
Which is weird because `5.4.0` is not a hash... Whatever is going on here, the error message needs to be fixed.
* Flake8 whitespace
* fix parser
* Removed xfails
* cleaned up debug print statements
* make use of these changes in gcc
* Added comment explaining unreachable line, line left for added protection
* Sphinx no longer supports Python 2.6
* Update vendored sphinxcontrib.programoutput from 0.9.0 to 0.10.0
* Documentation cannot be built in parallel
* Let Travis install programoutput for us
* Remove vendored sphinxcontrib-programoutput
Recent updates to the sphinx package prevent the vendored version
from being found in sys.path. We don't vendor sphinx, so it doesn't
make sense to vendor sphinxcontrib-programoutput either.
PR #3367 inadvertently changed the semantics of _find_recursive and
_find_non_recursive so that the returned list are not ordered as the
input search list. This commit restores the original semantic, and adds
tests to verify it.
Added DFLAGS to the `make.inc` file being written.
These macros are also added to the language specific variables
like CFLAGS, CXXFLAGS and FCFLAGS. Changed `spec.satisfies('foo')`
with `'foo' in spec` in `intel-mkl`, see #4135. Added a basic
build interface to `intel-mpi`.
It seems that parse_anonymous_spec may fail if more than one part
(variant, version range, etc.) is given to the function. Added tests to
code against to fix the problem in #4144.
- Full help is now only generated lazily, when needed.
- Executing specific commands doesn't require loading all of them.
- All commands are only loaded if we need them for help.
- There is now short and long help:
- short help (spack help) shows only basic spack options
- long help (spack help -a) shows all spack options
- Both divide help on commands into high-level sections
- Commands now specify attributes from which help is auto-generated:
- description: used in help to describe the command.
- section: help section
- level: short or long
- Clean up command descriptions
- Add a `spack docs` command to open full documentation
in the browser.
- move `spack doc` command to `spack pydoc` for clarity
- Add a `spack --spec` command to show documentation on
the spec syntax.
* SV variants are evaluated correctly in `when=` statements fixes#4113
The problem here was tricky:
```python
spec.satisfies(other)
```
changes already the MV variants in others into SV variants (where
necessary) if spec is concrete. If it is not concrete it does
nothing because we may be acting at a pure syntactical level.
When evaluating a `when=` keyword spec is for sure not concrete
as it is in the middle of the concretization process. In this case we
have to trigger manually the substitution in other to not end up
comparing a MV variant "foo=bar" to a SV variant "foo=bar" and having
False in return. Which is wrong.
* sv variants: improved error message for typos in "when=" statements
Modifications:
- added support for multi-valued variants
- refactored code related to variants into variant.py
- added new generic features to AutotoolsPackage that leverage multi-valued variants
- modified openmpi to use new features
- added unit tests for the new semantics
This allows people on systems that don't have all the fetchers to still
run Spack tests. Mark tests that require git, subversion, or mercurial to
be skipped if they're not installed.
* Filter all system paths introduced by dependencies from PATH
* Make sure path filtering works *even* for trailing slashes
* Revert some of the changes to `filter_system_paths`
* Yes, `bin64` is a real thing (sigh)
* add tests: /usr, /usr/, /usr/local/../bin, etc.
* Convert from rST to Google-style docstrings
The required hash of a submodule might point to the
non-HEAD commit of the current main branch and hence
would lead to a "no such remote ref" at checkout in
a shallow submodule.
## Motivation
Python installations are both important and unfortunately inconsistent. Depending on the Python version, OS, and the strength of the Earth's magnetic field when it was installed, the name of the Python executable, directory containing its libraries, library names, and the directory containing its headers can vary drastically.
I originally got into this mess with #3274, where I discovered that Boost could not be built with Python 3 because the executable is called `python3` and we were telling it to use `python`. I got deeper into this mess when I started hacking on #3140, where I discovered just how difficult it is to find the location and name of the Python libraries and headers.
Currently, half of the packages that depend on Python and need to know this information jump through hoops to determine the correct information. The other half are hard-coded to use `python`, `spec['python'].prefix.lib`, and `spec['python'].prefix.include`. Obviously, none of these packages would work for Python 3, and there's no reason to duplicate the effort. The Python package itself should contain all of the information necessary to use it properly. This is in line with the recent work by @alalazo and @davydden with respect to `spec['blas'].libs` and friends.
## Prefix
For most packages in Spack, we assume that the installation directory is `spec['python'].prefix`. This generally works for anything installed with Spack, but gets complicated when we include external packages. Python is a commonly used external package (it needs to be installed just to run Spack). If it was installed with Homebrew, `which python` would return `/usr/local/bin/python`, and most users would erroneously assume that `/usr/local` is the installation directory. If you peruse through #2173, you'll immediately see why this is not the case. Homebrew actually installs Python in `/usr/local/Cellar/python/2.7.12_2` and symlinks the executable to `/usr/local/bin/python`. `PYTHONHOME` (and presumably most things that need to know where Python is installed) needs to be set to the actual installation directory, not `/usr/local`.
Normally I would say, "sounds like user error, make sure to use the real installation directory in your `packages.yaml`". But I think we can make a special case for Python. That's what we decided in #2173 anyway. If we change our minds, I would be more than happy to simplify things.
To solve this problem, I created a `spec['python'].home` attribute that works the same way as `spec['python'].prefix` but queries Python to figure out where it was actually installed. @tgamblin Is there any way to overwrite `spec['python'].prefix`? I think it's currently immutable.
## Command
In general, Python 2 comes with both `python` and `python2` commands, while Python 3 only comes with a `python3` command. But this is up to the OS developers. For example, `/usr/bin/python` on Gentoo is actually Python 3. Worse yet, if someone is using an externally installed Python, all 3 commands may exist in the same directory! Here's what I'm thinking:
If the spec is for Python 3, try searching for the `python3` command.
If the spec is for Python 2, try searching for the `python2` command.
If neither are found, try searching for the `python` command.
## Libraries
Spack installs Python libraries in `spec['python'].prefix.lib`. Except on openSUSE 13, where it installs to `spec['python'].prefix.lib64` (see #2295 and #2253). On my CentOS 6 machine, the Python libraries are installed in `/usr/lib64`. Both need to work.
The libraries themselves change name depending on OS and Python version. For Python 2.7 on macOS, I'm seeing:
```
lib/libpython2.7.dylib
```
For Python 3.6 on CentOS 6, I'm seeing:
```
lib/libpython3.so
lib/libpython3.6m.so.1.0
lib/libpython3.6m.so -> lib/libpython3.6m.so.1.0
```
Notice the `m` after the version number. Yeah, that's a thing.
## Headers
In Python 2.7, I'm seeing:
```
include/python2.7/pyconfig.h
```
In Python 3.6, I'm seeing:
```
include/python3.6m/pyconfig.h
```
It looks like all Python 3 installations have this `m`. Tested with Python 3.2 and 3.6 on macOS and CentOS 6
Spack has really nice support for libraries (`find_libraries` and `LibraryList`), but nothing for headers. Fixed.
When a compiler was not found a stacktrace was displayed to user because
there were three arguments to be substituted in a string with only two
substitutions to be done.
Fixes#4026#1167 updated Database.reindex to keep old installation records to
support external packages. However, when a user manually removes a
prefix and reindexes this kept the records so the packages were
still installed according to "spack find" etc. This adds a check
for non-external packages to ensure they are properly installed
according to the directory layout.
- add Version.__format__ to support new-style formatting.
- Python3 doesn't handle this well -- it delegates to
object.__format__(), which raises an error for fancy format strings.
- not sure why it doesn't call str(self).__format__ instead, but that's
hwo things are.
* Properly ignore flake8 F811 redefinition errors
* Add unit tests for flake8 command
* Allow spack flake8 to work on systems with older git
* Skip flake8 unit tests for Python 2.6 and 3.3
