- Failed to catch all instances of modify_module when it was renamed
to setup_dependent_package.
- Refactored remaining modify_module calls.
- Also modified Python's setup_dependent_package slightly: only
creates empty site-packages directory for Python extensions now, not
for all dependents.
- Fixed in package.py
- Fixed wrong prototypes in packages that use it.
- Fixed build_environment to set module variables properly
- added hacky fix to ensure spec/package consistency in build processes.
- Need to think about defensive spec copy done by `Repo.get`. May be
time to think about an immutable spec implementation.
- Gave setup_environment and setup_dependent_environment more similar
signatures. They now allows editing the Spack env and the runtime
env for *this* package and dependents, respectively.
- modify_module renamed to setup_dependent_python_module for symmetry
with setup_dependent_environment and to avoid confusion with
environment modules.
- removed need for patching Package objects at runtime.
- adjust packages to reflect these changes.
- `_expand_virtual_packages` now gets a candidate list and will try
all the candidates.
- Good news: If the first virtual in the list conflicts with something else in
the spec, we'll keep trying until we find a good one.
- Bad news: Only looks as far as the next normalize(); can't see
conflicts further ahead than that if they're inevitable
some other virtual expansion.
- Refactor `concretize.py` to keep all the nasty spec graph stitching in
`spec.py`. This is more similar to before externals support.
- `concretize.py` now just returns a list of candidates sorted by
ABI compatibility to `_expand_virtual_packages`, and `spec.py`
handles testing the candidates.
- Refactor the way external paths are handled in `config.py` and `concretize.py`:
- previously, `spec_externals` returned spec/path pairs. Now it
returns specs with `external` set. Makes code in `concretize.py`
more natural.
- Shouldn't call .package from within things like normalize() and
concretize() beacuse spec may be inconsistent.
- Add `.package_class` property so that we can get at package metadata
without constructing a Package with a Spec.
- should be faster than `.package` was, anyway. Use where possible.
not change the behavior for existing packages that subclass from
spack.Package.
If a package subclasses spack.StagedPackage instead of spack.Package,
the install() phase (run inside a forked process) is now separated
into sub-stages:
a) spconfig: Generate a script spconfig.py that will configure the
package (eg by running CMake or ./configure) This is for use if
the user wishes to build externally from Spack. Therefore, the
Spack compiler wrappers are NOT used here. Currently, that
means that RPATH support is up to the user.
b) configure: Configure the project (eg: runs configure, CMake,
etc). This will configure it for use within Spack, using the
Spack wrapper.
c) build: eg: "make"
d) install: eg: "install"
If one chooses to use StagedPackage instead of Package, then one must
implement each of the install sub-stages as a separate method.
StagedPackage.install() then calls each of the sub-stages as
appropriate.
StagedPackage can be configured to only run certain sub-stages. This
is done by setting the optional kwarg install_phases when calling
do_install(). Setting install_phases() ONLY has an effect on
StagedPackage, not on any existing packages. By default,
install_phases is set to make StagedPackage run the same stages that
are normally run for any package: configure, build, install (and NOT
spconfig).
The ability for Spack to run stages selectively for StagedPackage
instances will enable new functionality. For example, explicit
CMake/Autotools helpers that allow Spack to help configure a user's
project without fetching, building or installing it.
-------------
One implementation of StagedPackage is provided, CMakePackage. This
has the following advantage for CMake-based projects over using the
standard Package class:
a) By separating out the phases, it enables future new functionality
for packages that use it.
b) It provides an implementation of intall_spconfig(), which will
help users configure their CMake-based projects.
CMakePackage expects users to implement configure_args() and
configure_env(). These methods provide the package-specific arguments
and environment needed to properly configure the package. They are
placed in separated functions because they are used in both the
spconfig and configure stages.
TODO:
1. Generate spconfig.py scripts that are more readable. This allows
the users to understand how their project is configured.
2. Provide a practical way for the user to ACCESS the spconfig stage
without building the project through Spack.
3. The CMAKE_TRANSITIVE_INCLUDE_PATH stuff needs to be reworked; it
should be considered provisional for now.
4. User of Autotools might wish to make a similar ConfigurePackage
subclass of StagedPackage.
---------------
One package using CMakePackage is introduced. See ibmisc/package.py.
- Move `Spec.__cmp__` out of spec, into concretize as `cmp_specs`.
- `Spec.__cmp__` was never called (except explicitly) due to rich
comparison operators from `key_ordering`
- Refactor `_find_other_spec` to free function `find_spec`. Add a test
for it to make sure it works.
- Had attempted to add more functionality by assigning different
meanign None, True, and False values "keep_stage" (where False was
"always delete").
- Turns out that's not really worth the complexity. Having the third
"always delete" sense is hardly ever useful but makes the code hard
to understand.
- Fix bug introduced during merge of stage refactor.
- install prefix was not created before build_environment.fork()
- build_environment.fork() calls setup_dependent_environment
- python's setup_dependent_environment can inadvertently create
the install prefix before directory_layout expects it.
- Clean up Package.do_install:
- simplify control flow: parent process now entirely responsible for
creating/destroying the install prefix. cleanup is now in one place.
- Hoisting cleanup out of the child improves nesting of try/catch in
`real_work`.
- `real_work` renamed to `build_process`
- Allows skipping the expand step for downloads.
- Fixed stage so that it knows expansion didn't fail when there is a
no-expand URLFetchStrategy.
- Updated docs to reflect new option, and provided an example.
- package.py uses context manager more effectively.
- Stage.__init__ has easier to understand method signature now.
- keep can be used to override the default behavior either to keep
the stage ALL the time or to delete the stage ALL the time.
- This reverts commit c5d9ee8924.
- merged too soon before
- reverting and fixing bugs now.
Conflicts:
lib/spack/spack/mirror.py
lib/spack/spack/package.py
This does several things:
- Add `sbang`: a script to run scripts with long shebang lines.
- Documentation for `sbang` is in `bin/sbang`.
- Add an `sbang` hook that filters the `bin` directory after install
and modifies any scripts wtih shebangs that are too long to use
`sbang` instead.
- `sbang` is at the top level, so it should be runnable (not much we
can do if spack itself is too deep for shebang)
- `sbang`, when used as the interpreter, runs the *second* shebang
line it finds in a script.
- shoud fix issues with too long shebang paths.
Compiler detection was not getting triggered properly with some of the
new config logic. Adjust the conditions under which Spack will serach
for compilers.
- Recent external repo refactoring made `repo_for_pkg` raise an error
when the package was not known.
- Correct behavior is to return the highest precedence repo.
- Old test: did not attempt to actually fetch mirrored packages.
- New test:
1. Creates a temporary mirror,
2. Registers it with spack,
3. Fetches from it, and
4. Verifies that the fetched archive matches the original
- This test will hopefully mean that `spack mirror` is less brittle
from now on.
- simplify output, error, and input redirection
- `return_output=True` is now `output=str`
- `return_output=True` will still work for the time being but is
deprecated.
- previously you could say `return_output=True` and `output=<stream>`,
which wouldn't actually write to the stream. Now you actually can't
specify erroneous cases since there is only one parameter with
mutually exclusive options..
This is necessary for the NAG Fortran compiler, which prints its version message to STDERR instead of STDOUT. This message was previously being ignored, preventing spack from finding the compiler's version or automatically adding it to the compilers.yaml configuration file.
* develop: (77 commits)
Make libxcb compile with gcc 4.9.
disable parallel install for glib (found races)
disable cairo-trace, which is incompatible with older libiberty.h.
Fix LaunchMon on newer gcc versions.
Simplify output redirection in spack.util.executable
Allow completely empty config files.
Update cmake package to use http in all URLs.
Fix create, diy, edit, and repo commands to use multiple repos.
Add namespace option to find command.
Temporary fix: Clang is the default compiler on Mac OS X.
Netcdf requires cmake >=2.8.12
Mbedtls depends on cmake
Update libtool to 2.4.6
Update Julia to 0.4.3
Make binutils build on OS X
Add verbs and psm variants
py-matplotlib: depend on freetype
py-numpy: make blas/lapack optional dependencies
qhull: apply patch to work with libc++'s standard library
New package GNU tar
...
- By default inherit parent's input/output descriptor
- Only use pipes if we need to return output.
- Allows subprocesses (like emacsclient) to detect terminal correctly
- This moves var/spack/packages to var/spack/repos/builtin/packages.
- Packages that did not exist in the source branch, or were changed in
develop, were moved into var/spack/repos/builtin/packages as part of
the integration.
Conflicts:
lib/spack/spack/test/unit_install.py
var/spack/repos/builtin/packages/clang/package.py
- mirrors.yaml now uses dict order for precedence, instead of lists of
dicts.
- spack.cmd now specifies default scope for add/remove and for list
with `default_modify_scope` and `default_list_scope`.
- commands that only read or list default to all scopes (merged)
- commands that modify configs modify user scope (highest
precedence) by default
- These vars are used in setup_paraser for mirror/repo/compiler.
- Spack's argparse supports aliases now.
- added 'rm' alias for `spack [repo|compiler|mirror] remove`
This update significantly reworks the llvm and clang packages. The llvm
package now includes variants allowing it to build and install any and
all of:
* clang
* lldb
* llvm's libunwind (why, WHY did they name it this?!?)
* polly (including building it directly into the clang tools, 3.7.0 only)
* clang extra tools
* compiler-rt (sanitizers)
* clang lto (the gold linker plugin that allows same to work)
* libcxx/libcxxabi
* libopenmp, also setting the default openmp runtime to same, when
parameters happen this shoudl be an option of libomp or libgomp
Ideally, this should have rpath setup like the gcc package does, but
clang's driver has no support for specs as such, and no clearly
equivalent mechanism either. If anyone has ideas on this, they would be
welcome.
One significant note related to gcc though, if you test this on LLNL
systems, or anywhere that has multiple GCCs straddling the dwarf2
boundary and sharing a libstdc++, build a gcc with spack and use that to
build clang. If you use a gcc4.8+ to build this with an older
libstdc++ it will fail on missing unwind symbols because of the
discrepancy.
Resource handling has been changed slightly to move the unpacked archive
into the target rather than use symlinks, because symlinks break certain
kinds of relative paths, and orders resource staging such that nested
resources are unpacked after outer ones.
This solution doesn't really make me happy, but does seem to work. It
sorts the resources by the length of the string representing their
destination. Since any nested resource must contain another resource's
name in its path, it seems that should work, but there should be a
better way to do this.
This allows resources to be placed into subdirectory trees that may not
exist in the base package, and may depend on other resources to be
staged later.
- All of these work:
- `spack mirror add`
- `spack mirror remove`
- `spack mirror list`
- `spack mirror` subcommands (except create) now have their own
--scope argument.
- Mirror config is now stored sanely as an ordered list.
- `spack compiler` subcommands now take an optional --scope argument.
- no more `remove_from_config` in `config.py` -- `update` just
overwrites b/c it's easier to just call `get_config`, modify YAML
structures directly, and then call `update`.
- Implemented `spack compiler remove`.
- Configs are now parsed with `spack.util.spack_yaml.load/dump`
- Parser annotates returned data with `_start_mark` and `_end_mark`
properties, so that we can recover what lines/files they came from.
- Parser uses `OrderedDict` instead of `dict`. This will help
maintain some sanity when round-tripping config files.
- User and site config are now kept separately in memory.
- Merging is done on demand when client code requests the configuration.
- Allows user/site config to be updated independently of each other by commands.
- simplifies config logic (no more tracking merged files)
- Stage and fetcher were not being set up properly when fetching using
a different fetch strategy than the default one for the package.
- This is fixed but fetch/stage/mirror logic is still too complicated
and long-term needs a rethink.
- Spack will now print a warning when fetching a checksum-less tarball
from a mirror -- users should be careful to use https or local
filesystem mirrors for this.
- Move `find_versions_of_archive` from spack.package to `spack.util.web`.
- `spider` funciton now just uses the link parsing it already does to
return links. We evaluate actual links found in the scraped pages
instead of trying to reconstruct them naively.
- Add `spack url-parse` command, which you can use to show how Spack
interprets the name and version in a URL.
Versions found by wildcard URLs are different from versions found by
parse_version, etc. The wildcards are constructed more haphazardly
than the very specific URL patterns in url.py, so they can get things
wrong. e.g., for this URL:
https://software.lanl.gov/MeshTools/trac/attachment/wiki/WikiStart/mstk-2.25rc1.tgz
We miss the 'rc' and only return 2.25r as the version if we ONLY use
URL wildcards.
Future: Maybe use the regexes from url.py to scrape web pages, and
then compare them for similarity with the original URL, instead of
trying to make a structured wildcard URL pattern? This might yield
better results.
- remove getcwd() check (seems arbitrary -- if users set their TMPDIR
to this why stop them?)
- try a number of common locations and try per-user directories in
them first.
- Adding `preferred=True` to a version directive will change its sort
order in concretization.
- This provides us a rudimentary ability to keep the Spack stack
stable as new versions are added.
- Having multiple stacks will come next, but this at least allows us
to specify default versions of things instead of always taking the
newest.
This changes the compiler wrappers so that they are called by the same
name as the wrapped compiler. Many builds make assumptions about
compiler names, and we need the spack compilers to be recognizable so
that build systems will get their flags right.
This adds per-compiler subdirectories to lib/spack/spack/env directory
that contain symlinks to cc for the C, C++, F77, and F90
compilers. The build now sets CC, CXX, F77, and F90 to point to these
links instead of to the generically named cc, c++, f77, and f90
wrappers.
- Extension logic didn't take conditional deps into account.
- Extension methods now check for whether the extnesion is
in the extendee map AND whether the dependency is actually present
in the spec yet.
This changes the hash algorithm so that it does much less object
allocation and copying, and so that it is correct.
The old version of `_cmp_key()` would call `sorted_deps`, which would
call `flat_dependencies` to get a list of dependencies so that it
could sort them in alphabetical order. This isn't necessary in the
`_cmp_key()`, and in fact we want more DAG structure than that to be
included in the `_cmp_key()`.
The new version constructs a tuple without copying the Spec DAG, and
the tuple contains hashes of sub-DAGs that are computed recursively
in-place. This is way faster than the previous algorithm and reduces
the numebr of copies significantly. It is also a correct DAG hash.
Example timing and copy counts for the different hashing algorithms
we've tried:
Original (wrong) Spec hash:
```
106,170 copies
real 0m5.024s
user 0m4.949s
sys 0m0.104s
```
Spec hash using YAML `dag_hash()`:
```
3,794 copies
real 0m5.024s
user 0m4.949s
sys 0m0.104s
New no-copy, no-YAML hash:
```
3,594 copies
real 0m2.543s
user 0m2.435s
sys 0m0.104s
```
So now we have a hash that is correct AND faster.
The remaining ~3k copies happen mostly during concretization, and as
all packages are initially loaded. I believe this is because Spack
currently has to load all packages to figure out virtual dependency
information; it could also be becasue there ar a lot of lookups of
partial specs in concretize. I can investigate this further.
- _cross_provider_maps() had suffered some bit rot (map returned was
ill-formed but still worked for cases with one vdep)
- ProviderIndex.satisfies() was only checking whether the result map
was non-empty. It should check whether all common vdeps are *in*
the result map, as that indicates there is *some* way to satisfy
*all* of them. We were checking whether there was some way to
satisfy *any one* of them, which is wrong.
- Above would cause a problem when there is more than one vdep provider.
- Added test that covers this case.
- Added `constrained()` method to Spec. Analogous to `normalized()`:
`constrain():constrained() :: normalize():normalized()`
Added test for, e.g.:
import spack.pkg.builtin.mock.mpich
import spack.pkg.builtin.mock.mpich as mpich
from spack.pkg.builtin.mock.mpich import Mpich
Among others. These ensure that direct package imports work so that
packages can be extended.
Package repositories now look like this:
top-level-dir/
repo.yaml
packages/
libelf/
package.py
mpich/
package.py
...
This leaves room at the top level for additional metadata, source,
per-repo configs, indexes, etc., and it makes it easy to see that
something is a spack repo (just look for repo.yaml and packages).
unit tests, so tracking tests with sets wouldn't work unless I extracted the
details relevant to the particular test. For now a simple count will work so
using a set was unnecessary anyways.
1. Adding a plugin to keep track of the total number of tests run as well as the
number of tests with failures/errors.
2. Some nose plugins (including xunit which will be added in a future commit)
assign stdout to a stream object that does not have a .fileno attribute.
spack.util.executable.Executable now avoids passing stdout to subprocess (and
always uses subprocess.PIPE)
TODO:
1. Still need to figure out how to activate the plugin (as of now it is
being ignored by nose). Newer versions of nose appear to make this simpler
(e.g. the "addplugins" argument to nose.run)
2. Need to include new version of nose in order to use xunit
python install and lib/spack/external have the same library installed. This
requires modifying the names of some modules in lib/spack/external in cases
where both the system python and backported features of future python versions
(i.e. after 2.6) are used (previously distinguished by "from external import X"
and "import X").
Flags are passed from the command line all the way through
build environments to environment variables.
Flags are specified using +name=value and values are quoted
using escaped quotes when necessary.
Future work includes using the flags in the compiler wrapper script
and hopefully updating the parser for a gentler user experience of
the spec language.
- `remove_prefix` was modified to remove from the DB, but the package
may not have been added to the DB yet when `remove_prefix` is called
from `cleanup`.
- Made `remove_prefix` a pure utility function (it just removes the prefix)
- Added `installed_db.remove()` call only after the `remove_prefix` in
`uninstall`.
- Code simplification ignored case where exception was raised.
- If LockError was raised, read and write counts were incremented erroneously.
- updated lock test.
output path is specified. Test output files are placed in this directory.
Furthermore the filenames now have the prefix "test" (but otherwise are the
string representation of the spec ID as before).
- llnl.util.lock now uses fcntl.lockf instead of flock
- purported to have more NFS compatibility.
- Added an extensive test case for locks.
- tests acquiring, releasing, upgrading, timeouts, shared, & exclusive cases.
convert individual bytes, so therefore incorrectly converts utf-8 encoded
characters. Decoding byte strings to unicode objects results in correct HTML
number encodings.
2. The parent of a failed child process in build_environment.fork no longer
calls sys.exit - instead it raises a CommandError (from [1])
3. test-install command now attempts to install all packages even if one fails
command by having it consume exactly 1 positional argument (i.e. by removing
"nargs=argparse.REMAINDER") but this does not work when configuring dependencies
of a top-level package (which show up as additional positional args). Instead
now there is an explicit check to ensure there is only 1 top-level package.
argument). The default path is [package id].xml in the CWD where test-install
is called from.
2. Fixed a bug with package.build_log_path (which was added in this branch).
3. keep_stage for package.do_install is now set. This allows uninstalling and
reinstalling packages without (re) downloading them.
Also even though I calculated which installs are new (e.g. vs. packages that
have already been installed by a previous command) I forgot to make use of that
in create_test_output (so I was always generating test output even if a package
had been installed before running the test-install command).
Note to avoid confusion: the 'handled' variable (removed in this commit) did not
serve the same purpose as 'newInstalls': it was originally required because the
recursive approach would visit the same dependency twice if more than one
package depended on it.
packages are specified and a prior one fails, it will prevent any of the others
from succeeding (and generating test output) even if they don't share
dependencies.
the original intent was to generate output as if each package was a unit test,
but I noticed that I was only generating test output for top-level packages.
Still need to add output formatting (in a commonly parse-able format like Junit
or TAP). May want to adjust how the build log is accessed in case of a build
failure.
1. Database stores a file version, so we can add to it in the future.
2. Database indexed by hashes and not numerical indexes.
3. Specs built by database have consistent hashes and it's checked.
4. minor naming and whitespace changes.
Most importantly wrote the Lock, Read_Lock_Instance, and Write_Lock_Instance classes in lock.py
Updated the locking in database.py
TODO: Lock on larger areas
Replaced them all with references to the database
Implemented caching in the database. The database now only re-reads data
if the database file exists and was changed since this file last wrote to it.
Added the installed_db field to the spack instance
Left the call to all_specs from testdirectory_layout.py for now.
No methods use the database so far.
Also, a bug fix:
Previous version did not remove the staging directory on a failed install
This led to spack refusing to uninstall dependencies of the failed install
Added to cleanup() to blow away the staging directory on failed install.
Certain remote protocols don't support the `--depth` option. Since this can't
be checked by URL type or in any sane way locally, this version attempts to
clone git repositories with the --depth option, and if that fails attempts the
clone again without it.
Ensures all tags are ready before checkout, using `--branch` if possible and
an extra pull if that is not available. Also adds `--depth 1` to create
shallow clones if the git version is sufficient.
Fixes#64.
It is currently less painful to pull the source from github, compile it into a
gem, then install the gem, than it is to download a gem and install it. This
still lacks an activation mechanism, but `spack use tmuxinator` is functional.
- This can result in the user being prompted to download an unsafe
version.
- Avoids overly strict errors when something *could* be satisfiable
but we don't know about hte version.
- The following now work differently:
spec['mpi']
spec['blas']
This can return a spec for openmpi, mpich, mvapich, etc., EVEN if
the spec is already concretized. This means that in a package that
`depends_on('mpi')`, you can do `spec['mpi']` to see what it was
concretized to. This should simplify MPI and BLAS packages.
'mpi' in spec
'blas' in spec
Previously, if the spec had been concretized, these would be `False`
because there was not a dependency in the DAG with either of these
names. These will now be `True` even if the spec has been
concretized. So, e.g., this will print "YES"
s = Spec('callpath ^mpich')
if 'mpi' in spec:
print "YES"
- Similarly, this will be True:
Spec('mpich').satisfies('mpi')
- Because of the way virtual dependencies are currently implemented,
the above required some fiddling around with `package.py` so that it
would never call `Spec.__contains__` (and result in endless
recursion).
- This should be fixed by allowing virutal dependnecies to have their
own package class.
- This would allow a quicker check for vdeps, without a call to
`all_packages`.
- For the time being, `package.py` shouldn't call `__contains__`
- Expanding archvies like MAGMA 1.6.2 creates extra hidden files that
confuse Spack's staging mechanism.
- Added a special case to ignore hidden files when checking whether
the tarball exploded.
- spack install suppresses build output by default.
- use install -v to show build output on the console too
- package.py uses log_output context to redirect output and log it to a file
- filters color codes out of output written to file
- optionally echos to the terminal
- YAML directory layout knows about its build log.
- can get path to install build log to from directory layout
- Package.install now copies the build log to $prefix/.spack/build.out
- Error message from failed install execution now includes build log location
This includes:
- Much better variant support (+debug/-debug)
- Optional dependency support (depends_on(... , when='<condition>')
- New config file format (YAML in ~/.spack)
- New Spec format (YAML in $prefix/.spack/spec.yaml)
- Can depend conditionally based on variant, compiler, arch, deps, etc
- normalize() is not iterative yet: no chaining depends_ons
- really need a SAT solver, but iterative will at least handle
simple cases.
- Added "strict" option to Spec.satisfies()
- strict checks that ALL of other's constraints are met (not just
the ones self shares)
- Consider splitting these out into two methods: could_satisfy() and
satisfies()
- didn't do this yet as it would require changing code that uses
satisfies()
- Changed semantics of __contains__ to use strict satisfaction (SPACK-56)
- Added tests for optional dependencies.
- The constrain() method on Specs, compilers, versions, etc. now
returns whether the spec changed as a result of the call.
- Variant concretization is tricky:
- During concretization, a spec without variants (e.g., mpich) means
"don't care". So, Spec('mpich').satisfies('mpich+debug') is true
because it *could* still be built that way.
- After concretization, a spec without a particular variant means
"don't know", as that wasn't part of the spec, so the opposite
relationship is true. Assume 'spec' is already installed:
spec.satisfies('mpich+debug')
this is false beacuse the `debug` variant didn't exist when spec
was built, so we can't satisfy the explicit request for +debug.
- Variants are now declarable in packages using the variant() directive.
- Variants are checked - you can't just ask for a random variant, it has to be declared.
- conditional logic (@when, if, '+debug' in spec, etc.) still required in package to
implement variant.
Suppress python stack trace on build error UNLESS in debug mode (spack -d).
Now spack shows errors with a single red arrow, and it's easier to find the actual build output.
