* Allow dashes in command names and fix command name handling
- Command should allow dashes in their names like the reest of spack,
e.g. `spack log-parse`
- It might be too late for `spack build-cache` (since it is already
called `spack buildcache`), but we should try a bit to avoid
inconsistencies in naming conventions
- The code was inconsistent about where commands should be called by
their python module name (e.g. `log_parse`) and where the actual
command name should be used (e.g. `log-parse`).
- This made it hard to make a command with a dash in the name, and it
made `SpackCommand` fail to recognize commands with dashes.
- The code now uses the user-facing name with dashes for function
parameters, then converts that the module name when needed.
* Improve performance of log parsing
- A number of regular expressions from ctest_log_parser have really poor
performance, most due to untethered expressions with * or + (i.e., they
don't start with ^, so the repetition has to be checked for every
position in the string with Python's backtracking regex implementation)
- I can't verify that CTest's regexes work with an added ^, so I don't
really want to touch them. I tried adding this and found that it
caused some tests to break.
- Instead of using only "efficient" regular expressions, Added a
prefilter() class that allows the parser to quickly check a
precondition before evaluating any of the expensive regexes.
- Preconditions do things like check whether the string contains "error"
or "warning" (linear time things) before evaluating regexes that would
require them. It's sad that Python doesn't use Thompson string
matching (see https://swtch.com/~rsc/regexp/regexp1.html)
- Even with Python's slow implementation, this makes the parser ~200x
faster on the input we tried it on.
* Add `spack log-parse` command and improve the display of parsed logs
- Add better coloring and line wrapping to the log parse output. This
makes nasty build output look better with the line numbers.
- `spack log-parse` allows the log parsing logic used at the end of
builds to be executed on arbitrary files, which is handy even outside
of spack.
- Also provides a profile option -- we can profile arbitrary files and
show which regular expressions in the magic CTest parser take the most
time.
* Parallelize log parsing
- Log parsing now uses multiple threads for long logs
- Lines from logs are divided into chnks and farmed out to <ncpus>
- Add -j option to `spack log-parse`
* libdwarf: fix linker flags
* libdwarf: fix header install names
* libdwarf: add zlib dependency
Prior to this commit, libdwarf was linked to the system zlib. This
commit links it to spack-installed zlib.
* libdwarf: fix dynamic library suffix for Darwin
* Adding support for the Accelerate library on OS X
* Fixed guard on variable and added another depends_on statement to make sure that openblas gets selected even when openmp_blas variant is not turned on.
* Added a package for the MDAnalysis toolkit.
* Added some flags to fix the build RPATHs and OpenMP for LBANN on OS X.
* Fixed Hydrogen branch of Elemental to find OpenMP on OS X.
* Marking database tests as slow
* Marking url command tests as slow
* Marking every test that uses database as slow
* Marking tests that import files as slow
* Marking gpg tests as slow
* Marking all versions and one list tests as slow
* Added more markers to unit tests + cli option to skip slow tests
Following a discussion with Axel, the generic 'slowtest' marker has been
split into 'db', 'network' and 'maybeslow'. A brief description of the
meaning of each marker has been added to pytest.ini.
A command line option to run only fast tests has been added to
'spack test'
* Don't use classes to group tests together
Reverted grouping tests under a class, as required in the review
* Minor style changes
This attempts to address one of the complaints at #5996 (comment):
> Repo currently caches package instances by Spec, and those Package instances have a Spec.
> This is unnecessary and causes confusion. I think I thought that we'd need to cache instances
> after loading package classes, but really just caching the classes is fine.
With this update, Repo's package cache is removed and Specs cache the package reference themselves. One consequence is that Specs which compare as equal will store separate instances of a Package class (not doing this creates issues for #4595 (comment)).
There were several references to Spec.package that could be replaced with Spec.package_class without any additional modifications. There are still a couple remaining references to Spec.package in Spec that would require adding functionality before replacing (e.g. calling Package.provides and Package.installed).
Note this makes it difficult to mock fetchers for tests which invokes code that reconstructs specs. test_packaging was one example of this where the updates caused a failure (in that case the error was avoided by not making an unnecessary call).
Details:
* Replace instances of spec.package with spec.package_class where a class method is being called
* Remove instances of Repo.get where Spec.package_class can be used in its place
* remove Repo.get caching instances of Package class for specs
* remove redundant check (which is also incorrect now that each spec stores its own copy of its package)
* avoid creating mirror with specs because it copies specs and those copies dont refer to the mocked fetcher (and it is also not required for the test)
* remove checks that are no longer necessary since repo doesn't cache specs
* Cleaned up JUnit report generation on install
The generation of a JUnit report was previously part of the install
command. This commit factors the logic into its own module, and uses
a template for the generation of the report.
It also improves report generation, that now can deal with multiple
specs installed at once. Finally, extending the list of supported
formats is much easier than before, as it entails just writing a
new template.
* Polished report generation + added tests for failures and errors
The generation of a JUnit report has been polished, so that the
stacktrace is correctly displayed with Jenkins JUnit plugin. Standard
error is still not used.
Added unit tests to cover for installation failures and installation
errors.
* libtool: fix linking of libtool on darwin
* binutils, libtool: fix conflicts w/ BSD tools, v2
Fix namespace conflicts with BSD tools in a more elegant fashion,
using a program prefix, similar to Homebrew.
* Tell R's configure about tcltk config
Add configure arguments that specify the location of
the tcl and tk config scripts.
Fixes#7072
* Flake8 cleanup
Looking at build logs, it seems that prior to this commit bzip2 was
using 'cc', and relying on the PATH to be set correctly to find Spack
wrappers. This commit improves the robustness of the recipe, by using
the absolute path of the wrapper.
Avoid adding an "outside" (local home's) python user site directory
during python package installs.
Implements #6611
Fixes packages with auto-find side effects such as `py-setuptools`
that cause `py-matplotlib` to fail to build #6558
On Darwin and other BSD systems, the system 'libtool' and 'libtoolize'
are BSD libtool and libtoolize, respectively. Some build systems
require the GNU versions of these commands, so BSD package systems
tend to name the GNU versions 'glibtool' and 'glibtoolize',
respectively, to avoid namespace collisions.
A problem with the current libtool package is that it installs the GNU
libtool commands as 'libtool' and 'libtoolize', respectively, but
build systems on BSD platforms will attempt to run 'glibtool' and
'glibtoolize'. The expedient solution to this problem is to symlink
'libtool' to 'glibtool' and 'libtoolize' to 'glibtoolize', because
attempting to patch the detection logic one build system at a time
seems impractical.
* Added new link line flag
* Refactored some of the common components out of the different
version's build commands. Also corrected the when commands to
properly capture non-numeric names such as local.
* Added support to the Elemental package for the Hydrogen branch.
* Turning off default Qt support. Restricting Qt version to <= 4.9 else geant4 won't build
* Making vecgeom optional
* Differentiating between cxx11 and cxx14 support
* Making vecgeom optional. Part 2. forgot to move usolids option
* Adding explicit CXX11 and CXX14 support
* Adding more clhep version to support newer GEANT4 versions
* Adding GEANT4 versions 10.03.p03 and 10.04
* Checkpoint. Adding opengl and x11 variant. Lead up to more changes to enable X11 support
* Adding dependency on fixesproto for newer versions
* Making the X11 dependent pieces work without requiring X11 on the system
* Adding motif
