df758e1cfc
* Allow dashes in command names and fix command name handling
- Command should allow dashes in their names like the reest of spack,
e.g. `spack log-parse`
- It might be too late for `spack build-cache` (since it is already
called `spack buildcache`), but we should try a bit to avoid
inconsistencies in naming conventions
- The code was inconsistent about where commands should be called by
their python module name (e.g. `log_parse`) and where the actual
command name should be used (e.g. `log-parse`).
- This made it hard to make a command with a dash in the name, and it
made `SpackCommand` fail to recognize commands with dashes.
- The code now uses the user-facing name with dashes for function
parameters, then converts that the module name when needed.
* Improve performance of log parsing
- A number of regular expressions from ctest_log_parser have really poor
performance, most due to untethered expressions with * or + (i.e., they
don't start with ^, so the repetition has to be checked for every
position in the string with Python's backtracking regex implementation)
- I can't verify that CTest's regexes work with an added ^, so I don't
really want to touch them. I tried adding this and found that it
caused some tests to break.
- Instead of using only "efficient" regular expressions, Added a
prefilter() class that allows the parser to quickly check a
precondition before evaluating any of the expensive regexes.
- Preconditions do things like check whether the string contains "error"
or "warning" (linear time things) before evaluating regexes that would
require them. It's sad that Python doesn't use Thompson string
matching (see https://swtch.com/~rsc/regexp/regexp1.html)
- Even with Python's slow implementation, this makes the parser ~200x
faster on the input we tried it on.
* Add `spack log-parse` command and improve the display of parsed logs
- Add better coloring and line wrapping to the log parse output. This
makes nasty build output look better with the line numbers.
- `spack log-parse` allows the log parsing logic used at the end of
builds to be executed on arbitrary files, which is handy even outside
of spack.
- Also provides a profile option -- we can profile arbitrary files and
show which regular expressions in the magic CTest parser take the most
time.
* Parallelize log parsing
- Log parsing now uses multiple threads for long logs
- Lines from logs are divided into chnks and farmed out to <ncpus>
- Add -j option to `spack log-parse`
|
||
|---|---|---|
| bin | ||
| etc/spack/defaults | ||
| lib/spack | ||
| share/spack | ||
| templates | ||
| var/spack | ||
| .codecov.yml | ||
| .coveragerc | ||
| .flake8 | ||
| .flake8_packages | ||
| .gitignore | ||
| .mailmap | ||
| .travis.yml | ||
| CODE_OF_CONDUCT.md | ||
| conftest.py | ||
| CONTRIBUTING.md | ||
| issue_template.md | ||
| LICENSE | ||
| NOTICE | ||
| pytest.ini | ||
| README.md | ||
Spack
Spack is a multi-platform package manager that builds and installs multiple versions and configurations of software. It works on Linux, macOS, and many supercomputers. Spack is non-destructive: installing a new version of a package does not break existing installations, so many configurations of the same package can coexist.
Spack offers a simple "spec" syntax that allows users to specify versions and configuration options. Package files are written in pure Python, and specs allow package authors to write a single script for many different builds of the same package. With Spack, you can build your software all the ways you want to.
See the Feature Overview for examples and highlights.
To install spack and your first package, make sure you have Python. Then:
$ git clone https://github.com/spack/spack.git
$ cd spack/bin
$ ./spack install libelf
Documentation
Full documentation for Spack is the first place to look.
Try the Spack Tutorial, to learn how to use spack, write packages, or deploy packages for users at your site.
See also:
- Technical paper and slides on Spack's design and implementation.
- Short presentation from the Getting Scientific Software Installed BOF session at Supercomputing 2015.
Get Involved!
Spack is an open source project. Questions, discussion, and contributions are welcome. Contributions can be anything from new packages to bugfixes, or even new core features.
Mailing list
If you are interested in contributing to spack, join the mailing list. We're using Google Groups for this:
Slack channel
Spack has a Slack channel where you can chat about all things Spack:
Sign up here to get an invitation mailed to you.
Contributions
Contributing to Spack is relatively easy. Just send us a
pull request.
When you send your request, make develop the destination branch on the
Spack repository.
Your PR must pass Spack's unit tests and documentation tests, and must be PEP 8 compliant. We enforce these guidelines with Travis CI. To run these tests locally, and for helpful tips on git, see our Contribution Guide.
Spack uses a rough approximation of the
Git Flow
branching model. The develop branch contains the latest
contributions, and master is always tagged and points to the latest
stable release.
Authors
Many thanks go to Spack's contributors.
Spack was created by Todd Gamblin, tgamblin@llnl.gov.
Citing Spack
If you are referencing Spack in a publication, please cite the following paper:
- Todd Gamblin, Matthew P. LeGendre, Michael R. Collette, Gregory L. Lee, Adam Moody, Bronis R. de Supinski, and W. Scott Futral. The Spack Package Manager: Bringing Order to HPC Software Chaos. In Supercomputing 2015 (SC’15), Austin, Texas, November 15-20 2015. LLNL-CONF-669890.
Release
Spack is released under an LGPL license. For more details see the NOTICE and LICENSE files.
LLNL-CODE-647188