* First draft for SC17 build systems portion
Added tutorial_buildsystems.rst file as well as example files under
the tutorial/ directory.
* Remove floating `
* Add requested changes, and examples of subclasses
Added in the requested changes to the documentation. Also added in
information about the subclasses and the defaults that they provide.
Also fixed some phrasing issues, formatting and punctuation.
* Flake8 fixes and new files for classes
Made flake8 fixes to pass tests and also added files to demonstrate code
in the classes.
* Minor edits
Edits in formatting and made some sentence changes
* Flake8 fixes
More flake8 fixes
* Flake8 fix
* Change section order on tutorial and minor edits
Placed the section at the appropriate section for the tutorial and then
added some minor edits that were requested.
* Add requested changes and more details
Added more details to Cmake, Makefile and Python Packages.
* Fixed formatting and minor edits
* Fix doc build error
* Allow types and 'any' in variant definitions.
- Previously variant values had to be a tuple or a callable predicate.
- This allows 'any' as shorthand for `lambda x: True` and type objects
as shorthand for "any value of this type".
- Makes variant definitions more readable, keeps lambdas out of
packages for common cases.
* Update packaging tutorial
* Fix bad file reference in packaging tutorial
* First draft of the advanced packaging tutorial
* advanced packaging tutorial: improved phrasing
Thanks Denis and Hartzell!
* Fixed typos + reworded a couple of sentences
* Reworked module file tutorial section
First draft for the SC17 update. This includes:
- adding an introduction on module files + Spack's module
generation blueprints
- adding a set-up section and provide a docker image for easy set-up
- updating all the relevant snippets
- extending a bit some of the concepts that were already touched
* Added reference to #5582 + committed Dockerfiles
Also fixed a couple of typos spotted by Denis.
* module file tutorial: added section on template customization
* module file tutorial: fixed minor typos + rephrased a sentence
* module file tutorial: made explicit that Docker image comes with software
* module file tutorial: improved phrasing and layout.
Thanks Hartzell!
* module file tutorial: added vim and nano to editors
* module file tutorial: fixed typo
* Fixed typos
Thanks Adam!
* module file tutorial: updated Dockerfile + minor changes in introduction
Fixes#6154
For compiler options which set values using the syntax "-flag value"
(with a space between the flag and the flag's value) the flag and
value were treated as separate and reordered. This updates Spack's
logic to treat the flag and value as a single unit, even if there is
a space between them. It assumes that all flags are prefixed with
"-" and that all flag values are not.
* Skip rewriting files that are links (this also avoids issues with parsing
the output of the 'file' command for symlinks)
* Fail rather than warn for several gpg signing issues (e.g. if there is no
key available to sign with)
* Installation with 'buildcache install' only retrieves link and run
dependencies (this matches 'buildcache create' which only creates tarballs
of link/run dependencies)
* Don't rewrite RPATHs for a binary if the binary cached package was created
with relative RPATHs
* Refactor relocate_binary to operate on multiple binaries (given as a
collection of paths). This was likewise done for relocate_text and
make_binary_relative
- This isn't one of those autogenerated SVGs from a drawing program!
- This is a completely re-traced, minimalist SVG file with clearly
delineated pieces so that your favorite renderer can draw a Spack logo
at whatever resolution you want.
- Included versions with text, as well.
Fixes#6126#3183 removed the print_help function from the "modules" module.
This adds it back so that if a user invokes 'spack load foo' without
having sourced an environment setup script, they will be prompted
to do so. This also improves the placeholder message to tell the
user to invoke 'spack' as shell function rather than as an executable.
Part of the resource staging process is to place downloaded/expanded
files in the root stage. This was not happening when a resource stage
was restaged.
Fixes#5778.
Spack uses 'gcc -dumpversion' to determine the full version of gcc.
'gcc -dumpversion' no longer gives the full version on gcc 7.2.0.
'gcc -dumpfullversion' is required instead. This PR detects when
'gcc -dumpversion' gives a truncated version of '7' and in that
case retrieves the full version with 'gcc -dumpfullversion'
The name of the debug log written by the cc compiler wrapper was given
by Spec.short_spec, which includes the architecture. Somewhere along
the line Spec.format started adding spaces around the architecture
property so the filename started including spaces; the cc wrapper
script appears to ignore this, so files like spack-cc-bzip2-....in.log
(which record the wrapped compiler invocations) were not being
generated. This uses a different format string from the spec to
generate the wrapper log file names (which does not include spaces).
* when a user-provided spec refers to an already-installed package, packages with patches applied were causing validation errors based on the recorded variants in the package's class
* avoid checks on all reserved variants (not just 'patches')
* basic functionality to install spec from a binary cache when it's available; this spiders each cache for each package and could likely be more efficient by caching the results of the first check
* add spec to db after installing from binary cache
* cache (in memory) spec listings retrieved from binary caches
* print a warning vs. failing when no mirrors are configured to retrieve pre-built Spack packages
* make automatic retrieval of pre-built spack packages from mirrors optional
* no code was using the links stored in the dictionary returned by get_specs, so this simplifies the logic to return only a set of specs
* print package prefix after installing from binary cache
* provide more information to user on install progress
This updates the logic for Package.extends so that if the spec
associated with the package is not concrete, it will report true if
the package *could extend* the given spec; generally speaking a
package could extend a spec as long as none of the details associated
with its extendee spec conflict with the given spec. When the spec
associated with the package is concrete, this function will only
report whether the package *does extend* the given spec. When both
the specs are concrete, the semantics are the same as before.
* When creating a tar of a package for a build cache, symlinks are
preserved (the corresponding path in the newly-created tarfile will
be a symlink rather than a copy of the file)
* Dont add external packages to a build cache
* When installing from binary cache, don't create install prefix until
verification is complete
* Fixes#5754
Previously when RepoPath.repo_for_pkg was invoked with a string,
it did not check if the string included a namespace. Any
namespace-qualified package provided as a string would not be found
(at which point the behavior was to return the highest-precedence
repository).
* handle nested namespaces for packages specified as strings in repo_for_pkg
* add preliminary repository tests
* add test which replicates #5754
* refactor repo tests with fixtures
* define repo_path equivalent at test-level scope for repo tests
* add tests for unknown namespace/package
* rename fixture function (no longer prefixed with 'test_')
Internally we work against a branch named 'llnl/develop', which
mirrors the public repository's `develop` branch.
It's useful to be able to run flake8 on our changes, using
`llnl/develop` as the base branch instead of `develop`.
Internally the flake8 subcommand generates the list of changes files
using a hardcoded range of `develop...`.
This makes the base of that range a command line option, with a
default of `develop`.
That lets us do this:
```
spack flake8 --base llnl/develop
```
which uses a range of `llnl/develop...`.
'spack install' can now reinstall a spec even if it has dependents, via
the --overwrite option. This option moves the current installation in a
temporary directory. If the reinstallation is successful the temporary
is removed, otherwise a rollback is performed.
- new E741 flake8 checks disallow 'l', 'O', and 'I' as variable names
- rework parts of the code that use this, to be compliant
- we could add exceptions for this, but we're trying to mostly keep up
with PEP8 and we already have more than a few exceptions.
- When you don't use wildcards, flake8 will find places where you used an
undefined name.
- This commit has all the bugfixes resulting from this static check.
There are now separate flake8 configs for core vs. packages:
- core has a smaller set of flake8 exceptions
- packages allow `from spack import *` and module globals
- Allows core to take advantage of static checking for undefined names
- Allows packages to keep using Spack tricks like `from spack import *`
and dependencies setting globals for dependents.
* Update Getting Started docs to clarify that full Xcode suite is required for qt
* Better error message when only the command-line tools are installed
I'm tracking down a problem with the perl package that's been
generating this error:
```
OSError: OSError: [Errno 2] No such file or directory: '/blah/blah/blah/lib/5.24.1/x86_64-linux/Config.pm~'
```
The real problem is upstream, but it's being masked by an exception
raised in `filter_file`s finally block.
In my case, `backup` is `False`.
The backup is created around line 127, the `re.sub()` calls
fails (working on that), the `except` block fires and moves the backup
file back, then the finally block tries to remove the non-existent
backup file.
This change just avoids trying to remove the non-existent file.
- The shell script uses arrays and hence only works on sophisticated shells and not the default `sh`. For clarity the shebang `#!/bin/bash` has been used instead.
- This fakes out GitFetchStrategy to try code paths for different git
versions.
- This allows us to test code paths for old versions using a newer git
version.
- Tests use a session-scoped mock stage directory so as not to interfere
with the real install.
- Every test is forced to clean up after itself with an additional check.
We now automatically assert that no new files have been added to
`spack.stage_path` during each test.
- This means that tests that fail installs now need to clean up their
stages, but in all other cases the check is useful.
- Be explicit about stage creation during the install process.
- This avoids accidental creation of stages
- e.g., during `spack install --fake`, stages were erroneously recreated
after being destroyed
- This prevents the main spack install from being clusttered by
invocations of `spack test`.
- This uses a session-scoped stage fixture to ensure tests don't
interfere.
- Spack's core package interface was previously overly stateful, in that
calling methods like `do_stage()` could change your working directory.
- This removes Stage's `chdir` and `chdir_to_source` methods and replaces
their usages with `with working_dir(stage.path)` and `with
working_dir(stage.source_path)`, respectively. These ensure the
original working directory is preserved.
- This not only makes the API more usable, it makes the tests more
deterministic, as previously a test could leave the current working
directory in a bad state and cause subsequent tests to fail
mysteriously.
- Assertion would search for root through all possible paths.
- It's also really slow.
- This isn't needed anymore. We're pretty good at ensuring single-rooted
DAGs, and this assertion has never been thrown.
- This shaves another 6 seconds off r-rminer concretization
- This reduces concretization time for r-rminer from over 1 minute to
only 16 seconds.
- OrderedDict ends up taking a *lot* of time to compare larger specs.
- An OrderedDict isn't actually needed here. It's actually not possible
to find duplicates, and we end up sorting the contents of the
OrderedDict anyway.
- This is an optimization to the way traverse_edges iterates over
successors.
- Previous version called dependencies_dict(), which involved a lot of
redundant work (creating dicts and calling caonical_deptype)
- Spack ends up constructing compilers frequently from YAML data.
- This caches the result of parsing the compiler config
- The logic in compilers/__init__.py could use a bigger cleanup, but this
makes concretization much faster for now.
- on macOS, this also ensures that xcrun is called only twice, as opposed
to every time a new compiler object is constructed.
This adds a workflow section on how to use spack on Docker.
It provides an example on the best-practices I collected over the
last months and circumvents the common pitfalls I tapped in.
Works with MPI, CUDA, Modules, execution as root, etc.
Background: Developed initially for PIConGPU.
* Make --trusted default when running spack gpg list
Currently running `spack gpg list` with no arguments returns nothing. You must supply either the `--trusted` or the `--signing` options. The idea here is to return some initial data to the user when the command is run. The alternative is to return an error, telling the user to select one of the two options.
* Add an extra test case for the empty list command
Fixes the issue with code coverage
This isn't a rework of the concretizer but it speeds things up a LOT.
The main culprits were:
1. Variant code, `provider_index`, and `concretize.py` were calling
`spec.package` when they could use `spec.package_class`
- `spec.package` looks up a package instance by `Spec`, which requires a
(fast-ish but not that fast) DAG compare.
- `spec.package_class` just looks up the package's class by name, and you
should use this when all you need is metadata (most of the time).
- not really clear that the current way packages are looked up is
necessary -- we can consider refactoring that in the future.
2. `Repository.repo_for_pkg` parses a `str` argument into a `Spec` when
called with one, via `@_autospec`, but this is not needed.
- Add some faster code to handle strings directly and avoid parsing
This speeds up concretization 3-9x in my limited tests. Still not super
fast but much more bearable:
Before:
- `spack spec xsdk` took 33.6s
- `spack spec dealii` took 1m39s
After:
- `spack spec xsdk` takes 6.8s
- `spack spec dealii` takes 10.8s
* Do not call setup_package for fake installs
- setup package could fail if ``setup_dependent_environment`` or other
routines expected to use executables from dependencies
- xpetsc and boost try to get python config variables in
`setup_dependent_package`; this would cause them not to be
fake-installable
* Remove vestigial deptype_query argument to Spec.traverse()
- The `deptype_query` argument isn't used anymore -- it's only passed
around and causes confusion when calling traverse.
- Get rid of it and just keep the `deptypes` argument
* Don't print redundant messages when installing dependencies
- `do_install()` was originally depth-first recursive, and printed "<pkg>
already installed in ..." multiple times for packages as recursive
calls encountered them.
- For much cleaner output, use spec.traverse(order='post') to install
dependencies instead
* Add package for aspell and ass't dictionaries
Add a package definition for aspell.
Add a handful of dictionaries to convince myself that the support for
a bunch of dictionaries works.
* Flake8 cleanup
* Use six's version of urlparse
`urlparse` is not python3 friendly. This works around it (stolen from
`.../cmd/md5.py`).
* Fix incorrect trimming regexp
* Clean up dictionary build
- more parsimonious use of `which` (`make()` has already been made)
- use `sh` instead of `bash`
* Use a helper method to generate info for variants
I figured out my issues with static methods. I *think* that it this
is pythonic.
* Convert aspell to an extendable package
Convert aspell to be extendable and rework the dictionaries to be
extensions.
As it stands, there's a great deal of cut and paste in the
dictionaries, I'll abstract that out next.
The {de,}activate methods copy a great deal of code out of
package.py. Perhaps there's a better way....
* Create AspellDictPackage and use it for the dictionaries
Reduce the repeated code, pull it into a base class.
I'm confused about why 'from spack import *' wasn't more useful in the
base class.
* Oops, -de & -es should be AspellDictPackages too
* Typo: pakcage -> package
* Address some commentary
* Update copyright dates, 2016->2017
* spec and spec.package.spec can refer to different objects in the
database. When these two instances of spec differ in terms of
the value of the 'concrete' property, Spec._mark_concrete can
fail when checking Spec.package.installed (which requires
package.spec to be concrete). This skips the check for
spec.package.installed when _mark_concrete is called with
'True' (in other words, when the database is marking all specs
as being concrete).
* add test to confirm this fixes#5293
