* Added installation date and time to the database
Information on the date and time of installation of a spec is recorded
into the database. The information is retained on reindexing.
* Expose the possibility to query for installation date
The DB can now be queried for specs that have been installed in a given
time window. This query possibility is exposed to command line via two
new options of the `find` command.
* Extended docstring for Database._add
* Use timestamps since the epoch instead of formatted date in the DB
* Allow 'pretty date' formats from command line
* Substituted kwargs with explicit arguments
* Simplified regex for pretty date strings. Added unit tests.
This updates architecture concretization to
* Search for the nearest parent in the DAG for architecture information
rather than defaulting to the root of the DAG
* Propagate architecture settings transitively, such that if for
example the target is set at the root of the dag it will set the
same target on indirect dependencies (assuming no intermediate
dependency specifies a separate target). Previously this occurred
in general but under some conditions did not, for example if an
intermediate dependency specified some subset of architecture
properties.
* Create mirror for system with different compilers
Spack concretizes the spec provided by the user in
"spack mirror create" to ensure downloading the right
dependencies. Under normal circumstances concretization
requires that the chosen compiler exists on the system,
but this is not required when creating download mirrors
for other systems, so this requirement is removed in that
case.
* Add test for disabling compiler existence check
* Update compiler existence checking logic
* improve test for disabling compiler existence check
* make py-setuptools a run-time-only dep for py-basemap and patch python package to only apply setuptools flag for build deps
* py-qtconsole does not require setuptools
* This allows Spack to work with MD5 hashes on machines with openssl in FIPS mode.
* We are still using MD5 for validation in many places, and a later PR will replace all uses of MD5 with SHA256.
* This is a quick fix until that happens.
- transitive dependencies were not being handled correctly
- restructure code to do recursion and mark visited packages properly
- add `-V` option to *not* expand virtuals in spack dependencies
This re-adds the option to create and install unsigned tarballs, now
with the -u option (--unsigned) rather than the -y option.
This also changes the "keys" command, replacing the -y/--yes-to-all
option with the -t/--trust option (which has the same effect but is
more-clearly named).
Fixes#7130
shutil.move expects a source path like "/x/y/" to be a directory and
fails if "/x/y" is a symlink. This invokes realpath on the source
path to avoid the issue.
Fixes#7356
In some cases OperatingSystem (e.g. LinuxDistro) was getting
instantiated with a version that contains dashes. This breaks because
the concretizer later converts this value to a string and re-parses
it, and the '-' character is used to separate architecture components.
This adds a guard in the initializer to convert '-' to '_'.
Fixes#7237Fixes#6404Fixes#6418Fixes#6369
Identify when binary relocation fails to remove all instances of the
source prefix (and report an error to the user unless they specify
-a to allow the old root to appear). This check occurs at two stages:
during "bincache create" all instances of the root are replaced with
a special placeholder string ("@@@@..."), and a failure occurs if the
root is detected at this point; when the binary package is extracted
there is a second check. This addresses #7237 and #6418.
This is intended to be compatible with previously-created binary
packages.
This also adds:
* Better error messages for "spack install --use-cache" (#6404)
* Faster relocation on Mac OS (using a single call to
install_name_tool for all files rather than a call for each file)
* Clean up when "buildcache create" fails (addresses #6369)
* Explicit error message when the spack instance extracting the binary
package uses a different install layout than the spack instance that
created the binary package (since this is currently not supported)
* Remove the option to create unsigned binary packages with -y
This updates Cray.setup_platform_environment to use cray-specific
pkgconfig paths so that all providers of 'pkgconfig' have access
to them (previously the 'pkg-config' provider had this but the
'pkgconf' provider did not).
* [SPACK/spec.py] When a query through ForwardQueryToPackage returns
'None', treat that as query failure and raise RuntimeError with
suitable message. This overrides the current behavior to raise an
AttributeError which is now triggered only when no suitable query
property is found and there is no default handler.
* [spack/spec.py] Fix style.
* [SPACK/spec.py] In case of query failure, i.e. property returning
'None', raise AttributeError instead of RuntimeError in order to
pass the unit test. Also, small update in the logic distinguishing
query failure and lack of relevant property/attribute handling.
This updates the fix_darwin_install_name function to use the Spack
Executable object to run install_name_tool, which ensures that
process output is formatted as a 'str' for python2 and python3.
Originally fix_darwin_install_name was invoking subprocess.Popen
directly.
Fixes#5189
When working with non-normalized paths containing ".." on some
file systems, Spack was found to encounter a permission error when
writing to the path. This normalizes a path written by the
intel-parallel-studio package and also normalizes all paths
written by the license install hook (for all packages) to avoid
this issue for intel-parallel-studio.
Following the discussion with Todd and Adam, find has been modified to
accept glob expressions. This should not affect performance as every
glob implementation I inspected has 3 cases (no wildcard, wildcard but
no directories involved, wildcard and directories involved) and uses
fnmatch underneath.
Mixins have been changed to do by default a non-recursive search (but
a recursive search can still be triggered using the recursive keyword).
Following a comment from Todd, the search path for the files listed in
`filter_compiler_wrappers` can now be narrowed. Anyhow, the function
implementation still makes use of `find`, the rationale being that we
have already seen packages that install artifacts in e.g. architecture
dependent folders. The possibility to have a relative search path might
be a good compromise between the previous approach and the one suggested
in the review.
Also: 'ignore_absent' and 'backup' keyword arguments can be optionally
forwarded to `filter_file`.
Following comments from Todd:
- the call to tty.debug has been moved deeper, to log the filtering of each file
- the shadowing on the name "kwargs" is avoided
Implemented a declarative syntax for the additional behavior that can
get attached to classes. Implemented a function to filter compiler
wrappers that uses the mechanism above.
- command reference now includes usage for all Spack commands as output
by `spack help`. Each command usage links to any related section in
the docs.
- added `spack commands` command which can list command names,
subcommands, and generate RST docs for commands.
- added `llnl.util.argparsewriter`, which analyzes an argparse parser and
calls hooks for description, usage, options, and subcommands
- Shorten Spack command usage for short options. Short options are now
shown as [-abc] instead of as [-a] [-b] [-c]
- fix bug that mixed long and short options for top-level `spack help`
- Add proper help for `spack buildcache` subcommands
- Reorganize the help categories of Spack commands so that buildcache is
in packaging and diy and setup are now in build.
- previously commands with this argument showed a long list of choices
that were platform specific.
- use a better metavar: {defaults,system,site,user}[/PLATFORM]
Fixes#7159
When activating extensions in external views, the --ignore-conflicts
option was being ignored. In this particular issue the conflict was
for the duplicate __init__ file for multiple python packages in the
same namespace, but in general any conflict for extensions would
cause an error whether or not --ignore-conflicts was set.
This also renames the 'force' option of do_activate to
'with_dependencies' and updates views to call do_activate with this
set to False (since it traverses the dependency dag anyway). This
isn't strictly required, it just avoids redundant calls.
This reorganizes most sections and rewords a significant portion of
the content (including all introductions) but keeps all the examples.
* Remove section 'What happens at subscript time' from tutorial:
it is too detailed for a tutorial
* Move the 'Extra query parameters' and 'Attach attributes to other
packages' sections into a separate grouping 'Other packaging topics'
* move the 'Set variables at build time yourself' section after
'Set environment variables in dependents' section since the latter
is more motivating
* start the 'set environment variables at build-time for yourself'
section with qt as an example
* renamed section 'specs build interface' to 'retrieving library
information' and updated section introduction
* renamed section 'a motivating example' to 'accessing library
dependencies'; split out the material which deals with implementing
.libs for netlib-lapack into a separate section called 'providing
libraries to dependents'. consolidated in material from the section
'single package providing multiple virtual specs' since
netlib-lapack is an example of this (this removes the material
about intel-parallel studio)
* Allow dashes in command names and fix command name handling
- Command should allow dashes in their names like the reest of spack,
e.g. `spack log-parse`
- It might be too late for `spack build-cache` (since it is already
called `spack buildcache`), but we should try a bit to avoid
inconsistencies in naming conventions
- The code was inconsistent about where commands should be called by
their python module name (e.g. `log_parse`) and where the actual
command name should be used (e.g. `log-parse`).
- This made it hard to make a command with a dash in the name, and it
made `SpackCommand` fail to recognize commands with dashes.
- The code now uses the user-facing name with dashes for function
parameters, then converts that the module name when needed.
* Improve performance of log parsing
- A number of regular expressions from ctest_log_parser have really poor
performance, most due to untethered expressions with * or + (i.e., they
don't start with ^, so the repetition has to be checked for every
position in the string with Python's backtracking regex implementation)
- I can't verify that CTest's regexes work with an added ^, so I don't
really want to touch them. I tried adding this and found that it
caused some tests to break.
- Instead of using only "efficient" regular expressions, Added a
prefilter() class that allows the parser to quickly check a
precondition before evaluating any of the expensive regexes.
- Preconditions do things like check whether the string contains "error"
or "warning" (linear time things) before evaluating regexes that would
require them. It's sad that Python doesn't use Thompson string
matching (see https://swtch.com/~rsc/regexp/regexp1.html)
- Even with Python's slow implementation, this makes the parser ~200x
faster on the input we tried it on.
* Add `spack log-parse` command and improve the display of parsed logs
- Add better coloring and line wrapping to the log parse output. This
makes nasty build output look better with the line numbers.
- `spack log-parse` allows the log parsing logic used at the end of
builds to be executed on arbitrary files, which is handy even outside
of spack.
- Also provides a profile option -- we can profile arbitrary files and
show which regular expressions in the magic CTest parser take the most
time.
* Parallelize log parsing
- Log parsing now uses multiple threads for long logs
- Lines from logs are divided into chnks and farmed out to <ncpus>
- Add -j option to `spack log-parse`
* Marking database tests as slow
* Marking url command tests as slow
* Marking every test that uses database as slow
* Marking tests that import files as slow
* Marking gpg tests as slow
* Marking all versions and one list tests as slow
* Added more markers to unit tests + cli option to skip slow tests
Following a discussion with Axel, the generic 'slowtest' marker has been
split into 'db', 'network' and 'maybeslow'. A brief description of the
meaning of each marker has been added to pytest.ini.
A command line option to run only fast tests has been added to
'spack test'
* Don't use classes to group tests together
Reverted grouping tests under a class, as required in the review
* Minor style changes
This attempts to address one of the complaints at #5996 (comment):
> Repo currently caches package instances by Spec, and those Package instances have a Spec.
> This is unnecessary and causes confusion. I think I thought that we'd need to cache instances
> after loading package classes, but really just caching the classes is fine.
With this update, Repo's package cache is removed and Specs cache the package reference themselves. One consequence is that Specs which compare as equal will store separate instances of a Package class (not doing this creates issues for #4595 (comment)).
There were several references to Spec.package that could be replaced with Spec.package_class without any additional modifications. There are still a couple remaining references to Spec.package in Spec that would require adding functionality before replacing (e.g. calling Package.provides and Package.installed).
Note this makes it difficult to mock fetchers for tests which invokes code that reconstructs specs. test_packaging was one example of this where the updates caused a failure (in that case the error was avoided by not making an unnecessary call).
Details:
* Replace instances of spec.package with spec.package_class where a class method is being called
* Remove instances of Repo.get where Spec.package_class can be used in its place
* remove Repo.get caching instances of Package class for specs
* remove redundant check (which is also incorrect now that each spec stores its own copy of its package)
* avoid creating mirror with specs because it copies specs and those copies dont refer to the mocked fetcher (and it is also not required for the test)
* remove checks that are no longer necessary since repo doesn't cache specs
