Fixes#2306
Any dependency explicitly mentioned in a spec string ended up with the
build and link deptypes unconditionally. This fixes dependency
resolution to ensure that packages which are mentioned in the spec
string have their deptypes determined by the dependency information
in the package.py files. For example if a package has cmake as a build
dependency, and cmake is mentioned as a dependency in the spec string
for the package, then it ends up with just the build deptype.
Packages built targeting a backend may depend on packages like cmake
which can be built against the frontend. With this commit, any build
dependency or child of a build dependency will target the frontend by
default. In compiler concretization when packages copy compilers from
nearby packages, build dependencies use compiler information from
other build dependencies, and link dependencies avoid using compiler
information from build dependencies.
* Use JSON for the database instead of YAML.
- JSON is much faster than YAML *and* can preserve ordered keys.
- 170x+ faster than Python YAML when using unordered dicts
- 55x faster than Python YAML (both using OrderedDicts)
- 8x faster than C YAML (with OrderedDicts)
- JSON is built into Python, unlike C YAML, so doesn't add a dependency.
- Don't need human readability for the package database.
- JSON requires no major changes to the code -- same object model as YAML.
- add to_json, from_json methods to spec.
* Add tests to ensure JSON and YAML don't need to be ordered in DB.
* Write index.json first time it's not found instead of requiring reindex.
* flake8 bug.
* Added some notes about how multiarch detection could be fixed.
* Implemented a preliminary version of the "spack.spec.ArchSpec" class.
* Updated the "spack.spec.Spec" class to use "ArchSpec" instead of "Arch".
* Fixed a number of small bugs in the "spack.spec.ArchSpec" class.
* Fixed the 'Concretizer.concretize_architecture' method so that it uses the new architecture specs.
* Updated the package class to properly use arch specs.
Removed a number of unused architecture functions.
* Fixed up a number of bugs that were causing the regression tests to fail.
Added a couple of additional regression tests related to architecture parsing/specification.
Fixed a few bugs with setting reserved os/target values on "ArchSpec" objects.
Removed a number of unnecessary functions in the "spack.architecture" and "spack.concretize" modules.
* Fixed a few bugs with reading architecture information from specs.
Updated the tests to use a uniform architecture to improve reliability.
Fixed a few minor style issues.
* Adapted the compiler component of Spack to use arch specs.
* Implemented more test cases for the extended architecture spec features.
Improved error detection for multiple arch components in a spec.
* Fix for backwards compatibility with v0.8 and prior
* Changed os to unknown for compatibility specs
* Use `spack09` instead of `spackcompat` for the platform of old specs.
Different compilers have different flags for PIC (position-independent
code). This patch provides a common ground to accessing it inside specs.
See discussions in #508 and #2373. This patch does not address the issue
of mixed compilers as mentioned in #508.
Package provides a 'list_url' attribute which may be searched to find
download links. #1822 created a slowdown for all tests by always
searching this URL. This reenables dynamic search only in cases where
all other fetchers fail. This also only enables dynamic search when
'mirror_only' is set to false.
The option -s now causes file and line number information to be printed
along with any invocation of msg, info, etc...
This will greatly ease debugging.
- Seems like older git versions won't be able to clone an LFS repo.
- Reverting to an external link for the slides to avoid storing an 8MB
file in the repo and to avoid using git LFS.
* Allow compiler wrapper to modify environment
This adds the ability to set environment variables in the compiler
wrappers. These are specified as part of the compilers.yaml config.
The user may also specify RPATHs in compilers.yaml that should be
added.
* Minor doc tweak
* Waste less space when fetching cached archives, simplify fetch messages.
- Just symlink cached files into the stage instead of copying them with curl.
- Don't copy linked files back into the cache when done fetching.
* Fixes for review.
* more updates
* last update
Not the desired eventual behavior, but cflag subset matching is not currently working for anonymous specs and this provides a temporary solution by restricting the feature until it is fixed.
* Wordsmithing/minor-edits to module tutorial
A small set of wordsmithing, spell checking and minor edits to the fancy
new modules tutorial!
* Fix typo (sneaky z key...)
* Fix "S:" and "manual<" typos
Some packages which include resources fetched from source control
repositories terminated package installs because they failed to
archive; specifically, this included all SCM resources which identify
a specific state of the repo - for example a revision in svn or a
tag/revision in git. This is because the resource stage creation
logic did not choose an appropriate archive name for these kinds of
resources.
* module files tutorial : first complete draft
- first complete draft for module files tutorial
- minor corrections to module file reference
* module file tutorial : first batch of corrections
- module avail spelled out fully
- typos from @adamjstewart
- rewording of a few sentences
* module file tutorial : first batch of corrections
- emphasized lines in yaml files
* module file tutorial : fixes according to @citibeth and @adamjstewart reviews
- used long format for command options
- reworded unclear sentence on tokes
- reworked table in reference manual to make it clearer
* module file tutorial : implemented corrections collected on site from @schlyfts
* module file tutorial : removed comment (@hartzell suggestion)
* Add options for hashes, tree depth, and YAML to `spack spec`.
- Can now display hashes with `spack spec`, like `spack find`.
- Removed the old "ids" argument to `spack spec` (which
printed numerical values)b
- Can output YAML spec from `spack spec` with `-y`
- Can control depth of DAG traversal with --cover=[nodes|edges|paths]
- Can print install status (installed, missing, not installed) with -I
* Don't use YAML aliases in specs.
- Force Spack's YAML dumper to ignore aliases.
- aliases cause non-canonical YAML to be used in DAG hash, and result in
redundant hashes.
- add a test to ensure this behavior stays
* spack install: forward sys.stdin to child processes fixes#2140
- [ ] redirection process is spawned in __enter__ instead of __init__
- [ ] sys.stdin is forwarded to child processes
* log: wrapped __init__ definition
1) list gfortran as a fc and f77 compiler that can work with clang
2) allow compatible gfortran to ./spack compiler find with clang by matching version numbers
This is based on the discussions in
https://github.com/LLNL/spack/issues/237https://github.com/dealii/dealii/wiki/deal.II-in-Spack#mixing-gcc-and-clang-on-osx
This is not a long term solution but something to get us through the next months until the compiler
infrastructure is reworked to allow mixing and matching for C/C++ and Fortran compilers
Funded-by: IDEAS
Project: IDEAS/xSDK
Time: 1.5 hours
- Detailed debug information is now handed back to the parent process
from builds, for *any* type of exception.
- previously this only worked for Spack ProcessErrors, but now it works
for any type of error raised in a child.
- Spack will print an error message and source code context for build
errors by default.
- It will print a stack trace when using `spack -d`, even when the error
occurred in the child process.
- Ported old run-flake8-tests qa script to `spack flake8` command.
- New command does not modify files in the source tree
- Copies files to a temp stage modifies them there, and runs tests.
- Updated docs and `run-flake8-tests` script to call `spack flake8`.
- generalized and fixed to work with any key in YAML file
- simplified schema writing, as well
- add more unit tests for the config system
- Rename test/yaml.py to test/spack_yaml.py
- Add test/yaml.pyc to ignored pyc files.
- Added a schema for config.yaml
- Moved install tree configuration to config.yaml
- Moved etc/spack/install.yaml to etc/spack/defaults/config.yaml
- renamed install_area to "store", to use a term in common with guix/nix.
- in `config.yaml` file, it's called the `install_tree` to be more
intuitive to users.
- `install_tree` might've worked in the code, but `install_tree` is
already a global function in the spack namespace, from
llnl.util.filesystem.
Merge #2030 added a cyclic dependency between the Cray platform needing
to read a `targets.yaml` config file and `config.py` needing to get the
platform names.
This commit removes the cyclic dependency in favor of the more general
config scheme. It also removes the now functionless `targets.yaml`
config file. This breaks 'frontend' targets on the Cray platform but
all architecture targets, including the frontend, that are provided by
CrayPE are added to the Platform anyway so users can be explicit about
the architecture targeted by the Cray compiler wrappers:
```
spack spec libelf arch=cray-CNL-frontend
```
becomes
```
spack spec libelf arch=cray-CNL-mc8 # on an XK7 or
spack spec libelf arch=cray-CNL-sandybridge # on an older XC30, etc..
```
The only way the 'frontend' target can be defined after this commit is
through target environment variables.
* module file support: major rework of docs
* module file support: fixed issues found by @adamjstewart
- list or enumeration should not be indented
- use console instead of bash or csh in things that are not scripts
- other typos
* module file support: fixed other issues found by @adamjstewart
- tables should not be indented
- substitute lines with pyobject to import an entire function
- get help output running commands
- typos
* module file support: fixes according to review comments
- @citibeth moved `spack module loads` after `spack load`
- @glennpj tried to clarify installation table + changes to language
- @tgamblin Removed top level section and moved the whole thing into the reference manual
* module file support: moved directive before spack module loads
External packages do not have an spec.yaml file so don't check for it.
Without this change any time a package depends on an external package
when the new package is installed you will get the error
Install prefix exists but contains no spec.yaml
This problem has also haunted me since I started using Spack since PETSc
depends on Python and I used an external python but fortunately it
was relatively easy to debug once I could reproduce it at will.
Funded-by: IDEAS
Project: IDEAS/xSDK
Time: 1 hour
This replaces a custom token-based substitution format with calls to
Spec.format in modules.py
This also resolves a couple issues:
- LmodModules set configuration globally instead of in its initializer
which meant test-specific configuration was not applied
- Added support for setting hash_length=0 for LmodModules. This only
affects the module filename and not the directory names for the
hierarchy tokens in the path. This includes an additional unit test.
* add filter_system_paths()
* filter system paths in set_build_environment_variables()
* two function: lib/inc + bin
* reverse order in bin
* fix order
* minor
* improvements of the code
* more cleanup
* alternative solution for filter_bins
* fiddle base alalazo's approach
* minor
* minor
* Fixed a bug causing config-specified compiler flags to be ignored.
Updated the compiler config so all flags are in a separate section.
* Updated the documentation for the `compilers.yaml` file spec.
* Implemented basic testing for the 'flags' section of compiler config.
* Fixed a few minor problems with the manual compiler config documentation.
* Add new version property to handle joined version numbers
* Add unit test for new joined property
* Add documentation on version.up_to() and version.joined
* Removes the extra argument from Package.do_install while maintaining the changes in behavior pulled in #1603
* install : removed -i and -d shorthands (breaks backward compatibility)
* Change ':' to ','
* build_environment: allow compilers to set up an environment
* clang: mock up a toolchain directory for xcode
Some projects ignore CC and CXX flags and instead use xcode to find the
toolchain. Clang on Apple should set up the environment properly.
Arguably, every compiler could do this on Apple, but let's see how this
works out just for AppleClang for now.
The Documentation directory is ~1.7G and the excluded platforms add up
to about 7G. Ignoring swift saves another 500M. The resulting Xcode.app
copy is in the 2G range.
* compiler: set member variables early
This is required so that later methods can query things such as the
version of the compiler.
* compiler: support finding the real path of the compiler
On Apple, the /usr/bin compilers are actually wrapping tools themselves
which query xcrun for the currently selected Xcode installation. Pierce
this veil and get the real, full path the to underlying compilers
instead.
* icu4c: install with rpath
On macOS, icu installs with a library ID of the library name. Enabling
rpath makes its ID its full installed path which lets Qt5 link against
it successfully.
* qt: no -no-gtkstyle flag on Qt5 on macOS
* Rebase and merging using platform.system
Rebasing and merging using platform.system instead of uname -a.
* Add missing import platform statement
* Remove subprocess import
Remove ununsed import subprocess to make changes flak8 compliant
- Locks now use fcntl range locks on a single file.
How it works for prefixes:
- Each lock is a byte range lock on the nth byte of a file.
- The lock file is ``spack.installed_db.prefix_lock`` -- the DB tells us
what to call it and it lives alongside the install DB. n is the
sys.maxsize-bit prefix of the DAG hash.
For stages, we take the sha1 of the stage name and use that to select a
byte to lock.
With 100 concurrent builds, the likelihood of a false lock collision is
~5.36e-16, so this scheme should retain more than sufficient paralellism
(with no chance of false negatives), and get us reader-writer lock
semantics with a single file, so no need to clean up lots of lock files.
- Closing and re-opening to upgrade to write will lose all existing read
locks on this process.
- If we didn't allow ranges, sleeping until no reads would work.
- With ranges, we may never be able to take some legal write locks
without invalidating all reads. e.g., if a write lock has distinct
range from all reads, it should just work, but we'd have to close the
file, reopen, and re-take reads.
- It's easier to just check whether the file is writable in the first
place and open for writing from the start.
- Lock now only opens files read-only if we *can't* write them.
A use case where the previous approach was failing is :
- more than one spack process running on compute nodes
- stage directory is a link to fast LOCAL storage
In this case the processes may try to unlink something that is "dead" for them, but actually used by other processes on storage they cannot see.
* This fixes a bug in concretization. Before the recent change to the
algorithm, the intent was that the @develop version, although
"greater" than numberic versions, is never preferred BY DEFAULT over
numeric versions.
To test this... suppose you have a package with no `preferred=True` in
it, and nothing in `packages.yaml`, but with a `develop` version. For
the sake of this example, I've hacked my `python/package.py` to work
this way.
Without bugfix (WRONG: user should never get develop by default):
```
python@develop%clang@7.3.0-apple~tk~ucs4 arch=darwin-elcapitan-x86_64
...
```
With bugfix (RIGHT: largest numeric version selected):
```
python@3.5.2%clang@7.3.0-apple~tk~ucs4 arch=darwin-elcapitan-x86_64
...
```
* Documented version selection in concretization algo.
* Fix typos
* flake8
* Fix various documentation bugs
* Keep long option names, but don't include in Command Index
* Use long option name
* Explicitly designate sections to be listed in the Command Index
* Consistent menu bar titles
Input/output/error streams not directed to None or 'str' were not being closed
because `close()` method was being called on the argument value (a string type)
instead of the open file descriptor object.
* Fix bug in handling of precedence of preferred=True vs. versions given in packages.yaml (#1556)
* Standardized comparison of versions: numeric versions are always greater than non-numeric versions; and non-numeric versions are sorted alphabetically.
This is
a) simple
b) ensures that non-numeric versions (such as 'develop') in package.py are not chosen ahead of numeric versions, when nothing is specified in packages.yaml
Fixes Issue #1557
* Removed debugging output
* Fix variable shadowing bug
* Ensure develop < numeric version.
* Bug fix.
* Passes all unit tests in versions.py
* flake8 fixes
* flake8 fixes
* Changed type test to be more correct.
See http://stackoverflow.com/questions/8203336/difference-between-int-and-numbers-integral-in-python
This closes#1757 which provides an example of a url scheme where the
version appears after the extension. Instead of extending the parsing
logic to handle this case, this commit allows the user to specify
their extension type. This helps Spack choose the appropriate
decompressor and mirror archive filename.
* Transferred pending changes from efischer/develop
* 1. Rewrite of "Getting Started": everything you need to set up Spack, even on old/ornery systems. This is not a reference manual section; items covered here are covered more systematically elsewhere in the manual. Some sections were moved here from elsewhere.
2. Beginning to write three methods of application developer support. Two methods were moved from elsewhere.
* Edits...
* Moved sections in preparation for additional text to be added from old efischer/docs branch.
* Moved 2 more sections.
* Avoid accid
* Applied proofreading edits from @adamjstewart
* Fixed non-standard section characters.
* Moved section on profiling to the developer's guide.
* Still working on Spack workflows...
* Finished draft of packaging_guide.rst
* Renamed sample projects.
* Updates to docstrings
* Added documentation to resolve#638 (content taken from #846)
* Added section on resolving inconsistent run dependencies. Addresses #645
* Showed how to build Python extensions only compatible with certain versions of Python.
* Added examples of getting the right behavior from depends_on(). See #1035
* Added section on Intel compilers and their GCC masquerading feature. Addresses #638, #1687.
* Fixed formatting
* Added fixes to filesystem views. Added a caveats section to ``spack setup``.
* Updated section on Intel compiler configuration because compiler flags currently do not work (see #1687)
* Defined trusted downloads, and updated text based on them. (See #1696)
* Added workflow to deal with buggy upstream software. See #1683
* Added proper separation between Spack Docs vs. Reference Manual
* Renamed spack_workflows to workflows. Resolves a conflict with the .gitignore file.
* Removed repeated section.
* Created new "Vendor Specific Compiler Configuration" section and organized existing Intel section into it. Added new PGI and NAG sections; but they need to be expanded / rewritten based on the existing text plus research through Spack issues on GitHub.
* Fixed text on `spack load --dependencies` to conform to reality. See #1662
* Added patching as option for upstream bugfixes.
* Added section on using licensed compilers.
* Added section on non-downloadable tarballs.
* Wrote sections on NAG and PGI. Arranged compilers in alphabetical order.
* Fix indent.
* Fixed typos.
* Clarified dependency types.
* Applied edits from Adam J. Stewart. Spellchecked workflows and getting_started.
* Removed spurious header
* Fixed Sphinx errors
* Fixed erroneous symbol in docstring.
* Fix many typos and formatting problems.
* Spacing changes
* Added section on fixing Git problems. See #1779
* Fixed signature of install() method.
* Addressed system packages in greater detail. See #1794#1795
* Fixed typos
* Fixed quotes
* Duplicate section on Spack profiling removed from configuration.rst. It had earlier been moved to developer_guide.rst, where it fits better.
* Minor edits
- Tweak supported platform language.
- Various small changes to the new getting started guide.
* Fixed bug with quotes.
- Fixed up dependency management so that:
- build deps go in PATH and -I
- link deps go in -L args
- only *immediate* link deps are RPATH'd
The latter reduces the number of libraries that need to be added to
DT_NEEDED / LC_RPATH. This removes redundant RPATHs to transitive
dependencies.
* Fixes#1720: spack reindex fails with invalid ref count.
- Database graph wasn't being built properly; dependencies were set up
incorrectly in the nodes that ended up in the graph on reindex.
- Reworked _add to increment ref count properly and to always build
bottom-up to make the logic simpler to understand.
* Add checks to ensure DB is a valid merkle tree.
- Distinguish between static (package) and dynamic (spec) graphs.
- static graphs ignore conditions and multiple instances (hashes) and
plot raw dependencies among packages.
- dynamic graphs include information from particular specs (instances of
packages) and can have multiple instances with hashes.
- Allow graphing all packages in the install DB.
- useful for debugging.
* Perform shebang fix for all files
* Fix sbang for perl scripts
Otherwise perl would look at the #! line and call sbang again, resulting
in an infinite loop.
* Turned <provider>_libs into an iterable
Modifications :
- added class LibraryList + unit tests
- added convenience functions `find_libraries` and `dedupe`
- modifed non Intel blas/lapack providers
- modified packages using blas_shared_libs and similar functions
* atlas : added pthread variant
* intel packages : added lapack_libs and blas_libs
* find_library_path : removed unused function
* PR review : fixed last issues
* LibraryList : added test on __add__ return type
* LibraryList : added __radd__ fixed unit tests
fix : failing unit tests due to missing `self`
* cp2k and dependecies : fixed blas-lapack related statements in package.py
* Fixed bug propagating --dirty flag to sub-installs.
* Fix syntax error
* Allow --dirty flag to be set with SPACK_DIRTY env var.
* Added dirty flag to `spack diy` and `spack setup`, as is currently in `spack install`
* flake8
Includes :
- treatment of a generic hierarchy (i.e. lapack + mpi + compiler)
- possibility to specify which compilers are to be considered Core
- correct treatment of the 'family' directive
- unit tests for most new features
