- Allows hashes to be specified after other parts of the spec
- Does not allow other parts of the spec to be specified after the hash
- The hash must either end input or be followed by another separate spec
- The next spec cannot be an anonymous spec (it must start with a package name or a hash)
See #2769 (after it was merged) for further discussion of this interface addition. That discussion resulted in these requirements:
```
python # 1 spec
/abc123 # 1 spec
python /abc123 # 1 spec
/456789 # 1 spec
python /abc123 /456789 # 2 specs
python /456789 /abc123 # 2 specs
/abc123 /456789 # 2 specs
/456789 /abc123 # 2 specs
/456789 /abc123 python # 3 specs
```
assuming `abc123` and `456789` are both hashes of different python specs.
* Add support for IBM threaded compilers, xl*_r
Added new compiler class, xl_r; added default flags to the compilers.yaml file.
* Add cppflags to the set of default flags to be added to the compilers stanza in compiler.yaml.
These flags are optional. Only defined flags will be listed in the compilers.yaml file.
* Fix scripting warnings revealed by flake8.
Updated __init__.py and xl_r.py to conform with flake8 rules.
* Add justification to the definition of the XL default compiler flags.
* PackageMeta: `run_before` is an alias of `precondition`, `run_after` an alias of `sanity_check`
* PackageMeta: removed `precondition` and `sanity_check`
* PackageMeta: decorators are now free-standing
* package: modified/added docstrings. Fixed the semantics of `on_package_attributes`.
* package: added unit test assertion as side effects of install
* build_systems: factored build-time test running into base class
* r: updated decorators in package.py
* docs: updated decorator names
* documentation: reworked packaging guide to add build-system phases
* documentation: improvements to AutotoolsPackage autodocs
* build_systems: updated autodocs
* run-tests: added a few information on how to run tests fixes#2606 fixes#2605
* documentation: fixed items brought up by @davydden
* typos in docs
* consistent use of 'build system' (i.e. removed 'build-system' from docs)
* added a note on possible default implementations for build-time tests
* documentation: fixed items brought up by @citibeth
* added note to explain the difference between build system and language used in a package
* capitalized bullet items
* added link to API docs
* documentation: fixed multiple cross-references after rebase
* documentation: fixed minor issues raised by @tgamblin
* documentation: added entry in table for the `PythonPackage` class
* docs: fixed issues brought up by @citybeth in the second review
Previously, fix_darwin_install_name would only handle dependencies that have no path set, and it ignore dependencies that have the build directory as path baked in. Catch this, and replace it by the install directory.
- Add a PythonPackage class with build system support.
- Support build phases in PythonPackage
- Add a custom sanity check for PythonPackages
- Get rid of nolink dependencies in python packages
- Update spack create to use new PythonPackage class
- Port most of Python packages to new PythonPackage class
- Conducted a massive install and activate of Python packages.
- Fixed bugs introduced by install and activate.
- Update API docs on PythonPackage
* Initial changes to spack create command
* Get 'spack create <url>' working again
* Simplify call to BuildSystemGuesser
* More verbose output of spack create
* Remove duplicated code from spack create and spack checksum
* Add better documentation to spack create docstrings
* Fix pluralization bug
* Flake8
* Update documentation on spack create and deprecate spack edit --force
* Make it more obvious when we are renaming a package
* Further deprecate spack edit --force
* Fix unit tests
* Rename default template to generic template
* Don't add automake/autoconf deps to Autotools packages
* Remove changes to default $EDITOR
* Completely remove all traces of spack edit --force
* Remove grammar changes to make the PR easier to review
* Fixed parser to eliminate need for escape quotes. TODO: Fix double call to shlex, fix spaces in spec __str__
* Fixed double shlex
* cleanup
* rebased on develop
* Fixed parsing for multiple specs; broken since #360
* Revoked elimination of the `-` sigil in the syntax, and added it back into tests
* flake8
* more flake8
* Cleaned up dead code and added comments to parsing code
* bugfix for spaces in arguments; new bug found in testing
* Added unit tests for kv pairs in parsing/lexing
* Even more flake8
* ... yet another flake8
* Allow multiple specs in install
* unfathomable levels of flake8
* Updated documentation to match parser fix
* Added customization for make targets in 'build' and 'install' phases for CMakePackage
* Use rst in build system docs so that Sphinx generates nice API docs
* Allow AutotoolsPackages to be built in a different directory
* Flake8
* Fix missing import
* Allow configure to be located in different directory
* Update espressopp to use build targets
* Flake8
* Sphinx fix, lists must be a new paragraph
* Back out change that allowed a configure script in a different directory than build_directory
* Add missing deps, build in parallel
* Missing space for rst list
* Removing the nobuild, nolink, and alldeps dependency types in favor of being explicit.
* This will help with maintenance going forward, as adding more dependency types won't affect existing declared dependencies in weird ways.
* default deptype is still `('build', 'link')`
* Rename packages
* Upcasing depends_on() in packages.
* Downcased extends('r')
* Fixed erroneously changed URL that had slipped through.
* Fixed typo
* Fixed link from documentation into package source code.
* Fixed another doc problem.
* Changed underscores to dashes in package names.
* Added test to enforce lowercase, no-underscore naming convention.
* Fix r-xgboost
* Downcase more instances of 'R' in package auto-creation.
* Fix test.
* Converted unit test packages to use dashes not underscores
* Downcase `r` in the docs.
* Update module_file_support.rst
Fix r->R for class R.
* Porting: substitute nose with ytest
This huge commit substitutes nose with pytest as a testing system. Things done here:
* deleted external/nose as it is no longer used
* moved mock resources in their own directory 'test/mock/'
* ported two tests (cmd/find, build_system) to pytest native syntax as an example
* build_environment, log: used monkeypatch instead of try/catch
* moved global mocking of fetch_cache to an auto-used fixture
* moved global mocking from test/__init__.py to conftest.py
* made `spack test` a wrapper around pytest
* run-unit-tests: avoid running python 2.6 tests under coverage to speed them up
* use `pytest --cov` instead of coverage run to cut down testing time
* mock/packages_test: moved mock yaml configuration to files instead of leaving it in the code as string literals
* concretize.py: ported tests to native pytest, reverted multiprocessing in pytest.ini as it was creating the wrong report for coveralls
* conftest.py, fixtures: added docstrings
* concretize_preferences.py: uses fixtures instead of subclassing MockPackagesTest
* directory_layout.py: uses fixtures instead of subclassing MockPackagesTest
* install.py: uses fixtures instead of subclassing MockPackagesTest
* optional_deps.py: uses fixtures instead of subclassing MockPackagesTest
optional_deps.py: uses fixtures instead of subclassing MockPackagesTest
* packages.py: uses fixtures instead of subclassing MockPackagesTest
* provider_index.py: uses fixtures instead of subclassing MockPackagesTest
* spec_yaml.py: uses fixtures instead of subclassing MockPackagesTest
* multimethod.py: uses fixtures instead of subclassing MockPackagesTest
* install.py: now uses mock_archive_url
* git_fetch.py: uses fixtures instead of subclassing MockPackagesTest
* hg_fetch.py: uses fixtures instead of subclassing MockPackagesTest
* svn_fetch.py, mirror.py: uses fixtures instead of subclassing MockPackagesTest
repo.py: deleted
* test_compiler_cmd.py: uses fixtures instead of subclassing MockPackagesTest
* cmd/module.py, cmd/uninstall.py: uses fixtures instead of subclassing MockDatabase
* database.py: uses fixtures instead of subclassing MockDatabase, removed mock/database
* pytest: uncluttering fixture implementations
* database: changing the scope to 'module'
* config.py: uses fixtures instead of subclassing MockPackagesTest
* spec_dag.py, spec_semantics.py: uses fixtures instead of subclassing MockPackagesTest
* stage.py: uses fixtures instead of subclassing MockPackagesTest. Removed mock directory
* pytest: added docstrings to all the fixtures
* pytest: final cleanup
* build_system_guess.py: fixed naming and docstrings as suggested by @scheibelp
* spec_syntax.py: added expected failure on parsing multiple specs closes#1976
* Add pytest and pytest-cov to Spack externals.
* Make `spack flake8` ignore externals.
* run-unit-tests runs spack test and not pytest.
* Remove all the special stuff for `spack test`
- Remove `conftest.py` magic and all the special case stuff in `bin/spack`
- Spack commands can optionally take unknown arguments, if they want to
handle them.
- `spack test` is now a command like the others.
- `spack test` now just delegates its arguments to `pytest`, but it does
it by receiving unknown arguments and NOT taking an explicit
help argument.
* Fix error in fixtures.
* Improve `spack test` command a bit.
- Now supports an approximation of the old simple interface
- Also supports full pytest options if you want them.
* Use external coverage instead of pytest-cov
* Make coverage use parallel-mode.
* change __init__.py docs to include pytest
* inheritance of directives: using meta-classes to inject attributes coming from directives into packages + lazy directives
* _dep_types -> dependency_types
* using a meta-class to inject directives into packages
* directives are lazy
fixes#2466
* directives.py: allows for multiple inheritance. Added blank lines as suggested by @tgamblin
* directives.py: added a test for simple inheritance of directives
* Minor improvement requested by @tgamblin
CMakePackage: importing names from spack.directives
directives: wrap __new__ to respect pep8
* Refactoring requested by @tgamblin
directives: removed global variables in favor of class variables. Simplified the interface for directives (they return a callable on a package or a list of them).
* Ensure that every package has a license
Also fixes URLs with http://http:// doubled.
This is a continuation of #2656.
* Add license to every file in Spack
* Make sure Todd is the author of all packages
* Fix flake8 tests
* Don't license external Sphinx docs
* Don't display licenses in tutorial example packages
Also fixes typos and converts command-line examples
from tcsh to bash, which is more common
That's because in set_build_environment_variables()
the funciton filter_system_paths() is actually applied to
package prefixes, and not to prefix/lib or prefix/include.
The primary goal of #2292 was to use the frontend compiler to make
build dependencies like cmake on HPC platforms. It turns out that
while this works in some cases, it did not handle cases where a
package was a link dependency of the root and of a build dependency
(and could produce incorrect concretizations which would not build).
* Better output for disambiguate_specs()
* Fix wrong exception name.
* Fix satsifies(): concrete specs require matching by hash.
- Fixes uninstall by hash and other places where we need to match a
specific spec.
- Fix an error in provider_index (satisfies() call was backwards)
- Fix an error in satisfies_dependencies(): checks were too shallow.
* Fix default args in Spec.tree()
* Move installed_dependents() to DB to avoid unknown package error.
* Make `spack find` and `sapck.store.db.query()` faster for hashes.
* Add a test to ensure satisfies() respects concrete Specs' hashes.
* Customization for make targets in build and test phases for AutotoolsPackage
* Updated Blitz++ to use customized make build and test targets
* Removed flake8 error
* Removed make test customization, added make install customization, need to figure out issues with multiple make targets
* Changed build_targets and install_targets to normal attributes
* MakefilePackage: changed build_args and install_args for consistency with #2464
openblas: derives from MakefilePackage
* MakefilePackage: changed default edit behavior
* Better cxx11/14 flags for GNU/Clang/Intel
- GCC 4.8 only supports -std=c++1y for C++14
- Use CMake's rules for AppleClang to set cxx11 and cxx14 flags based on
Apple Xcode/LLVM version
- Use CMake's rules for Intel to add support for cxx14 flags based on
Intel version.
* Add cxx17_flag property
Implement property in compiler for c++17 as per those for c++11/14.
Add concrete support for GNU/Clang:
- Return -std=c++1z for GCC 5 and above per GCC documentation
- Return -std=c++1z for Clang 3.5 and above per Clang documentation
- Return -std=c++1z for Apple LLVM 6.1 and above per CMake's rules
* Update `spack setup` and `spack graph` to be consistent with c557e765 and 9347f869. Fixes#2316.
* Added another "fix" necessary to make `spack setup` work.
* Added another "fix" necessary to make `spack setup` work. (reverted from commit 7f0d3ecb38c97ec00491d7cd66b4266e3018b1ca)
* Add documentation for repositories and namespaces.
* Update and extend repository documentation per review.
- Also add `-N` argument for `spack spec`
The advanced [Uninstalling Packages](spack.readthedocs.io/en/latest/tutorial_sc16_spack_basics.html#uninstalling-packages) via hash has a couple missing `.. code-block:: console` directives ;)
I have no idea what branch to direct this to though...
* clang: do xcode mockup iff requested by a package
* add a note
* add pkg to setup_custom_environment() and decide whether or not to use mockup XCode there based on the package
* Make targets an attribute of compilers, similar to OS. Allows users to use `spack compiler find` to add compilers for the same platform/os but different targets when spack is installed on a file system mounted on machines with different targets.
* Changed get_compilers() method to treat old compilers with no target as target='any'
* flake8 changes
* Address change requests for comments/code clarity
Fixes#2306
Any dependency explicitly mentioned in a spec string ended up with the
build and link deptypes unconditionally. This fixes dependency
resolution to ensure that packages which are mentioned in the spec
string have their deptypes determined by the dependency information
in the package.py files. For example if a package has cmake as a build
dependency, and cmake is mentioned as a dependency in the spec string
for the package, then it ends up with just the build deptype.
Packages built targeting a backend may depend on packages like cmake
which can be built against the frontend. With this commit, any build
dependency or child of a build dependency will target the frontend by
default. In compiler concretization when packages copy compilers from
nearby packages, build dependencies use compiler information from
other build dependencies, and link dependencies avoid using compiler
information from build dependencies.
* Use JSON for the database instead of YAML.
- JSON is much faster than YAML *and* can preserve ordered keys.
- 170x+ faster than Python YAML when using unordered dicts
- 55x faster than Python YAML (both using OrderedDicts)
- 8x faster than C YAML (with OrderedDicts)
- JSON is built into Python, unlike C YAML, so doesn't add a dependency.
- Don't need human readability for the package database.
- JSON requires no major changes to the code -- same object model as YAML.
- add to_json, from_json methods to spec.
* Add tests to ensure JSON and YAML don't need to be ordered in DB.
* Write index.json first time it's not found instead of requiring reindex.
* flake8 bug.
* Added some notes about how multiarch detection could be fixed.
* Implemented a preliminary version of the "spack.spec.ArchSpec" class.
* Updated the "spack.spec.Spec" class to use "ArchSpec" instead of "Arch".
* Fixed a number of small bugs in the "spack.spec.ArchSpec" class.
* Fixed the 'Concretizer.concretize_architecture' method so that it uses the new architecture specs.
* Updated the package class to properly use arch specs.
Removed a number of unused architecture functions.
* Fixed up a number of bugs that were causing the regression tests to fail.
Added a couple of additional regression tests related to architecture parsing/specification.
Fixed a few bugs with setting reserved os/target values on "ArchSpec" objects.
Removed a number of unnecessary functions in the "spack.architecture" and "spack.concretize" modules.
* Fixed a few bugs with reading architecture information from specs.
Updated the tests to use a uniform architecture to improve reliability.
Fixed a few minor style issues.
* Adapted the compiler component of Spack to use arch specs.
* Implemented more test cases for the extended architecture spec features.
Improved error detection for multiple arch components in a spec.
* Fix for backwards compatibility with v0.8 and prior
* Changed os to unknown for compatibility specs
* Use `spack09` instead of `spackcompat` for the platform of old specs.
Different compilers have different flags for PIC (position-independent
code). This patch provides a common ground to accessing it inside specs.
See discussions in #508 and #2373. This patch does not address the issue
of mixed compilers as mentioned in #508.
Package provides a 'list_url' attribute which may be searched to find
download links. #1822 created a slowdown for all tests by always
searching this URL. This reenables dynamic search only in cases where
all other fetchers fail. This also only enables dynamic search when
'mirror_only' is set to false.
The option -s now causes file and line number information to be printed
along with any invocation of msg, info, etc...
This will greatly ease debugging.
- Seems like older git versions won't be able to clone an LFS repo.
- Reverting to an external link for the slides to avoid storing an 8MB
file in the repo and to avoid using git LFS.
* Allow compiler wrapper to modify environment
This adds the ability to set environment variables in the compiler
wrappers. These are specified as part of the compilers.yaml config.
The user may also specify RPATHs in compilers.yaml that should be
added.
* Minor doc tweak
* Waste less space when fetching cached archives, simplify fetch messages.
- Just symlink cached files into the stage instead of copying them with curl.
- Don't copy linked files back into the cache when done fetching.
* Fixes for review.
* more updates
* last update
Not the desired eventual behavior, but cflag subset matching is not currently working for anonymous specs and this provides a temporary solution by restricting the feature until it is fixed.
* Wordsmithing/minor-edits to module tutorial
A small set of wordsmithing, spell checking and minor edits to the fancy
new modules tutorial!
* Fix typo (sneaky z key...)
* Fix "S:" and "manual<" typos
Some packages which include resources fetched from source control
repositories terminated package installs because they failed to
archive; specifically, this included all SCM resources which identify
a specific state of the repo - for example a revision in svn or a
tag/revision in git. This is because the resource stage creation
logic did not choose an appropriate archive name for these kinds of
resources.
* module files tutorial : first complete draft
- first complete draft for module files tutorial
- minor corrections to module file reference
* module file tutorial : first batch of corrections
- module avail spelled out fully
- typos from @adamjstewart
- rewording of a few sentences
* module file tutorial : first batch of corrections
- emphasized lines in yaml files
* module file tutorial : fixes according to @citibeth and @adamjstewart reviews
- used long format for command options
- reworded unclear sentence on tokes
- reworked table in reference manual to make it clearer
* module file tutorial : implemented corrections collected on site from @schlyfts
* module file tutorial : removed comment (@hartzell suggestion)
* Add options for hashes, tree depth, and YAML to `spack spec`.
- Can now display hashes with `spack spec`, like `spack find`.
- Removed the old "ids" argument to `spack spec` (which
printed numerical values)b
- Can output YAML spec from `spack spec` with `-y`
- Can control depth of DAG traversal with --cover=[nodes|edges|paths]
- Can print install status (installed, missing, not installed) with -I
* Don't use YAML aliases in specs.
- Force Spack's YAML dumper to ignore aliases.
- aliases cause non-canonical YAML to be used in DAG hash, and result in
redundant hashes.
- add a test to ensure this behavior stays
* spack install: forward sys.stdin to child processes fixes#2140
- [ ] redirection process is spawned in __enter__ instead of __init__
- [ ] sys.stdin is forwarded to child processes
* log: wrapped __init__ definition
1) list gfortran as a fc and f77 compiler that can work with clang
2) allow compatible gfortran to ./spack compiler find with clang by matching version numbers
This is based on the discussions in
https://github.com/LLNL/spack/issues/237https://github.com/dealii/dealii/wiki/deal.II-in-Spack#mixing-gcc-and-clang-on-osx
This is not a long term solution but something to get us through the next months until the compiler
infrastructure is reworked to allow mixing and matching for C/C++ and Fortran compilers
Funded-by: IDEAS
Project: IDEAS/xSDK
Time: 1.5 hours
- Detailed debug information is now handed back to the parent process
from builds, for *any* type of exception.
- previously this only worked for Spack ProcessErrors, but now it works
for any type of error raised in a child.
- Spack will print an error message and source code context for build
errors by default.
- It will print a stack trace when using `spack -d`, even when the error
occurred in the child process.
- Ported old run-flake8-tests qa script to `spack flake8` command.
- New command does not modify files in the source tree
- Copies files to a temp stage modifies them there, and runs tests.
- Updated docs and `run-flake8-tests` script to call `spack flake8`.
- generalized and fixed to work with any key in YAML file
- simplified schema writing, as well
- add more unit tests for the config system
- Rename test/yaml.py to test/spack_yaml.py
- Add test/yaml.pyc to ignored pyc files.
- Added a schema for config.yaml
- Moved install tree configuration to config.yaml
- Moved etc/spack/install.yaml to etc/spack/defaults/config.yaml
- renamed install_area to "store", to use a term in common with guix/nix.
- in `config.yaml` file, it's called the `install_tree` to be more
intuitive to users.
- `install_tree` might've worked in the code, but `install_tree` is
already a global function in the spack namespace, from
llnl.util.filesystem.
Merge #2030 added a cyclic dependency between the Cray platform needing
to read a `targets.yaml` config file and `config.py` needing to get the
platform names.
This commit removes the cyclic dependency in favor of the more general
config scheme. It also removes the now functionless `targets.yaml`
config file. This breaks 'frontend' targets on the Cray platform but
all architecture targets, including the frontend, that are provided by
CrayPE are added to the Platform anyway so users can be explicit about
the architecture targeted by the Cray compiler wrappers:
```
spack spec libelf arch=cray-CNL-frontend
```
becomes
```
spack spec libelf arch=cray-CNL-mc8 # on an XK7 or
spack spec libelf arch=cray-CNL-sandybridge # on an older XC30, etc..
```
The only way the 'frontend' target can be defined after this commit is
through target environment variables.
* module file support: major rework of docs
* module file support: fixed issues found by @adamjstewart
- list or enumeration should not be indented
- use console instead of bash or csh in things that are not scripts
- other typos
* module file support: fixed other issues found by @adamjstewart
- tables should not be indented
- substitute lines with pyobject to import an entire function
- get help output running commands
- typos
* module file support: fixes according to review comments
- @citibeth moved `spack module loads` after `spack load`
- @glennpj tried to clarify installation table + changes to language
- @tgamblin Removed top level section and moved the whole thing into the reference manual
* module file support: moved directive before spack module loads
External packages do not have an spec.yaml file so don't check for it.
Without this change any time a package depends on an external package
when the new package is installed you will get the error
Install prefix exists but contains no spec.yaml
This problem has also haunted me since I started using Spack since PETSc
depends on Python and I used an external python but fortunately it
was relatively easy to debug once I could reproduce it at will.
Funded-by: IDEAS
Project: IDEAS/xSDK
Time: 1 hour
This replaces a custom token-based substitution format with calls to
Spec.format in modules.py
This also resolves a couple issues:
- LmodModules set configuration globally instead of in its initializer
which meant test-specific configuration was not applied
- Added support for setting hash_length=0 for LmodModules. This only
affects the module filename and not the directory names for the
hierarchy tokens in the path. This includes an additional unit test.
* add filter_system_paths()
* filter system paths in set_build_environment_variables()
* two function: lib/inc + bin
* reverse order in bin
* fix order
* minor
* improvements of the code
* more cleanup
* alternative solution for filter_bins
* fiddle base alalazo's approach
* minor
* minor
* Fixed a bug causing config-specified compiler flags to be ignored.
Updated the compiler config so all flags are in a separate section.
* Updated the documentation for the `compilers.yaml` file spec.
* Implemented basic testing for the 'flags' section of compiler config.
* Fixed a few minor problems with the manual compiler config documentation.
* Add new version property to handle joined version numbers
* Add unit test for new joined property
* Add documentation on version.up_to() and version.joined
* Removes the extra argument from Package.do_install while maintaining the changes in behavior pulled in #1603
* install : removed -i and -d shorthands (breaks backward compatibility)
* Change ':' to ','
* build_environment: allow compilers to set up an environment
* clang: mock up a toolchain directory for xcode
Some projects ignore CC and CXX flags and instead use xcode to find the
toolchain. Clang on Apple should set up the environment properly.
Arguably, every compiler could do this on Apple, but let's see how this
works out just for AppleClang for now.
The Documentation directory is ~1.7G and the excluded platforms add up
to about 7G. Ignoring swift saves another 500M. The resulting Xcode.app
copy is in the 2G range.
* compiler: set member variables early
This is required so that later methods can query things such as the
version of the compiler.
* compiler: support finding the real path of the compiler
On Apple, the /usr/bin compilers are actually wrapping tools themselves
which query xcrun for the currently selected Xcode installation. Pierce
this veil and get the real, full path the to underlying compilers
instead.
* icu4c: install with rpath
On macOS, icu installs with a library ID of the library name. Enabling
rpath makes its ID its full installed path which lets Qt5 link against
it successfully.
* qt: no -no-gtkstyle flag on Qt5 on macOS
* Rebase and merging using platform.system
Rebasing and merging using platform.system instead of uname -a.
* Add missing import platform statement
* Remove subprocess import
Remove ununsed import subprocess to make changes flak8 compliant
- Locks now use fcntl range locks on a single file.
How it works for prefixes:
- Each lock is a byte range lock on the nth byte of a file.
- The lock file is ``spack.installed_db.prefix_lock`` -- the DB tells us
what to call it and it lives alongside the install DB. n is the
sys.maxsize-bit prefix of the DAG hash.
For stages, we take the sha1 of the stage name and use that to select a
byte to lock.
With 100 concurrent builds, the likelihood of a false lock collision is
~5.36e-16, so this scheme should retain more than sufficient paralellism
(with no chance of false negatives), and get us reader-writer lock
semantics with a single file, so no need to clean up lots of lock files.
- Closing and re-opening to upgrade to write will lose all existing read
locks on this process.
- If we didn't allow ranges, sleeping until no reads would work.
- With ranges, we may never be able to take some legal write locks
without invalidating all reads. e.g., if a write lock has distinct
range from all reads, it should just work, but we'd have to close the
file, reopen, and re-take reads.
- It's easier to just check whether the file is writable in the first
place and open for writing from the start.
- Lock now only opens files read-only if we *can't* write them.
A use case where the previous approach was failing is :
- more than one spack process running on compute nodes
- stage directory is a link to fast LOCAL storage
In this case the processes may try to unlink something that is "dead" for them, but actually used by other processes on storage they cannot see.
* This fixes a bug in concretization. Before the recent change to the
algorithm, the intent was that the @develop version, although
"greater" than numberic versions, is never preferred BY DEFAULT over
numeric versions.
To test this... suppose you have a package with no `preferred=True` in
it, and nothing in `packages.yaml`, but with a `develop` version. For
the sake of this example, I've hacked my `python/package.py` to work
this way.
Without bugfix (WRONG: user should never get develop by default):
```
python@develop%clang@7.3.0-apple~tk~ucs4 arch=darwin-elcapitan-x86_64
...
```
With bugfix (RIGHT: largest numeric version selected):
```
python@3.5.2%clang@7.3.0-apple~tk~ucs4 arch=darwin-elcapitan-x86_64
...
```
* Documented version selection in concretization algo.
* Fix typos
* flake8
* Fix various documentation bugs
* Keep long option names, but don't include in Command Index
* Use long option name
* Explicitly designate sections to be listed in the Command Index
* Consistent menu bar titles
Input/output/error streams not directed to None or 'str' were not being closed
because `close()` method was being called on the argument value (a string type)
instead of the open file descriptor object.
* Fix bug in handling of precedence of preferred=True vs. versions given in packages.yaml (#1556)
* Standardized comparison of versions: numeric versions are always greater than non-numeric versions; and non-numeric versions are sorted alphabetically.
This is
a) simple
b) ensures that non-numeric versions (such as 'develop') in package.py are not chosen ahead of numeric versions, when nothing is specified in packages.yaml
Fixes Issue #1557
* Removed debugging output
* Fix variable shadowing bug
* Ensure develop < numeric version.
* Bug fix.
* Passes all unit tests in versions.py
* flake8 fixes
* flake8 fixes
* Changed type test to be more correct.
See http://stackoverflow.com/questions/8203336/difference-between-int-and-numbers-integral-in-python
This closes#1757 which provides an example of a url scheme where the
version appears after the extension. Instead of extending the parsing
logic to handle this case, this commit allows the user to specify
their extension type. This helps Spack choose the appropriate
decompressor and mirror archive filename.
* Transferred pending changes from efischer/develop
* 1. Rewrite of "Getting Started": everything you need to set up Spack, even on old/ornery systems. This is not a reference manual section; items covered here are covered more systematically elsewhere in the manual. Some sections were moved here from elsewhere.
2. Beginning to write three methods of application developer support. Two methods were moved from elsewhere.
* Edits...
* Moved sections in preparation for additional text to be added from old efischer/docs branch.
* Moved 2 more sections.
* Avoid accid
* Applied proofreading edits from @adamjstewart
* Fixed non-standard section characters.
* Moved section on profiling to the developer's guide.
* Still working on Spack workflows...
* Finished draft of packaging_guide.rst
* Renamed sample projects.
* Updates to docstrings
* Added documentation to resolve#638 (content taken from #846)
* Added section on resolving inconsistent run dependencies. Addresses #645
* Showed how to build Python extensions only compatible with certain versions of Python.
* Added examples of getting the right behavior from depends_on(). See #1035
* Added section on Intel compilers and their GCC masquerading feature. Addresses #638, #1687.
* Fixed formatting
* Added fixes to filesystem views. Added a caveats section to ``spack setup``.
* Updated section on Intel compiler configuration because compiler flags currently do not work (see #1687)
* Defined trusted downloads, and updated text based on them. (See #1696)
* Added workflow to deal with buggy upstream software. See #1683
* Added proper separation between Spack Docs vs. Reference Manual
* Renamed spack_workflows to workflows. Resolves a conflict with the .gitignore file.
* Removed repeated section.
* Created new "Vendor Specific Compiler Configuration" section and organized existing Intel section into it. Added new PGI and NAG sections; but they need to be expanded / rewritten based on the existing text plus research through Spack issues on GitHub.
* Fixed text on `spack load --dependencies` to conform to reality. See #1662
* Added patching as option for upstream bugfixes.
* Added section on using licensed compilers.
* Added section on non-downloadable tarballs.
* Wrote sections on NAG and PGI. Arranged compilers in alphabetical order.
* Fix indent.
* Fixed typos.
* Clarified dependency types.
* Applied edits from Adam J. Stewart. Spellchecked workflows and getting_started.
* Removed spurious header
* Fixed Sphinx errors
* Fixed erroneous symbol in docstring.
* Fix many typos and formatting problems.
* Spacing changes
* Added section on fixing Git problems. See #1779
* Fixed signature of install() method.
* Addressed system packages in greater detail. See #1794#1795
* Fixed typos
* Fixed quotes
* Duplicate section on Spack profiling removed from configuration.rst. It had earlier been moved to developer_guide.rst, where it fits better.
* Minor edits
- Tweak supported platform language.
- Various small changes to the new getting started guide.
* Fixed bug with quotes.
- Fixed up dependency management so that:
- build deps go in PATH and -I
- link deps go in -L args
- only *immediate* link deps are RPATH'd
The latter reduces the number of libraries that need to be added to
DT_NEEDED / LC_RPATH. This removes redundant RPATHs to transitive
dependencies.
* Fixes#1720: spack reindex fails with invalid ref count.
- Database graph wasn't being built properly; dependencies were set up
incorrectly in the nodes that ended up in the graph on reindex.
- Reworked _add to increment ref count properly and to always build
bottom-up to make the logic simpler to understand.
* Add checks to ensure DB is a valid merkle tree.
- Distinguish between static (package) and dynamic (spec) graphs.
- static graphs ignore conditions and multiple instances (hashes) and
plot raw dependencies among packages.
- dynamic graphs include information from particular specs (instances of
packages) and can have multiple instances with hashes.
- Allow graphing all packages in the install DB.
- useful for debugging.
* Perform shebang fix for all files
* Fix sbang for perl scripts
Otherwise perl would look at the #! line and call sbang again, resulting
in an infinite loop.
* Turned <provider>_libs into an iterable
Modifications :
- added class LibraryList + unit tests
- added convenience functions `find_libraries` and `dedupe`
- modifed non Intel blas/lapack providers
- modified packages using blas_shared_libs and similar functions
* atlas : added pthread variant
* intel packages : added lapack_libs and blas_libs
* find_library_path : removed unused function
* PR review : fixed last issues
* LibraryList : added test on __add__ return type
* LibraryList : added __radd__ fixed unit tests
fix : failing unit tests due to missing `self`
* cp2k and dependecies : fixed blas-lapack related statements in package.py
* Fixed bug propagating --dirty flag to sub-installs.
* Fix syntax error
* Allow --dirty flag to be set with SPACK_DIRTY env var.
* Added dirty flag to `spack diy` and `spack setup`, as is currently in `spack install`
* flake8
Includes :
- treatment of a generic hierarchy (i.e. lapack + mpi + compiler)
- possibility to specify which compilers are to be considered Core
- correct treatment of the 'family' directive
- unit tests for most new features
When re-using previously downloaded tarballs, spack copies from
`var/spack/stage/PACKAGE-VERSION-HASH/downloaded_file` to
`var/spack/cache/PACKAGE/downloaded_file`. This fails if the source is owned by
a different user (`shutil.copy` tries to retain all meta data including file
ownership). Change to a non-meta-data copy function (`shutil.copyfile`).
This closes#1308, where fetching a non-expanded resource from a mirror
will cause an error.
This also ensures that when a URL resource is fetched from a mirror,
that it will be named as though it were retrieved from the original
URL. This is particularly useful for non-expanded resources since it
ensures that the resource name is consistent for the installation
(this is less important for expanded resources because the build takes
place inside the expanded resource).
- Spec._dup() incorrectly copied cached hashes and normal/concrete values
even when dependency structure was not preserved.
- Now these are only copied when *all* dependencies are copied.
- Currently, build dependencies are not currently hashed; we are waiting
to hash these until we have smarter concretization that can reuse more
installed specs. The layout needs to account for this when checking
whethert things are installed.
- Transaction logic had gotten complicated -- DB would not reindex when
corrupt, rather the error would be reported (ugh).
- DB will now print the error and force a rebuild when errors are
detected reading the old databse.
- Spack currently not hashing build deps (to allow more reuse of packages
and less frequent re-installing)
- Fast in-memory hash should still hash *all* deptypes, and installed
specs will only reflect link and run deps.
- We'll revert this when we can concretize more liberally based on what
is already installed.
- can now pass these to Spec.copy() and Spec._dup():
- deps=True
- deps=False
- deps=(list of deptypes)
- Makes it easy to filter out only part of a spec.
Everything in the __all__ list in the spack module is from some other
module, so only do their documentation in their original location. This
also avoids issues like the fact that some directive names shadow spack
core module names.
The header append lines were too long, so I just converted it into a
multi-line string template so it's all one piece now instead of a bunch
of appends.
Major stuff:
- Created a FileCache for managing user cache files in Spack. Currently just
handles virtuals.
- Moved virtual cache from the repository to the home directory so that users do
not need write access to Spack repositories to use them.
- Refactored `Transaction` class in `database.py` -- moved it to
`LockTransaction` in `lock.py` and made it reusable by other classes.
Other additions:
- Added tests for file cache and transactions.
- Added a few more tests for database
- Fixed bug in DB where writes could happen even if exceptions were raised
during a transaction.
- `spack uninstall` now attempts to repair the database when it discovers that a
prefix doesn't exist but a DB record does.
- Spack will check if the index needs updating, and will only parse
all package files if it does.
- Spack tries to parse as few package files as necessary.
- Don't need to list all packages unless we have to.
- Only use the list of all packages for existence checks if we have
generated it for some other purpose.
This PR will catch the error where the url can not be determined from a
VCS URL, such as git. It will print a message to the console and move on
because it should not be a fatal error at this point in the process.
This should fix#1459.
The archive_version variable should only get set if versions will be
checked and that is only if there is a list_url in the package file. For
VCS repos setting the variable triggers an error from web.py as it
parses the default_fetcher object.
This should fix#1422.
Assuming a bash interactive environment will be correctly formed on login, we
should prefer to probe the environment using a shell that reports itself as
`bash` instead of `sh` which may not source files that set the
environment modules in statements like:
```
case "$is" in
bash) test -s /etc/bash.bashrc.local && . /etc/bash.bashrc.local ;;
ksh) test -s /etc/ksh.kshrc.local && . /etc/ksh.kshrc.local ;;
zsh) test -s /etc/zsh.zshrc.local && . /etc/zsh.zshrc.local ;;
ash) test -s /etc/ash.ashrc.local && . /etc/ash.ashrc.local
esac
test -s /etc/sh.shrc.local && . /etc/sh.shrc.local
```
The list of default environment modules is obtained by calling
`module list -lt`
from a subshell with a wiped environment. This allows `/etc/profile` and other
init scripts to be fully sourced which should generally include loading the
default modules.
The list of default modules is then parsed for the first acceptable CPU target,
assumed to be the back_end target.
A platform to generically cover all Cray systems is introduced to avoid having
specific platforms for each of XK (XE, XT), XC, and future systems using CrayPE
and CNL.
The platform searches for 'front_end' and 'back_end' targets, in order, from:
* Environment variables 'SPACK_FRONT_END' and 'SPACK_BACK_END'
* A spack configuration file 'targets.yaml'
* Parsing `/etc/bash.bashrc.local` for first loaded CrayPE CPU target.
If a back_end target is not found through one of these methods, an exception is
raised. Otherwise, the back_end is set as the default target. The shell init
script search is based on recommendations in Crayports case #144359.
No automagic way of determining front_end targets has been found (to date) so if
a front_end is not specified through configuration, it is ignored which should
leave a spack instance in a consistant state.
In addition to `list_url` make sure the package has the `url` attribute
set before attempting to add urls from a list. This is to cover the case
where there may be a `list_url` specified in tandem with a
`url_for_version`.
This commit will make urls from list_url only checked if `list_url` is
set in the package file. This makes more sense as there is no need to
check for those if the attribute is not present. If `url` is present and
`list_url` is not then it would result in the same url. If
`url_for_version` is used then that will not work anyway.
This PR allows archive file retrieval from urls derived from the
`list_url` setting in a package file. This allows for continued
retrieval of checksummed archive files even when they are moved to a new
remote location when a package is updated upstream.
Set up the environment for the Intel compilers and tools. This commit
does the following:
- Unset variables that were incorrect from the auto guess prefix
inspections.
- Add a RemovePath environment_modifications_formats for dotkit.
- Set the module environment variables appropriate for the different
variants.
- Change the component logic so that the '+all' variant works. It was
getting split by letter and leaving COMPONENTS empty.
- Added a variant checking function.
- Added NONRPM_DB_DIR to the silent.cfg so that the product database
goes to the installation directory.
- With the product database in prefix the code to remove the product
database file from the home directory is no longer needed and was
removed.
- Reformat the 'tools' variant description.
There are probably more variables needed for the '+tools' for the
'professional' product version but I do not have access to that.
- Default scope is versioned with spack and can be overridden by site
or user config.
- Default scope provides sensible default concretization preferences
for all of Spack.
- per-platform concretization scope can be added later (to force a
particular MPI on, e.g., Cray systems)
- Consolidated code to read spec.yaml and database index.yaml into
one method (`read_yaml_dep_specs()`) in spec.py.
- Code understands old hash format, tuple format, and dicts for dep
specs, for backward compatibility.
- Spec YAML now uses a dict with keys to represent dep specs (this is
more future-proof).
- Dep specs no longer contain !!py-tuple entries in YAML (only lists
properly YAML-ize)
- bump database version.
deptypes: allow for different queries
For things like Python modules, they are required for the build and
runtime, but rather than adding a new parameter for what to query across
the dependency DAG, just expose the recursive query parameter.
When var/spack/stage is a symlink, the tests fail since realpath is used
on the resulting path, but not the original path, so the string compare
fails. Smarter path testing might be a better option.
This list was not formatted correctly on [the ReadTheDocs site](http://software.llnl.gov/spack/basic_usage.html#specs-dependencies).
I'm not a .rst expert, but I think that it was improperly indented.
The example includes an `arch=...` string but *arch* is not listed in the valid compiler flag specifiers or architecture specifiers. Should it be, or is it considered an "optional variant specifier?
Modifications :
- added EditableMakefile to PackageBase subclasses
- astyle modified as an example
- preliminary hook to stop at a certain phase of install
- add macports to things that are cleaned out of the environment.
- linker incompatibilities cause issues with packages like OpenSSL.
- also clean up NOQA stuff in OpenSSL
- Common cli arguments now are in their own module
- Moved utilities that can be reused by different commands into spack.cmd.__init__.py
- Modifications to `spack module loads`
Removed examples of Cray config files and added them under Spack on
Cray. Any users of Cray can just look at the Spack on Cray section to
get an idea of what's used. If they want more information they can click
links to the sections.
# Conflicts:
# lib/spack/docs/packaging_guide.rst
# lib/spack/spack/package.py
Mostly minor/formatting issues in lib/spack/spack/package.py (the heavyweight merge was already done recently). Only one serious issue: it looks like the feature branch had accidentally deleted the line `spack.hooks.post_install(self)`. This got added back in from develop.
config :
- added `hash_length` under the modules section
EnvModules :
- take into consideration hash_length when constructing `file_name`
- added logic to warn and skip module file writing in case of file name clash
If you want to bootstrap from a fork, the `--remote` option may be used
to select it.
Also limit the branches to 'develop' and 'master' if the remote is
'origin' since those are the actual integration branches used (other
branches on 'origin' are just PR branches).
spack module :
- refresh accepts a constraint
- find and refresh share common cli options
- ask for confirmation before refreshing
- deleting the module file tree is now optional
Updating how to use compilers.yaml on Cray as well as setting up
external packages. Also includes what needs to be set up for usage on
Cray and explains what needs to be done to get Spack properly working.
Also explain the architecture spec and what it does.
Building OpenMPI without fortran is possible, and was already handled.
Raising an install error we effectively renounce at clang+OpenMPI.
build_environment : sets 'CC', 'CXX', and similar variables only if a compiler is found
This does two things:
1. By default `spack find` no longer shows variants. You have to
supply `-v` to get that
2. This improves the `colify` implementation so that it no longer pads
the rightmost column. This avoids the issue where if one spec was
too long in the output, *all* specs would have space padding added
to that width, and it would look like the output of `spack find`
was double spaced. This no longer happens -- the one bad line
wraps around and the other lines are now single-spaced when you use
`-v` with boost.
- Lexer is the same for every Spec parser in spack, so don't build it
every time.
- This improves time to import package.py files a lot, as a Lexer
doesn't have to be constructed for every spc in the packages.
- To concretize dealii:
- Before: ~20 sec
- After: ~6 sec
- `spack spec dealii` now takse seconds rather than 18 minutes.
- Slow hash (`dag_hash`) was accidentally inserted into the fast hash
(`__hash__`).
- Do not put the slow hash in the fast hash.
- Tagging a version to include all that's happened since 0.8.17
- This is in anticipation of merging `features/cflags` and
`features/newarch`, which will break the spec format.
- 1.0 will likely come after those.
Adds a new attribute in the database to track whether a package was
installed explicitly or not, where explicitly is the user running `spack
install <package>` and implicitly is it being installed as a dependency.
It also adds arguments to `spack find` to find these packages such that
it should be possible to query the packages that were installed
implicitly and are not currently depended upon any longer.
This commit includes:
* a new go package that uses gccgo to bootstrap the go toolchain
* env support added to Executable
* a new Go fetch strategy that uses `go get` to fetch a package and all
of its deps
* A platinum searcher package that leverages the new go package and
fetch strategy
This is a complete rework of the lua package, it also allows the
environment modification classes to handle paths that are not separated
by colons, and uses the support for same in TCL modules as well.
The biggest difference is the handling for lua extension packages, which
now have their paths set correctly by the lua parent package, and have
access to both lua and luarocks as installation tools. See the luaposix
package for what should be required for most lua packages after this.
I addressed them by factoring the code better to follow the visitor
pattern. This will allow actions to be easily added in the future.
These may not even be file sytsem views. One could add actions to
generate shell init scripts, JSON DAG-dumpers, GraphViz DOT file
generators, etc (yes, some of these are alread in there - just to give
the idea).
Also added is a top-level test
$ source share/spack/setup-env.sh
$ ./share/spack/examples/test_view.sh
Read the top of that script first.
This allows command line arguments with spaces to be shown. The quoting
madness is because a single quote cannot appear within a single quoted
argument on the command line. To do so, you have to stop the single
quote argument, double quote the single quote, then open the single
quote again:
$ echo 'before'"'"'after'
before'after
Fixes#174
Added R as a build system so that the create template will have the
correct configure line.
Added a regex for version parsing as the R URLs are a little odd.
This version actually pulls the path through the package to deliver it
to each stage on creation when passed through the command. This is
necessary due to the new StageComposite class that makes setting the
path directly on the stage impractical, it also takes the logic out of
package for the most part, which seems like an improvement.
Allow users to use spack to stage a, potentially complex, package into a
given path. This is nice for packages with multiple resources that must
be placed, for example LLVM with all sub-projects.
- ld -r doesn't work with RPATH on OS X.
- for GNU ld, the -r option only means 'relocatable',
and doesn't affect RPATH.
- This adds special handling to omit RPATHs for ld -r on OS X
- removed a lot of old logic that was only still needed for tests.
- Added better unit tests for dependency RPATH, -L, and -I args
- tests now check whether the compiler omits -I args in link mode.
- Consolidate this in one place so that we don't have to do it in every build.
- Will update further once better OS support is committed. Shoudl
really be an attribute of the forthcoming `Platform` class.
- Failed to catch all instances of modify_module when it was renamed
to setup_dependent_package.
- Refactored remaining modify_module calls.
- Also modified Python's setup_dependent_package slightly: only
creates empty site-packages directory for Python extensions now, not
for all dependents.
- Fixed in package.py
- Fixed wrong prototypes in packages that use it.
- Fixed build_environment to set module variables properly
- added hacky fix to ensure spec/package consistency in build processes.
- Need to think about defensive spec copy done by `Repo.get`. May be
time to think about an immutable spec implementation.
- Gave setup_environment and setup_dependent_environment more similar
signatures. They now allows editing the Spack env and the runtime
env for *this* package and dependents, respectively.
- modify_module renamed to setup_dependent_python_module for symmetry
with setup_dependent_environment and to avoid confusion with
environment modules.
- removed need for patching Package objects at runtime.
- adjust packages to reflect these changes.
- `_expand_virtual_packages` now gets a candidate list and will try
all the candidates.
- Good news: If the first virtual in the list conflicts with something else in
the spec, we'll keep trying until we find a good one.
- Bad news: Only looks as far as the next normalize(); can't see
conflicts further ahead than that if they're inevitable
some other virtual expansion.
- Refactor `concretize.py` to keep all the nasty spec graph stitching in
`spec.py`. This is more similar to before externals support.
- `concretize.py` now just returns a list of candidates sorted by
ABI compatibility to `_expand_virtual_packages`, and `spec.py`
handles testing the candidates.
- Refactor the way external paths are handled in `config.py` and `concretize.py`:
- previously, `spec_externals` returned spec/path pairs. Now it
returns specs with `external` set. Makes code in `concretize.py`
more natural.
- Shouldn't call .package from within things like normalize() and
concretize() beacuse spec may be inconsistent.
- Add `.package_class` property so that we can get at package metadata
without constructing a Package with a Spec.
- should be faster than `.package` was, anyway. Use where possible.
not change the behavior for existing packages that subclass from
spack.Package.
If a package subclasses spack.StagedPackage instead of spack.Package,
the install() phase (run inside a forked process) is now separated
into sub-stages:
a) spconfig: Generate a script spconfig.py that will configure the
package (eg by running CMake or ./configure) This is for use if
the user wishes to build externally from Spack. Therefore, the
Spack compiler wrappers are NOT used here. Currently, that
means that RPATH support is up to the user.
b) configure: Configure the project (eg: runs configure, CMake,
etc). This will configure it for use within Spack, using the
Spack wrapper.
c) build: eg: "make"
d) install: eg: "install"
If one chooses to use StagedPackage instead of Package, then one must
implement each of the install sub-stages as a separate method.
StagedPackage.install() then calls each of the sub-stages as
appropriate.
StagedPackage can be configured to only run certain sub-stages. This
is done by setting the optional kwarg install_phases when calling
do_install(). Setting install_phases() ONLY has an effect on
StagedPackage, not on any existing packages. By default,
install_phases is set to make StagedPackage run the same stages that
are normally run for any package: configure, build, install (and NOT
spconfig).
The ability for Spack to run stages selectively for StagedPackage
instances will enable new functionality. For example, explicit
CMake/Autotools helpers that allow Spack to help configure a user's
project without fetching, building or installing it.
-------------
One implementation of StagedPackage is provided, CMakePackage. This
has the following advantage for CMake-based projects over using the
standard Package class:
a) By separating out the phases, it enables future new functionality
for packages that use it.
b) It provides an implementation of intall_spconfig(), which will
help users configure their CMake-based projects.
CMakePackage expects users to implement configure_args() and
configure_env(). These methods provide the package-specific arguments
and environment needed to properly configure the package. They are
placed in separated functions because they are used in both the
spconfig and configure stages.
TODO:
1. Generate spconfig.py scripts that are more readable. This allows
the users to understand how their project is configured.
2. Provide a practical way for the user to ACCESS the spconfig stage
without building the project through Spack.
3. The CMAKE_TRANSITIVE_INCLUDE_PATH stuff needs to be reworked; it
should be considered provisional for now.
4. User of Autotools might wish to make a similar ConfigurePackage
subclass of StagedPackage.
---------------
One package using CMakePackage is introduced. See ibmisc/package.py.
- Move `Spec.__cmp__` out of spec, into concretize as `cmp_specs`.
- `Spec.__cmp__` was never called (except explicitly) due to rich
comparison operators from `key_ordering`
- Refactor `_find_other_spec` to free function `find_spec`. Add a test
for it to make sure it works.
- Had attempted to add more functionality by assigning different
meanign None, True, and False values "keep_stage" (where False was
"always delete").
- Turns out that's not really worth the complexity. Having the third
"always delete" sense is hardly ever useful but makes the code hard
to understand.
- Fix bug introduced during merge of stage refactor.
- install prefix was not created before build_environment.fork()
- build_environment.fork() calls setup_dependent_environment
- python's setup_dependent_environment can inadvertently create
the install prefix before directory_layout expects it.
- Clean up Package.do_install:
- simplify control flow: parent process now entirely responsible for
creating/destroying the install prefix. cleanup is now in one place.
- Hoisting cleanup out of the child improves nesting of try/catch in
`real_work`.
- `real_work` renamed to `build_process`
- Allows skipping the expand step for downloads.
- Fixed stage so that it knows expansion didn't fail when there is a
no-expand URLFetchStrategy.
- Updated docs to reflect new option, and provided an example.
- package.py uses context manager more effectively.
- Stage.__init__ has easier to understand method signature now.
- keep can be used to override the default behavior either to keep
the stage ALL the time or to delete the stage ALL the time.
- This reverts commit c5d9ee8924.
- merged too soon before
- reverting and fixing bugs now.
Conflicts:
lib/spack/spack/mirror.py
lib/spack/spack/package.py
This does several things:
- Add `sbang`: a script to run scripts with long shebang lines.
- Documentation for `sbang` is in `bin/sbang`.
- Add an `sbang` hook that filters the `bin` directory after install
and modifies any scripts wtih shebangs that are too long to use
`sbang` instead.
- `sbang` is at the top level, so it should be runnable (not much we
can do if spack itself is too deep for shebang)
- `sbang`, when used as the interpreter, runs the *second* shebang
line it finds in a script.
- shoud fix issues with too long shebang paths.
Compiler detection was not getting triggered properly with some of the
new config logic. Adjust the conditions under which Spack will serach
for compilers.
- Recent external repo refactoring made `repo_for_pkg` raise an error
when the package was not known.
- Correct behavior is to return the highest precedence repo.
- Old test: did not attempt to actually fetch mirrored packages.
- New test:
1. Creates a temporary mirror,
2. Registers it with spack,
3. Fetches from it, and
4. Verifies that the fetched archive matches the original
- This test will hopefully mean that `spack mirror` is less brittle
from now on.
- simplify output, error, and input redirection
- `return_output=True` is now `output=str`
- `return_output=True` will still work for the time being but is
deprecated.
- previously you could say `return_output=True` and `output=<stream>`,
which wouldn't actually write to the stream. Now you actually can't
specify erroneous cases since there is only one parameter with
mutually exclusive options..
This is necessary for the NAG Fortran compiler, which prints its version message to STDERR instead of STDOUT. This message was previously being ignored, preventing spack from finding the compiler's version or automatically adding it to the compilers.yaml configuration file.
* develop: (77 commits)
Make libxcb compile with gcc 4.9.
disable parallel install for glib (found races)
disable cairo-trace, which is incompatible with older libiberty.h.
Fix LaunchMon on newer gcc versions.
Simplify output redirection in spack.util.executable
Allow completely empty config files.
Update cmake package to use http in all URLs.
Fix create, diy, edit, and repo commands to use multiple repos.
Add namespace option to find command.
Temporary fix: Clang is the default compiler on Mac OS X.
Netcdf requires cmake >=2.8.12
Mbedtls depends on cmake
Update libtool to 2.4.6
Update Julia to 0.4.3
Make binutils build on OS X
Add verbs and psm variants
py-matplotlib: depend on freetype
py-numpy: make blas/lapack optional dependencies
qhull: apply patch to work with libc++'s standard library
New package GNU tar
...
- By default inherit parent's input/output descriptor
- Only use pipes if we need to return output.
- Allows subprocesses (like emacsclient) to detect terminal correctly
- This moves var/spack/packages to var/spack/repos/builtin/packages.
- Packages that did not exist in the source branch, or were changed in
develop, were moved into var/spack/repos/builtin/packages as part of
the integration.
Conflicts:
lib/spack/spack/test/unit_install.py
var/spack/repos/builtin/packages/clang/package.py
- mirrors.yaml now uses dict order for precedence, instead of lists of
dicts.
- spack.cmd now specifies default scope for add/remove and for list
with `default_modify_scope` and `default_list_scope`.
- commands that only read or list default to all scopes (merged)
- commands that modify configs modify user scope (highest
precedence) by default
- These vars are used in setup_paraser for mirror/repo/compiler.
- Spack's argparse supports aliases now.
- added 'rm' alias for `spack [repo|compiler|mirror] remove`
This update significantly reworks the llvm and clang packages. The llvm
package now includes variants allowing it to build and install any and
all of:
* clang
* lldb
* llvm's libunwind (why, WHY did they name it this?!?)
* polly (including building it directly into the clang tools, 3.7.0 only)
* clang extra tools
* compiler-rt (sanitizers)
* clang lto (the gold linker plugin that allows same to work)
* libcxx/libcxxabi
* libopenmp, also setting the default openmp runtime to same, when
parameters happen this shoudl be an option of libomp or libgomp
Ideally, this should have rpath setup like the gcc package does, but
clang's driver has no support for specs as such, and no clearly
equivalent mechanism either. If anyone has ideas on this, they would be
welcome.
One significant note related to gcc though, if you test this on LLNL
systems, or anywhere that has multiple GCCs straddling the dwarf2
boundary and sharing a libstdc++, build a gcc with spack and use that to
build clang. If you use a gcc4.8+ to build this with an older
libstdc++ it will fail on missing unwind symbols because of the
discrepancy.
Resource handling has been changed slightly to move the unpacked archive
into the target rather than use symlinks, because symlinks break certain
kinds of relative paths, and orders resource staging such that nested
resources are unpacked after outer ones.
This solution doesn't really make me happy, but does seem to work. It
sorts the resources by the length of the string representing their
destination. Since any nested resource must contain another resource's
name in its path, it seems that should work, but there should be a
better way to do this.
This allows resources to be placed into subdirectory trees that may not
exist in the base package, and may depend on other resources to be
staged later.
- All of these work:
- `spack mirror add`
- `spack mirror remove`
- `spack mirror list`
- `spack mirror` subcommands (except create) now have their own
--scope argument.
- Mirror config is now stored sanely as an ordered list.
- `spack compiler` subcommands now take an optional --scope argument.
- no more `remove_from_config` in `config.py` -- `update` just
overwrites b/c it's easier to just call `get_config`, modify YAML
structures directly, and then call `update`.
- Implemented `spack compiler remove`.
- Configs are now parsed with `spack.util.spack_yaml.load/dump`
- Parser annotates returned data with `_start_mark` and `_end_mark`
properties, so that we can recover what lines/files they came from.
- Parser uses `OrderedDict` instead of `dict`. This will help
maintain some sanity when round-tripping config files.
- User and site config are now kept separately in memory.
- Merging is done on demand when client code requests the configuration.
- Allows user/site config to be updated independently of each other by commands.
- simplifies config logic (no more tracking merged files)
- Stage and fetcher were not being set up properly when fetching using
a different fetch strategy than the default one for the package.
- This is fixed but fetch/stage/mirror logic is still too complicated
and long-term needs a rethink.
- Spack will now print a warning when fetching a checksum-less tarball
from a mirror -- users should be careful to use https or local
filesystem mirrors for this.
- Move `find_versions_of_archive` from spack.package to `spack.util.web`.
- `spider` funciton now just uses the link parsing it already does to
return links. We evaluate actual links found in the scraped pages
instead of trying to reconstruct them naively.
- Add `spack url-parse` command, which you can use to show how Spack
interprets the name and version in a URL.
Versions found by wildcard URLs are different from versions found by
parse_version, etc. The wildcards are constructed more haphazardly
than the very specific URL patterns in url.py, so they can get things
wrong. e.g., for this URL:
https://software.lanl.gov/MeshTools/trac/attachment/wiki/WikiStart/mstk-2.25rc1.tgz
We miss the 'rc' and only return 2.25r as the version if we ONLY use
URL wildcards.
Future: Maybe use the regexes from url.py to scrape web pages, and
then compare them for similarity with the original URL, instead of
trying to make a structured wildcard URL pattern? This might yield
better results.
- remove getcwd() check (seems arbitrary -- if users set their TMPDIR
to this why stop them?)
- try a number of common locations and try per-user directories in
them first.
- Adding `preferred=True` to a version directive will change its sort
order in concretization.
- This provides us a rudimentary ability to keep the Spack stack
stable as new versions are added.
- Having multiple stacks will come next, but this at least allows us
to specify default versions of things instead of always taking the
newest.
This changes the compiler wrappers so that they are called by the same
name as the wrapped compiler. Many builds make assumptions about
compiler names, and we need the spack compilers to be recognizable so
that build systems will get their flags right.
This adds per-compiler subdirectories to lib/spack/spack/env directory
that contain symlinks to cc for the C, C++, F77, and F90
compilers. The build now sets CC, CXX, F77, and F90 to point to these
links instead of to the generically named cc, c++, f77, and f90
wrappers.
- Extension logic didn't take conditional deps into account.
- Extension methods now check for whether the extnesion is
in the extendee map AND whether the dependency is actually present
in the spec yet.
This changes the hash algorithm so that it does much less object
allocation and copying, and so that it is correct.
The old version of `_cmp_key()` would call `sorted_deps`, which would
call `flat_dependencies` to get a list of dependencies so that it
could sort them in alphabetical order. This isn't necessary in the
`_cmp_key()`, and in fact we want more DAG structure than that to be
included in the `_cmp_key()`.
The new version constructs a tuple without copying the Spec DAG, and
the tuple contains hashes of sub-DAGs that are computed recursively
in-place. This is way faster than the previous algorithm and reduces
the numebr of copies significantly. It is also a correct DAG hash.
Example timing and copy counts for the different hashing algorithms
we've tried:
Original (wrong) Spec hash:
```
106,170 copies
real 0m5.024s
user 0m4.949s
sys 0m0.104s
```
Spec hash using YAML `dag_hash()`:
```
3,794 copies
real 0m5.024s
user 0m4.949s
sys 0m0.104s
New no-copy, no-YAML hash:
```
3,594 copies
real 0m2.543s
user 0m2.435s
sys 0m0.104s
```
So now we have a hash that is correct AND faster.
The remaining ~3k copies happen mostly during concretization, and as
all packages are initially loaded. I believe this is because Spack
currently has to load all packages to figure out virtual dependency
information; it could also be becasue there ar a lot of lookups of
partial specs in concretize. I can investigate this further.
- _cross_provider_maps() had suffered some bit rot (map returned was
ill-formed but still worked for cases with one vdep)
- ProviderIndex.satisfies() was only checking whether the result map
was non-empty. It should check whether all common vdeps are *in*
the result map, as that indicates there is *some* way to satisfy
*all* of them. We were checking whether there was some way to
satisfy *any one* of them, which is wrong.
- Above would cause a problem when there is more than one vdep provider.
- Added test that covers this case.
- Added `constrained()` method to Spec. Analogous to `normalized()`:
`constrain():constrained() :: normalize():normalized()`
Added test for, e.g.:
import spack.pkg.builtin.mock.mpich
import spack.pkg.builtin.mock.mpich as mpich
from spack.pkg.builtin.mock.mpich import Mpich
Among others. These ensure that direct package imports work so that
packages can be extended.
Package repositories now look like this:
top-level-dir/
repo.yaml
packages/
libelf/
package.py
mpich/
package.py
...
This leaves room at the top level for additional metadata, source,
per-repo configs, indexes, etc., and it makes it easy to see that
something is a spack repo (just look for repo.yaml and packages).
unit tests, so tracking tests with sets wouldn't work unless I extracted the
details relevant to the particular test. For now a simple count will work so
using a set was unnecessary anyways.
1. Adding a plugin to keep track of the total number of tests run as well as the
number of tests with failures/errors.
2. Some nose plugins (including xunit which will be added in a future commit)
assign stdout to a stream object that does not have a .fileno attribute.
spack.util.executable.Executable now avoids passing stdout to subprocess (and
always uses subprocess.PIPE)
TODO:
1. Still need to figure out how to activate the plugin (as of now it is
being ignored by nose). Newer versions of nose appear to make this simpler
(e.g. the "addplugins" argument to nose.run)
2. Need to include new version of nose in order to use xunit
python install and lib/spack/external have the same library installed. This
requires modifying the names of some modules in lib/spack/external in cases
where both the system python and backported features of future python versions
(i.e. after 2.6) are used (previously distinguished by "from external import X"
and "import X").
Flags are passed from the command line all the way through
build environments to environment variables.
Flags are specified using +name=value and values are quoted
using escaped quotes when necessary.
Future work includes using the flags in the compiler wrapper script
and hopefully updating the parser for a gentler user experience of
the spec language.
- `remove_prefix` was modified to remove from the DB, but the package
may not have been added to the DB yet when `remove_prefix` is called
from `cleanup`.
- Made `remove_prefix` a pure utility function (it just removes the prefix)
- Added `installed_db.remove()` call only after the `remove_prefix` in
`uninstall`.
- Code simplification ignored case where exception was raised.
- If LockError was raised, read and write counts were incremented erroneously.
- updated lock test.
output path is specified. Test output files are placed in this directory.
Furthermore the filenames now have the prefix "test" (but otherwise are the
string representation of the spec ID as before).
- llnl.util.lock now uses fcntl.lockf instead of flock
- purported to have more NFS compatibility.
- Added an extensive test case for locks.
- tests acquiring, releasing, upgrading, timeouts, shared, & exclusive cases.
convert individual bytes, so therefore incorrectly converts utf-8 encoded
characters. Decoding byte strings to unicode objects results in correct HTML
number encodings.
2. The parent of a failed child process in build_environment.fork no longer
calls sys.exit - instead it raises a CommandError (from [1])
3. test-install command now attempts to install all packages even if one fails
command by having it consume exactly 1 positional argument (i.e. by removing
"nargs=argparse.REMAINDER") but this does not work when configuring dependencies
of a top-level package (which show up as additional positional args). Instead
now there is an explicit check to ensure there is only 1 top-level package.
argument). The default path is [package id].xml in the CWD where test-install
is called from.
2. Fixed a bug with package.build_log_path (which was added in this branch).
3. keep_stage for package.do_install is now set. This allows uninstalling and
reinstalling packages without (re) downloading them.
Also even though I calculated which installs are new (e.g. vs. packages that
have already been installed by a previous command) I forgot to make use of that
in create_test_output (so I was always generating test output even if a package
had been installed before running the test-install command).
Note to avoid confusion: the 'handled' variable (removed in this commit) did not
serve the same purpose as 'newInstalls': it was originally required because the
recursive approach would visit the same dependency twice if more than one
package depended on it.
packages are specified and a prior one fails, it will prevent any of the others
from succeeding (and generating test output) even if they don't share
dependencies.
