Commit graph

983 commits

Author SHA1 Message Date
Todd Gamblin
fa3e91bf2a Revert "Add lmod files to MODULEPATH" (#3917)
* Revert "Add lmod files to MODULEPATH (#3912)"

This reverts commit 186d1f4511.
2017-04-20 03:05:44 -07:00
scheibelp
a65c37f15d Override partial installs by default (#3530)
* Package install remove prior unfinished installs

Depending on how spack is terminated in the middle of building a
package it may leave a partially installed package in the install
prefix. Originally Spack treated the package as being installed if
the prefix was present, in which case the user would have to
manually remove the installation prefix before restarting an
install. This commit adds a more thorough check to ensure that a
package is actually installed. If the installation prefix is present
but Spack determines that the install did not complete, it removes
the installation prefix and starts a new install; if the user has
enabled --keep-prefix, then Spack reverts to its old behavior.

* Added test for partial install handling

* Added test for restoring DB

* Style fixes

* Restoring 2.6 compatibility

* Relocated repair logic to separate function

* If --keep-prefix is set, package installs will continue an install from an existing prefix if one is present

* check metadata consistency when continuing partial install

* Added --force option to make spack reinstall a package (and all dependencies) from scratch

* Updated bash completion; removed '-f' shorthand for '--force' for install command

* dont use multiple write modes for completion file
2017-04-19 21:59:18 -07:00
Adam J. Stewart
186d1f4511 Add lmod files to MODULEPATH (#3912) 2017-04-19 21:30:08 -07:00
Adam J. Stewart
50df071ad9 Overhaul Spack's URL parsing (#2972)
* Remove fake URLs from Spack

* Ignore long lines for URLs that start with ftp:

* Preliminary changes to version regexes

* New redesign of version regexes

* Allow letters in version-only

* Fix detection of versions that end in Final

* Rearrange a few regexes and add examples

* Add tests for common download repositories

* Add test cases for common tarball naming schemes

* Finalize version regexes

* spack url test -> spack url summary

* Clean up comments

* Rearrange suffix checks

* Use query strings for name detection

* Remove no longer necessary url_for_version functions

* Strip off extraneous information after package name

* Add one more test

* Dot in square brackets does not need to be escaped

* Move renaming outside of parse_name_offset

* Fix versions for a couple more packages

* Fix flake8 and doc tests

* Correctly parse Python, Lua, and Bio++ package names

* Use effective URLs for mfem

* Add checksummed version to mitos

* Remove url_for_version from STAR-CCM+ package

* Revert changes to version numbers with underscores and dashes

* Fix name detection for tbb

* Correctly parse Ruby gems

* Reverted mfem back to shortened URLs.

* Updated instructions for better security

* Remove preferred=True from newest version

* Add tests for new `spack url list` flags

* Add tests for strip_name_suffixes

* Add unit tests for version separators

* Fix bugs related to parseable name but in parseable version

* Remove dead code, update docstring

* Ignore 'binary' at end of version string

* Remove platform from version

* Flip libedit version numbers

* Re-support weird NCO alpha/beta versions

* Rebase and remove one new fake URL

* Add / to beginning of regex to avoid picking up similarly named packages

* Ignore weird tar versions

* Fix bug in url parse --spider when no versions found

* Less strict version matching for spack versions

* Don't rename Python packages

* Be a little more selective, version must begin with a digit

* Re-add fake URLs

* Fix up several other packages

* Ignore more file endings

* Add parsing support for Miniconda

* Update tab completion

* XFAILS are now PASSES for 2 web tests
2017-04-03 15:34:16 -07:00
Adam J. Stewart
c0cfaacbc8 Set default providers for all virtual dependencies (#3634)
* Set default providers for everything

* Add default OpenFOAM provider
2017-04-02 14:48:27 -05:00
Mark Olesen
9e1abb13dc support OpenFOAM package(s) (#3528)
* ENH: add package for building OpenFOAM (1612) from www.openfoam.com
- provide 'openfoam' as virtual package.
- package as openfoam-com to reflect the distribution point.

This initial spack packaging for OpenFOAM supports a number of possible
variants and should handle 64-bit labels properly now that the scotch
package has been updated accordingly.

* ENH: update package for foam-extend (extend-project.de)

- provide 'openfoam' as virtual package.

- much of the build is now aligned with how the openfoam-com package
  looks, with the aim of future refactoring.

- avoid installing intermediate targets.

- contains its own environment sourcing script for the build, for more
  flexibility and robustness (doesn't touch the python build environ)

* ENH: added package for building from openfoam.org

- provide 'openfoam' as a virtual package.

- this is largely a direct copy of the openfoam-com package.
  It has been supplied as a courtesy for users and to ensure maximum
  consistency in quality and naming between the foam-extend,
  openfoam-com and openfoam-org packages.

* CONFIG: add openfoam into bash completion providers list

* ENH: have openfoam-com use spack as USERMPI

- also simplify the generation of mplib/compiler rules

* ENH: have openfoam-org use spack as SYSTEMMPI

- this setup requires more environment settings than USERMPI
  (openfoam-com), but is currently the only means of integration
  for openfoam-org

- simplify generation of mplib/compiler rules

* ENH: simplify generation of mplib/compiler rules (foam-extend)

- rename mpi rules from SPACK,SPACKMPI to USER,USERMPI for consistency
  with openfoam-com and to generalize for any build system.

* STYLE: record spack tree as a log file (openfoam)

- can be useful for future diagnostics and general record keeping
2017-03-30 16:35:57 -07:00
Adam J. Stewart
f0e99456ed Fix tab completion of directories and filenames (#3184) 2017-02-20 15:44:58 -08:00
Adam J. Stewart
c2d210568c Fix tab completion after change to uninstall flag (#3175) 2017-02-18 13:08:59 -06:00
Adam J. Stewart
941dfcbe43 Bash Programmable Completion for Spack (#3026) 2017-02-06 12:34:35 -08:00
Michael Kuhn
258dfc707d Export spack function so it works in subshells (#2908) 2017-01-23 15:29:25 -08:00
Massimiliano Culpo
34d23c617c tests: coverage combine run only if tests succeeded (#2712) 2017-01-01 13:05:33 -08:00
Todd Gamblin
b5537553f2 Run coverage for the spack install command as well as spack test.
- Also don't run coverage on OSX.
2016-12-30 17:18:19 -08:00
Massimiliano Culpo
7ea10e768e unit tests: replace nose with pytest (#2502)
* Porting: substitute nose with ytest

This huge commit substitutes nose with pytest as a testing system. Things done here:

* deleted external/nose as it is no longer used
* moved mock resources in their own directory 'test/mock/'
* ported two tests (cmd/find, build_system) to pytest native syntax as an example
* build_environment, log: used monkeypatch instead of try/catch
* moved global mocking of fetch_cache to an auto-used fixture
* moved global mocking from test/__init__.py to conftest.py
* made `spack test` a wrapper around pytest
* run-unit-tests: avoid running python 2.6 tests under coverage to speed them up
* use `pytest --cov` instead of coverage run to cut down testing time

* mock/packages_test: moved mock yaml configuration to files instead of leaving it in the code as string literals

* concretize.py: ported tests to native pytest, reverted multiprocessing in pytest.ini as it was creating the wrong report for coveralls

* conftest.py, fixtures: added docstrings

* concretize_preferences.py: uses fixtures instead of subclassing MockPackagesTest

* directory_layout.py: uses fixtures instead of subclassing MockPackagesTest

* install.py: uses fixtures instead of subclassing MockPackagesTest

* optional_deps.py: uses fixtures instead of subclassing MockPackagesTest

optional_deps.py: uses fixtures instead of subclassing MockPackagesTest

* packages.py: uses fixtures instead of subclassing MockPackagesTest

* provider_index.py: uses fixtures instead of subclassing MockPackagesTest

* spec_yaml.py: uses fixtures instead of subclassing MockPackagesTest

* multimethod.py: uses fixtures instead of subclassing MockPackagesTest

* install.py: now uses mock_archive_url

* git_fetch.py: uses fixtures instead of subclassing MockPackagesTest

* hg_fetch.py: uses fixtures instead of subclassing MockPackagesTest

* svn_fetch.py, mirror.py: uses fixtures instead of subclassing MockPackagesTest
repo.py: deleted

* test_compiler_cmd.py: uses fixtures instead of subclassing MockPackagesTest

* cmd/module.py, cmd/uninstall.py: uses fixtures instead of subclassing MockDatabase

* database.py: uses fixtures instead of subclassing MockDatabase, removed mock/database

* pytest: uncluttering fixture implementations

* database: changing the scope to 'module'

* config.py: uses fixtures instead of subclassing MockPackagesTest

* spec_dag.py, spec_semantics.py: uses fixtures instead of subclassing MockPackagesTest

* stage.py: uses fixtures instead of subclassing MockPackagesTest. Removed mock directory

* pytest: added docstrings to all the fixtures

* pytest: final cleanup

* build_system_guess.py: fixed naming and docstrings as suggested by @scheibelp

* spec_syntax.py: added expected failure on parsing multiple specs closes #1976

* Add pytest and pytest-cov to Spack externals.

* Make `spack flake8` ignore externals.

* run-unit-tests runs spack test and not pytest.

* Remove all the special stuff for `spack test`

- Remove `conftest.py` magic and all the special case stuff in `bin/spack`

- Spack commands can optionally take unknown arguments, if they want to
  handle them.

- `spack test` is now a command like the others.

- `spack test` now just delegates its arguments to `pytest`, but it does
  it by receiving unknown arguments and NOT taking an explicit
  help argument.

* Fix error in fixtures.

* Improve `spack test` command a bit.

- Now supports an approximation of the old simple interface
- Also supports full pytest options if you want them.

* Use external coverage instead of pytest-cov

* Make coverage use parallel-mode.

* change __init__.py docs to include pytest
2016-12-29 07:48:48 -08:00
Adam J. Stewart
6cc9b31747 Allow shell support to work when set -u is set (#2418) 2016-11-29 17:04:29 -08:00
Gregory Becker
7201053f2a Basic usage step-by-step tutorial 2016-11-12 16:55:37 -07:00
Massimiliano Culpo
296a349d49 setup script for bash : fixes #2209 for sh setup (#2212) 2016-11-04 12:00:01 -07:00
Todd Gamblin
1b7f9e24f4 Add spack flake8 command. (#2186)
- Ported old run-flake8-tests qa script to `spack flake8` command.
- New command does not modify files in the source tree
- Copies files to a temp stage modifies them there, and runs tests.
- Updated docs and `run-flake8-tests` script to call `spack flake8`.
2016-10-31 11:40:20 -07:00
Joseph Ciurej
82fcc5e1de Feature Proposal : Make All Python Extensions User Configuration Independent (#1435)
* Updated all Python extension packages to use 'setup_py' on install.

* Fixed a few minor style issues with the updated Python packages.
2016-10-06 14:43:49 -07:00
Adam J. Stewart
d5e5e4ed1a Add missing dependencies for Flake8 (#1883)
* Warn user if flake8 can't find setuptools

* Add missing dependencies of flake8

* Updates to py-autopep8, make packages activateable

* Check for presence of setuptools for Sphinx too

* Fix bug in order of commands
2016-10-05 10:00:31 -07:00
Adam J. Stewart
2ce1b388c9 Run make clean to prevent warning messages (#1742)
* Run make clean to prevent warning messages

* Don't delete temporary files after completion
2016-10-02 18:51:09 -07:00
Adam J. Stewart
64194a823a Remove duplicate ICU package (#1837)
* Remove duplicate ICU package

* Ignore deleted files during flake8 tests

* Rename Boost ICU variant
2016-09-23 14:16:59 -07:00
Adam J. Stewart
06d8a3ad23 Add missing dependency for run-doc-tests (#1691) 2016-09-08 09:02:38 -07:00
Adam J. Stewart
f6d07b54f1 Always run spack unit tests 2016-08-30 16:01:00 -05:00
Adam J. Stewart
e04b76c2c3 Add spack to the PATH for doc tests 2016-08-30 15:35:58 -05:00
Adam J. Stewart
a0c8aca3a2 Remove py-mercurial, going with mercurial package instead 2016-08-30 15:35:58 -05:00
Adam J. Stewart
f856952728 Always run Documentation tests 2016-08-30 15:32:54 -05:00
Adam J. Stewart
c36f13e44d Added more missing dependencies 2016-08-30 15:21:15 -05:00
Adam J. Stewart
7f9d098c2f Add mercurial package, used as test dependency 2016-08-30 15:21:15 -05:00
Adam J. Stewart
1fc14fd7ed Only run unit tests when core Spack framework is modified 2016-08-30 15:21:15 -05:00
Adam J. Stewart
d2d6c91b66 Run documentation tests when documentation is modified 2016-08-30 15:21:15 -05:00
Adam J. Stewart
679f787a65 Add generic changed_files script 2016-08-30 15:21:15 -05:00
Adam J. Stewart
732c1985ef Overhaul Spack's CI Infrastructure 2016-08-30 15:20:03 -05:00
Adam J. Stewart
d39322e278 Run flake8 checks on new untracked files (#1510) 2016-08-28 19:35:09 -07:00
Adam J. Stewart
87d0a7c315 Always clean up tmp files, even if killed 2016-08-11 09:56:41 -05:00
Adam J. Stewart
09c9786fab Allow run-flake8 to be run from anywhere 2016-08-11 09:53:04 -05:00
Adam J. Stewart
1edfc82123 Run flake8 checks on changed uncommitted files 2016-08-11 09:53:04 -05:00
Todd Gamblin
bf1072c902 Make Spack core PEP8 compliant. 2016-08-10 16:33:37 -07:00
alalazo
f4422dc165 qa : fixes #1370 (no flake8 on python 2.6, coveralls only python 2.7) 2016-07-30 18:49:23 +02:00
Joseph Ciurej
fa92f58167 Adding temporary script that fixes all Python extensions w/ 'setup_py'. 2016-07-22 11:17:19 -07:00
Michael Kuhn
732c24f603 Fix recursive module loading. 2016-07-21 13:15:53 +02:00
alalazo
a96eb7096d Merge branch 'develop' of https://github.com/LLNL/spack into features/module_refresh
Conflicts:
	share/spack/setup-env.sh
2016-07-18 14:12:47 +02:00
becker33
36f818b9fb Merge pull request #1246 from suraia/improve-shell-integration
Improve shell integration.
2016-07-15 14:23:51 -07:00
Adam J. Stewart
a4a2f179c7 Updates to run-flake8 test 2016-07-15 12:11:41 -05:00
Michael Kuhn
64d91943ec Support --dependencies for recursively loading modules. 2016-07-14 20:14:06 +02:00
Michael Kuhn
fcfe2618d5 Improve shell integration.
Enable sh emulation for Zsh and allow loading modules recursively.
2016-07-13 20:33:14 +02:00
alalazo
b71d430af6 module : can regenerate single module files, homogenized cli options
spack module :
- refresh accepts a constraint
- find and refresh share common cli options
- ask for confirmation before refreshing
- deleting the module file tree is now optional
2016-06-28 19:28:18 +02:00
Adam J. Stewart
ea80c8d3c3 Ignore E501 for depends_on and extends 2016-06-15 14:38:53 -05:00
Joseph Ciurej
761c5c8450 Added line limit exemptions for 'variant' and 'version' directives.
Updated the 'scotch' package to use oneline 'variant' directives.
2016-05-16 12:14:57 -07:00
Joseph Ciurej
52c8bedb9c Updated style checking to ignore refinition errors for '@when' fxns.
Updated the Scotch package to conform to PEP8 standards and to test style checking improvements.
2016-05-16 11:09:44 -07:00
Todd Gamblin
e7ced54369 Correct LLNL LGPL license template for clarity. 2016-05-11 21:22:25 -07:00
alalazo
78ae5d7723 Merge branch 'develop' of https://github.com/LLNL/spack into features/custom_modulefile_from_config
Conflicts:
	lib/spack/spack/config.py
2016-05-11 16:11:19 +02:00
Todd Gamblin
78d25ad337 Add run-flake8 script.
- was missing the obvious.
2016-05-11 02:01:24 -07:00
Todd Gamblin
d7847ff768 Use just one flake8 file; mark package.py files to avoid E501 w/url. 2016-05-11 01:16:41 -07:00
Todd Gamblin
2aa4387eba Update code style checking.
- Exempt overlong URL lines from checks.
- Omit some of the more painful PEP items.
2016-05-11 00:07:42 -07:00
alalazo
9eee71edf7 Merge branch 'develop' of https://github.com/LLNL/spack into features/custom_modulefile_from_config 2016-05-10 11:13:19 +02:00
Tom Scogland
584501a1cc switched test to use -d for location check 2016-04-29 09:34:18 -07:00
Tom Scogland
10d8ed4d98 check that a target directory exists before committing to cd 2016-04-28 15:43:05 -07:00
alalazo
c69acfa5c8 naming work correctly 2016-04-18 13:22:04 +02:00
Erik Schnetter
f5f7d47be1 Support Bash "-u" mode 2016-03-19 18:42:39 -04:00
Tom Scogland
926efed31c another portability fix, this time for zsh/non-bash sh
Yay for non-portable declaration syntax.  After the previous screwiness
I ran this through a number of shells, and found that this is the most
portable version I coudl seem to get.
2015-12-30 10:55:52 -08:00
Tom Scogland
8726d2fe2e removing ill-fated array check for non-portability 2015-12-30 10:29:11 -08:00
Todd Gamblin
3e3667f254 Revert "removing check as it can cause failures in zsh"
This reverts commit 496e485691.

Undo overly large set of changes -- LLVM wasn't ready.  Tom can
recommit just the zsh stuff.
2015-12-30 10:20:27 -08:00
Tom Scogland
496e485691 removing check as it can cause failures in zsh 2015-12-30 10:13:46 -08:00
Tom Scogland
d7607973fa fix for array handling in ancient bash
fixes #284
2015-12-29 09:04:54 -08:00
Mike Nolta
527154e6df fix env module sys_type on non-LLNL systems 2015-12-19 16:34:24 -05:00
John Westlund
3e1ba6cc24 remove possible trailing slash in _sp_prefix or _sp_share_dir 2015-12-18 21:28:44 -08:00
Todd Gamblin
a173324069 Merge pull request #205 from trws/shell-arg-handling-fix
fix for the shell arg handling issues
2015-12-17 23:01:46 -08:00
Todd Gamblin
c41b9b7ddc Change github.com/scalability-llnl to github.com/llnl everywhere. 2015-12-09 01:24:15 -08:00
Tom Scogland
3ce99cf4e1 fix for the shell arg handling issues
The setup-env.sh script now saves off the argument array before shifting off
all the arguments so that the final command invocation doesn't arbitrarily
requote the arguments.  All external quoting is preserved now.  This has been
thoroughly tested with zsh, but should probably be tested with bash etc.
before acceptance.
2015-11-25 11:02:51 -08:00
Charles A. Reynolds
f106da0d09 Added missing $_sp_flags to spack.csh so options -d -k -m -p -v get passed on to spack proper. 2015-11-06 10:53:22 -08:00
Todd Gamblin
3c788147ca Add Spack logo. 2015-10-18 19:14:40 -07:00
Tom Scogland
b98454b8b2 fixing regular expression matches to use regular expression syntax instead of globbing syntax 2015-06-09 19:38:35 -07:00
Alfredo Gimenez
d95d48bbe6 py-mako and fix for setup-env.sh 2015-02-06 16:43:21 -08:00
Todd Gamblin
70475d08c0 Bugfix for spack cd -h 2014-09-29 22:39:36 -07:00
Todd Gamblin
d87a652582 Add spack cd and spack location commands.
- Better shell support for cd'ing into directories
- Fix some csh weirdness with nested aliases.
2014-08-22 11:00:19 -07:00
Todd Gamblin
eb5efed421 Merge branch 'features/postgresql' into develop
- add spack cd command.
- Fix bug in modules hook

Conflicts:
	lib/spack/spack/cmd/stage.py
	lib/spack/spack/hooks/dotkit.py
	share/spack/setup-env.bash
2014-08-21 22:59:39 -07:00
Todd Gamblin
b601fd08ca Bugfixes for csh environment modules. 2014-08-17 01:41:32 -07:00
Todd Gamblin
776560f8ce Add csh/tcsh support for modules
- csh scripting is a GIANT pain in the ass
- hopefully the thin script layer doesn't get much more complex.
2014-08-16 14:58:15 -07:00
Todd Gamblin
221cf6acb9 Consolidate most module code into spack.modules and spack.cmd.module
- One file with all the module classes (spack/modules.py)
  - Has an EnvModule superclass that does most of the work and consolidates common code
  - Subclasses have specializations for different module systems (TclModule, Dotkit)

- One command (spack module) for all the types of modules to use
  - the one command is used by the scripts, only need to maintain in one place
  - has some subcommands for different module types, but they're handled mostly generically.

- Consolidate zsh support into a single setup-env.sh script.
2014-08-16 14:53:57 -07:00
Todd Gamblin
abc7d401e2 Add "spack cd" shell support to cd directly into the staged archive. 2014-08-11 22:47:23 -07:00
David Beckingsale
57ddbd282a Fixed up module support 2014-08-04 07:53:40 -07:00
David Beckingsale
94c5c9667c Added inital module support 2014-08-04 07:53:40 -07:00
Todd Gamblin
295ffd8c50 Adding dotkit support to TAU.
- New spack.hooks package
  - contains modules with pre and post install hooks

- New dotkit hook module
  - generates/removes dotkits on install/uninstall

- New spack use, spack unuse commands
  - use same syntax as install/uninstall

- New setup-env.bash script
  - Sets up path, dotkit support

- new spack dotkit command
  - used by script to parse specs, generate
    specs of installed pckages for dotkit file names
2014-07-08 01:58:20 -07:00