Commit graph

6984 commits

Author SHA1 Message Date
Todd Gamblin
313771c734 Improve grammar in build log error message. (#5214)
- "1 error found" instead of "1 errors found"

- don't print any build log context if no errors were found; just refer
  the user to the build log.
2017-08-25 04:07:42 -07:00
Mark C. Miller
a0f39397c1 Add contributors link to Sphinx docs (#5213) 2017-08-25 02:57:44 -07:00
Sergey Kosukhin
fda3249c8b Respect --insecure (-k) flag when fetching list_url. (#5178)
* Respect --insecure when fetching list_url.
* Ensure support for Python 2.6, and that urlopen works for python versions prior 2.7.9 and between 3.0 and 3.4.3.
2017-08-25 02:15:46 -07:00
Denis Davydov
c0d3d33b58 dealii: add nanoflann, sundials and adol-c plus fixes on macOS (#5175)
* expat: fix build on macOS

* nanoflann: add new package

* sundials: fix compilation on macOS with clang+gfortran

* adol-c: add 2.6.3

* dealii: add adol-c, nanoflann, sundials and 8.5.1

* sundials: minor cleanup
2017-08-24 20:18:06 -05:00
Michael Kuhn
61cc55985b pkgconf: New package (#5202)
pkgconf is a new pkg-config implementation with additional features and
no external dependencies.

For a detailed comparison, see: http://pkgconf.org/features.html
2017-08-24 17:54:14 -07:00
Audrey Thoma
6ff82ff5a3 ray: new package (#5200) 2017-08-24 16:31:21 -07:00
Ben Boeckel
0e7071befe test/gpg: init from the testing key directory
The old testing pattern set an attribute on the parser directly. Now
that there is a parsed flag, use it instead.
2017-08-24 16:17:35 -07:00
Ben Boeckel
f564b2abf7 gpg: add an argument for the import directory
This is hidden because it is only meant for use by the tests.
2017-08-24 16:17:35 -07:00
Sergey Kosukhin
a48b5a627c Make jpeg a virtual dependency. (#5190)
* Make jpeg a virtual dependency.

* Make 'libjpeg-turbo' the default implementation of 'jpeg'.
2017-08-24 10:30:35 +02:00
sknigh
49a9d63be7 Added options to libfabrics (#5191) 2017-08-23 15:47:28 -07:00
Massimiliano Culpo
5d7901b312 Improve Spec literals, add Spec.from_dict() (#5151)
* Simplified Spec.__init__ signature by removing the *dep_like argument.

The `*dep_like` argument of `Spec.__init__` is used only for tests. This
PR removes it from the call signature and introduces an equivalent
fixture to be used in tests.

* Refactored ``spec_from_dict`` to be a static method of ``Spec``

The fixture ``spec_from_dict`` has been refactored to be a static method
of ``Spec``. Test code has been updated accordingly. Added tests for
exceptional paths.

* Renamed argument `unique` to `normal` + added LazySpecCache class

As requested in the review the argument `unique` of `Spec.from_literal`
has been renamed to `normal`. To avoid eager evaluations of
`Spec(spec_like)` expressions a subclass of `collections.defaultdict`
has been introduced.

* Spec object can be keys of the dictionary for a spec literal.

Added back the possibility use a spec directly as a key. This permits
to build DAGs that are partially normalized.
2017-08-23 14:20:40 -07:00
Christoph Junghans
fa1d0a8a4d Add --source option to spack install (#4102)
- -- source will copy source into prefix along with the package.
- added a test for --source, as well
2017-08-23 14:08:52 -07:00
Audrey Thoma
359b21c888 pgdspider: new package (#5186)
* pgdspider: new package

* couple of fixes
2017-08-23 08:34:12 -05:00
Axel Huebl
0daa639788 Boost: develop (#5184)
Adds a development version of boost, based on their github master
repo.
Git clones the submodules forever, but installs! :)

Useful for testing since boost tends to break complex software
dependencies quite often. This helps testing it before releases.
2017-08-23 07:31:34 -05:00
healther
15186d4ae8 add test for lua and node patching in sbang (#5169)
* add test for lua and node handling in sbang patching (cf #5086)
2017-08-23 01:05:36 -07:00
Todd Gamblin
40e9171390 Add testing for new build output.
- Update handling of ChildError so that its output is capturable from a
  SpackCommand

- Update cmd/install test to make sure Python and build log output is
  being displayed properly.
2017-08-22 16:14:38 -07:00
Todd Gamblin
f51b541ef8 Make install command reusable within single Spack run
- install and probably other commands were designed to run once, but now
  we can to test them from within Spack with SpackCommand

- cmd/install.py assumed that it could modify do_install in PackageBase
  and leave it that way; this makes the decorator temporary

- package.py didn't properly initialize its stage if the same package had
  been built successfully before (and the stage removed).
  - manage stage lifecycle better and remember when Package needs to
    re-create the stage
2017-08-22 16:14:38 -07:00
Todd Gamblin
fa1faa61c4 SpackCommand uses log_output to capture command output. 2017-08-22 16:14:38 -07:00
Todd Gamblin
4f444c5f58 log_ouptut can take either a filename or a file object 2017-08-22 16:14:38 -07:00
Todd Gamblin
139d5bfa6b Display build output on ProcessError, instead of Python context.
- If a failure comes from an external command and NOT the Python code,
  display errors highlighted with some context.

- Add some rudimentary support for parsing errors out of the build log
  (not very sophisticated yet).

- Build errors in Python code will still display with Python context.
2017-08-22 16:14:38 -07:00
Todd Gamblin
d54110d208 Limit package context to 3 lines and colorize in error output. 2017-08-22 16:14:38 -07:00
Massimiliano Culpo
4600d106e2 Config scopes are now returning OrderedDicts instead of dicts. (#5183)
It seems 8f21332fec introduced a bug
in that normal dictionaries are returned from ConfigScope objects
instead of OrderedDicts. This is fixed here.
2017-08-22 14:20:19 -07:00
Audrey Thoma
79df4db8e5 tcoffee: new package (#5185) 2017-08-22 13:48:23 -05:00
Axel Huebl
61c5b5faa7 Boost: 1.65.0 (#5182)
Adds the latest boost release.
Downloads seem to be pointing to bintray from the official homepage.
2017-08-22 13:46:54 -05:00
Sergey Kosukhin
f52bdeda2a Package 'libaec': no need to specify per version urls. (#5176) 2017-08-22 09:27:03 +02:00
Sergey Kosukhin
e55efdf8ed Updates for package 'cdo'. (#5179)
* Package 'cdo': removed obsolete version 1.6.9.

* Package 'cdo': updated urls.

* Package 'cdo': added version 1.9.0.

* Package 'cdo': no need to specify per version urls.

* Package 'cdo': switched from Package to AutotoolsPackage.

* Package 'cdo': added new variant 'openmp'.

* Package 'cdo': variants 'curl' and 'magics' by default.
2017-08-22 08:42:55 +02:00
Matthew Scott Krafczyk
48bf1e276b Add environment variables to path substitution
Update documentation on config file variable substitutions and
add expansion of environment variables in config files.
2017-08-21 18:35:00 -07:00
Audrey Thoma
e0ebf44239 py-pyrad: new package (#5181) 2017-08-21 17:58:05 -05:00
Massimiliano Culpo
581f70ff6f Added custom messages for conflicts directive. fixes #4965 (#5083)
Users can now add an optional custom message to the conflicts directive.
Layout on screen has been changed to improve readability and the long
spec is shown in tree format. Two conflicts in `espresso` have been
modified to showcase the feature.
2017-08-21 09:20:07 -07:00
Axel Huebl
bd8ac0a738 Protobuf: 3.4 (#5177) 2017-08-21 07:29:47 -05:00
Todd Gamblin
10bb681b57 Unbuffer so that output from packages appears when redirecting
- Python I/O would not properly interleave (or appear) with output from
  subcommands.

- Add a flusing wrapper around sys.stdout and sys.stderr when
  redirecting, so that Python output is synchronous with that of
  subcommands.
2017-08-20 16:51:10 -07:00
Todd Gamblin
79045afada Add tests for output redirection. 2017-08-20 16:51:10 -07:00
Todd Gamblin
11196e7b69 Preserve verbosity across installs when 'v' is pressed.
- 'v' toggle was previously only good for the current install.
- subsequent installs needed user to press 'v' again.
- 'v' state is now preserved across dependency installs.
2017-08-20 16:51:10 -07:00
Todd Gamblin
48440766df Fix exit call in SpackError.die()
- Previously we would use `os._exit()` in to avoid Spack error handling
  in the parent process when build processes failed.  This isn't
  necessary anymore since build processes propagate their exceptions to
  the parent process.

- Use `sys.exit` instead of `os._exit`. This has the advantage of
  automatically flushing output streams on quit, so output from child
  processes is not lost when Spack exits.
2017-08-20 16:51:10 -07:00
Todd Gamblin
05cc6c966f Rework output redirection in Spack.
- Simplify interface to log_output. New interface requires only one
  context handler instead of two.  Before:

      with log_output('logfile.txt') as log_redirection:
           with log_redirection:
               # do things ... output will be logged

  After:

      with log_output('logfile.txt'):
          # do things ... output will be logged

  If you also want the output to be echoed to ``stdout``, use the
  `echo` parameter::

      with log_output('logfile.txt', echo=True):
          # do things ... output will be logged and printed out

  And, if you just want to echo *some* stuff from the parent, use
  ``force_echo``:

      with log_output('logfile.txt', echo=False) as logger:
          # do things ... output will be logged

          with logger.force_echo():
              # things here will be echoed *and* logged

  A key difference between this and the previous implementation is that
  *everything* in the context handler is logged.  Previously, things like
  `Executing phase 'configure'` would not be logged, only output to the
  screen, so understanding phases in the build log was difficult.

- The implementation of `log_output()` is different in two major ways:

  1. This implementation avoids race cases by using only one pipe (before
     we had a multiprocessing pipe and a unix pipe).  The logger daemon
     stops naturally when the input stream is closed, which avoids a race
     in the previous implementation where we'd miss some lines of output
     because the parent would shut the daemon down before it was done
     with all output.

  2. Instead of turning output redirection on and off, which prevented
     some things from being logged, this version uses control characters
     in the output stream to enable/disable forced echoing.  We're using
     the time-honored xon and xoff codes, which tell the daemon to echo
     anything between them AND write it to the log.  This is how
     `logger.force_echo()` works.

- Fix places where output could get stuck in buffers by flushing more
  aggressively.  This makes the output printed to the terminal the same
  as that which would be printed through a pipe to `cat` or to a file.
  Previously these could be weirdly different, and some output would be
  missing when redirecting Spack to a file or pipe.

- Simplify input and color handling in both `build_environment.fork()`
  and `llnl.util.tty.log.log_output()`.  Neither requires an input_stream
  parameter anymore; we assume stdin will be forwarded if possible.

- remove `llnl.util.lang.duplicate_stream()` and remove associated
  monkey-patching in tests, as these aren't needed if you just check
  whether stdin is a tty and has a fileno attribute.
2017-08-20 16:51:10 -07:00
Todd Gamblin
e0dd55e090 Make SpackCommand a bit more testable
- add fail_on_error argument
- record exception and return code when the command fails
2017-08-20 16:51:10 -07:00
Todd Gamblin
e77c1a20c5 Fix issue with color formatting regular expression. (#5171)
- Fix issue with color formatting regular expression.
  - _separators regex in spec.py could be constructed such that '^' came
    first in the character matcher, e.g. '[^@#/]'.  This inverts the match
    and causes transient KeyErrors.

  - Fixed to escape all characters in the constructed regex.

  - This bug comes up in Python3 due to its more randomized hash iteration
    order, but it could probably also happen in a Python 2 implementation.

- also clean up variable docstrings in spec.py
2017-08-20 15:34:35 -07:00
Massimiliano Culpo
99fb394ac1 Group Travis CI jobs in stages (#5104)
- This should speed-up Travis CI tests and refers to #5049

- Travis uses build-stages to group tests together
    - The idea is to let fast tests fail first, then move to longer ones.
    - Added external perl to avoid download failure from CPAN and reduce build time
    - Disabling perl-dbi: continues to fail with (504 Gateway Time-out) on Travis

- We now cover all the build systems in tests:
    - Add back `openblas` to Travis as a separate package.
    - Switched `openblas` for `astyle` to build a simpler MakefilePackage.
    - Added 'tut' (WafPackage)
    - Added 'py-setuptools' (PythonPackage)
    - Added 'perl-dbi' (PerlPackage)
    - Added 'build_systems' directory to the ones for which we get a summary
    - Added 'openjpeg' (CMakePackage)
    - Added 'r-rcpp' (RPackage)
    - Added comments to build tests to show the covered build system
2017-08-19 14:52:27 -07:00
Todd Gamblin
bb10bc39ab Make test_inspect_path work on Mac OS X without XCode (#5168)
- Mac OS X Sierra has no /usr/include by default
- Instead of assuming there's an include directory in /usr, mock up a directory that looks like we expect.
2017-08-19 14:48:24 -07:00
健美猫
be0e051870 Add version 1.7.1 for cub. (#5164) 2017-08-19 09:22:19 -05:00
Adam Moody
d80c70538f lbann: add depends_on hwloc (#5165) 2017-08-19 01:00:52 -05:00
Audrey Thoma
fa39afb598 cufflinks: new package (#5158) 2017-08-18 16:02:15 -05:00
Audrey Thoma
3d1c5f679f muscle: fix (#5160) 2017-08-18 16:01:30 -05:00
Kelly (KT) Thompson
c41c5bc9b8 For Tk, also provide a symlink for wishX.X. (#5161)
+ gitk (and possibly other tools) expect to find the Tk executable `wish`, so
  add a symlink with this name that points to the the versioned filename.
+ Example: `wish --> wish8.6`
2017-08-18 16:01:02 -05:00
healther
d6d2dff324 sbang support: add node-js and fix lua
This adds sbang hook support for node-js and fixes the sbang filter
for lua (the character class exclusion was swallowing newlines and
reporting a false positive if lua was mentioned anywhere in the 
file).
2017-08-18 11:57:52 -07:00
healther
6d155833c5 build boost with graph library even when +mpi
Previously, when +graph and +mpi were enabled, the graph library
was replaced with the graph_parallel library. This alters the
logic for +graph+mpi to build both libraries.
2017-08-18 11:19:55 -07:00
Audrey Thoma
9ad8e79332 corset: new package (#5157) 2017-08-18 11:02:58 -05:00
Audrey Thoma
6abe8e7d91 parsimonator: new package (#5143)
* parsimonator: new package

* adding variant conflict
2017-08-18 09:36:35 -05:00
Audrey Thoma
5fb23b21e0 vegas2: new package (#5150)
* vegas2: new package

* small version things
2017-08-18 09:36:18 -05:00
George Hartzell
9af953a4cb Add info for nextflow@v0.25.6 (#5148) 2017-08-18 09:36:02 -05:00