CEED 1.0 (#7423)

* ceed bundle package including OCCA, MAGMA, Nek5000, mfem, and PETSc

Very preliminary, a lot of ad hoc decisions, fragile, missing functionality
All packages do build on my MacOS laptop with Cuda

Funded-by: ECP
Project: CEED
Time: 4 hours
Reported-by: Tzanio Kolev <kolev1@llnl.gov>

* Some adjustments in a few CEED-related packages.

This is still very preliminary, but with these changes I'm able to build from
scratch on a Mac laptop and a Linux desktop (RHEL7).

Note that there doesn't seem to be a good way to support CUDA in Spack, so
you'll have to install that manually yourself. You will also need a Fortran
compiler, in particular on a Mac where you also have to make sure that it
is being picked up in ~/.spack/darwin/compilers.yaml.

* ceed bundle package including OCCA, MAGMA, Nek5000, mfem, and PETSc

Very preliminary, a lot of ad hoc decisions, fragile, missing functionality
All packages do build on my MacOS laptop with Cuda

Funded-by: ECP
Project: CEED
Time: 4 hours
Reported-by: Tzanio Kolev <kolev1@llnl.gov>

* Reverting Mac-related CUDA checks in the OCCA and MAGMA packages.

A much better solution is to install CUDA manually and add it to
~/.spack/darwin/packages.yaml:

packages:
    cuda:
        paths:
            cuda@8.0.54: /usr/local/cuda

* Fix flake8 warnings

Funded-by: CEED
Project: CEED/Spack

* remove unneeded commented out code

Funded-by: CEED
Project: CEED/Spack

* Don't have PETSc use MUMPS to speedup install time, try parallel installs for Magma

Funded-by: CEED
Project: CEED/Spack

* Added libceed package and made changes to get the ceed metapackage up and running.

* Get the ceed package up and running and added installing the .h file in the libceed package.

* All packages are now pointed at specific versions (either tagged or git hashed.

* Changed some of the specific versions in the CEED metapackage.

* Some version updates.

* Applied Cameron Smith's patch for the latest pumi.

* Merge

* Nek5000: Use tarball instead of git tag

* Major update of the mfem package - needs more testing. Add a testing
shell script for mfem, test_builds.sh, in the mfem directory. It builds
a number of mfem configurations to ensure they all work.

Add 'suite-sparse' variant to petsc.

Add 'develop' version of libceed.

Add 'metis' variant of laghos.

Update the dependencies of ceed.

* Changes in petsc's handling of zlib to make the latest build work on Mac.

* [MFEM] When using '+libunwind', mfem needs '-ldl'.

* [OCCA] Remove debug print statements.

* [PETSc] Replace the check ('zlib' in spec) with ('+zlib' in spec) - the
latter checks if 'zlib' variant of petsc is enabled which is incorrect;
the former checks if petsc is configured to depend on 'zlib'.

* [MFEM] Add conduit variant.

* [libCEED] Enable testing using 'make prove'.

* [MFEM] Tweak test_builds.sh - all builds pass on fedora 27.

* Add a post install test

* Add a post installation check for NekCEM

* pumi: bump git hash, remove unused patch

* Fixed testing of Laghos package

* Adding variants in CEED package

* Empty file for URL didn't work on LLNL/LC. Replacing with bundle-package.txt -> README.md.

* [MFEM] If the spec defines the 'cxxflags' compiler flags, use that value
to set the CXXFLAGS value in the mfem build system.

* Disabling HDF5 by default.

You can still build it with "spack install ceed+hdf5".

* libceed: fix use of prefix versus DESTDIR

* Added cuda variant for libCEED (off by default)

* [libCEED] When building v0.1, fix ceed.pc before installing it.

* [CEED] Add variants for OCCA and PUMI. Replace the hdf5 variant with
a variant called quickbuild that disables variants in packages to
speedup the build - currently hdf5, boost, and superlu-dist in PETSc.

* [libCEED] Remove unused 'import os'.

* [MAGMA] Add version 2.3.0.

* [OCCA] Major update of occa/package.py.

* [libCEED] Several improvements.

* [CEED] Use fixed versions for occa and libceed.

* [OCCA] Fix a bug in CUDA setup.

* NekCEM doesn't need depends_on('python')

* [OCCA,libCEED,MFEM,CEED] Update occa and libceed versions plus a few
tweaks in occa and mfem.

* Switching to gslib-1.0.2 to fix summitdev bug.

* [Nek5000] Use the Fortran 77 compiler instead of the 'fc' compiler.
Propagate any 'fflags' and 'cflags' specifications to the Nek tools.

* [MFEM] Tweak for 'ppc64le'.

* attempt to override fips compatibility error for md5

* Compiler option changes in libCEED to fix summitdev build.

* Disable the nekcem install test until it can be fixed.

* [NekCEM] Major update of the package to fix various issues.

* [NekCEM] Fix formatting

* zoltan: remove non-portable -g0 flag

the pgi compiler does not accept it

* pumi: specify dependency on cmake > 3

* [Nek5000] Add dependency on libx11 when needed. Various hacks to
support IBM XL compilers.

* [NekCEM] Link with the pthreads library - it is required.

* [MAGMA] Add a patch for magma v2.3.0 when building with gcc <= 4.8.
Add dependency on BLAS.

* [CEED] Update to magma 2.3.0. When building with gcc < 4.9 constrain
the used suite-sparse version to <= 5.1.0 - starting with v5.2.0,
suite-sparse requires gcc >= 4.9.

* Small updates

* [libceed, gslib] Fix style

* [Nek5000] Need 'libxt' as a dependency as well.

* [MUMPS] Temporary workaround for ray.

* Updating occa and libceed to laters pre-1.0.0 and pre-0.2 hashes

* petsc: add pkg-config patch for variable quoting

Variable definitions in pkg-config are processed more like make than a
shell, so don't need quoting.  Older versions of pkg-config (<= 0.28)
did not remove the quotes when printing the value.

Freedesktop #67904 (https://bugs.freedesktop.org/show_bug.cgi?id=67904)

Reported-by: Tzanio Kolev <kolev1@llnl.gov>

* petsc: add 3.8.4 and myself as co-maintainer

* Updating libceed to latest pre-0.2 hash.

* [PETSc] Add the 'headers' and 'libs' properties.

* [MFEM] Explicitly add rpaths to link flags for external packages.
This is necessary when MFEM's exported options (in config.mk) are
used outside of Spack for linking against MFEM.

* Unset MFEM_DIR before building MFEM

* [PETSc] As observed by @jedbrown, there is no need to define the 'libs'
property explicitly - the default handler works fine too.

* Build MAGMA shared by default, install a few additional headers

* Forgot a self

* Switching to OCCA tag v1.0.0-alpha.4.

Adding urls for OCCA and libCEED ('spack uninstall' complains otherwise).

* Removed FIXMEs for OCCA tag

* [occa] Cleaned up tagged versions

* [libceed] Updated occa to v1.0.0-alpha.5

* [ceed] Updated occa to v1.0.0-alpha.5

* [libCEED] Always define the 'NDEBUG' makefile option based on the
'debug' variant setting - this should always work regardless of the
default setting inside the libceed makefile.

* [MUMPS] Revert a temporary workaround.

* [CEED, libCEED] Minor tweaks.

* libCEED v0.2 release

* [CEED] Use version 0.2 of libCEED.

* [HPGMG] Remove duplicate version.

* [CEED] Update the hpgmg version

* hpgmg: use tarball for 0.3

The Git repository contains somewhat heavy documentation so the tarballs
are much faster.

* hpgmg: +fe was default for the 0.3 release

* hpgmg: explicitly name build directory to avoid use of ambient PETSC_ARCH

* [HPGMG] Fix flake8 formatting

* libceed@0.2: work around occaFree issue

var/spack/repos/builtin/packages/ceed/README.md

@@ -0,0 +1,3 @@
+This is a bundle for the CEED software suite.
+
+See https://ceed.exascaleproject.org for details.

var/spack/repos/builtin/packages/ceed/package.py

@@ -0,0 +1,111 @@
+##############################################################################
+# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC.
+# Produced at the Lawrence Livermore National Laboratory.
+#
+# This file is part of Spack.
+# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
+# LLNL-CODE-647188
+#
+# For details, see https://github.com/llnl/spack
+# Please also see the LICENSE file for our notice and the LGPL.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License (as
+# published by the Free Software Foundation) version 2.1, February 1999.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
+# conditions of the GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+##############################################################################
+from spack import *
+import os
+
+
+class Ceed(Package):
+    """Ceed is a collection of benchmarks, miniapps, software libraries and
+       APIs for efficient high-order finite element and spectral element
+       discretizations for exascale applications developed in the Department of
+       Energy (DOE) and partially supported by the Exascale Computing Project
+       (ECP). This is a Spack bundle package that installs the CEED software
+       components."""
+
+    homepage = "https://ceed.exascaleproject.org"
+
+    url  = 'file://' + os.path.dirname(__file__) + '/README.md'
+    sha1 = 'b2eefd95c09ba573f663a761b84811a2d7e39788'
+
+    version('1.0.0', sha1, expand=False)
+
+    variant('cuda', default=False,
+            description='Build MAGMA; enable CUDA support in libCEED and OCCA')
+    variant('mfem', default=True, description='Build MFEM and Laghos')
+    variant('nek', default=True,
+            description='Build Nek5000, GSLIB, Nekbone, and NekCEM')
+    variant('occa', default=True,
+            description='Build OCCA; enable OCCA support in libCEED')
+    variant('petsc', default=True,
+            description='Build PETSc and HPGMG')
+    variant('pumi', default=True,
+            description='Build PUMI')
+    variant('quickbuild', default=True,
+            description='Speed-up the build by disabling variants in packages')
+    # TODO: Add 'int64' variant?
+
+    # LibCEED
+    depends_on('libceed@0.2~cuda', when='@1.0.0~cuda')
+    depends_on('libceed@0.2+cuda', when='@1.0.0+cuda')
+    depends_on('libceed@0.2+occa', when='@1.0.0+occa')
+    depends_on('libceed@0.2~occa', when='@1.0.0~occa')
+
+    # OCCA
+    depends_on('occa@v1.0.0-alpha.5~cuda', when='@1.0.0+occa~cuda')
+    depends_on('occa@v1.0.0-alpha.5+cuda', when='@1.0.0+occa+cuda')
+
+    # Nek5000, GSLIB, Nekbone, and NekCEM
+    depends_on('nek5000@17.0', when='@1.0.0+nek')
+    depends_on('gslib@1.0.2', when='@1.0.0+nek')
+    depends_on('nekbone@17.0', when='@1.0.0+nek')
+    depends_on('nekcem@0b8bedd', when='@1.0.0+nek')
+
+    # PETSc, HPGMG
+    # For a +quickbuild we disable boost, hdf5, and superlu-dist in PETSc.
+    # Ideally, these can be turned into recommendations to Spack for
+    # concretizing the PETSc spec, if Spack ever supports recommendations.
+    depends_on('petsc@3.8.3~boost~hdf5~superlu-dist',
+               when='@1.0.0+petsc+quickbuild')
+    depends_on('petsc@3.8.3+mpi+double~int64', when='@1.0.0+petsc~mfem')
+    # The mfem petsc examples need the petsc variants +hypre, +suite-sparse,
+    # and +mumps:
+    depends_on('petsc@3.8.3+mpi+hypre+suite-sparse+mumps+double~int64',
+               when='@1.0.0+petsc+mfem')
+    depends_on('hpgmg@a0a5510df23b+fe', when='@1.0.0+petsc')
+
+    # MAGMA
+    depends_on('magma@2.3.0', when='@1.0.0+cuda')
+
+    # PUMI
+    depends_on('pumi@2.1.0', when='@1.0.0+pumi')
+
+    # MFEM, Laghos
+    depends_on('mfem@3.3.2+mpi+examples+miniapps', when='@1.0.0+mfem~petsc')
+    depends_on('mfem@3.3.2+mpi+petsc+examples+miniapps',
+               when='@1.0.0+mfem+petsc')
+    depends_on('laghos@1.0', when='@1.0.0+mfem')
+    # The next line seems to be necessary because the concretizer somehow
+    # decides that mfem requires 'hypre+internal-superlu' even though the mfem
+    # package lists simply 'hypre' as dependency. This is only an issue because
+    # petsc explicitly requires 'hypre~internal-superlu' which for the
+    # concretizer is a conflict.
+    depends_on('hypre~internal-superlu', when='@1.0.0+mfem')
+
+    # If using gcc version <= 4.8 build suite-sparse version <= 5.1.0
+    depends_on('suite-sparse@:5.1.0', when='@1.0.0%gcc@:4.8+mfem+petsc')
+
+    # Dummy install
+    def install(self, spec, prefix):
+        install('README.md', prefix)

var/spack/repos/builtin/packages/gslib/package.py

@@ -31,10 +31,9 @@ class Gslib(Package):
     homepage = "https://github.com/gslib/gslib"
     url      = "https://github.com/gslib/gslib"
 
-    version('1.0.1', git='https://github.com/gslib/gslib.git',
+    version('1.0.2', git='https://github.com/gslib/gslib.git', tag='v1.0.2')
-        commit='d16685f24551b7efd69e58d96dc76aec75239ea3')
+    version('1.0.1', git='https://github.com/gslib/gslib.git', tag='v1.0.1')
-    version('1.0.0', git='https://github.com/gslib/gslib.git',
+    version('1.0.0', git='https://github.com/gslib/gslib.git', tag='v1.0.0')
-        commit='9533e652320a3b26a72c36487ae265b02072cd48')
 
     variant('mpi', default=True, description='Build with MPI')
     variant('mpiio', default=True, description='Build with MPI I/O')
@@ -85,6 +84,6 @@ def install(self, spec, prefix):
                 mkdir(prefix.lib)
                 install(libname, prefix.lib)
 
-        # Should only install the headers (this will
+        # Should only install the headers (this will be fixed in gslib on
-        # be fixed in gslib on future releases.
+        # future releases).
         install_tree(srcDir, prefix.include)

var/spack/repos/builtin/packages/hpgmg/package.py

@@ -26,22 +26,22 @@
 
 
 class Hpgmg(Package):
-    """HPGMG implements full multigrid (FMG) algorithms using
+    """HPGMG implements full multigrid (FMG) algorithms using finite-volume and
-    finite-volume and finite-element methods.
+       finite-element methods.  Different algorithmic variants adjust the
-    Different algorithmic variants adjust the arithmetic intensity
+       arithmetic intensity and architectural properties that are tested. These
-    and architectural properties that are tested. These FMG methods
+       FMG methods converge up to discretization error in one F-cycle, thus may
-    converge up to discretization error in one F-cycle,
+       be considered direct solvers. An F-cycle visits the finest level a total
-    thus may be considered direct solvers. An F-cycle visits
+       of two times, the first coarsening (8x smaller) 4 times, the second
-    the finest level a total of two times,
+       coarsening 6 times, etc.
-    the first coarsening (8x smaller) 4 times,
+    """
-    the second coarsening 6 times, etc."""
 
     homepage = "https://bitbucket.org/hpgmg/hpgmg"
     url      = "https://hpgmg.org/static/hpgmg-0.tar.gz"
     tags     = ['proxy-app']
 
-    version('a0a5510df23b', 'b9c50f25e541428d4735fb07344d1d0ed9fc821bdde918d8e0defa78c0d9b4f9')
     version('develop', git='https://bitbucket.org/hpgmg/hpgmg.git', branch='master')
+    version('a0a5510df23b', 'b9c50f25e541428d4735fb07344d1d0ed9fc821bdde918d8e0defa78c0d9b4f9')
+    version('0.3',          '12a65da216fec91daea78594ae4b5a069c8f1a700f1ba21eed9f45a79a68c793')
 
     variant(
         'fe', default=False, description='Build finite element solver')
@@ -61,7 +61,7 @@ class Hpgmg(Package):
 
     def configure_args(self):
         args = []
-        if '+fe' in self.spec:
+        if '+fe' in self.spec and not ('@0.3' in self.spec):
             args.append('--fe')
 
         if 'fv=serial' in self.spec:
@@ -74,7 +74,9 @@ def configure_args(self):
         if 'fv=none' in self.spec:
             args.append('--no-fv')
         else:
-            cflags.append(self.compiler.openmp_flag)
+            # Apple's Clang doesn't support OpenMP
+            if not (self.spec.satisfies('%clang') and self.compiler.is_apple):
+                cflags.append(self.compiler.openmp_flag)
 
         if '+debug' in self.spec:
             cflags.append('-g')
@@ -83,13 +85,17 @@ def configure_args(self):
 
         args.append('--CFLAGS=' + ' '.join(cflags))
 
+        # Explicitly specify the build directory to ensure that
+        # PETSC_ARCH is not used from the environment.
+        args.append('--arch=build')
+
         return args
 
     def configure(self, spec, prefix):
         configure(*self.configure_args())
 
     def build(self, spec, prefix):
-        make()
+        make('-C', 'build')
 
     def install(self, spec, prefix):
         install_tree('build/bin', prefix.bin)

var/spack/repos/builtin/packages/laghos/package.py

@@ -40,8 +40,12 @@ class Laghos(MakefilePackage):
     version('1.0', '4c091e115883c79bed81c557ef16baff')
     version('develop', git=git, branch='master')
 
-    depends_on('mpi')
+    variant('metis', default=True, description='Enable/disable METIS support')
-    depends_on('mfem@laghos-v1.0', when='@1.0')
+
+    depends_on('mfem@develop+mpi+metis', when='@develop+metis')
+    depends_on('mfem@develop+mpi~metis', when='@develop~metis')
+    depends_on('mfem@laghos-v1.0,3.3.2:+mpi+metis', when='@1.0+metis')
+    depends_on('mfem@laghos-v1.0,3.3.2:+mpi~metis', when='@1.0~metis')
 
     @property
     def build_targets(self):
@@ -49,14 +53,24 @@ def build_targets(self):
         spec = self.spec
 
         targets.append('MFEM_DIR=%s' % spec['mfem'].prefix)
-        targets.append('CONFIG_MK=%s' % join_path(spec['mfem'].prefix,
+        targets.append('CONFIG_MK=%s' % spec['mfem'].package.config_mk)
-                       'share/mfem/config.mk'))
+        targets.append('TEST_MK=%s' % spec['mfem'].package.test_mk)
-        targets.append('TEST_MK=%s' % join_path(spec['mfem'].prefix,
-                       'share/mfem/test.mk'))
         targets.append('CXX=%s' % spec['mpi'].mpicxx)
 
         return targets
 
+    # See lib/spack/spack/build_systems/makefile.py
+    def check(self):
+        targets = []
+        spec = self.spec
+
+        targets.append('MFEM_DIR=%s' % spec['mfem'].prefix)
+        targets.append('CONFIG_MK=%s' % spec['mfem'].package.config_mk)
+        targets.append('TEST_MK=%s' % spec['mfem'].package.test_mk)
+
+        with working_dir(self.build_directory):
+            make('test', *targets)
+
     def install(self, spec, prefix):
         mkdirp(prefix.bin)
         install('laghos', prefix.bin)

var/spack/repos/builtin/packages/libceed/occaFree-0.2.diff

@@ -0,0 +1,41 @@
+diff --git a/backends/occa/ceed-occa-basis.c b/backends/occa/ceed-occa-basis.c
+index 85ec292..86a46ee 100644
+--- a/backends/occa/ceed-occa-basis.c
++++ b/backends/occa/ceed-occa-basis.c
+@@ -293,10 +293,6 @@ static int CeedBasisDestroy_Occa(CeedBasis basis) {
+   const Ceed ceed = basis->ceed;
+   CeedBasis_Occa *data = basis->data;
+   dbg("[CeedBasis][Destroy]");
+-  occaFree(data->kZero);
+-  occaFree(data->kInterp);
+-  occaFree(data->kGrad);
+-  occaFree(data->kWeight);
+   occaFree(data->qref1d);
+   occaFree(data->qweight1d);
+   occaFree(data->interp1d);
+diff --git a/backends/occa/ceed-occa-qfunction.c b/backends/occa/ceed-occa-qfunction.c
+index a2776c3..abf7de0 100644
+--- a/backends/occa/ceed-occa-qfunction.c
++++ b/backends/occa/ceed-occa-qfunction.c
+@@ -154,7 +154,6 @@ static int CeedQFunctionDestroy_Occa(CeedQFunction qf) {
+   CeedQFunction_Occa *data=qf->data;
+   free(data->oklPath);
+   dbg("[CeedQFunction][Destroy]");
+-  occaFree(data->kQFunctionApply);
+   if (data->ready) {
+     if (!data->op) occaFree(data->d_q);
+     occaFree(data->d_u);
+diff --git a/backends/occa/ceed-occa-restrict.c b/backends/occa/ceed-occa-restrict.c
+index 6b7786c..c5360dc 100644
+--- a/backends/occa/ceed-occa-restrict.c
++++ b/backends/occa/ceed-occa-restrict.c
+@@ -95,9 +95,6 @@ static int CeedElemRestrictionDestroy_Occa(CeedElemRestriction r) {
+   const Ceed ceed = r->ceed;
+   CeedElemRestriction_Occa *data = r->data;
+   dbg("[CeedElemRestriction][Destroy]");
+-  for (int i=0; i<9; i++) {
+-    occaFree(data->kRestrict[i]);
+-  }
+   ierr = CeedFree(&data); CeedChk(ierr);
+   return 0;
+ }

var/spack/repos/builtin/packages/libceed/package.py

@@ -0,0 +1,81 @@
+##############################################################################
+# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC.
+# Produced at the Lawrence Livermore National Laboratory.
+#
+# This file is part of Spack.
+# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
+# LLNL-CODE-647188
+#
+# For details, see https://github.com/llnl/spack
+# Please also see the LICENSE file for our notice and the LGPL.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License (as
+# published by the Free Software Foundation) version 2.1, February 1999.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
+# conditions of the GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+##############################################################################
+from spack import *
+
+
+class Libceed(Package):
+    """The CEED API Library: Code for Efficient Extensible Discretizations
+    """
+
+    homepage = "https://github.com/CEED/libCEED"
+    url      = 'https://github.com/CEED/libCEED'
+
+    version('develop', git='https://github.com/CEED/libCEED.git',
+            branch='master')
+    version('0.2', git='https://github.com/CEED/libCEED.git', tag='v0.2')
+    version('0.1', git='https://github.com/CEED/libCEED.git', tag='v0.1')
+
+    variant('occa', default=True, description='Enable OCCA backends')
+    variant('cuda', default=False, description='Enable CUDA support')
+    variant('debug', default=False, description='Enable debug build')
+
+    depends_on('occa@v1.0.0-alpha.5,develop', when='+occa')
+    depends_on('occa@develop', when='@develop+occa')
+    depends_on('occa+cuda', when='+occa+cuda')
+    depends_on('occa~cuda', when='+occa~cuda')
+
+    # occa: do not occaFree kernels
+    # Repeated creation and freeing of kernels appears to expose a caching
+    # bug in Occa.
+    patch('occaFree-0.2.diff', when='@0.2')
+
+    phases = ['build', 'install']
+
+    def build(self, spec, prefix):
+        # Note: The occa package exports OCCA_DIR in the environment
+
+        makeopts = ['V=1']
+        makeopts += ['NDEBUG=%s' % ('' if '+debug' in spec else '1')]
+        make(*makeopts)
+
+        if self.run_tests:
+            make('prove', *makeopts, parallel=False)
+
+    def install(self, spec, prefix):
+        make('install', 'prefix=%s' % prefix, parallel=False)
+
+    @when('@0.1')
+    def install(self, spec, prefix):
+        mkdirp(prefix.include)
+        install('ceed.h', prefix.include)
+        mkdirp(prefix.lib)
+        install('libceed.%s' % dso_suffix, prefix.lib)
+        filter_file('^prefix=.*$', 'prefix=%s' % prefix, 'ceed.pc')
+        filter_file('^includedir=\$\{prefix\}$',
+                    'includedir=${prefix}/include', 'ceed.pc')
+        filter_file('^libdir=\$\{prefix\}$', 'libdir=${prefix}/lib', 'ceed.pc')
+        filter_file('Version:.*$', 'Version: 0.1', 'ceed.pc')
+        mkdirp(prefix.lib.pkgconfig)
+        install('ceed.pc', prefix.lib.pkgconfig)

var/spack/repos/builtin/packages/magma/magma-2.3.0-gcc-4.8.patch

@@ -0,0 +1,24 @@
+diff -ru magma-2.3.0/testing/testings.h magma-2.3.0-patched/testing/testings.h
+--- magma-2.3.0/testing/testings.h	2017-11-14 21:34:00.000000000 -0800
++++ magma-2.3.0-patched/testing/testings.h	2018-03-23 20:41:16.459934643 -0700
+@@ -269,4 +269,20 @@
+     typename blas::traits<FloatT>::real_t* sigma,
+     FloatT* A, magma_int_t lda );
+ 
++// This overload for the case sigma = nullptr is a workaround for an issue
++// when building with gcc 4.8.5. This is not an issue with gcc 4.9.2.
++template< typename FloatT >
++void magma_generate_matrix(
++    magma_opts& opts,
++    magma_int_t m, magma_int_t n,
++    std::nullptr_t sigma,
++    FloatT* A, magma_int_t lda )
++{
++    magma_generate_matrix<FloatT>(
++        opts,
++        m, n,
++        (typename blas::traits<FloatT>::real_t*) sigma,
++        A, lda );
++}
++
+ #endif /* TESTINGS_H */

var/spack/repos/builtin/packages/magma/package.py

@@ -22,28 +22,37 @@
 # License along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
+
 from spack import *
 
 
 class Magma(CMakePackage):
-    """The MAGMA project aims to develop a dense linear algebra library
+    """The MAGMA project aims to develop a dense linear algebra library similar to
-    similar to LAPACK but for heterogeneous/hybrid architectures,
+       LAPACK but for heterogeneous/hybrid architectures, starting with current
-    starting with current "Multicore+GPU" systems.
+       "Multicore+GPU" systems.
     """
 
     homepage = "http://icl.cs.utk.edu/magma/"
     url = "http://icl.cs.utk.edu/projectsfiles/magma/downloads/magma-2.2.0.tar.gz"
 
+    version('2.3.0', '9aaf85a338d3a17303e0c69f86f0ec52')
     version('2.2.0', '6c1ebf4cdf63eb302ff6258ff8c49217')
 
     variant('fortran', default=True,
             description='Enable Fortran bindings support')
+    variant('shared', default=True,
+            description='Enable shared library')
 
+    depends_on('blas')
     depends_on('lapack')
-    depends_on('cuda@9.0:', when='%gcc@6.0:6.9.9')
+    depends_on('cuda')
-    depends_on('cuda@8.0:', when='%gcc@5.0:')
+
+    conflicts('%gcc@6:', when='^cuda@:8')
+    conflicts('%gcc@7:', when='^cuda@:9')
+
     patch('ibm-xl.patch', when='@2.2:%xl')
     patch('ibm-xl.patch', when='@2.2:%xl_r')
+    patch('magma-2.3.0-gcc-4.8.patch', when='@2.3.0%gcc@:4.8')
 
     def cmake_args(self):
         spec = self.spec
@@ -52,10 +61,17 @@ def cmake_args(self):
         options.extend([
             '-DCMAKE_INSTALL_PREFIX=%s' % prefix,
             '-DCMAKE_INSTALL_NAME_DIR:PATH=%s/lib' % prefix,
-            '-DLAPACK_LIBRARIES=%s;%s' % (spec['blas'].libs,
+            '-DBLAS_LIBRARIES=%s' % spec['blas'].libs.joined(';'),
-                                          spec['lapack'].libs)
+            # As of MAGMA v2.3.0, CMakeLists.txt does not use the variable
+            # BLAS_LIBRARIES, but only LAPACK_LIBRARIES, so we need to
+            # explicitly add blas to LAPACK_LIBRARIES.
+            '-DLAPACK_LIBRARIES=%s' %
+            (spec['lapack'].libs + spec['blas'].libs).joined(';')
         ])
 
+        options += ['-DBUILD_SHARED_LIBS=%s' %
+                    ('ON' if ('+shared' in spec) else 'OFF')]
+
         if '+fortran' in spec:
             options.extend([
                 '-DUSE_FORTRAN=yes'
@@ -66,8 +82,16 @@ def cmake_args(self):
                 ])
 
         if spec.satisfies('^cuda@9.0:'):
-            options.extend([
+            if '@:2.2.0' in spec:
-                '-DGPU_TARGET=sm30'
+                options.extend(['-DGPU_TARGET=sm30'])
-            ])
+            else:
+                options.extend(['-DGPU_TARGET=sm_30'])
 
         return options
+
+    @run_after('install')
+    def post_install(self):
+        install('magmablas/atomics.cuh', self.prefix.include)
+        install('control/magma_threadsetting.h', self.prefix.include)
+        install('control/pthread_barrier.h', self.prefix.include)
+        install('control/magma_internal.h', self.prefix.include)

var/spack/repos/builtin/packages/mfem/package.py

@@ -22,22 +22,31 @@
 # License along with this program; if not, write to the Free Software
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
-import re
 from spack import *
+import os
+import shutil
 
 
 class Mfem(Package):
     """Free, lightweight, scalable C++ library for finite element methods."""
 
+    tags = ['FEM', 'finite elements', 'high-order', 'AMR', 'HPC']
+
     homepage = 'http://www.mfem.org'
     url      = 'https://github.com/mfem/mfem'
 
+    maintainers = ['goxberry', 'tzanio', 'markcmiller86', 'acfisher',
+                   'v-dobrev']
+
+    # Recommended mfem builds to test when updating this file: see the shell
+    # script 'test_builds.sh' in the same directory as this file.
+
     # mfem is downloaded from a URL shortener at request of upstream
     # author Tzanio Kolev <tzanio@llnl.gov>.  See here:
     #     https://github.com/mfem/mfem/issues/53
     #
     # The following procedure should be used to verify security when a
-    # new verison is added:
+    # new version is added:
     #
     # 1. Verify that no checksums on old versions have changed.
     #
@@ -51,6 +60,11 @@ class Mfem(Package):
     # If this quick verification procedure fails, additional discussion
     # will be required to verify the new version.
 
+    # 'develop' is a special version that is always larger (or newer) than any
+    # other version.
+    version('develop',
+            git='https://github.com/mfem/mfem', branch='master')
+
     version('3.3.2',
             '01a762a5d0a2bc59ce4e2f59009045a4',
             url='https://goo.gl/Kd7Jk8', extension='.tar.gz',
@@ -71,8 +85,19 @@ class Mfem(Package):
             '841ea5cf58de6fae4de0f553b0e01ebaab9cd9c67fa821e8a715666ecf18fc57',
             url='http://goo.gl/xrScXn', extension='.tar.gz')
 
+    variant('static', default=True,
+        description='Build static library')
+    variant('shared', default=False,
+        description='Build shared library')
     variant('mpi', default=True,
         description='Enable MPI parallelism')
+    # Can we make the default value for 'metis' to depend on the 'mpi' value?
+    variant('metis', default=True,
+        description='Enable METIS support')
+    # TODO: The 'hypre' variant is the same as 'mpi', we may want to remove it.
+    #       For now, keep the 'hypre' variant while ignoring its setting. This
+    #       is done to preserve compatibility with other packages that refer to
+    #       it, e.g. xSDK.
     variant('hypre', default=True,
         description='Required for MPI parallelism')
     variant('openmp', default=False,
@@ -83,10 +108,13 @@ class Mfem(Package):
             ' May cause minor performance issues.'))
     variant('superlu-dist', default=False,
         description='Enable MPI parallel, sparse direct solvers')
+    # Placeholder for STRUMPACK, support added in mfem v3.3.2:
+    # variant('strumpack', default=False,
+    #     description='Enable support for STRUMPACK')
     variant('suite-sparse', default=False,
         description='Enable serial, sparse direct solvers')
     variant('petsc', default=False,
-        description='Enable PETSc solvers, preconditioners, etc..')
+        description='Enable PETSc solvers, preconditioners, etc.')
     variant('sundials', default=False,
         description='Enable Sundials time integrators')
     variant('mpfr', default=False,
@@ -97,43 +125,77 @@ class Mfem(Package):
         description='Build debug instead of optimized version')
     variant('netcdf', default=False,
         description='Enable Cubit/Genesis reader')
+    variant('conduit', default=False,
+        description='Enable binary data I/O using Conduit')
     variant('gzstream', default=True,
         description='Support zip\'d streams for I/O')
+    variant('gnutls', default=False,
+        description='Enable secure sockets using GnuTLS')
+    variant('libunwind', default=False,
+        description='Enable backtrace on error support using Libunwind')
+    variant('timer', default='auto',
+        values=('auto', 'std', 'posix', 'mac', 'mpi'),
+        description='Timing functions to use in mfem::StopWatch')
     variant('examples', default=False,
         description='Build and install examples')
     variant('miniapps', default=False,
         description='Build and install miniapps')
 
-    conflicts('+mpi', when='~hypre')
+    conflicts('+shared', when='@:3.3.2')
-    conflicts('+suite-sparse', when='~lapack')
+    conflicts('~static~shared')
-    conflicts('+superlu-dist', when='@:3.1')
+    conflicts('~threadsafe', when='+openmp')
-    conflicts('+netcdf', when='@:3.1')
 
-    depends_on('hypre', when='+hypre')
+    conflicts('+netcdf', when='@:3.1')
-    depends_on('blas', when='+lapack')
+    conflicts('+superlu-dist', when='@:3.1')
-    depends_on('blas', when='+suite-sparse')
+    conflicts('+gnutls', when='@:3.1')
-    depends_on('lapack', when='+lapack')
+    conflicts('+gzstream', when='@:3.2')
-    depends_on('lapack', when='+suite-sparse')
+    conflicts('+mpfr', when='@:3.2')
+    conflicts('+petsc', when='@:3.2')
+    conflicts('+sundials', when='@:3.2')
+    conflicts('timer=mac', when='@:3.3.0')
+    conflicts('timer=mpi', when='@:3.3.0')
+    conflicts('~metis+mpi', when='@:3.3.0')
+    conflicts('+metis~mpi', when='@:3.3.0')
+    conflicts('+conduit', when='@:3.3.2')
+
+    conflicts('+superlu-dist', when='~mpi')
+    conflicts('+petsc', when='~mpi')
+    conflicts('timer=mpi', when='~mpi')
 
     depends_on('mpi', when='+mpi')
-    depends_on('metis')
+    depends_on('hypre', when='+mpi')
-    depends_on('parmetis', when='+superlu-dist')
+    depends_on('metis', when='+metis')
-    depends_on('metis@5:', when='+superlu-dist')
+    depends_on('blas', when='+lapack')
-    depends_on('metis@5:', when='+suite-sparse ^suite-sparse@4.5:')
+    depends_on('lapack', when='+lapack')
 
-    depends_on('sundials@2.7:+hypre', when='+sundials')
+    depends_on('sundials@2.7.0', when='@:3.3.0+sundials~mpi')
+    depends_on('sundials@2.7.0+mpi+hypre', when='@:3.3.0+sundials+mpi')
+    depends_on('sundials@2.7.0:', when='@3.3.2:+sundials~mpi')
+    depends_on('sundials@2.7.0:+mpi+hypre', when='@3.3.2:+sundials+mpi')
     depends_on('suite-sparse', when='+suite-sparse')
-    depends_on('superlu-dist', when='@3.2: +superlu-dist')
+    depends_on('superlu-dist', when='+superlu-dist')
-    depends_on('petsc@3.8:', when='+petsc')
+    # The PETSc tests in MFEM will fail if PETSc is not configured with
-
+    # SuiteSparse and MUMPS. On the other hand, if we require the variants
+    # '+suite-sparse+mumps' of PETSc, the xsdk package concretization fails.
+    depends_on('petsc@3.8:+mpi+double+hypre', when='+petsc')
+    # Recommended when building outside of xsdk:
+    # depends_on('petsc@3.8:+mpi+double+hypre+suite-sparse+mumps',
+    #            when='+petsc')
     depends_on('mpfr', when='+mpfr')
-    depends_on('netcdf', when='@3.2: +netcdf')
+    depends_on('netcdf', when='+netcdf')
-    depends_on('zlib', when='@3.2: +netcdf')
+    depends_on('libunwind', when='+libunwind')
-    depends_on('hdf5', when='@3.2: +netcdf')
-    depends_on('libunwind', when='+debug')
     depends_on('zlib', when='+gzstream')
+    depends_on('gnutls', when='+gnutls')
+    depends_on('conduit@0.3.1:', when='+conduit')
+    depends_on('conduit+mpi', when='+conduit+mpi')
 
-    patch('mfem_ppc_build.patch', when='@3.2:3.3 arch=ppc64le')
+    patch('mfem_ppc_build.patch', when='@3.2:3.3.0 arch=ppc64le')
+
+    phases = ['configure', 'build', 'install']
+
+    def setup_environment(self, spack_env, run_env):
+        spack_env.unset('MFEM_DIR')
+        spack_env.unset('MFEM_BUILD_DIR')
 
     #
     # Note: Although MFEM does support CMake configuration, MFEM
@@ -142,30 +204,42 @@ class Mfem(Package):
     # likely to be up to date in supporting *all* of MFEM's
     # configuration options. So, don't use CMake
     #
-    def install(self, spec, prefix):
+    def configure(self, spec, prefix):
 
         def yes_no(varstr):
             return 'YES' if varstr in self.spec else 'NO'
 
-        metis5_str = 'NO'
+        # We need to add rpaths explicitly to allow proper export of link flags
-        if '+superlu-dist' in spec or  \
+        # from within MFEM.
-            spec.satisfies('+suite-sparse ^suite-sparse@4.5:') or \
-                spec['metis'].satisfies('@5:'):
-                metis5_str = 'YES'
 
-        threadsafe_str = 'NO'
+        # Similar to spec[pkg].libs.ld_flags but prepends rpath flags too.
-        if '+openmp' in spec or '+threadsafe' in spec:
+        def ld_flags_from_LibraryList(libs_list):
-            threadsafe_str = 'YES'
+            flags = ['-Wl,-rpath,%s' % dir for dir in libs_list.directories]
+            flags += [libs_list.ld_flags]
+            return ' '.join(flags)
+
+        def ld_flags_from_dirs(pkg_dirs_list, pkg_libs_list):
+            flags = ['-Wl,-rpath,%s' % dir for dir in pkg_dirs_list]
+            flags += ['-L%s' % dir for dir in pkg_dirs_list]
+            flags += ['-l%s' % lib for lib in pkg_libs_list]
+            return ' '.join(flags)
+
+        metis5_str = 'NO'
+        if ('+metis' in spec) and spec['metis'].satisfies('@5:'):
+            metis5_str = 'YES'
 
         options = [
             'PREFIX=%s' % prefix,
             'MFEM_USE_MEMALLOC=YES',
             'MFEM_DEBUG=%s' % yes_no('+debug'),
+            # NOTE: env['CXX'] is the spack c++ compiler wrapper. The real
+            # compiler is defined by env['SPACK_CXX'].
             'CXX=%s' % env['CXX'],
-            'MFEM_USE_LIBUNWIND=%s' % yes_no('+debug'),
+            'MFEM_USE_LIBUNWIND=%s' % yes_no('+libunwind'),
             'MFEM_USE_GZSTREAM=%s' % yes_no('+gzstream'),
+            'MFEM_USE_METIS=%s' % yes_no('+metis'),
             'MFEM_USE_METIS_5=%s' % metis5_str,
-            'MFEM_THREAD_SAFE=%s' % threadsafe_str,
+            'MFEM_THREAD_SAFE=%s' % yes_no('+threadsafe'),
             'MFEM_USE_MPI=%s' % yes_no('+mpi'),
             'MFEM_USE_LAPACK=%s' % yes_no('+lapack'),
             'MFEM_USE_SUPERLU=%s' % yes_no('+superlu-dist'),
@@ -174,115 +248,175 @@ def yes_no(varstr):
             'MFEM_USE_PETSC=%s' % yes_no('+petsc'),
             'MFEM_USE_NETCDF=%s' % yes_no('+netcdf'),
             'MFEM_USE_MPFR=%s' % yes_no('+mpfr'),
-            'MFEM_USE_OPENMP=%s' % yes_no('+openmp')]
+            'MFEM_USE_GNUTLS=%s' % yes_no('+gnutls'),
+            'MFEM_USE_OPENMP=%s' % yes_no('+openmp'),
+            'MFEM_USE_CONDUIT=%s' % yes_no('+conduit')]
+
+        cxxflags = spec.compiler_flags['cxxflags']
+        if cxxflags:
+            # The cxxflags are set by the spack c++ compiler wrapper. We also
+            # set CXXFLAGS explicitly, for clarity, and to properly export the
+            # cxxflags in the variable MFEM_CXXFLAGS in config.mk.
+            options += ['CXXFLAGS=%s' % ' '.join(cxxflags)]
+
+        if '~static' in spec:
+            options += ['STATIC=NO']
+        if '+shared' in spec:
+            options += ['SHARED=YES', 'PICFLAG=%s' % self.compiler.pic_flag]
 
         if '+mpi' in spec:
             options += ['MPICXX=%s' % spec['mpi'].mpicxx]
-
-        if '+hypre' in spec:
             hypre = spec['hypre']
-            hypre_flag_list = (hypre.libs +
+            # The hypre package always links with 'blas' and 'lapack'.
-                               hypre['lapack'].libs +
+            all_hypre_libs = hypre.libs + hypre['lapack'].libs + \
-                               hypre['blas'].libs)
+                hypre['blas'].libs
-            options += ['HYPRE_DIR=%s' % hypre.prefix,
+            options += [
-                        'HYPRE_OPT=-I%s' % hypre.prefix.include,
+                'HYPRE_OPT=-I%s' % hypre.prefix.include,
-                        'HYPRE_LIB=%s' % hypre_flag_list.ld_flags]
+                'HYPRE_LIB=%s' % ld_flags_from_LibraryList(all_hypre_libs)]
+
+        if '+metis' in spec:
+            options += [
+                'METIS_OPT=-I%s' % spec['metis'].prefix.include,
+                'METIS_LIB=%s' %
+                ld_flags_from_dirs([spec['metis'].prefix.lib], ['metis'])]
 
         if '+lapack' in spec:
-            lapack_lib = (spec['lapack'].libs + spec['blas'].libs).ld_flags  # NOQA: ignore=E501
+            lapack_blas = spec['lapack'].libs + spec['blas'].libs
             options += [
-                'LAPACK_OPT=-I%s' % spec['lapack'].prefix.include,
+                # LAPACK_OPT is not used
-                'LAPACK_LIB=%s' % lapack_lib]
+                'LAPACK_LIB=%s' % ld_flags_from_LibraryList(lapack_blas)]
 
         if '+superlu-dist' in spec:
-            metis_lib = '-L%s -lparmetis -lmetis' % spec['parmetis'].prefix.lib
             options += [
-                'METIS_DIR=%s' % spec['parmetis'].prefix,
+                'SUPERLU_OPT=-I%s -I%s' %
-                'METIS_OPT=-I%s' % spec['parmetis'].prefix.include,
+                (spec['superlu-dist'].prefix.include,
-                'METIS_LIB=%s' % metis_lib]
+                 spec['parmetis'].prefix.include),
-            superlu_lib = '-L%s' % spec['superlu-dist'].prefix.lib
+                'SUPERLU_LIB=-L%s -L%s -lsuperlu_dist -lparmetis' %
-            superlu_lib += ' -lsuperlu_dist'
+                (spec['superlu-dist'].prefix.lib,
-            options += [
+                 spec['parmetis'].prefix.lib)]
-                'SUPERLU_DIR=%s' % spec['superlu-dist'].prefix,
-                'SUPERLU_OPT=-I%s' % spec['superlu-dist'].prefix.include,
-                'SUPERLU_LIB=%s' % superlu_lib]
-        else:
-            metis_lib = '-L%s -lmetis' % spec['metis'].prefix.lib
-            options += [
-                'METIS_DIR=%s' % spec['metis'].prefix,
-                'METIS_OPT=-I%s' % spec['metis'].prefix.include,
-                'METIS_LIB=%s' % metis_lib]
 
         if '+suite-sparse' in spec:
-            ssp = spec['suite-sparse'].prefix
+            ss_spec = 'suite-sparse:' + self.suitesparse_components
-            ss_lib = '-L%s' % ssp.lib
-            if '@3.2:' in spec:
-                ss_lib += ' -lklu -lbtf'
-            ss_lib += (' -lumfpack -lcholmod -lcolamd' +
-                       ' -lamd -lcamd -lccolamd -lsuitesparseconfig')
-            no_rt = spec.satisfies('platform=darwin')
-            if not no_rt:
-                ss_lib += ' -lrt'
-            ss_lib += (' ' + metis_lib + ' ' + lapack_lib)
             options += [
-                'SUITESPARSE_DIR=%s' % ssp,
+                'SUITESPARSE_OPT=-I%s' % spec[ss_spec].prefix.include,
-                'SUITESPARSE_OPT=-I%s' % ssp.include,
+                'SUITESPARSE_LIB=%s' %
-                'SUITESPARSE_LIB=%s' % ss_lib]
+                ld_flags_from_LibraryList(spec[ss_spec].libs)]
 
         if '+sundials' in spec:
-            sundials_libs = (
+            sun_spec = 'sundials:' + self.sundials_components
-                '-lsundials_arkode -lsundials_cvode'
-                ' -lsundials_nvecserial -lsundials_kinsol')
-            if '+mpi' in spec:
-                sundials_libs += (
-                    ' -lsundials_nvecparhyp -lsundials_nvecparallel')
             options += [
-                'SUNDIALS_DIR=%s' % spec['sundials'].prefix,
+                'SUNDIALS_OPT=%s' % spec[sun_spec].headers.cpp_flags,
-                'SUNDIALS_OPT=-I%s' % spec['sundials'].prefix.include,
+                'SUNDIALS_LIB=%s' %
-                'SUNDIALS_LIB=-L%s %s' % (spec['sundials'].prefix.lib,
+                ld_flags_from_LibraryList(spec[sun_spec].libs)]
-                                          sundials_libs)]
 
         if '+petsc' in spec:
-            f = open('%s/lib/pkgconfig/PETSc.pc' % spec['petsc'].prefix, 'r')
+            # options += ['PETSC_DIR=%s' % spec['petsc'].prefix]
-            for line in f:
-                if re.search('^\s*Cflags: ', line):
-                    petsc_opts = re.sub('^\s*Cflags: (.*)', '\\1', line)
-                elif re.search('^\s*Libs.*: ', line):
-                    petsc_libs = re.sub('^\s*Libs.*: (.*)', '\\1', line)
-            f.close()
             options += [
-                'PETSC_DIR=%s' % spec['petsc'].prefix,
+                'PETSC_OPT=%s' % spec['petsc'].headers.cpp_flags,
-                'PETSC_OPT=%s' % petsc_opts,
+                'PETSC_LIB=%s' %
-                'PETSC_LIB=-L%s -lpetsc %s' %
+                ld_flags_from_LibraryList(spec['petsc'].libs)]
-                (spec['petsc'].prefix.lib, petsc_libs)]
 
         if '+netcdf' in spec:
-            np = spec['netcdf'].prefix
-            zp = spec['zlib'].prefix
-            h5p = spec['hdf5'].prefix
-            nlib = '-L%s -lnetcdf ' % np.lib
-            nlib += '-L%s -lhdf5_hl -lhdf5 ' % h5p.lib
-            nlib += '-L%s -lz' % zp.lib
             options += [
-                'NETCDF_DIR=%s' % np,
+                'NETCDF_OPT=-I%s' % spec['netcdf'].prefix.include,
-                'HDF5_DIR=%s' % h5p,
+                'NETCDF_LIB=%s' %
-                'ZLIB_DIR=%s' % zp,
+                ld_flags_from_dirs([spec['netcdf'].prefix.lib], ['netcdf'])]
-                'NETCDF_OPT=-I%s' % np.include,
+
-                'NETCDF_LIB=%s' % nlib]
+        if '+gzstream' in spec:
+            if "@:3.3.2" in spec:
+                options += ['ZLIB_DIR=%s' % spec['zlib'].prefix]
+            else:
+                options += [
+                    'ZLIB_OPT=-I%s' % spec['zlib'].prefix.include,
+                    'ZLIB_LIB=%s' %
+                    ld_flags_from_LibraryList(spec['zlib'].libs)]
 
         if '+mpfr' in spec:
-            options += ['MPFR_LIB=-L%s -lmpfr' % spec['mpfr'].prefix.lib]
+            options += [
+                'MPFR_OPT=-I%s' % spec['mpfr'].prefix.include,
+                'MPFR_LIB=%s' %
+                ld_flags_from_dirs([spec['mpfr'].prefix.lib], ['mpfr'])]
+
+        if '+gnutls' in spec:
+            options += [
+                'GNUTLS_OPT=-I%s' % spec['gnutls'].prefix.include,
+                'GNUTLS_LIB=%s' %
+                ld_flags_from_dirs([spec['gnutls'].prefix.lib], ['gnutls'])]
+
+        if '+libunwind' in spec:
+            libunwind = spec['libunwind']
+            headers = find_headers('libunwind', libunwind.prefix.include)
+            headers.add_macro('-g')
+            libs = find_libraries('libunwind', libunwind.prefix.lib,
+                                  shared=True, recursive=True)
+            if not libs:
+                libs = find_libraries('libunwind', libunwind.prefix.lib,
+                                      shared=False, recursive=True)
+            # When mfem uses libunwind, it also needs 'libdl'.
+            libs += LibraryList(find_system_libraries('libdl'))
+            options += [
+                'LIBUNWIND_OPT=%s' % headers.cpp_flags,
+                'LIBUNWIND_LIB=%s' % ld_flags_from_LibraryList(libs)]
 
         if '+openmp' in spec:
-            options += ['OPENMP_OPT = %s' % self.compiler.openmp_flag]
+            options += ['OPENMP_OPT=%s' % self.compiler.openmp_flag]
 
-        make('config', *options)
+        timer_ids = {'std': '0', 'posix': '2', 'mac': '4', 'mpi': '6'}
+        timer = spec.variants['timer'].value
+        if timer != 'auto':
+            options += ['MFEM_TIMER_TYPE=%s' % timer_ids[timer]]
+
+        if '+conduit' in spec:
+            conduit = spec['conduit']
+            headers = HeaderList(find(conduit.prefix.include, 'conduit.hpp',
+                                      recursive=True))
+            conduit_libs = ['libconduit', 'libconduit_relay',
+                            'libconduit_blueprint']
+            libs = find_libraries(conduit_libs, conduit.prefix.lib,
+                                  shared=('+shared' in conduit))
+            libs += LibraryList(find_system_libraries('libdl'))
+            if '+hdf5' in conduit:
+                hdf5 = conduit['hdf5']
+                headers += find_headers('hdf5', hdf5.prefix.include)
+                libs += hdf5.libs
+            options += [
+                'CONDUIT_OPT=%s' % headers.cpp_flags,
+                'CONDUIT_LIB=%s' % ld_flags_from_LibraryList(libs)]
+
+        make('config', *options, parallel=False)
+        make('info', parallel=False)
+
+    def build(self, spec, prefix):
         make('lib')
 
-        if self.run_tests:
+    @run_after('build')
-            make('check')
+    def check_or_test(self):
+        # Running 'make check' or 'make test' may fail if MFEM_MPIEXEC or
+        # MFEM_MPIEXEC_NP are not set appropriately.
+        if not self.run_tests:
+            # check we can build ex1 (~mpi) or ex1p (+mpi).
+            make('-C', 'examples', 'ex1p' if ('+mpi' in self.spec) else 'ex1',
+                 parallel=False)
+            # make('check', parallel=False)
+        else:
+            make('all')
+            make('test', parallel=False)
 
-        make('install')
+    def install(self, spec, prefix):
+        make('install', parallel=False)
+
+        # TODO: The way the examples and miniapps are being installed is not
+        # perfect. For example, the makefiles do not work.
+
+        install_em = ('+examples' in spec) or ('+miniapps' in spec)
+        if install_em and ('+shared' in spec):
+            make('examples/clean', 'miniapps/clean')
+            # This is a hack to get the examples and miniapps to link with the
+            # installed shared mfem library:
+            with working_dir('config'):
+                os.rename('config.mk', 'config.mk.orig')
+                shutil.copyfile(str(self.config_mk), 'config.mk')
+                shutil.copystat('config.mk.orig', 'config.mk')
 
         if '+examples' in spec:
             make('examples')
@@ -291,3 +425,64 @@ def yes_no(varstr):
         if '+miniapps' in spec:
             make('miniapps')
             install_tree('miniapps', join_path(prefix, 'miniapps'))
+
+        if install_em:
+            install_tree('data', join_path(prefix, 'data'))
+
+    @property
+    def suitesparse_components(self):
+        """Return the SuiteSparse components needed by MFEM."""
+        ss_comps = 'umfpack,cholmod,colamd,amd,camd,ccolamd,suitesparseconfig'
+        if self.spec.satisfies('@3.2:'):
+            ss_comps = 'klu,btf,' + ss_comps
+        return ss_comps
+
+    @property
+    def sundials_components(self):
+        """Return the SUNDIALS components needed by MFEM."""
+        sun_comps = 'arkode,cvode,nvecserial,kinsol'
+        if '+mpi' in self.spec:
+            sun_comps += ',nvecparhyp,nvecparallel'
+        return sun_comps
+
+    @property
+    def headers(self):
+        """Export the main mfem header, mfem.hpp.
+        """
+        hdrs = HeaderList(find(self.prefix.include, 'mfem.hpp',
+                               recursive=False))
+        return hdrs or None
+
+    @property
+    def libs(self):
+        """Export the mfem library file.
+        """
+        libs = find_libraries('libmfem', root=self.prefix.lib,
+                              shared=('+shared' in self.spec), recursive=False)
+        return libs or None
+
+    @property
+    def config_mk(self):
+        """Export the location of the config.mk file.
+           This property can be accessed using spec['mfem'].package.config_mk
+        """
+        dirs = [self.prefix, self.prefix.share.mfem]
+        for d in dirs:
+            f = join_path(d, 'config.mk')
+            if os.access(f, os.R_OK):
+                return FileList(f)
+        return FileList(find(self.prefix, 'config.mk', recursive=True))
+
+    @property
+    def test_mk(self):
+        """Export the location of the test.mk file.
+           This property can be accessed using spec['mfem'].package.test_mk.
+           In version 3.3.2 and newer, the location of test.mk is also defined
+           inside config.mk, variable MFEM_TEST_MK.
+        """
+        dirs = [self.prefix, self.prefix.share.mfem]
+        for d in dirs:
+            f = join_path(d, 'test.mk')
+            if os.access(f, os.R_OK):
+                return FileList(f)
+        return FileList(find(self.prefix, 'test.mk', recursive=True))

var/spack/repos/builtin/packages/mfem/test_builds.sh

@@ -0,0 +1,72 @@
+#!/bin/bash
+
+builds=(
+    # preferred version:
+    'mfem'
+    'mfem~mpi~metis~gzstream'
+    'mfem+mpi+superlu-dist+suite-sparse+petsc \
+        +sundials+mpfr+netcdf+gzstream+gnutls+libunwind \
+        ^hypre~internal-superlu ^petsc~boost+suite-sparse+mumps'
+    'mfem~mpi+suite-sparse+sundials+mpfr+netcdf \
+        +gzstream+gnutls+libunwind'
+    # develop version:
+    'mfem@develop+shared~static'
+    'mfem@develop+shared~static~mpi~metis~gzstream'
+    # TODO: Replace '^conduit~python~hdf5' with '^conduit~python' when conduit
+    # is fixed to accept '^hdf5+mpi'.
+    'mfem@develop+shared~static+mpi \
+        +superlu-dist+suite-sparse+petsc+sundials+mpfr+netcdf+gzstream \
+        +gnutls+libunwind+conduit ^hypre~internal-superlu \
+        ^petsc~boost+suite-sparse+mumps ^conduit~python~hdf5'
+    # TODO: The options '^netcdf~mpi ^hdf5@1.8.19~mpi' are added just to make
+    # conduit happy.
+    'mfem@develop+shared~static~mpi \
+        +suite-sparse+sundials+mpfr+netcdf+gzstream+gnutls+libunwind \
+        +conduit ^conduit~python ^netcdf~mpi ^hdf5@1.8.19~mpi'
+)
+
+builds2=(
+    # preferred version
+    'mfem+superlu-dist'
+    'mfem+suite-sparse~mpi'
+    'mfem+suite-sparse'
+    'mfem+sundials~mpi'
+    'mfem+sundials'
+    'mfem+netcdf~mpi'
+    'mfem+netcdf'
+    'mfem+mpfr'
+    'mfem+gnutls'
+    'mfem+petsc+mpi ^hypre~internal-superlu ^petsc~boost+suite-sparse+mumps'
+    # develop version
+    'mfem@develop+superlu-dist'
+    'mfem@develop+suite-sparse~mpi'
+    'mfem@develop+suite-sparse'
+    'mfem@develop+sundials~mpi'
+    'mfem@develop+sundials'
+    'mfem@develop+netcdf~mpi'
+    'mfem@develop+netcdf'
+    'mfem@develop+mpfr'
+    'mfem@develop+gnutls'
+    'mfem@develop+conduit~mpi ^conduit~python'
+    'mfem@develop+conduit ^conduit~python'
+    'mfem@develop+petsc+mpi ^hypre~internal-superlu \
+        ^petsc~boost+suite-sparse+mumps'
+)
+
+trap 'printf "\nScript interrupted.\n"; exit 33' INT
+
+SEP='=========================================================================='
+sep='--------------------------------------------------------------------------'
+
+for bld in "${builds[@]}" "${builds2[@]}"; do
+    printf "\n%s\n" "${SEP}"
+    printf "    %s\n" "${bld}"
+    printf "%s\n" "${SEP}"
+    eval bbb="\"${bld}\""
+    spack spec -I $bbb || exit 1
+    printf "%s\n" "${sep}"
+    spack install --test=root $bbb || exit 2
+done
+
+# Uninstall all mfem builds:
+# spack uninstall --all mfem

var/spack/repos/builtin/packages/nek5000/package.py

@@ -25,6 +25,7 @@
 from spack import *
 
 import numbers
+import os
 
 
 def is_integral(x):
@@ -46,8 +47,9 @@ class Nek5000(Package):
     tags = ['cfd', 'flow', 'hpc', 'solver', 'navier-stokes',
             'spectral-elements', 'fluid']
 
-    version('17.0',    git='https://github.com/Nek5000/Nek5000.git',
+    version('17.0', '6a13bfad2ce023897010dd88f54a0a87',
-        commit='469daf94d3f9aa3ba9d258d8eee9ebde6893a702')
+            url="https://github.com/Nek5000/Nek5000/releases/download/"
+                    "v17.0/Nek5000-v17.0.tar.gz")
     version('develop', git='https://github.com/Nek5000/Nek5000.git',
         branch='master')
 
@@ -56,6 +58,9 @@ class Nek5000(Package):
     variant('profiling', default=True, description='Build with profiling data.')
     variant('visit',     default=False, description='Build with Visit.')
 
+    # TODO: add a variant 'blas' or 'external-blas' to enable the usage of
+    #       Spack installed/configured blas.
+
     # Variant for MAXNEL, we need to read this from user
     variant(
         'MAXNEL',
@@ -76,27 +81,106 @@ class Nek5000(Package):
 
     # Dependencies
     depends_on('mpi', when="+mpi")
+    depends_on('libx11', when="+prenek")
+    depends_on('libx11', when="+postnek")
+    # libxt is needed for X11/Intrinsic.h but not for linking
+    depends_on('libxt', when="+prenek")
+    depends_on('libxt', when="+postnek")
     depends_on('visit', when="+visit")
 
     @run_before('install')
     def fortran_check(self):
-        if not self.compiler.fc:
+        if not self.compiler.f77:
-            msg = 'Cannot build Nek5000 without a Fortran compiler.'
+            msg = 'Cannot build Nek5000 without a Fortran 77 compiler.'
             raise RuntimeError(msg)
 
+    @run_after('install')
+    def test_install(self):
+        currentDir = os.getcwd()
+        eddyDir = 'short_tests/eddy'
+        os.chdir(eddyDir)
+
+        os.system(join_path(self.prefix.bin, 'makenek') + ' eddy_uv')
+        if not os.path.isfile(join_path(os.getcwd(), 'nek5000')):
+            msg = 'Cannot build example: short_tests/eddy.'
+            raise RuntimeError(msg)
+
+        os.chdir(currentDir)
+
     def install(self, spec, prefix):
         toolsDir   = 'tools'
         binDir     = 'bin'
 
-        FC  = self.compiler.fc
+        # Do not use the Spack compiler wrappers.
+        # Use directly the compilers:
+        FC  = self.compiler.f77
         CC  = self.compiler.cc
 
+        fflags = spec.compiler_flags['fflags']
+        cflags = spec.compiler_flags['cflags']
+        if ('+prenek' in spec) or ('+postnek' in spec):
+            libx11_h = find_headers('Xlib', spec['libx11'].prefix.include,
+                                    recursive=True)
+            if not libx11_h:
+                raise RuntimeError('Xlib.h not found in %s' %
+                                   spec['libx11'].prefix.include)
+            cflags += ['-I%s' % os.path.dirname(libx11_h.directories[0])]
+            libxt_h = find_headers('Intrinsic', spec['libxt'].prefix.include,
+                                   recursive=True)
+            if not libxt_h:
+                raise RuntimeError('X11/Intrinsic.h not found in %s' %
+                                   spec['libxt'].prefix.include)
+            cflags += ['-I%s' % os.path.dirname(libxt_h.directories[0])]
+        if self.compiler.name in ['xl', 'xl_r']:
+            # Use '-qextname' to add underscores.
+            # Use '-WF,-qnotrigraph' to fix an error about a string: '... ??'
+            fflags += ['-qextname', '-WF,-qnotrigraph']
+        fflags = ' '.join(fflags)
+        cflags = ' '.join(cflags)
+
         # Build the tools, maketools copy them to Nek5000/bin by default.
         # We will then install Nek5000/bin under prefix after that.
         with working_dir(toolsDir):
             # Update the maketools script to use correct compilers
-            filter_file(r'^FC\s*=.*', 'FC="{0}"'.format(FC), 'maketools')
+            filter_file(r'^#FC\s*=.*', 'FC="{0}"'.format(FC), 'maketools')
-            filter_file(r'^CC\s*=.*', 'CC="{0}"'.format(CC), 'maketools')
+            filter_file(r'^#CC\s*=.*', 'CC="{0}"'.format(CC), 'maketools')
+            if fflags:
+                filter_file(r'^#FFLAGS=.*', 'FFLAGS="{0}"'.format(fflags),
+                            'maketools')
+            if cflags:
+                filter_file(r'^#CFLAGS=.*', 'CFLAGS="{0}"'.format(cflags),
+                            'maketools')
+
+            if self.compiler.name in ['xl', 'xl_r']:
+                # Patch 'maketools' to use '-qextname' when checking for
+                # underscore becasue 'xl'/'xl_r' use this option to enable the
+                # addition of the underscore.
+                filter_file(r'^\$FC -c ', '$FC -qextname -c ', 'maketools')
+
+            libx11_lib = find_libraries('libX11', spec['libx11'].prefix.lib,
+                                        shared=True, recursive=True)
+            if not libx11_lib:
+                libx11_lib = \
+                    find_libraries('libX11', spec['libx11'].prefix.lib64,
+                                   shared=True, recursive=True)
+            if not libx11_lib:
+                raise RuntimeError('libX11 not found in %s/{lib,lib64}' %
+                                   spec['libx11'].prefix)
+            # There is no other way to set the X11 library path except brute
+            # force:
+            filter_file(r'-L\$\(X\)', libx11_lib.search_flags,
+                        join_path('prenek', 'makefile'))
+            filter_file(r'-L\$\(X\)', libx11_lib.search_flags,
+                        join_path('postnek', 'makefile'))
+
+            if self.compiler.name in ['xl', 'xl_r']:
+                # Use '-qextname' when compiling mxm.f
+                filter_file('\$\(OLAGS\)', '-qextname $(OLAGS)',
+                            join_path('postnek', 'makefile'))
+            # Define 'rename_' function that calls 'rename'
+            with open(join_path('postnek', 'xdriver.c'), 'a') as xdriver:
+                xdriver.write('\nvoid rename_(char *from, char *to)\n{\n'
+                              '   rename(from, to);\n}\n')
 
             maxnel = self.spec.variants['MAXNEL'].value
             filter_file(r'^#MAXNEL\s*=.*', 'MAXNEL=' + maxnel, 'maketools')
@@ -106,8 +190,9 @@ def install(self, spec, prefix):
             # Build the tools
             if '+genbox' in spec:
                 makeTools('genbox')
-            if '+int_tp' in spec and self.version == Version('17.0.0-beta2'):
+            # "ERROR: int_tp does not exist!"
-                makeTools('int_tp')
+            # if '+int_tp' in spec:
+            #     makeTools('int_tp')
             if '+n2to3' in spec:
                 makeTools('n2to3')
             if '+postnek' in spec:
@@ -138,11 +223,26 @@ def install(self, spec, prefix):
 
             # Update the makenek to use correct compilers and
             # Nek5000 source.
-            if self.version >= Version('17.0'):
+            filter_file(r'^#FC\s*=.*', 'FC="{0}"'.format(FC), 'makenek')
-                filter_file(r'^#FC\s*=.*', 'FC="{0}"'.format(FC), 'makenek')
+            filter_file(r'^#CC\s*=.*', 'CC="{0}"'.format(CC), 'makenek')
-                filter_file(r'^#CC\s*=.*', 'CC="{0}"'.format(CC), 'makenek')
+            filter_file(r'^#SOURCE_ROOT\s*=\"\$H.*',  'SOURCE_ROOT=\"' +
-                filter_file(r'^#SOURCE_ROOT\s*=\"\$H.*',  'SOURCE_ROOT=\"' +
+                        prefix.bin.Nek5000 + '\"',  'makenek')
-                            prefix.bin.Nek5000 + '\"',  'makenek')
+            if fflags:
+                filter_file(r'^#FFLAGS=.*', 'FFLAGS="{0}"'.format(fflags),
+                            'makenek')
+            if cflags:
+                filter_file(r'^#CFLAGS=.*', 'CFLAGS="{0}"'.format(cflags),
+                            'makenek')
+
+        with working_dir('core'):
+            if self.compiler.name in ['xl', 'xl_r']:
+                # Patch 'core/makenek.inc' and 'makefile.template' to use
+                # '-qextname' when checking for underscore becasue 'xl'/'xl_r'
+                # use this option to enable the addition of the underscore.
+                filter_file(r'^\$FCcomp -c ', '$FCcomp -qextname -c ',
+                            'makenek.inc')
+                filter_file(r'\$\(FC\) -c \$\(L0\)',
+                            '$(FC) -c -qextname $(L0)', 'makefile.template')
 
         # Install Nek5000/bin in prefix/bin
         install_tree(binDir, prefix.bin)

var/spack/repos/builtin/packages/nekcem/package.py

@@ -24,6 +24,7 @@
 ##############################################################################
 from spack import *
 import os
+import json
 
 
 class Nekcem(Package):
@@ -39,10 +40,11 @@ class Nekcem(Package):
 
     # We only have a development version
     version('develop', git='https://github.com/NekCEM/NekCEM.git')
+    version('0b8bedd', git='https://github.com/NekCEM/NekCEM.git',
+            commit='0b8beddfdcca646bfcc866dfda1c5f893338399b')
 
     # dependencies
     depends_on('mpi', when='+mpi')
-    depends_on('python@2.7:')
     depends_on('blas')
     depends_on('lapack')
 
@@ -52,39 +54,77 @@ def fortran_check(self):
             msg = 'NekCEM can not be built without a Fortran compiler.'
             raise RuntimeError(msg)
 
+    @run_after('install')
+    def test_install(self):
+        NekCEM_test = join_path(self.prefix.bin, 'NekCEM', 'tests', '2dboxpec')
+        with working_dir(NekCEM_test):
+            makenek = Executable(join_path(self.prefix.bin, 'makenek'))
+            makenek(os.path.basename(NekCEM_test))
+            if not os.path.isfile('nekcem'):
+                msg = 'Cannot build example: %s' % NekCEM_test
+                raise RuntimeError(msg)
+
     def install(self, spec, prefix):
         binDir = 'bin'
         nek = 'nek'
         cNek = 'configurenek'
         mNek = 'makenek'
 
-        FC = self.compiler.fc
+        FC = self.compiler.f77
         CC = self.compiler.cc
 
+        fflags = spec.compiler_flags['fflags']
+        cflags = spec.compiler_flags['cflags']
+        ldflags = spec.compiler_flags['ldflags']
+
         if '+mpi' in spec:
             FC = spec['mpi'].mpif77
             CC = spec['mpi'].mpicc
 
         with working_dir(binDir):
-            filter_file(r'^FC\s*=.*', 'FC=\"' + FC + '\"', 'makenek')
+            fflags = ['-O3'] + fflags
-            filter_file(r'^CC\s*=.*', 'CC=\"' + CC + '\"', 'makenek')
+            cflags = ['-O3'] + cflags
-            filter_file(r'^NEK\s*=.*', 'NEK=\"' + prefix.bin.NekCEM +
+            fflags += ['-I.']
-                        '\"', 'makenek')
+            cflags += ['-I.', '-DGLOBAL_LONG_LONG']
 
+            if self.compiler.name == 'gcc' or self.compiler.name == 'clang':
+                # assuming 'clang' uses 'gfortran'
+                fflags += ['-fdefault-real-8', '-fdefault-double-8']
+                cflags += ['-DUNDERSCORE']
+            elif self.compiler.name == 'intel':
+                fflags += ['-r8']
+                cflags += ['-DUNDERSCORE']
+            elif self.compiler.name == 'xl' or self.compiler.name == 'xl_r':
+                fflags += ['-qrealsize=8']
+                cflags += ['-DPREFIX=jl_', '-DIBM']
+            elif self.compiler.name == 'pgi':
+                fflags += ['-r8']
+                cflags += ['-DUNDERSCORE']
+
+            if '+mpi' in spec:
+                fflags += ['-DMPI', '-DMPIIO']
+                cflags += ['-DMPI', '-DMPIIO']
             blasLapack = spec['lapack'].libs + spec['blas'].libs
-            ldFlags = blasLapack.ld_flags
+            pthread_lib = find_system_libraries('libpthread')
-            # Temporary workaround, we should use LDFLAGS when
+            ldflags += (blasLapack + pthread_lib).ld_flags.split()
-            # configurenek in Nekcem is fixed.
+            all_arch = {
-            # See issue: https://github.com/NekCEM/NekCEM/issues/200
+                'spack-arch': {
-            filter_file(r'^EXTRALDFLAGS\s*=.*', 'EXTRALDFLAGS=\"' + ldFlags +
+                    'FC': FC, 'FFLAGS': fflags,
-                        '\"', 'makenek')
+                    'CC': CC, 'CFLAGS': cflags,
+                    'LD': FC, 'LDFLAGS': ldflags
+                }
+            }
+            os.rename('arch.json', 'arch.json.orig')
+            with open('arch.json', 'w') as file:
+                file.write(json.dumps(all_arch))
+            filter_file(r'^ARCH=.*$', 'ARCH=spack-arch', 'makenek')
+            filter_file(r'^NEK=.*', 'NEK="%s"' % prefix.bin.NekCEM,
+                        'makenek')
 
         # Install NekCEM in prefix/bin
         install_tree('../NekCEM', prefix.bin.NekCEM)
         # Create symlinks to makenek, nek and configurenek scripts
-        os.symlink(os.path.join(prefix.bin.NekCEM, binDir, mNek),
+        with working_dir(prefix.bin):
-                   os.path.join(prefix.bin, mNek))
+            os.symlink(os.path.join('NekCEM', binDir, mNek), mNek)
-        os.symlink(os.path.join(prefix.bin.NekCEM, binDir, cNek),
+            os.symlink(os.path.join('NekCEM', binDir, cNek), cNek)
-                   os.path.join(prefix.bin, cNek))
+            os.symlink(os.path.join('NekCEM', binDir, nek), nek)
-        os.symlink(os.path.join(prefix.bin.NekCEM, binDir, nek),
-                   os.path.join(prefix.bin, nek))

var/spack/repos/builtin/packages/occa/package.py

@@ -0,0 +1,138 @@
+##############################################################################
+# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC.
+# Produced at the Lawrence Livermore National Laboratory.
+#
+# This file is part of Spack.
+# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
+# LLNL-CODE-647188
+#
+# For details, see https://github.com/llnl/spack
+# Please also see the LICENSE file for our notice and the LGPL.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License (as
+# published by the Free Software Foundation) version 2.1, February 1999.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
+# conditions of the GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+##############################################################################
+from spack import *
+import os
+import shutil
+
+
+class Occa(Package):
+    """OCCA is an open-source (MIT license) library used to program current
+       multi-core/many-core architectures. Devices (such as CPUs, GPUs,
+       Intel's Xeon Phi, FPGAs, etc) are abstracted using an offload-model
+       for application development and programming for the devices is done
+       through a C-based (OKL) or Fortran-based kernel language (OFL).
+       OCCA gives developers the ability to target devices at run-time by
+       using run-time compilation for device kernels.
+    """
+
+    homepage = "http://libocca.org"
+    url      = 'https://github.com/libocca/occa'
+    git_url  = 'https://github.com/libocca/occa.git'
+
+    version('develop', git='https://github.com/libocca/occa.git')
+    version('v1.0.0-alpha.5', git=git_url, tag='v1.0.0-alpha.5')
+    version('v0.2.0', git=git_url, tag='v0.2.0')
+    version('v0.1.0', git=git_url, tag='v0.1.0')
+
+    variant('cuda',
+            default=True,
+            description='Activates support for CUDA')
+    variant('openmp',
+            default=True,
+            description='Activates support for OpenMP')
+    variant('opencl',
+            default=True,
+            description='Activates support for OpenCL')
+
+    depends_on('cuda', when='+cuda')
+
+    conflicts('%gcc@6:', when='^cuda@:8')
+    conflicts('%gcc@7:', when='^cuda@:9')
+
+    def install(self, spec, prefix):
+        # The build environment is set by the 'setup_environment' method.
+        # Copy the source to the installation directory and build OCCA there.
+        for file in os.listdir('.'):
+            dest = join_path(prefix, os.path.basename(file))
+            if os.path.isdir(file):
+                shutil.copytree(file, dest)
+            else:
+                shutil.copy2(file, dest)
+        make('-C', prefix)
+
+        if self.run_tests:
+            make('-C', prefix, 'test', parallel=False)
+
+    def _setup_runtime_flags(self, s_env):
+        spec = self.spec
+        s_env.set('OCCA_DIR', self.prefix)
+        s_env.set('OCCA_CXX', self.compiler.cxx)
+
+        cxxflags = spec.compiler_flags['cxxflags']
+        if cxxflags:
+            # Run-time compiler flags:
+            s_env.set('OCCA_CXXFLAGS', ' '.join(cxxflags))
+
+        if '+cuda' in spec:
+            cuda_dir = spec['cuda'].prefix
+            # Run-time CUDA compiler:
+            s_env.set('OCCA_CUDA_COMPILER',
+                      join_path(cuda_dir, 'bin', 'nvcc'))
+
+    def setup_environment(self, spack_env, run_env):
+        spec = self.spec
+        # The environment variable CXX is automatically set to the Spack
+        # compiler wrapper.
+
+        # The cxxflags, if specified, will be set by the Spack compiler wrapper
+        # while the environment variable CXXFLAGS will remain undefined.
+        # We define CXXFLAGS in the environment to tell OCCA to use the user
+        # specified flags instead of its defaults. This way the compiler will
+        # get the cxxflags twice - once from the Spack compiler wrapper and
+        # second time from OCCA - however, only the second one will be seen in
+        # the verbose output, so we keep both.
+        cxxflags = spec.compiler_flags['cxxflags']
+        if cxxflags:
+            spack_env.set('CXXFLAGS', ' '.join(cxxflags))
+
+        # For the cuda, openmp, and opencl variants, set the environment
+        # variable OCCA_{CUDA,OPENMP,OPENCL}_ENABLED only if the variant is
+        # disabled. Otherwise, let OCCA autodetect what is available.
+
+        if '+cuda' in spec:
+            cuda_dir = spec['cuda'].prefix
+            cuda_libs_list = ['libcuda', 'libcudart', 'libOpenCL']
+            cuda_libs = find_libraries(cuda_libs_list,
+                                       cuda_dir,
+                                       shared=True,
+                                       recursive=True)
+            spack_env.set('OCCA_INCLUDE_PATH', cuda_dir.include)
+            spack_env.set('OCCA_LIBRARY_PATH', ':'.join(cuda_libs.directories))
+        else:
+            spack_env.set('OCCA_CUDA_ENABLED', '0')
+
+        if '~opencl' in spec:
+            spack_env.set('OCCA_OPENCL_ENABLED', '0')
+
+        # Setup run-time environment for testing.
+        spack_env.set('OCCA_VERBOSE', '1')
+        self._setup_runtime_flags(spack_env)
+        # The 'run_env' is included in the Spack generated module files.
+        self._setup_runtime_flags(run_env)
+
+    def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
+        # Export OCCA_* variables for everyone using this package from within
+        # Spack.
+        self._setup_runtime_flags(spack_env)

var/spack/repos/builtin/packages/petsc/package.py

@@ -37,7 +37,7 @@ class Petsc(Package):
     homepage = "http://www.mcs.anl.gov/petsc/index.html"
     url = "http://ftp.mcs.anl.gov/pub/petsc/release-snapshots/petsc-3.5.3.tar.gz"
 
-    maintainers = ['balay', 'barrysmith']
+    maintainers = ['balay', 'barrysmith', 'jedbrown']
 
     version('develop', git='https://bitbucket.org/petsc/petsc.git', tag='master')
     version('xsdk-0.2.0', git='https://bitbucket.org/petsc/petsc.git', tag='xsdk-0.2.0')
@@ -88,6 +88,8 @@ class Petsc(Package):
     variant('clanguage', default='C', values=('C', 'C++'),
             description='Specify C (recommended) or C++ to compile PETSc',
             multi=False)
+    variant('suite-sparse', default=False,
+            description='Activates support for SuiteSparse')
 
     # 3.8.0 has a build issue with MKL - so list this conflict explicitly
     conflicts('^intel-mkl', when='@3.8.0')
@@ -98,6 +100,7 @@ class Petsc(Package):
     if sys.platform == "darwin":
         patch('macos-clang-8.1.0.diff',
               when='@3.7.5%clang@8.1.0:')
+    patch('pkg-config-3.7.6-3.8.4.diff', when='@3.7.6:3.8.4')
 
     # Virtual dependencies
     # Git repository needs sowing to build Fortran interface
@@ -150,6 +153,7 @@ class Petsc(Package):
     depends_on('trilinos@12.6.2:', when='@3.7.0:+trilinos+mpi')
     depends_on('trilinos@xsdk-0.2.0', when='@xsdk-0.2.0+trilinos+mpi')
     depends_on('trilinos@develop', when='@xdevelop+trilinos+mpi')
+    depends_on('suite-sparse', when='+suite-sparse')
 
     def mpi_dependent_options(self):
         if '~mpi' in self.spec:
@@ -226,7 +230,7 @@ def install(self, spec, prefix):
 
         # Activates library support if needed
         for library in ('metis', 'boost', 'hdf5', 'hypre', 'parmetis',
-                        'mumps', 'trilinos', 'zlib'):
+                        'mumps', 'trilinos'):
             options.append(
                 '--with-{library}={value}'.format(
                     library=library, value=('1' if library in spec else '0'))
@@ -251,6 +255,29 @@ def install(self, spec, prefix):
             options.append(
                 '--with-superlu_dist=0'
             )
+        # SuiteSparse: configuring using '--with-suitesparse-dir=...' has some
+        # issues, so specify directly the include path and the libraries.
+        if '+suite-sparse' in spec:
+            ss_spec = 'suite-sparse:umfpack,klu,cholmod,btf,ccolamd,colamd,' \
+                'camd,amd,suitesparseconfig'
+            options.extend([
+                '--with-suitesparse-include=%s' % spec[ss_spec].prefix.include,
+                '--with-suitesparse-lib=%s' % spec[ss_spec].libs.ld_flags,
+                '--with-suitesparse=1'
+            ])
+        else:
+            options.append('--with-suitesparse=0')
+
+        # zlib: configuring using '--with-zlib-dir=...' has some issues with
+        # SuiteSparse so specify directly the include path and the libraries.
+        if 'zlib' in spec:
+            options.extend([
+                '--with-zlib-include=%s' % spec['zlib'].prefix.include,
+                '--with-zlib-lib=%s'     % spec['zlib'].libs.ld_flags,
+                '--with-zlib=1'
+            ])
+        else:
+            options.append('--with-zlib=0')
 
         python('configure', '--prefix=%s' % prefix, *options)
 
@@ -301,3 +328,10 @@ def setup_dependent_environment(self, spack_env, run_env, dependent_spec):
         # Set up PETSC_DIR for everyone using PETSc package
         spack_env.set('PETSC_DIR', self.prefix)
         spack_env.unset('PETSC_ARCH')
+
+    @property
+    def headers(self):
+        return find_headers('petsc', self.prefix.include, recursive=False) \
+            or None  # return None to indicate failure
+
+    # For the 'libs' property - use the default handler.

var/spack/repos/builtin/packages/petsc/pkg-config-3.7.6-3.8.4.diff

@@ -0,0 +1,29 @@
+diff --git c/config/PETSc/Configure.py w/config/PETSc/Configure.py
+index ca65368e54..995afd0799 100644
+--- c/config/PETSc/Configure.py
++++ w/config/PETSc/Configure.py
+@@ -174,19 +174,19 @@ class Configure(config.base.Configure):
+ 
+     self.setCompilers.pushLanguage('C')
+     fd.write('ccompiler='+self.setCompilers.getCompiler()+'\n')
+-    fd.write('cflags_extra="'+self.setCompilers.getCompilerFlags().strip()+'"\n')
+-    fd.write('cflags_dep="'+self.compilers.dependenciesGenerationFlag.get('C','')+'"\n')
+-    fd.write('ldflag_rpath="'+self.setCompilers.CSharedLinkerFlag+'"\n')
++    fd.write('cflags_extra='+self.setCompilers.getCompilerFlags().strip()+'\n')
++    fd.write('cflags_dep='+self.compilers.dependenciesGenerationFlag.get('C','')+'\n')
++    fd.write('ldflag_rpath='+self.setCompilers.CSharedLinkerFlag+'\n')
+     self.setCompilers.popLanguage()
+     if hasattr(self.compilers, 'C++'):
+       self.setCompilers.pushLanguage('C++')
+       fd.write('cxxcompiler='+self.setCompilers.getCompiler()+'\n')
+-      fd.write('cxxflags_extra="'+self.setCompilers.getCompilerFlags().strip()+'"\n')
++      fd.write('cxxflags_extra='+self.setCompilers.getCompilerFlags().strip()+'\n')
+       self.setCompilers.popLanguage()
+     if hasattr(self.compilers, 'FC'):
+       self.setCompilers.pushLanguage('FC')
+       fd.write('fcompiler='+self.setCompilers.getCompiler()+'\n')
+-      fd.write('fflags_extra="'+self.setCompilers.getCompilerFlags().strip()+'"\n')
++      fd.write('fflags_extra='+self.setCompilers.getCompilerFlags().strip()+'\n')
+       self.setCompilers.popLanguage()
+ 
+     fd.write('\n')

var/spack/repos/builtin/packages/pumi/package.py

@@ -23,7 +23,6 @@
 # Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 ##############################################################################
 from spack import *
-import sys
 
 
 class Pumi(CMakePackage):
@@ -39,17 +38,15 @@ class Pumi(CMakePackage):
     homepage = "https://www.scorec.rpi.edu/pumi"
     url      = "https://github.com/SCOREC/core.git"
 
-    version('0.0.1', git='https://github.com/SCOREC/core.git',
+    version('2.1.0', git='https://github.com/SCOREC/core.git',
-        commit='0c315e82b3f2478dc18bdd6cfa89f1cddb85cd6a')
+        commit='840fbf6ec49a63aeaa3945f11ddb224f6055ac9f')
     version('develop', git='https://github.com/SCOREC/core.git',
         branch='master')
 
-    if sys.platform == 'darwin':
-        patch('phiotimer.cc.darwin.patch', level=0)  # !clock_gettime
-
     variant('zoltan', default=False, description='Enable Zoltan Features')
 
     depends_on('mpi')
+    depends_on('cmake@3:', type='build')
     depends_on('zoltan', when='+zoltan')
 
     def cmake_args(self):

var/spack/repos/builtin/packages/pumi/phiotimer.cc.darwin.patch

@@ -1,21 +0,0 @@
-*** ./phasta/phiotimer.cc	2017-05-03 12:26:25.000000000 -0700
---- ./phasta/phiotimer.cc.patched	2017-05-03 12:26:11.000000000 -0700
-***************
-*** 56,63 ****
-  }
-  #else
-  void phastaio_time(phastaioTime* t) {
-!   int err;
-!   err = clock_gettime(CLOCK_MONOTONIC,t);
-    PCU_ALWAYS_ASSERT(!err);
-  }
-  /*return elapsed time in micro seconds*/
---- 56,63 ----
-  }
-  #else
-  void phastaio_time(phastaioTime* t) {
-!   int err=0;
-!   /*err = clock_gettime(CLOCK_MONOTONIC,t);*/
-    PCU_ALWAYS_ASSERT(!err);
-  }
-  /*return elapsed time in micro seconds*/

var/spack/repos/builtin/packages/zoltan/package.py

@@ -33,7 +33,7 @@ class Zoltan(Package):
     """The Zoltan library is a toolkit of parallel combinatorial algorithms
        for parallel, unstructured, and/or adaptive scientific
        applications.  Zoltan's largest component is a suite of dynamic
-       load-balancing and paritioning algorithms that increase
+       load-balancing and partitioning algorithms that increase
        applications' parallel performance by reducing idle time.  Zoltan
        also has graph coloring and graph ordering algorithms, which are
        useful in task schedulers and parallel preconditioners.
@@ -75,7 +75,7 @@ def install(self, spec, prefix):
         ]
         config_cflags = [
             '-O0' if '+debug' in spec else '-O3',
-            '-g' if '+debug' in spec else '-g0',
+            '-g' if '+debug' in spec else '',
         ]
 
         if '+shared' in spec: