Deprecate spack setup (#18240)

This commit is contained in:
Adam J. Stewart 2020-09-01 20:07:48 -05:00 committed by GitHub
parent ae3f3887a6
commit 17f7b23783
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
6 changed files with 5 additions and 526 deletions

View file

@ -99,7 +99,7 @@ username is not already in the path, Spack will append the value of ``$user`` to
the selected ``build_stage`` path.
.. warning:: We highly recommend specifying ``build_stage`` paths that
distinguish between staging and other activities to ensure
distinguish between staging and other activities to ensure
``spack clean`` does not inadvertently remove unrelated files.
Spack prepends ``spack-stage-`` to temporary staging directory names to
reduce this risk. Using a combination of ``spack`` and or ``stage`` in
@ -223,7 +223,7 @@ To build all software in serial, set ``build_jobs`` to 1.
--------------------
When set to ``true`` Spack will use ccache to cache compiles. This is
useful specifically in two cases: (1) when using ``spack setup``, and (2)
useful specifically in two cases: (1) when using ``spack dev-build``, and (2)
when building the same package with many different variants. The default is
``false``.

View file

@ -48,16 +48,3 @@ A workaround is to explicitly activate the variants of dependencies as well:
See https://github.com/spack/spack/issues/267 and
https://github.com/spack/spack/issues/2546 for further details.
----------------------------
``spack setup`` doesn't work
----------------------------
**Status:** Work in progress
Spack provides a ``setup`` command that is useful for the development of
software outside of Spack. Unfortunately, this command no longer works.
See https://github.com/spack/spack/issues/2597 and
https://github.com/spack/spack/issues/2662 for details. This is expected
to be fixed by https://github.com/spack/spack/pull/2664.

View file

@ -4677,119 +4677,3 @@ might write:
DWARF_PREFIX = $(spack location --install-dir libdwarf)
CXXFLAGS += -I$DWARF_PREFIX/include
CXXFLAGS += -L$DWARF_PREFIX/lib
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Build System Configuration Support
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Imagine a developer creating a CMake or Autotools-based project in a
local directory, which depends on libraries A-Z. Once Spack has
installed those dependencies, one would like to run ``cmake`` with
appropriate command line and environment so CMake can find them. The
``spack setup`` command does this conveniently, producing a CMake
configuration that is essentially the same as how Spack *would have*
configured the project. This can be demonstrated with a usage
example:
.. code-block:: console
$ cd myproject
$ spack setup myproject@local
$ mkdir build; cd build
$ ../spconfig.py ..
$ make
$ make install
Notes:
* Spack must have ``myproject/package.py`` in its repository for
this to work.
* ``spack setup`` produces the executable script ``spconfig.py`` in
the local directory, and also creates the module file for the
package. ``spconfig.py`` is normally run from the user's
out-of-source build directory.
* The version number given to ``spack setup`` is arbitrary, just
like ``spack diy``. ``myproject/package.py`` does not need to
have any valid downloadable versions listed (typical when a
project is new).
* spconfig.py produces a CMake configuration that *does not* use the
Spack wrappers. Any resulting binaries *will not* use RPATH,
unless the user has enabled it. This is recommended for
development purposes, not production.
* ``spconfig.py`` is human readable, and can serve as a developer
reference of what dependencies are being used.
* ``make install`` installs the package into the Spack repository,
where it may be used by other Spack packages.
* CMake-generated makefiles re-run CMake in some circumstances. Use
of ``spconfig.py`` breaks this behavior, requiring the developer
to manually re-run ``spconfig.py`` when a ``CMakeLists.txt`` file
has changed.
^^^^^^^^^^^^
CMakePackage
^^^^^^^^^^^^
In order to enable ``spack setup`` functionality, the author of
``myproject/package.py`` must subclass from ``CMakePackage`` instead
of the standard ``Package`` superclass. Because CMake is
standardized, the packager does not need to tell Spack how to run
``cmake; make; make install``. Instead the packager only needs to
create (optional) methods ``configure_args()`` and ``configure_env()``, which
provide the arguments (as a list) and extra environment variables (as
a dict) to provide to the ``cmake`` command. Usually, these will
translate variant flags into CMake definitions. For example:
.. code-block:: python
def cmake_args(self):
spec = self.spec
return [
'-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'),
'-DBUILD_PYTHON=%s' % ('YES' if '+python' in spec else 'NO'),
'-DBUILD_GRIDGEN=%s' % ('YES' if '+gridgen' in spec else 'NO'),
'-DBUILD_COUPLER=%s' % ('YES' if '+coupler' in spec else 'NO'),
'-DUSE_PISM=%s' % ('YES' if '+pism' in spec else 'NO')
]
If needed, a packager may also override methods defined in
``StagedPackage`` (see below).
^^^^^^^^^^^^^
StagedPackage
^^^^^^^^^^^^^
``CMakePackage`` is implemented by subclassing the ``StagedPackage``
superclass, which breaks down the standard ``Package.install()``
method into several sub-stages: ``setup``, ``configure``, ``build``
and ``install``. Details:
* Instead of implementing the standard ``install()`` method, package
authors implement the methods for the sub-stages
``install_setup()``, ``install_configure()``,
``install_build()``, and ``install_install()``.
* The ``spack install`` command runs the sub-stages ``configure``,
``build`` and ``install`` in order. (The ``setup`` stage is
not run by default; see below).
* The ``spack setup`` command runs the sub-stages ``setup``
and a dummy install (to create the module file).
* The sub-stage install methods take no arguments (other than
``self``). The arguments ``spec`` and ``prefix`` to the standard
``install()`` method may be accessed via ``self.spec`` and
``self.prefix``.
^^^^^^^^^^^^^
GNU Autotools
^^^^^^^^^^^^^
The ``setup`` functionality is currently only available for
CMake-based packages. Extending this functionality to GNU
Autotools-based packages would be easy (and should be done by a
developer who actively uses Autotools). Packages that use
non-standard build systems can gain ``setup`` functionality by
subclassing ``StagedPackage`` directly.
.. Emacs local variables
Local Variables:
fill-column: 79
End:

View file

@ -703,400 +703,6 @@ environments:
Administrators might find things easier to maintain without the
added "heavyweight" state of a view.
------------------------------
Developing Software with Spack
------------------------------
For any project, one needs to assemble an
environment of that application's dependencies. You might consider
loading a series of modules or creating a filesystem view. This
approach, while obvious, has some serious drawbacks:
1. There is no guarantee that an environment created this way will be
consistent. Your application could end up with dependency A
expecting one version of MPI, and dependency B expecting another.
The linker will not be happy...
2. Suppose you need to debug a package deep within your software DAG.
If you build that package with a manual environment, then it
becomes difficult to have Spack auto-build things that depend on
it. That could be a serious problem, depending on how deep the
package in question is in your dependency DAG.
3. At its core, Spack is a sophisticated concretization algorithm that
matches up packages with appropriate dependencies and creates a
*consistent* environment for the package it's building. Writing a
list of ``spack load`` commands for your dependencies is at least
as hard as writing the same list of ``depends_on()`` declarations
in a Spack package. But it makes no use of Spack concretization
and is more error-prone.
4. Spack provides an automated, systematic way not just to find a
packages's dependencies --- but also to build other packages on
top. Any Spack package can become a dependency for another Spack
package, offering a powerful vision of software re-use. If you
build your package A outside of Spack, then your ability to use it
as a building block for other packages in an automated way is
diminished: other packages depending on package A will not
be able to use Spack to fulfill that dependency.
5. If you are reading this manual, you probably love Spack. You're
probably going to write a Spack package for your software so
prospective users can install it with the least amount of pain.
Why should you go to additional work to find dependencies in your
development environment? Shouldn't Spack be able to help you build
your software based on the package you've already written?
In this section, we show how Spack can be used in the software
development process to greatest effect, and how development packages
can be seamlessly integrated into the Spack ecosystem. We will show
how this process works by example, assuming the software you are
creating is called ``mylib``.
^^^^^^^^^^^^^^^^^^^^^
Write the CMake Build
^^^^^^^^^^^^^^^^^^^^^
For now, the techniques in this section only work for CMake-based
projects, although they could be easily extended to other build
systems in the future. We will therefore assume you are using CMake
to build your project.
The ``CMakeLists.txt`` file should be written as normal. A few caveats:
1. Your project should produce binaries with RPATHs. This will ensure
that they work the same whether built manually or automatically by
Spack. For example:
.. code-block:: cmake
# enable @rpath in the install name for any shared library being built
# note: it is planned that a future version of CMake will enable this by default
set(CMAKE_MACOSX_RPATH 1)
# Always use full RPATH
# http://www.cmake.org/Wiki/CMake_RPATH_handling
# http://www.kitware.com/blog/home/post/510
# use, i.e. don't skip the full RPATH for the build tree
SET(CMAKE_SKIP_BUILD_RPATH FALSE)
# when building, don't use the install RPATH already
# (but later on when installing)
SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)
# add the automatically determined parts of the RPATH
# which point to directories outside the build tree to the install RPATH
SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
# the RPATH to be used when installing, but only if it's not a system directory
LIST(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${CMAKE_INSTALL_PREFIX}/lib" isSystemDir)
IF("${isSystemDir}" STREQUAL "-1")
SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
ENDIF("${isSystemDir}" STREQUAL "-1")
2. Spack provides a CMake variable called
``SPACK_TRANSITIVE_INCLUDE_PATH``, which contains the ``include/``
directory for all of your project's transitive dependencies. It
can be useful if your project ``#include``s files from package B,
which ``#include`` files from package C, but your project only
lists project B as a dependency. This works in traditional
single-tree build environments, in which B and C's include files
live in the same place. In order to make it work with Spack as
well, you must add the following to ``CMakeLists.txt``. It will
have no effect when building without Spack:
.. code-block:: cmake
# Include all the transitive dependencies determined by Spack.
# If we're not running with Spack, this does nothing...
include_directories($ENV{SPACK_TRANSITIVE_INCLUDE_PATH})
.. note::
Note that this feature is controversial and could break with
future versions of GNU ld. The best practice is to make sure
anything you ``#include`` is listed as a dependency in your
CMakeLists.txt (and Spack package).
.. _write-the-spack-package:
^^^^^^^^^^^^^^^^^^^^^^^
Write the Spack Package
^^^^^^^^^^^^^^^^^^^^^^^
The Spack package also needs to be written, in tandem with setting up
the build (for example, CMake). The most important part of this task
is declaring dependencies. Here is an example of the Spack package
for the ``mylib`` package (ellipses for brevity):
.. code-block:: python
class Mylib(CMakePackage):
"""Misc. reusable utilities used by Myapp."""
homepage = "https://github.com/citibeth/mylib"
url = "https://github.com/citibeth/mylib/tarball/123"
version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9')
version('develop', git='https://github.com/citibeth/mylib.git',
branch='develop')
variant('everytrace', default=False,
description='Report errors through Everytrace')
...
extends('python')
depends_on('eigen')
depends_on('everytrace', when='+everytrace')
depends_on('proj', when='+proj')
...
depends_on('cmake', type='build')
depends_on('doxygen', type='build')
def cmake_args(self):
spec = self.spec
return [
'-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'),
'-DUSE_PROJ4=%s' % ('YES' if '+proj' in spec else 'NO'),
...
'-DUSE_UDUNITS2=%s' % ('YES' if '+udunits2' in spec else 'NO'),
'-DUSE_GTEST=%s' % ('YES' if '+googletest' in spec else 'NO')]
This is a standard Spack package that can be used to install
``mylib`` in a production environment. The list of dependencies in
the Spack package will generally be a repeat of the list of CMake
dependencies. This package also has some features that allow it to be
used for development:
1. It subclasses ``CMakePackage`` instead of ``Package``. This
eliminates the need to write an ``install()`` method, which is
defined in the superclass. Instead, one just needs to write the
``configure_args()`` method. That method should return the
arguments needed for the ``cmake`` command (beyond the standard
CMake arguments, which Spack will include already). These
arguments are typically used to turn features on/off in the build.
2. It specifies a non-checksummed version ``develop``. Running
``spack install mylib@develop`` the ``@develop`` version will
install the latest version off the develop branch. This method of
download is useful for the developer of a project while it is in
active development; however, it should only be used by developers
who control and trust the repository in question!
3. The ``url``, ``url_for_version()`` and ``homepage`` attributes are
not used in development. Don't worry if you don't have any, or if
they are behind a firewall.
^^^^^^^^^^^^^^^^
Build with Spack
^^^^^^^^^^^^^^^^
Now that you have a Spack package, you can use Spack to find its
dependencies automatically. For example:
.. code-block:: console
$ cd mylib
$ spack setup mylib@local
The result will be a file ``spconfig.py`` in the top-level
``mylib/`` directory. It is a short script that calls CMake with the
dependencies and options determined by Spack --- similar to what
happens in ``spack install``, but now written out in script form.
From a developer's point of view, you can think of ``spconfig.py`` as
a stand-in for the ``cmake`` command.
.. note::
You can invent any "version" you like for the ``spack setup``
command.
.. note::
Although ``spack setup`` does not build your package, it does
create and install a module file, and mark in the database that
your package has been installed. This can lead to errors, of
course, if you don't subsequently install your package.
Also... you will need to ``spack uninstall`` before you run
``spack setup`` again.
You can now build your project as usual with CMake:
.. code-block:: console
$ mkdir build; cd build
$ ../spconfig.py .. # Instead of cmake ..
$ make
$ make install
Once your ``make install`` command is complete, your package will be
installed, just as if you'd run ``spack install``. Except you can now
edit, re-build and re-install as often as needed, without checking
into Git or downloading tarballs.
.. note::
The build you get this way will be *almost* the same as the build
from ``spack install``. The only difference is, you will not be
using Spack's compiler wrappers. This difference has not caused
problems in our experience, as long as your project sets
RPATHs as shown above. You DO use RPATHs, right?
^^^^^^^^^^^^^^^^^^^^
Build Other Software
^^^^^^^^^^^^^^^^^^^^
Now that you've built ``mylib`` with Spack, you might want to build
another package that depends on it --- for example, ``myapp``. This
is accomplished easily enough:
.. code-block:: console
$ spack install myapp ^mylib@local
Note that auto-built software has now been installed *on top of*
manually-built software, without breaking Spack's "web." This
property is useful if you need to debug a package deep in the
dependency hierarchy of your application. It is a *big* advantage of
using ``spack setup`` to build your package's environment.
If you feel your software is stable, you might wish to install it with
``spack install`` and skip the source directory. You can just use,
for example:
.. code-block:: console
$ spack install mylib@develop
.. _release-your-software:
^^^^^^^^^^^^^^^^^^^^^
Release Your Software
^^^^^^^^^^^^^^^^^^^^^
You are now ready to release your software as a tarball with a
numbered version, and a Spack package that can build it. If you're
hosted on GitHub, this process will be a bit easier.
#. Put tag(s) on the version(s) in your GitHub repo you want to be
release versions. For example, a tag ``v0.1.0`` for version 0.1.0.
#. Set the ``url`` in your ``package.py`` to download a tarball for
the appropriate version. GitHub will give you a tarball for any
commit in the repo, if you tickle it the right way. For example:
.. code-block:: python
url = 'https://github.com/citibeth/mylib/tarball/v0.1.2'
#. Use Spack to determine your version's hash, and cut'n'paste it into
your ``package.py``:
.. code-block:: console
$ spack checksum mylib 0.1.2
==> Found 1 versions of mylib
0.1.2 https://github.com/citibeth/mylib/tarball/v0.1.2
How many would you like to checksum? (default is 5, q to abort)
==> Downloading...
==> Trying to fetch from https://github.com/citibeth/mylib/tarball/v0.1.2
######################################################################## 100.0%
==> Checksummed new versions of mylib:
version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9')
#. You should now be able to install released version 0.1.2 of your package with:
.. code-block:: console
$ spack install mylib@0.1.2
#. There is no need to remove the `develop` version from your package.
Spack concretization will always prefer numbered version to
non-numeric versions. Users will only get it if they ask for it.
^^^^^^^^^^^^^^^^^^^^^^^^
Distribute Your Software
^^^^^^^^^^^^^^^^^^^^^^^^
Once you've released your software, other people will want to build
it; and you will need to tell them how. In the past, that has meant a
few paragraphs of prose explaining which dependencies to install. But
now you use Spack, and those instructions are written in executable
Python code. But your software has many dependencies, and you know
Spack is the best way to install it:
#. First, you will want to fork Spack's ``develop`` branch. Your aim
is to provide a stable version of Spack that you KNOW will install
your software. If you make changes to Spack in the process, you
will want to submit pull requests to Spack core.
#. Add your software's ``package.py`` to that fork. You should submit
a pull request for this as well, unless you don't want the public
to know about your software.
#. Prepare instructions that read approximately as follows:
#. Download Spack from your forked repo.
#. Install Spack; see :ref:`getting_started`.
#. Set up an appropriate ``packages.yaml`` file. You should tell
your users to include in this file whatever versions/variants
are needed to make your software work correctly (assuming those
are not already in your ``packages.yaml``).
#. Run ``spack install mylib``.
#. Run this script to generate the ``module load`` commands or
filesystem view needed to use this software.
#. Be aware that your users might encounter unexpected bootstrapping
issues on their machines, especially if they are running on older
systems. The :ref:`getting_started` section should cover this, but
there could always be issues.
^^^^^^^^^^^^^^^^^^^
Other Build Systems
^^^^^^^^^^^^^^^^^^^
``spack setup`` currently only supports CMake-based builds, in
packages that subclass ``CMakePackage``. The intent is that this
mechanism should support a wider range of build systems; for example,
GNU Autotools. Someone well-versed in Autotools is needed to develop
this patch and test it out.
Python Distutils is another popular build system that should get
``spack setup`` support. For non-compiled languages like Python,
``spack diy`` may be used. Even better is to put the source directory
directly in the user's ``PYTHONPATH``. Then, edits in source files
are immediately available to run without any install process at all!
^^^^^^^^^^
Conclusion
^^^^^^^^^^
The ``spack setup`` development workflow provides better automation,
flexibility and safety than workflows relying on environment modules
or filesystem views. However, it has some drawbacks:
#. It currently works only with projects that use the CMake build
system. Support for other build systems is not hard to build, but
will require a small amount of effort for each build system to be
supported. It might not work well with some IDEs.
#. It only works with packages that sub-class ``StagedPackage``.
Currently, most Spack packages do not. Converting them is not
hard; but must be done on a package-by-package basis.
#. It requires that users are comfortable with Spack, as they
integrate Spack explicitly in their workflow. Not all users are
willing to do this.
-------------------------------------
Using Spack to Replace Homebrew/Conda
-------------------------------------

View file

@ -37,6 +37,7 @@ def setup_parser(subparser):
cd_group = subparser.add_mutually_exclusive_group()
arguments.add_common_arguments(cd_group, ['clean', 'dirty'])
subparser.epilog = 'DEPRECATED: use `spack dev-build` instead'
def write_spconfig(package, dirty):
@ -98,6 +99,8 @@ def cmdlist(str):
def setup(self, args):
tty.warn('DEPRECATED: use `spack dev-build` instead')
if not args.spec:
tty.die("spack setup requires a package spec argument.")

View file

@ -466,7 +466,6 @@ def log(pkg):
packages_dir = spack.store.layout.build_packages_path(pkg.spec)
# Remove first if we're overwriting another build
# (can happen with spack setup)
try:
# log and env install paths are inside this
shutil.rmtree(packages_dir)