Deprecate spack setup (#18240)

2020-09-01 20:07:48 -05:00 · 2020-09-01 20:07:48 -05:00 · 17f7b23783
commit 17f7b23783
parent ae3f3887a6
6 changed files with 5 additions and 526 deletions
--- a/lib/spack/docs/config_yaml.rst
+++ b/lib/spack/docs/config_yaml.rst
@ -99,7 +99,7 @@ username is not already in the path, Spack will append the value of ``$user`` to
 the selected ``build_stage`` path.
 .. warning:: We highly recommend specifying ``build_stage`` paths that
-   distinguish between staging and other activities to ensure 
+   distinguish between staging and other activities to ensure
   ``spack clean`` does not inadvertently remove unrelated files.
   Spack prepends ``spack-stage-`` to temporary staging directory names to
   reduce this risk.  Using a combination of ``spack`` and or ``stage`` in
@ -223,7 +223,7 @@ To build all software in serial, set ``build_jobs`` to 1.
 --------------------
 When set to ``true`` Spack will use ccache to cache compiles. This is
-useful specifically in two cases: (1) when using ``spack setup``, and (2)
+useful specifically in two cases: (1) when using ``spack dev-build``, and (2)
 when building the same package with many different variants. The default is
 ``false``.
--- a/lib/spack/docs/known_issues.rst
+++ b/lib/spack/docs/known_issues.rst
@ -48,16 +48,3 @@ A workaround is to explicitly activate the variants of dependencies as well:
 See https://github.com/spack/spack/issues/267 and
 https://github.com/spack/spack/issues/2546 for further details.
 ----------------------------
 ``spack setup`` doesn't work
 ----------------------------
 **Status:** Work in progress
 Spack provides a ``setup`` command that is useful for the development of
 software outside of Spack. Unfortunately, this command no longer works.
 See https://github.com/spack/spack/issues/2597 and
 https://github.com/spack/spack/issues/2662 for details. This is expected
 to be fixed by https://github.com/spack/spack/pull/2664.
--- a/lib/spack/docs/packaging_guide.rst
+++ b/lib/spack/docs/packaging_guide.rst
@ -4677,119 +4677,3 @@ might write:
   DWARF_PREFIX = $(spack location --install-dir libdwarf)
   CXXFLAGS += -I$DWARF_PREFIX/include
   CXXFLAGS += -L$DWARF_PREFIX/lib
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Build System Configuration Support
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 Imagine a developer creating a CMake or Autotools-based project in a
 local directory, which depends on libraries A-Z.  Once Spack has
 installed those dependencies, one would like to run ``cmake`` with
 appropriate command line and environment so CMake can find them.  The
 ``spack setup`` command does this conveniently, producing a CMake
 configuration that is essentially the same as how Spack *would have*
 configured the project.  This can be demonstrated with a usage
 example:
 .. code-block:: console
   $ cd myproject
   $ spack setup myproject@local
   $ mkdir build; cd build
   $ ../spconfig.py ..
   $ make
   $ make install
 Notes:
 * Spack must have ``myproject/package.py`` in its repository for
  this to work.
 * ``spack setup`` produces the executable script ``spconfig.py`` in
  the local directory, and also creates the module file for the
  package.  ``spconfig.py`` is normally run from the user's
  out-of-source build directory.
 * The version number given to ``spack setup`` is arbitrary, just
  like ``spack diy``.  ``myproject/package.py`` does not need to
  have any valid downloadable versions listed (typical when a
  project is new).
 * spconfig.py produces a CMake configuration that *does not* use the
  Spack wrappers.  Any resulting binaries *will not* use RPATH,
  unless the user has enabled it.  This is recommended for
  development purposes, not production.
 * ``spconfig.py`` is human readable, and can serve as a developer
  reference of what dependencies are being used.
 * ``make install`` installs the package into the Spack repository,
  where it may be used by other Spack packages.
 * CMake-generated makefiles re-run CMake in some circumstances.  Use
  of ``spconfig.py`` breaks this behavior, requiring the developer
  to manually re-run ``spconfig.py`` when a ``CMakeLists.txt`` file
  has changed.
 ^^^^^^^^^^^^
 CMakePackage
 ^^^^^^^^^^^^
 In order to enable ``spack setup`` functionality, the author of
 ``myproject/package.py`` must subclass from ``CMakePackage`` instead
 of the standard ``Package`` superclass.  Because CMake is
 standardized, the packager does not need to tell Spack how to run
 ``cmake; make; make install``.  Instead the packager only needs to
 create (optional) methods ``configure_args()`` and ``configure_env()``, which
 provide the arguments (as a list) and extra environment variables (as
 a dict) to provide to the ``cmake`` command.  Usually, these will
 translate variant flags into CMake definitions.  For example:
 .. code-block:: python
   def cmake_args(self):
       spec = self.spec
       return [
           '-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'),
           '-DBUILD_PYTHON=%s' % ('YES' if '+python' in spec else 'NO'),
           '-DBUILD_GRIDGEN=%s' % ('YES' if '+gridgen' in spec else 'NO'),
           '-DBUILD_COUPLER=%s' % ('YES' if '+coupler' in spec else 'NO'),
           '-DUSE_PISM=%s' % ('YES' if '+pism' in spec else 'NO')
       ]
 If needed, a packager may also override methods defined in
 ``StagedPackage`` (see below).
 ^^^^^^^^^^^^^
 StagedPackage
 ^^^^^^^^^^^^^
 ``CMakePackage`` is implemented by subclassing the ``StagedPackage``
 superclass, which breaks down the standard ``Package.install()``
 method into several sub-stages: ``setup``, ``configure``, ``build``
 and ``install``.  Details:
 * Instead of implementing the standard ``install()`` method, package
  authors implement the methods for the sub-stages
  ``install_setup()``, ``install_configure()``,
  ``install_build()``, and ``install_install()``.
 * The ``spack install`` command runs the sub-stages ``configure``,
  ``build`` and ``install`` in order.  (The ``setup`` stage is
  not run by default; see below).
 * The ``spack setup`` command runs the sub-stages ``setup``
  and a dummy install (to create the module file).
 * The sub-stage install methods take no arguments (other than
  ``self``).  The arguments ``spec`` and ``prefix`` to the standard
  ``install()`` method may be accessed via ``self.spec`` and
  ``self.prefix``.
 ^^^^^^^^^^^^^
 GNU Autotools
 ^^^^^^^^^^^^^
 The ``setup`` functionality is currently only available for
 CMake-based packages.  Extending this functionality to GNU
 Autotools-based packages would be easy (and should be done by a
 developer who actively uses Autotools).  Packages that use
 non-standard build systems can gain ``setup`` functionality by
 subclassing ``StagedPackage`` directly.
 .. Emacs local variables
   Local Variables:
   fill-column: 79
   End:
--- a/lib/spack/docs/workflows.rst
+++ b/lib/spack/docs/workflows.rst
@ -703,400 +703,6 @@ environments:
  Administrators might find things easier to maintain without the
  added "heavyweight" state of a view.
 ------------------------------
 Developing Software with Spack
 ------------------------------
 For any project, one needs to assemble an
 environment of that application's dependencies.  You might consider
 loading a series of modules or creating a filesystem view.  This
 approach, while obvious, has some serious drawbacks:
 1. There is no guarantee that an environment created this way will be
   consistent.  Your application could end up with dependency A
   expecting one version of MPI, and dependency B expecting another.
   The linker will not be happy...
 2. Suppose you need to debug a package deep within your software DAG.
   If you build that package with a manual environment, then it
   becomes difficult to have Spack auto-build things that depend on
   it.  That could be a serious problem, depending on how deep the
   package in question is in your dependency DAG.
 3. At its core, Spack is a sophisticated concretization algorithm that
   matches up packages with appropriate dependencies and creates a
   *consistent* environment for the package it's building.  Writing a
   list of ``spack load`` commands for your dependencies is at least
   as hard as writing the same list of ``depends_on()`` declarations
   in a Spack package.  But it makes no use of Spack concretization
   and is more error-prone.
 4. Spack provides an automated, systematic way not just to find a
   packages's dependencies --- but also to build other packages on
   top.  Any Spack package can become a dependency for another Spack
   package, offering a powerful vision of software re-use.  If you
   build your package A outside of Spack, then your ability to use it
   as a building block for other packages in an automated way is
   diminished: other packages depending on package A will not
   be able to use Spack to fulfill that dependency.
 5. If you are reading this manual, you probably love Spack.  You're
   probably going to write a Spack package for your software so
   prospective users can install it with the least amount of pain.
   Why should you go to additional work to find dependencies in your
   development environment?  Shouldn't Spack be able to help you build
   your software based on the package you've already written?
 In this section, we show how Spack can be used in the software
 development process to greatest effect, and how development packages
 can be seamlessly integrated into the Spack ecosystem.  We will show
 how this process works by example, assuming the software you are
 creating is called ``mylib``.
 ^^^^^^^^^^^^^^^^^^^^^
 Write the CMake Build
 ^^^^^^^^^^^^^^^^^^^^^
 For now, the techniques in this section only work for CMake-based
 projects, although they could be easily extended to other build
 systems in the future.  We will therefore assume you are using CMake
 to build your project.
 The ``CMakeLists.txt`` file should be written as normal.  A few caveats:
 1. Your project should produce binaries with RPATHs.  This will ensure
   that they work the same whether built manually or automatically by
   Spack.  For example:
 .. code-block:: cmake
   # enable @rpath in the install name for any shared library being built
   # note: it is planned that a future version of CMake will enable this by default
   set(CMAKE_MACOSX_RPATH 1)
   # Always use full RPATH
   # http://www.cmake.org/Wiki/CMake_RPATH_handling
   # http://www.kitware.com/blog/home/post/510
   # use, i.e. don't skip the full RPATH for the build tree
   SET(CMAKE_SKIP_BUILD_RPATH  FALSE)
   # when building, don't use the install RPATH already
   # (but later on when installing)
   SET(CMAKE_BUILD_WITH_INSTALL_RPATH FALSE)
   # add the automatically determined parts of the RPATH
   # which point to directories outside the build tree to the install RPATH
   SET(CMAKE_INSTALL_RPATH_USE_LINK_PATH TRUE)
   # the RPATH to be used when installing, but only if it's not a system directory
   LIST(FIND CMAKE_PLATFORM_IMPLICIT_LINK_DIRECTORIES "${CMAKE_INSTALL_PREFIX}/lib" isSystemDir)
   IF("${isSystemDir}" STREQUAL "-1")
      SET(CMAKE_INSTALL_RPATH "${CMAKE_INSTALL_PREFIX}/lib")
   ENDIF("${isSystemDir}" STREQUAL "-1")
 2. Spack provides a CMake variable called
   ``SPACK_TRANSITIVE_INCLUDE_PATH``, which contains the ``include/``
   directory for all of your project's transitive dependencies.  It
   can be useful if your project ``#include``s files from package B,
   which ``#include`` files from package C, but your project only
   lists project B as a dependency.  This works in traditional
   single-tree build environments, in which B and C's include files
   live in the same place.  In order to make it work with Spack as
   well, you must add the following to ``CMakeLists.txt``.  It will
   have no effect when building without Spack:
   .. code-block:: cmake
      # Include all the transitive dependencies determined by Spack.
      # If we're not running with Spack, this does nothing...
      include_directories($ENV{SPACK_TRANSITIVE_INCLUDE_PATH})
   .. note::
      Note that this feature is controversial and could break with
      future versions of GNU ld.  The best practice is to make sure
      anything you ``#include`` is listed as a dependency in your
      CMakeLists.txt (and Spack package).
 .. _write-the-spack-package:
 ^^^^^^^^^^^^^^^^^^^^^^^
 Write the Spack Package
 ^^^^^^^^^^^^^^^^^^^^^^^
 The Spack package also needs to be written, in tandem with setting up
 the build (for example, CMake).  The most important part of this task
 is declaring dependencies.  Here is an example of the Spack package
 for the ``mylib`` package (ellipses for brevity):
 .. code-block:: python
   class Mylib(CMakePackage):
       """Misc. reusable utilities used by Myapp."""
       homepage = "https://github.com/citibeth/mylib"
       url = "https://github.com/citibeth/mylib/tarball/123"
       version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9')
       version('develop', git='https://github.com/citibeth/mylib.git',
           branch='develop')
       variant('everytrace', default=False,
               description='Report errors through Everytrace')
       ...
       extends('python')
       depends_on('eigen')
       depends_on('everytrace', when='+everytrace')
       depends_on('proj', when='+proj')
       ...
       depends_on('cmake', type='build')
       depends_on('doxygen', type='build')
       def cmake_args(self):
           spec = self.spec
           return [
               '-DUSE_EVERYTRACE=%s' % ('YES' if '+everytrace' in spec else 'NO'),
               '-DUSE_PROJ4=%s' % ('YES' if '+proj' in spec else 'NO'),
               ...
               '-DUSE_UDUNITS2=%s' % ('YES' if '+udunits2' in spec else 'NO'),
               '-DUSE_GTEST=%s' % ('YES' if '+googletest' in spec else 'NO')]
 This is a standard Spack package that can be used to install
 ``mylib`` in a production environment.  The list of dependencies in
 the Spack package will generally be a repeat of the list of CMake
 dependencies.  This package also has some features that allow it to be
 used for development:
 1. It subclasses ``CMakePackage`` instead of ``Package``.  This
   eliminates the need to write an ``install()`` method, which is
   defined in the superclass.  Instead, one just needs to write the
   ``configure_args()`` method.  That method should return the
   arguments needed for the ``cmake`` command (beyond the standard
   CMake arguments, which Spack will include already).  These
   arguments are typically used to turn features on/off in the build.
 2. It specifies a non-checksummed version ``develop``.  Running
   ``spack install mylib@develop`` the ``@develop`` version will
   install the latest version off the develop branch.  This method of
   download is useful for the developer of a project while it is in
   active development; however, it should only be used by developers
   who control and trust the repository in question!
 3. The ``url``, ``url_for_version()`` and ``homepage`` attributes are
   not used in development.  Don't worry if you don't have any, or if
   they are behind a firewall.
 ^^^^^^^^^^^^^^^^
 Build with Spack
 ^^^^^^^^^^^^^^^^
 Now that you have a Spack package, you can use Spack to find its
 dependencies automatically.  For example:
 .. code-block:: console
   $ cd mylib
   $ spack setup mylib@local
 The result will be a file ``spconfig.py`` in the top-level
 ``mylib/`` directory.  It is a short script that calls CMake with the
 dependencies and options determined by Spack --- similar to what
 happens in ``spack install``, but now written out in script form.
 From a developer's point of view, you can think of ``spconfig.py`` as
 a stand-in for the ``cmake`` command.
 .. note::
   You can invent any "version" you like for the ``spack setup``
   command.
 .. note::
   Although ``spack setup`` does not build your package, it does
   create and install a module file, and mark in the database that
   your package has been installed.  This can lead to errors, of
   course, if you don't subsequently install your package.
   Also... you will need to ``spack uninstall`` before you run
   ``spack setup`` again.
 You can now build your project as usual with CMake:
 .. code-block:: console
   $ mkdir build; cd build
   $ ../spconfig.py ..   # Instead of cmake ..
   $ make
   $ make install
 Once your ``make install`` command is complete, your package will be
 installed, just as if you'd run ``spack install``.  Except you can now
 edit, re-build and re-install as often as needed, without checking
 into Git or downloading tarballs.
 .. note::
   The build you get this way will be *almost* the same as the build
   from ``spack install``.  The only difference is, you will not be
   using Spack's compiler wrappers.  This difference has not caused
   problems in our experience, as long as your project sets
   RPATHs as shown above.  You DO use RPATHs, right?
 ^^^^^^^^^^^^^^^^^^^^
 Build Other Software
 ^^^^^^^^^^^^^^^^^^^^
 Now that you've built ``mylib`` with Spack, you might want to build
 another package that depends on it --- for example, ``myapp``.  This
 is accomplished easily enough:
 .. code-block:: console
   $ spack install myapp ^mylib@local
 Note that auto-built software has now been installed *on top of*
 manually-built software, without breaking Spack's "web."  This
 property is useful if you need to debug a package deep in the
 dependency hierarchy of your application.  It is a *big* advantage of
 using ``spack setup`` to build your package's environment.
 If you feel your software is stable, you might wish to install it with
 ``spack install`` and skip the source directory.  You can just use,
 for example:
 .. code-block:: console
   $ spack install mylib@develop
 .. _release-your-software:
 ^^^^^^^^^^^^^^^^^^^^^
 Release Your Software
 ^^^^^^^^^^^^^^^^^^^^^
 You are now ready to release your software as a tarball with a
 numbered version, and a Spack package that can build it.  If you're
 hosted on GitHub, this process will be a bit easier.
 #. Put tag(s) on the version(s) in your GitHub repo you want to be
   release versions.  For example, a tag ``v0.1.0`` for version 0.1.0.
 #. Set the ``url`` in your ``package.py`` to download a tarball for
   the appropriate version.  GitHub will give you a tarball for any
   commit in the repo, if you tickle it the right way.  For example:
   .. code-block:: python
      url = 'https://github.com/citibeth/mylib/tarball/v0.1.2'
 #. Use Spack to determine your version's hash, and cut'n'paste it into
   your ``package.py``:
   .. code-block:: console
      $ spack checksum mylib 0.1.2
      ==> Found 1 versions of mylib
        0.1.2     https://github.com/citibeth/mylib/tarball/v0.1.2
      How many would you like to checksum? (default is 5, q to abort)
      ==> Downloading...
      ==> Trying to fetch from https://github.com/citibeth/mylib/tarball/v0.1.2
      ######################################################################## 100.0%
      ==> Checksummed new versions of mylib:
            version('0.1.2', '3a6acd70085e25f81b63a7e96c504ef9')
 #. You should now be able to install released version 0.1.2 of your package with:
   .. code-block:: console
      $ spack install mylib@0.1.2
 #. There is no need to remove the `develop` version from your package.
   Spack concretization will always prefer numbered version to
   non-numeric versions.  Users will only get it if they ask for it.
 ^^^^^^^^^^^^^^^^^^^^^^^^
 Distribute Your Software
 ^^^^^^^^^^^^^^^^^^^^^^^^
 Once you've released your software, other people will want to build
 it; and you will need to tell them how.  In the past, that has meant a
 few paragraphs of prose explaining which dependencies to install.  But
 now you use Spack, and those instructions are written in executable
 Python code.  But your software has many dependencies, and you know
 Spack is the best way to install it:
 #. First, you will want to fork Spack's ``develop`` branch.  Your aim
   is to provide a stable version of Spack that you KNOW will install
   your software.  If you make changes to Spack in the process, you
   will want to submit pull requests to Spack core.
 #. Add your software's ``package.py`` to that fork.  You should submit
   a pull request for this as well, unless you don't want the public
   to know about your software.
 #. Prepare instructions that read approximately as follows:
   #. Download Spack from your forked repo.
   #. Install Spack; see :ref:`getting_started`.
   #. Set up an appropriate ``packages.yaml`` file.  You should tell
      your users to include in this file whatever versions/variants
      are needed to make your software work correctly (assuming those
      are not already in your ``packages.yaml``).
   #. Run ``spack install mylib``.
   #. Run this script to generate the ``module load`` commands or
      filesystem view needed to use this software.
 #. Be aware that your users might encounter unexpected bootstrapping
   issues on their machines, especially if they are running on older
   systems.  The :ref:`getting_started` section should cover this, but
   there could always be issues.
 ^^^^^^^^^^^^^^^^^^^
 Other Build Systems
 ^^^^^^^^^^^^^^^^^^^
 ``spack setup`` currently only supports CMake-based builds, in
 packages that subclass ``CMakePackage``.  The intent is that this
 mechanism should support a wider range of build systems; for example,
 GNU Autotools.  Someone well-versed in Autotools is needed to develop
 this patch and test it out.
 Python Distutils is another popular build system that should get
 ``spack setup`` support.  For non-compiled languages like Python,
 ``spack diy`` may be used.  Even better is to put the source directory
 directly in the user's ``PYTHONPATH``.  Then, edits in source files
 are immediately available to run without any install process at all!
 ^^^^^^^^^^
 Conclusion
 ^^^^^^^^^^
 The ``spack setup`` development workflow provides better automation,
 flexibility and safety than workflows relying on environment modules
 or filesystem views.  However, it has some drawbacks:
 #. It currently works only with projects that use the CMake build
   system.  Support for other build systems is not hard to build, but
   will require a small amount of effort for each build system to be
   supported.  It might not work well with some IDEs.
 #. It only works with packages that sub-class ``StagedPackage``.
   Currently, most Spack packages do not.  Converting them is not
   hard; but must be done on a package-by-package basis.
 #. It requires that users are comfortable with Spack, as they
   integrate Spack explicitly in their workflow.  Not all users are
   willing to do this.
 -------------------------------------
 Using Spack to Replace Homebrew/Conda
 -------------------------------------
--- a/lib/spack/spack/cmd/setup.py
+++ b/lib/spack/spack/cmd/setup.py
@ -37,6 +37,7 @@ def setup_parser(subparser):
    cd_group = subparser.add_mutually_exclusive_group()
    arguments.add_common_arguments(cd_group, ['clean', 'dirty'])
    subparser.epilog = 'DEPRECATED: use `spack dev-build` instead'
 def write_spconfig(package, dirty):
@ -98,6 +99,8 @@ def cmdlist(str):
 def setup(self, args):
    tty.warn('DEPRECATED: use `spack dev-build` instead')
    if not args.spec:
        tty.die("spack setup requires a package spec argument.")
--- a/lib/spack/spack/installer.py
+++ b/lib/spack/spack/installer.py
@ -466,7 +466,6 @@ def log(pkg):
    packages_dir = spack.store.layout.build_packages_path(pkg.spec)
    # Remove first if we're overwriting another build
    # (can happen with spack setup)
    try:
        # log and env install paths are inside this
        shutil.rmtree(packages_dir)