From 701fc1fdb15253f771aeec330a66b992d8c55dce Mon Sep 17 00:00:00 2001 From: Peter Scheibel Date: Tue, 12 May 2020 17:08:08 -0700 Subject: [PATCH] Update docs on "spack external find" (#16482) This improves the documentation for `spack external find` in several ways: * Provide a code example of implementing `determine_spec_details` for a package * Explain how to define executables to look for (and also e.g. that they are treated as regular expressions and so can pull in unexpected files). * Add the "why" for a couple of constraints (i.e. explain that this logic only works for build/run deps because it examines `PATH` for executables) * Spread the docs between build customization and packaging sections * Add cross-references * Add a label so that `spack external find` is linked from the command reference. --- lib/spack/docs/build_settings.rst | 24 +++++++---- lib/spack/docs/packaging_guide.rst | 64 ++++++++++++++++++++++++++++++ 2 files changed, 80 insertions(+), 8 deletions(-) diff --git a/lib/spack/docs/build_settings.rst b/lib/spack/docs/build_settings.rst index dee892f272..9f67d8c14f 100644 --- a/lib/spack/docs/build_settings.rst +++ b/lib/spack/docs/build_settings.rst @@ -158,11 +158,13 @@ Spack can then use any of the listed external implementations of MPI to satisfy a dependency, and will choose depending on the compiler and architecture. +.. _cmd-spack-external-find: + ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Automatically Find External Packages ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -A user can run the :ref:`spack external find ` command +You can run the :ref:`spack external find ` command to search for system-provided packages and add them to ``packages.yaml``. After running this command your ``packages.yaml`` may include new entries: @@ -177,17 +179,23 @@ Generally this is useful for detecting a small set of commonly-used packages; for now this is generally limited to finding build-only dependencies. Specific limitations include: -* A package must define ``executables`` and ``determine_spec_details`` - for Spack to locate instances of that package. -* This is currently intended to find build dependencies rather than - library packages. +* Packages are not discoverable by default: For a package to be + discoverable with ``spack external find``, it needs to add special + logic. See :ref:`here ` for more details. +* The current implementation only collects and examines executable files, + so it is typically only useful for build/run dependencies (in some cases + if a library package also provides an executable, it may be possible to + extract a meaningful Spec by running the executable - for example the + compiler wrappers in MPI implementations). +* The logic does not search through module files, it can only detect + packages with executables defined in ``PATH``; you can help Spack locate + externals which use module files by loading any associated modules for + packages that you want Spack to know about before running + ``spack external find``. * Spack does not overwrite existing entries in the package configuration: If there is an external defined for a spec at any configuration scope, then Spack will not add a new external entry (``spack config blame packages`` can help locate all external entries). -* Currently this logic is focused on examining ``PATH`` and does not - search through modules (although it should find the package if a - module is loaded for it). .. _concretization-preferences: diff --git a/lib/spack/docs/packaging_guide.rst b/lib/spack/docs/packaging_guide.rst index 7abadaac82..840c29454b 100644 --- a/lib/spack/docs/packaging_guide.rst +++ b/lib/spack/docs/packaging_guide.rst @@ -4048,6 +4048,70 @@ File functions :py:func:`touch(path) ` Create an empty file at ``path``. +.. _make-package-findable: + +---------------------------------------------------------- +Making a package discoverable with ``spack external find`` +---------------------------------------------------------- + +To make a package discoverable with +:ref:`spack external find ` you must +define one or more executables associated with the package and must +implement a method to generate a Spec when given an executable. + +The executables are specified as a package level ``executables`` +attribute which is a list of strings (see example below); each string +is treated as a regular expression (e.g. 'gcc' would match 'gcc', 'gcc-8.3', +'my-weird-gcc', etc.). + +The method ``determine_spec_details`` has the following signature: + +.. code-block:: python + + def determine_spec_details(prefix, exes_in_prefix): + # exes_in_prefix = a set of paths, each path is an executable + # prefix = a prefix that is common to each path in exes_in_prefix + + # return None or [] if none of the exes represent an instance of + # the package. Return one or more Specs for each instance of the + # package which is thought to be installed in the provided prefix + +``determine_spec_details`` takes as parameters a set of discovered +executables (which match those specified by the user) as well as a +common prefix shared by all of those executables. The function must +return one or more Specs associated with the executables (it can also +return ``None`` to indicate that no provided executables are associated +with the package). + +Say for example we have a package called ``foo-package`` which +builds an executable called ``foo``. ``FooPackage`` would appear as +follows: + +.. code-block:: python + + class FooPackage(Package): + homepage = "..." + url = "..." + + version(...) + + # Each string provided here is treated as a regular expression, and + # would match for example 'foo', 'foobar', and 'bazfoo'. + executables = ['foo'] + + @classmethod + def determine_spec_details(cls, prefix, exes_in_prefix): + candidates = list(x for x in exes_in_prefix + if os.path.basename(x) == 'foo') + if not candidates: + return + # This implementation is lazy and only checks the first candidate + exe_path = candidates[0] + exe = spack.util.executable.Executable(exe_path) + output = exe('--version') + version_str = ... # parse output for version string + return Spec('foo-package@{0}'.format(version_str)) + .. _package-lifecycle: -----------------------------