hlrs-software-stack/spack
7
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Bugfix/docs: correct and expand smoke test documentation (#20278)

Browse source
This commit is contained in:
Tamara Dahlgren 2020-12-15 08:38:00 -08:00 committed by GitHub
parent d6bfc104bf
commit 21f30e3074
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
1 changed files with 153 additions and 72 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

225
lib/spack/docs/packaging_guide.rst
View file

@ -3956,111 +3956,192 @@ Install Tests
The API for adding and running install tests is not yet considered
stable and may change drastically in future releases. Packages with
upstreamed tests will be refactored to match changes to the API.
install tests will be refactored to match changes to the API.
While build-tests are integrated with the build system, install tests
may be added to Spack packages to be run independently of the install
method.
While build tests are integrated with the build system, install tests
are for testing package installations independent of the build process.
Install tests may be added by defining a ``test`` method with the following signature:
Install tests are executed in a test stage directory that defaults to
``~/.spack/test``. You can change the location in the high-level ``config``
by adding the ``test_stage`` path as follows:
.. code-block:: yaml
config:
test_stage: /path/to/stage
""""""""""""""""""""
Adding install tests
""""""""""""""""""""
Install tests are added to a package by defining a ``test`` method
with the following signature:
.. code-block:: python
def test(self):
These tests will be run in an environment set up to provide access to
this package and all of its dependencies, including ``test``-type
dependencies. Inside the ``test`` method, standard python ``assert``
statements and other error reporting mechanisms can be used. Spack
will report any errors as a test failure.
These tests run in an environment that provides access to the
package and all of its dependencies, including ``test``-type
dependencies.
Inside the test method, individual tests can be run separately (and
continue transparently after a test failure) using the ``run_test``
method. The signature for the ``run_test`` method is:
Standard python ``assert`` statements and other error reporting
mechanisms can be used in the ``test`` method. Spack will report
such errors as test failures.
You can implement multiple tests (or test parts) within the ``test``
method, where each is run separately and testing continues after
failures by using the ``run_test`` method. The signature for the
method is:
.. code-block:: python
def run_test(self, exe, options=[], expected=[], status=0, installed=False,
purpose='', skip_missing=False, work_dir=None):
This method will operate in ``work_dir`` if one is specified. It will
search for an executable in the ``PATH`` variable named ``exe``, and
if ``installed=True`` it will fail if that executable does not come
from the prefix of the package being tested. If the executable is not
found, it will fail the test unless ``skip_missing`` is set to
``True``. The executable will be run with the options specified, and
the return code will be checked against the ``status`` argument, which
can be an integer or list of integers. Spack will also check that
every string in ``expected`` is a regex matching part of the output of
the executable. The ``purpose`` argument is recorded in the test log
for debugging purposes.
The test fails if there is no executable named ``exe`` found in the
paths of the ``PATH`` variable **unless** ``skip_missing`` is ``True``.
The test also fails if the resulting path is not within the prefix of
the package being tested when ``installed`` is ``True``.
""""""""""""""""""""""""""""""""""""""
Install tests that require compilation
""""""""""""""""""""""""""""""""""""""
The executable runs in ``work_dir``, when specified, using the provided
``options``. The return code is checked against the ``status`` argument,
which can be an integer or list of integers representing status codes
corresponding to successful execution. Spack also checks that every string
in ``expected`` is a regex matching part of the output from the test run.
Output from the test is written to its log file. The ``purpose`` argument
serves as the heading in text logs to highlight the start of each test part.
"""""""""""""""""""""""""
Enabling test compilation
"""""""""""""""""""""""""
Some tests may require access to the compiler with which the package
was built, especially to test library-only packages. To ensure the
compiler is configured as part of the test environment, set the
attribute ``tests_require_compiler = True`` on the package. The
compiler will be available through the canonical environment variables
(``CC``, ``CXX``, ``FC``, ``F77``) in the test environment.
was built, especially to test library-only packages. You must enable
loading the package's compiler configuration by setting the attribute
``test_requires_compiler`` to ``True``. Doing so makes the compiler
available in the test environment through the canonical environment
variables (``CC``, ``CXX``, ``FC``, ``F77``).
""""""""""""""""""""""""""""""""""""""""""""""""
Install tests that require build-time components
""""""""""""""""""""""""""""""""""""""""""""""""
"""""""""""""""""""""""
Adding build-time files
"""""""""""""""""""""""
Some packages cannot be easily tested without components from the
build-time test suite. For those packages, the
``cache_extra_test_sources`` method can be used.
.. note::
We highly recommend the re-use of build-time tests and input files
for testing installed software. These files are easier to keep
synchronized with the software than creating custom install tests.
You can use the ``cache_extra_test_sources`` method copy directories
and or files from the build stage directory to the package's installation
directory.
For example, a package method for copying everything in the ``tests``
subdirectory plus the ``foo.c`` and ``bar.c`` files from ``examples``
can be implemented as follows:
.. code-block:: python
@run_after('install')
def cache_test_sources(self):
srcs = ['./tests/foo.c', './tests/bar.c']
srcs = ['tests', 'examples/foo.c', 'examples/bar.c']
self.cache_extra_test_sources(srcs)
This method will copy the listed methods into the metadata directory
of the package at the end of the install phase of the build. They will
be available to the test method in the directory
``self._extra_tests_path``.
The use of the ``run_after`` directive ensures the associated files
are copied **after** the package is installed during the build process.
While source files are generally recommended, for many packages
binaries may also technically be cached in this way for later testing.
The method copies files to the package's metadata directory under
the ``self.install_test_root``. All files in the package source's
``tests`` directory for the example above will be copied to the
``join_path(self.install_test_root, 'tests')`` directory. The files
will be copied to the ``join_path(self.install_test_root, 'examples')``
directory.
.. note::
While source and input files are generally recommended, binaries
may also be cached by the build process for install testing.
"""""""""""""""""""
Adding custom files
"""""""""""""""""""
Some tests may require additional files not available from the build.
Examples include:
- test source files
- test input files
- test build scripts
- expected test output
These extra files should be added to the ``test`` subdirectory of the
package in the repository. Spack will automaticaly copy any files in
that directory to the test staging directory during install testing.
The ``test`` method can access those files from the
``self.test_suite.current_test_data_dir`` directory.
.. _cmd-spack-test:
"""""""""""""""""""
``spack test list``
"""""""""""""""""""
Packages available for install testing can be found using the
``spack test list`` command. The command outputs all installed
packages that have defined ``test`` methods.
""""""""""""""""""
``spack test run``
""""""""""""""""""
Install tests can be run for one or more installed packages using
the ``spack test run`` command. A ``test suite`` is created from
the provided specs. If no specs are provided it will test all specs
in the active environment or all specs installed in Spack if no
environment is active.
Test suites can be named using the ``--alias`` option. Unaliased
Test suites will use the content hash of their specs as their name.
Some of the more commonly used debugging options are:
- ``--fail-fast`` stops testing each package after the first failure
- ``--fail-first`` stops testing packages after the first failure
Test output is written to a text log file by default but ``junit``
and ``cdash`` are outputs are available through the ``--log-format``
option.
""""""""""""""""""""""
``spack test results``
""""""""""""""""""""""
The ``spack test results`` command shows results for all completed
test suites. Providing the alias or content hash limits reporting
to the corresponding test suite.
The ``--logs`` option includes the output generated by the associated
test(s) to facilitate debugging.
The ``--failed`` option limits results shown to that of the failed
tests, if any, of matching packages.
"""""""""""""""""""
``spack test find``
"""""""""""""""""""
The ``spack test find`` command lists the aliases or content hashes
of all test suites whose results are available.
"""""""""""""""""""""
Running install tests
``spack test remove``
"""""""""""""""""""""
Install tests can be run using the ``spack test run`` command. The
``spack test run`` command will create a ``test suite`` out of the
specs provided to it, or if no specs are provided it will test all
specs in the active environment, or all specs installed in Spack if no
environment is active. Test suites can be named using the ``--alias``
option; test suites not aliased will use the content hash of their
specs as their name.
Packages to install test can be queried using the ``spack test list``
command, which outputs all installed packages with defined ``test``
methods.
Test suites can be found using the ``spack test find`` command. It
will list all test suites that have been run and have not been removed
using the ``spack test remove`` command. The ``spack test remove``
command will remove tests to declutter the test stage. The ``spack
test results`` command will show results for completed test suites.
The test stage is the working directory for all install tests run with
Spack. By default, Spack uses ``~/.spack/test`` as the test stage. The
test stage can be set in the high-level config:
.. code-block:: yaml
config:
test_stage: /path/to/stage
The ``spack test remove`` command removes test suites to declutter
the test results directory. You are prompted to confirm the removal
of each test suite **unless** you use the ``--yes-to-all`` option.
---------------------------
File manipulation functions