Archived

This repository has been archived on 2023-11-20. You can view files and clone it, but cannot push or open issues or pull requests.

Martin Beaudoin 568b982954 testHarness: improving the documentation for the test harness

2015-09-23 15:18:26 +01:00

14 KiB

Raw Blame History

foam-extend-3.2: A guide to test harness

A guide to the foam-extend test harness

###############################################################################

A guide to the foam-extend test harness

This file is a guide to the foam-extend test harness.

The test harness from foam-extend version 3.2 was used as a reference implementation for this documentation.

The original version of this file is located at $WM\_PROJECT\_DIR/doc/testHarness/testHarness.org.

A plain text version of this file is located at $WM\_PROJECT\_DIR/doc/testHarness/testHarness.txt.

A MediaWiki version of this file is located at

In case any converted versions (text, MediaWiki, etc.) of this file differ from the original testHarness.org file, the latter should always be considered the reference version. But we do appreciate your feedback and comments for improving this text.

For instance, if you ever decide to improve the Mediawiki version only, please consider contributing your changes back to the testHarness.org file as well. Otherwise, your Mediawiki changes might get overwritten by an updated version of the .org reference file.

Introduction

The foam-extend test harness scripts are based mainly on Kitware CMake/CTest utilities http://www.cmake.org/.

In its current version, the test harness is designed for running the full suite of foam-extend tutorials, and report the fail/success results to a CDash Web service one can consult using a simple browser.

The CDash Web service for foam-extend 3.2 is available here: http://foam-extend.sourceforge.net/CDash/index.php

For a quick introduction of CMake/CTest, see here: http://www.cmake.org/Wiki/CMake/Testing_With_CTest

For a more complete introduction to CMake and CTest, one can also refer to the Kitware book `Mastering CMake: A Cross-Platform Build System` http://www.kitware.com/products/books.php

For more information on Kitware CDash, see here: http://www.cdash.org/

The rest of this document will provide suficient information so you can run the foam-extend test harness on your own.

Files location and a short description

The foam-extend test harness is currently built from the following files:

$WM_PROJECT_DIR/CMakeLists.txt: This is the main CMake file driving the configuration of the test harness. When processed though the cmake utility, all the necessary Makefiles and CTest scripts will get generated.
$WM_PROJECT_DIR/CTestConfig.cmake: This file contains the various variables and definitions necessary to publish your test harness results over the CDash Web service for foam-extend 3.2.
$FOAM_SITE_DIR/etc/CTestConfig.site.cmake: This file allow you to override the values from the main $WM\_PROJECT\_DIR/CTestConfig.cmake. This file should be used if you are hosting your own site CDash Web service.
$WM_PROJECT_USER_DIR/etc/CTestConfig.user.cmake: This file allow you to override the values from the main $WM\_PROJECT\_DIR/CTestConfig.cmake and the site file $FOAM\_SITE\_DIR/etc/CTestConfig.site.cmake. This file should be used if you are hosting your own user space personal CDash Web service.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/CMakeLists.txt: This file is a copy of the main $WM\_PROJECT\_DIR/CMakeLists.txt file.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/CTestConfig.cmake.foam-extend: This file is a copy of the main $WM\_PROJECT\_DIR/CTestConfig.cmake file.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/Dashboard_Tutorials.cmake.in: This file is a template for generating the file `ctest -S` file $WM\_PROJECT\_DIR/testHarness/foam-extend/3.2/runDir/Dashboard\_Tutorials.cmake CMake will take care of automatically generating the Dashboard\_Tutorials.cmake file.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/CMakeFiles/FOAM_Tutorials.cmake: This file will configure a copy the foam-extend tutorials for running under the test harness. The copy will be made under $WM\_PROJECT\_DIR/testHarness/foam-extend/3.2/runDir.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allclean: Cleanup script. Will remove all the automatically generated files and directories for the test harness. Should be called prior to running the Allrun\_\* scripts.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allrun_CMakeOnly: This script will generate all the necessary files for running the test harness on the foam-extend tutorials. One can then invoke `make help` to see all the available make options.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allrun_Experimental: Same script as `Allrun\_CMakeOnly`, but will also call `make Experimental`.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir/Allrun_Nightly: Same script as `Allrun\_CMakeOnly`, but will also call `make Nightly`.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/AdditionalRunFunctions: Additional `bash` macros for the tutorial Allrun files.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/Allrun.default: Default Allrun script for the tutorial when none are provided.
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/addMissingAllrunFileToTutorial.sh: Bash script for adding a default Allrun file to the tutorials that do not have one. The test harness only run tutorials with an existing Allrun file
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/prepareCasesForOneTimeStep.sh: This script will modify the test cases system/controlDict in order for the case to run for only 1 time step
$WM_PROJECT_DIR/testHarness/foam-extend/3.2/scripts/prepareCasesForTestHarness.sh: This script will modify the test cases Allrun file so it can run properly under the test harness.

Configuring your $WM\_PROJECT\_DIR/etc/prefs.sh file for the test harness

The following environment variables are used for configuring the test harness. You should use your $WM\_PROJECT\_DIR/etc/prefs.sh file to initialize these variables.

CDASH_SUBMIT_LOCAL_HOST_ID: System identifier for the FOAM CDash test harness on foam-extend. By default, your system FQN/hostname will be used as the system identifier when publishing your test harness results on the FOAM CDash server on foam-extend. You can override your identifier using this environment variable.
CDASH_SCM_INFO: Buildname suffix for the FOAM CDash test harness on foam-extend. By default, the git branch name and git revision number will be appended to the CDash build name. Otherwise, for users not using git, or wanting to provide additionnal information, simply initialize the CDASH\_SCM\_INFO with the proper information.
WM_NCOMPPROCS: Specify the number of cores to use for the parallel execution of the test harness.
FOAM_TUTORIALS: Directory where the original test cases are located. For foam-extend, this would be by default $WM\_PROJECT\_DIR/tutorials.

The main dashboards : Experimental, Nightly and Continuous

The result of a test run, reformatted for easy review, is called a `dashboard`. A dashboard can be submitted to a central server, like CDash. Once properly configured, the test harness will offer 3 main dashboards.

Experimental: Will test the current state of the project. An experimental submission can be performed at any time, usually interactively from the current working copy of a developer.
Nightly: Is similar to Experimental, except that the source tree will be set to the state it was in at a specific nightly time. This ensures that all `nightly` submissions correspond to the state of the project at the same point in time. `Nightly` builds are usually done automatically at a preset time of day. Nightly build will also update your source code to the latest available revision. So it is best not to run a Nightly dashboard on source code that is not yet committed.
Continuous: Means that the source tree is updated to the latest revision, and a build / test cycle is performed only if any files were actually updated. Like `Nightly` builds, `Continuous` ones are usually done automatically and repeatedly in intervals.

There are also `intermediary dashboards` that allow you to select a specific test harness intermediary step. The command `make help` will show you that list:

... ContinuousBuild
... ContinuousConfigure
... ContinuousCoverage
... ContinuousMemCheck
... ContinuousStart
... ContinuousSubmit
... ContinuousTest
... ContinuousUpdate
... ExperimentalBuild
... ExperimentalConfigure
... ExperimentalCoverage
... ExperimentalMemCheck
... ExperimentalStart
... ExperimentalSubmit
... ExperimentalTest
... ExperimentalUpdate
... NightlyBuild
... NightlyConfigure
... NightlyCoverage
... NightlyMemCheck
... NightlyMemoryCheck
... NightlyStart
... NightlySubmit
... NightlyTest
... NightlyUpdate

Note: For foam-extend, the MemCheck and Coverage dashboards are not supported.

Running the test harness

Running the test harness is pretty simple:

These commands will configure and run the `Experimental` version of the test harness:

   cd $WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir
   ./Allclean
   ./Allrun_Experimental

To run the `Nightly` version of the test harness:

   cd $WM_PROJECT_DIR/testHarness/foam-extend/3.2/runDir
   ./Allclean
   ./Allrun_Nightly

To see the full range of available options, run the command:

make help

Selecting a subset of test cases to run

Instead of using the `make` command to run the test harness, one can also use command `ctest`. The commmand `ctest` offers additionnal options to select or limit the number of tests to run.

The following command will provide all the available options for ctest:

ctest -N

Here is a list of useful ctest options:

ctest -N: this command will list the available tests by their name and number. By default, all the tests are run in succession following the numerical order shown.
ctest -R <regex>: run all the tests whose name are matching the supplied regular expression. For instance, to run all the tests related to cfMesh, one can use the following command: `ctest -R cfMesh`
ctest -E <regex>: run all the tests, but exclude the ones whose name is matching the supplied regular expression. For instance, to run all the tests except those for cfMesh, one can use the following command: `ctest -E cfMesh`
ctest -I [Start,End,Stride,test#,test#]: run individual tests by number. `ctest -I 3,5` will run test 3, 4 and 5. `ctest -I 4,4,,4,7,13` will run tests 4, 7 and 13.
ctest -D dashboard: run a specific dashboard test. For example, the command `make Experimental` can be replaced by the following suite of ctest commands:

ctest -D ExperimentalStart
ctest -D ExperimentalConfigure
ctest -D ExperimentalBuild
ctest -D ExperimentalTest
ctest -D ExperimentalSubmit

Here is a more complete example where we will configure, build, test and submit the test harness results, but only for the incompressible tutorials:

ctest -D Experimental -R incompressible

Browsing the CDash service

The results of the test harness run will be published on the CDash dashboard on foam-extend.

To see your results: URL : http://foam-extend.sourceforge.net/CDash/index.php?project=foam-extend-3.2

On this interactive Web site, one can then point and click various buttons and menus to explore the various reports uploaded from your test harness runs.

Configuring the test harness for using your own site or personal CDash service

The foam-extend source code comes with a set pre-configured parameters for uploading your dashboards results on the main project CDash server. One can also choose to host their own CDash service, either as a site service, or as a personnal service running in your own user space. Your CDash administrator can generate a file similar to $WM\_PROJECT\_DIR/CTestConfig.cmake where all the necessary parameters for connecting to your local service are specified.

In order to use your site CDash service, simply copy your site CTestConfig.cmake file to $FOAM\_SITE\_DIR/etc/CTestConfig.site.cmake

In order to use your personnal CDash service, simply copy your personnal CTestConfig.cmake file to $WM\_PROJECT\_USER\_DIR/etc/CTestConfig.user.cmake

As usual, your site configuration will override the default parameters from the main configuration file $WM\_PROJECT\_DIR/CTestConfig.cmake.

Likewise, your personnal configuration will override the default parameters from the main configuration file $WM\_PROJECT\_DIR/CTestConfig.cmake and the default parameters from the site file $FOAM\_SITE\_DIR/etc/CTestConfig.site.cmake.

New features for the test harness (foam-extend 3.2):

Running the test harness in parallel: It is now possible to run the test harness in parallel over a single node or computer. The environment variable WM\_NCOMPPROCS will specify the number of cores to use for running the test harness. For `n` cores specified, `n` tutorials will be running in parallel on your computer. Since all the tests will still run on the same computer, make sure you have enough ressources to run `n` tutorials in parallel. Depending on the number of cores available, one might have to tweak some of the shell `limits` values. The command `ulimit -a` will show you the actual limits values imposed on your shell. Some limit values like `open files` (ulimit -n) or `max user processes` (ulimit -u) might need to be adjusted to some higher values. In doubt, consult with your local sysadmin.

Notes

The MediaWiki version of this file was generated using the following command:

   pandoc testHarness.org -s -S -f org -t mediawiki -o testHarness.mediawiki

The ASCII version of this file was generated using the following command:

   pandoc testHarness.org -N -s -S -f org -t asciidoc -o testHarness.txt

14 KiB Raw Blame History