A new design for the ThirdParty packages compilation and installation process:
==============================================================================
The main purpose of this new development is to build a complete ThirParty
packages set for foam-extend-3.1 using only the original package source
tarball and some patch files (when necessary).
A useful by-product of this development is also to provide some kind of
binary packaging of the ThirdParty packages. There are currently two types of
binary packaging generated by this compilation process: RPMs and compressed
tarballs (TGZs)
The RPM package manager was selected to develop a first prototype. RPM is
available for multiple flavors of Unix/Linux, and offers all the necessary
functionnality to configured, patch, compile and install source code packages.
One of the main requirement for this new development is that the whole process
needs to run and install in user-space, without the need to be root for
installing the packages.
Here is what's available:
a: A set of RPM spec files for specific ThirdParty packages.
b: A set of bash scripts to automate the complete sequence of downloading,
compiling, installing and generating RPMs.
c: An directory structure pre-configured and ready to proceed with the
download, compilation and installation of chosen ThirdParty packages for
OF-1.6-ext.
1: Quick description of the main scripts:
-----------------------------------------
a: AllMake:
Main wrapper script that will call AllMake.stage0 to AllMake.stage4
scripts in sequence.
b: AllMake.stage0:
This script is useful only for populating what I am calling the local "RPM
vault" with pre-generated RPMs.
This is the script written to address the following use case:
"I have some pre-generated RPM files, now what"
Basically, you call this script with a list of RPMs already generated by
whichever of the AllMake.stage(1-4) in order to populate the local RPMS
vault. Once in place in the RPM vault, these are the RPMs that will get
directly installed by the AllMake.stage(1-4) scripts, instead of being
regenerated by the normal compilation and install process of the
ThirdParty packages.
c: AllMake.stage1:
This script is taking care of the basic ThirdParty tools like compilers,
flex, bison, cmake , python, etc.
This compilation stage will generate .sh and .csh files that will be
sourced by your file settings.sh or settings.csh in order to initialize
the PATH and LD_LIBRARY_PATH environment variable for the stage1 packages.
d: AllMake.stage2:
This script is taking care of the MPI communication libraries. Right now,
only OpenMPI is supported.
This compilation stage will generate .sh and .csh files that will be
sourced by your file settings.sh or settings.csh in order to initialize
the PATH and LD_LIBRARY_PATH environment variable for the stage2 packages.
e: AllMake.stage3:
This script is taking care of the "standard" ThirdParty libraries like
metis, scotch, mesquite, etc.
This compilation stage will generate .sh and .csh files that will be
sourced by your file settings.sh or settings.csh in order to initialize
the PATH and LD_LIBRARY_PATH environment variable for the stage3 packages.
f: AllMake.stage4:
This script is taking care of Paraview and QT (and this step will take an
awfull long time to compile, honest...).
This compilation stage will generate .sh and .csh files that will be
sourced by your file settings.sh or settings.csh in order to initialize
the PATH and LD_LIBRARY_PATH environment variable for the stage4 packages.
g: tools/makeThirdPartyFunctionsForRPM:
A set of bash functions useful for wrapping the rpmbuild and rpm commands
2: Important notice:
--------------------
Every single ThirdParty package will be installed with a set of companion .sh
and .csh files one needs to source in order to properly initialize the PATH and
LD_LIBRARY_PATH environment variables for running the packages.
Some packages will also create environments variables necessary for the
compilation of some foam-extend libraries.
The scripts AllMake.stage[1-4] are taking care of refreshing the environment
variable after the installation of every single package, so you will not need to
manually refresh your foam-extend environment yourself when building individual
ThirdParty packages through the AllMake.stage[1-4] scripts.
The main Allwmake script for the compilation of foam-extend will also take care of
refreshing the environment variable before moving on to the compilation of the
main foam-extend libraries and applications.
However, if you decide to compile the libraries and/or applications manually, or
through the src/Allwmake or applications/Allwmake scripts, YOU MUST MAKE SURE
YOUR ENVIRONMENT VARIABLES FOR THE THIRDPARTY PACKAGES ARE UPDATED FIRST!
In that case, simply resource your bashrc or cshrc file once, before moving on
with the rest of your compilation activities.
3: The building and installation process overview:
--------------------------------------------------
For every ThirdParty packages, this is the basic process we will be going
through when starting the compilation from scratch:
a: Verify in the local "RPM vault" if a RPM file is available for the
required ThirdParty package.
b: If the RPM is available, and the package already installed, then there is
nothing to do for this package, and we proceed with the next package.
c: If the RPM is available, and the package is NOT already installed, then
we simply install it using the RPM and move on to the next package.
d: if the RPM is absent:
1: we verify if the source tarbal is available from the SOURCES "vault"
2: if it is not, we download the tarball using the specified URL.
3: we then proceed with the extraction, patching, configuration,
compilation, RPM generation, TGZ generation and installation of the
package.
Important notices:
- The generated RPM is always used for the installation, not the TGZ file.
- The compressed tarball (.tgz) file is generated for systems where
using rpm for installing packages as a non-root user might be a problem.
- The .rpm files will be located under the directory
$WM_THIRD_PARTY_DIR/rpmBuild/RPMS
- The .tgz files will be located under the directory
$WM_THIRD_PARTY_DIR/rpmBuild/TGZS
e: The default installation root directory is "$WM_THIRD_PARTY_DIR".
This can be overridden though when installing the RPM.
f: Through the modification of the AllMake.stage[1-4] scripts, it is
possible to specify a modified version of a .spec file for a given
package in order to tweak its local compilation or installation.
The modified .spec file will have to be installed under the SPECS
directory, and the corresponding invocation or the command rpm_make in
the AllMake.stage[1-4] file will need to be modified accordingly.
4: Relocating the RPM root installation directory:
--------------------------------------------------
Warning: This section is a bit advanced, and as of yet, I have no evidence that
this feature has ever been used by anybody. So please feel free to skip
this section if you don't need to relocate your ThirdParty packages
installation directory, which you should not have do on most
installation anyway.
a: All the generated RPMs can be relocated, meaning that you can override
the hard-coded root installation directory when using those RPMs for
installation.
b: You can check that the RPM is relocatable by using the command rpm -qip
thePackage.rpm. For example, from the cmake-2.8.3 package generated on one
of my machines, I will get:
> rpm -qip cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm| grep Relocations
Name : cmake Relocations: /home/beaudoin/foam/foam-extend-3.1/ThirdParty
The Relocations path is the actual location pointed by the
$WM_THIRD_PARTY_DIR on my virtual machine when I generated the RPM. It is
the indication that the RPM is relocatable. This path will end up being
hardcoded in the RPM because the environment variable was expanded before
generating the RPM.
This is the default root directory where the RPM will install its
"payload". This can be overridden using the 'rpm' command-line parameter
--relocate OLDPATH=NEWPATH .
For example, let's say I want to install my cmake-2.8.3 RPM under the root
directory /tmp/someDir instead. I will need to call the 'rpm' command like
this:
rpm -ivh ./cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm \
--relocate /home/beaudoin/foam/foam-extend-3.1/ThirdParty=/tmp/someDir
Even better, you can dig down the hard-coded path even deeper in order to
relocate the whole installation directory, down to the last hard-coded
subdirectory.
Just specify the whole path when using the --relocate parameter
So basically, you can install the RPM right under /usr if you want, hence
bypassing the default sequence of package subdirectories I have chosen in
order to stay close to the "traditional" ThirdParty layout.
5: Things to do:
----------------
a: Testing testing testing. This prototype was tested on the following
platforms:
Centos 5.5 64-bit (RPM based)
Mac OS X 10.6 (Snow Leopard) (non RPM based)
OpenSUSE 11.3 64-bit (RPM based)
RedHat Enterprise Linux 6 64-bit (RPM based)
Ubuntu 10.04 32-bit (non RPM based)
Ubuntu 10.04 64-bit (non RPM based)
b: Maybe adding some RPM dependencies might be useful. I have not explored
this yet.
c: Supplying compilation recipes with the source code for various OSes.
Lots of traffic on the Forum about this. This does not need to be that
complicated.
6: Gotchas and solutions:
-------------------------
1: It was reported that version 5.2.0 of rpm does not work well with the provided scripts.
rpm version 5.2.1 is working nicely.
2: Error messages from rpm:
Problem : error: can't create transaction lock on /var/lock/rpm/transaction
Solution: add the following entry in your file ~/.rpmmacros
%_rpmlock_path %{_dbpath}/__db.000
7: Reporting errors:
--------------------
Please report ThirdParty packages related bugs to the Bug tracking system
for foam-extend-3.1
http://sourceforge.net/apps/mantisbt/openfoam-extend/main_page.php
Martin Beaudoin
Rev 0: 05/2011
Rev 1: 08/2011
Rev 2: 05/2014
be16ba3f2e