exaFOAM/foam-extend4.1-coherent-io
Archived
1
1
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

ThirdParty packages: modification to the README.ThirdParty file

Browse source
This commit is contained in:
Martin Beaudoin 2011-08-14 13:01:30 -07:00
parent f3533582f8
commit 50b3827c85
1 changed files with 203 additions and 84 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

287
ThirdParty/README.ThirdParty vendored
View file

@ -1,129 +1,248 @@
A new design for the ThirdParty packages:
=========================================
A new design for the ThirdParty packages compilation and installation process:
==============================================================================
The main purpose of this new development is to build a complete ThirParty packages set for OpenFOAM 1.6-ext using only the original package source
tarball and some patch files ( when necessary).
The main purpose of this new development is to build a complete ThirParty
packages set for OpenFOAM 1.6-ext using only the original package source
tarball and some patch files (when necessary).
A useful by-product of this development is also to provide some kind of binary packaging of the ThirdParty packages.
A useful by-product of this development is also to provide some kind of
binary packaging of the ThirdParty packages. There are currently two types of
binary packaging generated by this compilation process: RPMs and compressed
tarballs (TGZs)
The RPM suite of tools was selected to develop a first prototype.
The whole process needs to run and install in user-space, without the need to be root for installing the packages.
The RPM package manager was selected to develop a first prototype. RPM is
available for multiple flavors of Unix/Linux, and offers all the necessary
functionnality to configured, patch, compile and install source code packages.
One of the main requirement for this new development is that the whole process
needs to run and install in user-space, without the need to be root for
installing the packages.
Here is what's available:
a: A set of rpm spec files for specific ThirdParty packages.
b: A suite of bash scripts to automate the complete sequence of downloading, compiling, installing and generating RPMs.
c: An empty directory structure pre-configured and ready to proceed with the download, compilation and installation of
chosen ThirdParty packages for OF-1.6-ext.
a: A set of RPM spec files for specific ThirdParty packages.
b: A set of bash scripts to automate the complete sequence of downloading,
compiling, installing and generating RPMs.
c: An directory structure pre-configured and ready to proceed with the
download, compilation and installation of chosen ThirdParty packages for
OF-1.6-ext.
Quick description of the main scripts:
--------------------------------------
1: Quick description of the main scripts:
-----------------------------------------
a: AllMake:
Main wrapper script that will call AllMake.stage0 to AllMake.stage4 scripts in sequence.
Main wrapper script that will call AllMake.stage0 to AllMake.stage4
scripts in sequence.
b: AllMake.stage0:
This script is useful only for populating what I am calling the local "RPM vault" with pre-generated RPMs.
This is the script written to address the use-case: "I have some pre-generated RPM files, now what"
Basically, you call this script with a list of RPMs generated by the AllMake.stage(1-4) in order to populate the local RPMS vault.
Once in place, these are the RPMs will be installed instead of proceeding with the standard compilation process.
This script is useful only for populating what I am calling the local "RPM
vault" with pre-generated RPMs.
This is the script written to address the following use case:
"I have some pre-generated RPM files, now what"
Basically, you call this script with a list of RPMs already generated by
whichever of the AllMake.stage(1-4) in order to populate the local RPMS
vault. Once in place in the RPM vault, these are the RPMs that will get
directly installed by the AllMake.stage(1-4) scripts, instead of being
regenerated by the normal compilation and install process of the
ThirdParty packages.
c: AllMake.stage1:
This script is taking care of the basic ThirdParty tools like compilers, cmake , python, etc.
If we ever need to override the local version of flex or bison, this is where we will add those additional packages.
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
If you need to compile the rest of the ThirdParty packages with a new gcc compiler, you will need to source those .sh or .csh file in before activating the other
AllMake.stage(2-4) scripts.
This script is taking care of the basic ThirdParty tools like compilers,
flex, bison, cmake , python, etc.
This compilation stage will generate .sh and .csh files that will be
sourced by your file settings.sh or settings.csh in order to initialize
the PATH and LD_LIBRARY_PATH environment variable for the stage1 packages.
d: AllMake.stage2:
This script is taking care of the MPI communication libraries. Right now, only OpenMPI is supported.
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
You will need to source those .sh or .csh file in before activating the other AllMake.stage(3-4) scripts because some packages depends on the communication library.
This script is taking care of the MPI communication libraries. Right now,
only OpenMPI is supported.
This compilation stage will generate .sh and .csh files that will be
sourced by your file settings.sh or settings.csh in order to initialize
the PATH and LD_LIBRARY_PATH environment variable for the stage2 packages.
e: AllMake.stage3:
This script is taking care of the "standard" ThirdParty libraries like metis, scotch, mesquite, etc.
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
You will need to source those .sh or .csh file in before compiling OpenFOAM because some "Make/options" files will refer to environment variable that are
specific to those packages
This script is taking care of the "standard" ThirdParty libraries like
metis, scotch, mesquite, etc.
This compilation stage will generate .sh and .csh files that will be
sourced by your file settings.sh or settings.csh in order to initialize
the PATH and LD_LIBRARY_PATH environment variable for the stage3 packages.
f: AllMake.stage4:
This script is taking care of Paraview and QT (and takes an awfull long time to compile, honest...).
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
You will need to source those .sh or .csh file in before compiling OpenFOAM because some "Make/options" files will refer to environment variable that are specific to those packages
This script is taking care of Paraview and QT (and this step will take an
awfull long time to compile, honest...).
This compilation stage will generate .sh and .csh files that will be
sourced by your file settings.sh or settings.csh in order to initialize
the PATH and LD_LIBRARY_PATH environment variable for the stage4 packages.
g: tools/makeThirdPartyFunctionsForRPM:
A suite of bash functions useful for wrapping the rpmbuild and rpm commands
For every packages, this is the basic process we will be going through when starting building the ThirdParty packages from scratch::
a: Verify in the local "RPM vault" if a RPM file is available for a given package
b: If it is, simply install this RPM and move on to the next package
c: if the RPM is absent:
i: verify if the source tarbal is available from the SOURCES "vault"
ii: if it is not, download the tarball using the specified URL.
iii: proceed with the extraction, patching, configuration, compilation, RPM generation and installation of the package.
The generated RPM is always used for installation.
A set of bash functions useful for wrapping the rpmbuild and rpm commands
Important notice:
Please note that a compressed tarball (.tgz) is now also generated for systems where using rpm for installing packages
as a non-root user might be a problem.
The .rpm files will be located under the directory $WM_THIRD_PARTY_DIR/rpmBuild/RPMS
The .tgz files will be located under the directory $WM_THIRD_PARTY_DIR/rpmBuild/TGZS
2: Important notice:
--------------------
Every single ThirdParty package will be installed with a set of companion .sh
and .csh files one needs to source in order to properly initialize the PATH and
LD_LIBRARY_PATH environment variables for running the packages.
d: The default installation root directory is "$WM_THIRD_PARTY_DIR". This can be overriden though when installing the RPM.
Some packages will also create environments variables necessary for the
compilation of some OpenFOAM libraries.
The scripts AllMake.stage[1-4] are taking care of refreshing the environment
variable after the installation of every single package, so you will not need to
manually refresh your OpenFOAM environment yourself when building individual
ThirdParty packages through the AllMake.stage[1-4] scripts.
The main Allwmake script for the compilation of OpenFOAM will also take care of
refreshing the environment variable before moving on to the compilation of the
main OpenFOAM libraries and applications.
However, if you decide to compile the libraries and/or applications manually, or
through the src/Allwmake or applications/Allwmake scripts, YOU MUST MAKE SURE
YOUR ENVIRONMENT VARIABLES FOR THE THIRDPARTY PACKAGES ARE UPDATED FIRST!
In that case, simply resource your bashrc or cshrc file once, before moving on
with the rest of your compilation activities.
Relocating the RPM root installation directory:
a: All the generated RPMs can be relocated, meaning that you can override the hard-coded root installation directory when using those RPMs for installation.
b: You can check that the RPM is relocatable by using the command rpm -qip thePackage.rpm. For example, from the cmake-2.8.3 package generated on one of my machine:
> rpm -qip cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm| grep Relocations
Name : cmake Relocations: /Users/beaudoin/Projets/SAMH/OpenFOAM/ThirdParty-1.6-ext-RPM-based
3: The building and installation process overview:
--------------------------------------------------
For every ThirdParty packages, this is the basic process we will be going
through when starting the compilation from scratch:
The Relocations path is the actual location pointed by the $WM_THIRD_PARTY_DIR on my laptop when I generated the RPM.
It is the indication that the RPM is relocatable
This path will end up being hardcoded in the RPM because the environment variable was expanded before generating the RPM..
This is the default root directory where the RPM will install its "payload"
This can be overridden using the 'rpm' command-line parameter --relocate OLDPATH=NEWPATH .
a: Verify in the local "RPM vault" if a RPM file is available for the
required ThirdParty package.
b: If the RPM is available, and the package already installed, then there is
nothing to do for this package, and we proceed with the next package.
c: If the RPM is available, and the package is NOT already installed, then
we simply install it using the RPM and move on to the next package.
d: if the RPM is absent:
1: we verify if the source tarbal is available from the SOURCES "vault"
2: if it is not, we download the tarball using the specified URL.
3: we then proceed with the extraction, patching, configuration,
compilation, RPM generation, TGZ generation and installation of the
package.
Important notices:
- The generated RPM is always used for the installation, not the TGZ file.
- The compressed tarball (.tgz) file is generated for systems where
using rpm for installing packages as a non-root user might be a problem.
- The .rpm files will be located under the directory
$WM_THIRD_PARTY_DIR/rpmBuild/RPMS
- The .tgz files will be located under the directory
$WM_THIRD_PARTY_DIR/rpmBuild/TGZS
e: The default installation root directory is "$WM_THIRD_PARTY_DIR".
This can be overridden though when installing the RPM.
For example, let's say you want to install the RPM under the root directory /tmp/someDir instead.
You will need to call the 'rpm' command like this:
f: Through the modification of the AllMake.stage[1-4] scripts, it is
possible to specify a modified version of a .spec file for a given
package in order to tweak its local compilation or installation.
The modified .spec file will have to be installed under the SPECS
directory, and the corresponding invocation or the command rpm_make in
the AllMake.stage[1-4] file will need to be modified accordingly.
4: Relocating the RPM root installation directory:
--------------------------------------------------
Warning: This section is a bit advanced, and as of yet, I have no evidence that
this feature has ever been used by anybody. So please feel free to skip
this section if you don't need to relocate your ThirdParty packages
installation directory, which you should not have do on most
installation anyway.
a: All the generated RPMs can be relocated, meaning that you can override
the hard-coded root installation directory when using those RPMs for
installation.
b: You can check that the RPM is relocatable by using the command rpm -qip
thePackage.rpm. For example, from the cmake-2.8.3 package generated on one
of my machines, I will get:
rpm -ivh ./cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm --relocate /Users/beaudoin/Projets/SAMH/OpenFOAM/ThirdParty-1.6-ext-RPM-based=/tmp/someDir
> rpm -qip cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm| grep Relocations
Name : cmake Relocations: /home/beaudoin/OpenFOAM/OpenFOAM-1.6-ext/ThirdParty
Even better, you can dig down the hard-coded path even deeper in order to relocate the whole installation directory, down to the last hard-coded subdirectory.
Just specify the whole path when using the --relocate parameter
So basically, you can install the RPM right under /usr if you want, hence bypassing the default sequence of package subdirectories I have chosen in order to stay
close to the "traditional" ThirdParty layout.
The Relocations path is the actual location pointed by the
$WM_THIRD_PARTY_DIR on my virtual machine when I generated the RPM. It is
the indication that the RPM is relocatable. This path will end up being
hardcoded in the RPM because the environment variable was expanded before
generating the RPM.
This is the default root directory where the RPM will install its
"payload". This can be overridden using the 'rpm' command-line parameter
--relocate OLDPATH=NEWPATH .
9: Things to do:
a: Testing testing testing. This prototype was tested on the following platforms:
For example, let's say I want to install my cmake-2.8.3 RPM under the root
directory /tmp/someDir instead. I will need to call the 'rpm' command like
this:
Mac OS X 10.6 (Snow Leopard) (non RPM based)
Ubuntu 10.04 32bit (non RPM based)
Centos 5.5 64bit (RPM based)
OpenSUSE 11.3 64bit (RPM based)
rpm -ivh ./cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm \
--relocate /home/beaudoin/OpenFOAM/OpenFOAM-1.6-ext/ThirdParty=/tmp/someDir
Even better, you can dig down the hard-coded path even deeper in order to
relocate the whole installation directory, down to the last hard-coded
subdirectory.
Just specify the whole path when using the --relocate parameter
So basically, you can install the RPM right under /usr if you want, hence
bypassing the default sequence of package subdirectories I have chosen in
order to stay close to the "traditional" ThirdParty layout.
5: Things to do:
----------------
a: Testing testing testing. This prototype was tested on the following
platforms:
Centos 5.5 64-bit (RPM based)
Mac OS X 10.6 (Snow Leopard) (non RPM based)
OpenSUSE 11.3 64-bit (RPM based)
RedHat Enterprise Linux 6 64-bit (RPM based)
Ubuntu 10.04 32-bit (non RPM based)
Ubuntu 10.04 64-bit (non RPM based)
b: Maybe adding some RPM dependencies might be useful. I have not explored this yet.
b: Maybe adding some RPM dependencies might be useful. I have not explored
this yet.
10: Gotchas and solutions:
c: Supplying compilation recipes with the source code for various OSes.
Lots of traffic on the Forum about this. This does not need to be that
complicated.
6: Gotchas and solutions:
-------------------------
1: It was reported that version 5.2.0 of rpm does not work well with the provided scripts.
rpm version 5.2.1 works nicely.
rpm version 5.2.1 is working nicely.
2: Error messages from rpm:
---------------------------
Problem : error: can't create transaction lock on /var/lock/rpm/transaction
Solution: add the following entry in your file ~/.rpmmacros
Problem : error: can't create transaction lock on /var/lock/rpm/transaction
Solution: add the following entry in your file ~/.rpmmacros
%_rpmlock_path %{_dbpath}/__db.000
7: Reporting errors:
--------------------
To be continued...
Please report ThirdParty packages related bugs to the Bug tracking system
for OpenFOAM-1.6-ext.
http://sourceforge.net/apps/mantisbt/openfoam-extend/main_page.php
Martin Beaudoin
May 2011
Rev 0: 05/2011
Rev 1: 08/2011