ThirdParty packages: modification to the README.ThirdParty file
This commit is contained in:
Martin Beaudoin
parent
f3533582f8
commit
50b3827c85
1 changed files with 203 additions and 84 deletions
267
ThirdParty/README.ThirdParty
vendored
267
ThirdParty/README.ThirdParty
vendored
|
|
@ -1,129 +1,248 @@
|
||||||
A new design for the ThirdParty packages:
|
A new design for the ThirdParty packages compilation and installation process:
|
||||||
=========================================
|
==============================================================================
|
||||||
|
|
||||||
The main purpose of this new development is to build a complete ThirParty packages set for OpenFOAM 1.6-ext using only the original package source
|
The main purpose of this new development is to build a complete ThirParty
|
||||||
|
packages set for OpenFOAM 1.6-ext using only the original package source
|
||||||
tarball and some patch files (when necessary).
|
tarball and some patch files (when necessary).
|
||||||
|
|
||||||
A useful by-product of this development is also to provide some kind of binary packaging of the ThirdParty packages.
|
A useful by-product of this development is also to provide some kind of
|
||||||
|
binary packaging of the ThirdParty packages. There are currently two types of
|
||||||
|
binary packaging generated by this compilation process: RPMs and compressed
|
||||||
|
tarballs (TGZs)
|
||||||
|
|
||||||
The RPM suite of tools was selected to develop a first prototype.
|
The RPM package manager was selected to develop a first prototype. RPM is
|
||||||
The whole process needs to run and install in user-space, without the need to be root for installing the packages.
|
available for multiple flavors of Unix/Linux, and offers all the necessary
|
||||||
|
functionnality to configured, patch, compile and install source code packages.
|
||||||
|
|
||||||
|
One of the main requirement for this new development is that the whole process
|
||||||
|
needs to run and install in user-space, without the need to be root for
|
||||||
|
installing the packages.
|
||||||
|
|
||||||
Here is what's available:
|
Here is what's available:
|
||||||
|
|
||||||
a: A set of rpm spec files for specific ThirdParty packages.
|
a: A set of RPM spec files for specific ThirdParty packages.
|
||||||
b: A suite of bash scripts to automate the complete sequence of downloading, compiling, installing and generating RPMs.
|
b: A set of bash scripts to automate the complete sequence of downloading,
|
||||||
c: An empty directory structure pre-configured and ready to proceed with the download, compilation and installation of
|
compiling, installing and generating RPMs.
|
||||||
chosen ThirdParty packages for OF-1.6-ext.
|
c: An directory structure pre-configured and ready to proceed with the
|
||||||
|
download, compilation and installation of chosen ThirdParty packages for
|
||||||
|
OF-1.6-ext.
|
||||||
|
|
||||||
Quick description of the main scripts:
|
|
||||||
--------------------------------------
|
1: Quick description of the main scripts:
|
||||||
|
-----------------------------------------
|
||||||
a: AllMake:
|
a: AllMake:
|
||||||
Main wrapper script that will call AllMake.stage0 to AllMake.stage4 scripts in sequence.
|
Main wrapper script that will call AllMake.stage0 to AllMake.stage4
|
||||||
|
scripts in sequence.
|
||||||
|
|
||||||
b: AllMake.stage0:
|
b: AllMake.stage0:
|
||||||
This script is useful only for populating what I am calling the local "RPM vault" with pre-generated RPMs.
|
This script is useful only for populating what I am calling the local "RPM
|
||||||
This is the script written to address the use-case: "I have some pre-generated RPM files, now what"
|
vault" with pre-generated RPMs.
|
||||||
Basically, you call this script with a list of RPMs generated by the AllMake.stage(1-4) in order to populate the local RPMS vault.
|
|
||||||
Once in place, these are the RPMs will be installed instead of proceeding with the standard compilation process.
|
This is the script written to address the following use case:
|
||||||
|
"I have some pre-generated RPM files, now what"
|
||||||
|
|
||||||
|
Basically, you call this script with a list of RPMs already generated by
|
||||||
|
whichever of the AllMake.stage(1-4) in order to populate the local RPMS
|
||||||
|
vault. Once in place in the RPM vault, these are the RPMs that will get
|
||||||
|
directly installed by the AllMake.stage(1-4) scripts, instead of being
|
||||||
|
regenerated by the normal compilation and install process of the
|
||||||
|
ThirdParty packages.
|
||||||
|
|
||||||
c: AllMake.stage1:
|
c: AllMake.stage1:
|
||||||
This script is taking care of the basic ThirdParty tools like compilers, cmake , python, etc.
|
This script is taking care of the basic ThirdParty tools like compilers,
|
||||||
If we ever need to override the local version of flex or bison, this is where we will add those additional packages.
|
flex, bison, cmake , python, etc.
|
||||||
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
|
This compilation stage will generate .sh and .csh files that will be
|
||||||
If you need to compile the rest of the ThirdParty packages with a new gcc compiler, you will need to source those .sh or .csh file in before activating the other
|
sourced by your file settings.sh or settings.csh in order to initialize
|
||||||
AllMake.stage(2-4) scripts.
|
the PATH and LD_LIBRARY_PATH environment variable for the stage1 packages.
|
||||||
|
|
||||||
d: AllMake.stage2:
|
d: AllMake.stage2:
|
||||||
This script is taking care of the MPI communication libraries. Right now, only OpenMPI is supported.
|
This script is taking care of the MPI communication libraries. Right now,
|
||||||
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
|
only OpenMPI is supported.
|
||||||
You will need to source those .sh or .csh file in before activating the other AllMake.stage(3-4) scripts because some packages depends on the communication library.
|
This compilation stage will generate .sh and .csh files that will be
|
||||||
|
sourced by your file settings.sh or settings.csh in order to initialize
|
||||||
|
the PATH and LD_LIBRARY_PATH environment variable for the stage2 packages.
|
||||||
|
|
||||||
e: AllMake.stage3:
|
e: AllMake.stage3:
|
||||||
This script is taking care of the "standard" ThirdParty libraries like metis, scotch, mesquite, etc.
|
This script is taking care of the "standard" ThirdParty libraries like
|
||||||
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
|
metis, scotch, mesquite, etc.
|
||||||
You will need to source those .sh or .csh file in before compiling OpenFOAM because some "Make/options" files will refer to environment variable that are
|
This compilation stage will generate .sh and .csh files that will be
|
||||||
specific to those packages
|
sourced by your file settings.sh or settings.csh in order to initialize
|
||||||
|
the PATH and LD_LIBRARY_PATH environment variable for the stage3 packages.
|
||||||
|
|
||||||
f: AllMake.stage4:
|
f: AllMake.stage4:
|
||||||
This script is taking care of Paraview and QT (and takes an awfull long time to compile, honest...).
|
This script is taking care of Paraview and QT (and this step will take an
|
||||||
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
|
awfull long time to compile, honest...).
|
||||||
You will need to source those .sh or .csh file in before compiling OpenFOAM because some "Make/options" files will refer to environment variable that are specific to those packages
|
This compilation stage will generate .sh and .csh files that will be
|
||||||
|
sourced by your file settings.sh or settings.csh in order to initialize
|
||||||
|
the PATH and LD_LIBRARY_PATH environment variable for the stage4 packages.
|
||||||
|
|
||||||
g: tools/makeThirdPartyFunctionsForRPM:
|
g: tools/makeThirdPartyFunctionsForRPM:
|
||||||
A suite of bash functions useful for wrapping the rpmbuild and rpm commands
|
A set of bash functions useful for wrapping the rpmbuild and rpm commands
|
||||||
|
|
||||||
|
|
||||||
For every packages, this is the basic process we will be going through when starting building the ThirdParty packages from scratch::
|
2: Important notice:
|
||||||
a: Verify in the local "RPM vault" if a RPM file is available for a given package
|
--------------------
|
||||||
b: If it is, simply install this RPM and move on to the next package
|
Every single ThirdParty package will be installed with a set of companion .sh
|
||||||
c: if the RPM is absent:
|
and .csh files one needs to source in order to properly initialize the PATH and
|
||||||
i: verify if the source tarbal is available from the SOURCES "vault"
|
LD_LIBRARY_PATH environment variables for running the packages.
|
||||||
ii: if it is not, download the tarball using the specified URL.
|
|
||||||
iii: proceed with the extraction, patching, configuration, compilation, RPM generation and installation of the package.
|
|
||||||
The generated RPM is always used for installation.
|
|
||||||
|
|
||||||
Important notice:
|
Some packages will also create environments variables necessary for the
|
||||||
Please note that a compressed tarball (.tgz) is now also generated for systems where using rpm for installing packages
|
compilation of some OpenFOAM libraries.
|
||||||
as a non-root user might be a problem.
|
|
||||||
|
|
||||||
The .rpm files will be located under the directory $WM_THIRD_PARTY_DIR/rpmBuild/RPMS
|
The scripts AllMake.stage[1-4] are taking care of refreshing the environment
|
||||||
The .tgz files will be located under the directory $WM_THIRD_PARTY_DIR/rpmBuild/TGZS
|
variable after the installation of every single package, so you will not need to
|
||||||
|
manually refresh your OpenFOAM environment yourself when building individual
|
||||||
|
ThirdParty packages through the AllMake.stage[1-4] scripts.
|
||||||
|
|
||||||
d: The default installation root directory is "$WM_THIRD_PARTY_DIR". This can be overriden though when installing the RPM.
|
The main Allwmake script for the compilation of OpenFOAM will also take care of
|
||||||
|
refreshing the environment variable before moving on to the compilation of the
|
||||||
|
main OpenFOAM libraries and applications.
|
||||||
|
|
||||||
Relocating the RPM root installation directory:
|
However, if you decide to compile the libraries and/or applications manually, or
|
||||||
a: All the generated RPMs can be relocated, meaning that you can override the hard-coded root installation directory when using those RPMs for installation.
|
through the src/Allwmake or applications/Allwmake scripts, YOU MUST MAKE SURE
|
||||||
b: You can check that the RPM is relocatable by using the command rpm -qip thePackage.rpm. For example, from the cmake-2.8.3 package generated on one of my machine:
|
YOUR ENVIRONMENT VARIABLES FOR THE THIRDPARTY PACKAGES ARE UPDATED FIRST!
|
||||||
|
|
||||||
|
In that case, simply resource your bashrc or cshrc file once, before moving on
|
||||||
|
with the rest of your compilation activities.
|
||||||
|
|
||||||
|
|
||||||
|
3: The building and installation process overview:
|
||||||
|
--------------------------------------------------
|
||||||
|
For every ThirdParty packages, this is the basic process we will be going
|
||||||
|
through when starting the compilation from scratch:
|
||||||
|
|
||||||
|
a: Verify in the local "RPM vault" if a RPM file is available for the
|
||||||
|
required ThirdParty package.
|
||||||
|
|
||||||
|
b: If the RPM is available, and the package already installed, then there is
|
||||||
|
nothing to do for this package, and we proceed with the next package.
|
||||||
|
|
||||||
|
c: If the RPM is available, and the package is NOT already installed, then
|
||||||
|
we simply install it using the RPM and move on to the next package.
|
||||||
|
|
||||||
|
d: if the RPM is absent:
|
||||||
|
1: we verify if the source tarbal is available from the SOURCES "vault"
|
||||||
|
2: if it is not, we download the tarball using the specified URL.
|
||||||
|
3: we then proceed with the extraction, patching, configuration,
|
||||||
|
compilation, RPM generation, TGZ generation and installation of the
|
||||||
|
package.
|
||||||
|
|
||||||
|
Important notices:
|
||||||
|
- The generated RPM is always used for the installation, not the TGZ file.
|
||||||
|
|
||||||
|
- The compressed tarball (.tgz) file is generated for systems where
|
||||||
|
using rpm for installing packages as a non-root user might be a problem.
|
||||||
|
|
||||||
|
- The .rpm files will be located under the directory
|
||||||
|
$WM_THIRD_PARTY_DIR/rpmBuild/RPMS
|
||||||
|
|
||||||
|
- The .tgz files will be located under the directory
|
||||||
|
$WM_THIRD_PARTY_DIR/rpmBuild/TGZS
|
||||||
|
|
||||||
|
e: The default installation root directory is "$WM_THIRD_PARTY_DIR".
|
||||||
|
This can be overridden though when installing the RPM.
|
||||||
|
|
||||||
|
f: Through the modification of the AllMake.stage[1-4] scripts, it is
|
||||||
|
possible to specify a modified version of a .spec file for a given
|
||||||
|
package in order to tweak its local compilation or installation.
|
||||||
|
|
||||||
|
The modified .spec file will have to be installed under the SPECS
|
||||||
|
directory, and the corresponding invocation or the command rpm_make in
|
||||||
|
the AllMake.stage[1-4] file will need to be modified accordingly.
|
||||||
|
|
||||||
|
|
||||||
|
4: Relocating the RPM root installation directory:
|
||||||
|
--------------------------------------------------
|
||||||
|
|
||||||
|
Warning: This section is a bit advanced, and as of yet, I have no evidence that
|
||||||
|
this feature has ever been used by anybody. So please feel free to skip
|
||||||
|
this section if you don't need to relocate your ThirdParty packages
|
||||||
|
installation directory, which you should not have do on most
|
||||||
|
installation anyway.
|
||||||
|
|
||||||
|
a: All the generated RPMs can be relocated, meaning that you can override
|
||||||
|
the hard-coded root installation directory when using those RPMs for
|
||||||
|
installation.
|
||||||
|
|
||||||
|
b: You can check that the RPM is relocatable by using the command rpm -qip
|
||||||
|
thePackage.rpm. For example, from the cmake-2.8.3 package generated on one
|
||||||
|
of my machines, I will get:
|
||||||
|
|
||||||
> rpm -qip cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm| grep Relocations
|
> rpm -qip cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm| grep Relocations
|
||||||
Name : cmake Relocations: /Users/beaudoin/Projets/SAMH/OpenFOAM/ThirdParty-1.6-ext-RPM-based
|
Name : cmake Relocations: /home/beaudoin/OpenFOAM/OpenFOAM-1.6-ext/ThirdParty
|
||||||
|
|
||||||
The Relocations path is the actual location pointed by the $WM_THIRD_PARTY_DIR on my laptop when I generated the RPM.
|
The Relocations path is the actual location pointed by the
|
||||||
It is the indication that the RPM is relocatable
|
$WM_THIRD_PARTY_DIR on my virtual machine when I generated the RPM. It is
|
||||||
This path will end up being hardcoded in the RPM because the environment variable was expanded before generating the RPM..
|
the indication that the RPM is relocatable. This path will end up being
|
||||||
This is the default root directory where the RPM will install its "payload"
|
hardcoded in the RPM because the environment variable was expanded before
|
||||||
This can be overridden using the 'rpm' command-line parameter --relocate OLDPATH=NEWPATH .
|
generating the RPM.
|
||||||
|
|
||||||
For example, let's say you want to install the RPM under the root directory /tmp/someDir instead.
|
This is the default root directory where the RPM will install its
|
||||||
You will need to call the 'rpm' command like this:
|
"payload". This can be overridden using the 'rpm' command-line parameter
|
||||||
|
--relocate OLDPATH=NEWPATH .
|
||||||
|
|
||||||
rpm -ivh ./cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm --relocate /Users/beaudoin/Projets/SAMH/OpenFOAM/ThirdParty-1.6-ext-RPM-based=/tmp/someDir
|
For example, let's say I want to install my cmake-2.8.3 RPM under the root
|
||||||
|
directory /tmp/someDir instead. I will need to call the 'rpm' command like
|
||||||
|
this:
|
||||||
|
|
||||||
Even better, you can dig down the hard-coded path even deeper in order to relocate the whole installation directory, down to the last hard-coded subdirectory.
|
rpm -ivh ./cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm \
|
||||||
|
--relocate /home/beaudoin/OpenFOAM/OpenFOAM-1.6-ext/ThirdParty=/tmp/someDir
|
||||||
|
|
||||||
|
Even better, you can dig down the hard-coded path even deeper in order to
|
||||||
|
relocate the whole installation directory, down to the last hard-coded
|
||||||
|
subdirectory.
|
||||||
Just specify the whole path when using the --relocate parameter
|
Just specify the whole path when using the --relocate parameter
|
||||||
So basically, you can install the RPM right under /usr if you want, hence bypassing the default sequence of package subdirectories I have chosen in order to stay
|
So basically, you can install the RPM right under /usr if you want, hence
|
||||||
close to the "traditional" ThirdParty layout.
|
bypassing the default sequence of package subdirectories I have chosen in
|
||||||
|
order to stay close to the "traditional" ThirdParty layout.
|
||||||
|
|
||||||
9: Things to do:
|
|
||||||
a: Testing testing testing. This prototype was tested on the following platforms:
|
|
||||||
|
|
||||||
|
5: Things to do:
|
||||||
|
----------------
|
||||||
|
a: Testing testing testing. This prototype was tested on the following
|
||||||
|
platforms:
|
||||||
|
|
||||||
|
Centos 5.5 64-bit (RPM based)
|
||||||
Mac OS X 10.6 (Snow Leopard) (non RPM based)
|
Mac OS X 10.6 (Snow Leopard) (non RPM based)
|
||||||
Ubuntu 10.04 32bit (non RPM based)
|
OpenSUSE 11.3 64-bit (RPM based)
|
||||||
Centos 5.5 64bit (RPM based)
|
RedHat Enterprise Linux 6 64-bit (RPM based)
|
||||||
OpenSUSE 11.3 64bit (RPM based)
|
Ubuntu 10.04 32-bit (non RPM based)
|
||||||
|
Ubuntu 10.04 64-bit (non RPM based)
|
||||||
|
|
||||||
b: Maybe adding some RPM dependencies might be useful. I have not explored this yet.
|
b: Maybe adding some RPM dependencies might be useful. I have not explored
|
||||||
|
this yet.
|
||||||
|
|
||||||
|
c: Supplying compilation recipes with the source code for various OSes.
|
||||||
|
Lots of traffic on the Forum about this. This does not need to be that
|
||||||
|
complicated.
|
||||||
|
|
||||||
|
|
||||||
10: Gotchas and solutions:
|
|
||||||
|
6: Gotchas and solutions:
|
||||||
|
-------------------------
|
||||||
|
|
||||||
1: It was reported that version 5.2.0 of rpm does not work well with the provided scripts.
|
1: It was reported that version 5.2.0 of rpm does not work well with the provided scripts.
|
||||||
rpm version 5.2.1 works nicely.
|
rpm version 5.2.1 is working nicely.
|
||||||
|
|
||||||
|
|
||||||
2: Error messages from rpm:
|
2: Error messages from rpm:
|
||||||
---------------------------
|
|
||||||
|
|
||||||
Problem : error: can't create transaction lock on /var/lock/rpm/transaction
|
Problem : error: can't create transaction lock on /var/lock/rpm/transaction
|
||||||
|
|
||||||
Solution: add the following entry in your file ~/.rpmmacros
|
Solution: add the following entry in your file ~/.rpmmacros
|
||||||
|
|
||||||
%_rpmlock_path %{_dbpath}/__db.000
|
%_rpmlock_path %{_dbpath}/__db.000
|
||||||
|
|
||||||
|
|
||||||
|
7: Reporting errors:
|
||||||
|
--------------------
|
||||||
|
|
||||||
|
Please report ThirdParty packages related bugs to the Bug tracking system
|
||||||
|
for OpenFOAM-1.6-ext.
|
||||||
|
http://sourceforge.net/apps/mantisbt/openfoam-extend/main_page.php
|
||||||
|
|
||||||
To be continued...
|
|
||||||
|
|
||||||
|
|
||||||
Martin Beaudoin
|
Martin Beaudoin
|
||||||
May 2011
|
Rev 0: 05/2011
|
||||||
|
Rev 1: 08/2011
|
||||||
|
|
|
||||||
Reference in a new issue