ThirdParty packages: modification to the README.ThirdParty file
This commit is contained in:
Martin Beaudoin
parent
f3533582f8
commit
50b3827c85
1 changed files with 203 additions and 84 deletions
293
ThirdParty/README.ThirdParty
vendored
293
ThirdParty/README.ThirdParty
vendored
|
|
@ -1,129 +1,248 @@
|
|||
A new design for the ThirdParty packages:
|
||||
=========================================
|
||||
A new design for the ThirdParty packages compilation and installation process:
|
||||
==============================================================================
|
||||
|
||||
The main purpose of this new development is to build a complete ThirParty packages set for OpenFOAM 1.6-ext using only the original package source
|
||||
tarball and some patch files ( when necessary).
|
||||
The main purpose of this new development is to build a complete ThirParty
|
||||
packages set for OpenFOAM 1.6-ext using only the original package source
|
||||
tarball and some patch files (when necessary).
|
||||
|
||||
A useful by-product of this development is also to provide some kind of binary packaging of the ThirdParty packages.
|
||||
A useful by-product of this development is also to provide some kind of
|
||||
binary packaging of the ThirdParty packages. There are currently two types of
|
||||
binary packaging generated by this compilation process: RPMs and compressed
|
||||
tarballs (TGZs)
|
||||
|
||||
The RPM suite of tools was selected to develop a first prototype.
|
||||
The whole process needs to run and install in user-space, without the need to be root for installing the packages.
|
||||
The RPM package manager was selected to develop a first prototype. RPM is
|
||||
available for multiple flavors of Unix/Linux, and offers all the necessary
|
||||
functionnality to configured, patch, compile and install source code packages.
|
||||
|
||||
One of the main requirement for this new development is that the whole process
|
||||
needs to run and install in user-space, without the need to be root for
|
||||
installing the packages.
|
||||
|
||||
Here is what's available:
|
||||
|
||||
a: A set of rpm spec files for specific ThirdParty packages.
|
||||
b: A suite of bash scripts to automate the complete sequence of downloading, compiling, installing and generating RPMs.
|
||||
c: An empty directory structure pre-configured and ready to proceed with the download, compilation and installation of
|
||||
chosen ThirdParty packages for OF-1.6-ext.
|
||||
a: A set of RPM spec files for specific ThirdParty packages.
|
||||
b: A set of bash scripts to automate the complete sequence of downloading,
|
||||
compiling, installing and generating RPMs.
|
||||
c: An directory structure pre-configured and ready to proceed with the
|
||||
download, compilation and installation of chosen ThirdParty packages for
|
||||
OF-1.6-ext.
|
||||
|
||||
Quick description of the main scripts:
|
||||
--------------------------------------
|
||||
|
||||
1: Quick description of the main scripts:
|
||||
-----------------------------------------
|
||||
a: AllMake:
|
||||
Main wrapper script that will call AllMake.stage0 to AllMake.stage4 scripts in sequence.
|
||||
Main wrapper script that will call AllMake.stage0 to AllMake.stage4
|
||||
scripts in sequence.
|
||||
|
||||
b: AllMake.stage0:
|
||||
This script is useful only for populating what I am calling the local "RPM vault" with pre-generated RPMs.
|
||||
This is the script written to address the use-case: "I have some pre-generated RPM files, now what"
|
||||
Basically, you call this script with a list of RPMs generated by the AllMake.stage(1-4) in order to populate the local RPMS vault.
|
||||
Once in place, these are the RPMs will be installed instead of proceeding with the standard compilation process.
|
||||
This script is useful only for populating what I am calling the local "RPM
|
||||
vault" with pre-generated RPMs.
|
||||
|
||||
This is the script written to address the following use case:
|
||||
"I have some pre-generated RPM files, now what"
|
||||
|
||||
Basically, you call this script with a list of RPMs already generated by
|
||||
whichever of the AllMake.stage(1-4) in order to populate the local RPMS
|
||||
vault. Once in place in the RPM vault, these are the RPMs that will get
|
||||
directly installed by the AllMake.stage(1-4) scripts, instead of being
|
||||
regenerated by the normal compilation and install process of the
|
||||
ThirdParty packages.
|
||||
|
||||
c: AllMake.stage1:
|
||||
This script is taking care of the basic ThirdParty tools like compilers, cmake , python, etc.
|
||||
If we ever need to override the local version of flex or bison, this is where we will add those additional packages.
|
||||
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
|
||||
If you need to compile the rest of the ThirdParty packages with a new gcc compiler, you will need to source those .sh or .csh file in before activating the other
|
||||
AllMake.stage(2-4) scripts.
|
||||
This script is taking care of the basic ThirdParty tools like compilers,
|
||||
flex, bison, cmake , python, etc.
|
||||
This compilation stage will generate .sh and .csh files that will be
|
||||
sourced by your file settings.sh or settings.csh in order to initialize
|
||||
the PATH and LD_LIBRARY_PATH environment variable for the stage1 packages.
|
||||
|
||||
d: AllMake.stage2:
|
||||
This script is taking care of the MPI communication libraries. Right now, only OpenMPI is supported.
|
||||
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
|
||||
You will need to source those .sh or .csh file in before activating the other AllMake.stage(3-4) scripts because some packages depends on the communication library.
|
||||
This script is taking care of the MPI communication libraries. Right now,
|
||||
only OpenMPI is supported.
|
||||
This compilation stage will generate .sh and .csh files that will be
|
||||
sourced by your file settings.sh or settings.csh in order to initialize
|
||||
the PATH and LD_LIBRARY_PATH environment variable for the stage2 packages.
|
||||
|
||||
e: AllMake.stage3:
|
||||
This script is taking care of the "standard" ThirdParty libraries like metis, scotch, mesquite, etc.
|
||||
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
|
||||
You will need to source those .sh or .csh file in before compiling OpenFOAM because some "Make/options" files will refer to environment variable that are
|
||||
specific to those packages
|
||||
This script is taking care of the "standard" ThirdParty libraries like
|
||||
metis, scotch, mesquite, etc.
|
||||
This compilation stage will generate .sh and .csh files that will be
|
||||
sourced by your file settings.sh or settings.csh in order to initialize
|
||||
the PATH and LD_LIBRARY_PATH environment variable for the stage3 packages.
|
||||
|
||||
f: AllMake.stage4:
|
||||
This script is taking care of Paraview and QT (and takes an awfull long time to compile, honest...).
|
||||
This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable.
|
||||
You will need to source those .sh or .csh file in before compiling OpenFOAM because some "Make/options" files will refer to environment variable that are specific to those packages
|
||||
This script is taking care of Paraview and QT (and this step will take an
|
||||
awfull long time to compile, honest...).
|
||||
This compilation stage will generate .sh and .csh files that will be
|
||||
sourced by your file settings.sh or settings.csh in order to initialize
|
||||
the PATH and LD_LIBRARY_PATH environment variable for the stage4 packages.
|
||||
|
||||
g: tools/makeThirdPartyFunctionsForRPM:
|
||||
A suite of bash functions useful for wrapping the rpmbuild and rpm commands
|
||||
A set of bash functions useful for wrapping the rpmbuild and rpm commands
|
||||
|
||||
|
||||
For every packages, this is the basic process we will be going through when starting building the ThirdParty packages from scratch::
|
||||
a: Verify in the local "RPM vault" if a RPM file is available for a given package
|
||||
b: If it is, simply install this RPM and move on to the next package
|
||||
c: if the RPM is absent:
|
||||
i: verify if the source tarbal is available from the SOURCES "vault"
|
||||
ii: if it is not, download the tarball using the specified URL.
|
||||
iii: proceed with the extraction, patching, configuration, compilation, RPM generation and installation of the package.
|
||||
The generated RPM is always used for installation.
|
||||
2: Important notice:
|
||||
--------------------
|
||||
Every single ThirdParty package will be installed with a set of companion .sh
|
||||
and .csh files one needs to source in order to properly initialize the PATH and
|
||||
LD_LIBRARY_PATH environment variables for running the packages.
|
||||
|
||||
Important notice:
|
||||
Please note that a compressed tarball (.tgz) is now also generated for systems where using rpm for installing packages
|
||||
as a non-root user might be a problem.
|
||||
Some packages will also create environments variables necessary for the
|
||||
compilation of some OpenFOAM libraries.
|
||||
|
||||
The .rpm files will be located under the directory $WM_THIRD_PARTY_DIR/rpmBuild/RPMS
|
||||
The .tgz files will be located under the directory $WM_THIRD_PARTY_DIR/rpmBuild/TGZS
|
||||
The scripts AllMake.stage[1-4] are taking care of refreshing the environment
|
||||
variable after the installation of every single package, so you will not need to
|
||||
manually refresh your OpenFOAM environment yourself when building individual
|
||||
ThirdParty packages through the AllMake.stage[1-4] scripts.
|
||||
|
||||
d: The default installation root directory is "$WM_THIRD_PARTY_DIR". This can be overriden though when installing the RPM.
|
||||
The main Allwmake script for the compilation of OpenFOAM will also take care of
|
||||
refreshing the environment variable before moving on to the compilation of the
|
||||
main OpenFOAM libraries and applications.
|
||||
|
||||
Relocating the RPM root installation directory:
|
||||
a: All the generated RPMs can be relocated, meaning that you can override the hard-coded root installation directory when using those RPMs for installation.
|
||||
b: You can check that the RPM is relocatable by using the command rpm -qip thePackage.rpm. For example, from the cmake-2.8.3 package generated on one of my machine:
|
||||
However, if you decide to compile the libraries and/or applications manually, or
|
||||
through the src/Allwmake or applications/Allwmake scripts, YOU MUST MAKE SURE
|
||||
YOUR ENVIRONMENT VARIABLES FOR THE THIRDPARTY PACKAGES ARE UPDATED FIRST!
|
||||
|
||||
> rpm -qip cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm| grep Relocations
|
||||
Name : cmake Relocations: /Users/beaudoin/Projets/SAMH/OpenFOAM/ThirdParty-1.6-ext-RPM-based
|
||||
|
||||
The Relocations path is the actual location pointed by the $WM_THIRD_PARTY_DIR on my laptop when I generated the RPM.
|
||||
It is the indication that the RPM is relocatable
|
||||
This path will end up being hardcoded in the RPM because the environment variable was expanded before generating the RPM..
|
||||
This is the default root directory where the RPM will install its "payload"
|
||||
This can be overridden using the 'rpm' command-line parameter --relocate OLDPATH=NEWPATH .
|
||||
|
||||
For example, let's say you want to install the RPM under the root directory /tmp/someDir instead.
|
||||
You will need to call the 'rpm' command like this:
|
||||
|
||||
rpm -ivh ./cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm --relocate /Users/beaudoin/Projets/SAMH/OpenFOAM/ThirdParty-1.6-ext-RPM-based=/tmp/someDir
|
||||
|
||||
Even better, you can dig down the hard-coded path even deeper in order to relocate the whole installation directory, down to the last hard-coded subdirectory.
|
||||
Just specify the whole path when using the --relocate parameter
|
||||
So basically, you can install the RPM right under /usr if you want, hence bypassing the default sequence of package subdirectories I have chosen in order to stay
|
||||
close to the "traditional" ThirdParty layout.
|
||||
|
||||
9: Things to do:
|
||||
a: Testing testing testing. This prototype was tested on the following platforms:
|
||||
|
||||
Mac OS X 10.6 (Snow Leopard) (non RPM based)
|
||||
Ubuntu 10.04 32bit (non RPM based)
|
||||
Centos 5.5 64bit (RPM based)
|
||||
OpenSUSE 11.3 64bit (RPM based)
|
||||
|
||||
b: Maybe adding some RPM dependencies might be useful. I have not explored this yet.
|
||||
In that case, simply resource your bashrc or cshrc file once, before moving on
|
||||
with the rest of your compilation activities.
|
||||
|
||||
|
||||
10: Gotchas and solutions:
|
||||
3: The building and installation process overview:
|
||||
--------------------------------------------------
|
||||
For every ThirdParty packages, this is the basic process we will be going
|
||||
through when starting the compilation from scratch:
|
||||
|
||||
a: Verify in the local "RPM vault" if a RPM file is available for the
|
||||
required ThirdParty package.
|
||||
|
||||
b: If the RPM is available, and the package already installed, then there is
|
||||
nothing to do for this package, and we proceed with the next package.
|
||||
|
||||
c: If the RPM is available, and the package is NOT already installed, then
|
||||
we simply install it using the RPM and move on to the next package.
|
||||
|
||||
d: if the RPM is absent:
|
||||
1: we verify if the source tarbal is available from the SOURCES "vault"
|
||||
2: if it is not, we download the tarball using the specified URL.
|
||||
3: we then proceed with the extraction, patching, configuration,
|
||||
compilation, RPM generation, TGZ generation and installation of the
|
||||
package.
|
||||
|
||||
Important notices:
|
||||
- The generated RPM is always used for the installation, not the TGZ file.
|
||||
|
||||
- The compressed tarball (.tgz) file is generated for systems where
|
||||
using rpm for installing packages as a non-root user might be a problem.
|
||||
|
||||
- The .rpm files will be located under the directory
|
||||
$WM_THIRD_PARTY_DIR/rpmBuild/RPMS
|
||||
|
||||
- The .tgz files will be located under the directory
|
||||
$WM_THIRD_PARTY_DIR/rpmBuild/TGZS
|
||||
|
||||
e: The default installation root directory is "$WM_THIRD_PARTY_DIR".
|
||||
This can be overridden though when installing the RPM.
|
||||
|
||||
f: Through the modification of the AllMake.stage[1-4] scripts, it is
|
||||
possible to specify a modified version of a .spec file for a given
|
||||
package in order to tweak its local compilation or installation.
|
||||
|
||||
The modified .spec file will have to be installed under the SPECS
|
||||
directory, and the corresponding invocation or the command rpm_make in
|
||||
the AllMake.stage[1-4] file will need to be modified accordingly.
|
||||
|
||||
|
||||
4: Relocating the RPM root installation directory:
|
||||
--------------------------------------------------
|
||||
|
||||
Warning: This section is a bit advanced, and as of yet, I have no evidence that
|
||||
this feature has ever been used by anybody. So please feel free to skip
|
||||
this section if you don't need to relocate your ThirdParty packages
|
||||
installation directory, which you should not have do on most
|
||||
installation anyway.
|
||||
|
||||
a: All the generated RPMs can be relocated, meaning that you can override
|
||||
the hard-coded root installation directory when using those RPMs for
|
||||
installation.
|
||||
|
||||
b: You can check that the RPM is relocatable by using the command rpm -qip
|
||||
thePackage.rpm. For example, from the cmake-2.8.3 package generated on one
|
||||
of my machines, I will get:
|
||||
|
||||
> rpm -qip cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm| grep Relocations
|
||||
Name : cmake Relocations: /home/beaudoin/OpenFOAM/OpenFOAM-1.6-ext/ThirdParty
|
||||
|
||||
The Relocations path is the actual location pointed by the
|
||||
$WM_THIRD_PARTY_DIR on my virtual machine when I generated the RPM. It is
|
||||
the indication that the RPM is relocatable. This path will end up being
|
||||
hardcoded in the RPM because the environment variable was expanded before
|
||||
generating the RPM.
|
||||
|
||||
This is the default root directory where the RPM will install its
|
||||
"payload". This can be overridden using the 'rpm' command-line parameter
|
||||
--relocate OLDPATH=NEWPATH .
|
||||
|
||||
For example, let's say I want to install my cmake-2.8.3 RPM under the root
|
||||
directory /tmp/someDir instead. I will need to call the 'rpm' command like
|
||||
this:
|
||||
|
||||
rpm -ivh ./cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm \
|
||||
--relocate /home/beaudoin/OpenFOAM/OpenFOAM-1.6-ext/ThirdParty=/tmp/someDir
|
||||
|
||||
Even better, you can dig down the hard-coded path even deeper in order to
|
||||
relocate the whole installation directory, down to the last hard-coded
|
||||
subdirectory.
|
||||
Just specify the whole path when using the --relocate parameter
|
||||
So basically, you can install the RPM right under /usr if you want, hence
|
||||
bypassing the default sequence of package subdirectories I have chosen in
|
||||
order to stay close to the "traditional" ThirdParty layout.
|
||||
|
||||
|
||||
5: Things to do:
|
||||
----------------
|
||||
a: Testing testing testing. This prototype was tested on the following
|
||||
platforms:
|
||||
|
||||
Centos 5.5 64-bit (RPM based)
|
||||
Mac OS X 10.6 (Snow Leopard) (non RPM based)
|
||||
OpenSUSE 11.3 64-bit (RPM based)
|
||||
RedHat Enterprise Linux 6 64-bit (RPM based)
|
||||
Ubuntu 10.04 32-bit (non RPM based)
|
||||
Ubuntu 10.04 64-bit (non RPM based)
|
||||
|
||||
b: Maybe adding some RPM dependencies might be useful. I have not explored
|
||||
this yet.
|
||||
|
||||
c: Supplying compilation recipes with the source code for various OSes.
|
||||
Lots of traffic on the Forum about this. This does not need to be that
|
||||
complicated.
|
||||
|
||||
|
||||
|
||||
6: Gotchas and solutions:
|
||||
-------------------------
|
||||
|
||||
1: It was reported that version 5.2.0 of rpm does not work well with the provided scripts.
|
||||
rpm version 5.2.1 works nicely.
|
||||
rpm version 5.2.1 is working nicely.
|
||||
|
||||
|
||||
2: Error messages from rpm:
|
||||
---------------------------
|
||||
|
||||
Problem : error: can't create transaction lock on /var/lock/rpm/transaction
|
||||
Solution: add the following entry in your file ~/.rpmmacros
|
||||
Problem : error: can't create transaction lock on /var/lock/rpm/transaction
|
||||
|
||||
Solution: add the following entry in your file ~/.rpmmacros
|
||||
|
||||
%_rpmlock_path %{_dbpath}/__db.000
|
||||
|
||||
|
||||
7: Reporting errors:
|
||||
--------------------
|
||||
|
||||
Please report ThirdParty packages related bugs to the Bug tracking system
|
||||
for OpenFOAM-1.6-ext.
|
||||
http://sourceforge.net/apps/mantisbt/openfoam-extend/main_page.php
|
||||
|
||||
To be continued...
|
||||
|
||||
|
||||
Martin Beaudoin
|
||||
May 2011
|
||||
|
||||
Rev 0: 05/2011
|
||||
Rev 1: 08/2011
|
||||
|
|
|
|||
Reference in a new issue