diff --git a/ThirdParty/README.ThirdParty b/ThirdParty/README.ThirdParty index 4305ee83b..d7c141b0a 100644 --- a/ThirdParty/README.ThirdParty +++ b/ThirdParty/README.ThirdParty @@ -1,129 +1,248 @@ -A new design for the ThirdParty packages: -========================================= +A new design for the ThirdParty packages compilation and installation process: +============================================================================== -The main purpose of this new development is to build a complete ThirParty packages set for OpenFOAM 1.6-ext using only the original package source -tarball and some patch files ( when necessary). +The main purpose of this new development is to build a complete ThirParty +packages set for OpenFOAM 1.6-ext using only the original package source +tarball and some patch files (when necessary). -A useful by-product of this development is also to provide some kind of binary packaging of the ThirdParty packages. +A useful by-product of this development is also to provide some kind of +binary packaging of the ThirdParty packages. There are currently two types of +binary packaging generated by this compilation process: RPMs and compressed +tarballs (TGZs) -The RPM suite of tools was selected to develop a first prototype. -The whole process needs to run and install in user-space, without the need to be root for installing the packages. +The RPM package manager was selected to develop a first prototype. RPM is +available for multiple flavors of Unix/Linux, and offers all the necessary +functionnality to configured, patch, compile and install source code packages. + +One of the main requirement for this new development is that the whole process +needs to run and install in user-space, without the need to be root for +installing the packages. Here is what's available: - a: A set of rpm spec files for specific ThirdParty packages. - b: A suite of bash scripts to automate the complete sequence of downloading, compiling, installing and generating RPMs. - c: An empty directory structure pre-configured and ready to proceed with the download, compilation and installation of - chosen ThirdParty packages for OF-1.6-ext. + a: A set of RPM spec files for specific ThirdParty packages. + b: A set of bash scripts to automate the complete sequence of downloading, + compiling, installing and generating RPMs. + c: An directory structure pre-configured and ready to proceed with the + download, compilation and installation of chosen ThirdParty packages for + OF-1.6-ext. -Quick description of the main scripts: --------------------------------------- + +1: Quick description of the main scripts: +----------------------------------------- a: AllMake: - Main wrapper script that will call AllMake.stage0 to AllMake.stage4 scripts in sequence. + Main wrapper script that will call AllMake.stage0 to AllMake.stage4 + scripts in sequence. b: AllMake.stage0: - This script is useful only for populating what I am calling the local "RPM vault" with pre-generated RPMs. - This is the script written to address the use-case: "I have some pre-generated RPM files, now what" - Basically, you call this script with a list of RPMs generated by the AllMake.stage(1-4) in order to populate the local RPMS vault. - Once in place, these are the RPMs will be installed instead of proceeding with the standard compilation process. + This script is useful only for populating what I am calling the local "RPM + vault" with pre-generated RPMs. + + This is the script written to address the following use case: + "I have some pre-generated RPM files, now what" + + Basically, you call this script with a list of RPMs already generated by + whichever of the AllMake.stage(1-4) in order to populate the local RPMS + vault. Once in place in the RPM vault, these are the RPMs that will get + directly installed by the AllMake.stage(1-4) scripts, instead of being + regenerated by the normal compilation and install process of the + ThirdParty packages. c: AllMake.stage1: - This script is taking care of the basic ThirdParty tools like compilers, cmake , python, etc. - If we ever need to override the local version of flex or bison, this is where we will add those additional packages. - This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable. - If you need to compile the rest of the ThirdParty packages with a new gcc compiler, you will need to source those .sh or .csh file in before activating the other - AllMake.stage(2-4) scripts. - + This script is taking care of the basic ThirdParty tools like compilers, + flex, bison, cmake , python, etc. + This compilation stage will generate .sh and .csh files that will be + sourced by your file settings.sh or settings.csh in order to initialize + the PATH and LD_LIBRARY_PATH environment variable for the stage1 packages. + d: AllMake.stage2: - This script is taking care of the MPI communication libraries. Right now, only OpenMPI is supported. - This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable. - You will need to source those .sh or .csh file in before activating the other AllMake.stage(3-4) scripts because some packages depends on the communication library. + This script is taking care of the MPI communication libraries. Right now, + only OpenMPI is supported. + This compilation stage will generate .sh and .csh files that will be + sourced by your file settings.sh or settings.csh in order to initialize + the PATH and LD_LIBRARY_PATH environment variable for the stage2 packages. e: AllMake.stage3: - This script is taking care of the "standard" ThirdParty libraries like metis, scotch, mesquite, etc. - This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable. - You will need to source those .sh or .csh file in before compiling OpenFOAM because some "Make/options" files will refer to environment variable that are - specific to those packages + This script is taking care of the "standard" ThirdParty libraries like + metis, scotch, mesquite, etc. + This compilation stage will generate .sh and .csh files that will be + sourced by your file settings.sh or settings.csh in order to initialize + the PATH and LD_LIBRARY_PATH environment variable for the stage3 packages. f: AllMake.stage4: - This script is taking care of Paraview and QT (and takes an awfull long time to compile, honest...). - This stage will also generate a .sh and .csh file one needs to source in order to initialize the PATH and LD_LIBRARY_PATH env. variable. - You will need to source those .sh or .csh file in before compiling OpenFOAM because some "Make/options" files will refer to environment variable that are specific to those packages + This script is taking care of Paraview and QT (and this step will take an + awfull long time to compile, honest...). + This compilation stage will generate .sh and .csh files that will be + sourced by your file settings.sh or settings.csh in order to initialize + the PATH and LD_LIBRARY_PATH environment variable for the stage4 packages. g: tools/makeThirdPartyFunctionsForRPM: - A suite of bash functions useful for wrapping the rpmbuild and rpm commands - - -For every packages, this is the basic process we will be going through when starting building the ThirdParty packages from scratch:: - a: Verify in the local "RPM vault" if a RPM file is available for a given package - b: If it is, simply install this RPM and move on to the next package - c: if the RPM is absent: - i: verify if the source tarbal is available from the SOURCES "vault" - ii: if it is not, download the tarball using the specified URL. - iii: proceed with the extraction, patching, configuration, compilation, RPM generation and installation of the package. - The generated RPM is always used for installation. + A set of bash functions useful for wrapping the rpmbuild and rpm commands - Important notice: - Please note that a compressed tarball (.tgz) is now also generated for systems where using rpm for installing packages - as a non-root user might be a problem. - The .rpm files will be located under the directory $WM_THIRD_PARTY_DIR/rpmBuild/RPMS - The .tgz files will be located under the directory $WM_THIRD_PARTY_DIR/rpmBuild/TGZS +2: Important notice: +-------------------- +Every single ThirdParty package will be installed with a set of companion .sh +and .csh files one needs to source in order to properly initialize the PATH and +LD_LIBRARY_PATH environment variables for running the packages. - d: The default installation root directory is "$WM_THIRD_PARTY_DIR". This can be overriden though when installing the RPM. +Some packages will also create environments variables necessary for the +compilation of some OpenFOAM libraries. + +The scripts AllMake.stage[1-4] are taking care of refreshing the environment +variable after the installation of every single package, so you will not need to +manually refresh your OpenFOAM environment yourself when building individual +ThirdParty packages through the AllMake.stage[1-4] scripts. + +The main Allwmake script for the compilation of OpenFOAM will also take care of +refreshing the environment variable before moving on to the compilation of the +main OpenFOAM libraries and applications. + +However, if you decide to compile the libraries and/or applications manually, or +through the src/Allwmake or applications/Allwmake scripts, YOU MUST MAKE SURE +YOUR ENVIRONMENT VARIABLES FOR THE THIRDPARTY PACKAGES ARE UPDATED FIRST! + +In that case, simply resource your bashrc or cshrc file once, before moving on +with the rest of your compilation activities. -Relocating the RPM root installation directory: - a: All the generated RPMs can be relocated, meaning that you can override the hard-coded root installation directory when using those RPMs for installation. - b: You can check that the RPM is relocatable by using the command rpm -qip thePackage.rpm. For example, from the cmake-2.8.3 package generated on one of my machine: - > rpm -qip cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm| grep Relocations - Name : cmake Relocations: /Users/beaudoin/Projets/SAMH/OpenFOAM/ThirdParty-1.6-ext-RPM-based +3: The building and installation process overview: +-------------------------------------------------- +For every ThirdParty packages, this is the basic process we will be going +through when starting the compilation from scratch: - The Relocations path is the actual location pointed by the $WM_THIRD_PARTY_DIR on my laptop when I generated the RPM. - It is the indication that the RPM is relocatable - This path will end up being hardcoded in the RPM because the environment variable was expanded before generating the RPM.. - This is the default root directory where the RPM will install its "payload" - This can be overridden using the 'rpm' command-line parameter --relocate OLDPATH=NEWPATH . + a: Verify in the local "RPM vault" if a RPM file is available for the + required ThirdParty package. + + b: If the RPM is available, and the package already installed, then there is + nothing to do for this package, and we proceed with the next package. + + c: If the RPM is available, and the package is NOT already installed, then + we simply install it using the RPM and move on to the next package. + + d: if the RPM is absent: + 1: we verify if the source tarbal is available from the SOURCES "vault" + 2: if it is not, we download the tarball using the specified URL. + 3: we then proceed with the extraction, patching, configuration, + compilation, RPM generation, TGZ generation and installation of the + package. + + Important notices: + - The generated RPM is always used for the installation, not the TGZ file. + + - The compressed tarball (.tgz) file is generated for systems where + using rpm for installing packages as a non-root user might be a problem. + + - The .rpm files will be located under the directory + $WM_THIRD_PARTY_DIR/rpmBuild/RPMS + + - The .tgz files will be located under the directory + $WM_THIRD_PARTY_DIR/rpmBuild/TGZS + + e: The default installation root directory is "$WM_THIRD_PARTY_DIR". + This can be overridden though when installing the RPM. - For example, let's say you want to install the RPM under the root directory /tmp/someDir instead. - You will need to call the 'rpm' command like this: + f: Through the modification of the AllMake.stage[1-4] scripts, it is + possible to specify a modified version of a .spec file for a given + package in order to tweak its local compilation or installation. + + The modified .spec file will have to be installed under the SPECS + directory, and the corresponding invocation or the command rpm_make in + the AllMake.stage[1-4] file will need to be modified accordingly. + + +4: Relocating the RPM root installation directory: +-------------------------------------------------- + +Warning: This section is a bit advanced, and as of yet, I have no evidence that + this feature has ever been used by anybody. So please feel free to skip + this section if you don't need to relocate your ThirdParty packages + installation directory, which you should not have do on most + installation anyway. + + a: All the generated RPMs can be relocated, meaning that you can override + the hard-coded root installation directory when using those RPMs for + installation. + + b: You can check that the RPM is relocatable by using the command rpm -qip + thePackage.rpm. For example, from the cmake-2.8.3 package generated on one + of my machines, I will get: - rpm -ivh ./cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm --relocate /Users/beaudoin/Projets/SAMH/OpenFOAM/ThirdParty-1.6-ext-RPM-based=/tmp/someDir + > rpm -qip cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm| grep Relocations + Name : cmake Relocations: /home/beaudoin/OpenFOAM/OpenFOAM-1.6-ext/ThirdParty - Even better, you can dig down the hard-coded path even deeper in order to relocate the whole installation directory, down to the last hard-coded subdirectory. - Just specify the whole path when using the --relocate parameter - So basically, you can install the RPM right under /usr if you want, hence bypassing the default sequence of package subdirectories I have chosen in order to stay - close to the "traditional" ThirdParty layout. + The Relocations path is the actual location pointed by the + $WM_THIRD_PARTY_DIR on my virtual machine when I generated the RPM. It is + the indication that the RPM is relocatable. This path will end up being + hardcoded in the RPM because the environment variable was expanded before + generating the RPM. + + This is the default root directory where the RPM will install its + "payload". This can be overridden using the 'rpm' command-line parameter + --relocate OLDPATH=NEWPATH . -9: Things to do: - a: Testing testing testing. This prototype was tested on the following platforms: + For example, let's say I want to install my cmake-2.8.3 RPM under the root + directory /tmp/someDir instead. I will need to call the 'rpm' command like + this: - Mac OS X 10.6 (Snow Leopard) (non RPM based) - Ubuntu 10.04 32bit (non RPM based) - Centos 5.5 64bit (RPM based) - OpenSUSE 11.3 64bit (RPM based) + rpm -ivh ./cmake-2.8.3-darwinIntelGccDPOpt.i386.rpm \ + --relocate /home/beaudoin/OpenFOAM/OpenFOAM-1.6-ext/ThirdParty=/tmp/someDir + + Even better, you can dig down the hard-coded path even deeper in order to + relocate the whole installation directory, down to the last hard-coded + subdirectory. + Just specify the whole path when using the --relocate parameter + So basically, you can install the RPM right under /usr if you want, hence + bypassing the default sequence of package subdirectories I have chosen in + order to stay close to the "traditional" ThirdParty layout. + + +5: Things to do: +---------------- + a: Testing testing testing. This prototype was tested on the following + platforms: + + Centos 5.5 64-bit (RPM based) + Mac OS X 10.6 (Snow Leopard) (non RPM based) + OpenSUSE 11.3 64-bit (RPM based) + RedHat Enterprise Linux 6 64-bit (RPM based) + Ubuntu 10.04 32-bit (non RPM based) + Ubuntu 10.04 64-bit (non RPM based) - b: Maybe adding some RPM dependencies might be useful. I have not explored this yet. - + b: Maybe adding some RPM dependencies might be useful. I have not explored + this yet. -10: Gotchas and solutions: + c: Supplying compilation recipes with the source code for various OSes. + Lots of traffic on the Forum about this. This does not need to be that + complicated. + + + +6: Gotchas and solutions: +------------------------- 1: It was reported that version 5.2.0 of rpm does not work well with the provided scripts. - rpm version 5.2.1 works nicely. + rpm version 5.2.1 is working nicely. + 2: Error messages from rpm: - --------------------------- - Problem : error: can't create transaction lock on /var/lock/rpm/transaction - Solution: add the following entry in your file ~/.rpmmacros + Problem : error: can't create transaction lock on /var/lock/rpm/transaction + + Solution: add the following entry in your file ~/.rpmmacros %_rpmlock_path %{_dbpath}/__db.000 +7: Reporting errors: +-------------------- -To be continued... + Please report ThirdParty packages related bugs to the Bug tracking system + for OpenFOAM-1.6-ext. + http://sourceforge.net/apps/mantisbt/openfoam-extend/main_page.php + Martin Beaudoin -May 2011 - +Rev 0: 05/2011 +Rev 1: 08/2011