67 lines
3.6 KiB
Markdown
67 lines
3.6 KiB
Markdown
## Coherent I/O format for OpenFOAM
|
|
The new "coherent" OpenFOAM file format is designed for high-performance parallel file systems input/output (I/O), optimizing data access patterns for modern supercomputers. This format relies on the coherent mesh representation [1], ensuring consistency across various topological dimensions, including grid cells, faces, and points. The data coherence allows for efficient partitioning and selection of continuous mesh and field data chunks. Beyond HPC file systems, this optimized format extends its benefits to users with diverse compute resources, streamlining pre- and post-processing and contributing to an accelerated time-to-solution. These developments were implemented in a reduced version of foam-extend 4.1, encompassing the core foam and finiteVolume libraries.
|
|
|
|
### Installation
|
|
For the reduced version the same steps apply as for foam-extend-4.1, see
|
|
https://openfoamwiki.net/index.php/Installation/Linux/foam-extend-4.1.
|
|
|
|
In particular, use `Allwmake.firstInstall` to download and install the dependency to ADIOS2 in the ThirdParty folder.
|
|
|
|
```
|
|
source etc/bashrc [WM_MPLIB=XXX MPI_ROOT=/path/to/mpi]
|
|
Allwmake.firstInstall
|
|
```
|
|
|
|
Sourcing the bashrc for a second time enables usage of ADIOS2 tools like `bpls`.
|
|
|
|
### Usage
|
|
Check out the `lid-driven-cavity-3D` case in the `tutorials` folder. The following workflow works straightforward:
|
|
|
|
```
|
|
blockMesh
|
|
[decomposePar]
|
|
mpirun -n X icoFoam -parallel
|
|
```
|
|
|
|
Note that `decomposePar` actually is optional because a *naive* decomposition can be used during start-up phase of `icoFoam`. Also, if the number of processors in `decomposeParDict` does not match the number of ranks in `mpirun -n X`, the *naive* decomposition is used. Hence, `decomposePar` becomes optional and restarts on arbitrary number of MPI ranks is possible.
|
|
|
|
After the first run the time folders contain a ADIOS2 bp-file. The data can be observed using the command line tool `bpls`.
|
|
|
|
```
|
|
bpls 0.001/data.bp/
|
|
double U/internalField {3000000}
|
|
double p/internalField {1000000}
|
|
double phi/boundaryField/movingWall/value {10000}
|
|
double phi/internalField {2970000}
|
|
```
|
|
|
|
Please also refer to https://adios2.readthedocs.io/en/v2.9.1/ about viewing and extracting data from ADIOS2 files.
|
|
|
|
If the `system/controlDict` sets `writeBulkData yes`, the the data of all time steps is stored in a case-global ADIOS2 bp-file. The time steps share one common `data.bp` file in the case folder. In that case, asynchronous output can be activate in the `system/config.xml`:
|
|
|
|
```
|
|
<parameter key="AsyncOpen" value="true"/>
|
|
<parameter key="AsyncWrite" value="true"/>
|
|
```
|
|
|
|
The activated asynchronous output into the case-global data file can hide the overheads of the I/O latencies.
|
|
|
|
#### Reference
|
|
|
|
[1] "Coherent Mesh Representation for Parallel I/O of Unstructured Polyhedral Meshes", R. G. Weiss, S. Lesnik, F. C. C. Galeazzo, A. Ruopp, H. Rusche <https://doi.org/10.21203/rs.3.rs-3897818/v1>
|
|
|
|
#### Contributors
|
|
The work has been carried out in Task 3.4 — Parallel I/O — of the exaFOAM project.
|
|
Participating partners (partner in **bold** is the task lead): **HLRS**, Wikki GmbH
|
|
|
|
HLRS: Gregor Weiss, Flavio Galeazzo, and Andreas Ruopp
|
|
|
|
Wikki GmbH: Sergey Lesnik and Henrik Rusche
|
|
|
|
#### Acknowledgment
|
|
This application has been developed as part of the exaFOAM Project
|
|
https://www.exafoam.eu, which has received funding from the European
|
|
High-Performance Computing Joint Undertaking (JU) under grant agreement No
|
|
956416. The JU receives support from the European Union's Horizon 2020 research
|
|
and innovation programme and France, Germany, Italy, Croatia, Spain, Greece,
|
|
and Portugal.
|