reduced-foam-extend-coherent-io/README.md

## Coherent I/O format for OpenFOAM
The new "coherent" OpenFOAM file format is designed for high-performance parallel file systems input/output (I/O), optimizing data access patterns for modern supercomputers. This format relies on the coherent mesh representation [1], ensuring consistency across various topological dimensions, including grid cells, faces, and points. The data coherence allows for efficient partitioning and selection of continuous mesh and field data chunks. Beyond HPC file systems, this optimized format extends its benefits to users with diverse compute resources, streamlining pre- and post-processing and contributing to an accelerated time-to-solution. These developments were implemented in a reduced version of foam-extend 4.1, encompassing the core foam and finiteVolume libraries.

### Installation
For the reduced version the same steps apply as for foam-extend-4.1, see
https://openfoamwiki.net/index.php/Installation/Linux/foam-extend-4.1.

In particular, use `Allwmake.firstInstall` to download and install the dependency to ADIOS2 in the ThirdParty folder.

```
source etc/bashrc [WM_MPLIB=XXX MPI_ROOT=/path/to/mpi]
Allwmake.firstInstall
```

Sourcing the bashrc for a second time enables usage of ADIOS2 tools like `bpls`.

### Usage
Check out the `lid-driven-cavity-3D` case in the `tutorials` folder. The following workflow works straightforward:

```
blockMesh
[decomposePar]
mpirun -n X icoFoam -parallel
```

Note that `decomposePar` actually is optional because a *naive* decomposition can be used during start-up phase of `icoFoam`. Also, if the number of processors in `decomposeParDict` does not match the number of ranks in `mpirun -n X`, the *naive* decomposition is used. Hence, `decomposePar` becomes optional and restarts on arbitrary number of MPI ranks is possible.

After the first run the time folders contain a ADIOS2 bp-file. The data can be observed using the command line tool `bpls`.

```
bpls 0.001/data.bp/
  double   U/internalField                     {3000000}
  double   p/internalField                     {1000000}
  double   phi/boundaryField/movingWall/value  {10000}
  double   phi/internalField                   {2970000}
```

Please also refer to https://adios2.readthedocs.io/en/v2.9.1/ about viewing and extracting data from ADIOS2 files.

If the `system/controlDict` sets `writeBulkData yes`, the the data of all time steps is stored in a case-global ADIOS2 bp-file. The time steps share one common `data.bp` file in the case folder. In that case, asynchronous output can be activate in the `system/config.xml`:

```
<parameter key="AsyncOpen" value="true"/>
<parameter key="AsyncWrite" value="true"/>
```

The activated asynchronous output into the case-global data file can hide the overheads of the I/O latencies.

#### Reference

[1] "Coherent Mesh Representation for Parallel I/O of Unstructured Polyhedral Meshes", R. G. Weiss, S. Lesnik, F. C. C. Galeazzo, A. Ruopp, H. Rusche <https://doi.org/10.21203/rs.3.rs-3897818/v1>

#### Contributors
The work has been carried out in Task 3.4 — Parallel I/O — of the exaFOAM project.
Participating partners (partner in **bold** is the task lead): **HLRS**, Wikki GmbH

HLRS: Gregor Weiss, Flavio Galeazzo, and Andreas Ruopp

Wikki GmbH: Sergey Lesnik and Henrik Rusche

#### Acknowledgment
This application has been developed as part of the exaFOAM Project
https://www.exafoam.eu, which has received funding from the European
High-Performance Computing Joint Undertaking (JU) under grant agreement No
956416. The JU receives support from the European Union's Horizon 2020 research
and innovation programme and France, Germany, Italy, Croatia, Spain, Greece,
and Portugal.