Skip to content

Commit

Permalink
Add/Improve the ReadMe.md files in examples directory
Browse files Browse the repository at this point in the history
  • Loading branch information
spyridon97 committed Sep 27, 2023
1 parent 47efe10 commit 65390e3
Show file tree
Hide file tree
Showing 20 changed files with 334 additions and 138 deletions.
16 changes: 16 additions & 0 deletions examples/ReadMe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# ADIOS2 Examples

This directory contains examples of how to use ADIOS2 in different scenarios.
The examples are written mostly in C++, but also C, Fortran, Python, and Julia.

They can be found in the following subdirectories, and they should be explored in the order that they are listed:

1. [hello](hello): The _hello_ examples are meant to introduce you to ADIOS2's IO capabilities and engines.
2. [basics](basics): The _basics_ examples are meant to introduce you to basic concepts of ADIOS2, such as
global/joined/local arrays, values, and variables shapes.
3. [useCases](useCases): The _useCases_ examples are meant to demonstrate how to use ADIOS2 in different scenarios,
such as in situ visualization, and fides schema.
4. [plugins](plugins): The _plugins_ examples are meant to introduce you to the plugin capabilities of ADIOS2, such as
how to develop your own engine or operators.
5. [simulations](simulations): The _simulations_ examples are meant to demonstrate how to integrate ADIOS2 within your
simulation code to read, write and/or stream your simulation data.
29 changes: 29 additions & 0 deletions examples/basics/ReadMe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
## ADIOS2 basics examples

The _basics_ examples are meant to introduce you to basic concepts of ADIOS2, such as
global/joined/local arrays, values, and variables shapes.

They can be found in the following subdirectories, and they should be explored in the order that they are listed:

1. [globalArray1D](globalArray1D): The _globalArray1D_ example demonstrates how to read and write an
1-D global array with constant dimensions over time from multiple processors using ADIOS2's BP engine.
* Languages: C, Fortran
2. [globalArrayND](globalArrayND): The _globalArrayND_ example demonstrates how to write an N-D global array with
constant dimensions over time from multiple processors using ADIOS2's BP engine.
* Languages: C++
3. [localArray](localArray): The _localArray_ example demonstrates how to write and read a local array per processor
with the same name from multiple processors using ADIOS2's BP engine.
* Languages: C++
4. [joinedArray](joinedArray): The _joinedArray_ example demonstrates how to write local array that is different only in
one dimension so that it can be joined into a global array with the same name from multiple processors at read time
using ADIOS2's ADIOS2's BP engine.
* Languages: C++
5. [values](values): The _values_ example demonstrates how to write and read a multiple types of variables with a single
value, such as global constant, global value, local constant, and local value using ADIOS2's BP engine.
* Languages: C++, Fortran
6. [variablesShapes](variablesShapes): The _variablesShapes_ example demonstrates how to write supported variables
shapes using stepping and ADIOS2's BP engine.
* Languages: C++, C++ using high-level API
7. [queryWorker](queryWorker): The _queryWorker_ example demonstrates how to read variables using ADIOS2's BP engine
and perform queries on the read data and streams the results.
* Languages: C++
11 changes: 11 additions & 0 deletions examples/basics/queryWorker/ReadMe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
### ADIOS2 queryWorker example

The _queryWorker_ example demonstrates how to read variables using ADIOS2's BP engine
and perform queries on the read data and streams the results.

1. q1.json is a json query file for one var

2. q2.json is a composite query

3. The bp*xml are the from heat transfer example with bp4 engine, and <adios-query> tag is added at the end for xml
query spec
18 changes: 0 additions & 18 deletions examples/hello/README.md

This file was deleted.

69 changes: 69 additions & 0 deletions examples/hello/ReadMe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
## ADIOS2 hello examples

The _hello_ examples are meant to introduce you to ADIOS2's IO capabilities and engines.

They can be found in the following subdirectories, and they should be explored in the order that they are listed:

1. [helloWorld](helloWorld): The _helloWorld_ example demonstrates how to write a simple word and read it back using
ADIOS2's BP engine.
* Languages: C++, C++ using high-level API, C, Python, Python using high-level API
2. [bpWriter](bpWriter): The _bpWriter_ examples demonstrate how to write a variable to using ADIOS2's BP engine.
* Languages: C++, C, Fortran, Python
3. [bpReader](bpReader): The _bpReader_ examples demonstrate how to read a 1D/2D/3D variable using ADIOS2's BP engine.
* Languages: C++, Fortran, Python
4. [bpFWriteCRead](bpFWriteCRead): The _bpFWriteCRead_ example demonstrates how to write a 2D variable with Fortran and
read a subset of it with C++, and vice versa using ADIOS2's BP engine.
* Languages: C++, Fortran
5. [bpTimeWriter](bpTimeWriter): The _bpTimeWriter_ example demonstrates how to write two Variables (one is timestep)
using time aggregation using ADIOS2's BP engine.
* Languages: C++, Python
6. [bpAttributeWriter](bpAttributeWriter): The _bpAttributeWriter_ example demonstrates how to write attributes using
ADIOS2's BP engine.
* Languages: C++
7. [bpFlushWriter](bpFlushWriter): The _bpFlushWriter_ example demonstrates how to flush a variable using ADIOS2's BP
engine.
* Languages: C++
8. [bpWriteReadCuda](bpWriteReadCuda): The _bpWriteReadCuda_ example demonstrates how to write and read a variable with
multiple time steps using ADIOS2's BP engine and leveraging CUDA.
* Languages: C++
9. [bpWriteReadHip](bpWriteReadHip): The _bpWriteReadHip_ example demonstrates how to write and read a variable with
multiple time steps using ADIOS2's BP engine and leveraging HIP.
* Languages: C++
10. [bpWriteReadKokkos](bpWriteReadKokkos): The _bpWriteReadOmp_ example demonstrates how to write and read a variable
with multiple time steps using ADIOS2's BP engine and leveraging Kokkos.
* Languages: C++
11. [datamanReader](datamanReader): The _datamanReader_ example demonstrates how to read variables in real-time WAN
streams using ADIOS's DataMan engine.
* Languages: C++, Python
12. [datamanWriter](datamanWriter): The _datamanWriter_ example demonstrates how to write variables in real-time WAN
streams using ADIOS's DataMan engine.
* Languages: C++, Python
13. [dataspacesReader](dataspacesReader): The _dataspacesReader_ example demonstrates how to read a variable using
ADIOS2's DATASPACES engine.
* Languages: C++
14. [dataspacesWriter](dataspacesWriter): The _dataspacesWriter_ example demonstrates how to write a variable using
ADIOS2's DATASPACES engine.
* Languages: C++
15. [hdf5Reader](hdf5Reader): The _hdf5Reader_ example demonstrates how to read variables using ADIOS2's HDF5 engine.
* Languages: C++
16. [hdf5Writer](hdf5Writer): The _hdf5Writer_ example demonstrates how to write variables using ADIOS2's HDF5 engine.
* Languages: C++
17. [hdf5SubFile](hdf5SubFile): The _hdf5SubFile_ example demonstrates how to write variables using ADIOS2's parallel
HDF5 engine leveraging the subfile feature.
* Languages: C++
18. [inlineMWE](inlineMWE): The _inlineMWE_ example demonstrates how to write and read a variable using ADIOS2's inline
engine.
* Languages: C++
19. [inlineFWriteCppRead](inlineFWriteCppRead): The _inlineFWriteCppRead_ example demonstrates how to write a 2D
variable with Fortran and read it back a subset of it with C++ using ADIOS2's inline engine.
* Languages: C++, Fortran
20. [inlineReaderWriter](inlineReaderWriter): The _inlineReaderWriter_ example demonstrates how to write two Variables
(one is timestep) using time aggregation and ADIOS2's inline engine.
* Languages: C++
21. [sstReader](sstReader): The _sstReader_ example demonstrates how to read a variable using ADIOS2's SST engine.
* Languages: C++
22. [sstWriter](sstWriter): The _sstWriter_ example demonstrates how to write a variable using ADIOS2's SST engine.
* Languages: C++
23. [skeleton](skeleton): The _skeleton_ example demonstrates how to write and read a variable using an ADIOS2 skeleton
engine.
* Languages: C++
13 changes: 13 additions & 0 deletions examples/plugins/ReadMe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
## ADIOS2 plugins examples

The _plugins_ examples are meant to introduce you to the plugin capabilities of ADIOS2, such as
how to develop your own engine or operators.

They can be found in the following subdirectories, and they should be explored in the order that they are listed:

1. [engine](engine): The _engine_ example demonstrates how to develop the read/write capabilities of your ADIOS2 engine
and use it to read and write variables.
* Languages: C++
2. [operator](operator): The _operator_ example demonstrates how to read and write variables using the
ADIOS2/plugins/operators/EncryptionOperator and ADIOS2's BP engine.
* Languages: C++
71 changes: 43 additions & 28 deletions examples/simulations/GrayScott.jl/ReadMe.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
# ADIOS2 GrayScott.jl example
### ADIOS2 GrayScott.jl example

Julia version of [the gray-scott C++ and Python](https://github.com/ornladios/ADIOS2/blob/master/examples/simulations/gray-scott/)
example.
Julia version
of [the gray-scott C++ and Python](https://github.com/ornladios/ADIOS2/blob/master/examples/simulations/gray-scott/)
example.

This is a 3D 7-point stencil code to simulate the following [Gray-Scott
reaction diffusion model](https://doi.org/10.1126/science.261.5118.189):
Expand All @@ -11,27 +12,29 @@ u_t = Du * (u_xx + u_yy + u_zz) - u * v^2 + F * (1 - u) + noise * randn(-1,1)
v_t = Dv * (v_xx + v_yy + v_zz) + u * v^2 - (F + k) * v
```

This version contains:
This version contains:

- CPU threaded solver using Julia's [multithreading]](https://docs.julialang.org/en/v1/manual/multi-threading/)
- GPU solvers using [CUDA.jl](https://github.com/JuliaGPU/CUDA.jl), [AMDGPU.jl](https://github.com/JuliaGPU/AMDGPU.jl)
- Parallel I/O using the [ADIOS2.jl](https://github.com/eschnett/ADIOS2.jl) Julia bindings to [ADIOS2](https://github.com/ornladios/ADIOS2)
- Parallel I/O using the [ADIOS2.jl](https://github.com/eschnett/ADIOS2.jl) Julia bindings
to [ADIOS2](https://github.com/ornladios/ADIOS2)
- Message passing interface (MPI) using [MPI.jl](https://github.com/JuliaParallel/MPI.jl) Julia bindings to MPI
- Easily switch between float- (Float32) and double- (Float64) precision in the configuration file

## How to run
#### How to run

Currently only the simulation part is ported from C++, the data analysis is work-in-progress.
Currently only the simulation part is ported from C++, the data analysis is work-in-progress.

Pre-requisites:

- A recent Julia version: v1.8.5 or v1.9.0-beta3 as of January 2023 from [julialang.org/downloads](https://julialang.org/downloads/)
- A recent Julia version: v1.8.5 or v1.9.0-beta3 as of January 2023
from [julialang.org/downloads](https://julialang.org/downloads/)

### Run locally
##### Run locally

1. **Set up dependencies**

From the `GrayScott.jl` directory instantiate and use MPI artifact jll (preferred method).
From the `GrayScott.jl` directory instantiate and use MPI artifact jll (preferred method).
To use a system provided MPI, see [here](https://juliaparallel.org/MPI.jl/latest/configuration/#using_system_mpi)

```julia
Expand All @@ -50,7 +53,8 @@ julia> MPIPreferences.use_jll_binary()
julia> exit()
```

Julia manages its own packages using [Pkg.jl](https://pkgdocs.julialang.org/v1/), the above would create platform-specific `LocalPreferences.toml` and `Manifest.toml` files.
Julia manages its own packages using [Pkg.jl](https://pkgdocs.julialang.org/v1/), the above would create
platform-specific `LocalPreferences.toml` and `Manifest.toml` files.

2. **Set up the examples/settings-files.json configuration file**

Expand Down Expand Up @@ -81,8 +85,8 @@ Julia manages its own packages using [Pkg.jl](https://pkgdocs.julialang.org/v1/)
```

The file is nearly identical to the C++ original example.
Not all options are currently supported, but two Julia-only options are added:
The file is nearly identical to the C++ original example.
Not all options are currently supported, but two Julia-only options are added:

- "precision": either Float32 or Float64 in the array simulation (including GPUs)
- "backend": "CPU", "CUDA" or "AMDGPU"
Expand All @@ -103,37 +107,48 @@ Not all options are currently supported, but two Julia-only options are added:

This would generate an adios2 file from the output entry in the configuration file (e.g. `gs-julia-1MPI-64L-F32.bp`)
that can be visualized with ParaView with either the VTX or the FIDES readers.
**Important**: the AMDGPU.jl implementation of `randn` is currently work in progress.
**Important**: the AMDGPU.jl implementation of `randn` is currently work in progress.
See related issue [here](https://github.com/JuliaGPU/AMDGPU.jl/issues/378)


4. **Running on OLCF Summit and Crusher systems**
The code was tested on the Oak Ridge National Laboratory Leadership Computing Facilities (OLCF): [Summit](https://docs.olcf.ornl.gov/systems/summit_user_guide.html) and [Crusher](https://docs.olcf.ornl.gov/systems/crusher_quick_start_guide.html). Both are used testing a recent version of Julia [v1.9.0-beta3](https://julialang.org/downloads/#upcoming_release) and a `JULIA_DEPOT_PATH` is required to install packages and artifacts. **DO NOT USE your home directory**. We are providing configuration scripts in `scripts/config_XXX.sh` showing the plumming required for these systems. They need to be executed only once per session from the login nodes.

To reuse these file the first 3 entries must be modified and run on login-nodes and the PATH poiting at a downloaded Julia binary for the corresponding PowerPC (Summit) and x86-64 (Crusher) architectures. Only "CPU" and "CUDA" backends are supported on Summit, while "CPU" and "AMDGPU" backends are supported on Crusher.
The code was tested on the Oak Ridge National Laboratory Leadership Computing Facilities (
OLCF): [Summit](https://docs.olcf.ornl.gov/systems/summit_user_guide.html)
and [Crusher](https://docs.olcf.ornl.gov/systems/crusher_quick_start_guide.html). Both are used testing a recent
version of Julia [v1.9.0-beta3](https://julialang.org/downloads/#upcoming_release) and a `JULIA_DEPOT_PATH` is
required to install packages and artifacts. **DO NOT USE your home directory**. We are providing configuration
scripts in `scripts/config_XXX.sh` showing the plumming required for these systems. They need to be executed only
once per session from the login nodes.

To reuse these file the first 3 entries must be modified and run on login-nodes and the PATH poiting at a downloaded
Julia binary for the corresponding PowerPC (Summit) and x86-64 (Crusher) architectures. Only "CPU" and "CUDA" backends
are supported on Summit, while "CPU" and "AMDGPU" backends are supported on Crusher.

```
# Replace these 3 entries
### Replace these 3 entries
PROJ_DIR=/gpfs/alpine/proj-shared/csc383
export JULIA_DEPOT_PATH=$PROJ_DIR/etc/summit/julia_depot
GS_DIR=$PROJ_DIR/wgodoy/ADIOS2/examples/simulations/GrayScott.jl
...
# and the path
### and the path
export PATH=$PROJ_DIR/opt/summit/julia-1.9.0-beta3/bin:$PATH
```

## To-do list
#### To-do list

1. Add support including random number on device kernel code on `AMDGPU.jl`
2. Set the domain size `L` in the configuration file as a multiple of 6 for Summit, and a multiple of 4 on Crusher
3. Add data analysis: PDF for u and v and Julia 2D plotting capabilities: Plots.jl, Makie.jl
4. Add interactive computing with Pluto.jl notebooks
1. Add support including random number on device kernel code on `AMDGPU.jl`
2. Set the domain size `L` in the configuration file as a multiple of 6 for Summit, and a multiple of 4 on Crusher
3. Add data analysis: PDF for u and v and Julia 2D plotting capabilities: Plots.jl, Makie.jl
4. Add interactive computing with Pluto.jl notebooks

#### Acknowledgements

## Acknowledgements
This research was supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy’s Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation’s exascale computing imperative.
This research was supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of
Energy’s Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale
ecosystem, including software, applications, and hardware technology, to support the nation’s exascale computing
imperative.

This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.
This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which
is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

Thanks to the Exascale Computing Project PROTEAS-TUNE and ADIOS subprojects, and the ASCR Bluestone.
Thanks to all the Julia community members, packages developers and maintainers for their great work.
30 changes: 30 additions & 0 deletions examples/simulations/ReadMe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
## ADIOS2 simulations examples

The _simulations_ examples are meant to demonstrate how to integrate ADIOS2 within your
simulation code to read, write and/or stream your simulation data.

They can be found in the following subdirectories, and they should be explored in the order that they are listed:

1. [korteweg-de-vries](korteweg-de-vries): The _korteweg-de-vries_ example demonstrates how to solve the initial value
problem for the Korteweg de-Vries equation via the Zabusky and Krustal scheme and integrate ADIOS2 for its IO
capabilities.
* Languages: C++, Python
2. [lorenz_ode](lorenz_ode): The _lorenz_ode_ example demonstrates how to solve the Lorenz system of ordinary
differential equations and integrate ADIOS2 for its IO capabilities.
* Languages: C++
3. [gray-scott](gray-scott): The _gray-scott_ example demonstrates how to perform simulation with the Gray-Scott
reaction diffusion model and integrate ADIOS2 for its IO capabilities.
* Languages: C++
4. [gray-scott-struct](gray-scott-struct): The _gray-scott-struct_ example demonstrates how to perform a simulation with
the Gray-Scott reaction diffusion model using a struct that defines user defined data types and integrate ADIOS2 for
its IO capabilities.
* Languages: C++
5. [gray-scott-kokkos](gray-scott-kokkos): The _gray-scott-kokkos_ example demonstrates how to perform a simulation with
the Gray-Scott reaction diffusion model using Kokkos and integrate ADIOS2 for its IO capabilities.
* Languages: C++
6. [GrayScott.jl](GrayScott.jl): The _GrayScott.jl_ example demonstrates how to perform a simulation with the Gray-Scott
reaction diffusion model and integrate ADIOS2 for its IO capabilities.
* Languages: Julia
7. [heatTransfer](heatTransfer): The _heatTransfer_ example demonstrates how to solve a 2D Poisson equation for
temperature in homogeneous media using finite differences and integrate ADIOS2 for its IO capabilities.
* Languages: C++
2 changes: 0 additions & 2 deletions examples/simulations/gray-scott-kokkos/README.md

This file was deleted.

9 changes: 9 additions & 0 deletions examples/simulations/gray-scott-kokkos/ReadMe.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
### ADIOS2 gray-scott-kokkos example

This is a 3D 7-point stencil code to simulate the following [Gray-Scott
reaction diffusion model](https://doi.org/10.1126/science.261.5118.189):

#### Differences from the gray-scott example

It uses Kokkos to run the simulation on GPU.
Please use the installed Gray-Scott example directory but run adios2_simulations_gray-scott-kokkos binary.
Loading

0 comments on commit 65390e3

Please sign in to comment.