From 2f1c5a10b9df18b25ad479c80caca3bb7af34d36 Mon Sep 17 00:00:00 2001 From: William F Godoy Date: Mon, 13 Jan 2020 19:06:03 -0500 Subject: [PATCH 1/2] add documentation for null engine and bp3 option for nvram --- docs/user_guide/source/conf.py | 3 ++- docs/user_guide/source/engines/bp3.rst | 3 +++ docs/user_guide/source/engines/engines.rst | 2 ++ docs/user_guide/source/engines/null.rst | 7 +++++++ 4 files changed, 14 insertions(+), 1 deletion(-) create mode 100644 docs/user_guide/source/engines/null.rst diff --git a/docs/user_guide/source/conf.py b/docs/user_guide/source/conf.py index 98d93dbfea..b7ece7e318 100644 --- a/docs/user_guide/source/conf.py +++ b/docs/user_guide/source/conf.py @@ -138,7 +138,8 @@ 'engines/hdf5.rst', 'engines/bp3.rst', 'engines/sst.rst', - 'engines/inline.rst' + 'engines/inline.rst', + 'engines/null.rst', # 'engines/engines.rst', # 'faq/faq.rst', ] diff --git a/docs/user_guide/source/engines/bp3.rst b/docs/user_guide/source/engines/bp3.rst index 61bb6cc7fc..6259edc7ec 100644 --- a/docs/user_guide/source/engines/bp3.rst +++ b/docs/user_guide/source/engines/bp3.rst @@ -50,6 +50,8 @@ This engine allows the user to fine tune the buffering operations through the fo 9. **SubStreams**: (MPI-only) users can select how many sub-streams (``M`` sub-files) are produced during a run, ranges between 1 and the number of mpi processes from ``MPI_Size`` (``N``), adios2 will internally aggregate data buffers (``N-to-M``) to output the required number of sub-files. If Substream is out of bounds it will pick either 1 (``SubStreams`` < ``1 -> N-to-1``) or ``N`` ((``SubStreams`` > ``N -> N-to-N``) and ADIOS2 will issue a WARNING message. Use for performance tuning. +10. **Node-Local**: For distributed file system. Every writer process must make sure the .bp/ directory is created on the local file system. Required for using local disk/SSD/NVMe in a cluster. + ==================== ===================== =========================================================== **Key** **Value Format** **Default** and Examples ==================== ===================== =========================================================== @@ -62,6 +64,7 @@ This engine allows the user to fine tune the buffering operations through the fo BufferGrowthFactor float > 1 **1.05**, 1.01, 1.5, 2 FlushStepsCount integer > 1 **1**, 5, 1000, 50000 SubStreams integer >= 1 **MPI_Size (N-to-N)**, ``MPI_Size``/2, ... , 2, (N-to-1) 1 + Node-Local string On/Off **Off**, On ==================== ===================== =========================================================== diff --git a/docs/user_guide/source/engines/engines.rst b/docs/user_guide/source/engines/engines.rst index 42478cc5b8..56b89073ec 100644 --- a/docs/user_guide/source/engines/engines.rst +++ b/docs/user_guide/source/engines/engines.rst @@ -23,3 +23,5 @@ Parameters are passed at: .. include:: dataman.rst .. include:: inline.rst .. include:: insitu_mpi.rst +.. include:: null.rst + diff --git a/docs/user_guide/source/engines/null.rst b/docs/user_guide/source/engines/null.rst new file mode 100644 index 0000000000..7b1156595d --- /dev/null +++ b/docs/user_guide/source/engines/null.rst @@ -0,0 +1,7 @@ +**** +Null +**** + +The ``Null`` Engine by-passes any heavy I/O operations that other Engines might potentially execute, for example, memory allocations, buffering, transport data movement. Calls to the Null engine would effectively return immediately without doing any effective operations. + +The overall goal is to provide a mechanism to isolate an application behavior without the ADIOS 2 footprint. Use this engine to have an idea of the overhead cost of using a certain ADIOS 2 Engine (similar to writing to `/dev/null`) in an application. From 70796c09511c545b4acec799cf40badc917eb54d Mon Sep 17 00:00:00 2001 From: William F Godoy Date: Mon, 13 Jan 2020 19:48:40 -0500 Subject: [PATCH 2/2] added virtual engines documentation --- docs/user_guide/source/engines/engines.rst | 2 +- docs/user_guide/source/engines/virtual_engines.rst | 7 +++---- 2 files changed, 4 insertions(+), 5 deletions(-) diff --git a/docs/user_guide/source/engines/engines.rst b/docs/user_guide/source/engines/engines.rst index 56b89073ec..1ca3e9e20f 100644 --- a/docs/user_guide/source/engines/engines.rst +++ b/docs/user_guide/source/engines/engines.rst @@ -24,4 +24,4 @@ Parameters are passed at: .. include:: inline.rst .. include:: insitu_mpi.rst .. include:: null.rst - +.. include:: virtual_engines.rst diff --git a/docs/user_guide/source/engines/virtual_engines.rst b/docs/user_guide/source/engines/virtual_engines.rst index e00c66edd9..e532c62a25 100644 --- a/docs/user_guide/source/engines/virtual_engines.rst +++ b/docs/user_guide/source/engines/virtual_engines.rst @@ -8,9 +8,9 @@ The following I/O uses cases are supported by virtual engine names: 1. ``File``: File I/O (Default engine). - This sets up the I/O for files. If the file name passed in Open() ends with ".bp", then the BP4 engine will be used. + This sets up the I/O for files. If the file name passed in Open() ends with ".bp", then the BP4 engine will be used starting in v2.5.0. If it ends with ".h5", the HDF5 engine will be used. For old .bp files (BP version 3 format), the BP3 engine - will be used for reading. + will be used for reading (v2.4.0 and below). 2. ``FileStream``: Online processing via files. @@ -46,8 +46,7 @@ Virtual Engine Setups These are the actual settings in ADIOS when a virtual engine is selected. The parameters below can be modified before the Open call. -1. ``File``. Engine can be ``BP4`` or ``HDF5`` depending on file extension (".h5" for HDF5, otherwise always BP4), - or it can be ``BP3`` in reading if the input file has an older BP version. +1. ``File``. Refer to the parameter settings for these engines of ``BP4``, ``BP3`` and ``HDF5`` engines earlier in this section. 2. ``FileStream``. The engine is ``BP4``. The parameters are set to: