Skip to content

Commit

Permalink
Merge pull request #4184 from vicentebolea/backports-from-master
Browse files Browse the repository at this point in the history 
Backports from master
  • Loading branch information
@vicentebolea
vicentebolea authored May 29, 2024
2 parents c2a9464 + 260166d commit f691736
Show file tree
Hide file tree
Showing 37 changed files with 2,243 additions and 703 deletions.
6 changes: 3 additions & 3 deletions bindings/C/adios2/c/adios2_c_io.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -329,8 +329,8 @@ adios2_error adios2_remove_all_attributes(adios2_io *io);
* MPI Collective function as it calls MPI_Comm_dup
* @param io engine owner
* @param name unique engine identifier
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append, and
* adios2_mode_readRandomAccess
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append
* and adios2_mode_readRandomAccess
* @return success: handler, failure: NULL
*/
adios2_engine *adios2_open(adios2_io *io, const char *name, const adios2_mode mode);
Expand All @@ -341,7 +341,7 @@ adios2_engine *adios2_open(adios2_io *io, const char *name, const adios2_mode mo
* MPI Collective function as it calls MPI_Comm_dup
* @param io engine owner
* @param name unique engine identifier
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append, and
* @param mode adios2_mode_write, adios2_mode_read, adios2_mode_append and
* adios2_mode_readRandomAccess
* @param comm communicator other than adios' handler comm. MPI only.
* @return success: handler, failure: NULL
Expand Down
2 changes: 1 addition & 1 deletion bindings/CXX11/adios2/cxx11/Variable.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -214,7 +214,7 @@ class Variable
* variable.Count() = {Ny,Nx}, then memoryCount = {Ny+2,Nx+2}
* </pre>
*/
void SetMemorySelection(const adios2::Box<adios2::Dims> &memorySelection);
void SetMemorySelection(const adios2::Box<adios2::Dims> &memorySelection = {{}, {}});

/**
* Sets a step selection modifying current startStep, countStep
Expand Down
37 changes: 36 additions & 1 deletion docs/user_guide/source/components/anatomy.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,10 @@ named `adios2::Mode::ReadRandomAccess`. `adios2::Mode::Read` mode allows data ac
current timestep. `ReadRandomAccess` can only be used with file engines and involves loading all the file metadata at
once. So it can be more memory intensive than `adios2::Mode::Read` mode, but allows reading data from any timestep using
`SetStepSelection()`. If you use `adios2::Mode::ReadRandomAccess` mode, be sure to allocate enough memory to hold
multiple steps of the variable content.
multiple steps of the variable content. Note that ADIOS streaming
engines (like SST, DataMan, etc.) do not support `ReadRandomAccess`
mode. Also newer file Engines like BP5 to not allow
`BeginStep/EndStep` calls in `ReadRandomAccess` mode.

.. code:: C++

Expand All @@ -134,3 +137,35 @@ multiple steps of the variable content.
| |--> IO goes out of scope
|
|--> ADIOS goes out of scope or adios2_finalize()
Previously we explored how to read using the input mode `adios2::Mode::Read`. Nonetheless, ADIOS has another input mode
named `adios2::Mode::ReadRandomAccess`. `adios2::Mode::Read` mode allows data access only timestep by timestep using
`BeginStep/EndStep`, but generally it is more memory efficient as ADIOS is only required to load metadata for the
current timestep. `ReadRandomAccess` can only be used with file engines and involves loading all the file metadata at
once. So it can be more memory intensive than `adios2::Mode::Read` mode, but allows reading data from any timestep using
`SetStepSelection()`. If you use `adios2::Mode::ReadRandomAccess` mode, be sure to allocate enough memory to hold
multiple steps of the variable content. Note that ADIOS streaming
engines (like SST, DataMan, etc.) do not support `ReadRandomAccess`
mode. Also newer file Engines like BP5 to not allow
`BeginStep/EndStep` calls in `ReadRandomAccess` mode.

.. code:: C++

ADIOS adios("config.xml", MPI_COMM_WORLD);
|
| IO io = adios.DeclareIO(...);
| |
| | Engine e = io.Open("InputFileName.bp", adios2::Mode::ReadRandomAccess);
| | |
| | | Variable var = io.InquireVariable(...)
| | | | var.SetStepSelection()
| | | | e.Get(var, datapointer);
| | | |
| | |
| | e.Close();
| |
| |--> IO goes out of scope
|
|--> ADIOS goes out of scope or adios2_finalize()

2 changes: 1 addition & 1 deletion docs/user_guide/source/components/io.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -196,7 +196,7 @@ A particular ``Engine`` type is set to the current ``IO`` component with the ``I
Engine polymorphism is handled internally by the ``IO`` class, which allows subclassing future derived ``Engine`` types without changing the basic API.

``Engine`` objects are created in various modes.
The available modes are ``adios2::Mode::Read``, ``adios2::Mode::Write``, ``adios2::Mode::Append``, ``adios2::Mode::Sync``, ``adios2::Mode::Deferred``, and ``adios2::Mode::Undefined``.
The available modes are ``adios2::Mode::Read``, ``adios2::Mode::ReadRandomAccess``, ``adios2::Mode::Write``, ``adios2::Mode::Append``, ``adios2::Mode::Sync``, ``adios2::Mode::Deferred``, and ``adios2::Mode::Undefined``.


.. code-block:: c++
Expand Down
10 changes: 10 additions & 0 deletions docs/user_guide/source/engines/bp5.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,14 @@ This engine allows the user to fine tune the buffering operations through the fo

#. **Threads**: Read side: Specify how many threads one process can use to speed up reading. The default value is *0*, to let the engine estimate the number of threads based on how many processes are running on the compute node and how many hardware threads are available on the compute node but it will use maximum 16 threads. Value *1* forces the engine to read everything within the main thread of the process. Other values specify the exact number of threads the engine can use. Although multithreaded reading works in a single *Get(adios2::Mode::Sync)* call if the read selection spans multiple data blocks in the file, the best parallelization is achieved by using deferred mode and reading everything in *PerformGets()/EndStep()*.

#. **FlattenSteps**: This is a writer-side parameter specifies that the
reader should interpret multiple writer-created timesteps as a
single timestep, essentially flattening all Put()s into a single step.

#. **IgnoreFlattenSteps**: This is a reader-side parameter that
tells the reader to ignore any FlattenSteps parameter supplied
to the writer.

============================== ===================== ===========================================================
**Key** **Value Format** **Default** and Examples
============================== ===================== ===========================================================
Expand All @@ -156,6 +164,8 @@ This engine allows the user to fine tune the buffering operations through the fo
StatsLevel integer, 0 or 1 **1**, 0
MaxOpenFilesAtOnce integer >= 0 **UINT_MAX**, 1024, 1
Threads integer >= 0 **0**, 1, 32
FlattenSteps boolean **off**, on, true, false
IgnoreFlattenSteps boolean **off**, on, true, false
============================== ===================== ===========================================================


Expand Down
5 changes: 3 additions & 2 deletions docs/user_guide/source/setting_up/source/cmake.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ To build ADIOS v2.x, clone the repository and invoke the canonical CMake build s
$ git clone https://github.com/ornladios/ADIOS2.git ADIOS2
$ mkdir adios2-build && cd adios2-build
$ cmake ../ADIOS2 cmake -DADIOS2_BUILD_EXAMPLES=ON
$ cmake ../ADIOS2 -DADIOS2_BUILD_EXAMPLES=ON
-- The C compiler identification is GNU 9.4.0
-- The CXX compiler identification is GNU 9.4.0
...
Expand Down Expand Up @@ -111,10 +111,11 @@ Optionally, run the tests (need to configure with ``-DBUILD_TESTING=ON`` cmake f
Total Test time (real) = 95.95 sec
And finally, use the standard invocation to install:
And finally, use the standard invocation to install (setting the install path beforehand):

.. code-block:: bash
$ cmake ../ADIOS2 -DCMAKE_INSTALL_PREFIX=/path/to/where/adios/will/be/installed
$ make install
Expand Down
3 changes: 2 additions & 1 deletion docs/user_guide/source/tutorials/helloWorld.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -86,14 +86,15 @@ Start editing the skeleton file `ADIOS2/examples/hello/helloWorld/hello-world_tu

adios2::Engine reader = io.Open("hello-world-cpp.bp", adios2::Mode::Read);
std::string greeting;
reader.BeginStep();
reader.Get(varGreeting, greeting);
reader.EndStep();
reader.Close();
return greeting;

.. note::

The ``BeginStep`` and ``EndStep`` calls are required when **reading** one step and multiple steps. We will see in
In Mode::Read, the ``BeginStep`` and ``EndStep`` calls are required when **reading** one step and multiple steps. We will see in
another tutorial how to read multiple steps. It's important to note that the ``BeginStep`` should be called **before**
all ``Inquire*`` / ``Available*`` function calls.

Expand Down
40 changes: 20 additions & 20 deletions source/adios2/common/ADIOSMacros.h
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,7 @@
</pre>
*/
#define ADIOS2_FOREACH_ATTRIBUTE_PRIMITIVE_STDTYPE_1ARG(MACRO) \
MACRO(char) \
MACRO(int8_t) \
MACRO(int16_t) \
MACRO(int32_t) \
Expand All @@ -43,24 +44,24 @@
MACRO(uint64_t) \
MACRO(float) \
MACRO(double) \
MACRO(long double) \
MACRO(char)
MACRO(long double)

#define ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO) \
ADIOS2_FOREACH_ATTRIBUTE_PRIMITIVE_STDTYPE_1ARG(MACRO) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>)

#define ADIOS2_FOREACH_ATTRIBUTE_STDTYPE_1ARG(MACRO) \
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO) \
MACRO(std::string)
MACRO(std::string) \
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO)

#define ADIOS2_FOREACH_STDTYPE_1ARG(MACRO) \
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO) \
MACRO(std::string)
MACRO(std::string) \
ADIOS2_FOREACH_PRIMITIVE_STDTYPE_1ARG(MACRO)

#define ADIOS2_FOREACH_TYPE_1ARG(MACRO) \
MACRO(std::string) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(short) \
Expand All @@ -75,10 +76,10 @@
MACRO(double) \
MACRO(long double) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>) \
MACRO(char)
MACRO(std::complex<double>)

#define ADIOS2_FOREACH_PRIMITIVE_TYPE_1ARG(MACRO) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(short) \
Expand All @@ -93,18 +94,17 @@
MACRO(double) \
MACRO(long double) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>) \
MACRO(char)
MACRO(std::complex<double>)

#define ADIOS2_FOREACH_COMPLEX_PRIMITIVE_TYPE_1ARG(MACRO) \
MACRO(float) \
MACRO(double) \
MACRO(long double)

#define ADIOS2_FOREACH_CHAR_TYPE_1ARG(MACRO) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(char)
MACRO(unsigned char)

#define ADIOS2_FOREACH_NUMERIC_TYPE_1ARG(MACRO) \
MACRO(short) \
Expand All @@ -123,6 +123,7 @@

#define ADIOS2_FOREACH_ATTRIBUTE_TYPE_1ARG(MACRO) \
MACRO(std::string) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(short) \
Expand All @@ -137,10 +138,10 @@
MACRO(double) \
MACRO(long double) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>) \
MACRO(char)
MACRO(std::complex<double>)

#define ADIOS2_FOREACH_ATTRIBUTE_PRIMITIVE_TYPE_1ARG(MACRO) \
MACRO(char) \
MACRO(signed char) \
MACRO(unsigned char) \
MACRO(short) \
Expand All @@ -155,8 +156,7 @@
MACRO(double) \
MACRO(long double) \
MACRO(std::complex<float>) \
MACRO(std::complex<double>) \
MACRO(char)
MACRO(std::complex<double>)

/**
<pre>
Expand Down Expand Up @@ -185,6 +185,7 @@
#define ADIOS2_FOREACH_ATTRIBUTE_STDTYPE_2ARGS(MACRO) \
MACRO(std::string, string) \
MACRO(int8_t, int8) \
MACRO(char, char) \
MACRO(uint8_t, uint8) \
MACRO(int16_t, int16) \
MACRO(uint16_t, uint16) \
Expand All @@ -196,11 +197,11 @@
MACRO(double, double) \
MACRO(long double, ldouble) \
MACRO(std::complex<float>, cfloat) \
MACRO(std::complex<double>, cdouble) \
MACRO(char, char)
MACRO(std::complex<double>, cdouble)

#define ADIOS2_FOREACH_PRIMITVE_STDTYPE_2ARGS(MACRO) \
MACRO(int8_t, int8) \
MACRO(char, char) \
MACRO(uint8_t, uint8) \
MACRO(int16_t, int16) \
MACRO(uint16_t, uint16) \
Expand All @@ -212,8 +213,7 @@
MACRO(double, double) \
MACRO(long double, ldouble) \
MACRO(std::complex<float>, cfloat) \
MACRO(std::complex<double>, cdouble) \
MACRO(char, char)
MACRO(std::complex<double>, cdouble)

#define ADIOS2_FOREACH_MINMAX_STDTYPE_2ARGS(MACRO) \
MACRO(int8_t, int8) \
Expand Down
7 changes: 7 additions & 0 deletions source/adios2/core/VariableBase.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -221,6 +221,13 @@ void VariableBase::SetMemorySelection(const Box<Dims> &memorySelection)
const Dims &memoryStart = memorySelection.first;
const Dims &memoryCount = memorySelection.second;

if (memoryStart.empty() && memoryCount.empty())
{
m_MemoryStart.clear();
m_MemoryCount.clear();
return;
}

if (m_SingleValue)
{
helper::Throw<std::invalid_argument>("Core", "VariableBase", "SetMemorySelection",
Expand Down
Loading

0 comments on commit f691736

Please sign in to comment.