ADIOS v2.8.1 Production Patch Release

Summary

This is a patch release to address many of the issues identified with the v2.8.0 release after it was published.
For the new BP5 engine the fixes are

• SelectSteps were always showing the first step in the file even if not selected
• AppendAfterStep created a corrupted index file
• bpls did not show the actual values of Global Values and Local Values (shown as 1D arrays)
• bpls was not able to dump data of variables beyond the first step

Changes / Fixes

PR #3237
PR #3236
PR #3234
PR #3228
PR #3227
PR #3221
PR #3213
PR #3211
PR #3207
PR #3206
PR #3186
PR #3176
PR #3162
PR #3161
PR #3156
PR #3155
PR #3145
PR #3153

ADIOS v2.8.0 Production Release

Summary

This is a major production release with a variety of new features and bug fixes.

Changes to the API

• adios2::Mode::ReadRandomAccess mode is introduced for reading files with access to all steps. BeginStep/EndStep calls are NOT allowed. SetStepSelection() can be used to access specific steps in the file.
• adios2::Mode::Read mode now requires using BeginStep/EndStep loop to access steps serially one by one. Variable inquiry fails outside BeginStep/EndStep sections. You need to modify your Open() statement to use the random-access mode if your code is accessing all steps in an existing file. Technically, this is only true for the new BP5 files, you can still read BP3/BP4/HDF5 files with your existing code but keep in mind that your code is not going to work when moving to BP5.
• adios2::ADIOS::EnterComputationBlock(), adios2::ADIOS::ExitComputationBlock() are hints to ADIOS that a process is in a computing (i.e. non-communicating) phase. BP5 asynchronous I/O operations can schedule writing during such phases to avoid interfering with the application’s own communication.
• GPU-aware I/O supports passing device-memory data pointers to the Put/Get functions, and ADIOS2 will automatically download/upload data from/to the device during I/O. Alternatively, an extra member function of the Variable class, SetMemorySpace(const adios2::MemorySpace mem) can explicitly tell ADIOS2 whether the pointer points to device memory or host memory.

Notable new features

BP5

BP5 is a new data format and there is a new engine, BP5, that writes and reads in this format. See https://adios2.readthedocs.io/en/latest/engines/engines.html for runtime options. There are four main advantages using this over the previous engine BP4.

• Lower memory consumption. Deferred Puts use user buffer for I/O wherever possible thus saving on a memory copy. Aggregation uses a fixed-size shared-memory segment on each compute node instead of using MPI to send data from one process to another. Memory consumption can save 50% of extra memory usage compared to BP4 in some cases.
• Faster metadata management improves write/read performance where hundreds or more variables are added to the output.
• Improved functionality around appending many output steps into the same file. Better performance than writing new files each step. Restart can append to an existing series by truncating unwanted steps. Readers can filter out unwanted steps to only see and process a limited set of steps. Just like as in BP4, existing steps cannot be corrupted by appending new steps. Existing steps can be read by another application while new steps are written and the reader can wait for incoming steps, i.e., in situ processing through files works identically to using staging engines.
• Asynchronous writing is an option with BP5, which can further improve performance in applications. This is a simple runtime option to turn it on/off, there is no need to tinker with I/O calls in the source code. This asynchronous I/O implementation does NOT need a multithreaded MPI library.

GPU Support

• IO: ADIOS can now write and read data to and from CUDA device memory by passing a valid device pointer to the appropriate Put and Get API calls. Current support is limited to CUDA on NVIDIA GPUs but future releases will support both AMD and Intel GPUs.
• Compression: The ZFP floating point compression operator has additional configurations options and now supports compression and decompression operations on NVIDIA GPUs for variable data on either host or device memory.

Further details on GPU-Aware I/O can be found in the ADIOS user guide

Plugin architecture for Engines and Operators

Engines and Operators can now be developed outside of ADIOS and loaded at runtime. For engine plugins, the engine should inherit from the PluginEngineInterface class. The engine library should also contain EngineCreate() and EngineDestroy() symbols that ADIOS will be able to use to create and destroy your engine. Similarly, Operator plugins should inherit from the PluginOperatorInterface class and the library should contain OperatorCreate() and OperatorDestroy() symbols. To help ADIOS find user plugin libraries, the ADIOS2_PLUGIN_PATH environment variable can be set. Further details on writing and using plugins can be found in the ADIOS user guide.

Compression operators backward-compatibility feature

From version 2.8.0 onward Operators manage their own metadata and their future versions will provide information on the (compression) libraries used in older datasets in case they cannot decompress the data with the current compression library it is built with. Users then may be able to rebuild the future ADIOS version with an older, compatible compression library that can read back their data. The downside of the redesign of Operators is that datasets written with lossy compression cannot be read back with ADIOS 2.8.0 and newer versions, so you need to build and use 2.7.1 for that.

Full Changelog: v2.7.1...v2.8.0

NWChem-ADIOS2-R-Vis

This is an intermediate release for the sole purpose of having a tagged easily reproducible build for the "Interleaving R with Simulations and Visualization for streaming on Large Distributed Systems via the ADIOS2 Data Transport Framework" paper submitted to the Journal of Statistical Software

ADIOS v2.7.1 Production Patch Release

Summary

This is a minor patch release to address outstanding bugs that arose after the 2.7.0 release was deployed.

Bug fixes

• #2582 Installation of bp2h5 wrappers fails on Windows due to symlinks
  • Fixed by #2583
• #2585 Update documentation for using both serial and MPI libraries from a single build.
• #2018 Blosc compression doesn't support variables over 2GiB
  • Fixed by #2592
• #2596 Python shouldn't be required if testing and Python bindings are disabled

ADIOS v2.7.0 Production Release

Summary

This is a major production release with a variety of new features and bug fixes in the areas of staging, HDF5 compatibility, and writer aggregation.

Features

New features

• Hierarchical traversing for reading ADIOS files using the new Group object
  • This is an optional, alternative way to go over the variables and attributes in a file/stream in a similar fashion as one traverses HDF5 files.
• Default aggregation mode has changed to 1 sub-file per compute node (from one sub-file per process)
• DataMan engine supports compression (used for wide-area-network data transfers)
• SSC engine supports flexible I/O pattern
• Step selection for reading
• Simplified wrapper bp2h5 for the adios2_reorganize utility for converting ADIOS BP files to HDF5 files.

HDF5 Virtual Object Layer (VOL) Plugin for ADIOS

• Added a VOL plugin to enable ADIOS I/O through the HDF5 API. Basic functionalities are covered.
• Enables existing HDF5 clients to create and access ADIOS files with minimal effort.

API Changes

• Double pointer Get API for the Inline engine
• AggregatorRatio option to set number of sub-files as a ratio of number of processes. NumAggregators option is used to set a fixed number of sub-files. See options for https://adios2.readthedocs.io/en/latest/engines/engines.html#bp4

ADIOS v2.6.0 Production Release

Summary

This is a major release with a wide variety of features in all areas including staging, file I/O, build system, and many others.

Features

Build System

• When enabling MPI the build system now generates both serial and parallel libraries. One installation of ADIOS is needed only for building both parallel and serial applications
• CMake targets now support components for each supported language, i.e. instead of just adios2 there are now adios2::cxx11 and adios2::cxx11_mpi
• All third party dependencies can now be provided externally rather than use internal copies

Engines

• BP4 Engine: Burst buffer support for writing. File on the final target file system can still be read step-by-step in situ by other apps. Appending to an existing file, however, is NOT supported.
• New engine SSC: MPI-based engine for code coupling and in situ processing
• DataMan engine: new reliable mode, that guarantees that every step arrives at the reader
• Inline engine: is now being elevated from experimental to beta.
• SST engine:
  • With the non-RDMA transport, IP Port range now defaults to "any", rather than "26000-26100". (The desired range can be specified with the ADIOS2_PORT_RANGE environment variable, see sst_conn_tool documentation for more information.)
  • SST now has a "push mode" where written data is pro-actively sent to readers without waiting for specific requests. This "push mode" is either speculative (based on an assumption that data will be used) or learned (based on actual usage).
    • Speculative push mode is enabled automatically in certain circumstances, managed by SST engine parameters SpeculativePreloadMode and SpecAutoNodeThreshold.
    • Learned preload mode is triggered when the reader and writer indicate that their data consumption/production patterns are static by calling LockReaderSelections and LockWriterDefinitions, respectively.

API Changes (do not break compilation)

• DebugMode now deprecated from the public constructor and factory interfaces. In previous releases this flag controlled enabling or disabling additional error checking. Through extensive bench-marking of API calls it was determined that the savings was negligible so the appropriate error checking (null pointer, bounds, etc.) is now always performed within API calls and throwing exceptions (in C++) or returning error codes (C and Fortran) where appropriate.
• Python available_variables() now has a filter list argument: available_variables(['type']) will return faster but only provides names and types of variables. For backward compatibility, available_variables() without arguments is identical to a call with the full list available_variables(['Type', 'Min', 'Max', 'SingleValue','Shape','AvailableStepsCount']). The filter names are case-insensitive.
• YAML configuration file. adios2::ADIOS constructor can now accept a configuration file using the YAML format and it must have the *.yaml or *.yml extension. See section in docs.

ADIOS v2.5.0 Production Release

Summary

ADIOS v2.5.0, September 30th 2019 This release should be considered stable and suitable for deployment. We highly recommend migration to this version for long term support and future API compatibility. This is a major update which includes the introduction of a new directory-based file format, BP4, for the default Engine targeting streaming through files applications.

Features

• Virtual Engines (File, FileStream, InSituAnalysis, InSituVisualization, CodeCoupling)
• BP4 file engine has a constant append time of new steps. Faster than creating new output at every step.
• Default is now the BP4 file engine that supports reading streaming concurrently while the writer is still producing new steps
• Zero-copy access to the engine buffer memory in Write mode. Uses Variable::Span per Engine Put block, based on upcoming C++20 std::span. Available in BP3 (since v2.4.0) and BP4 (since v2.5.0) Engines C++ APIs only.

API Breakage

• Python high-level API: ability to retrieve single global values as numpy's 0-dimension array This breaks API compatibility with v2.4.0 Python high-level API

ADIOS v2.4.0 Production Release

Summary

ADIOS v2.4.0, June 30th 2019. This release should be considered stable and suitable for deployment. We highly recommend migration to this version for long term support and future API compatibility.

Features

• Automatic conversion to fixed width types (int -> int32_t), in all bindings to facilitate portable types and bpls output
• MPI library "serial" functionality. ADIOS2 library compiled with MPI can run with code not invoking MPI_Init and MPI_Finalize running in serial. Requires linking with MPI for symbols.
• Stable high-level APIs in Python and C++ supporting local variables
• Zero-copy access to the engine buffer memory in Write mode. Uses Variable<T>::Span per Engine Put block, based on upcoming C++20 std::span. Available in BP3 Engine C++ APIs only.
• New functionality in Fortran and C APIs for safer handling of subroutines/functions outputting strings
• Addition of no-op backends to disable selected functionality and / or evaluate the performance impact of various layers
  • null Engine : Bypass all library actions as soon as possible, often in the high-level language bindings before the core library is even reached
  • nullcore Engine: Implemented in the core of the library allowing the front-end data management and variable operations to occur but to transparently ignore all write and buffering requests.
  • null Transport : For use with the BP engines, this will bypass all disk I/O operations. All of the data management and buffering will still occur but there will be no actual data written to a file.
• Optimization of data exchanges in the SST engine when a fixed data exchange pattern is specified via LockWriterDefinitions/LockReaderSelections
• PreciousFirstTimestep engine mode allows saving mesh and other one-time-output in a stream, so that late-comer readers can properly initialize from the first input step. Available in SST engine only.
• Added support for MemorySelection for reading into non-contiguous memory (e.g. ghost cells). Available in BP3 Engine only.
• Fixed several bugs: substreams aggregation, endian interoperability, local variables and bpls
• Complete redesign of the user guide documentation in readthedocs
• Added several new lossless compression plugins in BP buffer: bzip2, libpng, blosc. Available in BP and SST engines only.
• Availability of tools interface with connection to TAU 2.28 or greater.

Changes

• LatestAvailable reading mode has been removed from BeginStep() and is now specified at Open() with AlwaysProvideLatestTimestep engine parameter. This allows for more optimized queueing behavior in staging engines. This breaks API compatibility with v2.3.x

• IO.LockDefinitions() function has been removed and replaced with Engine.LockWriterDefinitions() and Engine.LockReaderSelections() to improve clarity and separate functionality. As before, calling these functions informs ADIOS2 that the write/read pattern between two applications in staging is unchanging from timestep to timestep (after the calls). This allows certain optimizations, such as eliminating some metadata exchanges and pre-loading data to the readers. This breaks API compatibility with v2.3.x

• Data queueing in the SST engine has been redesigned, adding more explicit support for retaining timesteps for late-arriving readers (see the ReserveQueueLimit engine parameter), but also changing the way timesteps are handled when there are no readers present. In the no-readers case, specifying QueueFullPolicy:Blocking will no longer cause SST to pause until a reader arrives. This behavior is a change from v2.3.x

ADIOS v2.4.0-RC1 Release Candidate

Summary

This is a release candidate for the upcoming v2.4.0 release in June 30th 2019. The v2.4.0-rc1 release should be considered stable and suitable for testing deployment as a module on HPC platforms. We recommend users to migrate to this version to test for the upcoming v2.4.0 and report any outstanding issue.

Features

• Moved completely to fix width types in all bindings to facilitate portable types and bpls output
• Stable high-level APIs for Python and C++ supporting local variables
• Added NULL Engine to measure Engine overall overhead
• Variable<T>::Span provide access to the engine buffer memory in Write mode (C++ APIs) similar to C++20 std::span of a std::vector
• New functionality in Fortran and C APIs for handling strings in a safer mode
• Fixed several bugs: Substreams aggregation, endian interoperability, bpls
• Update documentation: adios2.readthedocs.io

ADIOS v2.3.1 Production Release

Summary

This is a bug-fix release on top of the v2.3.0 release. The v2.3.1 release should be considered stable and suitable for deployment as a module on HPC platforms.

Features

Bug fixes beyond the v2.3.0 release include:

• Various CMake fixes
  • #1203 #1204
• Statically generate adios2-config at install time so cmake and python are no longer required
  • #1172 #1191
• Various bpls bug fixes
  • #1175 #1182 #1211
• Add missing attributes to high level API
  • #1146 #1197
• Various SST bug fixes
  • #1157 #1167 #1152 #1151 #1149 #1186 #1183 #1201 #1207 #1213
• HDF5 Single value attributes
  • #1159
• MGARD tweaks
  • #1196