ADIOS2: v2.7.0+ #927

ax3l · 2021-02-10T23:58:06Z

Increase ADIOS2 dependency to v2.7.0+

update docs and build logic @ax3l
bump CI to have access to newer ADIOS @ax3l
remove pure ADIOS<2.7.0 work-arounds from code-base @franzpoeschel

franzpoeschel · 2021-02-12T09:39:05Z

docs/source/backends/adios2.rst

@@ -22,6 +22,7 @@ Steps

 ADIOS2 is optimized towards organizing the process of reading/writing data into IO steps.
 In order to activate steps, it is imperative to use the :ref:`Streaming API <usage-streaming>` (which can be used for either file-based or streaming-based workflows).
+@franzpoeschel: let's remove these work-arounds :-)


Removing this workaround would be a breaking change to our written datasets, making them unreadable with old versions of the openPMD API. I'd postpone the removal of this workaround until switching to the new schema (I've re-set the defaults accordingly for the new schema in #855).

Sounds great!
Do you like to rephrase the section here then? It might be confusing for us and users to talk about 2.6.0 if we mean the openPMD 1 layout after this PR (because this PR drops ADIOS 2.6.0). Maybe you can use the same phrasing as in the doxygen comment (backwards compatible with 2.6.0 output that needed to work around a bug...)

Ah, I somehow missed this one. Updated it.

franzpoeschel · 2021-02-12T09:51:54Z

include/openPMD/IO/ADIOS/ADIOS2IOHandler.hpp

@@ -1245,6 +1245,8 @@ namespace detail
             *    is opt-in for now.
             * 2) Reading with the Streaming API any file that has been written
             *    without steps.
+             *
+             * @todo [franzpoeschel] let's remove these work-arounds :-)


The description in this one is a bit incomplete. As of ADIOS 2.7.0, it is still necessary to delay opening the first step in case a reader does not use steps at all.
But in any case, I'll need to update the description of this (and also mention somewhere that its difference to the state OutsideOfStep lies in the fact that requireActiveStep() will not open a step).

franzpoeschel · 2021-02-12T11:50:18Z

Hmm, I tried to activate the tests under Windows that I thought were failing because of ADIOS 2.6.0. Apparently, they're still failing..

franzpoeschel · 2021-02-17T10:04:46Z

I threw out the failing commit, but something else went wrong again

franzpoeschel · 2021-02-24T13:10:42Z

I've fixed the SSC issues. Also, rebased since there was a conflict.

ax3l · 2021-03-04T01:19:11Z

src/IO/ADIOS/ADIOS2PreloadAttributes.cpp

@@ -102,7 +102,10 @@ namespace detail
                {
                    numItems *= extent;
                }
-                new( dest ) T[ numItems ]{};
+                for( size_t i = 0; i < numItems; ++i )


That construct looks expensive, why do we need to loop this now?
Update: just saw #926 (comment)

Just to get a feeling, how large is numItems usually?
Update: I just learned what a placement new is, so it's probably not more expensive to loop here? :D

I trust that you know what you are doing: https://stackoverflow.com/a/36854950/2719194 :)

Ah yes, should have written something on that one here too.
In most cases, numItems is 1, for our unit dimension things it is 7. I think, the largest instances are for the VEC_STRING datatype since those are represented as 2D char vectors, so you'll get some 2-digit number there.

so it's probably not more expensive to loop here

I think it shouldn't be since this doesn't allocate anything.

Theoretically, we could also memset the whole thing to zero and use placement new only for non-basic datatypes, but that makes things very finicky to deal with and I don't think it's worth.

Also, I don't use delete[], see struct AttributeLocationDestroy in the same file for the destructors.

But yeah, this file was tricky to get right

Increase ADIOS2 dependency to v2.7.0+

This one does not yet have E4S caches, so build times might go up a bit.

To do this, add .ssc as a possible file ending and enlist it under the file endings.

This also fixes broken array placement new in MSVC

franzpoeschel · 2021-03-08T11:31:42Z

Rebased this again to get the frontend redesign (I use this as base for some other PRs).

ax3l added the backend: ADIOS2 label Feb 10, 2021

ax3l requested a review from franzpoeschel February 10, 2021 23:58

ax3l assigned ax3l and franzpoeschel Feb 10, 2021

ax3l force-pushed the adios-2.7.0 branch 2 times, most recently from ceefa82 to 76d4e2d Compare February 11, 2021 00:02

franzpoeschel reviewed Feb 12, 2021

View reviewed changes

franzpoeschel force-pushed the adios-2.7.0 branch from d5de68a to 76d4e2d Compare February 17, 2021 09:56

franzpoeschel force-pushed the adios-2.7.0 branch 2 times, most recently from befbd74 to b7009eb Compare February 24, 2021 13:10

franzpoeschel mentioned this pull request Feb 24, 2021

Resizable Datasets #829

Merged

6 tasks

franzpoeschel force-pushed the adios-2.7.0 branch from b7009eb to 80786d8 Compare February 24, 2021 13:57

ax3l commented Mar 4, 2021

View reviewed changes

franzpoeschel mentioned this pull request Mar 8, 2021

New attribute layout: Read from either int8_t or uint8_t #926

Merged

2 tasks

franzpoeschel force-pushed the adios-2.7.0 branch from 37de671 to 32916ae Compare March 8, 2021 11:15

franzpoeschel mentioned this pull request Mar 8, 2021

Use schema-based versioning in ADIOS2 backend #941

Merged

7 tasks

ax3l and others added 7 commits March 8, 2021 12:24

ADIOS2: v2.7.0+

a0676ea

Increase ADIOS2 dependency to v2.7.0+

CI: Newer Spack Version

6d9140e

This one does not yet have E4S caches, so build times might go up a bit.

[Draft] Remove Work-Arounds for ADIOS2<2.7.0

75dbdf4

Activate ADIOS2 SSC test

5781c8d

Update some comments on ADIOS2 workarounds

dc73321

Only run SSC tests if ADIOS2 has been built with SSC

5de0b20

To do this, add .ssc as a possible file ending and enlist it under the file endings.

Activate tests for new ADIOS2 schema on MSVC

3b0f40b

This also fixes broken array placement new in MSVC

franzpoeschel mentioned this pull request Mar 8, 2021

[WIP] ADIOS: 2.7.0+ #911

Closed

franzpoeschel force-pushed the adios-2.7.0 branch from 32916ae to 3b0f40b Compare March 8, 2021 12:20

Update documentation in adios2.rst

52de1b6

ax3l merged commit 969b430 into openPMD:dev Mar 8, 2021

ax3l deleted the adios-2.7.0 branch March 8, 2021 19:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ADIOS2: v2.7.0+ #927

ADIOS2: v2.7.0+ #927

ax3l commented Feb 10, 2021 •

edited by franzpoeschel

Loading

franzpoeschel Feb 12, 2021

ax3l Feb 24, 2021 •

edited

Loading

franzpoeschel Mar 8, 2021

franzpoeschel Feb 12, 2021

franzpoeschel commented Feb 12, 2021

franzpoeschel commented Feb 17, 2021

franzpoeschel commented Feb 24, 2021

ax3l Mar 4, 2021 •

edited

Loading

franzpoeschel Mar 8, 2021

franzpoeschel commented Mar 8, 2021

ADIOS2: v2.7.0+ #927

ADIOS2: v2.7.0+ #927

Conversation

ax3l commented Feb 10, 2021 • edited by franzpoeschel Loading

franzpoeschel Feb 12, 2021

Choose a reason for hiding this comment

ax3l Feb 24, 2021 • edited Loading

Choose a reason for hiding this comment

franzpoeschel Mar 8, 2021

Choose a reason for hiding this comment

franzpoeschel Feb 12, 2021

Choose a reason for hiding this comment

franzpoeschel commented Feb 12, 2021

franzpoeschel commented Feb 17, 2021

franzpoeschel commented Feb 24, 2021

ax3l Mar 4, 2021 • edited Loading

Choose a reason for hiding this comment

franzpoeschel Mar 8, 2021

Choose a reason for hiding this comment

franzpoeschel commented Mar 8, 2021

ax3l commented Feb 10, 2021 •

edited by franzpoeschel

Loading

ax3l Feb 24, 2021 •

edited

Loading

ax3l Mar 4, 2021 •

edited

Loading