BP5 AppendAfterSteps: Data corruption issues persist

[franzpoeschel]

Using AppendAfterSteps still results in errors like in #3225, but they're harder to trigger now on the master branch. I was not able to find a minimal ADIOS2-only example this time, so I'll upload different versions of the dataset instead.

Describe the bug
The dataset before appending:
before_appending.zip

$ bpls ../samples/append/append_groupbased.bp5/ -al
  string    /basePath                                  attr   = "/data/%T/"
  double    /data/dt                                   attr   = 1
  string    /data/meshes/E/axisLabels                  attr   = {"x"}
  string    /data/meshes/E/dataOrder                   attr   = "C"
  string    /data/meshes/E/geometry                    attr   = "cartesian"
  double    /data/meshes/E/gridGlobalOffset            attr   = {0}
  double    /data/meshes/E/gridSpacing                 attr   = {1}
  double    /data/meshes/E/gridUnitSI                  attr   = 1
  float     /data/meshes/E/timeOffset                  attr   = 0
  double    /data/meshes/E/unitDimension               attr   = {0, 0, 0, 0, 0, 0, 0}
  int32_t   /data/meshes/E/x                           10*{10} = 0 / 0
  double    /data/meshes/E/x/position                  attr   = {0}
  double    /data/meshes/E/x/unitSI                    attr   = 1
  uint64_t  /data/snapshot                             attr   = 11
  double    /data/time                                 attr   = 0
  double    /data/timeUnitSI                           attr   = 1
  string    /date                                      attr   = "2022-11-17 09:55:54 +0000"
  string    /iterationEncoding                         attr   = "variableBased"
  string    /iterationFormat                           attr   = "/data"
  string    /meshesPath                                attr   = "meshes/"
  string    /openPMD                                   attr   = "1.1.0"
  uint32_t  /openPMDextension                          attr   = 0
  string    /software                                  attr   = "openPMD-api"
  string    /softwareVersion                           attr   = "0.15.0-dev"
  uint64_t  __openPMD_groups/data                      attr   = 9
  uint64_t  __openPMD_groups/data/meshes               attr   = 9
  uint64_t  __openPMD_groups/data/meshes/E             attr   = 9
  uint64_t  __openPMD_internal/openPMD2_adios2_schema  attr   = 20220726
  uint8_t   __openPMD_internal/useSteps                attr   = 1

Appending without specifying AppendAfterSteps:
append_without_appendaftersteps.zip

$ bpls ../samples/append/append_groupbased.bp5/ -al
  string    /basePath                                  attr   = "/data/%T/"
  double    /data/dt                                   attr   = 1
  string    /data/meshes/E/axisLabels                  attr   = {"x"}
  string    /data/meshes/E/dataOrder                   attr   = "C"
  string    /data/meshes/E/geometry                    attr   = "cartesian"
  double    /data/meshes/E/gridGlobalOffset            attr   = {0}
  double    /data/meshes/E/gridSpacing                 attr   = {1}
  double    /data/meshes/E/gridUnitSI                  attr   = 1
  float     /data/meshes/E/timeOffset                  attr   = 0
  double    /data/meshes/E/unitDimension               attr   = {0, 0, 0, 0, 0, 0, 0}
  int32_t   /data/meshes/E/x                           12*{10} = 0 / 0
  double    /data/meshes/E/x/position                  attr   = {0}
  double    /data/meshes/E/x/unitSI                    attr   = 1
  uint64_t  /data/snapshot                             attr   = 5
  double    /data/time                                 attr   = 0
  double    /data/timeUnitSI                           attr   = 1
  string    /date                                      attr   = "2022-11-17 10:02:01 +0000"
  string    /iterationEncoding                         attr   = "variableBased"
  string    /iterationFormat                           attr   = "/data"
  string    /meshesPath                                attr   = "meshes/"
  string    /openPMD                                   attr   = "1.1.0"
  uint32_t  /openPMDextension                          attr   = 0
  string    /software                                  attr   = "openPMD-api"
  string    /softwareVersion                           attr   = "0.15.0-dev"
  uint64_t  __openPMD_groups/data                      attr   = 11
  uint64_t  __openPMD_groups/data/meshes               attr   = 11
  uint64_t  __openPMD_groups/data/meshes/E             attr   = 11
  uint64_t  __openPMD_internal/openPMD2_adios2_schema  attr   = 20220726
  uint8_t   __openPMD_internal/useSteps                attr   = 1

Appending with AppendAfterSteps = -3:
append_with_appendaftersteps.zip

$ bpls ../samples/append/append_groupbased.bp5/ -al
Failed to open with BPFile engine: [Thu Nov 17 10:07:26 2022] [ADIOS2 EXCEPTION] <Engine> <BP5Reader> <ParseMetadataIndex> : Record 9 (id = 115) has invalid length 48. We parsed 6914080 bytes for this record

Failed to open with HDF5 engine: [Thu Nov 17 10:07:26 2022] [ADIOS2 EXCEPTION] <Engine> <HDF5ReaderP> <HDF5ReaderP> : Invalid HDF5 file found


Error: Could not open this file with any ADIOS2 file reading engines

To Reproduce
As described above, I have no minimal example.
I used the following openPMD test case to trigger the error (special branch needed to support BP5 https://github.com/franzpoeschel/openPMD-api/tree/adios2-bugreport). The test case first appends several times normally, then a final time with specifying AppendAfterSteps = -3.

void append_mode(
    std::string const &filename,
    bool variableBased,
    ParseMode parseMode,
    std::string jsonConfig = "{}")
{
    if (auxiliary::directory_exists("../samples/append"))
    {
        auxiliary::remove_directory("../samples/append");
    }
    std::vector<int> data(10, 999);
    auto writeSomeIterations = [&data](
                                   WriteIterations &&writeIterations,
                                   std::vector<uint64_t> indices) {
        for (auto index : indices)
        {
            auto it = writeIterations[index];
            auto dataset = it.meshes["E"]["x"];
            dataset.resetDataset({Datatype::INT, {10}});
            dataset.storeChunk(data, {0}, {10});
            // test that it works without closing too
            it.close();
        }
    };
    {
        Series write(filename, Access::APPEND, jsonConfig);
        if (variableBased)
        {
            write.setIterationEncoding(IterationEncoding::variableBased);
        }
        writeSomeIterations(
            write.writeIterations(), std::vector<uint64_t>{0, 1});
    }
    {
        Series write(filename, Access::APPEND, jsonConfig);
        if (variableBased)
        {
            write.setIterationEncoding(IterationEncoding::variableBased);
        }

        writeSomeIterations(
            write.writeIterations(), std::vector<uint64_t>{3, 2});
        write.flush();
    }
    {
        using namespace std::chrono_literals;
        /*
         * Put a little sleep here to trigger writing of a different /date
         * attribute. ADIOS2 v2.7 does not like that so this test ensures that
         * we deal with it.
         */
        std::this_thread::sleep_for(1s);
        Series write(filename, Access::APPEND, jsonConfig);
        if (variableBased)
        {
            write.setIterationEncoding(IterationEncoding::variableBased);
        }

        writeSomeIterations(
            write.writeIterations(), std::vector<uint64_t>{4, 3, 10});
        write.flush();
    }
    {
        Series write(filename, Access::APPEND, jsonConfig);
        if (variableBased)
        {
            write.setIterationEncoding(IterationEncoding::variableBased);
        }

        writeSomeIterations(
            write.writeIterations(), std::vector<uint64_t>{7, 1, 11});
        write.flush();
    }

    if (auxiliary::ends_with(filename, ".bp5"))
    {
        {
            Series write(
                filename,
                Access::APPEND,
                json::merge(
                    jsonConfig,
                    R"({"adios2":{"engine":{"parameters":{"AppendAfterSteps": -3}}}})"));
            if (variableBased)
            {
                write.setIterationEncoding(IterationEncoding::variableBased);
            }

            writeSomeIterations(
                write.writeIterations(), std::vector<uint64_t>{4, 5});
            write.flush();
        }
        {
            Series read(filename, Access::READ_LINEAR);
            for (auto iteration : read.readIterations())
            {
                std::cout << "Seeing iteration " << iteration.iterationIndex << std::endl;
            }
        }
        if (!variableBased)
        {
            Series read(filename, Access::READ_ONLY);
            for (auto const &iteration : read.readIterations())
            {
                std::cout << "Seeing iteration " << iteration.iterationIndex << std::endl;
            }
        }
    }
}

TEST_CASE("append_mode", "[serial]")
{
    std::string jsonConfigNew = R"END(
{
    "adios2":
    {
        "schema": 20220726,
        "engine":
        {
            "usesteps" : true
        }
    }
})END";
    append_mode(
        "../samples/append/append_groupbased.bp5",
        true,
        ParseMode::WithSnapshot,
        jsonConfigNew);
}

Expected behavior
The last two steps of the original dataset should be truncated, the new steps should be added, the dataset shoud be readable.

Desktop (please complete the following information):

• OS/Platform: Noticed on various platforms, including Debian and Ubuntu
• Build: Gnu 11.1.0 Debug, Gnu 11.3.0 Release
• Git tag 9c9e4ca (recent master)

Additional context
Add any other context about the problem here.

Following up
Was the issue fixed? Please report back.

[pnorbert]

#3447 Should fix this issue.

[franzpoeschel]

I just tested this, it does!
The test that has been failing for months is now running green again. Same goes for the branch that I specified above.