Adios2 with Melissa

[mschouler]

Hi, I am a developer in the Melissa team and we are currently investigating adios2 as a way to improve performances of our software.

In order to give you a bit of context, Melissa is a framework that coordinates ensemble runs to perform large scale and online sensitivity analysis or deep surrogate training. Our usual workflow consists of multiple data producers (any MPI code instrumented with our API) which dynamically connect to a parallel Python server and send this server data as they are produced (either in an MxN or a round robin fashion). Our point is to avoid files by directly communicating data through our API based on ZeroMQ sockets.

The main inconvenient in our current configuration is that ZeroMQ does not enable to take full advantage of the data transport layer on supercomputers which is why adios2 seemed like a good alternative. In substituting our current API with adios2, we would still like to handle client connection and manage message reception somehow asynchronously on the server side.

So far we have managed to build a preliminary working version with adios2. However, it is pretty sequential in the sense that as a data producer is detected as running by the server rank 0, its id is broadcasted to all ranks and the corresponding engine is opened on all ranks at the same time. Regarding message extraction, we are basically looping over all active engines trying to extract data if they are holding any.

From the adios2 documentation, it seems that engine (SST in our case) Open/Close are collective functions (not sure about BeginStep/EndStep). It also says that adios2 should be considered as not thread-safe. Indeed, as soon as we are trying to optimize our approach with threads to handle engine opening for instance, the server runs into deadlocks (same with message extraction).

Given our preliminary tests, we are hence wondering if adios2 is actually well suited for our purpose or not. If so, what do you think would be the optimal way to handle client connection and message reception?

Best regards,
Marc

[eisenhauer]

I'm going to suggest that @anagainaru chime in because she has been working with some of these sorts of issues, but in the meantime, I can address the SST questions. Yes, in general Open is a collective operation across all the engines. ADIOS' focus is on optimizing collective IO and in some sense the sweet spot is supporting applications like parallel timestepped simulations where all ranks dump data at the same time and in the aggregate it forms a global data structure. So while BeginStep might or might not be collective, EndStep pretty much always is. That said, @anagainaru has been using SST in more of a job-distribution sort of way by having multiple readers connect to a single producer and using SST's "OnDemand" value for the StepDistributionMode parameter. Hopefully she can fill you in on the details.

[mschouler]

Thank you very much for your reply!

[anagainaru]

Yes, @eisenhauer is right, I have been looking at this exact scenario.

Typically, ADIOS was developed for simulations that generate data in steps. When you connect a producer with a consumer, each time a producer generates a step (which means all the puts between a BeginStep and EndStep) this step is streamed to all the consumers (for the default behavior). You are the one defining what a step looks like, for example for cancer research where multiple MRIs are used to train a classification model, each step is represented by a batch of MRIs. I would recommend defining the steps based on the data the producers are waiting to receive (the more data you include in one step the more work for the consumers but you might have to wait longer until the producer sends anything). Only once an EndStep call is executed by the producer, the step is streamed to consumers. We currently have 3 ways of doing this: all to all (all consumers receive all steps), round robin (consumers receive steps in round robin order) and on demand (consumers ping the producers when they need data and they receive the next available step). For AI applications where consumers train surrogate models I use the on demand strategy. The streaming looks sequential if you have one consumer but it will be concurrent for your MxN example.

I added a simple example for using on demand using SST here: https://github.com/anagainaru/ADIOS2-addons/tree/main/DataStreaming/ADIOS_ondemand

The adios.xml file is used in this example to specify the number of consumers and the streaming strategy (in this case 2 consumers and on demand):

        <parameter key="RendezvousReaderCount" value="2"/>
        <parameter key="StepDistributionMode" value="OnDemand"/>

To run the code:

./sstWriter 100 2 & ./sstReader 100 2 & ./sstReader 100 2

Let me know if this makes sense. I can take a look at code snippets if you'd like me to figure out why your testcase looks sequential.

[mschouler]

Hi @anagainaru, I had a bit of time and fixed a stupid error of my doing in the pool thread submission command.

The --thread_open option now works for a sequential server and multiple clients or for a parallel server and a single client. However it fails for a parallel server and multiple clients. I feel like it could be due to the server ranks trying to open different engines at the same time.

[anagainaru]

Hey @mschouler I was able to reproduce your problem using your code. I will try to debug it this week and come back to you.

[anagainaru]

OK, I think I understand what you need from adios now. You basically have multiple clients (running on multiple ranks) generating data that needs to be processes by one server (running on multiple ranks). To do this, yes you need multiple reader engines on the server side for each of the clients. Once you do that, you can periodically poll each of those engines with a BeginStep with zero timeout to see if they’ve got data ready for you.

So we believe the issue with your code is (as you guessed) the Opens when using multiple threads. Getting the ops for different Open’s intertwined on a single communicator is an issue that might cause the Open to fail. So my first question is, do you need to create threads for Opens? Would sequential Opens kill your performance? As long as all ranks call Opens on engines in the same order you will be ok.

A possible solution (that we did not test for but that I could look into if you really need to parallelize Opens) is to use different communicators in different threads for the Open.

On the read side, things are much nicer. Right now you are locking for receiving data, but you could create separate threads to read from each engine and merge the results somehow through a thread-safe queue.

[mschouler]

Thank you very much @anagainaru. This gives us a lot to think.

We will certainly do some testing at scale in order to assess to what extent Opens management affects performances. Using distinct communicators sounds like a really interesting lead, we will probably test it as well.

Overall, it seems like adios2 is very well suited for our needs. We will let you know how it goes as we progress in its integration with Melissa.

[anagainaru]

Sounds good, keep us posted