Replace RabbitMQ (concrete design notes)

[unkcpz]

• Used AEP template from AEP 0
• Status is draft
• Added type & status labels to PR
• Added AEP to README.md
• Provided github handles for authors

The #30 give the explanation and motivation on removing RabbitMQ. I extent on it as a concrete implementation AEP.
The prototype is put in https://github.com/unkcpz/tatzelwurm, I put it in private at the moment and shared only to aiida-core team members. Let me know if you want to take a look.

Go built aep page if need to read as whole post.

[unkcpz]

I am aware of chrisjsewell/aiida-process-coordinator#4, the comments by @giovannipizzi and @sphuber are addressed sporadically in the design notes.
I pick some what I think important to explicitly reply here.

The only remaining question that sprung to mind was the usage of broadcasting where processes broadcasted state changes. There are some basic usages of this functionality, for example verdi process watch which allows to watch a process' state changes, but these are not that important. What is important though, is how workchains used this functionality to quickly start running again once the child processes they were waiting for were, reach the terminated state:

This is mentioned by @sphuber. I think if the priority of tasks and different type of runners are implemented the problem can be addressed gracefully.
Once the child processes waited by the workchain are finished, the workchain will have relatively higher priority than workchains that still have the child processes are running therefore will be picked up and run first.

Followings are mentioned by @giovannipizzi:

I suggest to use PostgreSQL for stress testing

In the prototype, before I mock the task pool for se/de the task, I created fake async/sync sleep task in the worker and the throughput I tested can be > 100,000 tasks without any noticeable delay (the bottleneck is then my terminal can not consume the print of so many logs).
When the load is large, I saw the CPU usage scaled up to more than 1 which attribute to the Rust tokio and it is not possible if using python with GIL.
For a more realistic benchmark, I use surrealDB which has memory storage and disk DB and support SQL. It said has potentially an extremely good performance.
But it should be very easy to test scenarios you are interested by yourself by just download the binaries.

is there still a concept of "slots" per worker, like now? (i.e. how many AiiDA processes a worker can take care of)?

In my plan, after the distinguish of runners type and tasks type I think an single async runner is enough to handle > 10,000 tasks per second, which is way more large then the current use case of AiiDA (not for the future if AiiDA is plan to compete with workflow engines for bio-information and ML use cases). Then the slots concept is applied to block function runner only.