Development of custom scheduler plugin

[atztogo]

I have met two queueing systems on different supercomputing systems, one at Kyoto university and the other at NIMS/Japan. They use usual queueing systems (LSF/PBSPro) that are already supported by AiiDA-core, but unfortunately they put customized wrappers on the usual qsystem commands, so I can't use them directly by AiiDA. I write this issue, because this may be a typical case.

A few years ago, I solved this problem just modifying the existing plugin in AiiDA-core source code since the modification was expected not that much. But this should be avoided unless for personal use. So developing a new scheduler plugin should be a proper direction, for which I would like to know how to do it. Is there any recommended way to write it? I also appreciate If someone could tell me any existing documentation for it.

Of course, the other way to solve this problem is to discuss with engineers working for the supercomputer centres to let us allow to access bare qsystem commands. This I will if recommended.

Thanks,

[ltalirz]

Hi @atztogo , we just came across this issue in #3404 as well.
I think it would be great to get started with writing a simple example of a scheduler plugin.

I'm happy to provide a very first template to get you started (still leaving you with all the job to do but at least showing how to do it in principle) - just let me know how this plugin should be named (aiida-nims-schedulers? or something more specific perhaps?)

[ltalirz]

@sphuber
Since I don't expect these types of plugins to be very large, perhaps it even makes sense to have something like an aiida-scheduler-bundle plugin on aiidateam where interested people can add their own schedulers relatively easily (unless they prefer to develop them privately).

That would make it easier to migrate things in case there are changes to the scheduler API in the future (not sure whether there's changes planned already)

Anyhow, not entirely sure...

[sphuber]

@espenfl has also been working on schedulers, particularly on writing tests against actual schedulers, as suggested in #3805, such that API changes in new versions should hopefully be detected. This could potentially indeed be developed in a separate plugin where we can provide easy tools for people to extend existing ones and customize them.

[espenfl]

Yes. And when we are on the issue; we should reach out to https://github.com/Parsl/libsubmit and see if we can coordinate the efforts.

[atztogo]

Hi @atztogo , we just came across this issue in #3404 as well.
I think it would be great to get started with writing a simple example of a scheduler plugin.

I'm happy to provide a very first template to get you started (still leaving you with all the job to do but at least showing how to do it in principle) - just let me know how this plugin should be named (aiida-nims-schedulers? or something more specific perhaps?)

Thank you very much @ltalirz . aiida-nims-schedulers is nice.

[atztogo]

I close this issue. Thanks for your answers.

[ltalirz]

Just came across this custom scheduler plugin by @chrisjsewell
https://github.com/chrisjsewell/aiida-cx1scheduler

Also, @pzarabadip is working on a plugin for their PBS scheduler.
The plan is to eventually include integration tests using @espenfl 's docker containers with schedulers, if possible.

[atztogo]

Thanks. Very helpful. I will try to write it

[ezpzbz]

Hi, as @ltalirz mentioned, I also needed to apply couple of modifications to the AiiDA-shipped PBS scheduler as well as ssh transport plugin to be able running calculations on one of our clusters.
You may find the current version here (https://github.com/pzarabadip/aiida-scheduler-bundle).

We were discussing on the possibility of having all these custom schedulers gathered in one registered plugin package but it seems it brings more complications through introducing several entry points which not all of them are useful for every user and also it can result in having conflicts on entry point names.

Herein, we can discuss if we should:

• provide a tutorial-like page in AiiDA documentation that explains how to proceed in similar cases if one would require customization in scheduler/transport plugins. There we can point out users to the available customized plugins like the one by @chrisjsewell, @espenfl and mine as starting points.
• recommend a unique naming and folder structure for this type of scheduler plugin bundle
• add specific instruction to guide users making the draft of similar custom plugins using AiiDA plugin cutter

I am mentioning @giovannipizzi to ask about his thoughts on this matter too.

It's worth noting that during AiiDA Hackathon, Giovanni mentioned about MyQueue(https://myqueue.readthedocs.io/en/latest/) which seems its implementation as a meta scheduler in AiiDA can potentially solve couple of our issues with schedulers. I still have not found time to test it for my cases and I hope I will do it soon. However, if you have experience/thoughts on MyQueue or similar packages, your thoughts and opinions are highly appreciated.

[giovannipizzi]

I agree on the three bullet points - docs and plugin cookiecutter would be very appreciated, and explicitly or implicitly suggest the naming and folder structure

[ltalirz]

thanks @pzarabadip !
For the naming scheme, I have no strong feelings - e.g. aiida-metavo-schedulers would be fine with me, but if you might want to bundle more than just schedulers in there, aiida-metavo-cluster (or whatever is appropriate) would also work fine.

Inside the plugin, I think it makes sense to put scheduler-related stuff in to a file schedulers.py or a folder schedulers (in analogy to data/calculations in aiida-diff ).

As for creating a cookie cutter (or integrating it into the existing one):
If you think one can make this general enough to be useful, feel free to give this a try - otherwise, I think just being able to point to a well-done example would already go a very long way.

As for where to link to this from the docs, I suggest we make the "Entry Points" section a bit more useful by starting to add links to example plugins (or adding more instructions) for adding plugins for the individual entry point groups (I'll work on this over the next days)

[atztogo]

Finally I could make it. @chrisjsewell 's one was closer. So I started from it. Thanks.

[ltalirz]

@atztogo @pzarabadip
Just mentioning that you may want to consider registering your plugins on the AiiDA plugin registry as others may have similar issues to yours.

I understand, of course, that these are rather specific plugins that might not be very useful in many other places, but it could be helpful to have at least 1-2 plugin packages with scheduler plugins registered so people can imitate what you guys have been doing.