[dask] Support asynchronous workflows

[jmoralez]

Summary

Implementing coroutines for training and computing predictions with an asynchronous dask client.

Motivation

By having an asynchronous interface, LightGBM's distributed training with dask could be used in concurrent applications. One possible use case for this would be having a trained LightGBM model in a web application and get predictions from it in a non-blocking fashion. Another possibility could be having an API that takes in some configuration as a POST request, starts a remote cluster and trains a LightGBM model on it, having this interface would allow several models to be trained concurrently.

Description

This can be achieved by implementing coroutines for the train and predict functions and then using client.sync on them to get the synchronous variants.

References

• xgboost-train coroutine
• xgboost-train function
• xgboost-predict coroutine
• xgboost-predict function
• "Working with asyncio" (XGBoost Dask docs)

[jameslamb]

Thanks for opening this! Could you please provide a more specific XGBoost link, to the parts of their code that specifically allow async access?

Could you also add details on why someone would want to use LightGBM with Dask asynchronously?

[jmoralez]

Hi James. I've updated my comment with some cases I can think of. I realize this probably needs to come after some other building blocks but I'd like to work towards this, what do you think would be needed first?

[jameslamb]

Thanks very much for that. I added a link to XGBoost's docs on this topic as well: https://xgboost.readthedocs.io/en/latest/tutorials/dask.html#working-with-asyncio.

I realize this probably needs to come after some other building blocks but I'd like to work towards this

If you're interested in contributing further, we'd be very grateful! But to be honest, I think that a lot of fundamental pieces need to be added before we consider supporting asynchronous training / prediction in the Dask interface.

The Dask interface is still very new and is missing big features like init_score (#3807), the other boosting types (#3896), and the other distributed training modes (#3834), to name a few.

Adding support for init_score (#3807) or predict(raw_score=True) (#3793) are good next steps.

[jameslamb]

I've added this to #2302 , where we keep all feature requests.