Rewrite Dask interface.

[trivialfis]

This PR rewrites the dask interface based on dask-xgboost, with added support for evaluation metrics and slightly different interface.

• Implement functional train and predict.
• Implement Scikit-Learn wrapper for regression and classification. No ranking yet.
• Implement DaskDMatrix as a proxy for underlying distributed DMatrix.

Closes #4814.

[trivialfis]

@RAMitchell @hcho3 @CodingCat I can further abstract DMatrix to accept dask.dataframe, but with considerably more changes. WDYT?

[trivialfis]

I would like to see how many works are in common with #4656 . @thesuperzapper Could you also join the review after this PR no longer being WIP?

[RAMitchell]

@RAMitchell @hcho3 @CodingCat I can further abstract DMatrix to accept dask.dataframe, but with considerably more changes. WDYT?

No rush on this I think. Lets stabilise the dask interface first.

Whats your plan for assigning gpu_id in this interface?

[trivialfis]

@RAMitchell I have another branch that uses a utility I implemented. I need to somehow merge it here.

[mtjrider]

Separate the client from the current Python process by enabling a client override. Remove experimental new Python syntax. Use Dask's method for frame concatenation instead of dispatching the type yourself.

[mrocklin]

@RAMitchell asked me for my thoughts on the use of the distributed_dispatch decorator here.

Strictly from a Dask user's perspective I think that it's nice to have first class support for Dask within XGBoost. However, I can see how from an XGBoost maintainer's point of view, this might be concerning.

In general I think that API extensibility is good. For example it's nice that I can use np.sum on a numpy array, cupy array, dask array, and so on. However usually when making an API extensible it's common to establish a contract that anyone can implement, without having to directly modify code within the core project. For example maybe xgb.train looks for a .__xgb_train__ method on the passed-in arguments and calls that. That way, if some other project comes along and implements things correctly then they can be first class as well, even if they don't have strong ties to the XGBoost maintainers. Protocols like this tend to level the playing field.

That's all maybe a bit philosophical. I'm happy to get more into details here if folks want.

[mrocklin]

I'll also say, it would be nice to find protocols that would allow us to pass in Dask dataframes or Dask arrays directly, without creating the DaskDMatrix object. I'm not sure how easy that would be. Maybe using something like functools.singledispatch or multipledispatch?

[mrocklin]

The actual implementation looks like what we've all done historically. It's not super clean but there doesn't seem to be a better solution at the moment. I think it would be worth thinking about how to improve these sorts of workflows within Dask.

My guess is that that's out of scope for folks here (some core Dask devs probably need to spend some serious time thinking about that). Mostly I mention this so that you don't grow too attached to this implementation, and remain open to some day replacing it with something else.

[trivialfis]

The actual implementation looks like what we've all done historically

Yeah. I learned a lot from it. Especially the part of obtaining worker local data.

My guess is that that's out of scope for folks here

I'm interested in getting to know more about dask. So please keep me in the loop.

Mostly I mention this so that you don't grow too attached to this implementation

That won't be a problem. If there's an improvement we can change it anytime. I don't expect one commit can get everything right. ;-)

As for sklearn interface. I'm working on it.

[codecov-io]

Codecov Report

Merging #4819 into master will decrease coverage by 5.91%.
The diff coverage is 23.97%.

@@            Coverage Diff             @@
##           master    #4819      +/-   ##
==========================================
- Coverage   77.63%   71.72%   -5.92%     
==========================================
  Files          11       11              
  Lines        2039     2281     +242     
==========================================
+ Hits         1583     1636      +53     
- Misses        456      645     +189

Impacted Files	Coverage Δ
python-package/xgboost/sklearn.py	87.69% <ø> (-0.04%)	⬇️
python-package/xgboost/__init__.py	89.47% <100%> (ø)	⬆️
python-package/xgboost/dask.py	19.13% <19.13%> (-4.31%)	⬇️
python-package/xgboost/core.py	75.57% <25%> (-0.77%)	⬇️
python-package/xgboost/compat.py	54.4% <70.37%> (+2.97%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fc8c9b0...8be19b0. Read the comment docs.

[trivialfis]

@mrocklin @RAMitchell @mt-jones Please take a look again.

[RAMitchell]

LGTM. You will have to make sure all of these functions appear in the Python API documentation.

[trivialfis]

@mrocklin @TomAugspurger The original implementation of dask-xgboost is acknowledged in both doc and code header. Thank you so much!

[trivialfis]

@CodingCat @hcho3 Would you like to take a look?

[mtjrider]

Looks great to me.

[trivialfis]

Merging as we will have more time to test it. @hcho3 @CodingCat Try it out, I have grown to like dask now. ;-)

[trivialfis]

@mt-jones @mrocklin @RAMitchell Thanks!