Apache Spark 3.0 compatibility

[jkbradley]

This is meant to be a discussion around preparing xgboost for the upcoming Apache Spark 3.0 release.

I participate in the Apache Spark project and would like to make it easier for xgboost to adapt to Spark's upcoming major release. For reference, the current Spark dev list discussions indicate these rough dates:

• Fall 2019: Spark 3.0 Preview release
• Early 2020: Spark 3.0 release
• See http://apache-spark-developers-list.1001551.n3.nabble.com/Thoughts-on-Spark-3-release-or-a-preview-release-td27891.html and other "3.0 Preview" mentions in the dev list for reference.

Some questions to get started:

• What are reasonable timelines around this?
• Any comments about coordination with ongoing efforts around an XGBoost 1.0.0 release + PySpark support?

Notes about Spark 3.0:

• Let me know if there are more questions around Spark 3.0 previews, testing, etc.
• Overall, I would not expect the newer DataFrame-based API for ML in Spark to change much, but private APIs and the older RDD-based ML API may change.
• The other big change will be a shift to Scala 2.12.

Thanks!

[jkbradley]

CC @CodingCat and @mengxr :)

[CodingCat]

Hi, @jkbradley

One of the expectation from 1.0.0 release of XGB is that we should coordinate with Spark 3.0 if possible. specifically, we are looking forward to GPU scheduling, per stage spark.task.cpus, etc.

for the 1.0.0.preview of XGBoost, Scala 2.12 has been the way to go, so I don't worry about it. But I would think more about whether to support 3.0 preview or stay with 2.4 (or both)

PySpark is something I would like to finish reviewing soon, but i am out of bandwidth (maybe you or @mengxr can help on that ;) )

[jkbradley]

Sorry for the slow response! We've been a bit underwater ourselves. But I talked with Xiangrui, and he recommended we ask @WeichenXu123 to help out with reviewing the PySpark integration. I've messaged Weichen offline about that, and we can follow up once we know when that can be prioritized.

That's great about Scala 2.12. For Spark 3.0 coordination, let me (or the dev list) know if you have specific questions!

[hcho3]

Closing this, since we upgraded to Spark 3.0.0 for the latest source.