sparkxgb import for PySpark

[Ben-Epstein]

I've been following the threads for PySpark support and see here that adding the jars as pyFiles does the trick.

Unfortunately I'm unable to import from sparkxgb after following those steps on 1.0.0

from pyspark.sql import SparkSession
spark = SparkSession.builder.config('spark.jars','xgboost4j-spark_2.12-1.0.0.jar').config('spark.jars', 'xgboost4j_2.12-1.0.0.jar').getOrCreate()

spark.sparkContext.addPyFile("xgboost4j-spark_2.12-1.0.0.jar")
spark.sparkContext.addPyFile("./xgboost4j_2.12-1.0.0.jar")
from sparkxgb import XGBoostClassifier, XGBoostClassificationModel

ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-9-8a1823de6207> in <module>
----> 1 from sparkxgb import XGBoostClassifier, XGBoostClassificationModel

ModuleNotFoundError: No module named 'sparkxgb'

I'm running this on Jupyter, not sure if it matters or not.

Any suggestions?

Thanks!

[hcho3]

There is a new thread for PySpark: #5658.