[WIP] Scala 2.13 compatible build

[db-scnakandala]

No description provided.

[jsleight]

Think you need to alter the travis yaml to make scala 2.13 available https://github.com/combust/mleap/blob/master/.travis.yml#L25

[scheruku-in]

Hi
Is there any ETA on when this PR gets merged and 2.13 mleap libraries will be available?

Thanks

[scheruku-in]

Hi
Is there any ETA on when this PR gets merged and 2.13 mleap libraries will be available?

Thanks

[jsleight]

@db-scnakandala any rough timeline on your end? I'm happy to merge and release artifacts soon after the PR is complete

[db-scnakandala]

@db-scnakandala any rough timeline on your end? I'm happy to merge and release artifacts soon after the PR is complete

Didn't get a chance to go through the build failures. Will check again this week.

[prashil1996]

@db-scnakandala Hey, any updates on this? Is there something I can do to help you?

[prashil1996]

@jsleight Hey, you mentioned that updating the travis.yml to 2.13.11 should make it compatible to scala 2.13? Because I did the same the local compile was successful. I see this PR is taking quite some time and I need to make changes so we can make it compatible.

Please update, whenever you do get a chance as it would be really helpful.

[db-scnakandala]

@prashil1996 I am still trying to diagnose and fix several CI failures. Please let me know if you have any insights on the test failures.

[prashil1996]

@db-scnakandala Let me look into the build failures.

I see that the Mleap Executor & Mleap benchmark have passed. Let me look into the logs and come back to you.

[prashil1996]

@db-scnakandala There is an error while importing has this has been moved in Scala 2.13 to separate module [scala/scala-parallel-collection]
Need to update the library dependency such as for scala 2.13 onwards.
libraryDependencies += "org.scala-lang.modules" %% "scala-parallel-collections" % "1.0.0"

This is where one build is failing.
[error] /home/travis/build/combust/mleap/mleap-spark-extension/src/main/scala/org/apache/spark/ml/mleap/classification/OneVsRest.scala:26:34: object CollectionConverters is not a member of package scala.collection.parallel [error] import scala.collection.parallel.CollectionConverters._

[prashil1996]

@db-scnakandala Also, I was also facing this issue locally where the following were not found.
https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/snapshot/ml/dmlc/xgboost4j_2.12/2.0.0-SNAPSHOT/xgboost4j_${scala.binary.version}-2.0.0-20230817.090749-594.jar: not found: https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/snapshot/ml/dmlc/xgboost4j_2.12/2.0.0-SNAPSHOT/xgboost4j_${scala.binary.version}-2.0.0-20230817.090749-594.jar

Is there a way to resolve this dependency? @jsleight

[prashil1996]

And the other major error is due to NoSuchMethodError error.
Exception in thread "main" java.lang.NoSuchMethodError: scala.util.matching.Regex.<init>(Ljava/lang/String;Lscala/collection/Seq;)V
Can we make sure that the spark version we are using for scala 2.13.11, if its compatible or not? Becase the jdk version is at openjdk11 and the scala version being passed is 2.13.11, but as per the documentation openjdk11 supports only 2.13.0, so maybe there could be a version mismatch there.
https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html
@db-scnakandala

[jsleight]

Can we make sure that the spark version we are using for scala 2.13.11, if its compatible or not? Becase the jdk version is at openjdk11 and the scala version being passed is 2.13.11, but as per the documentation openjdk11 supports only 2.13.0, so maybe there could be a version mismatch there.
https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html

That link is giving the minimum versions. So any scala 2.13.x version should work with jdk11. It matters more for older scala versions -- e.g., you need scala 2.12.4 or newer for a scala 2.12 (i.e., scala 2.12.3 wouldn't work with jdk11).

Our spark version is compiled with scala 2.13.8 -- I'd expect any scala 2.13 version to work though.

[jsleight]

@db-scnakandala Also, I was also facing this issue locally where the following were not found.
https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/snapshot/ml/dmlc/xgboost4j_2.12/2.0.0-SNAPSHOT/xgboost4j_${scala.binary.version}-2.0.0-20230817.090749-594.jar: not found: https://s3-us-west-2.amazonaws.com/xgboost-maven-repo/snapshot/ml/dmlc/xgboost4j_2.12/2.0.0-SNAPSHOT/xgboost4j_${scala.binary.version}-2.0.0-20230817.090749-594.jar

Is there a way to resolve this dependency? @jsleight

We have xgboost 1.7.3, so I'm not sure why you would be seeing errors for xgboost 2.0, let alone for a snapshot version. Would likely be something wrong with your branch.

[jsleight]

we'll need to have both a scala 2.12 and a scala 2.13 distribution installed in travis (so the cross builds can run). Refer to travis docs for how to do this.

[prashil1996]

@db-scnakandala Any updates?