Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: fix Guava version issue in Azure Synapse and Databricks #1103

Merged
merged 12 commits into from
Jul 8, 2021
Merged

fix: fix Guava version issue in Azure Synapse and Databricks #1103

Changes from all commits
Commits
Show all changes
12 commits
Select commit Hold shift + click to select a range
b1745f8
fix: update Guava version
xuwq1993 Jun 25, 2021
3c01899
Merge branch 'master' into wenqx/updateGuava
xuwq1993 Jul 2, 2021
9df7460
Merge branch 'master' into wenqx/updateGuava
xuwq1993 Jul 6, 2021
e09729e
update: remove guava dependency
xuwq1993 Jul 6, 2021
ceafcdf
update: use newDirectExecutorService
xuwq1993 Jul 6, 2021
259ff97
update: update MoreExecutor
xuwq1993 Jul 7, 2021
ffcd7e6
Merge branch 'master' into wenqx/updateGuava
xuwq1993 Jul 7, 2021
fbeef4e
update: use reflection
xuwq1993 Jul 8, 2021
da5bdaf
update: use reflection
xuwq1993 Jul 8, 2021
cf2aa39
update: use reflection
xuwq1993 Jul 8, 2021
8ebebe7
update: use reflection
xuwq1993 Jul 8, 2021
d6781f0
update: use reflection
xuwq1993 Jul 8, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 17 additions & 1 deletion core/src/main/scala/com/microsoft/ml/spark/automl/TuneHyperparameters.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,11 @@ import org.apache.spark.mllib.util.MLUtils
import org.apache.spark.sql._
import org.apache.spark.sql.types.StructType

import java.lang.reflect.Method
import scala.collection.mutable.ListBuffer
import scala.concurrent.{Awaitable, ExecutionContext, Future}
import scala.concurrent.duration.Duration
import scala.reflect.internal.util.ScalaClassLoader
import scala.util.control.NonFatal

/** Tunes model hyperparameters
Expand Down Expand Up @@ -95,7 +97,21 @@ class TuneHyperparameters(override val uid: String) extends Estimator[TuneHyperp
private def getExecutionContext: ExecutionContext = {
getParallelism match {
case 1 =>
ExecutionContext.fromExecutorService(MoreExecutors.sameThreadExecutor())
val classPath = "com.google.common.util.concurrent.MoreExecutors"
val funcNameOld = "sameThreadExecutor"
val funcNameNew = "newDirectExecutorService"
val c = ScalaClassLoader(getClass.getClassLoader).tryToLoadClass(classPath)
val method: Method = {
try {
c.get.getMethod(funcNameNew)
}
catch {
case _: NoSuchMethodError => c.get.getMethod(funcNameOld)
case _: NoSuchMethodException => c.get.getMethod(funcNameOld)
}
}
val executorService = method.invoke(c.get).asInstanceOf[ExecutorService]
ExecutionContext.fromExecutorService(executorService)
case _ =>
val keepAliveSeconds = 60L
val prefix = s"${this.getClass.getSimpleName}-thread-pool"
Expand Down