Skip to content

[SPARK-30772][ML][SQL] avoid tuple assignment because it will circumvent the transient tag #27523

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
zhengruifeng wants to merge 2 commits into from
Closed

[SPARK-30772][ML][SQL] avoid tuple assignment because it will circumvent the transient tag #27523

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1de531c
init
zhengruifeng Feb 10, 2020
968ffb3
revert_sql
zhengruifeng Feb 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion mllib-local/src/main/scala/org/apache/spark/ml/stat/distribution/MultivariateGaussian.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -55,7 +55,9 @@ class MultivariateGaussian @Since("2.0.0") (
* rootSigmaInv = D^(-1/2)^ * U.t, where sigma = U * D * U.t
* u = log((2*pi)^(-k/2)^ * det(sigma)^(-1/2)^)
*/
@transient private lazy val (rootSigmaInv: BDM[Double], u: Double) = calculateCovarianceConstants
@transient private lazy val tuple = calculateCovarianceConstants
@transient private lazy val rootSigmaInv = tuple._1
@transient private lazy val u = tuple._2

/**
* Returns density of this multivariate Gaussian at given point, x
Expand Down
16 changes: 12 additions & 4 deletions mllib/src/main/scala/org/apache/spark/ml/classification/NaiveBayes.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -405,18 +405,26 @@ class NaiveBayesModel private[ml] (
* This precomputes log(1.0 - exp(theta)) and its sum which are used for the linear algebra
* application of this condition (in predict function).
*/
@transient private lazy val (thetaMinusNegTheta, piMinusThetaSum) = $(modelType) match {
@transient private lazy val thetaMinusNegTheta = $(modelType) match {
case Bernoulli =>
theta.map(value => value - math.log1p(-math.exp(value)))
case _ =>
// This should never happen.
throw new IllegalArgumentException(s"Invalid modelType: ${$(modelType)}. " +
"Variables thetaMinusNegTheta should only be precomputed in Bernoulli NB.")
}

@transient private lazy val piMinusThetaSum = $(modelType) match {
case Bernoulli =>
val thetaMinusNegTheta = theta.map(value => value - math.log1p(-math.exp(value)))
val negTheta = theta.map(value => math.log1p(-math.exp(value)))
val ones = new DenseVector(Array.fill(theta.numCols)(1.0))
val piMinusThetaSum = pi.toDense.copy
BLAS.gemv(1.0, negTheta, ones, 1.0, piMinusThetaSum)
(thetaMinusNegTheta, piMinusThetaSum)
piMinusThetaSum
case _ =>
// This should never happen.
throw new IllegalArgumentException(s"Invalid modelType: ${$(modelType)}. " +
"Variables thetaMinusNegTheta and negThetaSum should only be precomputed in Bernoulli NB.")
"Variables piMinusThetaSum should only be precomputed in Bernoulli NB.")
}

/**
Expand Down
4 changes: 3 additions & 1 deletion mllib/src/main/scala/org/apache/spark/mllib/stat/distribution/MultivariateGaussian.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,7 +60,9 @@ class MultivariateGaussian @Since("1.3.0") (
* rootSigmaInv = D^(-1/2)^ * U.t, where sigma = U * D * U.t
* u = log((2*pi)^(-k/2)^ * det(sigma)^(-1/2)^)
*/
@transient private lazy val (rootSigmaInv: DBM[Double], u: Double) = calculateCovarianceConstants
@transient private lazy val tuple = calculateCovarianceConstants
@transient private lazy val rootSigmaInv = tuple._1
@transient private lazy val u = tuple._2

/**
* Returns density of this multivariate Gaussian at given point, x
Expand Down