Skip to content

[SPARK-24349][SQL] Ignore setting token if using JDBC #21396

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
LantaoJin wants to merge 1 commit into from
Closed

[SPARK-24349][SQL] Ignore setting token if using JDBC #21396

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion core/src/main/scala/org/apache/spark/deploy/security/HiveDelegationTokenProvider.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,10 @@ private[spark] class HiveDelegationTokenProvider
val principal = conf.getTrimmed(principalKey, "")
require(principal.nonEmpty, s"Hive principal $principalKey undefined")
val metastoreUri = conf.getTrimmed("hive.metastore.uris", "")
require(metastoreUri.nonEmpty, "Hive metastore uri undefined")
if (metastoreUri.isEmpty) {
Copy link
Contributor

@vanzin vanzin May 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How is the code getting past the check in delegationTokensRequired? It's basically checking for the same thing.

Copy link
Contributor Author

@LantaoJin LantaoJin May 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

require() will throws IllegalArgumentException and exits the JVM here. Letting delegationTokensRequired return when metastoreUri is undefined (using JDBC to connect DB directly) only finishes this method (not set token in Credentials).

Copy link
Contributor

@jerryshao jerryshao May 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same question as @vanzin , delegationTokensRequired should already check whether hive.metastore.uris is empty or not, so it will not obtain the DT if this hive.metastore.uris is not configured.

  override def delegationTokensRequired(
      sparkConf: SparkConf,
      hadoopConf: Configuration): Boolean = {
    // Delegation tokens are needed only when:
    // - trying to connect to a secure metastore
    // - either deploying in cluster mode without a keytab, or impersonating another user
    //
    // Other modes (such as client with or without keytab, or cluster mode with keytab) do not need
    // a delegation token, since there's a valid kerberos TGT for the right user available to the
    // driver, which is the only process that connects to the HMS.
    val deployMode = sparkConf.get("spark.submit.deployMode", "client")
    UserGroupInformation.isSecurityEnabled &&
      hiveConf(hadoopConf).getTrimmed("hive.metastore.uris", "").nonEmpty &&
      (SparkHadoopUtil.get.isProxyUser(UserGroupInformation.getCurrentUser()) ||
        (deployMode == "cluster" && !sparkConf.contains(KEYTAB)))
  }

Copy link
Contributor Author

@LantaoJin LantaoJin May 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I know. If the metastore is undefined, it no needs to obtain the DT. Am I right?

Copy link
Contributor

@jerryshao jerryshao May 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before getting the DT, we will check if delegationTokensRequired returns true or false, if it is false, then we will not get DT. Here since "hive.metastore.uris" is not configured, then it should return false.

Copy link
Contributor Author

@LantaoJin LantaoJin May 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, my fault, delegationTokensRequired has been checked in SparkSQLCLIDriver.scala.

logInfo("Hive metastore uri undefined, ignore to obtain Credentials")
return None
}

val currentUser = UserGroupInformation.getCurrentUser()
logDebug(s"Getting Hive delegation token for ${currentUser.getUserName()} against " +
Expand Down