Skip to content

[SPARK-19720][CORE][BACKPORT-2.1] Redact sensitive information from SparkSubmit console #18765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
dmvieira wants to merge 2 commits into from
Closed

[SPARK-19720][CORE][BACKPORT-2.1] Redact sensitive information from SparkSubmit console #18765

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9e75782
[SPARK-19720][CORE] Redact sensitive information from SparkSubmit con…
markgrover Mar 2, 2017
f5362fc
Fixing 100 line length code
Jul 29, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion core/src/main/scala/org/apache/spark/deploy/SparkSubmit.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -670,7 +670,8 @@ object SparkSubmit {
if (verbose) {
printStream.println(s"Main class:\n$childMainClass")
printStream.println(s"Arguments:\n${childArgs.mkString("\n")}")
printStream.println(s"System properties:\n${sysProps.mkString("\n")}")
// sysProps may contain sensitive information, so redact before printing
printStream.println(s"System properties:\n${Utils.redact(sysProps).mkString("\n")}")
printStream.println(s"Classpath elements:\n${childClasspath.mkString("\n")}")
printStream.println("\n")
}
Expand Down
12 changes: 9 additions & 3 deletions core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,15 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
// scalastyle:off println
if (verbose) SparkSubmit.printStream.println(s"Using properties file: $propertiesFile")
Option(propertiesFile).foreach { filename =>
Utils.getPropertiesFromFile(filename).foreach { case (k, v) =>
val properties = Utils.getPropertiesFromFile(filename)
properties.foreach { case (k, v) =>
defaultProperties(k) = v
if (verbose) SparkSubmit.printStream.println(s"Adding default property: $k=$v")
}
// Property files may contain sensitive information, so redact before printing
if (verbose) {
Utils.redact(properties).foreach { case (k, v) =>
SparkSubmit.printStream.println(s"Adding default property: $k=$v")
}
}
}
// scalastyle:on println
Expand Down Expand Up @@ -318,7 +324,7 @@ private[deploy] class SparkSubmitArguments(args: Seq[String], env: Map[String, S
|
|Spark properties used, including those specified through
| --conf and those from the properties file $propertiesFile:
|${sparkProperties.mkString(" ", "\n ", "\n")}
|${Utils.redact(sparkProperties.toMap).mkString(" ", "\n ", "\n")}
""".stripMargin
}

Expand Down
18 changes: 18 additions & 0 deletions core/src/main/scala/org/apache/spark/util/Utils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -38,6 +38,7 @@ import scala.io.Source
import scala.reflect.ClassTag
import scala.util.Try
import scala.util.control.{ControlThrowable, NonFatal}
import scala.util.matching.Regex

import _root_.io.netty.channel.unix.Errors.NativeIoException
import com.google.common.cache.{CacheBuilder, CacheLoader, LoadingCache}
Expand Down Expand Up @@ -2571,6 +2572,23 @@ private[spark] object Utils extends Logging {
sparkJars.map(_.split(",")).map(_.filter(_.nonEmpty)).toSeq.flatten
}
}

private[util] val REDACTION_REPLACEMENT_TEXT = "*********(redacted)"
private[util] val SECRET_REDACTION_PATTERN = "(?i)secret|password".r
Copy link
Member

@gatorsmile gatorsmile Aug 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be a configurable SQLConf.

Copy link
Author

@dmvieira dmvieira Aug 1, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But I'm following UI pattern: https://github.com/apache/spark/blob/branch-2.1/core/src/main/scala/org/apache/spark/ui/env/EnvironmentPage.scala

Redact with configurable SQLConf is a port of this feature: https://issues.apache.org/jira/browse/SPARK-18535 . Isn't it?

Copy link
Member

@markgrover markgrover Aug 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what's really happening here is that we are backporting some changes introduced in SPARK-18535 while backporting this JIRA (SPARK-19720). SPARK-18535 is a dependency of this, so if we want to backport this, we should really be backporting SPARK-18535 as well.

Copy link
Author

@dmvieira dmvieira Aug 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @markgrover ! My intention here was only fix this security breach making spark-submit redact patten similar to UI redact pattern. I can change it, but it will be a new feature backport and not a bugfix backport

Copy link
Author

@dmvieira dmvieira Aug 1, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did another pull request with all feature: #18802

Copy link
Author

@dmvieira dmvieira Aug 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did PR but I don't know why Jenkins fail with access error... It sounds like permission issue.

Copy link
Author

@dmvieira dmvieira Aug 3, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did it work there... I tested here and UI and spark-submit already working. I think you can close this pull request and focus on #18802


def redact(kvs: Map[String, String]): Seq[(String, String)] = {
val redactionPattern = SECRET_REDACTION_PATTERN
redact(redactionPattern, kvs.toArray)
}

private def redact(redactPattern: Regex, kvs: Array[(String, String)]): Seq[(String, String)] = {
kvs.map { kv =>
redactPattern.findFirstIn(kv._1)
.map { _ => (kv._1, REDACTION_REPLACEMENT_TEXT) }
.getOrElse(kv)
}
}

}

private[util] object CallerContext extends Logging {
Expand Down