Skip to content

[SPARK-48678][CORE] Performance optimizations for SparkConf.get(ConfigEntry) #47049

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JoshRosen wants to merge 5 commits into from
Closed

[SPARK-48678][CORE] Performance optimizations for SparkConf.get(ConfigEntry) #47049

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
c5bfde3
Skip regex in ConfigReader.substitute for inputs that cannot possibly…
JoshRosen Jun 20, 2024
5a03cff
ConfigEntry collections operation optimizations
JoshRosen Jun 20, 2024
b4af01d
Add code comments.
JoshRosen Jun 20, 2024
9ee3ebb
Update core/src/main/scala/org/apache/spark/internal/config/ConfigEnt…
JoshRosen Jun 21, 2024
a18177f
Address review feedback: remove shadowing and some branches
JoshRosen Jun 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 8 additions & 8 deletions core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,14 +89,14 @@ private[spark] abstract class ConfigEntry[T] (
def defaultValueString: String

protected def readString(reader: ConfigReader): Option[String] = {
val values = Seq(
prependedKey.flatMap(reader.get(_)),
alternatives.foldLeft(reader.get(key))((res, nextKey) => res.orElse(reader.get(nextKey)))
).flatten
if (values.nonEmpty) {
Some(values.mkString(prependSeparator))
} else {
None
// SPARK-48678: performance optimization: this code could be expressed more succinctly
// using flatten and mkString, but doing so adds lots of Scala collections perf. overhead.
val maybePrependedValue: Option[String] = prependedKey.flatMap(reader.get)
val maybeValue: Option[String] = alternatives
.foldLeft(reader.get(key))((res, nextKey) => res.orElse(reader.get(nextKey)))
(maybePrependedValue, maybeValue) match {
case (Some(prependedValue), Some(value)) => Some(s"$prependedValue$prependSeparator$value")
case _ => maybeValue.orElse(maybePrependedValue)
}
}

Expand Down
4 changes: 3 additions & 1 deletion core/src/main/scala/org/apache/spark/internal/config/ConfigReader.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,9 @@ private[spark] class ConfigReader(conf: ConfigProvider) {
def substitute(input: String): String = substitute(input, Set())

private def substitute(input: String, usedRefs: Set[String]): String = {
if (input != null) {
// SPARK-48678: performance optimization: skip the costly regex processing
// if the string cannot possibly contain a variable reference:
if (input != null && input.contains("${")) {
ConfigReader.REF_RE.replaceAllIn(input, { m =>
val prefix = m.group(1)
val name = m.group(2)
Expand Down