Skip to content

Commit eb13f86

Browse files
yifhuamanuzhang
authored andcommitted
[HADP-52545][HADP-45851] Fix backward compatibility of alternative configs of Hadoop Filesystems to access (apache#245)
[HADP-45851] Fix backward compatibility of alternative configs of Hadoop Filesystems to access (apache#119) ### What changes were proposed in this pull request? Fix precedence of configs of Hadoop Filesystems to access. Before this PR ``` spark.kerberos.access.hadoopFileSystems -> spark.yarn.access.namenodes -> spark.yarn.access.hadoopFileSystems ``` After this PR ``` spark.kerberos.access.hadoopFileSystems -> spark.yarn.access.hadoopFileSystems -> spark.yarn.access.namenodes ``` ### Why are the changes needed? Before apache#23698, the precedence of configuring Hadoop Filesystems to access is ``` spark.yarn.access.hadoopFileSystems -> spark.yarn.access.namenodes ``` Afterwards, it's ``` spark.kerberos.access.hadoopFileSystems -> spark.yarn.access.namenodes -> spark.yarn.access.hadoopFileSystems ``` When both `spark.yarn.access.hadoopFileSystems` and `spark.yarn.access.namenodes` are configured with different values, the PR will break backward compatibility and cause application failure. ### Does this PR introduce _any_ user-facing change? Yes. Fix backward compatibility. ### How was this patch tested? Updated UT. Co-authored-by: tianlzhang <tianlzhang@ebay.com>
1 parent 414e8c3 commit eb13f86

File tree

2 files changed

+7
-4
lines changed

2 files changed

+7
-4
lines changed

core/src/main/scala/org/apache/spark/SparkConf.scala

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -750,8 +750,8 @@ private[spark] object SparkConf extends Logging {
750750
KERBEROS_RELOGIN_PERIOD.key -> Seq(
751751
AlternateConfig("spark.yarn.kerberos.relogin.period", "3.0")),
752752
KERBEROS_FILESYSTEMS_TO_ACCESS.key -> Seq(
753-
AlternateConfig("spark.yarn.access.namenodes", "2.2"),
754-
AlternateConfig("spark.yarn.access.hadoopFileSystems", "3.0")),
753+
AlternateConfig("spark.yarn.access.hadoopFileSystems", "3.0"),
754+
AlternateConfig("spark.yarn.access.namenodes", "2.2")),
755755
"spark.kafka.consumer.cache.capacity" -> Seq(
756756
AlternateConfig("spark.sql.kafkaConsumerCache.capacity", "3.0")),
757757
MAX_EXECUTOR_FAILURES.key -> Seq(

core/src/test/scala/org/apache/spark/SparkConfSuite.scala

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -276,8 +276,11 @@ class SparkConfSuite extends SparkFunSuite with LocalSparkContext with ResetSyst
276276
conf.set("spark.yarn.access.namenodes", "testNode")
277277
assert(conf.get(KERBEROS_FILESYSTEMS_TO_ACCESS) === Array("testNode"))
278278

279-
conf.set("spark.yarn.access.hadoopFileSystems", "testNode")
280-
assert(conf.get(KERBEROS_FILESYSTEMS_TO_ACCESS) === Array("testNode"))
279+
conf.set("spark.yarn.access.hadoopFileSystems", "testNode2")
280+
assert(conf.get(KERBEROS_FILESYSTEMS_TO_ACCESS) === Array("testNode2"))
281+
282+
conf.set("spark.yarn.access.namenodes", "testNode3")
283+
assert(conf.get(KERBEROS_FILESYSTEMS_TO_ACCESS) === Array("testNode2"))
281284
}
282285

283286
test("SPARK-13727") {

0 commit comments

Comments
 (0)