Commit eb13f86
[HADP-52545][HADP-45851] Fix backward compatibility of alternative configs of Hadoop Filesystems to access (apache#245)
[HADP-45851] Fix backward compatibility of alternative configs of Hadoop Filesystems to access (apache#119)
### What changes were proposed in this pull request?
Fix precedence of configs of Hadoop Filesystems to access.
Before this PR
```
spark.kerberos.access.hadoopFileSystems -> spark.yarn.access.namenodes -> spark.yarn.access.hadoopFileSystems
```
After this PR
```
spark.kerberos.access.hadoopFileSystems -> spark.yarn.access.hadoopFileSystems -> spark.yarn.access.namenodes
```
### Why are the changes needed?
Before apache#23698, the precedence of configuring Hadoop Filesystems to access is
```
spark.yarn.access.hadoopFileSystems -> spark.yarn.access.namenodes
```
Afterwards, it's
```
spark.kerberos.access.hadoopFileSystems -> spark.yarn.access.namenodes -> spark.yarn.access.hadoopFileSystems
```
When both `spark.yarn.access.hadoopFileSystems` and `spark.yarn.access.namenodes` are configured with different values, the PR will break backward compatibility and cause application failure.
### Does this PR introduce _any_ user-facing change?
Yes. Fix backward compatibility.
### How was this patch tested?
Updated UT.
Co-authored-by: tianlzhang <tianlzhang@ebay.com>1 parent 414e8c3 commit eb13f86
File tree
2 files changed
+7
-4
lines changed- core/src
- main/scala/org/apache/spark
- test/scala/org/apache/spark
2 files changed
+7
-4
lines changed| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
750 | 750 | | |
751 | 751 | | |
752 | 752 | | |
753 | | - | |
754 | | - | |
| 753 | + | |
| 754 | + | |
755 | 755 | | |
756 | 756 | | |
757 | 757 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
276 | 276 | | |
277 | 277 | | |
278 | 278 | | |
279 | | - | |
280 | | - | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
281 | 284 | | |
282 | 285 | | |
283 | 286 | | |
| |||
0 commit comments