[SPARK-48678][CORE] Performance optimizations for SparkConf.get(ConfigEntry)

[JoshRosen]

What changes were proposed in this pull request?

This PR proposes two micro-optimizations for SparkConf.get(ConfigEntry):

1. Avoid costly Regex.replaceAllIn for variable substitution: if the config value does not contain the substring ${ then it cannot possibly contain any variables, so we can completely skip the regex evaluation in such cases.
2. Avoid Scala collections operations, including List.flatten and AbstractIterable.mkString, for the common case where a configuration does not define a prepended configuration key.

Why are the changes needed?

Improve performance.

This is primarily motivated by unit testing and benchmarking scenarios but it will also slightly benefit production queries.

Spark tries to avoid excessive configuration reading in hot paths (e.g. via changes like #46979). If we do accidentally introduce such read paths, though, then this PR's optimizations will help to greatly reduce the associated perf. penalty.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Correctness should be covered by existing unit tests.

To measure performance, I did some manual benchmarking by running

val conf = new SparkConf()
conf.set("spark.network.crypto.enabled", "true")

followed by

conf.get(NETWORK_CRYPTO_ENABLED)

10 million times in a loop.

On my laptop, the optimized code is ~7.5x higher throughput than the original.

We can also compare the before-and-after flamegraphs from a while(true) configuration reading loop, showing a clear difference in hotspots before and after this change:

Before:

After:

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Github Copilot

[mridulm]

super nit: avoid shadowing value ? ... that would also allow us to simply return value for case 3 (and maybePrependedValue for case 2)

[JoshRosen]

Good idea.

I found that we can actually go a bit further and collapse all but the first case into a single case: if we don't match (Some(prependedValue), Some(value)) then we can just return value.orElse(prependedValue) since only one of the two options will be non-empty in that branch.

a18177f

[allisonwang-db]

👍

[HyukjinKwon]

Merged to master.