[SPARK-45278][YARN] Support executor bind address in Yarn executors for Spark 3.5

[gedeh]

What changes were proposed in this pull request?

Uptake --bind-address parameter in YarnCoarseGrainedExecutorBackend when launching new container in Yarn cluster. This PR also ensure YarnAllocator uses default hostname when its not configured.

Why are the changes needed?

We've came across issue with Spark running on Yarn in Istio enabled Kubernetes cluster. Previous PR #32633 is not merged because Spark 2.4 was EOL and 3.x branch didnt get enough traction.

Does this PR introduce any user-facing change?

Yes, new config specifically for Yarn cluster mode is added and relevant doc is updated.

How was this patch tested?

Tested in Kubenetes with Istio and added tests to YarnAllocatorSuite

Thanks!

[gedeh]

Same with #47892, but this one specifically for 3.5 branch

[dongjoon-hyun]

Thank you for making a PR, @gedeh .

However, Apache Spark community has a backporting policy which allows only bug-fixes. Since this is a new feature or improvement, we cannot accept this for branch-3.5.

[dongjoon-hyun]

Let me close this in order to prevent accidental merging. We can still continue to discuss on the closed PR.

[gedeh]

Gotcha, thanks @dongjoon-hyun OK I understand its new feature/improvement, its just that I have same PR during Spark 3.4.0-SNAPSHOT and sadly it was closed due to not much activity on that one.

Would be great to take a look for the upcoming Spark 4.0.0 version, although Spark 4.0.0 changes a lot compared to 3.5.0 branch. Thanks in advance

[gedeh]

Hi @dongjoon-hyun looking back again, is this still a feature improvement? It seems like a bug though that bind address which is exposed in configuration is not adhered by Yarn?

[dongjoon-hyun]

Adding a new configuration is definitely a new feature.