[SPARK-29670][CORE][2.4] Make executor bind address configurable for Spark 2.4

[gedeh]

What changes were proposed in this pull request?

Backporting changes from #26331 to Spark 2.4, in a similar fashion, executor's bindAddress is passed as an input parameter for RPCEnv.create.

This PR also ensure YarnAllocator use the config when allocating new executor and use default hostname when its not configured.

Why are the changes needed?

Recently we've came across this issue with Spark running on Yarn in Istio enabled Kubernetes cluster. As I understand it, Spark 2.4 is a stepping stone for teams moving away from Scala 2.11 to Scala 2.12 and then moving to Spark 3.

Does this PR introduce any user-facing change?

Yes, new config is added and relevant doc is updated.

How was this patch tested?

I will run this on Kubenetes and Istio.

[AmplabJenkins]

Can one of the admins verify this patch?

[srowen]

This could be OK, but I'm not sure there will be any more 2.4.x releases after 2.4.8 anyway

[gedeh]

This could be OK, but I'm not sure there will be any more 2.4.x releases after 2.4.8 anyway

Ow, I didn't aware of this. That's explains 2.4.9-SNAPSHOT changed to 2.4.8 in pom.xml

[dongjoon-hyun]

Thank you for making a PR, @gedeh .

However, this PR is invalid in a few ways.

1. SPARK-29670 is closed as Duplicated. You cannot ask a backport of unmerged JIRA.
2. The actual merged JIRA for feature is SPARK-24203 which was an Improvement at Apache Spark 3.0.0. Apache Spark community have a general policy which allows bug-fix backporting only. So, we cannot backport a new feature because we need to protect old branches.
3. Lastly, 2.4.8 was the EOL release already. Here is some explanation for that.

• https://spark.apache.org/versioning-policy.html

The last minor release within a major a release will typically be maintained for longer as an “LTS” release. For example, 2.4.0 was released in November 2nd 2018 and had been maintained for 31 months until 2.4.8 was released on May 2021. 2.4.8 is the last release and no more 2.4.x releases should be expected even for bug fixes.

Given that the above reasons, could you close this PR, @gedeh ?

[gedeh]

Thank you @dongjoon-hyun for clarification. I wasn't aware of EOL of Spark 2.4 closing this PR. Thanks!

[dongjoon-hyun]

Thank you so much, @gedeh !