[SPARK-33454][INFRA] Add GitHub Action job for Hadoop 2

[dongjoon-hyun]

What changes were proposed in this pull request?

This PR aims to protect Hadoop 2.x profile compilation in Apache Spark 3.1+.

Why are the changes needed?

Since Apache Spark 3.1+ switch our default profile to Hadoop 3, we had better prevent at least compilation error with Hadoop 2.x profile at the PR review phase. Although this is an additional workload, it will finish quickly because it's compilation only.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass the GitHub Action.

• This should be merged after [SPARK-33288][YARN][FOLLOW-UP][test-hadoop2.7] Fix type mismatch error #30375 .

[dongjoon-hyun]

We can upgrade from Hadoop 2.7.4 to Hadoop 2.10.x. So, the job name is only mentioning the major version.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35699/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35699/

[mridulm]

Thanks for making this change @dongjoon-hyun !
This applies only to github actions right ? I assumed that currently the procedure was to wait for either jenkins or github actions to pass before merge.
With this change, a 2.x issue would be caught only by github actions, not jenkins.

Or does jenkins leverage the same ?

[dongjoon-hyun]

Ya. That's true. However, GitHub Action is very fast and Hadoop 2 build compilation error is easily detected. I means the result comes in 15 minutes and it's hard to consider it as a flaky job.

For Jenkins, IIUC, it's difficult to apply this in PR Builder. Currently, all PR Builders (PRBuilder, K8s IT PRBuilder) are focusing on the default profile because the runs are too heavy. You may ask Shane for new PRBuilder. He knows better than me about the capacity and new PR Builder for Hadoop 2.

[dongjoon-hyun]

BTW, I have only read-only access privilege to see Jenkins configuration.

[mridulm]

+CC @shaneknapp
Will it be possible to do the same for jenkins ?
It will ensure consistency and no accidental merges.

[mridulm]

I means the result comes in 15 minutes and it's hard to consider it as a flaky job.

I am concerned sometimes we might just not see it - given it is listed at the bottom of the list (if jenkins gives green).
If we cant do it in both, atleast reordering so that build failures are called out first would be good. Thoughts @dongjoon-hyun ?

[dongjoon-hyun]

Sure! Given the importance of Hadoop 2, it sounds good to me.

[SparkQA]

Test build #131096 has finished for PR 30378 at commit f439d8a.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35701/

[SparkQA]

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35701/

[SparkQA]

Test build #131098 has finished for PR 30378 at commit 7840d67.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[HyukjinKwon]

There looks a legitimate failure but I like this idea +1.

[dongjoon-hyun]

Yes, @HyukjinKwon . The failure is the motive of this PR. We should prevent that happens again.

[dongjoon-hyun]

Since #30375 is merged now, I retriggered GitHub Action.

cc @mridulm , @HyukjinKwon , @wangyum

[viirya]

Looks okay, just one question.

[dongjoon-hyun]

It seems that there exists some delay on GitHub Action. I retriggered GitHub Action because the previous run fails with compilation error again.

[dongjoon-hyun]

BTW, @HyukjinKwon . Do you still want to use Java 8 in this PR?

Scala 2.13 build job is also using Java 11.

[dongjoon-hyun]

Thank you, @wangyum . I'm looking at both this PR and vanilla master branch, too.

The previous failure might happen by another unknown reason instead of the API issue.

[dongjoon-hyun]

It's weird. The new code is not consumed at the GitHub Action re-trigger. I'll rebase this to the master~

[error] /home/runner/work/spark/spark/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala:320:52: type mismatch;
706
[error]  found   : Long
707
[error]  required: Int
708
[error]         Resource.newInstance(resourcesWithDefaults.totalMemMiB, resourcesWithDefaults.cores)
709
[error]                                                    ^
710
[error] one error found

[wangyum]

Thank you @dongjoon-hyun.

[dongjoon-hyun]

Sorry for the rebasing after your approval, @HyukjinKwon , @wangyum , @viirya . It was inevitable to bring the latest master branch. (I didn't notice that GitHub Action doesn't use the latest master.)

Also, I switched to Java 8 according to @HyukjinKwon 's advice.

[dongjoon-hyun]

Now, the new job is green.

[dongjoon-hyun]

Could you approve once more please?

[HyukjinKwon]

Merged to master.

[dongjoon-hyun]

Yay! Thank you, @viirya and @HyukjinKwon ! 😄

[SparkQA]

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35735/

[SparkQA]

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/35735/

[SparkQA]

Test build #131132 has finished for PR 30378 at commit f26fc30.

• This patch fails due to an unknown error code, -9.
• This patch merges cleanly.
• This patch adds no public classes.

[wangyum]

Another related question, do you think we should test Java 14 by Github Action?

[HyukjinKwon]

There's a discussion thread in the mailing list about JDK 14 FYI: http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-on-JDK-14-td30348.html

[dongjoon-hyun]

Although it's important to track upstream, Java 14 is already EOL.
Only Java 8 / Java 11 / Java 17 (September 2021) are LTS.