[SPARK-40783][INFRA] Enable Spark on K8s integration test

[Yikun]

What changes were proposed in this pull request?

This patch enable the Spark on K8s integration test:

• scala2.12-java11-python3-ubuntu: Run scala / PySpark basic test
• scala2.12-java11-ubuntu: Run scala basic test
• scala2.12-java11-r-ubuntu: Run scala / SparkR basic test
• scala2.12-java11-python3-r-ubuntu: Run all K8s integration test

Currently, we use the local registry as a bridge between build and test:

• Build: generate the image and push to local registry
• Test: load to minikube docker, run K8s test using specific image

Due to the multi-platform images cannot be exported with the docker export type, the local registry (push) is used here rather than local build (load). Compare to ghcr it reduces the network transmition and permission required.

Also:

• Upgrade setup-qemu-action to v2
• Upgrade setup-buildx-action to v2
• Remove ununsed Image digest step

Why are the changes needed?

To ensure the quality of official dockerfiles.

Does this PR introduce any user-facing change?

No

How was this patch tested?

CI passed

[Yikun]

Yikun#4

hmm, it was passed in my local repo, but failed in upstream repo, a little bit wired. because it use the wrong old image

(later)

WARNING: No output specified with docker-container driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load

OK, use load to export local

(later)

ERROR: docker exporter does not currently support exporting manifest lists
Error: buildx failed with: ERROR: docker exporter does not currently support exporting manifest lists

looks like need the local registry: https://github.com/docker/build-push-action/blob/master/docs/advanced/local-registry.md

[Yikun]

@HyukjinKwon Thanks, will merge this!