[Bug]: Build Release Candidate stage_docker job fails with out of space error.

[lostluck]

What happened?

https://github.com/apache/beam/actions/workflows/build_release_candidate.yml

stage_docker job fails

#13 117.0 --- Logging error ---
4340#13 117.0 Traceback (most recent call last):
4341#13 117.0   File "/usr/local/gcloud/google-cloud-sdk/lib/googlecloudsdk/core/updater/installers.py", line 233, in DownloadAndExtractTar
4342#13 117.0     fp.write(chunk)
4343#13 117.0 OSError: [Errno 28] No space left on device
4344#13 117.0 
4345#13 117.0 During handling of the above exception, another exception occurred:
4346#13 117.0 
4347#13 117.0 Traceback (most recent call last):
4348#13 117.0   File "/usr/local/gcloud/google-cloud-sdk/lib/googlecloudsdk/core/updater/installers.py", line 362, in _InstallTar
4349#13 117.0     return DownloadAndExtractTar(
4350#13 117.0   File "/usr/local/gcloud/google-cloud-sdk/lib/googlecloudsdk/core/updater/installers.py", line 238, in DownloadAndExtractTar
4351#13 117.0     raise URLFetchError(e)
4352#13 117.0 googlecloudsdk.core.updater.installers.URLFetchError: [Errno 28] No space left on device
4353#13 117.0 
4354#13 117.0 During handling of the above exception, another exception occurred:
4355#13 117.0 
4356#13 117.0 Traceback (most recent call last):
4357#13 117.0   File "/usr/local/gcloud/google-cloud-sdk/lib/googlecloudsdk/calliope/cli.py", line 987, in Execute
4358Error: No space left on device : '/home/runner/runners/2.308.0/_diag/pages/33207c11-3b7a-47ff-8a8f-ec83a1ce1461_dca69a40-7968-5563-ee53-0628c33d568e_1.log'

Likely due to now building Multi-arch containers. We'll either somehow need runners with more space on them, or rearchitect the action to not be so space hungry.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

• Component: Python SDK
• Component: Java SDK
• Component: Go SDK
• Component: Typescript SDK
• Component: IO connector
• Component: Beam examples
• Component: Beam playground
• Component: Beam katas
• Component: Website
• Component: Spark Runner
• Component: Flink Runner
• Component: Samza Runner
• Component: Twister2 Runner
• Component: Hazelcast Jet Runner
• Component: Google Cloud Dataflow Runner

[lostluck]

cc: @damccorm for triage

[Abacn]

multiarch double the space needed for containers, now github action fails halfway, when build python3.10 container. Runners (flink etc) already pushed to dockerhub, e.g. https://hub.docker.com/r/apache/beam_flink1.15_job_server/tags

[damccorm]

With previous instances of the same problem, we've worked around it by pruning the images in the middle -

beam/build.gradle.kts

Line 597 in a3b1d0b

doLast {

Now that we're building/pushing multiarch, we probably just need to do that more often