The action runner EC2 image used for connector publication may not have enough resource

[tuliren]

Summary

Running integration tests source-mysql and source-mysql-strict-encrypt by the /test command completes without any problem, while running the /publish command for the same two connectors would stuck there for hours.

The most recent example is here: #16259

One difference between the /test and /publish command is that the former uses EC2 image ami-0d648081937c75a73 with 80GB disk, while the latter uses the default ami-0f23be2f917510c26. I suspect the latter does not have enough disk space or memory to run the integration tests. When I tried to use ami-0d648081937c75a73 to run the /publish command, it failed because there is no docker buildx in that image.

Could we update ami-0d648081937c75a73 to have a newer version of docker that supports docker buildx, or add more resource to the default ami-0f23be2f917510c26?

Here is a summary:

	/test	/publish
EC2 image	ami-0d648081937c75a73	ami-0f23be2f917510c26
What it does	Run integration test.	Run integration test + publish docker image.
Problem	Does not have docker buildx.	May not have enough resource (e.g. disk).
Request (either one)	Install a newer version of docker with docker buildx in this image.	Keep the dependencies. Add more resource to this image.

[tuliren]

@davinchia, FYI.

[davinchia]

Thanks Liren!

[evantahler]

We are in the Github beta which would allow us to use "very big" servers to run Github actions. We could skip all the AWS meshugas and maybe go back to using Github Actions. cc @supertopher

Details over in https://github.com/airbytehq/airbyte-cloud/issues/2737

[evantahler]

@git-phu can you let me know if the AMIs here have been updated with the work last week?

[git-phu]

@evantahler they have not

Last week we tried some experiments with preloading some things into runner AMIs but we did not find significant gains with the things we tested so we didn't change any current build images.

However, on a separate note I recently updated the cloud pipelines to use a newer AMI that has the latest version of docker/docker compose (and has a 100gb disk), so if we are still encountering the issues originally mentioned in this issue, we could try out this ami
https://github.com/airbytehq/airbyte-cloud/pull/3424

[evantahler]

Ok, great! We'll update to AMI ami-06fb57d8e7c033959. I think we probably should harmonize all of our tools using one AMI if we can - less to manage and worry about.

[evantahler]

@git-phu how much HDD space does the new image made in https://github.com/airbytehq/airbyte-cloud/pull/3424 have? It looks like the /test command and /publish command use different images, and perhaps the main difference is hard drive space?

airbyte/.github/actions/start-aws-runner/action.yml

Lines 10 to 13 in 894d672

	ec2-image-id:
	# github-self-hosted-runner-ubuntu-20-100g-disk-with-cypress-deps
	default: "ami-005924fb76f7477ce"
	required: true

vs

airbyte/.github/workflows/test-command.yml

Lines 64 to 65 in 894d672

	# 80 gb disk
	ec2-image-id: ami-06cf12549e3d9c522

[git-phu]

@evantahler the new image uses 100gb for disk.
I saw some of our older images use less than that, so I can check for the ones you linked

[git-phu]

@evantahler

ami-005924fb76f7477ce

airbyte/.github/actions/start-aws-runner/action.yml

Lines 10 to 13 in 894d672

	ec2-image-id:
	# github-self-hosted-runner-ubuntu-20-100g-disk-with-cypress-deps
	default: "ami-005924fb76f7477ce"
	required: true

has 100gb for disk

---

ami-06cf12549e3d9c522

airbyte/.github/workflows/test-command.yml

Lines 64 to 65 in 894d672

	# 80 gb disk
	ec2-image-id: ami-06cf12549e3d9c522

has 80gb for disk

[git-phu]

afaik 100gb was chosen semi arbitrarily too, so we could just use even bigger disks if that could help avoid some test failures