Consolidate / fix dockerfiles and docker builds

[deepthi]

Overview of the Issue

We have several different docker images we publish on dockerhub which are supposed to be "full" or comprehensive vitess installs: base, lite, k8s.
In addition, we use k8s as the base and build small images for individual binaries.

We have a few problems right now

• k8s images are used by helm charts, but everyone else uses lite images. It will be nice to eliminate the k8s image altogether and use lite as the base for all the smaller images.
• k8s images are not auto-built with releases unlike the lite image
• When we make a release only the mysql57 version of the lite image is being built.
• If we want to auto-build any other images (e.g. vtexplain) with a release, we currently don't have a way of making that wait until its base image with the same tag has been built.
• Somehow each push to master results in two docker builds with the same SHA. Given the number of variants we build, this slows the pipeline down quite a bit.
• There are some other issues like vtexplain being installed in a different place from vtctlclient which is confusing to users.

[derekperkins]

How committed are we to sticking with the name lite? There have been a number of questions about it in the past. If it is going to contain all the binaries, can we just call it vitess?

[deepthi]

vitess/vitess? It would be a breaking change for people who are already using lite. That is the main concern.
@enisoc do you want to comment?

[derekperkins]

Yes, vitess/vitess. We can always push the same image to lite too. vitess/lite isn't memorable and doesn't inspire confidence that you've downloaded the production version of Vitess.

https://hub.docker.com/r/cockroachdb/cockroach
https://hub.docker.com/r/mysql/mysql-server

On a similar note, we're artificially inflating the amount of installation and operational complexity by having all the separate binaries, since the majority of each binary is all the same code and a ton of dependencies. If we had a single binary that "shelled" out to all the existing binaries: e.g. vitess vtgate, vitess vttablet, etc, you'd have everything you needed in a single 100 MB binary instead of 10 80 MB files. There's a lot of value on the Cockroach side of having the install be a single binary.

curl https://binaries.cockroachdb.com/cockroach-v20.2.4.darwin-10.9-amd64.tgz | tar -xJ

That may be more than we want to tackle here, but I really think we could significantly simplify things.

[enisoc]

We have several different docker images we publish on dockerhub which are supposed to be "full" or comprehensive vitess installs: base, lite, k8s.

The names base and lite are historical cruft from when I was still figuring out how to use Docker in the first place, and I think we should move away from them at our earliest convenience. In more modern setups, what we call lite is just the normal image; what we call bootstrap is something like a build image; and base does not exist.

If we want to auto-build any other images (e.g. vtexplain) with a release, we currently don't have a way of making that wait until its base image with the same tag has been built.

Somehow each push to master results in two docker builds with the same SHA. Given the number of variants we build, this slows the pipeline down quite a bit.

In general, I think we should move off of Docker Hub's automation and use GitHub Actions. The main blocker for that has been that we didn't want to give GHA in our public repo the credentials to push the final images to Docker Hub, but someone recently explained that we can get around that by creating a private repo under the vitessio org and running the workflow there instead. That private repo could pull the public repo and run scripts that remain checked in on the public side.

We can always push the same image to lite too.

Good idea! And this will be trivial to do from GHA. You just run another docker tag and docker push. It should reuse the already-uploaded image.

If we had a single binary that "shelled" out to all the existing binaries

Nice idea! Might want to track that in a separate issue.

[empeje]

In addition to the issues/comments above, I'd like to add a point-of-view from the documentation side. As someone who is interested in Vitess, I found the documentation shows unnecessary steps. For example, in here, https://vitess.io/docs/get-started/local-docker/ we are encouraged to do make docker_local which builds Docker image in our local, I hope there's an out of the box Docker image ready so I just need to run docker-compose.yml or some other script to run Vitess in my local.

But again when I look around Docker Hub I found the same confusion as @derekperkins said

Yes, vitess/vitess. We can always push the same image to lite too. vitess/lite isn't memorable and doesn't inspire confidence that you've downloaded the production version of Vitess.