opam.ocaml.org takes a too long and unpredictible time to deploy

[kit-ty-kate]

opam2web only takes 30mins to build. However, ocurrent-deployer (for some ungodly reason) wants to push the resulting 22GB image to docker hub, just to pull it again right after. This whole process in the end takes 3+ hours, sometimes more if docker hub doesn’t like it, and somtimes even twice that when opam-repository maintainers merge PRs in the middle.

Could we please detach this whole "wait for docker hub twice" process and simply transfer the docker image to the target server directly?

[mtelvers]

opam2web does not take very long to build the tool, but it does take a long time to run once it has been built. Typically, the overall run is 90 minutes. As you note, there is some unpredictability in pushing to Docker hub: the process can timeout or just randomly fail as the layers are very large.

The machines opam-4.ocaml.org and opam-5.ocaml.org run Docker in swarm mode. We use docker service update to update the image. This mechanism only works when the image is held on a registry. You cannot use a local image. Therefore, as it stands, we push to Docker hub in order to be able to pull it again and use that image to update the service.

However, we don't need to push to Docker hub, just a Docker registry. We could build opam.ocaml.org on a specific machine with a Docker registry running locally. Running a Docker registry seems straightforward, and adding authentication isn't too hard. The push would then be local, not over the network. The update step on opam-4 and opam-5 would then pull from this registry rather than Docker hub. This removes the push over the network and should guarantee the performance of the push.

I have created a PR#211 to do this and tested it using a cluster worker with a large local disk (x86-bm-c9.sw.ocaml.org). @avsm @dra27

[kit-ty-kate]

Thank you so much @mtelvers! ❤️

[mtelvers]

This pipeline has now been deployed for a couple of weeks and seems to be giving us a consistent deployment time

[kit-ty-kate]

Thank you so much! ❤️