[CI] Reduce the number of times Docker images are built

[hcho3]

Currently, the CI server heavily uses Docker containers to manage suitable testing environments for the Linux platform. The original intent was to avoid the need for installing packages over and over again, by caching Docker images that were previously built. So the cost of building Docker containers would be amortized over multiple test jobs.

Unfortunately, in recent weeks, the Docker containers are being re-built from scratch very frequently. The reason is that, whenever the base container (nvidia/cuda) gets updated, all the cached Docker layers get invalidated, forcing our CI server to re-build the container from scratch. Edit. This is not the correct diagnosis. See the latest comment below.

I recognize the importance of that security patches, but having to re-build entire Docker containers from scratch is tremendously wasteful. (It takes 20-30 mins to complete the build.)

Proposed fix. We should have a cron job (or something equivalent) that builds new Docker containers from the Dockerfiles at a preset frequency (say, every 2 weeks). All CI jobs should now always pull pre-build containers, instead of attempting to build containers from Dockerfile. In short, our containers should be updated less frequently than the base cuda container is updated. The disadvantage is that updating Dockerfile won't cause rebuilding of the container immediately. Edit. See the latest comment below.

@trivialfis

[hcho3]

Example: the Jenkins CI server of the TVM project (https://github.com/apache/incubator-tvm) pulls pre-built images from the Docker hub. There exists a time gap between the modification of Dockerfile and the modification of the corresponding Docker container image. See https://tvm.apache.org/docs/contribute/pull_request.html#ci-environment. Also see the discussion in https://discuss.tvm.apache.org/t/ci-docker-how-to-match-docker-version-tags-in-source-control-with-the-ones-in-docker-hub/5152 for the rationale of avoiding Docker builds on the fly.

[hcho3]

Example where Docker containers gets re-built from scratch:

• https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/master/568/pipeline/211
• https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/master/568/pipeline/214
• https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/master/573/pipeline/211
• https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-6292/3/pipeline/207

[hcho3]

My diagnosis was wrong. The actual reason is that Docker containers with different CUDA versions (10.2, 11.0) get pushed to the same Docker registry (492475357299.dkr.ecr.us-west-2.amazonaws.com/xgb-ci.gpu). As a result, a container with CUDA 10.2 will cause a cache miss for a container with CUDA 11.0.

Fix. Create separate registries to cache containers with different CUDA versions, e.g. xgb-ci.gpu11.0, xgb-ci.gpu10.2 and so forth.