Add new Jenkins pipeline to build TVM CI docker images #62
Conversation
leandron
force-pushed
the
jenkins-docker-rebuild
branch
from
e98eb13
to
4316a95
Compare
* The new pipeline is able to build the TVM Docker images and upload them to Docker Hub * It makes sure that all images are guaranteed to use the same TVM git hash, using a timestamp and short hash, e.g. ci_cpu:2021_07_14-145016-c16d61b
| stage('Build') { | ||
| parallel { | ||
|
|
||
| stage('ci-cpu') { |
There was a problem hiding this comment.
just one question, is it possible to define these using e.g. a for loop over a dict mapping image name to node label?
There was a problem hiding this comment.
It would be cool, but I couldn't find a way to do it. We could do that in pure Groovy with a shared library I think, but that open a whole new way of dealing with the jobs, so I recommend we stick with a plain Jenkinsfile like this one.
| } // stages | ||
| post { | ||
| success { | ||
| discordSend description: "New images published on DockerHub with tag `${TVM_DOCKER_TAG}`. Use `docker pull tlcpackstaging/<image_type>:${TVM_DOCKER_TAG}` to download the images. Image types: `ci_arm`, `ci_cpu`, `ci_gpu`, `ci_i386`, `ci_lint`, `ci_qemu`, `ci_wasm`.", |
There was a problem hiding this comment.
would be great if we can source the list of image types from above dict
There was a problem hiding this comment.
The issue is that images are built in parallel, independently. I'm not seeing a simpler way to make them share some status.
| webhookURL: "${DISCORD_WEBHOOK}" | ||
| } | ||
| unsuccessful { | ||
| discordSend description: "Failed to generate Docker images using TVM hash `${TVM_GIT_REPO_URL}`. See logs at ${BUILD_URL}.", |
There was a problem hiding this comment.
what if some of them succeed?
There was a problem hiding this comment.
If not all of them succeed (this happens sometimes) we consider the build as failed, because one of the goals is to have a full set of images that represent a give hash on GitHub.
As the builds run in parallel, some of them might upload images to DockerHub as a side effect, and we can use if we want to, but we won't be doing further testing on partial builds.
leandron
force-pushed
the
jenkins-docker-rebuild
branch
from
4316a95
to
6163f55
Compare
|
ping @areusch for review when you have a minute |
|
Thanks for landing this @leandron !! amazing work, really appreciate you doing this, going to be transformative for the community. |
This is a draft Jenkins pipeline for us to daily build TVM CI docker images.