Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[v1.x] Fix nightly CD for python docker image releases #19774

Merged
merged 3 commits into from
Mar 2, 2021
Merged

[v1.x] Fix nightly CD for python docker image releases #19774

merged 3 commits into from
Mar 2, 2021

Conversation

mseth10
Copy link
Contributor

@mseth10 mseth10 commented Jan 21, 2021 •
edited
Loading

Description

This PR makes multiple changes to fix the nightly CD pipeline for python docker image releases.

  • Cherry-pick 34010ea
  • Change push target from dockerhub to ecr public repository
  • Update CUDA 11.2 base image for ubuntu16
  • Use nightly_v1.x prefix to differentiate from master branch docker tags
  • Remove unused mxnet_lib/dynamic pipeline

@mseth10 mseth10 requested a review from szha as a code owner January 21, 2021 23:43
@mxnet-bot
Copy link

Hey @mseth10 , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [windows-gpu, website, centos-cpu, edge, unix-gpu, sanity, windows-cpu, unix-cpu, miscellaneous, centos-gpu, clang]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@lanking520 lanking520 added the pr-awaiting-testing PR is reviewed and waiting CI build and test label Jan 21, 2021
@lanking520 lanking520 added pr-awaiting-merge Review and CI is complete. Ready to Merge and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels Jan 22, 2021
leezu
leezu reviewed Jan 22, 2021
View reviewed changes
Copy link
Contributor

@leezu leezu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As part of this work, can we reconsider the publication strategy? Currently nightly builds are pushed to dockerhub, which is not compatible with ASF policy. Instead, can you configure the job to only push on approved release tags?

@szha
Copy link
Member

szha commented Feb 8, 2021

we could also consider publishing nightly tags to ECR and make them available to only developers.

@mseth10
Copy link
Contributor Author

mseth10 commented Feb 26, 2021

@leezu @szha We have changed docker push target to a public ecr repo in master with this PR #19772 . Will backport it to v1.x too.

@mseth10 mseth10 force-pushed the update_nightly_docker_tag branch 2 times, most recently from 3249e55 to ec67ada Compare February 26, 2021 16:32
@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-awaiting-merge Review and CI is complete. Ready to Merge labels Feb 26, 2021
@mseth10 mseth10 changed the title differentiate nightly docker image tags [v1.x] Fix nightly CD for python docker image releases Feb 26, 2021
@lanking520 lanking520 added pr-awaiting-review PR is waiting for code review and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels Feb 26, 2021
@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test pr-work-in-progress PR is still work in progress and removed pr-awaiting-review PR is waiting for code review pr-awaiting-testing PR is reviewed and waiting CI build and test labels Feb 26, 2021
@lanking520 lanking520 added pr-awaiting-testing PR is reviewed and waiting CI build and test and removed pr-work-in-progress PR is still work in progress labels Feb 27, 2021
mseth10 and others added 2 commits February 27, 2021 04:17
@mseth10
Fix nightly CD for python docker image releases (apache#19772)
5a9b7d0 
* install wget

* test cd docker in ci

* install docker

* install python3-dev and gcc

* remove docker testing from ci

* remove python3-dev

* ecr target

* skip build test

* adding back python3-dev for make

* remove dynamic and pypi stages for testing

* install build-essential

* install zlib

* update python version

* update ld library path

* install openssl

* update test packages for python3.7

* remove call to deleted safe_docker_run.py

* hardcode region for public ecr repo

* use deadsnakes to install python

* revert dependency change

* refactor ecr login

* update ecr repo jenkins global var

* cleanup

* update docker authentication

* add ecr repo

* add back pypi and tests

* remove unused libmxnet pipeline

* update cu112 base docker

* update base docker images to ub18
@mseth10
differentiate tag prefix for v1.x
fdc789d
@lanking520 lanking520 added pr-awaiting-review PR is waiting for code review and removed pr-awaiting-testing PR is reviewed and waiting CI build and test labels Feb 27, 2021
@lanking520 lanking520 added pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test pr-awaiting-review PR is waiting for code review and removed pr-awaiting-review PR is waiting for code review pr-work-in-progress PR is still work in progress pr-awaiting-testing PR is reviewed and waiting CI build and test labels Feb 27, 2021
access2rohit
access2rohit approved these changes Mar 1, 2021
View reviewed changes
Copy link
Contributor

@access2rohit access2rohit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

waytrue17
waytrue17 approved these changes Mar 2, 2021
View reviewed changes
cd/Jenkinsfile_utils.groovy
// artifact (docker image, binary, pypi or maven package, etc.) that should
// be published.

STATE_UPDATE="State Update"

def trigger_release_job(job_name, job_type, mxnet_variants) {
def trigger_release_job(cd_release_job, job_name, job_type, mxnet_variants) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Curious, why do we pass the cd_release_job instead of using the env var?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CD_RELEASE_JOB_NAME is actually a parameter of the CD pipeline, so this is how it's supposed to be used. It is also set as a global env variable, which can potentially cause some issues.
For reference, it is addressed in master by this PR #17775

@mseth10 mseth10 merged commit 9b8a5e6 into apache:v1.x Mar 2, 2021
@mseth10 mseth10 deleted the update_nightly_docker_tag branch March 2, 2021 01:35
mseth10 added a commit to mseth10/incubator-mxnet that referenced this pull request Mar 5, 2021
@mseth10 @szha
[v1.x] Fix nightly CD for python docker image releases (apache#19774)
fd2c442 
* [CD] switch CD_RELEASE_JOB_NAME from global env var to job argument (apache#17775)

* Fix nightly CD for python docker image releases (apache#19772)

* install wget

* test cd docker in ci

* install docker

* install python3-dev and gcc

* remove docker testing from ci

* remove python3-dev

* ecr target

* skip build test

* adding back python3-dev for make

* remove dynamic and pypi stages for testing

* install build-essential

* install zlib

* update python version

* update ld library path

* install openssl

* update test packages for python3.7

* remove call to deleted safe_docker_run.py

* hardcode region for public ecr repo

* use deadsnakes to install python

* revert dependency change

* refactor ecr login

* update ecr repo jenkins global var

* cleanup

* update docker authentication

* add ecr repo

* add back pypi and tests

* remove unused libmxnet pipeline

* update cu112 base docker

* update base docker images to ub18

* differentiate tag prefix for v1.x

Co-authored-by: Sheng Zha <szha@users.noreply.github.com>
mseth10 added a commit that referenced this pull request Mar 8, 2021
@mseth10 @szha
[v1.8.x] Cherry-pick CD pipelines refactor with fix for docker images…
dd4661a 
… pipeline (#19989)

* [v1.x] Fix nightly CD for python docker image releases (#19774)

* [CD] switch CD_RELEASE_JOB_NAME from global env var to job argument (#17775)

* Fix nightly CD for python docker image releases (#19772)

* install wget

* test cd docker in ci

* install docker

* install python3-dev and gcc

* remove docker testing from ci

* remove python3-dev

* ecr target

* skip build test

* adding back python3-dev for make

* remove dynamic and pypi stages for testing

* install build-essential

* install zlib

* update python version

* update ld library path

* install openssl

* update test packages for python3.7

* remove call to deleted safe_docker_run.py

* hardcode region for public ecr repo

* use deadsnakes to install python

* revert dependency change

* refactor ecr login

* update ecr repo jenkins global var

* cleanup

* update docker authentication

* add ecr repo

* add back pypi and tests

* remove unused libmxnet pipeline

* update cu112 base docker

* update base docker images to ub18

* differentiate tag prefix for v1.x

Co-authored-by: Sheng Zha <szha@users.noreply.github.com>

* pass version param (#19982)

* remove cu112 changes

Co-authored-by: Sheng Zha <szha@users.noreply.github.com>
mseth10 added a commit to mseth10/incubator-mxnet that referenced this pull request Mar 15, 2021
@mseth10 @szha
[v1.8.x] Cherry-pick CD pipelines refactor with fix for docker images…
d45343c 
… pipeline (apache#19989)

* [v1.x] Fix nightly CD for python docker image releases (apache#19774)

* [CD] switch CD_RELEASE_JOB_NAME from global env var to job argument (apache#17775)

* Fix nightly CD for python docker image releases (apache#19772)

* install wget

* test cd docker in ci

* install docker

* install python3-dev and gcc

* remove docker testing from ci

* remove python3-dev

* ecr target

* skip build test

* adding back python3-dev for make

* remove dynamic and pypi stages for testing

* install build-essential

* install zlib

* update python version

* update ld library path

* install openssl

* update test packages for python3.7

* remove call to deleted safe_docker_run.py

* hardcode region for public ecr repo

* use deadsnakes to install python

* revert dependency change

* refactor ecr login

* update ecr repo jenkins global var

* cleanup

* update docker authentication

* add ecr repo

* add back pypi and tests

* remove unused libmxnet pipeline

* update cu112 base docker

* update base docker images to ub18

* differentiate tag prefix for v1.x

Co-authored-by: Sheng Zha <szha@users.noreply.github.com>

* pass version param (apache#19982)

* remove cu112 changes

Co-authored-by: Sheng Zha <szha@users.noreply.github.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@leezu leezu leezu left review comments

@waytrue17 waytrue17 waytrue17 approved these changes

@access2rohit access2rohit access2rohit approved these changes

@szha szha szha approved these changes

Assignees
No one assigned
Labels
pr-awaiting-review PR is waiting for code review
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@mseth10 @mxnet-bot @szha @leezu @access2rohit @waytrue17 @lanking520