Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

[v1.x] Backport edge pipeline #18375

Merged
merged 14 commits into from
May 26, 2020
Merged

[v1.x] Backport edge pipeline #18375

merged 14 commits into from
May 26, 2020

Conversation

nickguletskii
Copy link
Contributor

@nickguletskii nickguletskii commented May 20, 2020 •
edited
Loading

Description

Cherry pick commits from #17984 onto v1.x to fix #18298

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • Code is well-documented:
  • For user-facing API changes, API doc string has been updated.
  • For new C++ functions in header files, their functionalities and arguments are documented.
  • For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
  • Check the API doc at https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
  • To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

  • Cherry pick 36c87c3
  • Cherry pick 104074a
  • Updated the link to the IUS repository to fix CI jobs on CentOS: IUS CDN iusrepo/infrastructure#14
  • Cherry pick 6d1d122
  • Remove the parts of the backported commits that depend on C++17.
  • Cherry pick b1dd13c
  • Fix compilation errors related to the missing -stdlib flag.
  • Verify that the new QEMU tutorial works.

Comments

  • This PR is not just about the Jetson pipeline: the Android ARM pipelines were also upgraded.
  • The changes to RNNs mean that the same random seeds will produce different results before and after this patch.

@mxnet-bot
Copy link

Hey @nickguletskii , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [windows-cpu, clang, centos-gpu, unix-gpu, website, edge, miscellaneous, sanity, centos-cpu, unix-cpu, windows-gpu]

Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@nickguletskii nickguletskii force-pushed the bugfix/backport_edge_ci_pipeline branch from a929c8c to bcfa364 Compare May 20, 2020 21:07
@nickguletskii
Copy link
Contributor Author

@mxnet-bot run ci [sanity]

@mxnet-bot
Copy link

Jenkins CI successfully triggered : [sanity]

@leezu
Copy link
Contributor

leezu commented May 22, 2020 •
edited
Loading

@nickguletskii thank you for your help fixing the 1.x pipeline!

To address the current Jetson related failure, it would be sufficient to backport only the jetson changes (ie Dockerfile.build.jetson and the corresponding cmake files). @waytrue17 is attempting to do that in #18339 for the 1.7 branch.

However, backporting more of the changes (as in your PR) will help the CI stability on the 1.x branch. So once your PR passes CI, it's great to merge it. Thanks!

@leezu
Copy link
Contributor

leezu commented May 22, 2020

@mxnet-bot run ci [all]

@mxnet-bot
Copy link

Jenkins CI successfully triggered : [clang, centos-cpu, miscellaneous, centos-gpu, windows-cpu, windows-gpu, sanity, unix-cpu, website, unix-gpu, edge]

@nickguletskii
Copy link
Contributor Author

@mxnet-bot run ci [centos-gpu]

@mxnet-bot
Copy link

Jenkins CI successfully triggered : [centos-gpu]

@nickguletskii nickguletskii marked this pull request as ready for review May 24, 2020 07:54
@ChaiBapchya
Copy link
Contributor

@leezu looks like edge pipeline passes CI
Good to merge?

@ChaiBapchya
Copy link
Contributor

And in that case should 1.7.x branch have this PR cherry-picked or #18339 merged?

@leezu leezu merged commit fe90008 into apache:v1.x May 26, 2020
@leezu
Copy link
Contributor

leezu commented May 26, 2020

@nickguletskii thank you for your work on this PR!

@ChaiBapchya this would be up to @ciyongch given that AFAIK there is already a code-freeze for 1.7 and this PR is not a "minimal fix" but rather a more complete backport from the master branch.

@ChaiBapchya ChaiBapchya mentioned this pull request May 26, 2020
Merged
@ciyongch ciyongch mentioned this pull request May 27, 2020
Merged
@ciyongch
Copy link
Contributor

Yes, I agree with @leezu , for the current v1.7.x branch, let's keep the main code base and with necessary minimal fix to make the release process moving forward, as this PR seems to be quite a large one :)

@ChaiBapchya
Copy link
Contributor

Makes sense! Thanks @leezu @ciyongch

ChaiBapchya pushed a commit to ChaiBapchya/mxnet that referenced this pull request Aug 15, 2020
@nickguletskii @leezu @ChaiBapchya
[v1.x] Backport edge pipeline (apache#18375)
7c9c9fc 
* Update edge toolchain

* Support platforms without rand_r

* Fix the URL to the IUS repository

* compiler warnings

* Use a pre-c++17 way of distinguishing between device types

* Greatly simplify qemu setup

* Request the C++ standard library and extensions

* Upgrade dmlc-core to resolve build errors

* Remove leftovers from C++17 dev type check

* Fix CPU-only RRNOp Forward

* Change the ARM8 build to work like the ARM7 build

* Revert "Fix CPU-only RRNOp Forward"

This reverts commit 0a921a4.

* Hack around the lack of constexpr if

* Adjust the list of files to be packed in ARM jobs

Co-authored-by: Leonard Lausen <lausen@amazon.com>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@leezu leezu leezu approved these changes

@aaronmarkham aaronmarkham Awaiting requested review from aaronmarkham aaronmarkham is a code owner

@anirudh2290 anirudh2290 Awaiting requested review from anirudh2290 anirudh2290 is a code owner

@marcoabreu marcoabreu Awaiting requested review from marcoabreu marcoabreu is a code owner

@szha szha Awaiting requested review from szha szha is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@nickguletskii @mxnet-bot @leezu @ChaiBapchya @ciyongch