Nightly Tests For Clojure #13624

hellonico · 2018-12-12T07:55:49Z

Description

This is about running the inrtegration tests during nightly CI.

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Changes

Feature1, tests, (and when applicable, API doc)
Feature2, tests, (and when applicable, API doc)

Comments

If this change is a backward incompatible change, why must this change be made.
Interesting edge cases to note here

ci/docker/runtime_functions.sh

ci/jenkins/Jenkins_steps.groovy

hellonico · 2018-12-12T08:24:30Z

updated !

ci/jenkins/Jenkins_steps.groovy

tests/nightly/Jenkinsfile

hellonico · 2018-12-12T23:37:34Z

Moved 'Clojure Nightly: CPU' to JenkinsfileForBinaries
Renamed main function to: unittest_ubuntu_cpu_clojure_integration_tests

ci/jenkins/Jenkins_steps.groovy

tests/nightly/JenkinsfileForBinaries

hellonico · 2018-12-13T06:28:48Z

http://jenkins.mxnet-ci.amazon-ml.com/blue/organizations/jenkins/mxnet-validation%2Funix-cpu/detail/PR-13624/4/pipeline

Let me guess ... its fails because the workspace is different in JenkinsfileForBinaries ?

gigasquid · 2018-12-14T00:08:37Z

@hellonico - I'm still a little bit unsure of how all the CI/ Nightly tests fit together. I started another PR (temporary) to test my theories of how it all works so we can get a better understanding of it. Anything we learn from that, we can apply here. #13642

WIP apache#13624 clojure nightly tests

gigasquid · 2018-12-14T16:41:24Z

@marcoabreu I think we are at a good point where we understand that the integration tests can run on a node in the regular CI. We just need to move that node over to the nightly tests. Do you think that you could make a copy of the job to point to this branch so that we can test that?

WIP apache#13624 clojure nightly tests

sandeep-krishnamurthy · 2018-12-29T18:07:45Z

@marcoabreu - can you please help here? Thanks.

gigasquid · 2019-01-08T14:37:57Z

@marcoabreu So I've been thinking about this a bit more and have a couple of questions.

It seems like other languages have integration tests that run as part of regular CI https://github.com/apache/incubator-mxnet/blob/master/ci/jenkins/Jenkins_steps.groovy (do a search for integration). Since the Clojure one does not take too long (15 min for the suite), could we have it as part of the regular CI instead of nightly?
If we can't and have to do nightly, how do we get notified of breaking tests?

stu1130 · 2019-01-16T20:40:30Z

@marcoabreu ping again for the update! Thanks

gigasquid · 2019-01-18T22:55:41Z

started a dev discussion about this - https://lists.apache.org/thread.html/a22e40685d543cce4e2f4420046e8142126275888c1c15cb83a7a103@%3Cdev.mxnet.apache.org%3E

I think we can move forward on this by just adding it to CI and not nightlies right now until the notifications get implemented.

gigasquid · 2019-01-18T23:00:48Z

@hellonico if you have some bandwidth to move this forward I think we need:

Merge with master
Remove any changes from the nightlies directory
Keep our CI integration tests (the new node we added and tested)
Tweak the integration-tests.sh to have some sort of exclude list so we have a way to easily take out tests that become problematical until we can fix them.

Thanks for your patience in figuring this out :)

hellonico · 2019-01-19T02:02:53Z

Bonjour !

Working on it ...

Merge with master
Remove any changes from the nightlies directory
Keep our CI integration tests (the new node we added and tested)
Tweak the integration-tests.sh to have some sort of exclude list so we have a way to easily take out tests that become problematical until we can fix them.
Update infer test cases to run in the integration phase (simply with lein test).
- imageclassifier and predictor are ok
- My change to draw contour boxes on the objectdetection broke the tests. Nothing is returned in detect-images-in-dir and detect-single-image, so nothing to test against. 🥇

larroy · 2019-01-19T15:40:54Z

tests/nightly/JenkinsfileForBinaries

+      node(NODE_LINUX_GPU) {
+        ws('workspace/clojure') {
+          utils.unpack_and_init('gpu', mx_lib)
+          utils.docker_run('ubuntu_nightly_gpu', 'unittest_ubuntu_cpu_clojure_integration', false)


unittest_ubuntu_cpu_clojure_integration Are these CPU or GPU? maybe rename the function to XPU?

@larroy Merci ! Thanks for the comment ! Right now and as it is, only CPU but it's almost ready to run a few GPU tests
I don't see any reference to XPU in master, is another language already using the notation ?

In c++ we use the notion for operator definitions that are applicable for CPU as well as GPU, thus called XPU. I can look for a reference if you're interested

hellonico · 2019-01-21T02:24:06Z

Hmmm ... locally, I get

Ran 2 tests containing 11 assertions.
0 failures, 0 errors.

And CI shows:

lein test :only infer.objectdetector-example-test/test-batch-detection
FAIL in (test-batch-detection) (objectdetector_example_test.clj:68)
expected: (= #{"dog" "person" "bicycle" "car"} (set (mapv :class predictions)))
  actual: (not (= #{"dog" "person" "bicycle" "car"} #{"dog" "cow" "person" "horse"}))

Will do some digging later.

marcoabreu · 2019-01-21T04:57:43Z

Poor person, getting misclassified as cow 🐄 :(

Jokes aside. There's a guide in our confluence wiki under continuous integration named "reproducing test results" that allows you to locally reproduce the results. I'm on mobile right now, but let me know if you can't find it and I'll be happy to provide a link.

hellonico · 2019-01-21T04:59:25Z

@marcoabreu poor cow as well ;)

Thanks, will dig into the wiki !

hellonico · 2019-01-21T06:43:36Z

Update infer test cases to run in the integration phase (simply with lein test).

~~imageclassifier and~~ predictor is ok
fix objectdetection on local/CI
imageclassifier-example-test failing on CI

…r test

hellonico · 2019-01-21T07:39:57Z

@gigasquid @marcoabreu : Thank you so much for all the help and support. I promise no cows were harmed during the making of this PR.

Let's get this in !!!!!!!!!!!!!!!!!!!!!!!!!!

gigasquid · 2019-01-22T01:12:58Z

Thanks @hellonico - We are soo close!
There is a conflict that needs to be resolved and it looks like there is still a file change in tests/nightly/JenkinsfileForBinaries - For this PR, let's remove any changes to that file so that we keep our integration tests in the regular CI. We can revisit the nighties later after notifications are implemented.

gigasquid · 2019-01-22T01:08:27Z

contrib/clojure-package/integration-tests.sh

-  cd ${test_dir} && lein test
-done
+# use AWK pattern for blacklisting
+TEST_CASES=`find ${EXAMPLES_HOME} -name test | awk '!/dontselect1|dontselect2/'`


Nice thanks :)

gigasquid

Looks good 💯

This reverts commit eebdd5f.

marcoabreu · 2019-01-23T07:27:09Z

Thanks a lot for your great efforts @hellonico and @gigasquid!

What do you think about sending an announcement to dev@ that we are now testing Clojure examples? That's a big win!

lanking520 · 2019-01-23T17:52:01Z

ci/docker/runtime_functions.sh

+unittest_ubuntu_cpu_clojure_integration() {
+    set -ex
+    make scalapkg USE_OPENCV=1 USE_BLAS=openblas USE_DIST_KVSTORE=1 ENABLE_TESTCOVERAGE=1
+    make scalainstall USE_OPENCV=1 USE_BLAS=openblas USE_DIST_KVSTORE=1 ENABLE_TESTCOVERAGE=1


Just to inform that we plan to deprecate the make command for Scala package. Please use mvn install instead to cover the Scala portion

Generally I'd like to keep the native build, the scala build and the scala packaging separate. Right now, this job is done on demand and this leads to GPU instances compiling code without caching. So this sounds like a move in the right direction!

hellonico requested a review from marcoabreu as a code owner December 12, 2018 07:55

marcoabreu reviewed Dec 12, 2018

View reviewed changes

ci/docker/runtime_functions.sh Outdated Show resolved Hide resolved

ci/jenkins/Jenkins_steps.groovy Outdated Show resolved Hide resolved

marcoabreu reviewed Dec 12, 2018

View reviewed changes

ci/jenkins/Jenkins_steps.groovy Show resolved Hide resolved

marcoabreu reviewed Dec 12, 2018

View reviewed changes

tests/nightly/Jenkinsfile Outdated Show resolved Hide resolved

hellonico added a commit to hellonico/incubator-mxnet that referenced this pull request Dec 12, 2018

WIP apache#13624 clojure nightly tests

d03fb30

marcoabreu reviewed Dec 13, 2018

View reviewed changes

ci/jenkins/Jenkins_steps.groovy Outdated Show resolved Hide resolved

marcoabreu reviewed Dec 13, 2018

View reviewed changes

tests/nightly/JenkinsfileForBinaries Outdated Show resolved Hide resolved

gigasquid added Clojure pr-work-in-progress PR is still work in progress labels Dec 13, 2018

hellonico force-pushed the topic/nightlyclj branch from f6e5b4f to 1446ce5 Compare December 14, 2018 01:42

hellonico requested a review from gigasquid as a code owner December 14, 2018 01:42

hellonico added a commit to hellonico/incubator-mxnet that referenced this pull request Dec 14, 2018

WIP apache#13624 clojure nightly tests

d9ec761

hellonico force-pushed the topic/nightlyclj branch from 1446ce5 to e3a3978 Compare December 14, 2018 08:11

hellonico added a commit to hellonico/incubator-mxnet that referenced this pull request Dec 14, 2018

merged with gigasquid's understanding-ci branch

e3a3978

WIP apache#13624 clojure nightly tests

hellonico added a commit to hellonico/incubator-mxnet that referenced this pull request Dec 19, 2018

merged with gigasquid's understanding-ci branch

e980d47

WIP apache#13624 clojure nightly tests

hellonico force-pushed the topic/nightlyclj branch from e3a3978 to 1d6910b Compare January 19, 2019 01:19

hellonico added a commit to hellonico/incubator-mxnet that referenced this pull request Jan 19, 2019

apache#13624 clojure nightly tests

1d6910b

larroy reviewed Jan 19, 2019

View reviewed changes

hellonico force-pushed the topic/nightlyclj branch from 8b6f767 to 6507124 Compare January 21, 2019 01:06

hellonico added a commit to hellonico/incubator-mxnet that referenced this pull request Jan 21, 2019

apache#13624 clojure nightly tests

6507124

hellonico force-pushed the topic/nightlyclj branch from 02fff1a to e934050 Compare January 21, 2019 06:19

hellonico added a commit to hellonico/incubator-mxnet that referenced this pull request Jan 21, 2019

apache#13624 clojure nightly tests

e934050

hellonico added a commit to hellonico/incubator-mxnet that referenced this pull request Jan 21, 2019

Nightly Tests For Clojure apache#13624 : update batch image classifie…

1f7c03c

…r test

gigasquid changed the title ~~WIP: Nightly Tests For Clojure~~ Nightly Tests For Clojure Jan 22, 2019

gigasquid reviewed Jan 22, 2019

View reviewed changes

apache#13624 clojure nightly tests

2b4d4b1

hellonico force-pushed the topic/nightlyclj branch from 1f7c03c to 2b4d4b1 Compare January 22, 2019 01:21

gigasquid approved these changes Jan 22, 2019

View reviewed changes

gigasquid merged commit eebdd5f into apache:master Jan 22, 2019

piyushghai added a commit to piyushghai/incubator-mxnet that referenced this pull request Jan 22, 2019

Revert "apache#13624 clojure nightly tests (apache#13624)"

4e33e33

This reverts commit eebdd5f.

lanking520 reviewed Jan 23, 2019

View reviewed changes

jessr92 pushed a commit to jessr92/incubator-mxnet that referenced this pull request Jan 27, 2019

apache#13624 clojure nightly tests (apache#13624)

c218230

stephenrawls pushed a commit to stephenrawls/incubator-mxnet that referenced this pull request Feb 16, 2019

apache#13624 clojure nightly tests (apache#13624)

7a9c07d

haohuanw pushed a commit to haohuanw/incubator-mxnet that referenced this pull request Jun 23, 2019

apache#13624 clojure nightly tests (apache#13624)

417a117

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nightly Tests For Clojure #13624

Nightly Tests For Clojure #13624

hellonico commented Dec 12, 2018

hellonico commented Dec 12, 2018

hellonico commented Dec 12, 2018

hellonico commented Dec 13, 2018

gigasquid commented Dec 14, 2018

gigasquid commented Dec 14, 2018

sandeep-krishnamurthy commented Dec 29, 2018

gigasquid commented Jan 8, 2019

stu1130 commented Jan 16, 2019

gigasquid commented Jan 18, 2019

gigasquid commented Jan 18, 2019

hellonico commented Jan 19, 2019 •

edited

Loading

larroy Jan 19, 2019

hellonico Jan 21, 2019

marcoabreu Jan 21, 2019

hellonico commented Jan 21, 2019

marcoabreu commented Jan 21, 2019

hellonico commented Jan 21, 2019

hellonico commented Jan 21, 2019 •

edited

Loading

hellonico commented Jan 21, 2019 •

edited

Loading

gigasquid commented Jan 22, 2019

gigasquid Jan 22, 2019

gigasquid left a comment •

edited

Loading

marcoabreu commented Jan 23, 2019

lanking520 Jan 23, 2019

marcoabreu Jan 23, 2019

Nightly Tests For Clojure #13624

Nightly Tests For Clojure #13624

Conversation

hellonico commented Dec 12, 2018

Description

Checklist

Essentials

Changes

Comments

hellonico commented Dec 12, 2018

hellonico commented Dec 12, 2018

hellonico commented Dec 13, 2018

gigasquid commented Dec 14, 2018

gigasquid commented Dec 14, 2018

sandeep-krishnamurthy commented Dec 29, 2018

gigasquid commented Jan 8, 2019

stu1130 commented Jan 16, 2019

gigasquid commented Jan 18, 2019

gigasquid commented Jan 18, 2019

hellonico commented Jan 19, 2019 • edited Loading

larroy Jan 19, 2019

Choose a reason for hiding this comment

hellonico Jan 21, 2019

Choose a reason for hiding this comment

marcoabreu Jan 21, 2019

Choose a reason for hiding this comment

hellonico commented Jan 21, 2019

marcoabreu commented Jan 21, 2019

hellonico commented Jan 21, 2019

hellonico commented Jan 21, 2019 • edited Loading

Update infer test cases to run in the integration phase (simply with lein test).

hellonico commented Jan 21, 2019 • edited Loading

gigasquid commented Jan 22, 2019

gigasquid Jan 22, 2019

Choose a reason for hiding this comment

gigasquid left a comment • edited Loading

Choose a reason for hiding this comment

marcoabreu commented Jan 23, 2019

lanking520 Jan 23, 2019

Choose a reason for hiding this comment

marcoabreu Jan 23, 2019

Choose a reason for hiding this comment

hellonico commented Jan 19, 2019 •

edited

Loading

hellonico commented Jan 21, 2019 •

edited

Loading

hellonico commented Jan 21, 2019 •

edited

Loading

gigasquid left a comment •

edited

Loading