[Temporary] Multi-GPU predictor

[trivialfis]

Please ignore this PR. I'm attempting to resolve the bug in #3738 . Creating a temporary PR allows me to peek at Jenkins without polluting @canonizer 's branch.

[hcho3]

@trivialfis FYI, now that you are a committer, you have ability to interrupt any CI jobs as well. This may be helpful when you run into executor starvation on Jenkins.

[trivialfis]

@hcho3 I looked around the Jenkins interface, only relevant button I saw is "Restart Jenkins: Build & Test". Which, doens't do anything.

[hcho3]

@trivialfis Look for the square stop icon on the top right corner:

[hcho3]

Also, current setup involves 3 re-tries, so you may have to press the stop button multiple times.

[trivialfis]

@hcho3

[hcho3]

@trivialfis Ugh, I thought all committers had admin rights on Jenkins, but it doesn't seem to be the case. Let me fix this.

[trivialfis]

@hcho3 Thanks, that will be a great help!

[hcho3]

@trivialfis Looks like I had to manually add you to the admin list. Try it again.

[hcho3]

Actually, with a recent Jenkins update, the Stop button now properly works with the retry block. Pressing it once should stop all jobs.

[trivialfis]

@hcho3 I tried it again, it's "HTTP ERROR 404" now.

[hcho3]

Can you access the page https://xgboost-ci.net/blue/organizations/jenkins/xgboost/detail/PR-3819/4/pipeline/ ?

[trivialfis]

@hcho3 Yes, this one works.

[trivialfis]

@hcho3 But still no access permission (the stop button isn't there).

[hcho3]

Try stopping it and see if it works.

[trivialfis]

@hcho3 Oh, after a refresh now I see it. It works. Thanks! :)

[hcho3]

Good to hear that. You are now able to stop and re-start any CI jobs. Now let's see if we can figure out why multi-GPU test is failing.

[trivialfis]

@hcho3 I'm able to stop it, but not restart it. You can see from the log a java exception "java.lang.NullPointerException: Cannot invoke method getBuildName() on null object
" is thrown after restart. But I will trigger it by amend commit so it doesn't really matter.

Thanks a lot! Let me try to figure something out.

[hcho3]

@trivialfis That's curious. For now, let's focus on the multi-GPU test. Jenkins setup is relatively new, and it can be improved in the future.

[hcho3]

@trivialfis FYI, there are two buttons for restart. Make sure to use the one on the top (without text), not the one at bottom with text. I don't know why, but the bottom one never worked for me.

[trivialfis]

@hcho3 Got it.

[trivialfis]

@hcho3 Can appvayor be cancelled?

[hcho3]

@trivialfis Yes. Do you see this button on the top right?

If not, your account is not yet under the tqchen group. For this, we'll have to reach out to Tianqi.

[hcho3]

In the long run, I'm looking to migrate all Windows tests to Jenkins. You can run many tests in parallel, and each worker can be customized.

[trivialfis]

@hcho3 After logging out and log back, I can cancel the appvayor now. :)

That's a good plan, I am also thinking about if we can reduce or combine some expensive tests. Will discuss about it later on.

[trivialfis]

@hcho3 Last commit in this PR solved the problem. I will push it to original PR tomorrow. Thanks for all the help!

[hcho3]

@trivialfis No problem. BTW, does your fix work when n_row = 8? Otherwise we should raise an error whenever [number of data row] < [number of GPUs]. Alternatively, we can proactively change number of GPUs to fit the number of data rows.

[trivialfis]

@hcho3 Good point. Let me keep looking.

[hcho3]

@trivialfis I found out that Jenkins was showing 404 for anonymous users. I fixed the configuration to address this.

[trivialfis]

@hcho3 Glad to hear that. :)