Consider migrating tests from AppVeyor/Travis CI to Azure Pipelines

[hcho3]

Consider migrating all tests from AppVeyor/Travis CI to Azure Pipelines.

Why migrate tests?

• OSX tests on Travis CI are quite slow and have proven flaky from time to time (e.g. - fix issues with training with external memory on cpu #4487, add support for training on gpu with external memory #4448).
• AppVeyor allows only a single job to run at a time; Azure Pipelines allows for 10 parallel jobs
• Azure Pipelines can run both Windows and Mac OSX tests (note: no GPUs.)
• Azure Pipelines is provided free of charge to open-source projects

TODO. Test Azure Pipelines for performance and stability

cc @sriramch

[CodingCat]

+10000 on getting rid of AppVeyor

[hcho3]

I played with Azure Pipelines a bit, and so far it's been pleasure to work with.

• Builds start almost instantaneously and run fast.
• In a single YAML file, I can use Linux, Mac OSX, and Windows agents.
• Jobs run in parallel, but some jobs can be made dependent on other jobs.

See example: azure-pipelines.yml, build result

[trivialfis]

Can't wait.

[trams]

I am also interested in seeing Azure Pipelines instead of AppVeyor

[hcho3]

We've set up GitHub Actions (#5917) and moved some tests from AppVeyor to GitHub Actions (#5922). There is currently a nondeterministic test that's blocking a further migration away from AppVeyor.

We are keeping Travis CI because we need access to MacOS High Sierra to build XGBoost Python wheel. (Expectation is that it will be compatible with last three major releases, High Sierra, Mojave, and Catalina.) GitHub Actions only provides MacOS Catalina.

Closing this in favor of #5891.

[trivialfis]

@hcho3 I don't think we are running 3 versions of osx on travis right?

[trivialfis]

Another thing is, our test on Travis now takes 1.5+ hour to run for each PR. And multiple PRs cannot be run in parallel.

[hcho3]

And multiple PRs cannot be run in parallel.

Travis CI runs multiple Linux jobs just fine. It's just that their MacOS agent pool is really small, so our jobs get stuck in the global queue for a long time.

[hcho3]

I don't think we are running 3 versions of osx on travis right?

No. We assume backward compatibility: if we can build and run XGBoost on MacOS High Sierra, then the generated binary should also be compatible with MacOS Mojave/Catalina.

[trivialfis]

Got it. Thanks for the explanation.

[hcho3]

Reopening, since Travis CI seems to be quite slow nowadays. The tests remain in queue for 2-3 hours. (Once they start, they are usually fairly quick.)

A proposal: Migrate all MacOS tests to GitHub Actions. GitHub Actions will use MacOS Catalina (latest). Travis CI should only run on the master branch, to generate and upload nightly builds (Python wheels) with MacOS High Sierra.
Pros: We (developers) no longer get blocked waiting for test results.
Cons: Since the release is still using MacOS High Sierra, we may end up writing code that works in MacOS Catalina but fails in High Sierra. Hopefully this will be a rare occasion. We would need to monitor the test result of the master branch on Travis CI (MacOS High Sierra) and try to fix it if it breaks.

@trivialfis WDYT?

[trivialfis]

@hcho3 I'm not sure about the ecosystem on osx. I will leave it to others to decide. ;-) @CodingCat seems also fond of osx.

[CodingCat]

I think the backward compatibility of osx is generally guaranteed....

…

On Wed, Jul 22, 2020 at 6:43 PM Jiaming Yuan ***@***.***> wrote: @hcho3 <https://github.com/hcho3> I'm not sure about the ecosystem on osx. I will leave it to others to decide. ;-) @CodingCat <https://github.com/CodingCat> seems also fond of osx. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4498 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFFQ6HUV5ESUKI2WKT5TA3R46ITJANCNFSM4HPSVYFA> .

[hcho3]

Travis CI seems fine now.