Add Azure Pipelines configuration

[brcrista]

As an open-source project, Pip has some free concurrent jobs to use with Azure Pipelines. I've been working with @pfmoore and @dstufft to set up build pipelines for Pip:

1. Linux
2. Windows
3. macOS

Build results can be viewed at https://dev.azure.com/pypa/pip/_build.

[brcrista]

Since this doesn't add any user-facing functionality, should I add a news entry?

[dstufft]

Nah

[benoit-pierre]

Is there a point in duplicating Linux/Windows tests on another infrastructure?

[dstufft]

If the VSTS integrate works well, we might end up dropping the other infrastructure (Appveyor for instance tends to have no concurrency and is a lot slower). They can also test different Linux OSs or whatever.

[pfmoore]

Personally, I'd be against dropping Travis (it's the standard open source CI and familiar to basically everyone). I'd be willing to drop Appveyor if VSTS proves successful, because it's so slow, but even there I'd prefer to simply make it an advisory test rather than dropping it totally, again for reasons of familiarity for contributors).

[pfmoore]

Regarding "is there a point duplicating", I don't know. Does VSTS use a different Linux distro than Travis does? It'd be interesting to know if it did (or could), just for better coverage.

[brcrista]

@pfmoore I believe Travis provides 12.04 and 14.04 while Pipelines has 16.04.

• https://docs.travis-ci.com/user/reference/overview/#virtualisation-environment-vs-operating-system
• https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/hosted?view=vsts&tabs=yaml#use-a-microsoft-hosted-agent

[pfmoore]

@brcrista I guess that's Ubuntu - so the same distro, just a different version? Oh well...

[benoit-pierre]

The distribution does not really matter anyway, just the Python version, since tox is used, no?

[pradyunsg]

I had independently stumbled upon this a week back and I'm glad that someone made the time to look into this.

The Windows build times are definitely better.

[pradyunsg]

As for duplication, I feel that it's probably fine, as long as there's some utility to each. (contributor familiarity, platform/OS coverage etc)

I won't mind just having Azure Pipelines being the only required check for PRs, if it's covering the same things as our current setup and is overall faster than Appveyor.

[benoit-pierre]

Is there support for rolling builds (AppVeyor) / auto-cancelation (Travis)?

[brcrista]

@benoit-pierre there's support for scheduled builds. Queued PR builds should get cancelled when new changes are pushed

[dstufft]

I'm inclined to merge this to at least get some real world experience with it. We can always revert it if needed. Any objections @pypa/pip-committers ?

[pradyunsg]

Any objections @pypa/pip-committers ?

Not from me.

[pfmoore]

Any objections @pypa/pip-committers ?

Nor from me.

[pfmoore]

@benoit-pierre there's support for scheduled builds.

However, from the linked page (and previously discussed, although I believe only on private email)

Someone must view a page in your Azure DevOps organization regularly for CI and scheduled builds to run.

I'm -1 on having anything to do with scheduled builds until this limitation is removed. Open source projects simply don't have the sort of regular activity, let alone routine admin activity, that makes this a viable situation. (My understanding is that the comment there about CI builds is simply inaccurate - CI builds will run even if no-one ever visits the VSTS admin pages).

[dstufft]

I think that was just a confusion, Rolling Builds are what AppVeyor calls the thing where if the same PR/branch has 2 builds queued for different commits, it will only bother to test the latest commit for that PR/branch, much like the auto-cancellation feature in Travis.

I think @brcrista just assumed from the name that it was a scheduling feature, and not an auto-cancellation feature.

[pradyunsg]

I don't think this would cause any issues with prepping for 18.1. Let's merge this?

[dstufft]

Tried to delete it, but it said some stuff about being retained by a release, which I don't currently have time to dig too far into. If you want to delete that Docs Pipeline go for it.

[brcrista]

Sure, I'm on it

[dstufft]

Ok, this checks integration is cool.

[dstufft]

https://github.com/pypa/pip/pull/6027/checks?check_run_id=33909124 and hit "raw output" under the test that failed.

[dstufft]

Although it looks like it didn't get all of the output, it only sees, it missed the "captured stdout call" output that comes after it, which shows the actual failure was an intermittent 503 error.

[dstufft]

Oh nice, i can re-run a build right from inside the checks output too.

[pradyunsg]

actually I just learned you can map the orgs and it will propagate permissions, it just doesn't happen automatically :)

Ah great! That'll be one less thing to worry about then (like when onboarding new committers) then, once we set that up.

In other news, I've probably messed up my Microsoft accounts (seems like I've messed up which username is associated with which email and all that). It's ~2am here so I'll go to sleep now and come around to that tomorrow.

I probably won't be able to comment on much here rn, but then again, there's probably enough people on deck.

I know that Bernát (gaborbernat) set up the CI for virtualenv on Azure Pipelines, who was telling me about it during the Bloomberg Sprints, so it might be useful to take a look at that configuration. There's also like test coverage metrics and the ability to use JUnit XML reports that then are able to present test output in the Web UI (though I doubt it'd work out of the box).

Ok, this checks integration is cool.

+1

[pradyunsg]

Geez, those weren't proper sentences. Edited to actually be legible English (probably).

[pradyunsg]

Ok, this checks integration is cool.

Oh btw, Travis CI has checks integration on travis-ci.com (which is basically still free for OSS).

[pradyunsg]

^ blog post for the above: https://blog.travis-ci.com/2018-05-07-announcing-support-for-github-checks-api-on-travis-ci-com

[dstufft]

Yea, migration is just disabled at the moment.

[pradyunsg]

Yea, migration is just disabled at the moment.

Ah, I'd tried it out on a new repo.

[dstufft]

Looks like the Package job is failing on macOS, I think we just need to add stuff to https://github.com/pypa/pip/blob/master/.azure-pipelines/jobs/package.yml#L18 though.

[dstufft]

#6028 Will fix it I think.

[dstufft]

Hmm, maybe the re-run check button doesn't work with Azure?

[brcrista]

@pradyunsg, here's that test view: https://dev.azure.com/pypa/pip/_build/results?buildId=2478&view=ms.vss-test-web.test-result-details

I didn't set up coverage though.

[brcrista]

@dstufft unfortunately there's a known bug with re-run when doing a PR from a fork. We're working on a fix now

[brcrista]

If you rebuild on the Pipelines side, it should still post the status to GitHub:

[dstufft]

Cool that worked.

Not for nothing, but I already want ot disable Appveyor for being slow af.

[dstufft]

@brcrista Now that my fix is merged, can I retrigger the PRs that failed, or do they have to merge master into their branch to get the fixed pipeline yaml on their branch?

[brcrista]

When we build PRs we're looking at what's in their branch, so they'll need to merge master to take the change.

(If someone makes a change to the build YAML in a PR, you want to make sure it doesn't break.)

[pradyunsg]

Could someone add "Pradyun Gedam <pradyunsg@gmail.com>" on Azure Devops?

[brcrista]

Done

[xavfernandez]

Hello \o !
I'm interested also ^^ (

pip/AUTHORS.txt

Line 423 in 465261a

Xavier Fernandez <xav.fernandez@gmail.com>

)

[pradyunsg]

@xavfernandez Done!

@cjerdonek should be added as well, if he's interested.

[pradyunsg]

Let's have any further discussion about future enhancements, current setup issues etc at #6043.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.