Azure build jobs not being triggered

[mfansler]

Pull Requests in some feedstocks are not triggering Azure build jobs. This has been observed on both staged-recipes and individual feedstocks since at least 27-Oct 5:43 AM GMT. Restarting CI (close-open) does not resolve the issue. Travis CI jobs do appear to get triggered. Some feedstocks additionally trigger an (unusual) Appveyor job that fails.

Maintainers are reminded to double-check that "passing" checks on PRs include build jobs before considering merging.

It is hypothesized that this is related to an Azure wide issue. Please see https://status.dev.azure.com/_event/559260722.

[danielnachun]

I'll note here that I was able to bypass the restart CI issues by simply making an empty commit for a while, but that doesn't seem to work now.

Furthermore, for a while when restarting CI, Azure jobs would still get triggered but fail almost instantly due to what I believe was a "git status 128" error. I saw this happen sporadically once in staged-recipes but it went away after restarting CI. I'm not sure if these issues are connected to or informative for the current issue. Hopefully this gets resolved soon! Azure itself reports no outages right now.

[mfansler]

Fortunately, bot-automerge doesn't seem to proceed. 😅 Perhaps maintainers should be advised to convert affected PRs to "Draft", to guard against accidental untested merges?

[mfansler]

NB: Moving these examples out of the notification post since they were taking up Status page real estate.

Example PRs

• Add r-bshazard staged-recipes#28024
• [bot-automerge] Rebuild for r-base 4.4 and UCRT mingw toolchain r-startr-feedstock#18
• [bot-automerge] r-mlr3tuning v1.1.0 r-mlr3tuning-feedstock#30
• [bot-automerge] r-memisc v0.99.31.8.1 r-memisc-feedstock#8
• Fix include path r-strawr-feedstock#6
• Enable windows r-cairo-feedstock#41
• Enable windows build r-primme-feedstock#19
• MNT: rerender qt-wayland-feedstock#33
• MNT: rerender csp-feedstock#45
• [bot-automerge] r-hmisc v5.2_0 r-hmisc-feedstock#38
• [bot-automerge] polars v1.12.0 polars-feedstock#281

Feel free to edit.

[lesteve]

Just to say that a similar behaviour has been seen in scikit-learn and joblib so this is likely not conda-forge specific.

[jdblischak]

this is likely not conda-forge specific

Definitely not conda-forge specific. There is a known incident being tracked at https://status.dev.azure.com/_event/559260722

[xylar]

Meanwhile, users can try triggering the pipelines manually.

Sadly not an option for most maintainers here.

[h-vetinari]

Sadly not an option for most maintainers here.

And even if that were an option, we shouldn't use it. That's because triggering a run in the Azure Pipelines UI for commit from a PR has a big footgun built-in, in the sense that it would run as if it had been merged, in other words, packages would get published.

[xylar]

Oh, wow, good that it's not an option for maintainers, then!

[hmaarrfk]

Hmm so an azure build takes at least 2 hours!

[mfansler]

The hotfix has been rolled out as of 29-Oct 07:40 AM GMT. However, I have not yet been able to confirm a successful build. Close-Reopen is occasionally generating aberrant(?) AppVeyor jobs. Also, seeing failure to fetch the feedstock PRs, e.g.:

fatal: couldn't find remote ref refs/pull/30/merge
##[error]Git fetch failed with exit code: 128

[beckermr]

The appveyor jobs are unrelated.

[mfansler]

✅ Affected bot PRs marked bot-rerun come back working now.
✅ Affected staged-recipes PRs are working after merge from main.
❓ Other PRs may work again after pushing a commit (e.g., empty commit).

Seems like things are working again after the PRs "refresh" in some way.

[beckermr]

Great. I am going to close this one since Azure has marked it as done as well.