Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

OSX 10.15 images deprecated in azure, to be removed end of August #1798

Closed
h-vetinari opened this issue Jul 15, 2022 · 35 comments
Closed

OSX 10.15 images deprecated in azure, to be removed end of August #1798

h-vetinari opened this issue Jul 15, 2022 · 35 comments

Comments

@h-vetinari
Copy link
Member

h-vetinari commented Jul 15, 2022

Message started appearing in OSX jobs:

The macOS-10.15 environment is deprecated, consider switching to macos-11(macos-latest),
macos-12 instead. For more details see https://github.com/actions/virtual-environments/issues/5583

Quoting from the upstream issue:

Deprecation will begin on 5/31/2022 and the image will be fully unsupported by 8/30/2022

To raise awareness of the upcoming removal, we will temporarily fail jobs using macOS 10.15. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:

GitHub Actions:

  • July 21, 12:00 UTC - July 22, 18:00 UTC
  • July 27, 00:00 UTC - July 28, 00:00 UTC
  • August 3, 00:00 UTC - August 4, 00:00 UTC
  • August 15, 00:00 UTC - August 16, 00:00 UTC
  • August 26, 00:00 UTC - August 27, 00:00 UTC
  • August 29, 00:00 UTC - August 30, 00:00 UTC

Azure DevOps

  • August 1, 12:00 UTC - August 1, 14:00 UTC
  • August 9, 16:00 UTC - August 9, 18:00 UTC
  • August 17, 14:00 UTC - August 17, 18:00 UTC
  • August 25, 14:00 UTC - August 25, 20:00 UTC
  • September 2, 12:00 UTC - September 2, 22:00 UTC
  • September 8, 00:00 UTC - September 9, 00:00 UTC
  • September 14, 00:00 UTC - September 15, 00:00 UTC
  • September 20, 00:00 UTC - September 21, 00:00 UTC
  • September 26, 00:00 UTC - September 27, 00:00 UTC

Those brownouts get pretty extensive - 24h for all but the first on GHA, and the same a bit later on Azure - so it sounds we need to move until end of July with this.

@jakirkham
Copy link
Member

cc @conda-forge/core

@beckermr
Copy link
Member

Is this for azure or GitHub actions or both?

@h-vetinari
Copy link
Member Author

Is this for azure or GitHub actions or both?

The brownouts mentioned in actions/runner-images#5583 and the OP are for GHAs, and doesn't give a target date for azure, but the warning that caused me to open this issue was on azure

@ngam
Copy link
Contributor

ngam commented Jul 16, 2022

azure

It's the same for azure. Some builds will get this warnings, e.g. this one,

The macOS-10.15 environment is deprecated, consider switching to macos-11(macos-latest), macos-12 instead. For more details see actions/runner-images#5583

@h-vetinari
Copy link
Member Author

I just checked, the oldest SDK that macos-11 comes with is 10.15, plus xcode 11.7. As far as I can tell, that version still supports deployment target 10.9 (actually 10.6-10.15), so it doesn't sound like the image change forces us to change anything substantial about MacOS support 1

Footnotes

  1. It's worth noting though that 10.9 is EOL for more than 5 years already, but moving on from that is a separate discussion.

@beckermr
Copy link
Member

Thank you!

@chrisburr had some nice stats about where we go next on osx pulled from pypi usage. I forget but moving to somewhere around 10.12 would be pretty non-destructive IIRC.

@jakirkham
Copy link
Member

Would suggest we separate discussing the CI image change from updating the SDK used to build (if possible). Given CI is the primary driver of discussion here, let's focus on that. If we would like to discuss SDK changes, let's open a new issue (unless these are somehow intertwined).

In terms of CI image used (the primary concern here), moving macOS 11 is the next oldest thing we can use (as 10.15 is the last in the 10.x line).

@h-vetinari
Copy link
Member Author

h-vetinari commented Jul 21, 2022

Would suggest we separate discussing the CI image change from updating the SDK used to build (if possible).

AFAIU, the SDK is not really the issue, the MACOS_DEPLOYMENT_TARGET specifies the compatibility. That said, I had assumed that we were using one of the SDK's in the current 10.15 image, but that's not the case, because we're still using the 10.9 SDK, but I couldn't easily find (e.g. in smithy) where that is injected.

So with that misunderstanding out of the way - fine by me to leave the SDK untouched (assuming the current SDK injection mechanism works in the macos-11 image).

@jakirkham
Copy link
Member

All good. Just trying to help dial things in a bit so we can make decisions 🙂

@jaimergp
Copy link
Member

So we have roughly a month to:

  1. Update this block to macos-11 as soon as possible
  2. Hope a month is enough for people to rerender in time
  3. In September, remember users they still need to rerender to re-enable OSX builds

Does that sound correct?

@beckermr
Copy link
Member

We have not historically worried about rerendering in these cases. People should rerender almost always when making new PRs.

@beckermr
Copy link
Member

So we only need a PR to smithy.

@jaimergp
Copy link
Member

And we don't have a reason to suspect that the current way we are installing SDKs on the image would happen to fail on MacOS 11, right? I guess we can only try... I'll open a PR on conda-smithy to see what happens.

@jaimergp
Copy link
Member

Let's see: conda-forge/dav1d-feedstock#2

@jakirkham
Copy link
Member

Don't think so, but agree a test is a better way to check

@jaimergp
Copy link
Member

Everything looks ok:

   INFO: sysroot: '/Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/' files: '['usr/share/man/mann/zero.n', 'usr/share/man/mann/yencode.n', 'usr/share/man/mann/yaml.n', 'usr/share/man/mann/xsxp.n']'
   INFO (dav1d,bin/dav1d): lib/libdav1d.6.dylib found in this package
   INFO (dav1d,bin/dav1d): Needed DSO /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/libSystem.B.dylib found in $SYSROOT
   INFO (dav1d,lib/libdav1d.6.dylib): Needed DSO /Applications/Xcode_13.2.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk/usr/lib/libSystem.B.dylib found in $SYSROOT

@jaimergp
Copy link
Member

image

@jakirkham
Copy link
Member

Great! Thanks Jaime 😄

Sounds like we are ready for a conda-smithy PR & release. If we get it out today, we will have ~1 week to try things and catch/fix any issues 🙂

@jaimergp
Copy link
Member

We have til August 30th, don't we? Not that I have anything against doing it soon, but it's not as rushed as a week :)

@h-vetinari
Copy link
Member Author

h-vetinari commented Jul 22, 2022

We have til August 30th, don't we? Not that I have anything against doing it soon, but it's not as rushed as a week :)

Well the five 24h-long brown outs from July 27th until end of August sound pretty painful. During that time no OSX pipeline in conda-forge would run through.

@jaimergp
Copy link
Member

Ah, hadn't considered the brown-outs. I'll add the PR to conda-smithy, then.

@jakirkham
Copy link
Member

Ah sorry. Was thinking beginning of August for some reason 🤦‍♂️

Yeah brownouts are an issue too though

Thanks Jaime! 🙏

@jaimergp
Copy link
Member

Just realized we also need to do this for staged-recipes. Submitted PR here: conda-forge/staged-recipes#19800

After that, I think we can consider this solved and closed.

@ngam
Copy link
Contributor

ngam commented Jul 25, 2022

And we don't have a reason to suspect that the current way we are installing SDKs on the image would happen to fail on MacOS 11, right? I guess we can only try... I'll open a PR on conda-smithy to see what happens.

No, as you figured (we also have used variations of macos-11 and macos-12 in different feedstocks, one could set this in conda-forge.yml, e.g. see this PR from a while back: #1638 and it works fines).

However, it may be worth considering trying to better understand the SDKs situation. The website/repo we get stuff from is no longer maintained, and it doesn't have many of the newer SDKs. Fortunately, for now, most of these newer SDKs come with the images anyway so it's not a big deal. However, it would likely be best to think this through again and decide how best to proceed (i.e. use another repo as shown here: conda-forge/conda-forge-ci-setup-feedstock#201 or rely entirely on what the vmImages offer for example this is what macos-11 has: https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md#installed-sdks)

@ngam
Copy link
Contributor

ngam commented Jul 25, 2022

(unless these are somehow intertwined).

They're somewhat intertwined because the images come preconfigured with certain SDKs and that's exactly what we relied on in the pytorch feedstock to support MPS (metal). But yes, this should be more thoroughly discussed elsewhere. Just wanted to point out this was on the radar before and highlight how the SDKs and vmImages relate to a degree.

After that, I think we can consider this solved and closed.

Yes, from my previous testing, this is solved and closed after the two PRs

@jaimergp
Copy link
Member

Thanks @ngam!

@ngam
Copy link
Contributor

ngam commented Jul 25, 2022

Thank YOU for the PRs 😃

@ngam
Copy link
Contributor

ngam commented Jul 25, 2022

FYI @beckermr the change may take a bit to come online with conda-smithy (hence rerendering may not effect changes yet) because the CDN cloning has a "major outage"... https://conda-forge.org/status/

@ngam
Copy link
Contributor

ngam commented Jul 25, 2022

Also, before this is closed, we need to push a release of conda-smithy, I believe. Otherwise, the changes won't take place.

@h-vetinari
Copy link
Member Author

We're now in the first brownout on azure (dates of which have been added to the upstream issue, as well as the OP).

Could we have a new smithy release?

@ngam
Copy link
Contributor

ngam commented Aug 1, 2022

Could we have a new smithy release?

@conda-forge/core @conda-forge/conda-smithy

@h-vetinari
Copy link
Member Author

Felipe commented in conda-forge/conda-smithy#1645

I guess we need a new release with this ASAP.

To take the heat off a bit - it's not a crisis. The brownout is only two hours this time. But since that smithy PR has been merged (and I assume the repo is in a releasable state) it would still be good to get a new smithy release soon (i.e. before the next brownout in a week). It'll take a while anyway until feedstocks have been rerendered in large numbers.

@beckermr
Copy link
Member

beckermr commented Aug 1, 2022

release is done - should be live in an hour or so

@h-vetinari
Copy link
Member Author

With smithy & staged-recipes now done, I think we can close this. Thanks all! :)

@jakirkham
Copy link
Member

JFYI we recently saw an (admittedly somewhat niche) issue with Python 3.7 on macOS 11 ( conda-forge/python-feedstock#575 ). Mentioning here merely for awareness.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

No branches or pull requests

5 participants