macOS-10.15 agent uses Homebrew 3.0.10 which uses Bintray

[thisisthekap]

Description

According to Homebrew/discussions#691, Homebrew switched from Bintray to GitHub Packages. Homebrew 3.1.0 uses the new CDN. The Microsoft Hosted Agents should imho be updated accordingly to avoid issues when downloading/installing packages using Homebrew.

Area for Triage:
Artifacts, Packages, Servers

Question, Bug, or Feature?:
Bug

Virtual environments affected

• Ubuntu 16.04
• Ubuntu 18.04
• Ubuntu 20.04
• macOS 10.15
• macOS 11.0
• Windows Server 2016 R2
• Windows Server 2019

[maxim-lobanov]

Hello @thisisthekap , I think nothing is required from our side to switch CDN.
As I can see, 5 hours ago the new version of brew with support of new CDN was released: https://github.com/Homebrew/brew/releases/tag/3.1.0

We will update Homebrew to this version with new image rollout (approx middle of next week since this week rollout has been already started).

[thisisthekap]

@maxim-lobanov Thank you for the quick response!

Updating Homebrew to 3.1.0 is indeed the solution I was hoping for :)

Once again: Thank you very much 😎

[jsonfry]

Right now bintray have started what they're calling a 'brown out' https://status.bintray.com/incidents/4mzph0y8l6hk
but actually means they're failing all requests to bintray for the next few hours, so any brew installs in macos runners are failing / will fail.

Did you have to update brew as part of your ci job?

[thisisthekap]

I just execute brew update prior to the needed brew install call to make sure Homebrew is on the lastest version.

[jsonfry]

Hmm not ideal.

@maxim-lobanov I guess there's no chance of brew being updated before bintray finished their thing in about 6 hours?

[jonathanhood]

Some kind of guidance, or a notification of some kind on the actions status page, may be necessary. It took me, at least, a good portion of the morning chasing down links to find myself here. Such a notification, email, or whatever could save a lot of people a lot of time this morning - especially since the brownout doesn't end until 2pm pacific.

[ColinPitrat]

I'm seeing some "403 forbidden" when executing brew install today from GitHub Actions. Is this linked to this issue?

Adding logs to help discoverability of this issue:

Run brew install sdl2
==> Downloading https://homebrew.bintray.com/bottles/sdl2-2.0.14_1.catalina.bottle.tar.gz
curl: (22) The requested URL returned error: 403 Forbidden
Error: Failed to download resource "sdl2"

The workaround is to run brew update before brew install:

      - run: brew update
      - run: brew install sdl2

[jonathanhood]

Yes. To recap the issue:

1. Bintray is issuing 403 Forbidden errors today as part of a scheduled "brownout" in preparation for their take down of the service.
2. A version of brew is available that uses github packages instead. This version is not currently installed by default on the mac actions runners.

While we wait for the runners to be updated, you can do brew update manually in your github actions workflow and sidestep the issue.

[mathbagu]

When can we expect the runners to be updated, just to know if we should use the suggested workaround?

[miketimofeev]

@mathbagu we've decided to start the new rollout tomorrow to deliver the fix asap

[maxim-lobanov]

Issue is related to planned brown-out session on Bintray side.
Bintray is going to recover in a few hours so issue should disappear: https://status.bintray.com/incidents/4mzph0y8l6hk

With next image rollout, homebrew should stop using bintray. Updating runners take about 3-4 days usually.

[maxim-lobanov]

The deployment of new set of images has been already started and will be finished in a few days.

[miketimofeev]

@thisisthekap We have deployed macOS images with brew 3.1.1 on board! All the packages are now retrieved from GitHub Packages.
I'm going to close the issue. Please feel free to contact us if you have any concerns.
Thank you!