curl: (7) Failed to connect to more.musl.cc port 443: Connection timed out

[masnagam]

Hi,

I'm facing an issue that curl fails:
https://github.com/mirakc/mirakc/runs/7612735719?check_suite_focus=true#step:8:2450

The execution of the workflow was scheduled by on.schedule.cron:
https://github.com/mirakc/mirakc/blob/main/.github/workflows/daily-main.yml#L7

and ran on ubuntu-latest.

A PAT was used in order to trigger other workflows:
https://github.com/mirakc/mirakc/blob/main/.github/workflows/daily-main.yml#L12

The docker job tried to build a Docker image by using docker/build-push-action@v3.
buildenv.sh was executed and tried to download a toolchain from more.musl.cc by using curl, but the connection timed out.

You can see verbose logs from curl in the following page:
https://github.com/mirakc/mirakc/runs/7598905955?check_suite_focus=true#step:8:8956

I confirmed that:

• Another workflow built Docker images successfully, which was triggered by a push event
• Docker images can be built successfully on my local Linux PC by using a command described in README.md

Are there any mistakes in my scripts and/or the workflow definition?

Thank you for your help.

[gabek]

Same issue here.

      +crosscompiler | Connecting to musl.cc (musl.cc)|104.232.42.245|:443... failed: Operation timed out.

Some detail: It's not specifically curl, it's any connection to that host. For a few days it was failing intermittently but now it's consistently failing.

[zv-io]

For clarity, the reason it was failing "intermittently" is because I was shutting the web server down during huge traffic spikes (see https://github.com/orgs/community/discussions/27906#discussioncomment-3332440) just for a few minutes at a time to let the network recover.

Then, after this occurred several days in a row and again a week later, I blocked the Microsoft IP ranges.

This pattern should be rewritten: https://github.com/mirakc/mirakc/blob/b2d25a350da3877e511876c62f774f1b39a6e869/docker/build-scripts/buildenv.sh#L17-L20

[masnagam]

@zv-io
Thank you for your suggestion. I'll improve the script.
Currently, I'm considering to use a pre-built Docker image as a build environment which contains toolchains needed for cross compilations.

[airtower-luna]

Timeout without any connection makes me think of some kind of firewall, that's usually the cause of something like this. Your workflow seems to be downloading other data just fine, so it's probably not the Actions infrastructure in general.

Theoretically it could be anywhere along the route (I've seen issues with firewalls eating "fragmentation needed" packets), but I'm wondering if maybe there's some kind of rate-limiting involved that's enforced by firewall. I wonder if you could contact the admins of musl.cc and ask if they have any such rules, and if yes if there's any overlap between them and the IP ranges the GitHub API reports for Actions? 😅

[gabek]

I went ahead and emailed musl.cc and I'll report if I hear back!

[masnagam]

@airtower-luna @gabek Thank you for your help :)

[jdevfullstack]

hey guys, I am using cURL directly on GitHub Codespaces without anything being installed ! ( You also have there by default Git, Bash, Java etc. ) That is awesome ! and unlike local installation like this that can interfere with the functionalities, you don't worry about that using GitHub Codespaces,

anyways, it's a remote Linux machine but it's really that convenient

[gabek]

I'm not terribly knowledgeable about codespaces, but could you clarify how that is related to our GitHub actions issue? Thanks for chiming in!

[jdevfullstack]

I just saw above about the firewall and local setup that might interfere

that's why I mentioned about cURL and Codespaces

can you run that command separately ? manually doing in cURL ? just to be sure whehter it's working or not

[zv-io]

Hi, @gabek did email me. Thanks for reaching out and creating a discussion about this.

Currently, most Microsoft IP ranges are blocked from accessing the site. This is due to some careless, though probably unintentional, DDoS abuse. I'm happy to unblock these ranges but it's going to have to wait until September 01 for unrelated reasons.

At times, up to ~2k IP addresses had been hammering the site at the same time downloading the x86_64 and aarch64 toolchains over and over, all reporting some form of the curl user agent. I think so far approximately 30k unique Microsoft IP addresses had been used.

It wasn't clear to me initially whether this was a Microsoft-internal service deployment, some private Azure accounts, or GitHub Actions.

On my end, not just my site, but the upstream ISP, went offline or had service degradation that affected VoIP connections. This could potentially affect access to emergency services, so we've shaped our QoS policy to at least prevent that.

So I wholesale blocked most of the Microsoft IP ranges I could find. Sorry.

My position is this: if these toolchains (and maintenance, including new releases, more testing, etc.) are important to the GitHub community, I'd be happy to work out some sort of sponsored arrangement to prioritize this work, have more robust hosting (CDN or otherwise), and allow everyone to access these toolchains at full speed (10Gbps instead of 100Mbps).

There is absolutely no reason that these toolchains need to be downloaded from the upstream server every single time CI is run. Let's not forget that bandwidth isn't unlimited, free, etc.

[gabek]

This makes a lot of sense, thanks for clarifying. Personally I was downloading once a day during a nightly build not knowing I was adding to a growing problem for you. I'll certainly make changes to not have to rely on your server anymore.