Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tracking issue for spurious network failures on bots #40474

Open
alexcrichton opened this issue Mar 13, 2017 · 15 comments
Open

Tracking issue for spurious network failures on bots #40474

alexcrichton opened this issue Mar 13, 2017 · 15 comments
Labels
A-CI Area: Our Github Actions CI A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.

Comments

@alexcrichton
Copy link
Member

alexcrichton commented Mar 13, 2017 •
edited
Loading

I'm hoping to catalog network failures on the bots here just to keep an eye on the rate that they're happening as well as common failure modes. If they're common enough we should investigate solutions.

github errors

Submodule clone timeout

https://travis-ci.org/rust-lang/rust/jobs/210363616

Cloning into '/home/travis/build/rust-lang/rust/src/llvm'...

No output has been received in the last 30m0s, this potentially indicates a stalled build or something wrong with the build itself.

Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received

The build has been terminated

Fixed faiulres

Failure to fetch OpenSSL tarball

Fixed by #40545

https://travis-ci.org/rust-lang/rust/jobs/211285568

curl: (7) Failed to connect to www.openssl.org port 443: Connection refused


command did not execute successfully: "curl" "-o" "/checkout/obj/build/mipsel-unknown-linux-gnu/openssl/openssl-1.0.2k.tar.tmp" "https://www.openssl.org/source/openssl-1.0.2k.tar.gz"
expected success, got: exit code: 7

CentOS 5 urls changed

https://travis-ci.org/rust-lang/rust/jobs/218179477

hopefully fixed by #41045

CentOS 5.11 become EOL, apparently the base image failed to keep succeeding at yum update

Crates.io errors

Presumed bug in Cargo
@alexcrichton alexcrichton added the A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) label Mar 13, 2017
@eddyb eddyb mentioned this issue Mar 13, 2017
Merged
This was referenced Mar 14, 2017
Merged
Merged
Merged
alexcrichton added a commit to alexcrichton/rust that referenced this issue Mar 15, 2017
@alexcrichton
rustbuild: Retry downloads of OpenSSL source
5de0406 
We need this to compile Cargo and we download it at build time, but as like all
other network requests it has a chance of failing. This commit moves the source
of the tarball to a mirror (S3 seems semi-more-reliable most of the time) and
also wraps the download in a retry loop.

cc rust-lang#40474
@alexcrichton alexcrichton mentioned this issue Mar 15, 2017
Merged
@ishitatsuyuki
Copy link
Contributor

Travis uses GCE and I suppose Google won't have problems with their high quality networking, so I guess this is some kind of anti-DDoS triggered. One way is to mirror, and another way is to force them into build cache.

Psst... using Google Cloud Storage will make the transfers completely inside Google's Cloud and potentially decreases: latency, errors, and egress bill.

frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 17, 2017
@frewsxcv
Rollup merge of rust-lang#40545 - alexcrichton:retry-download-openssl…
870547b 
…, r=nikomatsakis

rustbuild: Retry downloads of OpenSSL source

We need this to compile Cargo and we download it at build time, but as like all
other network requests it has a chance of failing. This commit moves the source
of the tarball to a mirror (S3 seems semi-more-reliable most of the time) and
also wraps the download in a retry loop.

cc rust-lang#40474
frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 17, 2017
@frewsxcv
Rollup merge of rust-lang#40545 - alexcrichton:retry-download-openssl…
8e99263 
…, r=nikomatsakis

rustbuild: Retry downloads of OpenSSL source

We need this to compile Cargo and we download it at build time, but as like all
other network requests it has a chance of failing. This commit moves the source
of the tarball to a mirror (S3 seems semi-more-reliable most of the time) and
also wraps the download in a retry loop.

cc rust-lang#40474
frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 17, 2017
@frewsxcv
Rollup merge of rust-lang#40545 - alexcrichton:retry-download-openssl…
7e313ab 
…, r=nikomatsakis

rustbuild: Retry downloads of OpenSSL source

We need this to compile Cargo and we download it at build time, but as like all
other network requests it has a chance of failing. This commit moves the source
of the tarball to a mirror (S3 seems semi-more-reliable most of the time) and
also wraps the download in a retry loop.

cc rust-lang#40474
frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 17, 2017
@frewsxcv
Rollup merge of rust-lang#40545 - alexcrichton:retry-download-openssl…
88a1234 
…, r=nikomatsakis

rustbuild: Retry downloads of OpenSSL source

We need this to compile Cargo and we download it at build time, but as like all
other network requests it has a chance of failing. This commit moves the source
of the tarball to a mirror (S3 seems semi-more-reliable most of the time) and
also wraps the download in a retry loop.

cc rust-lang#40474
frewsxcv added a commit to frewsxcv/rust that referenced this issue Mar 17, 2017
@frewsxcv
Rollup merge of rust-lang#40545 - alexcrichton:retry-download-openssl…
b71e5f7 
…, r=nikomatsakis

rustbuild: Retry downloads of OpenSSL source

We need this to compile Cargo and we download it at build time, but as like all
other network requests it has a chance of failing. This commit moves the source
of the tarball to a mirror (S3 seems semi-more-reliable most of the time) and
also wraps the download in a retry loop.

cc rust-lang#40474
This was referenced Mar 23, 2017
Closed
Closed
Closed
Closed
@alexcrichton
Copy link
Member Author

Looks like we've got new failures reaching crates.io during the cargotest phase, and PRs to update that code to implement retries would be greatly appreciated!

@alexcrichton alexcrichton mentioned this issue Mar 25, 2017
Closed
@alexcrichton alexcrichton mentioned this issue Apr 2, 2017
Closed
This was referenced Apr 3, 2017
Merged
Closed
Closed
This was referenced Apr 3, 2017
Closed
Merged
Merged
Merged
Merged
Merged
Merged
@kennytm kennytm mentioned this issue Jun 25, 2018
Merged
This was referenced Jul 12, 2018
Merged
Merged
Merged
This was referenced Jul 27, 2018
Merged
Merged
Merged
Merged
@pietroalbini pietroalbini mentioned this issue Jul 31, 2018
Merged
This was referenced Aug 10, 2018
Merged
Merged
@nikomatsakis nikomatsakis mentioned this issue Aug 10, 2018
Merged
This was referenced Aug 11, 2018
Merged
Closed
Merged
This was referenced Aug 19, 2018
Merged
Merged
Closed
Merged
Merged
@davidtwco davidtwco mentioned this issue Sep 18, 2018
Merged
@pietroalbini pietroalbini mentioned this issue Sep 18, 2018
Merged
@QuietMisdreavus QuietMisdreavus mentioned this issue Sep 20, 2018
Merged
@wesleywiser wesleywiser mentioned this issue Sep 24, 2018
Merged
This was referenced Sep 28, 2018
Merged
Merged
Merged
@kennytm kennytm mentioned this issue Feb 11, 2019
Merged
@jieyouxu jieyouxu mentioned this issue Dec 6, 2024
Open
@jieyouxu jieyouxu added the A-CI Area: Our Github Actions CI label Dec 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
A-CI Area: Our Github Actions CI A-spurious Area: Spurious failures in builds (spuriously == for no apparent reason) C-tracking-issue Category: An issue tracking the progress of sth. like the implementation of an RFC T-infra Relevant to the infrastructure team, which will review and decide on the PR/issue.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@durka @alexcrichton @carols10cents @frewsxcv @aidanhs @pietroalbini @Mark-Simulacrum @ishitatsuyuki @bjorn3 @jieyouxu