Connectivity issue to GitHub on the CI

[davido]

I'm wondering why some tests are failing on Bazel CI.

For example in:

https://buildkite.com/bazel/bazel-bazel-github-presubmit/builds/6074#52c9f049-c069-4386-8353-82ec0e0a401d

where I added java_tools release on GitHub, it is failing to fetch:

https://storage.googleapis.com/bazel-untrusted-buildkite-artifacts/52c9f049-c069-4386-8353-82ec0e0a401d/src/test/shell/bazel/workspace_resolved_test/attempt_1.log

INFO: Repository remote_java_tools_linux_beta instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_buildkite/c3a616e1648c5e14a8ab09d0d59696c2/sandbox/darwin-sandbox/3279/execroot/io_bazel/_tmp/58d272c7f3dd803b2bcb2fc7be47d391/root/fb8b458bcc92813a6fcf57a0dbe6e8bd/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
WARNING: Download from https://github.com/davido/java_tools/releases/download/javac11-v11.0/java_tools_javac11_linux-v11.0.zip failed: class java.net.ConnectException Operation not permitted (connect failed)
ERROR: An error occurred during the fetch of repository 'remote_java_tools_linux_beta':
   java.io.IOException: Error downloading [https://github.com/davido/java_tools/releases/download/javac11-v11.0/java_tools_javac11_linux-v11.0.zip] to /private/var/tmp/_bazel_buildkite/c3a616e1648c5e14a8ab09d0d59696c2/sandbox/darwin-sandbox/3279/execroot/io_bazel/_tmp/58d272c7f3dd803b2bcb2fc7be47d391/root/fb8b458bcc92813a6fcf57a0dbe6e8bd/external/remote_java_tools_linux_beta/java_tools_javac11_linux-v11.0.zip: Operation not permitted (connect failed)
INFO: Repository remote_java_tools_darwin_beta instantiated at:
  no stack (--record_rule_instantiation_callstack not enabled)
Repository rule http_archive defined at:
  /private/var/tmp/_bazel_buildkite/c3a616e1648c5e14a8ab09d0d59696c2/sandbox/darwin-sandbox/3279/execroot/io_bazel/_tmp/58d272c7f3dd803b2bcb2fc7be47d391/root/fb8b458bcc92813a6fcf57a0dbe6e8bd/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
WARNING: Download from https://github.com/davido/java_tools/releases/download/javac11-v11.0/java_tools_javac11_darwin-v11.0.zip failed: class java.net.ConnectException Operation not permitted (connect failed)
ERROR: An error occurred during the fetch of repository 'remote_java_tools_darwin_beta':
   java.io.IOException: Error downloading [https://github.com/davido/java_tools/releases/download/javac11-v11.0/java_tools_javac11_darwin-v11.0.zip] to /private/var/tmp/_bazel_buildkite/c3a616e1648c5e14a8ab09d0d59696c2/sandbox/darwin-sandbox/3279/execroot/io_bazel/_tmp/58d272c7f3dd803b2bcb2fc7be47d391/root/fb8b458bcc92813a6fcf57a0dbe6e8bd/external/remote_java_tools_darwin_beta/java_tools_javac11_darwin-v11.0.zip: Operation not permitted (connect failed)
ERROR: java.io.IOException: Error downloading [https://github.com/davido/java_tools/releases/download/javac11-v11.0/java_tools_javac11_darwin-v11.0.zip] to /private/var/tmp/_bazel_buildkite/c3a616e1648c5e14a8ab09d0d59696c2/sandbox/darwin-sandbox/3279/execroot/io_bazel/_tmp/58d272c7f3dd803b2bcb2fc7be47d391/root/fb8b458bcc92813a6fcf57a0dbe6e8bd/external/remote_java_tools_darwin_beta/java_tools_javac11_darwin-v11.0.zip: Operation not permitted (connect failed)

What am I missing? Should this dependency be added to the bazel mirror?

//CC @meteorcloudy @philwo

[philwo]

@davido We run most tests on CI without network access and instead inject the external repositories from the outside. This saves a lot of network traffic and I/O (because the Bazel integration tests don't have to extract the repository archives again and again).

You can look at this PR where I explained yesterday to someone else how to fix test failures like this: bazelbuild/bazel#11436

[philwo]

I added an FAQ section to our docs and this question there now: https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md#my-tests-fail-on-bazel-ci-due-to-error-downloading :)

[davido]

Thanks @philwo for your help, code pointers and FAQ. Will adapt my PR correspondingly ;-)

Some verification jobs for my PR were successful. Why would that be the case? Is connectivity only disabled for some jobs, while the other jobs are still fetching the data over the wire? Should this be fixed to be consistent? And this issue could be re-opened with the changed title: "Consistently disable connectivity for all CI Bazel jobs"?

[davido]

Unrelated to this issue, but one crucial link description is missing in the documentation:

https://github.com/bazelbuild/continuous-integration/blob/master/buildkite/README.md#useful-links

If one test is failing, then the log can be downloaded from the buildkite site. It took me quite some time to figure this out, under the section artifacts, ... This would be very usefel addition to the documentation.

[philwo]

Some verification jobs for my PR were successful. Why would that be the case? Is connectivity only disabled for some jobs, while the other jobs are still fetching the data over the wire?

Yes, we cannot inject these repositories when running the tests on RBE, so tests running on that platform will fetch external repositories over the internet as before.

I think it's actually good to have both kind of tests - e.g. we get the speed-up and hermeticity testing on non-RBE platforms via injected repos, but we also test that without all that injection magic the tests still work fine (or fail when they should) on RBE.

[philwo]

If one test is failing, then the log can be downloaded from the buildkite site. It took me quite some time to figure this out, under the section artifacts, ... This would be very usefel addition to the documentation.

Thanks for the suggestion. Fixed, see the new page: https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#useful-links :)

[davido]

Yes, we cannot inject these repositories when running the tests on RBE, so tests running on that platform will fetch external repositories over the internet as before.

Thank for clarifying.

Thanks for the suggestion. Fixed, see the new page: https://github.com/bazelbuild/continuous-integration/tree/master/buildkite#useful-links :)

Great! Thanks ;-)