netrc doesn't work with UrlRewriter

[denyska]

Description of the problem / feature request:

When you have a ~/.netrc file with this content:

machine my.corp.com
  login something
  password something

And build --experimental_downloader_config=.bazel_download_config with content of . bazel_download_config as following:

rewrite github.com/(.*) my.corp.com/artifactory/github/$1
rewrite mirror.bazel.build/(.*) my.corp.com/artifactory/bazel-mirror/$1

WORKSPACE

http_archive(
    name = "com_github_bazelbuild_buildtools",
    sha256 = "0dba3995084990d557f3bbb7f7eca4ebcc71d5c9d758eca49342e69fc41e061c",
    strip_prefix = "buildtools-840218fa3afc7e7145c1eeb3bfeb612c497e67f7",
    urls = [
        "https://github.com/bazelbuild/buildtools/archive/840218fa3afc7e7145c1eeb3bfeb612c497e67f7.zip",
    ],
)

http_archive fails with 401.

The reason it fails is because use_netrc from repo/utils.bzl creates map with auth headers using hosts from urls param of http_archive. In this case netrc doesn't have gitbub.com, hence maps comes back empty. And download fails with 401.

On other hand, if I try to "workaround" problem and add github.com into netrc, then

• use_netrc will return map with one entry that corresponds to github.com.
• DownloadManager.java

  bazel/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/DownloadManager.java

  Lines 101 to 105 in 9804ba5

  	if (rewriter != null) {
  	urls = rewriter.amend(urls);
  	}

  
  first rewrite urls, therefore https://github.com/bazelbuild/buildtools/archive/840218fa3afc7e7145c1eeb3bfeb612c497e67f7.zip becomes https://my.corp.com/artifactory/bazelbuild/buildtools/archive/840218fa3afc7e7145c1eeb3bfeb612c497e67f7.zip. Then com.google.devtools.build.lib.bazel.repository.downloader.HttpConnectorMultiplexer#getHeaderFunction tries to extract headers based on the URL. At this point URL is https://my.corp.com/artifactory/bazelbuild/buildtools/archive/840218fa3afc7e7145c1eeb3bfeb612c497e67f7.zip, hence function returns empty map and no auth headers added to the http request.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

see above

What operating system are you running Bazel on?

macOS

What's the output of bazel info release?

release 4.0.0

The solution I propose is to modify use_netrc Starlark function to return map of auth headers based on .netrc file alone, i.e. don't filter it by urls supplied.

[keith]

Also see #13822

[denyska]

Also see #13822

@keith #13822 is great, however it requires storing creds in multiple places: ~/.netrc and .bazel_download_config. I also expect that most of the people will put .bazel_download_config into source control, therefore storing per-user credentials there is not practical.

[bjacklyn]

I just ran into this today when trying out --experimental_downloader_config which unfortunately makes this feature not usable in my org. We can't check in the downloader config file with credentials so Keith's workaround doesn't work for us (our setup is that each developer has a .netrc file with their own api key/encrypted password as the password).

Thank you for putting up a PR to fix this, hopefully someone from Google will have time to review it soon.

[Wyverald]

@bazel-io fork 5.1

[Wyverald]

This is fixed by #14066