http_archive is not able to fetch dl.bintray.com/boostorg

[yongtang]

Description of the problem / feature request:

While working on a tensorflow PR tensorflow/tensorflow#19461 to add the Parquet support, I needed to add the boost library. The download link of boost library (https://www.boost.org/users/download/) is in dl.bintray.com:
https://dl.bintray.com/boostorg/release/1.67.0/source/

However, bazel will generate a GET returned 403 Forbidden error without much information. See example below for details.

Feature requests: what underlying problem are you trying to solve with this feature?

N/A

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I have reduced the test case with the following example:

# WORKSPACE file
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
http_archive(
    name = "boost",
    urls = ["https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.gz"],
    sha256 = "8aa4e330c870ef50a896634c931adf468b21f8a69b77007e45c444151229f665"
)
# main/BUILD file
cc_binary(
    name = "hello-world",
    srcs = ["hello-world.cc"],
    deps = ["@boost"]
)

ubuntu@ubuntu:~/bazel$ bazel build -s --verbose_failures //main:hello-world
ERROR: /home/ubuntu/bazel/main/BUILD:1:1: no such package '@boost//': java.io.IOException: Error downloading [https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.gz] to /home/ubuntu/.cache/bazel/_bazel_ubuntu/ffdf15b7269c0b208fa140f8b561f825/external/boost/boost_1_67_0.tar.gz: GET returned 403 Forbidden and referenced by '//main:hello-world'
ERROR: Analysis of target '//main:hello-world' failed; build aborted: Analysis failed
INFO: Elapsed time: 14.736s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (1 packages loaded)
ubuntu@ubuntu:~/bazel$ 

I am able to download the package manually through curl (See full output at the end):

curl -vv -OL https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.gz

What operating system are you running Bazel on?

ubuntu@ubuntu:~/bazel$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 16.04.4 LTS
Release:	16.04
Codename:	xenial

What's the output of bazel info release?

ubuntu@ubuntu:~/bazel$ bazel info release
release 0.16.1

If bazel info release returns "development version" or "(@Non-Git)", tell us how you built Bazel.

N/A

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

N/A

Have you found anything relevant by searching the web?

No.

Any other information, logs, or outputs that you want to share?

ubuntu@ubuntu:~/bazel$ curl -v -OL https://dl.bintray.com/boostorg/release/1.67.0/source/boost_1_67_0.tar.gz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 108.168.243.150...
* Connected to dl.bintray.com (108.168.243.150) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.bintray.com (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: CN=*.bintray.com
* 	 start date: Thu, 09 Nov 2017 00:00:00 GMT
* 	 expire date: Sat, 09 Nov 2019 23:59:59 GMT
* 	 issuer: C=US,O=GeoTrust Inc.,OU=Domain Validated SSL,CN=GeoTrust DV SSL CA - G3
* 	 compression: NULL
* ALPN, server did not agree to a protocol
> GET /boostorg/release/1.67.0/source/boost_1_67_0.tar.gz HTTP/1.1
> Host: dl.bintray.com
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 302 
< Server: nginx
< Date: Sun, 19 Aug 2018 12:07:21 GMT
< Content-Length: 0
< Connection: keep-alive
< Location: https://d29vzk4ow07wi7.cloudfront.net/8aa4e330c870ef50a896634c931adf468b21f8a69b77007e45c444151229f665?response-content-disposition=attachment%3Bfilename%3D%22boost_1_67_0.tar.gz%22&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHAqOi8vZDI5dnprNG93MDd3aTcuY2xvdWRmcm9udC5uZXQvOGFhNGUzMzBjODcwZWY1MGE4OTY2MzRjOTMxYWRmNDY4YjIxZjhhNjliNzcwMDdlNDVjNDQ0MTUxMjI5ZjY2NT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPWF0dGFjaG1lbnQlM0JmaWxlbmFtZSUzRCUyMmJvb3N0XzFfNjdfMC50YXIuZ3olMjIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1MzQ2ODExNjF9LCJJcEFkZHJlc3MiOnsiQVdTOlNvdXJjZUlwIjoiMC4wLjAuMC8wIn19fV19&Signature=YYxSO8tT6mnm5HXFXDwnkcex--pEHJ2i3EXg~euxv1LCnhslk83vERvL--UBZwCI6f8X4T0FNPqxIE2RC8dojaoM9-kwQrbVfu5rOraOw79jaYhO9QEWkgRhuF0hhyiVU9C1Zneqf4wWISOlg8ZXR4UKX4mFFiFLngEOJv7-Xs6ZA40Ybk3D7b6YxZuBHBXcQSQNYcmCMSkl6kCoPKs4HSMC6GiWhx0swZYw4DFQbetV~A4lcWd62F3AxiFcSDvQEwPsHxOitmCAnm2CpNfw7gaK68En8jDOTdcrHZC68Tr8-m~NAKpBL6QvUFVc3W14hkfixZfcif699yU~KEI6dg__&Key-Pair-Id=APKAIFKFWOMXM2UMTSFA
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host dl.bintray.com left intact
* Issue another request to this URL: 'https://d29vzk4ow07wi7.cloudfront.net/8aa4e330c870ef50a896634c931adf468b21f8a69b77007e45c444151229f665?response-content-disposition=attachment%3Bfilename%3D%22boost_1_67_0.tar.gz%22&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHAqOi8vZDI5dnprNG93MDd3aTcuY2xvdWRmcm9udC5uZXQvOGFhNGUzMzBjODcwZWY1MGE4OTY2MzRjOTMxYWRmNDY4YjIxZjhhNjliNzcwMDdlNDVjNDQ0MTUxMjI5ZjY2NT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPWF0dGFjaG1lbnQlM0JmaWxlbmFtZSUzRCUyMmJvb3N0XzFfNjdfMC50YXIuZ3olMjIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1MzQ2ODExNjF9LCJJcEFkZHJlc3MiOnsiQVdTOlNvdXJjZUlwIjoiMC4wLjAuMC8wIn19fV19&Signature=YYxSO8tT6mnm5HXFXDwnkcex--pEHJ2i3EXg~euxv1LCnhslk83vERvL--UBZwCI6f8X4T0FNPqxIE2RC8dojaoM9-kwQrbVfu5rOraOw79jaYhO9QEWkgRhuF0hhyiVU9C1Zneqf4wWISOlg8ZXR4UKX4mFFiFLngEOJv7-Xs6ZA40Ybk3D7b6YxZuBHBXcQSQNYcmCMSkl6kCoPKs4HSMC6GiWhx0swZYw4DFQbetV~A4lcWd62F3AxiFcSDvQEwPsHxOitmCAnm2CpNfw7gaK68En8jDOTdcrHZC68Tr8-m~NAKpBL6QvUFVc3W14hkfixZfcif699yU~KEI6dg__&Key-Pair-Id=APKAIFKFWOMXM2UMTSFA'
*   Trying 52.85.146.22...
* Connected to d29vzk4ow07wi7.cloudfront.net (52.85.146.22) port 443 (#1)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_128_GCM_SHA256
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: *.cloudfront.net (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: RSA
* 	 certificate version: #3
* 	 subject: C=US,ST=Washington,L=Seattle,O=Amazon.com Inc.,CN=*.cloudfront.net
* 	 start date: Wed, 22 Nov 2017 00:00:00 GMT
* 	 expire date: Wed, 21 Nov 2018 12:00:00 GMT
* 	 issuer: C=US,O=DigiCert Inc,CN=DigiCert Global CA G2
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET /8aa4e330c870ef50a896634c931adf468b21f8a69b77007e45c444151229f665?response-content-disposition=attachment%3Bfilename%3D%22boost_1_67_0.tar.gz%22&Policy=eyJTdGF0ZW1lbnQiOiBbeyJSZXNvdXJjZSI6Imh0dHAqOi8vZDI5dnprNG93MDd3aTcuY2xvdWRmcm9udC5uZXQvOGFhNGUzMzBjODcwZWY1MGE4OTY2MzRjOTMxYWRmNDY4YjIxZjhhNjliNzcwMDdlNDVjNDQ0MTUxMjI5ZjY2NT9yZXNwb25zZS1jb250ZW50LWRpc3Bvc2l0aW9uPWF0dGFjaG1lbnQlM0JmaWxlbmFtZSUzRCUyMmJvb3N0XzFfNjdfMC50YXIuZ3olMjIiLCJDb25kaXRpb24iOnsiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE1MzQ2ODExNjF9LCJJcEFkZHJlc3MiOnsiQVdTOlNvdXJjZUlwIjoiMC4wLjAuMC8wIn19fV19&Signature=YYxSO8tT6mnm5HXFXDwnkcex--pEHJ2i3EXg~euxv1LCnhslk83vERvL--UBZwCI6f8X4T0FNPqxIE2RC8dojaoM9-kwQrbVfu5rOraOw79jaYhO9QEWkgRhuF0hhyiVU9C1Zneqf4wWISOlg8ZXR4UKX4mFFiFLngEOJv7-Xs6ZA40Ybk3D7b6YxZuBHBXcQSQNYcmCMSkl6kCoPKs4HSMC6GiWhx0swZYw4DFQbetV~A4lcWd62F3AxiFcSDvQEwPsHxOitmCAnm2CpNfw7gaK68En8jDOTdcrHZC68Tr8-m~NAKpBL6QvUFVc3W14hkfixZfcif699yU~KEI6dg__&Key-Pair-Id=APKAIFKFWOMXM2UMTSFA HTTP/1.1
> Host: d29vzk4ow07wi7.cloudfront.net
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Type: application/gzip
< Content-Length: 103363944
< Connection: keep-alive
< Date: Sun, 19 Aug 2018 11:53:34 GMT
< Last-Modified: Wed, 11 Apr 2018 14:47:19 GMT
< ETag: "4850fceb3f2222ee011d4f3ea304d2cb"
< Content-Disposition: attachment;filename="boost_1_67_0.tar.gz"
< Accept-Ranges: bytes
< Server: AmazonS3
< Age: 828
< X-Cache: Hit from cloudfront
< Via: 1.1 86335fa0218c5bd3b89dc26ce10431df.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: 8k3qZEp5hT60msMO9RkF8sFU7S57LHYdKJZXcWNGuvVTutt4UB7o2A==
< 
{ [15871 bytes data]
100 98.5M  100 98.5M    0     0  68.0M      0  0:00:01  0:00:01 --:--:--  105M
* Connection #1 to host d29vzk4ow07wi7.cloudfront.net left intact

[aehlig]

However, bazel will generate a `GET returned 403 Forbidden` error without much information. See example below for details. [...]

Thanks for the detailed bug report. That made it very easy to reproduce. By now I managed to narrow down the error to the mergeUrls function https://github.com/bazelbuild/bazel/blob/2c4a0b7bddc5cd9fa83b7c445afced1a984bb785/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpUtils.java#L73 In fact, replacing line https://github.com/bazelbuild/bazel/blob/2c4a0b7bddc5cd9fa83b7c445afced1a984bb785/src/main/java/com/google/devtools/build/lib/bazel/repository/downloader/HttpUtils.java#L66 by a simple `URL result = URI.create(newLocation).toURL();` makes the download succeed. Currently working on a fix for the mergeUrls function.

…

-- Klaus Aehlig Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado

[aehlig]

Pending fix: https://bazel-review.googlesource.com/c/bazel/+/71050

…

-- Klaus Aehlig Google Germany GmbH, Erika-Mann-Str. 33, 80636 Muenchen Registergericht und -nummer: Hamburg, HRB 86891 Sitz der Gesellschaft: Hamburg Geschaeftsfuehrer: Paul Terence Manicle, Halimah DeLaine Prado

[yongtang]

Thanks for the quick fix @aehlig !👍 🎉

[ittaiz]

Awesome! We (rules scala) were also severely impacted by this. Thank you both! cc @johnynek

…

On Mon, 20 Aug 2018 at 17:56 Yong Tang ***@***.***> wrote: Thanks for the quick fix @aehlig <https://github.com/aehlig> !👍 🎉 — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5932 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABUIFy2SsYzARrZ4rH07NaTgJvx_hEa1ks5uSs4ugaJpZM4WC8ap> .

[ccdavid]

I am facing this issue. How can the fix be used before it's part of a release? Even the Compiling Bazel from Source instructions require a distribution zip to be available.

[felderhof]

I see this is marked as closed but it does still seem to be an issue, for me at least. I imagine that this is just because I don't know what to do.

Is there a way to depend on an explicit previous version of the downloader (or target future version)?

I have just been using the packaged version of bazel on ubuntu. Do I need to build my own version from the latest source? Will that solve the problem? (I'm afraid that I don't fully understand how this became an issue in the first place).

I have also tried to copy the relevant tarball into the expected location by hand but it seems to be deleted each time. Maybe there is a temporary work around like this?

Many thanks in advance for any help.

[aehlig]

How can the fix be used before it's part of a release?

You can take the latest released bazel and then bazel build //src:bazel on the tree at head.

[felderhof]

I had a problem building the latest version from source but for the sake of anyone coming across this thread, upgrading to latest testing version of bazel on ubuntu according to the instructions here made the issue go away.

At time of writing, this was version 0.17.1rc1.

Many thanks for your help.