cli compiled against golang 1.8.3 cannot download app droplet

[kkallday]

Command

(from cf-acceptance-tests in cf-release 259)

cf curl -i /v2/apps/649e74de-c2a4-45f1-b93f-5d797017f6bc/droplet/download --output droplet.tgz

What occurred

The droplet.tgz file was not a tarball. It was a file that contained the following XML:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>ACCESSKEY</AWSAccessKeyId><StringToSign>GET

application/json
1497493686
/*DROPLET BUCKET*/0f/e8/0fe84d1c-a4f5-4cab-91cb-e084782bfaed/ffdcf5b508ba64897fe28de6f7234b31c40175ad</StringToSign><SignatureProvided>*SOME SIGNATURE*</SignatureProvided><StringToSignBytes>*SOME STRING TO SIGN*</StringToSignBytes><RequestId>* SOME REQUEST ID*</RequestId><HostId>*SOME HOST ID*</HostId></Error>

What you expected to occur

The droplet.tgz file to contain my app bits.

CLI Version

cf version 6.27.0+d26b32dcc.2017-06-08

CC API Endpoint Version

2.82.0

CF Trace

HTTP/1.1 403 Forbidden
Connection: close
Transfer-Encoding: chunked
Content-Type: application/xml
Date: Thu, 15 Jun 2017 23:45:46 GMT
Server: AmazonS3
X-Amz-Id-2: *SOME AMZ ID*
X-Amz-Request-Id: *SOME REQUEST ID*

Platform & Shell Details

Mac OS X and Linux

Any other relevant information

The cf curl reaches the Cloud Controller, which 302 redirects us to the S3 droplet bucket. We were able to successfully download the droplet when we did a normal curl using the same URL that CC redirects cf client to.

We've observed the same issue on Google Cloud. When running tcpdump we noticed that the failing request to Google Cloud Storage had a header of Content-Type: application/json. When we sent the same request without the Content-Type header, we were able to download the droplet.

We believe the root cause is due to a change in net/http (golang/go#19973) in Go 1.8. The client now resends the original request headers when following a redirect.

Thanks,

@acrmp and @kkallday

[cf-gitbot]

We have created an issue in Pivotal Tracker to manage this:

https://www.pivotaltracker.com/story/show/147310395

The labels on this github issue will be updated when the story is started.

[dkoper]

cf curl defaults to application/json for both content-type and accept.
cf CLI 6.27.0 works fine for me against an S3 blobstore based droplet storage so the following is making guesses based on your issue description:

It looks like cf curl -i /v2/apps/649e74de-c2a4-45f1-b93f-5d797017f6bc/droplet/download --output droplet.tgz happened to work before for you because the accept: application/json header did not reach your cloud storage due to the redirect, so it returned the binary as you expected. But with the golang change, it returns xml as it was requested to.

Does cf curl -i /v2/apps/649e74de-c2a4-45f1-b93f-5d797017f6bc/droplet/download -H "accept: application/zip" --output droplet.tgz work for you?

[kkallday]

Sorry about taking a while to get back to you, @dkoper.

You are correct - cf curl ... worked fine with previous golang because it did not send through headers on 302 redirection.

You mentioned that due to the golang change, the request "returns xml as it was requested to" but this isn't the case. I suspect this is what is happening:

1. cf curl sends request to Cloud Controller with Content-Type: application/json.
2. Cloud Controller generates a signed request letting S3/GCS know that a request will be arriving with a specific data (specific url, no headers, no body). I believe the request S3/GCS will accept is one without the Content-Type: application/json header.
3. cf curl follows the redirect sending Content-Type: application/json (since this is the golang change that was made)
4. S3/GCS says the request signature does not match the one that was provided (Cloud Controller provided the "signature"). The response is an XML error message:

<?xml version="1.0" encoding="UTF-8"?>
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message><AWSAccessKeyId>ACCESSKEY</AWSAccessKeyId><StringToSign>GET

application/json
1497493686
/*DROPLET BUCKET*/0f/e8/0fe84d1c-a4f5-4cab-91cb-e084782bfaed/ffdcf5b508ba64897fe28de6f7234b31c40175ad</StringToSign><SignatureProvided>*SOME SIGNATURE*</SignatureProvided><StringToSignBytes>*SOME STRING TO SIGN*</StringToSignBytes><RequestId>* SOME REQUEST ID*</RequestId><HostId>*SOME HOST ID*</HostId></Error>

Given that the presence of Content-Type: application/json is what causes the request to fail, I don't think accept: application/json will solve the issue but it is worth trying. I will give it a shot tomorrow to confirm.

[kkallday]

We found a commit in the bosh-cli repo that shows how they are dealing with the Go HTTP redirect header issue. In this commit they are removing all headers (except Authorization header) before following redirects.

This is the stdlib function they are using to deal with the issue (see CheckRedirect description).

Sorry about the late response.

[dkoper]

We're aware of the bosh-cli commit. Unfortunately the way cf curl was designed there is no such simple fix for us. We'd have to refactor the command first which will take time and currently not a priority.

You mentioned there was something else worth trying and you'd give it a shot?

[kkallday]

I tried adding a header of accept: application/zip as you suggested but the test is still failing.

When I run the test with cli v6.26.0, the test passes. When I run the test with cli v6.27.0, the test fails.

In v6.27.0, the release notes says it was compiled with go 1.8.3 and I'm assuming this is when the stdlib change was made.

[dkoper]

Thanks for trying!

[XenoPhex]

Closing this because we have an official work around (aka use non-cf curl to download the droplet).

Going forward - use the curl to download droplets - cf curl does not function with external blob stores.