Add support for S3

[dancannon]

As discussed in #2 I have worked on adding support for S3, I have also included the features requested in the mentioned ticket, to summarise this adds the following features:

• Detect S3 URLS in the following formats (this means that any S3 urls will no longer be fetched using the HTTP getter)
  • BUCKET.s3.amazonaws.com/PATH
  • BUCKET.s3-REGION.amazonaws.com/PATH
  • s3.amazonaws.com/BUCKET/PATH
  • s3-REGION.amazonaws.com/BUCKET/PATH
• Also supported is the ability to override the default authentication method and the version being "getted" (this is only supported by GetFile). For example:
  • s3-REGION.amazonaws.com/BUCKET/PATH?aws_access_key_id=foo&aws_access_key_secret=bar&aws_access_token=baz
  • s3-REGION.amazonaws.com/BUCKET/PATH?version=1234
• Using Get all files with the given prefix are fetched and stored in the destination path relative to the given prefix
• Using GetFile fetches the given file from S3 and is stored at the destination path. As mentioned it is also possible to fetch a versioned S3 object.

If you have any further questions/feature suggestions let me know, thanks.

[catsby]

looks like this was left over from a copy-paste

[dancannon]

Removing now, thanks.

[catsby]

Hey @dancannon – great start!
When I add my credentials to my environment, I get a single failure:

--- FAIL: TestS3Getter (0.00s)
    get_s3_test.go:21: err: URL is not a valid S3 URL

Is that expected, because I don't have access to that URL? It sounds like an invalid URL, not just a permissions thing

[dancannon]

It sounds like something else is going on, could you please paste a sample URL that has the same format?

[catsby]

This is the test and url that's failing

• https://github.com/hashicorp/go-getter/pull/3/files#diff-fe4fbe1b573926567435619dee064738R20

The URL is https://s3-eu-west-1.amazonaws.com/hailo-s3-test

cUrl suggest the url itself is fine:

$ curl -I https://s3-eu-west-1.amazonaws.com/hailo-s3-test
HTTP/1.1 200 OK

[dancannon]

Ah sorry about that!! I had added some more validation which wasnt quite correct. The tests should pass now.

[mitchellh]

Unused constant, remove?

[mitchellh]

The only thing that appears to be missing is loading credentials from environment, profile, etc. The AWS SDK does this for you but you have to make sure to just enable the credential lookup thing. If you already do this and I missed it or you need more guidance, let me know.

Your tests look great.

[dancannon]

Thanks for looking at this so quickly, regarding the AWS credentials I was under the impression that if you dont pass any Credentials to the config then DefaultChainCredentials is used by default which looks up the credentials. Also I tested this code with a private S3 bucket and it seemed to pick up the credentials.

If this is incorrect let me know and I will look into it again.

[mitchellh]

We should explicitly setup the credential chain. By using DefaultChainCredentials it allows an external caller to modify it and we don't really want to allow that. Once that fix is in, I'll merge

[dancannon]

I have made the suggested change, however now that you had added CI the tests are failing as travis does not have any AWS credentials.

Have you got any ideas for how we could fix this?

[mitchellh]

I'll take a look locally. Thanks!