Generates invalid URIs

[SergeyTsalkov]

An HTTP PUT request should be of the form

PUT path/to/file.jpg
Host: bucket.s3.amazonaws.com

However, this library generates PUT requests of the form

PUT http://bucket.s3.amazonaws.com/path/to/file.jpg
Host: bucket.s3.amazonaws.com

Although S3 handles this correctly, it violates both HTTP/1.1 and the spec here: http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html

This also breaks when using DreamObjects, an S3-compatible object store.

[jasdel]

Hi @SergeyTsalkov Thanks for reporting this issue. This is occurring due to the SDK usage of net/url's URL.Opaque field. This field is needed by the SDK to prevent the Go net/url library from mutating the URI by escaping and unescaping. URL.String() defines how Opaque is used when constructing a URL.

From what I found absolute URI for all methods except CONNECT is valid. According to the HTTP1.1 spec

To allow for transition to absoluteURIs in all requests in future versions of HTTP, all HTTP/1.1 servers MUST accept the absoluteURI form in requests, even though HTTP/1.1 clients will only generate them in requests to proxies.

The Go net/url seems to be taking this as being allowed to use absoluteURI when URL.Opaque is set. The SDK could be updated to use the Request.Host and only set the URL's path in URL.Opaque since URL.Host would be ignored when making requests. But this breaks the ability to call URL.String() to receive a useful URL string. Since URL.Host is ignored.

[lsegal]

@SergeyTsalkov is the report specifically regarding support for DreamObjects? I would imagine that if DreamObjects is "S3-compatible" as you mention, then it should support whatever is supported by S3, including absoluteURI syntax (otherwise it may not be entirely S3-compatible). Specifically, the SDK is not doing anything that another tool, or even a very well intentioned proxy, might be doing, and according to the HTTP/1.1 spec, web servers (including a server like DreamObjects) "MUST" support the absoluteURI form that the SDK is providing in order to be compliant-- even though you could argue that we "SHOULD" not be crafting the request in this way.

The problem is that, unfortunately, there is no other way to control encoding logic for URIs other than using the Opaque field, and since the URL value is part of our public API (req.HTTPRequest.URL), we want to ensure that it is a self-contained API; in other words, calling req.HTTPRequest.URL.String() returns the fully qualified endpoint without having to manually build it yourself by concatenating values from the URL and the HTTPRequest.Host, which can be confusing and complicated to do. Request pre-signing specifically needs this functionality, for example.

If the issue is really specifically regarding SDK compatibility with DreamObjects (since the SDK itself works fine with S3), I would recommend opening an issue with DreamObjects and requesting that they investigate support for the absoluteURI syntax in their web server logic-- according to the HTTP/1.1 spec, this is something they should ("MUST", technically) be supporting, and something they should be supporting "in all requests in future versions of HTTP", so it is explicitly not a legacy thing.

[SergeyTsalkov]

Guys, thanks for the quick and thoughtful responses! Yes, my complaint is entirely about DreamObjects compatibility. Clearly I was wrong about HTTP/1.1, so thanks for that lesson!

I'd hoped that fixing the problem on this end would be faster and easier than getting DreamObjects changed, but it sounds like this would be impractical.

[jasdel]

@SergeyTsalkov I'm going to close this issue since using the absolute URL within the HTTP request is apart of the HTTP spec, and valid. The Go net/http.URL auto (un)escapes values in the URL which are incompatible with the AWS services. The SDK needs to use the Opaque field of the URL to ensure the auto unescaping logic is not used, and the request are made how services need them.