net/http: NewRequest modifies path

[gopherbot]

by stuart.carnie:

What does 'go version' print?
go version go1.3.1 darwin/amd64

What steps reproduce the problem?

Unable to fetch the following image URL when using net/http.Get or manually constructing
a http.NewRequest.  
https://d2mckvlpm046l3.cloudfront.net/787968d58b958e4aea105ad017f9a1abb6cce99c/http:%2F%2Fd.pr%2Fi%2F4EN8%2B.


http://play.golang.org/p/ZhefDUqSNS

What happened?
Server responds with a 404 Not Found

What should have happened instead?
Image data should have been returned in the response Body, such as when using a browser
or CLI tools like wget or curl

[gopherbot]

Comment 1 by stuart.carnie:

URL as posted is confirmed to load in Chrome

[josharian]

Comment 2:

Thanks for the report.
Diagnosis: http.NewRequest parses the URL, which unescapes the :, /, and + in the path
-- see http://play.golang.org/p/HNK6Qa4RfH. curl, browsers, etc. send the original path.
cloudfront doesn't treat the unescaped and the escaped paths the same, which looks like
a violation of the RFC. On the other hand, it seems like the http client shouldn't
modify the path provided to it. Brad?

Labels changed: added repo-main.

Owner changed to @bradfitz.

[bradfitz]

Comment 3:

This is the dozenth bug in the same theme. It's been like this for ages.
The http client ultimately uses net/url.Parse which unescapes the path.
The workaround has been documented at http://golang.org/pkg/net/url/#URL but not on
url.Parse, nor on http.Get and friends.
Perhaps we need a new url.ParseRaw or something which is documented to guarantee to
round-trip (String->Parse->String), parsing into the url.Opaque field as needed to
preserve the guarantee.
Then we can adjust net/http to use url.ParseRaw instead.
Assigning to Russ for his opinion, and he can assign back to me to implement if desired.

Owner changed to @rsc.

[bradfitz]

Comment 4:

Update: I added an example at https://golang.org/cl/171620043