net/http/httputil: ReverseProxy does not set host header correctly

[broady]

The HTTP/2 client prefers Request.Host over Request.URL.Host:

go/src/net/http/h2_bundle.go

Lines 7947 to 7954 in a0d6420

	func (cc *http2ClientConn) encodeHeaders(req *Request, addGzipHeader bool, trailers string, contentLength int64) ([]byte, error) {
	cc.hbuf.Reset()
	host := req.Host
	if host == "" {
	host = req.URL.Host
	}
	host, err := httpguts.PunycodeHostPort(host)

ReverseProxy's default Director only sets Request.URL.Host, and not Request.Host, so the request would get proxied to the same host, rather than the proxy target. This resulted in an infinite loop on golang.org (see Issue #28134).

go/src/net/http/httputil/reverseproxy.go

Lines 105 to 121 in a0d6420

	func NewSingleHostReverseProxy(target *url.URL) *ReverseProxy {
	targetQuery := target.RawQuery
	director := func(req *http.Request) {
	req.URL.Scheme = target.Scheme
	req.URL.Host = target.Host
	req.URL.Path = singleJoiningSlash(target.Path, req.URL.Path)
	if targetQuery == "" || req.URL.RawQuery == "" {
	req.URL.RawQuery = targetQuery + req.URL.RawQuery
	} else {
	req.URL.RawQuery = targetQuery + "&" + req.URL.RawQuery
	}
	if _, ok := req.Header["User-Agent"]; !ok {
	// explicitly disable User-Agent so it's not set to default value
	req.Header.Set("User-Agent", "")
	}
	}
	return &ReverseProxy{Director: director}

[gopherbot]

Change https://golang.org/cl/141718 mentions this issue: godoc/proxy: workaround for infinite redirect on ReverseProxy

[bradfitz]

It's preferring the req.Host first per the docs on net/http.Request.Host:

        // For client requests Host optionally overrides the Host
        // header to send. If empty, the Request.Write method uses
        // the value of URL.Host. Host may contain an international
        // domain name.
        Host string

I worry it might not be safe to change this without breaking other people. (There's the usual forward-proxy vs reverse-proxy distinction that people often bring up, too, which is kinda related here.) But if we have different behavior already whether we use HTTP/1 vs HTTP/2 to the backend, then I could see changing it.

[broady]

Hm, would it be safe to set (or unset) Request.Host in the default director?

To note....

The workaround I tried for godoc actually didn't work in production (but worked locally). It 502s and logs this:

httputil: ReverseProxy read error during body copy: stream error: stream ID 5; PROTOCOL_ERROR

[bradfitz]

Well a PROTOCOL_ERROR is very interesting too.

If we have a nice way to capture stderr on one of those, GODEBUG=http2debug=2 would be very interesting. (but send it to me privately in case there are secrets: likely)

[gopherbot]

Change https://golang.org/cl/153858 mentions this issue: [release-branch.go1.11] godoc/proxy: remove use of httputil.ReverseProxy for /share

[ahmetb]

I also hit this! It applies to both http/1 and http/2 based on my experimentation.

I refactored both Director and Transport (http.RoundTripper) to set the Host header, yet the local hostname of my server leaks to the backend and therefore my program doesn't function correctly.

I basically rewrite the /get request to send to https://httpbin.org/get while setting the Host: httpbin.org and Scheme = "https". The echoed response shows that the Host header remained as localhost:8080.

I also asked this at https://stackoverflow.com/questions/55426924/go-httputil-reverseproxy-not-overriding-the-host-authority-header, my minimal repro below:

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
)

func main() {
    proxy := &httputil.ReverseProxy{
        Transport: roundTripper(rt),
        Director: func(req *http.Request) {
            req.URL.Scheme = "https"
            req.URL.Host = "httpbin.org"
            req.Header.Set("Host", "httpbin.org") // <--- I set it here first
        },
    }
    log.Fatal(http.ListenAndServe(":8080", proxy))
}

func rt(req *http.Request) (*http.Response, error) {
    log.Printf("request received. url=%s", req.URL)
    req.Header.Set("Host", "httpbin.org") // <--- I set it here as well
    defer log.Printf("request complete. url=%s", req.URL)

    return http.DefaultTransport.RoundTrip(req)
}


// roundTripper makes func signature a http.RoundTripper
type roundTripper func(*http.Request) (*http.Response, error)

func (f roundTripper) RoundTrip(req *http.Request) (*http.Response, error) { return f(req) }

[uucckk]

Goang's access to nginx through proxy can not correctly orientate the website, is it solved at present?

If it's solved, you can tell me which version, or solve it by other means.thinks.

[ahmetb]

Turns out I needed to set req.Host field instead and that solved it for me above.
https://stackoverflow.com/questions/55426924/go-httputil-reverseproxy-not-overriding-the-host-header

[thediveo]

Lost 1h searching for my downstream server returning 403 when going through the httputil.ReverseProxy with the stock Director, while curl worked as expected. Finally, I found this bug report and I can confirm that it is necessary to set req.Host to req.URL.Host for the reverse proxy to work and not cause endless recursion.

At least, the implementation of Director in NewReverseProxyForSingleHost should be fixed and properly documented.

[stevenh]

Confirmed this had me digging for some time too.

[docmerlin]

Welp, we just ran into this.

[domdom82]

Ran into the same issue in almost 2021 now. Fixed it by manually adding

req.Host = targetURL.Host

in the director function.. but why do I have to do this!?

[gopherbot]

Change https://go.dev/cl/407214 mentions this issue: net/http/httputil: add ReverseProxy.Rewrite