x/net/http2: Server rejects Transport with ErrCodeProtocol stream error

[bradfitz]

We're running a Go 1.17rc2 httputil.ReverseProxy in front of a Go go1.17rc2 net/http.Server.

We were seeing weird ErrCodeProtocol stream errors, so we stopped using the bundled x/net/http2 and switched both to using x/net/http2 explicitly (with http2.ConfigureTransports and http2.ConfigureServer, so the bundled one isn't used).

Our x/net git version is golang/net@aaa1db6, which is today's current master.

The logs we were getting weren't initially enough to tell us where the problem was:

2021/08/10 14:07:20 http: proxy error: stream error: stream ID 1293155; PROTOCOL_ERROR
2021/08/10 14:07:20 http: proxy error: stream error: stream ID 1293157; PROTOCOL_ERROR

But after hacking it up locally to add some expvar counters on which paths in http2.Server were returning the protocol error, we found it was here:

https://github.com/golang/net/blob/aaa1db679c0d7765d2b1cb1f92cac8ebf4d94c53/http2/server.go#L1835

// http://tools.ietf.org/html/rfc7540#section-5.1.2
// [...] Endpoints MUST NOT exceed the limit set by their peer. An
// endpoint that receives a HEADERS frame that causes their
// advertised concurrent stream limit to be exceeded MUST treat
// this as a stream error (Section 5.4.2) of type PROTOCOL_ERROR
// or REFUSED_STREAM.
if sc.curClientStreams+1 > sc.advMaxStreams {
        if sc.unackedSettings == 0 {
                // They should know better.
                return streamError(id, ErrCodeProtocol)
        }

So, the Go http2 client (Transport) and/or server is getting its concurrent/max streams accounting wrong.

Perhaps interesting: the backend (the http2.Server that the ReverseProxy's Transport is hitting) has extremely long-running http.Handlers (like: days or weeks longs) alongside many short ones. That might explain why this bug hasn't been reported by more people.

I see no recent commits that suggest this is a regression. This server is new, so I also can't say whether it worked previously.

/cc @neild @bcmills @fraenkel @dmitshur @dsnet @josharian

[fraenkel]

possible? #42777

[bradfitz]

Ah, yes. That's probably what we're hitting.

[dmitshur]

CC @neild.

[gopherbot]

Change https://golang.org/cl/347033 mentions this issue: http2: make Transport not reuse conns after a stream protocol error

[gopherbot]

Change https://golang.org/cl/356978 mentions this issue: [internal-branch.go1.16-vendor] http2: make Transport not reuse conns after a stream protocol error

[gopherbot]

Change https://golang.org/cl/357673 mentions this issue: [internal-branch.go1.17-vendor] http2: make Transport not reuse conns after a stream protocol error