Skip to content

net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) #63417

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
neild opened this issue Oct 6, 2023 · 18 comments
Assignees
Labels
NeedsFix The path to resolution is known, but the work has not been done. release-blocker Security
Milestone

Comments

@neild
Copy link
Contributor

neild commented Oct 6, 2023

A malicious HTTP/2 client which rapidly creates requests and
immediately resets them can cause excessive server resource consumption.
While the total number of requests is bounded to the
http2.Server.MaxConcurrentStreams setting, resetting an in-progress
request allows the attacker to create a new request while the existing
one is still executing.

HTTP/2 servers now bound the number of simultaneously executing
handler goroutines to the stream concurrency limit. New requests
arriving when at the limit (which can only happen after the client
has reset an existing, in-flight request) will be queued until a
handler exits. If the request queue grows too large, the server
will terminate the connection.

This issue is also fixed in golang.org/x/net/http2 v0.17.0,
for users manually configuring HTTP/2.

The default stream concurrency limit is 250 streams (requests)
per HTTP/2 connection. This value may be adjusted using the
golang.org/x/net/http2 package; see the Server.MaxConcurrentStreams
setting and the ConfigureServer function.

This is CVE-2023-39325 and Go issue https://go.dev/issue/63417.
This is also tracked by CVE-2023-44487.


This is a PRIVATE issue for CVE-2023-39325, tracked in http://b/303836512.

/cc @golang/security and @golang/release

@neild neild self-assigned this Oct 6, 2023
@seankhliao seankhliao added Security NeedsFix The path to resolution is known, but the work has not been done. release-blocker labels Oct 6, 2023
@seankhliao seankhliao added this to the Go1.22 milestone Oct 6, 2023
@neild
Copy link
Contributor Author

neild commented Oct 6, 2023

@gopherbot please open backport issues

@gopherbot
Copy link
Contributor

Backport issue(s) opened: #63426 (for 1.20), #63427 (for 1.21).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/534215 mentions this issue: http2: limit maximum handler goroutines to MaxConcurrentStreams

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/534235 mentions this issue: [release-branch.go1.21] net/http: regenerate h2_bundle.go

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/534255 mentions this issue: [release-branch.go1.20] net/http: regenerate h2_bundle.go

@dmitshur
Copy link
Member

Reopening to track bundling into the main tree, for Go 1.22.

@dmitshur dmitshur reopened this Oct 10, 2023
gopherbot pushed a commit that referenced this issue Oct 10, 2023
Pull in a security fix from x/net/http2:
http2: limit maximum handler goroutines to MaxConcurrentStreamso

For #63417
Fixes #63427
Fixes CVE-2023-39325

Change-Id: I70626734e6d56edf508f27a5b055ddf96d806eeb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047402
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Ian Cottrell <iancottrell@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/534235
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/534295 mentions this issue: all: pull in x/net v0.17.0 and its dependencies

gopherbot pushed a commit that referenced this issue Oct 10, 2023
Pull in a security fix from x/net/http2:
http2: limit maximum handler goroutines to MaxConcurrentStreamso

For #63417
Fixes #63426
Fixes CVE-2023-39325

Change-Id: I6e32397323cd9b4114c990fcc9d19557a7f5f619
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047401
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
TryBot-Result: Security TryBots <security-trybots@go-security-trybots.iam.gserviceaccount.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Ian Cottrell <iancottrell@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/534255
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
TryBot-Bypass: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
@dmitshur dmitshur changed the title security: fix CVE-2023-39325 net/http, x/net/http2: rapid stream resets can cause excessive work (CVE-2023-39325) Oct 10, 2023
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/534218 mentions this issue: [internal-branch.go1.21-vendor] http2: limit maximum handler goroutines to MaxConcurrentStreams

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/534236 mentions this issue: [internal-branch.go1.20-vendor] http2: limit maximum handler goroutines to MaxConcurrentStreams

gopherbot pushed a commit to golang/net that referenced this issue Oct 10, 2023
…es to MaxConcurrentStreams

When the peer opens a new stream while we have MaxConcurrentStreams
handler goroutines running, defer starting a handler until one
of the existing handlers exits.

For golang/go#63417.
For golang/go#63427.
For CVE-2023-39325.

Change-Id: If0531e177b125700f3e24c5ebd24b1023098fa6d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047391
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Ian Cottrell <iancottrell@google.com>
Reviewed-on: https://go-review.googlesource.com/c/net/+/534218
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
gopherbot pushed a commit to golang/net that referenced this issue Oct 10, 2023
…es to MaxConcurrentStreams

When the peer opens a new stream while we have MaxConcurrentStreams
handler goroutines running, defer starting a handler until one
of the existing handlers exits.

For golang/go#63417.
For golang/go#63426.
For CVE-2023-39325.

Change-Id: If0531e177b125700f3e24c5ebd24b1023098fa6d
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047553
Reviewed-by: Ian Cottrell <iancottrell@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
TryBot-Result: Security TryBots <security-trybots@go-security-trybots.iam.gserviceaccount.com>
Reviewed-on: https://go-review.googlesource.com/c/net/+/534236
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
@gopherbot
Copy link
Contributor

Change https://go.dev/cl/534297 mentions this issue: [release-branch.go1.20] all: tidy dependency versioning after release

@gopherbot
Copy link
Contributor

Change https://go.dev/cl/534415 mentions this issue: [release-branch.go1.21] all: tidy dependency versioning after release

smira added a commit to smira/tools that referenced this issue Oct 10, 2023
See golang/go#63417

Signed-off-by: Andrey Smirnov <andrey.smirnov@siderolabs.com>
awly pushed a commit to tailscale/go that referenced this issue Feb 7, 2024
Pull in a security fix from x/net/http2:
http2: limit maximum handler goroutines to MaxConcurrentStreamso

For golang#63417
Fixes golang#63427
Fixes CVE-2023-39325

Change-Id: I70626734e6d56edf508f27a5b055ddf96d806eeb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047402
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Ian Cottrell <iancottrell@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/534235
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
awly pushed a commit to tailscale/go that referenced this issue Feb 7, 2024
Done with:

go get golang.org/x/net@internal-branch.go1.21-vendor
go mod tidy
go mod vendor
go generate net/http  # zero diff since CL 534235 already did this

For golang#63417.
For golang#63427.
For CVE-2023-39325.

Change-Id: Ib258e0d8165760a1082e02c2f4c5ce7d2a3c3c90
Reviewed-on: https://go-review.googlesource.com/c/go/+/534415
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
JanDeDobbeleer pushed a commit to JanDeDobbeleer/go that referenced this issue Feb 12, 2024
Pull in a security fix from x/net/http2:
http2: limit maximum handler goroutines to MaxConcurrentStreamso

For golang#63417
Fixes golang#63427
Fixes CVE-2023-39325

Change-Id: I70626734e6d56edf508f27a5b055ddf96d806eeb
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/2047402
Reviewed-by: Tatiana Bradley <tatianabradley@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
Reviewed-by: Ian Cottrell <iancottrell@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/534235
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
JanDeDobbeleer pushed a commit to JanDeDobbeleer/go that referenced this issue Feb 12, 2024
Done with:

go get golang.org/x/net@internal-branch.go1.21-vendor
go mod tidy
go mod vendor
go generate net/http  # zero diff since CL 534235 already did this

For golang#63417.
For golang#63427.
For CVE-2023-39325.

Change-Id: Ib258e0d8165760a1082e02c2f4c5ce7d2a3c3c90
Reviewed-on: https://go-review.googlesource.com/c/go/+/534415
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
@tuk2
Copy link

tuk2 commented Aug 20, 2024

The issue with HTTP/2 rapid reset is still active. When an HTTP/2 server is attacked using rapid resets, it results in memory leaks (up to peak memory usage). This problem is especially noticeable when using net/http/httputil (reverse proxy). IMPORTANT: The memory leak occurs ONLY during rapid reset attacks.

photo_2024-08-20_23-26-23
profile002
Screenshot 2024-08-20 234726

@tuk2
Copy link

tuk2 commented Aug 20, 2024

@bradfitz

camilamacedo86 added a commit to camilamacedo86/catalogd that referenced this issue Dec 13, 2024
- Ensure HTTP/2 is disabled when the enable-http2 flag is set to false (default).
- Disabling HTTP/2 mitigates vulnerabilities associated with:
  - HTTP/2 Stream Cancellation (GHSA-qppj-fm5r-hxr3)
  - HTTP/2 Rapid Reset (GHSA-4374-p667-p6c8)
- While CVE fixes exist, they remain insufficient; disabling HTTP/2 helps reduce risks.
  For details, see:
  - GHSA-qppj-fm5r-hxr3
  - GHSA-4374-p667-p6c8
  - golang/go#63417
camilamacedo86 added a commit to camilamacedo86/catalogd that referenced this issue Dec 13, 2024
- Ensure HTTP/2 is disabled when the enable-http2 flag is set to false (default).
- Disabling HTTP/2 mitigates vulnerabilities associated with:
  - HTTP/2 Stream Cancellation (GHSA-qppj-fm5r-hxr3)
  - HTTP/2 Rapid Reset (GHSA-4374-p667-p6c8)
- While CVE fixes exist, they remain insufficient; disabling HTTP/2 helps reduce risks.
  For details, see:
  - GHSA-qppj-fm5r-hxr3
  - GHSA-4374-p667-p6c8
  - golang/go#63417
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsFix The path to resolution is known, but the work has not been done. release-blocker Security
Projects
None yet
Development

No branches or pull requests

10 participants