Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

forwarding requests subjected to Login MFA to the active node #15009

Merged
merged 3 commits into from
Apr 13, 2022
Merged

forwarding requests subjected to Login MFA to the active node #15009

merged 3 commits into from
Apr 13, 2022

Conversation

hghaf099
Copy link
Contributor

@hghaf099 hghaf099 commented Apr 12, 2022 •
edited
Loading

Fixing a panic that happens when a two-phase login request that is subject to MFA verification is serviced by perfStandby nodes. I was able to reproduce this on the ENT side, and going to add that test in that repo. The panic happens when the auth response is being saved into the priority queue. And in Standby nodes that Queue is not being initialized.

Below is the stack trace filed by the HCP team:

Apr 12 12:55:19.832	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/h2_bundle.go:5579 +0x510
Apr 12 12:55:19.832	vault	7-2	created by net/http.(*http2serverConn).processHeaders
Apr 12 12:55:19.832	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/h2_bundle.go:5849 +0x78
Apr 12 12:55:19.832	vault	7-2	net/http.(*http2serverConn).runHandler(0x12cf805, 0xc001ee7d00, 0xc002d6ec00, 0xc001ee7d00)
Apr 12 12:55:19.832	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:3480 +0x245
Apr 12 12:55:19.832	vault	7-2	net/http.initALPNRequest.ServeHTTP({{0x67593f0, 0xc002252f30}, 0xc001e4e000, {0xc001303ce0}}, {0x670f8b8, 0xc002630580}, 0xc002a0e700)
Apr 12 12:55:19.832	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2879 +0x43b
Apr 12 12:55:19.832	vault	7-2	net/http.serverHandler.ServeHTTP({0xc00011e000}, {0x670f8b8, 0xc002630580}, 0xc002a0e700)
Apr 12 12:55:19.832	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2047 +0x2f
Apr 12 12:55:19.831	vault	7-2	net/http.HandlerFunc.ServeHTTP(0x0, {0x670f8b8, 0xc002630580}, 0x8)
Apr 12 12:55:19.831	vault	7-2	/home/runner/go/pkg/mod/github.com/hashicorp/go-cleanhttp@v0.5.2/handlers.go:42 +0x98
Apr 12 12:55:19.831	vault	7-2	github.com/hashicorp/go-cleanhttp.PrintablePathCheckHandler.func1({0x670f8b8, 0xc002630580}, 0xc002a0e700)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2047 +0x2f
Apr 12 12:55:19.831	vault	7-2	net/http.HandlerFunc.ServeHTTP(0xc0006b1080, {0x670f8b8, 0xc002630580}, 0xc0020a8501)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/http/handler.go:422 +0x119c
Apr 12 12:55:19.831	vault	7-2	github.com/hashicorp/vault/http.wrapGenericHandler.func1({0x670f8b8, 0xc002630580}, 0xc002a0e700)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2047 +0x2f
Apr 12 12:55:19.831	vault	7-2	net/http.HandlerFunc.ServeHTTP(0x4ec88a0, {0x6702568, 0xc002252fc0}, 0x11)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/http/util_ent.go:83 +0x468
Apr 12 12:55:19.831	vault	7-2	github.com/hashicorp/vault/http.wrapDRSecondaryHandler.func1({0x6702568, 0xc002252fc0}, 0xc000c58420)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2047 +0x2f
Apr 12 12:55:19.831	vault	7-2	net/http.HandlerFunc.ServeHTTP(0x85b, {0x6702568, 0xc002252fc0}, 0x0)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/http/util.go:97 +0x9d0
Apr 12 12:55:19.831	vault	7-2	github.com/hashicorp/vault/http.rateLimitQuotaWrapping.func1({0x6702568, 0xc002252fc0}, 0xc002814500)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2047 +0x2f
Apr 12 12:55:19.831	vault	7-2	net/http.HandlerFunc.ServeHTTP(0xc000842800, {0x6702568, 0xc002252fc0}, 0xc001ff9e00)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/http/cors.go:29 +0x6e4
Apr 12 12:55:19.831	vault	7-2	github.com/hashicorp/vault/http.wrapCORSHandler.func1({0x6702568, 0xc002252fc0}, 0xc0015a3350)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2047 +0x2f
Apr 12 12:55:19.831	vault	7-2	net/http.HandlerFunc.ServeHTTP(0xd, {0x6702568, 0xc002252fc0}, 0xc0015a32e8)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/http/help.go:23 +0x129
Apr 12 12:55:19.831	vault	7-2	github.com/hashicorp/vault/http.wrapHelpHandler.func1({0x6702568, 0xc002252fc0}, 0xc002814500)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2425 +0x149
Apr 12 12:55:19.831	vault	7-2	net/http.(*ServeMux).ServeHTTP(0x3, {0x6702568, 0xc002252fc0}, 0xc002814500)
Apr 12 12:55:19.831	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2047 +0x2f
Apr 12 12:55:19.831	vault	7-2	net/http.HandlerFunc.ServeHTTP(0xc0015a31e0, {0x6702568, 0xc002252fc0}, 0x0)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/http/handler.go:817 +0x2b1
Apr 12 12:55:19.830	vault	7-2	github.com/hashicorp/vault/http.handleRequestForwarding.func1({0x6702568, 0xc002252fc0}, 0xc002814500)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/server.go:2047 +0x2f
Apr 12 12:55:19.830	vault	7-2	net/http.HandlerFunc.ServeHTTP(0x67593f0, {0x6702568, 0xc002252fc0}, 0xc00134a2c0)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/http/logical.go:341 +0xb6
Apr 12 12:55:19.830	vault	7-2	github.com/hashicorp/vault/http.handleLogicalInternal.func1({0x6702568, 0xc002252fc0}, 0xc002814500)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/http/handler.go:910 +0x86
Apr 12 12:55:19.830	vault	7-2	github.com/hashicorp/vault/http.request(0x8741a6, {0x6702568, 0xc002252fc0}, 0xc002814500, 0xc0016be000)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/vault/request_handling.go:420
Apr 12 12:55:19.830	vault	7-2	github.com/hashicorp/vault/vault.(*Core).HandleRequest(...)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/vault/request_handling.go:459 +0x51f
Apr 12 12:55:19.830	vault	7-2	github.com/hashicorp/vault/vault.(*Core).switchedLockHandleRequest(0xc000842800, {0x67593f0, 0xc002253110}, 0xc0016be000, 0xa0)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/vault/request_handling.go:640 +0x152e
Apr 12 12:55:19.830	vault	7-2	github.com/hashicorp/vault/vault.(*Core).handleCancelableRequest(0xc000842800, {0x67593f0, 0xc0022533e0}, 0xc0016be000)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/vault/request_handling.go:1533 +0x2c55
Apr 12 12:55:19.830	vault	7-2	github.com/hashicorp/vault/vault.(*Core).handleLoginRequest(0xc000842800, {0x67593f0, 0xc002253410}, 0xc0016be000)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/vault/core.go:3086 +0xa6
Apr 12 12:55:19.830	vault	7-2	github.com/hashicorp/vault/vault.(*Core).SaveMFAResponseAuth(0x4927ae0, 0xc0017abd10)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/vault-enterprise/vault-enterprise/vault/mfa_auth_resp_priority_queue.go:45 +0x39
Apr 12 12:55:19.830	vault	7-2	github.com/hashicorp/vault/vault.(*LoginMFAPriorityQueue).Push(0x0, 0xc000426150)
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/runtime/panic.go:1038 +0x215
Apr 12 12:55:19.830	vault	7-2	panic({0x49afec0, 0x962f2f0})
Apr 12 12:55:19.830	vault	7-2	/home/runner/actions-runner/_work/_tool/go/1.17.7/x64/src/net/http/h2_bundle.go:5842 +0x125
Apr 12 12:55:19.830	vault	7-2	net/http.(*http2serverConn).runHandler.func1()
Apr 12 12:55:19.830	vault	7-2	goroutine 436954 [running]:
Apr 12 12:55:19.567	vault	7-2	panic serving 45.139.15.115:58894: runtime error: invalid memory address or nil pointer dereference

@vercel vercel bot temporarily deployed to Preview – vault April 12, 2022 19:36 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook April 12, 2022 19:36 Inactive
@vercel vercel bot temporarily deployed to Preview – vault April 12, 2022 21:43 Inactive
@hghaf099 hghaf099 added this to the 1.10.1 milestone Apr 13, 2022
@hghaf099 hghaf099 merged commit e072acf into main Apr 13, 2022
@hghaf099 hghaf099 deleted the forward-login-MFA-requests-from-perf-standby branch April 13, 2022 14:11
Closed
hghaf099 added a commit that referenced this pull request Apr 13, 2022
@hghaf099
forwarding requests subjected to Login MFA to the active node (#15009)
1e3c147 
* forwarding requests subjected to Login MFA to the active node

* CL, and making fmt happy
hghaf099 added a commit that referenced this pull request Apr 13, 2022
@hghaf099
forwarding requests subjected to Login MFA to the active node (#15009) (
eddbbcb 
#15016)

* forwarding requests subjected to Login MFA to the active node

* CL, and making fmt happy
kitography pushed a commit that referenced this pull request Apr 24, 2022
@hghaf099 @kitography
forwarding requests subjected to Login MFA to the active node (#15009)
023be1f 
* forwarding requests subjected to Login MFA to the active node

* CL, and making fmt happy
schultz-is pushed a commit that referenced this pull request Apr 27, 2022
@hghaf099
forwarding requests subjected to Login MFA to the active node (#15009)
fb306fa 
* forwarding requests subjected to Login MFA to the active node

* CL, and making fmt happy
schultz-is pushed a commit that referenced this pull request May 2, 2022
@hghaf099
forwarding requests subjected to Login MFA to the active node (#15009)
a8a088c 
* forwarding requests subjected to Login MFA to the active node

* CL, and making fmt happy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raskchanky raskchanky raskchanky approved these changes

@ncabatoff ncabatoff Awaiting requested review from ncabatoff

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.10.1
Development

Successfully merging this pull request may close these issues.
2 participants
@hghaf099 @raskchanky