Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VAULT-5003] Use net/http client in Sys().RaftSnapshotRestore #14269

Merged
merged 12 commits into from
Mar 14, 2022

Conversation

VinnyHC
Copy link
Contributor

@VinnyHC VinnyHC commented Feb 25, 2022

What does this PR do?

Modifies (*client).Sys().RaftSnapshotRestore function to use a net/http client to prevent attempting to read the entire body of a request (which is the case in our (*client).RawRequestWithContext() )
The go-retryablehttp package makes a copy of the request body on construction of a retry-able request to be able to re-play the request. When large snapshots are streamed to the server, this is problematic for the API client’s Sys().RaftSnapshotRestore function.

Any useful background information?

This likely hasn't been identified as a problem in our CLI tool because we use files that go-retryablehttp can handle without reading the full value into memory.
Our (*client).Sys().RaftSnapshot method currently implements a net/http client.

How has this been tested?

Tests have been updated to use helper functions.
Analyzed a memory profile of our current code when writing a "large" snapshot, which copies the entire request body, where the new code does not. Large writes have not been included in the updated tests as to not gobble up unnecessary resources.
"large" data object:

	data := map[string]interface{}{"target": strings.Repeat("a", 1000000)}
	_, err := leaderClient.Logical().Write("secret/big", data)

@VinnyHC VinnyHC force-pushed the vinnyhc/VAULT-5003_retry branch from fc08616 to 73fc27b Compare February 25, 2022 00:11
@vercel vercel bot temporarily deployed to Preview – vault February 25, 2022 00:12 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook February 25, 2022 00:12 Inactive
@VinnyHC VinnyHC requested a review from a team February 25, 2022 00:17
@vercel vercel bot temporarily deployed to Preview – vault-storybook February 25, 2022 16:54 Inactive
@vercel vercel bot temporarily deployed to Preview – vault February 25, 2022 16:55 Inactive
api/sys_raft.go Outdated Show resolved Hide resolved
api/sys_raft.go Outdated Show resolved Hide resolved
api/sys_raft.go Outdated Show resolved Hide resolved
api/sys_raft.go Outdated Show resolved Hide resolved
@vercel vercel bot temporarily deployed to Preview – vault February 25, 2022 17:56 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook February 25, 2022 17:56 Inactive
@vercel vercel bot temporarily deployed to Preview – vault February 25, 2022 18:02 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook February 25, 2022 18:02 Inactive
@VinnyHC VinnyHC force-pushed the vinnyhc/VAULT-5003_retry branch from 9b5a79c to 09b26e4 Compare February 25, 2022 18:18
@vercel vercel bot temporarily deployed to Preview – vault February 25, 2022 18:18 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook February 25, 2022 18:18 Inactive
@VinnyHC VinnyHC force-pushed the vinnyhc/VAULT-5003_retry branch from 09b26e4 to 780782e Compare February 25, 2022 18:38
@vercel vercel bot temporarily deployed to Preview – vault-storybook February 25, 2022 18:38 Inactive
@vercel vercel bot temporarily deployed to Preview – vault February 25, 2022 18:38 Inactive
@VinnyHC VinnyHC changed the title (WIP) [VAULT-5003] Use net/http client in Sys().RaftSnapshotRestore [VAULT-5003] Use net/http client in Sys().RaftSnapshotRestore Feb 25, 2022
@VinnyHC VinnyHC requested a review from averche February 25, 2022 19:08
@vercel vercel bot temporarily deployed to Preview – vault-storybook February 25, 2022 20:22 Inactive
@vercel vercel bot temporarily deployed to Preview – vault February 25, 2022 20:22 Inactive
@VinnyHC VinnyHC force-pushed the vinnyhc/VAULT-5003_retry branch from 19a9775 to 4e329d9 Compare February 28, 2022 22:32
@vercel vercel bot temporarily deployed to Preview – vault-storybook February 28, 2022 22:32 Inactive
@vercel vercel bot temporarily deployed to Preview – vault February 28, 2022 22:32 Inactive
@VinnyHC VinnyHC requested a review from a team February 28, 2022 23:59
api/client.go Outdated Show resolved Hide resolved
api/client.go Outdated Show resolved Hide resolved
vault/external_tests/raft/raft_test.go Show resolved Hide resolved
api/client.go Outdated Show resolved Hide resolved
api/client.go Show resolved Hide resolved
api/client.go Outdated Show resolved Hide resolved
api/client.go Show resolved Hide resolved
@vercel vercel bot temporarily deployed to Preview – vault March 1, 2022 19:14 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook March 1, 2022 19:14 Inactive
@vercel vercel bot temporarily deployed to Preview – vault March 2, 2022 18:56 Inactive
Copy link
Contributor

@averche averche left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One optional comment, LGTM otherwise 👍

api/client.go Outdated Show resolved Hide resolved
@vercel vercel bot temporarily deployed to Preview – vault March 2, 2022 21:34 Inactive
@vercel vercel bot temporarily deployed to Preview – vault-storybook March 2, 2022 21:34 Inactive
api/client.go Outdated Show resolved Hide resolved
Copy link
Collaborator

@ncabatoff ncabatoff left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants