Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add etcd s3 config secret implementation #10340

Merged
merged 1 commit into from
Jul 10, 2024

Conversation

brandond
Copy link
Member

@brandond brandond commented Jun 11, 2024

Proposed Changes

Add implementation of etcd snapshot s3 config secret support, as covered by the ADR from #9364

  • Move snapshot structs and functions into pkg/etcd/snapshot
  • Move s3 client code and functions into pkg/etcd/s3
  • Refactor pkg/etcd to track snapshot and s3 moves
  • Add support for reading s3 client config from secret
  • Add minio client cache, since S3 client configuration can now be changed at runtime by modifying the secret, and don't want to have to create a new minio client every time we read config.

Types of Changes

New feature

Verification

See ADR for intended use

Testing

Yes, S3 integration test is updated

Linked Issues

User-Facing Change

A proxy can now be configured for use when uploading etcd snapshots to a s3-compatible storage service. This overrides any proxy settings passed via environment variables.
Credentials and endpoint configuration for storing etcd snapshots on a s3-compatible storage service can now be read from a Secret, instead of passing them via the CLI or config file. See https://github.com/k3s-io/k3s/blob/master/docs/adrs/etcd-s3-secret.md for more information.

Further Comments

Copy link

codecov bot commented Jun 11, 2024

Codecov Report

Attention: Patch coverage is 48.58841% with 346 lines in your changes missing coverage. Please review.

Project coverage is 43.58%. Comparing base (047664b) to head (5c55247).

Files Patch % Lines
pkg/etcd/snapshot/types.go 6.55% 114 Missing ⚠️
pkg/etcd/snapshot.go 4.08% 94 Missing ⚠️
pkg/etcd/s3/s3.go 80.58% 39 Missing and 21 partials ⚠️
pkg/etcd/etcd.go 13.79% 23 Missing and 2 partials ⚠️
pkg/etcd/snapshot_handler.go 0.00% 22 Missing ⚠️
pkg/cli/server/server.go 0.00% 13 Missing and 1 partial ⚠️
pkg/etcd/snapshot_controller.go 0.00% 10 Missing ⚠️
pkg/etcd/s3/config_secret.go 87.50% 6 Missing and 1 partial ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master   #10340      +/-   ##
==========================================
- Coverage   47.69%   43.58%   -4.11%     
==========================================
  Files         177      179       +2     
  Lines       14830    14915      +85     
==========================================
- Hits         7073     6501     -572     
- Misses       6407     7212     +805     
+ Partials     1350     1202     -148     
Flag Coverage Δ
e2etests 36.38% <2.37%> (-10.01%) ⬇️
inttests 36.69% <0.59%> (+16.77%) ⬆️
unittests 13.38% <46.50%> (+2.03%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@brandond brandond force-pushed the etcd-s3-secret-impl branch 3 times, most recently from 478031c to 2385ab0 Compare June 12, 2024 23:03
@brandond brandond marked this pull request as ready for review June 12, 2024 23:03
@brandond brandond requested a review from a team as a code owner June 12, 2024 23:03
@brandond brandond force-pushed the etcd-s3-secret-impl branch 15 times, most recently from ec9bd75 to 270d540 Compare June 18, 2024 20:05
galal-hussein
galal-hussein previously approved these changes Jul 9, 2024
* Move snapshot structs and functions into pkg/etcd/snapshot
* Move s3 client code and functions into pkg/etcd/s3
* Refactor pkg/etcd to track snapshot and s3 moves
* Add support for reading s3 client config from secret
* Add minio client cache, since S3 client configuration can now be
  changed at runtime by modifying the secret, and don't want to have to
  create a new minio client every time we read config.
* Add tests for pkg/etcd/s3

Signed-off-by: Brad Davidson <brad.davidson@rancher.com>
@brandond brandond merged commit c36db53 into k3s-io:master Jul 10, 2024
29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants