Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade code to use aws sdk go v2 #6486

Merged
merged 59 commits into from
Sep 12, 2023
Merged
Changes from 1 commit
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
3196b0d
oidc blocktype and rate limiter configuration
nopcoder Aug 21, 2023
30d8dc0
kvparams extract configuration params
nopcoder Aug 26, 2023
060799c
remove blank lines
nopcoder Aug 21, 2023
d85509c
wip
nopcoder Aug 21, 2023
d93e992
kvparams extract configuration params
nopcoder Aug 21, 2023
a14993e
fix missing fmt
nopcoder Aug 21, 2023
d3527db
remove blank lines
nopcoder Aug 21, 2023
00a9f0c
wip
nopcoder Aug 22, 2023
51e994b
wip
nopcoder Aug 22, 2023
4f26ee9
wip: code compile, no test, no run
nopcoder Aug 26, 2023
b721eb5
fix no such key
nopcoder Aug 24, 2023
ce41258
improve s3 blocker client cache
nopcoder Aug 25, 2023
1ee0ddf
support non multipart upload api
nopcoder Aug 25, 2023
b5fbc02
Add missing generated files
nopcoder Aug 25, 2023
c0081dd
gateway middleware test filename
nopcoder Aug 26, 2023
6c01a0b
fix err no such key and not found
nopcoder Aug 26, 2023
677f97e
fixing unit tests
nopcoder Aug 27, 2023
e31bb51
dynamodb max conn per host 10
nopcoder Aug 28, 2023
f21425c
fix s3 client cache behaviour
nopcoder Aug 27, 2023
3521484
remove s3 adapter stream
nopcoder Aug 27, 2023
ae4be31
remove streaming check size and timeout
nopcoder Aug 27, 2023
e28b0be
configurable client log mode
nopcoder Aug 27, 2023
436e3f1
Fix logging default test
nopcoder Aug 27, 2023
709a8ef
gtw sig use request context
nopcoder Aug 28, 2023
8182940
refactor sig interface and skip chunk test
nopcoder Aug 28, 2023
1a8300d
default s3 endpoint with http scheme
nopcoder Aug 28, 2023
5840bf8
fix test and apply put without seeker
nopcoder Aug 28, 2023
2036f81
fix test part number type
nopcoder Aug 28, 2023
39d01de
esti handle s3 endpoint with scheme
nopcoder Aug 28, 2023
fce82bb
dont log key and authenticator
nopcoder Aug 28, 2023
05f3ecd
no need to log constants
nopcoder Aug 28, 2023
01ff3fb
without swap compute payload sha256 for unsigned payload
nopcoder Aug 28, 2023
46fd9d3
fix esti build code with wrong go version
nopcoder Aug 29, 2023
999e2ad
Graveler test reduce branch update retry duration for specific test
nopcoder Aug 29, 2023
9132dda
use unsigned payload for put object and upload part
nopcoder Aug 29, 2023
2cc3704
update aws-actions/configure-aws-credentials to v3
nopcoder Aug 29, 2023
35c1836
comment and code style
nopcoder Aug 29, 2023
c7f5bf8
fix upload helper uses separator as part of key prefix
nopcoder Aug 30, 2023
e13681c
use NoBody on zero size content
nopcoder Aug 31, 2023
858b73a
upload object update api swagger order revert client sdk changes
nopcoder Aug 31, 2023
1ccc476
metadata client use web identity configuration
nopcoder Sep 1, 2023
19b787e
apply code review changes and additional lint issues
nopcoder Sep 5, 2023
dc5ee48
missing parts from previous commit
nopcoder Sep 5, 2023
ce58347
remove web identify from cloud metadata client
nopcoder Sep 5, 2023
b7c67d9
apply code review changes
nopcoder Sep 5, 2023
4d0c693
remove inventory
nopcoder Sep 5, 2023
0577667
Handle server value from put requests
nopcoder Sep 6, 2023
e081750
move sse header extract function to a different file
nopcoder Sep 6, 2023
9a0e575
limit upload to one attempt
nopcoder Sep 7, 2023
cd7a211
fix controller after rebase
nopcoder Sep 8, 2023
433e52c
fix rebase compilation error
nopcoder Sep 8, 2023
4940464
go mod tidy
nopcoder Sep 8, 2023
872dc6f
put max retry based on seekable reader
nopcoder Sep 8, 2023
dcae21e
Update docs and remove code of unuseed streaming_chunk_size and strea…
nopcoder Sep 10, 2023
d6f1e39
remove viper defaults
nopcoder Sep 10, 2023
914a53c
Presigned url expiry value consider credentials expiry
nopcoder Sep 10, 2023
e712b20
apply code review change
nopcoder Sep 11, 2023
955ac3f
Use session expiry window as part of expiry adjust
nopcoder Sep 11, 2023
204c1f4
Update comments based on code review
nopcoder Sep 12, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 30 additions & 8 deletions pkg/block/s3/adapter.go
Original file line number Diff line number Diff line change
Expand Up @@ -349,14 +349,25 @@ func (a *Adapter) GetWalker(uri *url.URL) (block.Walker, error) {
return NewS3Walker(a.clients.GetDefault()), nil
}

type CaptureExpiresPresigner struct {
Presigner s3.HTTPPresignerV4
CredentialsCanExpire bool
CredentialsExpireAt time.Time
}

func (c *CaptureExpiresPresigner) PresignHTTP(ctx context.Context, credentials aws.Credentials, r *http.Request, payloadHash string, service string, region string, signingTime time.Time, optFns ...func(*v4.SignerOptions)) (url string, signedHeader http.Header, err error) {
// capture credentials expiry
c.CredentialsCanExpire = credentials.CanExpire
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this could just capture credentials and make them available later on. No need to run the processing logic here. I prefer to keep the signing path as simple and unmodified as possible.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was worried about capturing something that holds a state that can change and instead capture the same information the signer accepted.
Let me know if you think. if the risk is low I'll update the code.

c.CredentialsExpireAt = credentials.Expires
return c.Presigner.PresignHTTP(ctx, credentials, r, payloadHash, service, region, signingTime, optFns...)
}

func (a *Adapter) GetPreSignedURL(ctx context.Context, obj block.ObjectPointer, mode block.PreSignMode) (string, time.Time, error) {
if a.disablePreSigned {
return "", time.Time{}, block.ErrOperationNotSupported
}

// TODO(barak): handle expiry window of the client credentials when pre-signed
// support enabled
expiry := time.Now().Add(a.preSignedExpiry)
expiry := time.Now().UTC().Add(a.preSignedExpiry)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

UTC is a timezone. The best kind of timezone, but still a timezone. Unless Golang defaults yo something line SQL TIMESTAMP (which is without TIMEZONE), I don't think UTC is needed or required here.
Timezones belong when formatting time, not when processing it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

true, I will remove it.
it was easy to debug as the aws credentials expire time was in this tz.


log := a.log(ctx).WithFields(logging.Fields{
"operation": "GetPreSignedURL",
Expand All @@ -371,28 +382,39 @@ func (a *Adapter) GetPreSignedURL(ctx context.Context, obj block.ObjectPointer,
}

client := a.clients.Get(ctx, bucket)
presigner := s3.NewPresignClient(client, func(options *s3.PresignOptions) {
options.Expires = a.preSignedExpiry
})
presigner := s3.NewPresignClient(client,
func(options *s3.PresignOptions) {
options.Expires = a.preSignedExpiry
})

captureExpiresPresigner := &CaptureExpiresPresigner{}
var req *v4.PresignedHTTPRequest
if mode == block.PreSignModeWrite {
putObjectInput := &s3.PutObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
req, err = presigner.PresignPutObject(ctx, putObjectInput)
req, err = presigner.PresignPutObject(ctx, putObjectInput, func(o *s3.PresignOptions) {
captureExpiresPresigner.Presigner = o.Presigner
o.Presigner = captureExpiresPresigner
})
} else {
getObjectInput := &s3.GetObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(key),
}
req, err = presigner.PresignGetObject(ctx, getObjectInput)
req, err = presigner.PresignGetObject(ctx, getObjectInput, func(o *s3.PresignOptions) {
captureExpiresPresigner.Presigner = o.Presigner
o.Presigner = captureExpiresPresigner
})
}
if err != nil {
log.WithError(err).Error("could not pre-sign request")
return "", time.Time{}, err
}
if captureExpiresPresigner.CredentialsCanExpire && captureExpiresPresigner.CredentialsExpireAt.Before(expiry) {
expiry = captureExpiresPresigner.CredentialsExpireAt
}
return req.URL, expiry, nil
}

Expand Down
Loading