RClone and LakeFS integration breaks without v2_auth #2445
Assignees
Labels
area/auth
IAM, authorization, authentication, audit, AAA, and integrations with all those
contributor
next
Target for next release
Comments
|
Thanks for the detailedreport! This should work. Edited to add: ...so I shall reproduce and attempt to diagnose. |
|
I'm sorry, I am unable to reproduce this with 1.56.0 and lakeFS 0.48.0. I am able to upload (and download) files, using SigV4, onto lakeFS. I verified sigV4 using My only issues were operator error (I don't really know how to use rclone...), I was unable to use it to create a new file, only to overwrite an existing file. I do not believe that this is related to your issue. Would appreciate more info, e.g. (redacted) lakeFS and rclone configuration files, the |
|
No problem. Rclone Config: LakeFS Config: |
|
Just to add I've also found some more odd behavior trying to copy data into LakeFS. If the filename has unicode characters it fails with the same message however works with files without the characters. Dumping the logs it seems RClone sends a HEAD request in to see if the file exists already and that's what LakeFS returns the 403 on. Oddly enough it sends that same HEAD requests for all the files which authorizes fine when those characters don't exist. The UI doesn't seem to mind them manually adding it nor does it have an issue with the PUT command which I tested with |
|
I remain unable to reproduce. Hmmm... perhaps MacOS? Will try to hunt down a Mac and try there. Update: Still works on MacOS. I configured a repo, hooked up rclone using: and copied a file to the name I am sorry, I am repeatedly drawing a blank on this. We know this is a big request, but having full debug logs (from both or either sides) might be helpful. E.g. if you can capture the output from |
|
Hmm interesting. Maybe then it's an issue with how my backend bucket is configured? Here is the dump of all those sections on the same requests I've been trying. With the other auth method. This one seems to fail on the put and not head.
|
|
Did also reverify that I can upload files without those characters which it seems like RClone is doing the same commands and that the user for the lakefs backing buckets has all S3 operations. |
Reproduction of SigV2 failure using the AWS CLIRequirementsAn aws v1 CLI. v2 of the CLI has already dropped support for SigV2. Configure your profile to use SigV2: Put this stanza in Now attempt to copy a file to something with a non-ASCII name: |
|
The value of Line 186 in ac76250 ""/moo/main/ש" when the original encoded value "/moo/main/%D7%A9" is needed.
|
|
So this diff appears to work. I will do some historical research to see why we had that code in the first place. diff --git a/pkg/gateway/sig/v2.go b/pkg/gateway/sig/v2.go
index 9e3c238c5..299666150 100644
--- a/pkg/gateway/sig/v2.go
+++ b/pkg/gateway/sig/v2.go
@@ -238,17 +238,11 @@ func (a *V2SigAuthenticator) Verify(creds *model.Credential, bareDomain string)
- QSA(Query String Arguments) - query arguments are searched for "interesting Resources".
*/
- /*
- URI encoding requirements for aws signature are different from what GO does.
- This logic is taken from https://docs.aws.amazon.com/AWSECommerceService/latest/DG/Query_QueryAuth.html
- These replacements are necessary for Java. There is no description about GO, but I found the '=' needs treatment as well
- */
+ // Prefer the raw path if it exists -- *this* is what SigV2 signs
+ url := a.r.URL
+ rawPath := url.EscapedPath()
- patchedPath := strings.ReplaceAll(a.r.URL.Path, "=", "%3D")
- patchedPath = strings.ReplaceAll(patchedPath, "+", "%20")
- patchedPath = strings.ReplaceAll(patchedPath, "*", "%2A")
- patchedPath = strings.ReplaceAll(patchedPath, "%7E", "~")
- path := buildPath(a.r.Host, bareDomain, patchedPath)
+ path := buildPath(a.r.Host, bareDomain, rawPath)
stringToSign := canonicalString(a.r.Method, a.r.URL.Query(), path, a.r.Header)
digest := signCanonicalString(stringToSign, []byte(creds.SecretAccessKey))
if !Equal(digest, a.ctx.signature) { |
Update#2464 in draft, based on the above, makes SigV2 handle these Unicode characters correctly. I am currently adding a gateway signature component test so as to have more confidence that this is indeed correct. It turns out that SigV2 is not only deprecated-but-in-use, it is also documented in subtly different terms for S3 and for (most of) the rest of AWS. Fun times! |
|
Thanks for running this down @arielshaqed! |
Thanks for finding these bugs and providing all needed information, and to @talSofer for reproducing (most Linux terminal emulators manage to strip Unicode before it ever hits the command-line programs...)! Please note that #2464 should fix SigV2. I do not expect any change to SigV4 authentication; indeed, I do not believe that we have managed to reproduce the SigV4 issue yet. So I am not closing the bug yet. I hope to release next week. We are not many few hours away from a fairly large holiday; releasing today would be a bit too adventurous even for me. 🤷♂️ |
|
Sounds good! I think the |
arielshaqed
added
area/auth
|
@arielshaqed I was able to upgrade to the latest release and the unicode error is indeed fixed for |
|
Interestingly is even though the unicode characters work I get the same signature mistmatch when it tries to do a multipart upload.. I set the |
Thanks for another great analysis! I can see an ETag mismatch but not a signature mismatch:
This is some very odd behaviour, I am not sure that If you still see specifically signature failures and not the ETag MD5 difference that I see, please open another one (or just re-open this bug). |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This might be intended and just needs an update on your website but I ran into a hard to debug issue trying to sync data into LakeFS with RClone. Note, I've followed the instructions previously on the site and this worked so not sure what LakeFS version it broke in. The behavior seems to point towards the client side however I haven't updated RClone at all since it did previously work.
LakeFS Version:
0.48.0(Also reproduced on0.47.0)RClone Version:
Error trying to copy local data into LakeFS using the suggested configuration:
To test that the credentials were correct I went the other way and used LakeFS as a source after adding a file via the UI
What put me onto it was the signature signing is this log line in the LakeFS stating it was using SigV4.
time="2021-09-03T01:51:31Z" level=warning msg="error verifying credentials for key" func=pkg/gateway.AuthenticationHandler.func1 file="build/pkg/gateway/middleware.go:54" authenticator=sigv4 error=SignatureDoesNotMatch key=AKIAJ6UDLXIPOISF7LKQAlso verified by dumping the RClone headers that it was using SigV4 authentication by dumping the headers.
Setting the V2 auth in rclone does fix this issue:
It's an easy enough work around to add
v2_auth = true, however I wanted to report it in case it is a bug since you have SigV4 listed as supported hereThe text was updated successfully, but these errors were encountered: