Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Grant locking #4117

Merged
merged 59 commits into from
Aug 21, 2023
Merged

Grant locking #4117

merged 59 commits into from
Aug 21, 2023

Conversation

butonic
Copy link
Contributor

@butonic butonic commented Aug 17, 2023 •
edited by micbar
Loading

The jsoncs3 share manager now uses etags instead of mtimes to determine when metadata needs to be updated.
As a precondtition we had to change decomposedfs as well: to consistently calculate the etag for the file content
we now store the mtime in the metadata and use the metadata lock for atomicity.

also fixes:

aduffeck and others added 30 commits August 16, 2023 15:51
@aduffeck @butonic
Improve tracing
9572ac9
@aduffeck @butonic
Lock the node before changing the grants
355dddf
@butonic
log cache diff on sync
4739429 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@aduffeck @butonic
WIP debugging
ef5732b
@aduffeck @butonic
Validae if-unmodified-since before finishing the upload
dfe1534
@aduffeck @butonic
WIP
53f21da
@butonic
etag based cache invalidation
69b39ca 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
providercache seems solid now
acb2853 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
log diffs as before ind after
fa873dd 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@aduffeck @butonic
Remove dead code
511d8a6
@aduffeck @butonic
Fix concurrent map writes
ec2f286
@aduffeck @butonic
Return the new etag after upload
e403ffd
@aduffeck @butonic
Lock on the provider level. fixes concurrent map accesses
fbaf93f
@aduffeck @butonic
Retry more often
f295c15
@aduffeck @butonic
Update in-memory etag with the one from the Upload response
f867426
@aduffeck @butonic
Improve logging
0da73c7
@butonic
ensure etag did not change during reading blob metadata and download
c5cd504 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
move mtime to metadata
855ea9e 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
openlock before reading metadata
7a07920 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
fix etag calculation
a8e599c 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
drop diff, remove unused variable
c1ab979 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
revert info level log from gateway storageprovider
0862c08 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
reduce logging in providercache
e28a818 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
crank up retries when adding shares
2a9f325 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
drop unused mtime in jsoncs3
d847475 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
bring back providercache persist debug log
52b40a6 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
update providercache tests
816daaf 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
update sharecache tests
7ef014c 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
make providercache logging more resilient
4f4daca 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
update jsoncs3 tests
6d5eb15 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic pushed a commit to aduffeck/ocis that referenced this pull request Aug 18, 2023
@aduffeck @butonic
Bump reva to test cs3org/reva#4117
d179013 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
fix unreachable file handle
5d52158 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic pushed a commit to aduffeck/ocis that referenced this pull request Aug 18, 2023
@aduffeck @butonic
Bump reva to test cs3org/reva#4117
440d553 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic pushed a commit to aduffeck/ocis that referenced this pull request Aug 18, 2023
@aduffeck @butonic
Bump reva to test cs3org/reva#4117
264b6d4 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
fix space grant updates
a01e432 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic pushed a commit to aduffeck/ocis that referenced this pull request Aug 18, 2023
@aduffeck @butonic
Bump reva to test cs3org/reva#4117
929942d 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
remove expired shares without checking permissions
dea0e12 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic pushed a commit to aduffeck/ocis that referenced this pull request Aug 18, 2023
@aduffeck @butonic
Bump reva to test cs3org/reva#4117
d68853c 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic added 2 commits August 19, 2023 01:25
@butonic
ignore expired grants when assembling permissions
e6955f9 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
note idea how to aggregate persist requests in jsoncs3
2d20d68 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic pushed a commit to aduffeck/ocis that referenced this pull request Aug 18, 2023
@aduffeck @butonic
Bump reva to test cs3org/reva#4117
9182d2f 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
no need to propagate share expiry as an etag change
2e7b923 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic pushed a commit to aduffeck/ocis that referenced this pull request Aug 21, 2023
@aduffeck @butonic
Bump reva to test cs3org/reva#4117
6985bfe 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic and others added 3 commits August 21, 2023 09:45
@butonic @micbar
Update changelog/unreleased/grant-locking.md
fd4f11e 
Co-authored-by: Michael Barz <michael.barz@zeitgestalten.eu>
@butonic
retry providercache removes more often
9281c2d 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
only retry on recoverable errors
31ad4cd 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic pushed a commit to aduffeck/ocis that referenced this pull request Aug 21, 2023
@aduffeck @butonic
Bump reva to test cs3org/reva#4117
43f5123 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
kobergj
kobergj requested changes Aug 21, 2023
View reviewed changes
Copy link
Contributor

@kobergj kobergj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. Just some questions

pkg/share/manager/jsoncs3/providercache/providercache.go Outdated Show resolved Hide resolved
pkg/storage/utils/metadata/cs3.go Show resolved Hide resolved
pkg/storage/utils/metadata/cs3.go Outdated Show resolved Hide resolved
pkg/storage/utils/metadata/cs3.go Outdated Show resolved Hide resolved
butonic added 3 commits August 21, 2023 15:20
@butonic
unify body close
cb8746f 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
dedupliacate metadata client download code
9f9b996 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic
do not magically retry downloads
1bb6cf7 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
butonic pushed a commit to aduffeck/ocis that referenced this pull request Aug 21, 2023
@aduffeck @butonic
Bump reva to test cs3org/reva#4117
8e62c45 
Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
@butonic butonic merged commit 010c817 into cs3org:edge Aug 21, 2023
@aduffeck aduffeck deleted the grant-locking branch September 4, 2023 06:25
@kobergj kobergj mentioned this pull request Mar 1, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@micbar micbar micbar left review comments

@kobergj kobergj kobergj approved these changes

@aduffeck aduffeck aduffeck approved these changes

@labkode labkode Awaiting requested review from labkode labkode is a code owner

@ishank011 ishank011 Awaiting requested review from ishank011

@glpatcern glpatcern Awaiting requested review from glpatcern glpatcern is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@butonic @aduffeck @micbar @kobergj