Skip to content

Commit

Permalink
Add a jsoncs3 share manager (#3148)
Browse files Browse the repository at this point in the history 
* initial jsoncs3 share manager

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* explore and note thoughts

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* more thoughts

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* Lay some groundwork

* WIP

* implement getByKey and getByID

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* GetShare

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* Unshare

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* UpdateShare

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* ListShares without filter

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* DRY up tests

* Extend and cleanup test suite

* Mininmal implementation for ListReceivedShares

* Add a basic cache for the received shares

* UpdateReceivedShare

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* filter by id

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* initial group accessCache

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* thoughts

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* refactor shareCache

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* add todos for permission changes

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* add sharecache implementation

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* use group cache for pending state

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* add plan for storage

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* add storage.Stat() and sketch out refresh from disk

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* read user cache from storage

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* write user share cache on share

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* persist on unshare

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* extract set/removeCreatedCache

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* replace provider space gcache with map

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* drop gcache and unused code

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* Increase test coverage

* Do not choke when the usercache is empty

* Load the created.json when there's no in-memory cache yet

* Add a ProviderCache

* Test that expired UserShareCaches are reloaded properly

* Extract ShareCache into a subpackage

* Extract ShareCache persistence into the sharecache subpackage

* Refactor

* Persist the provider cache

* Update the space mtime when adding/removing shares

* Fix persisting user shares

* Test that GetShare syncs the cache

* Reload cache if a Share wasn't found. Fix unmarshaling shares

* Persist the cache when adding/removing shares

* Implicitly persist the cache when adding/removing entries

* Also implicitly persist the ShareCache when adding/removing items

* Consider the resource id when looking for shares by key

* Add more tests

* Persist updates in the provider cache

* Update the MTime when persisting the caches

* Set the MTime when persisting a UserCache

* Fix reloading the provider cache from disk when outdated

* Cleanup

* Fix syncing the provider cache when listing shares

* Sync the providercache before listing received shares

* Add a cache for the received shares states

* Use the new userreceivedshares cache

* Increase test coverage of UpdateReceivedShare

* Fix listing group shares

* Get the individual shares from the cache instead of listing all of them first

* Fix unmarshaling group shares

* sync the GroupReceivedCache before listing received shares

* Increase test coverage regarding group shares

* Fix the path/filename of the groups received.json file in the storage

* Update docs

* Initialize storage

* Use a different shareid format to not clash with resource ids

* Fix group shares not being merged with the user states

* Fix hound issues

* Remove TODO comments that aren't relevant anymore/yet

* Add support for the IfUnmodifiedSince upload condition

* Do not overwrite newer data on the storage

* Allow for listing shares of other users as space manager

* Bump go-cs3apis

* Fix hound issues

* Log the sync processes

* Add changelog

* Fix even more hound issues

* Remove superfluous sync

* Regenerate mocks

* use utils.NowTS()

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* cleanup cache sync logs

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* introduce utils.TimeToTS

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* convert to timestamp instead of taking time seconds

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* prevent nil when trying to remove not existing grant

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* Update local mtime after writing to the cache

* WIP: use IfUnmodifiedSince flag when writing caches

* comment stat after upload, update docs

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* use new mtime for if-unmodified-since

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* add timestamps to precondition failed error

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* revert mtime change on error

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* add missing headers

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* lint fixes

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

* fix nextcloud mock

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>

Signed-off-by: Jörn Friedrich Dreyer <jfd@butonic.de>
Co-authored-by: Jörn Friedrich Dreyer <jfd@butonic.de>
  • Loading branch information
@aduffeck @butonic
aduffeck and butonic authored Aug 23, 2022
1 parent 15d0666 commit bc50fed
Show file tree
Hide file tree
Showing 25 changed files with 3,125 additions and 27 deletions.
6 changes: 6 additions & 0 deletions changelog/unreleased/jsoncs3-share-manager.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
Enhancement: Add new jsoncs3 share manager

We've added a new jsoncs3 share manager which splits the json file per storage
space and caches data locally.

https://github.com/cs3org/reva/pull/3148
2 changes: 1 addition & 1 deletion go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ require (
github.com/cheggaaa/pb v1.0.29
github.com/coreos/go-oidc v2.2.1+incompatible
github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e
github.com/cs3org/go-cs3apis v0.0.0-20220719130120-361e9f987d64
github.com/cs3org/go-cs3apis v0.0.0-20220818202316-e92afdddac6d
github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8
github.com/dgraph-io/ristretto v0.1.0
github.com/emvi/iso-639-1 v1.0.1
Expand Down
4 changes: 2 additions & 2 deletions go.sum
View file
Original file line number Diff line number Diff line change
Expand Up @@ -170,8 +170,8 @@ github.com/cpuguy83/go-md2man/v2 v2.0.0/go.mod h1:maD7wRr/U5Z6m/iR4s+kqSMx2CaBsr
github.com/creack/pty v1.1.9/go.mod h1:oKZEueFk5CKHvIhNR5MUki03XCEU+Q6VDXinZuGJ33E=
github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e h1:tqSPWQeueWTKnJVMJffz4pz0o1WuQxJ28+5x5JgaHD8=
github.com/cs3org/cato v0.0.0-20200828125504-e418fc54dd5e/go.mod h1:XJEZ3/EQuI3BXTp/6DUzFr850vlxq11I6satRtz0YQ4=
github.com/cs3org/go-cs3apis v0.0.0-20220719130120-361e9f987d64 h1:cFnankJOCWndnOns4sKRG7yzH61ammK2Am6rEGWCK40=
github.com/cs3org/go-cs3apis v0.0.0-20220719130120-361e9f987d64/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY=
github.com/cs3org/go-cs3apis v0.0.0-20220818202316-e92afdddac6d h1:toyZ7IsXlUdEPZ/IG8fg7hbM8HcLPY0bkX4FKBmgLVI=
github.com/cs3org/go-cs3apis v0.0.0-20220818202316-e92afdddac6d/go.mod h1:UXha4TguuB52H14EMoSsCqDj7k8a/t7g4gVP+bgY5LY=
github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8 h1:Z9lwXumT5ACSmJ7WGnFl+OMLLjpz5uR2fyz7dC255FI=
github.com/cubewise-code/go-mime v0.0.0-20200519001935-8c5762b177d8/go.mod h1:4abs/jPXcmJzYoYGF91JF9Uq9s/KL5n1jvFDix8KcqY=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
Expand Down
29 changes: 18 additions & 11 deletions internal/grpc/services/storageprovider/storageprovider.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -302,26 +302,33 @@ func (s *service) InitiateFileUpload(ctx context.Context, req *provider.Initiate
if err != nil {
return nil, err
}
switch sRes.Status.Code {
case rpc.Code_CODE_OK, rpc.Code_CODE_NOT_FOUND:
// Just continue with a normal upload
default:
return &provider.InitiateFileUploadResponse{
Status: sRes.Status,
}, nil
}

metadata := map[string]string{}
ifMatch := req.GetIfMatch()
if ifMatch != "" {
switch sRes.Status.Code {
case rpc.Code_CODE_OK:
if sRes.Info.Etag != ifMatch {
return &provider.InitiateFileUploadResponse{
Status: status.NewAborted(ctx, errors.New("etag mismatch"), "etag mismatch"),
}, nil
}
case rpc.Code_CODE_NOT_FOUND:
// Just continue with a normal upload
default:
if sRes.Info.Etag != ifMatch {
return &provider.InitiateFileUploadResponse{
Status: sRes.Status,
Status: status.NewFailedPrecondition(ctx, errors.New("etag mismatch"), "etag mismatch"),
}, nil
}
metadata["if-match"] = ifMatch
}
ifUnmodifiedSince := req.GetIfUnmodifiedSince()
if ifUnmodifiedSince != nil {
if utils.LaterTS(sRes.Info.Mtime, ifUnmodifiedSince) == sRes.Info.Mtime {
return &provider.InitiateFileUploadResponse{
Status: status.NewFailedPrecondition(ctx, errors.New("resource has been modified"), "resource has been modified"),
}, nil
}
}

ctx = ctxpkg.ContextSetLockID(ctx, req.LockId)

Expand Down
Loading

0 comments on commit bc50fed

Please sign in to comment.