-
Notifications
You must be signed in to change notification settings - Fork 34
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Restore improvement: transfers (#4054)
* chore(go.mod): bump SM sub packages versions * feat(rcserver): implement rate limit/transfers change on upload/download This commit allows for greater control over transfers and rate limit as a part of the upload/download rclone API used by backup/restore tasks. It also makes the changes more persistent in case of agent restarts. It also adds the core/transfers endpoint that can be used outside the upload/download rclone API in order to set the amount of transfers for the following rclone API calls. * feat(scyllaclient): add RcloneSet/GetTransfers to client This commit exposes the /rclone/core/transfers agent endpoint via agent client. * feat(scyllaclient): set transfers in RcloneCopyPaths This commit requires changes in the usage of RcloneCopyPaths in restore pkg. It also adds Transfers field to restore Target with the default value -1 meaning that transfers will be set to the amount specified in agent rclone server config. * feat(restore): support max transfers (0) and make it default Our experiments showed that the fastest file download was achieved for transfers=2*2*node_shard_cnt. In order to make it easier for the user to use, a new, special value of transfers=0 will take care of that. It is set as the default, because we aim to make not configured restore as fast as possible. * feat(command/restore): add --transfers flag This commit allows user to control the amount of rclone transfers used during restore. Moreover, during development days of restore task, the default value of --parallel flag was changed from 1 to 0, but we forgot to change it in the flag documentation. * feat(backup): add Transfers to Target This is the first step to control transfers in the context of backup. * feat(scyllaclient): set transfers in rcloneMoveOrCopyDir This commit requires changes in the usage of RcloneMoveDir in backup pkg. * feat(command/backup): add --transfers flag This commit allows user to control the amount of rclone transfers used during backup. * feat(restore_test): extend TestRestoreTablesPreparationIntegration with transfers This way this test also checks transfers before and after backup. It also checks transfers before, in the middle, when paused, when resumed, and after restore. This required some changes to the test like swapping src and dst cluster (increase batch count) and hanging on LAS instead of copy paths (transfer change is applied as a part of copy paths).
- Loading branch information
1 parent
ee47f3d
commit 2099e2e
Showing
48 changed files
with
840 additions
and
131 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.
Oops, something went wrong.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| // Copyright (C) 2024 ScyllaDB | ||
|
|
||
| package rcserver | ||
|
|
||
| import ( | ||
| "context" | ||
| "sync" | ||
|
|
||
| "github.com/pkg/errors" | ||
| "github.com/rclone/rclone/fs" | ||
| "github.com/rclone/rclone/fs/cache" | ||
| "github.com/rclone/rclone/fs/rc" | ||
| "go.uber.org/atomic" | ||
| ) | ||
|
|
||
| // globalConfigGuard is global configGuard that should be used | ||
| // by rc calls when performing global config changes. | ||
| var globalConfigGuard = &configGuard{} | ||
|
|
||
| // configGuard is a tool for performing global config changes. | ||
| // It supports setting transfers and bandwidth limit. | ||
| // It does not re-set config values if they are already | ||
| // set to the desired value. | ||
| type configGuard struct { | ||
| mu sync.Mutex | ||
| initialized atomic.Bool | ||
|
|
||
| defaultTransfers int | ||
| transfers int | ||
| bandwidthLimit string | ||
| } | ||
|
|
||
| func (cg *configGuard) init() { | ||
| if cg.initialized.CompareAndSwap(false, true) { | ||
| defaultTransfers := fs.GetConfig(context.Background()).Transfers | ||
| cg.defaultTransfers = defaultTransfers | ||
| cg.transfers = defaultTransfers | ||
| cg.bandwidthLimit = "" | ||
| } | ||
| } | ||
|
|
||
| // SetTransfers sets global transfers value in rclone config. | ||
| // It also clears fs.Fs cache, so that they can be re-initialized | ||
| // with the new transfers value. | ||
| func SetTransfers(transfers int) error { | ||
| globalConfigGuard.mu.Lock() | ||
| defer globalConfigGuard.mu.Unlock() | ||
| globalConfigGuard.init() | ||
|
|
||
| if transfers == -1 { | ||
| transfers = globalConfigGuard.defaultTransfers | ||
| } | ||
| if transfers <= 0 { | ||
| return errors.Errorf("transfers count must be greater than 0, got %d", transfers) | ||
| } | ||
| // Returns global config | ||
| ci := fs.GetConfig(context.Background()) | ||
| if transfers == globalConfigGuard.transfers { | ||
| // Safety check in case configguard is not in sync with global config | ||
| if transfers == ci.Transfers { | ||
| // Transfers are already set to specified value | ||
| return nil | ||
| } | ||
| } | ||
|
|
||
| globalConfigGuard.transfers = transfers | ||
| ci.Transfers = transfers | ||
| // The amount of transfers impacts fs.Fs initialization (e.g. pool.Pool and fs.Pacer), | ||
| // so fs.Fs cache should be cleared on transfers count change. | ||
| cache.Clear() | ||
| return nil | ||
| } | ||
|
|
||
| // SetBandwidthLimit sets global bandwidth limit in token bucket. | ||
| func SetBandwidthLimit(limit string) error { | ||
| globalConfigGuard.mu.Lock() | ||
| defer globalConfigGuard.mu.Unlock() | ||
| globalConfigGuard.init() | ||
|
|
||
| if limit == globalConfigGuard.bandwidthLimit { | ||
| return nil | ||
| } | ||
|
|
||
| in := rc.Params{ | ||
| "rate": limit, | ||
| } | ||
| _, err := rcCalls.Get("core/bwlimit").Fn(context.Background(), in) | ||
| if err != nil { | ||
| return errors.Wrapf(err, "set bandwidth to %s", limit) | ||
| } | ||
| return nil | ||
| } |
Oops, something went wrong.