command: add `sync` command. #340

ocakhasan · 2021-08-12T09:04:30Z

Sync command is implemented. The use case is similar to aws-cli and s3cmd.
Usage:

s5cmd sync <src> <target>

--delete flag can be used to delete the object belongs to remote but does not belong to source.
Sync command works in this way.

Get all of the source and destination objects.
Determine which ones belongs to only source, which ones only destination and which ones are in common both source and destination in parallel.
Only source objects are directly copied. There is no need to check if the object should be overridden.
Common objects (both in source and destination) needs to be checked to see if the corresponding destination object should be overridden.
sync command checks size and modification time default. If any of those is different, then the destination object should be overridden. However, if --size-only flag is set, then size becomes the only criteria in override operations.
Only destination objects are deleted if --delete flag is set.

Resolves #3

… validateCopy

… from storage

* add --size-only and --checksum flag logic to validation * change errors in storage/hash

* moved hash errors to error package * added usage examples to sync * created test cases for sync with --checksum flag

* deleted the deleteSource parameter in newSync function.

kuasha420 · 2021-08-26T05:39:58Z

Amazing work!

…p etc) * add test with wildcard operation

* delete --checksum option from sync * create an interface Strategy to compare objects * delete tests which includes --checksum flag * delete hash.go and hash_test.go

… and upload operations * move transfer logic into its own package * create a new internal folder to store all of internal packages used

* clean the unnecessary comments.

* Add comments to exported functions or structs

* add source to algorithm comment.

* delete --checksum and --size-only logic together

igungor · 2021-09-20T13:31:16Z

Current plan is to go with another approach: #347.

ocakhasan added 6 commits August 13, 2021 18:44

first commit sync with lists

899c10a

add --size-only and --delete flag to sync

f2a77cc

fix exported function names and add validateSyncCopy instead of using…

7fca122

… validateCopy

delete unused parameters

f51abcc

add comment and fix some line

757df70

create mock storage for new function ListSlice and fix some bugs in sync

cd0dc88

ocakhasan force-pushed the sync branch from 0bcc042 to cd0dc88 Compare August 16, 2021 06:54

ocakhasan added 13 commits August 16, 2021 11:52

use expandSource instead of listslice

7cf739c

change validation comments

538996b

remove unused parameters

ecf199f

get objects from channel then append to slice, delete listlice method…

0cba615

… from storage

delete unused senderrortoSlice method

42722e4

remove unused expandSourceList

9064392

check files with filepath.ToSlash to handle windows related issues

02a3ed2

change gofakes3 to get the sizes of objects

4a0a5ff

Merge branch 'master' into sync

f127013

currently working gofakes3

d2402f5

update igungor/gofakes3, fix ls and du tests and add sync tests

c8dd156

add remote -> remote tests

44dfa65

change last statement of tests with delete

fd95702

ocakhasan marked this pull request as ready for review August 18, 2021 11:34

ocakhasan requested a review from a team as a code owner August 18, 2021 11:34

ocakhasan requested review from igungor and sonmezonur and removed request for a team August 18, 2021 11:34

ocakhasan added 4 commits August 20, 2021 09:41

update gofakes3 and create hash

135b23b

Create --checksum flag which is used for checking file hashes.

ed2f73f

* add --size-only and --checksum flag logic to validation * change errors in storage/hash

fix staticcheck errors

ab00341

Add tests to sync with -checksum options

4d261f1

* moved hash errors to error package * added usage examples to sync * created test cases for sync with --checksum flag

ilkinulas requested a review from seruman August 22, 2021 14:21

ilkinulas requested review from ilkinulas and aykutfarsak August 22, 2021 14:21

ocakhasan and others added 2 commits August 23, 2021 15:59

Changed file structures to more nested in sync_test.go

2d10cbf

* deleted the deleteSource parameter in newSync function.

Merge branch 'peak:master' into sync

78b610c

ocakhasan added 7 commits August 27, 2021 09:50

Remove errors to be appended when creating channel objects (same as c…

7c28b35

…p etc) * add test with wildcard operation

Create new version of sync

3a636e9

* delete --checksum option from sync * create an interface Strategy to compare objects * delete tests which includes --checksum flag * delete hash.go and hash_test.go

Create a new transfer manager which will handle all of copy, download…

39751ff

… and upload operations * move transfer logic into its own package * create a new internal folder to store all of internal packages used

fix a windows path bug with filepath slash

2b360a3

* clean the unnecessary comments.

Use relative path instead of custom objectPath function

530aab1

* Add comments to exported functions or structs

Use sort.SliceStable instead of sort.Interface

40fd069

* add source to algorithm comment.

Delete unused errors

21210ac

* delete --checksum and --size-only logic together

igungor closed this Sep 20, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

command: add `sync` command. #340

command: add `sync` command. #340

ocakhasan commented Aug 12, 2021 •

edited

Loading

kuasha420 commented Aug 26, 2021

igungor commented Sep 20, 2021

command: add sync command. #340

command: add sync command. #340

Conversation

ocakhasan commented Aug 12, 2021 • edited Loading

kuasha420 commented Aug 26, 2021

igungor commented Sep 20, 2021

command: add `sync` command. #340

command: add `sync` command. #340

ocakhasan commented Aug 12, 2021 •

edited

Loading