-
Notifications
You must be signed in to change notification settings - Fork 39
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Sync fails when files being naively written to #24
Comments
Interesting problem. I'll think about if its safe to simply skip a file that fails the rclone transfer. For the individual file copy from path2 to path1 I know which file failed, but for the final rclone sync from path1 to path2 it seems unsafe. Hum... Also see issue #8. Note that there is a risk of data loss with --first-sync. Newer versions of files on Path2 will be lost. Be careful. I'm out of town for a bit. Perhaps next week I'll get back to this. |
Perhaps you could have such files to the filters file and entirely ignore them? |
Background: I'm trying to set up
rclonesync.py
for an HPC environment, meaning that good I/O practices can't be guaranteed. Users who don't know how to avoid hammering files, or who of necessity use programs that are not designed for HPC/don't handle I/O properly, could be writing to files many times per second at sync time.When a file is being constantly written to, though, running
rclonesync.py
fails hard enough that it must be rerun with--first-sync
before it can be run normally again. I can understand why it would fail--a file that's getting hammered isn't a good candidate for syncing--but is there anything preventing the failure from being graceful, i.e. allowing the next try to succeed without requiring--first-sync
? I know that bidirectional syncing is a hard problem, so I may be missing something obvious that means one simply has to be careful not to engage in foolishness likefor i in {1..99999}; do echo $i >> myfile; done
while syncing. However, if there is a way to more gracefully handle such cases, it would mean a more robustrclonesync.py
.The following case illustrates my point. In essence, I run rclonesync.py while a local file is being written to 100 times per second, and cause a critical error abort:
Is this fixable? If not, do you have any suggestions on how to handle cases like this? Since adding
--first-sync
does make the issue at least superficially go away (at least after the file hammering is finished), what would be the implications of using--first-sync
every timerclonesync.py
is run?The text was updated successfully, but these errors were encountered: