Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Local sync does not resume partial transfer #570

Open
Lobstros opened this issue Aug 1, 2021 · 13 comments · May be fixed by #571 or #572
Open

Local sync does not resume partial transfer #570

Lobstros opened this issue Aug 1, 2021 · 13 comments · May be fixed by #571 or #572
Labels
defect unison fails to meet its specification (but doesn't crash; see also "crash") effort-medium issue is likely resolvable with <= 20h of effort feedback Information has been requested; may be closed in 30 days if not provided. impact-medium medium importance

Comments

@Lobstros
Copy link

Lobstros commented Aug 1, 2021

Hey,

I seem to have tripped over some odd opinions that Unison has about behaviour of remote vs local syncing!

To elaborate, I want to sync files to and from a local FUSE mount; the code behind this mount then transparently encrypts them and sends them to a cloud store. However Unison appears to unexpectedly have separate code paths for remote and ostensibly "local" syncs:

  • Resuming partial transfers works flawlessly remotely, but mostly does not work locally:
    • When the base directory does not yet exist at the target, it restarts the entire directory.
    • When the base directory already exists at the target, it recognises completed files and folders one level down, but restarts any of these that were partially transferred.
  • Hitting ctrl-c (SIGINT, I think?) on the terminal cancels the whole process when remote syncing, but when local syncing just stops the current file and moves on to the next, leaving partial files.

I think I see in the tracker others stumbling into related differences (e.g. #472), though not these two specifically.

If fixing them would be very difficult, I am satisfied it's possible to work around by making Unison SSH to my local machine (thus forcing "remote" behaviour). However the "local" path appears to contradict what's in the manual (which is maybe written from a "remote-only" perspective?), so perhaps it could be handy to have a prominent warning somewhere about it.

@gdt gdt added the feedback Information has been requested; may be closed in 30 days if not provided. label Aug 1, 2021
@gdt
Copy link
Collaborator

gdt commented Aug 1, 2021

Please either close this and bring it up on unison-users or edit to be a crisp description of incorrect behavior vs the documentation, preferably with a repro recipe. And, if you can separate into multiple incorrect behaviors, please do. I think you have found a real issue and there is a bug lurking here -- but we're trying to have the tracker be clearly-articulated bugs and feature requests only. (I could guess at one, but I don't want to guess.)

@Lobstros
Copy link
Author

Lobstros commented Aug 1, 2021

Ok, no problem. Here's a few commands that should exhibit it.

Setup

Check that an SSHD is running locally with the current user's public key in authorized_keys.

Generate 1GB of fake data (100 × 10MB files) to transfer:

mkdir /tmp/test-source
for n in $(seq 1 100); do
    dd if=/dev/urandom of=/tmp/test-source/random-data-$n.bin bs=1M count=100
done

Verify expected "remote" behaviour:

unison -auto -batch /tmp/test-source ssh://localhost//tmp/test-destination-remote
  • "SIGINT" issue: ctrl-c this before complete; it should terminate properly.
  • "Resume" issue: Rerun the above command; it should resume from partial completion.

Reproduce aberrant "local" behaviour:

unison -auto -batch /tmp/test-source /tmp/test-destination-local
  • "Resume" issue: ctrl-z before completion, kill the background job, and repeat the command. It will restart from 0% each time.
  • "SIGINT" issue: Hit ctrl-c (after investigating the first issue); it doesn't stop the overall process.
    • N.B. if the transfer completes, state will be stored in ~/.unison indicating that it's not a first-time transfer, and I had to grep and remove files after this to reproduce either issue cleanly.

Let me know if this is enough or you want it redone somehow.

@Lobstros
Copy link
Author

Lobstros commented Aug 1, 2021

The part of the manual that this seemed to clash with is under "Making Unison faster on large files":

If a directory transfer is interrupted, the next run of Unison will automatically skip any files that were completely transferred before the interruption. (This behavior is always on: it does not depend on the setting of the copythreshold preference.) Note, though, that the new directory will not appear in the destination filesystem until everything has been transferred—partially transferred directories are kept in a temporary location (with names like .unison.DIRNAME....) until the transfer is complete.

It is in a section about customizing how Unison runs, although the wording seems to suggest that resume from partial transfer happens regardless.

(I notice now that this link could also be an old revision of the docs which I just happened to find when googling; do point me in the right direction if it has since changed)

@gdt gdt added defect unison fails to meet its specification (but doesn't crash; see also "crash") effort-medium issue is likely resolvable with <= 20h of effort impact-medium medium importance and removed feedback Information has been requested; may be closed in 30 days if not provided. labels Aug 1, 2021
@gdt
Copy link
Collaborator

gdt commented Aug 1, 2021

Thanks, that's clear and agreed the local behavior is contrary to docs (and to what obviously ought to be). Can you add a comment with unison version, ocaml version, OS type and version? (I don't expect this is ok with the most recent, but we only deal with the most recent release.)

@Lobstros
Copy link
Author

Lobstros commented Aug 1, 2021

Sure thing:

$ unison -version
unison version 2.51.4 (ocaml 4.12.0)

@Lobstros
Copy link
Author

Lobstros commented Aug 1, 2021

Oops, just missed the part about OS. I'm on Ubuntu 20.04.

@tleedjarv
Copy link
Contributor

I can confirm both issues that you've observed. And yes, there is a slightly different code path taken when copying files locally vs remotely.

As for the SIGINT issue, I propose to handle it separately. There already is #554; perhaps that can serve as the umbrella ticket for this specific issue.

This ticket can remain about the resume issue.

@tleedjarv tleedjarv linked a pull request Aug 2, 2021 that will close this issue
@tleedjarv
Copy link
Contributor

I've proposed a patch for the resume issue.
@Lobstros could you test it with your test cases? You can grab a build here or feel free to build yourself with the patch applied.

@Lobstros
Copy link
Author

Lobstros commented Aug 2, 2021

Thanks for looking at this! I can confirm that the linked build appears to recognise files that have already been transferred in a previously aborted "first-time" local sync. The terminal shows the following:

Unison 2.51.4 (ocaml 4.12.0) started propagating changes at 20:47:57.42 on 02 Aug 2021
[BGN] Copying  from /tmp/test-source to /tmp/test-destination-local2
/tmp/test-destination-local2/random-data-1.bin has already been transferred
/tmp/test-destination-local2/random-data-10.bin has already been transferred
/tmp/test-destination-local2/random-data-100.bin has already been transferred
/tmp/test-destination-local2/random-data-11.bin has already been transferred

etc.

@tleedjarv tleedjarv linked a pull request Aug 2, 2021 that will close this issue
@tleedjarv
Copy link
Contributor

@Lobstros if you are up for some more testing, there is now a different patch available. You can grab a build here.

@Lobstros
Copy link
Author

Lobstros commented Aug 2, 2021

@tleedjarv : Yep, seems to work fine still.

@gdt
Copy link
Collaborator

gdt commented Mar 19, 2023

@Lobstros can you test #572? It seems like we might be able to make progress here.

@gdt gdt added the feedback Information has been requested; may be closed in 30 days if not provided. label Mar 19, 2023
@gdt
Copy link
Collaborator

gdt commented Oct 9, 2023

@Lobstros Are you able to test #572?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defect unison fails to meet its specification (but doesn't crash; see also "crash") effort-medium issue is likely resolvable with <= 20h of effort feedback Information has been requested; may be closed in 30 days if not provided. impact-medium medium importance
Projects
None yet
3 participants