attic remote backup ends in broken pipe

[tobiaslangner]

Whenever I run a backup from my local machine (attic 0.16) to a remote server (attic 0.15), the backup starts nicely, runs for about 1-2 hours and then terminates with

Write failed: Broken pipe
attic: Error: Connection closed by remote host

If I list the contents of the repository later on, I see a checkpoint entry. Rerunning the backup then resumes from the point where the former backup stopped and gets a bit faster but then is also terminated with the same error.

I have tried increasing ServerAliveInterval in ~/.ssh/config for the remote server, but that didn't change much. I have read #233 which talks about a similar problem but if I got it right, it should have been fixed in version 0.15...

What can I do to run proper backups with attic?

[tobiaslangner]

Hmm, is there really nothing I can do? This is really a blocking issue for me as I cannot run a backup through attic. But I really want to do that as I like the software very much, aside from this issue...

[ThomasWaldmann]

Are you sure that this is an attic issue and not just your connection breaking down?
E.g. some DSL providers do a nightly disconnect.

Can you try doing the same backup, but using a local LAN connection?

[tobiaslangner]

I am fairly certain that this is not the issue, as the "broken pipe" happens at any time during the day, not just at night or so. On top of that, I never witnessed any network disconnects on any of the machines in the same network... I will try running it through LAN and get back at you.

[tobiaslangner]

When I run the backup through LAN, the problem does not occur. I did, however, now monitor the network connection during the backup and it is definitely not interrupted at any time during the backup, still the broken pipe occurs. Note also that the error that attic reports is "Connection closed by remote host" which indicates that something on the server side is wrong... Moreover, after this error occurs, I cannot connect to attic (say through attic info or the like) for a few hours. The call just hangs and waits and waits.

I do know that other users are also running attic backups on the same machine, though in different repositories. Could that be relevant to the issue?

[ThomasWaldmann]

That "hang" could be a wait for a lock on that repo maybe.

If the other users really use different repos, I don't think there should be side effects. Just make sure they are really different repos. If they use the same, they will lock the repo as long as they backup to it.

[tobiaslangner]

I am hundred percent sure that the other users use different repos...

[cjk]

Make sure your ssh-connection isn't timeing out. That can happen if your connection is slow/congested and/or you're currently backing up a large file.
I had the same issue until I increased the ssh-timeouts / keep-alive-intervalls and since then no more broken pipes.

[ThomasWaldmann]

@cjk that sounds really strange. I would not expect that ssh can time out while a lot of data is flowing over it and that the keep alive is only needed when there is no data flow otherwise.

[cjk]

@ThomasWaldmann I agree this sounds unintuitive. Though it's an easy change (detailed, for example here and worth a try.

In my case the broken pipe occured on backing up large (VM-) files and since I got no debug-output from attic, it could be well that compressing or other (local) processing (not the actual upload) took too long and SSH disconnected. That is assuming attic doesn't spend all the time transfering data.

If it works better on faster (LAN-?) connections that would rather point to bandwidth-/connection-problems.

[m0r0n]

@cjk I have the same issue and it didn't work to increase the "ClientAliveInterval" on server side. Broken pipe occurs more or less at the same file for me using 0.13 on Debian Jessie and 0.16 on the server, also Jessie.

Edit: Ok, as I just learned in my case it's too few RAM to handle the chunkcache

[tobiaslangner]

I tried the backup again with the settings

ServerAliveInterval 300
ServerAliveCountMax 5

but after some time I get the following error:

Timeout, server xyz.org not responding.
attic: Error: Connection closed by remote host

Note that it is not a broken pipe anymore. It is quite possible that the version of attic on the server got updated. Is there anything I can do on the clientside to solve the issue or to get more debug information?