Added: Trying to recover from error state like disk-full.

[kunitake]

This is same following issue.

• Snapshot open file descriptor permanently unusable after "filesystem full" condition #1744
• Added: Trying to recover from error state like disk-full. #2236

serf uses bufio library to write the snapshot file.
bufio library can not recover from error state like disk-full unless calling Reset() even if the amount of disk free space is increased.

https://golang.org/pkg/bufio/#Reader.Reset

But Reset() discards any buffered data, resets all state.
If the snapshot file is permissible to lose the old buffer data , this patch to fix this problem .

Best regards,

[simnyc]

Hello,

What is the status on this? We're running consul 0.6.4 and are facing this issue as well where the agent does not recover properly from a disk full condition and continuously emits messages like the one below until it is restarted:

serf: failed to flush snapshot: write /mnt/consul/serf/local.snapshot: no space left on device

If this is merged, can it be made available to the 0.6 branch?

Thanks

[slackpad]

Sorry this hasn't gotten merged yet. I'm worried that this might leave the snapshot corrupt and we will need to do something more to recover from this case, such as starting a new snapshot file, or removing the last line of the current snapshot in case it was cut off. If the snapshot reader can just skip a bad line then we may be ok as well, but I think we need to do some testing before we can pull this in.

[preetapan]

Closing this, we used the approach implemented in #477