Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added: Trying to recover from error state like disk-full. #428

Closed
wants to merge 1 commit into from

Conversation

kunitake
Copy link

This is same following issue.

serf uses bufio library to write the snapshot file.
bufio library can not recover from error state like disk-full unless calling Reset() even if the amount of disk free space is increased.

https://golang.org/pkg/bufio/#Reader.Reset

But Reset() discards any buffered data, resets all state.
If the snapshot file is permissible to lose the old buffer data , this patch to fix this problem .

Best regards,

@simnyc
Copy link

simnyc commented Mar 2, 2017

Hello,

What is the status on this? We're running consul 0.6.4 and are facing this issue as well where the agent does not recover properly from a disk full condition and continuously emits messages like the one below until it is restarted:

serf: failed to flush snapshot: write /mnt/consul/serf/local.snapshot: no space left on device

If this is merged, can it be made available to the 0.6 branch?

Thanks

@slackpad
Copy link
Contributor

slackpad commented Jun 3, 2017

Sorry this hasn't gotten merged yet. I'm worried that this might leave the snapshot corrupt and we will need to do something more to recover from this case, such as starting a new snapshot file, or removing the last line of the current snapshot in case it was cut off. If the snapshot reader can just skip a bad line then we may be ok as well, but I think we need to do some testing before we can pull this in.

@preetapan
Copy link

Closing this, we used the approach implemented in #477

@preetapan preetapan closed this Jul 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants