Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

irmin-pack.unix: review usage of fsync and add where needed #1971

Closed
metanivek opened this issue Jul 7, 2022 · 2 comments
Closed

irmin-pack.unix: review usage of fsync and add where needed #1971

metanivek opened this issue Jul 7, 2022 · 2 comments
Labels
feature/layered-store Related to the Layered Store

Comments

@metanivek
Copy link
Member

Let's review our usage of fsync in the new gc code and modify as needed.

@metanivek metanivek added the feature/layered-store Related to the Layered Store label Jul 7, 2022
@metanivek metanivek changed the title irmin-pack: review usage of fsync and add where needed irmin-pack.unix: review usage of fsync and add where needed Jul 7, 2022
@samoht
Copy link
Member

samoht commented Jul 18, 2022

From @icristescu and @metanivek discussion earlier today, a possible plan for crash consistency:

  • call fsync after snapshot import;
  • call fsync at the end of the gc, in the gc child process, to guarantee that all files created/updated by the gc are consistent
  • the gc needs to copy the commit object in the prefix as well (currently it is the first object in the suffix).
  • at startup if files are not in a consistent state:
    • check for the previous generation files;
    • ignore the suffix
    • reconstruct control file: from the mapping, and fstat dict.
    • restart the node from the commit in the prefix
  • the guarantee is that if the first gc succeeds then we have always a set of files (prefix, mapping and dict) in a consistent state from which we can build the control file and start with a fresh suffix.

@metanivek
Copy link
Member Author

the gc needs to copy the commit object in the prefix as well (currently it is the first object in the suffix).

will happen as a part of chunking the suffix. I moved the rest of the list to a new issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/layered-store Related to the Layered Store
Projects
None yet
Development

No branches or pull requests

2 participants