Skip to content
This repository was archived by the owner on Oct 18, 2023. It is now read-only.

flush frames and confirm snapshot completion before calling SQLite checkpoint #704

Merged
merged 2 commits into from
Sep 26, 2023

Conversation

Horusiath
Copy link
Contributor

This solves the issue when SQLite files (DB + WAL) are being modified before we confirm that they were backed up.

PS: don't merge yet, there's still an ongoing issue with tokio task never being scheduled causing a deadlock on subsequent checkpoints:

let handle = tokio::spawn(async move {
let start = Instant::now();
let body = match Self::maybe_compress_main_db_file(db_file, compression).await {
Ok(file) => file,
Err(e) => {
tracing::error!(
"Failed to compress db file (generation {}): {}",
generation,
e
);
let _ = snapshot_notifier.send(Err(e));
return;
}
};
let mut result = snapshot_req.body(body).send().await;
if let Err(e) = result {
tracing::error!(
"Failed to upload snapshot for generation {}: {:?}",
generation,
e
);
let _ = snapshot_notifier.send(Err(e.into()));
return;
}
result = change_counter_req.send().await;
if let Err(e) = result {
tracing::error!(
"Failed to upload change counter for generation {}: {:?}",
generation,
e
);
let _ = snapshot_notifier.send(Err(e.into()));
return;
}
let _ = snapshot_notifier.send(Ok(Some(generation)));
let elapsed = Instant::now() - start;
tracing::debug!("Snapshot upload finished (took {:?})", elapsed);
let _ = tokio::fs::remove_file(format!("db.{}", compression)).await;
});

@MarinPostma MarinPostma force-pushed the bottomless-flush-frames-before-checkpoint branch from c33b15c to ac9f0d5 Compare September 26, 2023 15:55
@Horusiath Horusiath marked this pull request as ready for review September 26, 2023 16:04
@Horusiath Horusiath added this pull request to the merge queue Sep 26, 2023
Merged via the queue into main with commit 0a42e6e Sep 26, 2023
@Horusiath Horusiath deleted the bottomless-flush-frames-before-checkpoint branch September 26, 2023 16:48
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants