Skip to content
This repository was archived by the owner on Oct 18, 2023. It is now read-only.

Namespace forking #646

Merged
merged 10 commits into from
Sep 8, 2023
Merged

Namespace forking #646

merged 10 commits into from
Sep 8, 2023

Conversation

MarinPostma
Copy link
Contributor

@MarinPostma MarinPostma commented Sep 1, 2023

This PR introduces a basic forking functionality. The admin API route POST /v1/namespaces/:from/fork/:to forks the namespace :from into a new namespace :to from the current point in time.

Forking works by creating a copy of the source database. This copy is performed from the replication log, to ensure point-in-time consistency. The forking algorithm works as follows:

  • Determine the point in time at which to fork. This is done by saving the current frame_no.
  • Replay the replication log from the start until the chosen point, and write that to a new database file
  • Create a new namespace by placing the database file in the correct directory
  • Let the recovery process take care of regenerating a replication log and replicating to bottomless

@MarinPostma MarinPostma changed the title Introduce dababase fork Namespace forking Sep 1, 2023
@MarinPostma MarinPostma marked this pull request as ready for review September 1, 2023 13:02
@MarinPostma MarinPostma requested review from Horusiath and LucioFranco and removed request for Horusiath September 1, 2023 13:03
@psarna
Copy link
Contributor

psarna commented Sep 4, 2023

Note: for a cheap copy, we could use XFS' (and some other fs') capabilities of copy-on-write, based on https://man7.org/linux/man-pages/man2/ioctl_ficlone.2.html . Probably not trivial with our current design where a restore also takes place, it's just an interesting concept - e.g. perhaps we could CoW-copy the wallog.

@MarinPostma
Copy link
Contributor Author

@psarna we will iterate on that design for sure

@haaawk
Copy link
Collaborator

haaawk commented Sep 7, 2023

Is anything blocking this PR? @psarna @MarinPostma ?

@MarinPostma
Copy link
Contributor Author

@haaawk nope, just waiting for review

@MarinPostma MarinPostma requested a review from psarna September 8, 2023 12:01
@@ -967,7 +967,7 @@ impl Replicator {
&mut self,
generation: Uuid,
utc_time: Option<NaiveDateTime>,
) -> Result<RestoreAction> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what does this bool mean? Please add a comment

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, now I went over the whole change and I'm even more confused - why was this parameter added at all? It looks like it's not used anywhere, you just ignore it above.

Copy link
Contributor Author

@MarinPostma MarinPostma Sep 8, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@MarinPostma MarinPostma requested a review from psarna September 8, 2023 12:55
@MarinPostma MarinPostma added this pull request to the merge queue Sep 8, 2023
Merged via the queue into main with commit af45314 Sep 8, 2023
@MarinPostma MarinPostma deleted the namespace-fork branch September 8, 2023 13:43
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants