feat: implement `takeover` for nydusd fusedev daemon #1434

loheagn · 2023-09-26T09:37:40Z

Relevant Issue (if applicable)

#1421

Details

This patch implements the save and restore functions in the fusedev_upgrade in the service crate.

To do this,

This patch adds a new crate named nydus-upgrade into the workspace. The nydus-upgrade crate has some util functions help to do serialization and deserialization for the rust structs using the versionize and snapshot crates. The crate also has a trait named StorageBackend which can be used to store and restore fuse session fds and state data for the upgrade action, and there's also an implementation named UdsStorageBackend which uses unix domain socket to do this.
as we have to use the same fuse session connection, backend file system mount commands, Vfs to re-mount the rafs for the new daemon (created for "hot upgrade" or failover), this patch add a new struct named FusedevState to hold these information. The FusedevState is serialized and stored into the UdsStorageBackend (which happens in the save function in the fusedev_upgrade module) before the new daemon is created, and the FusedevState is deserialized and restored from the UdsStorageBackend (which happens in the restore function in the fusedev_upgrade module) when the new daemon is triggered by takeover.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation Update (if none of the other choices apply)

Checklist

Go over all the following points, and put an x in all the boxes that apply.

I have updated the documentation accordingly.
I have added tests to cover my changes.

loheagn · 2023-09-26T09:40:31Z

As this patch depend on the cloud-hypervisor/fuse-backend-rs#149 ,
I keep the fuse-backend-rs crate use the local repo in my local devbox, and this pr may keep draft until the cloud-hypervisor/fuse-backend-rs#149 merged.

imeoer · 2023-09-27T09:00:28Z

@ccx1024cc Can we test the compatibility with the internal nydus implementation?

codecov · 2023-09-27T15:32:19Z

Codecov Report

Merging #1434 (ff79cdb) into master (d7b1851) will decrease coverage by 0.21%.
The diff coverage is 19.60%.

@@            Coverage Diff             @@
##           master    #1434      +/-   ##
==========================================
- Coverage   46.41%   46.20%   -0.21%     
==========================================
  Files         123      127       +4     
  Lines       38745    39033     +288     
  Branches    38745    39033     +288     
==========================================
+ Hits        17983    18036      +53     
- Misses      19789    20024     +235     
  Partials      973      973

Files	Coverage Δ
api/src/http.rs	`0.00% <ø> (ø)`
service/src/fusedev.rs	`0.00% <ø> (ø)`
upgrade/src/lib.rs	`100.00% <100.00%> (ø)`
upgrade/src/backend/mod.rs	`98.30% <98.30%> (ø)`
service/src/lib.rs	`75.64% <0.00%> (ø)`
service/src/fs_service.rs	`50.21% <0.00%> (-2.47%)`	⬇️
upgrade/src/backend/unix_domain_socket.rs	`0.00% <0.00%> (ø)`
upgrade/src/persist.rs	`0.00% <0.00%> (ø)`
service/src/upgrade.rs	`0.00% <0.00%> (ø)`

... and 3 files with indirect coverage changes

This patch implements the `save` and `restore` functions in the `fusedev_upgrade` in the service create. To do this, - This patch add a new create named `nydus-upgrade` into the workspace. The `nydus-upgrade` create has some util functions help to do serialization and deserialization for the rust structs using the versionize and snapshot crates. The crate also has a trait named `StorageBackend` which can be used to store and restore fuse session fds and state data for the upgrade action, and there's also an implementation named `UdsStorageBackend` which uses unix domain socket to do this. - as we have to use the same fuse session connection, backend file system mount commands, Vfs to re-mount the rafs for the new daemon (created for "hot upgrade" or failover), this patch add a new struct named `FusedevState` to hold these information. The `FusedevState` is serialized and stored into the `UdsStorageBackend` (which happens in the `save` function in the `fusedev_upgrade` module) before the new daemon is created, and the `FusedevState` is deserialized and restored from the `UdsStorageBackend` (which happens in the `restore` function in the `fusedev_upgrade` module) when the new daemon is triggered by `takeover`. Signed-off-by: Nan Li <loheagn@icloud.com> Signed-off-by: linan.loheagn3 <linan.loheagn3@bytedance.com>

kevinXYin · 2023-10-01T02:42:44Z

Thanks for the nice job! Can this work with the nydus-snapshotter ? I also have plans to support takeover feature for fscache mode, I think there are some process that can be reused.

kevinXYin · 2023-10-01T02:59:08Z

service/src/upgrade.rs

+            .session
+            .lock()
+            .unwrap()
+            .get_fuse_file()


If nydusd exit when there are inflight fuse requests, IO hung occurs because the mainline kernel does not support resending inflight fuse requests. If we want to support takeover for fusedev, how about store the fuse fd created by ioctl FUSE_DEV_IOC_CLONE, then when nydusd exit, the inflight requests will be dropped instead of blocking IO.

Get.

How about implementing this function which clones the fuse fd in fuse-backend-rs crate? For example, add a method named clone_fuse_file to FuseSession?

I'm not sure if macos can support clone, maybe we need to keep the API more generic.
Any suggestions? @imeoer @jiangliu

Maybe make it as a built-in feature in fuse-backend-rs? WDYT cc @jiangliu ?

Sure, let's get fuse-backend-rs ready for this:)

Sure, let's get fuse-backend-rs ready for this:)

@imeoer @jiangliu I might be able to help with this :)

cloud-hypervisor/fuse-backend-rs#177 may fix this

loheagn · 2023-10-06T08:36:29Z

Thanks for the nice job! Can this work with the nydus-snapshotter ? I also have plans to support takeover feature for fscache mode, I think there are some process that can be reused.

Here is a pr containerd/nydus-snapshotter#520 for nydus-snapshotter to do failover and hot upgrade. This pr adds some check code to ensure the upgrade process be done for all the old version nydus daemons, especially the deamons created during the upgrade stage.

I have tested the failover feature and more test cases should be added.

kevinXYin · 2023-10-07T07:27:09Z

Thanks for the nice job! Can this work with the nydus-snapshotter ? I also have plans to support takeover feature for fscache mode, I think there are some process that can be reused.

Here is a pr containerd/nydus-snapshotter#520 for nydus-snapshotter to do failover and hot upgrade. This pr adds some check code to ensure the upgrade process be done for all the old version nydus daemons, especially the deamons created during the upgrade stage.

I have tested the failover feature and more test cases should be added.

As I know , the nydus-snapshotter already support failover and hot upgrade workflow.

kevinXYin · 2023-11-11T06:22:31Z

I will send another pr also support takeover for fscache , which fold your patch , then we can start reviewing full takeover feature.

bergwolf · 2023-12-06T08:40:58Z

@kevinXYin The feature has been pending for too long. Let's do it step by step, by merging the fuse takeover first, and then you can proceed with fscache takeover.

loheagn · 2023-12-08T05:17:48Z

@kevinXYin The feature has been pending for too long. Let's do it step by step, by merging the fuse takeover first, and then you can proceed with fscache takeover.

@bergwolf #1487 has already contain the commits in this pr. This pr can be closed.

loheagn mentioned this pull request Sep 26, 2023

feat: add "persist" feature cloud-hypervisor/fuse-backend-rs#149

Merged

loheagn force-pushed the takeover-1 branch from 2e52431 to 6658d82 Compare September 27, 2023 15:22

loheagn force-pushed the takeover-1 branch 3 times, most recently from 2a8ab23 to ed7df0c Compare September 28, 2023 07:05

loheagn force-pushed the takeover-1 branch from ed7df0c to ff79cdb Compare September 28, 2023 07:28

kevinXYin reviewed Oct 1, 2023

View reviewed changes

loheagn marked this pull request as ready for review November 10, 2023 05:31

loheagn requested a review from a team as a code owner November 10, 2023 05:31

loheagn requested review from bergwolf, liubin and adamqqqplay and removed request for a team November 10, 2023 05:31

loheagn mentioned this pull request Nov 15, 2023

fusedev: add clone_fuse_file method to FuseSession cloud-hypervisor/fuse-backend-rs#177

Merged

kevinXYin mentioned this pull request Nov 18, 2023

implement takeover for nydus fscache/fusedev mode #1487

Merged

5 tasks

loheagn closed this Dec 8, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: implement `takeover` for nydusd fusedev daemon #1434

feat: implement `takeover` for nydusd fusedev daemon #1434

loheagn commented Sep 26, 2023 •

edited

Loading

loheagn commented Sep 26, 2023 •

edited

Loading

imeoer commented Sep 27, 2023

codecov bot commented Sep 27, 2023 •

edited

Loading

kevinXYin commented Oct 1, 2023

kevinXYin Oct 1, 2023

loheagn Oct 6, 2023

kevinXYin Oct 7, 2023

imeoer Nov 9, 2023

jiangliu Nov 13, 2023

loheagn Nov 13, 2023 •

edited

Loading

loheagn Nov 15, 2023

loheagn commented Oct 6, 2023 •

edited

Loading

kevinXYin commented Oct 7, 2023

kevinXYin commented Nov 11, 2023

bergwolf commented Dec 6, 2023

loheagn commented Dec 8, 2023

feat: implement takeover for nydusd fusedev daemon #1434

feat: implement takeover for nydusd fusedev daemon #1434

Conversation

loheagn commented Sep 26, 2023 • edited Loading

Relevant Issue (if applicable)

Details

Types of changes

Checklist

loheagn commented Sep 26, 2023 • edited Loading

imeoer commented Sep 27, 2023

codecov bot commented Sep 27, 2023 • edited Loading

Codecov Report

kevinXYin commented Oct 1, 2023

kevinXYin Oct 1, 2023

Choose a reason for hiding this comment

loheagn Oct 6, 2023

Choose a reason for hiding this comment

kevinXYin Oct 7, 2023

Choose a reason for hiding this comment

imeoer Nov 9, 2023

Choose a reason for hiding this comment

jiangliu Nov 13, 2023

Choose a reason for hiding this comment

loheagn Nov 13, 2023 • edited Loading

Choose a reason for hiding this comment

loheagn Nov 15, 2023

Choose a reason for hiding this comment

loheagn commented Oct 6, 2023 • edited Loading

kevinXYin commented Oct 7, 2023

kevinXYin commented Nov 11, 2023

bergwolf commented Dec 6, 2023

loheagn commented Dec 8, 2023

feat: implement `takeover` for nydusd fusedev daemon #1434

feat: implement `takeover` for nydusd fusedev daemon #1434

loheagn commented Sep 26, 2023 •

edited

Loading

loheagn commented Sep 26, 2023 •

edited

Loading

codecov bot commented Sep 27, 2023 •

edited

Loading

loheagn Nov 13, 2023 •

edited

Loading

loheagn commented Oct 6, 2023 •

edited

Loading