[Request] Incremental sync

[i2dcarrasco]

Hello,

I am migrating a Wordpress site to Cloud Run, and I am using GFuse for the statics part. This works, but is not stable so I am trying your plugin to remove the mount point and just use an Storage.

The first step is to sync all the media files, and I have noticed that the process seems to be full all the time, ignoring if any of the files was already uploaded. This is fine most of the time and is a good choice to ensure that all files are in sync, but can be a big problem when you have several files. I have got two different scenarios that ended in the same problem (GKE + NFS and Cloud Run + GFuse), so I am sure that is not a GFuse only problem. The problem is that when the process fails and some files were not uploaded, you have to fully restart the process. For example, now I am uploading +5000 files, which takes several hours. In case of an error (which for sure will happens because the GFuse), I am not able to check if any of the uploads has failed and then to be sure I have to fully check the media manager, or to resync several times.

My request is the possiblitity of to track which files were already uploader (keeping an status database or checking if the media URL matches the final URL for example), and then skip the already processed files of course keeping the possibility of fully sync again. Maybe now is doing it in any way, but is not the sensation I have because it take a very long time and looks like it was doing the whole process again.

Best regards.