Support FileProvider API (VFS) in macOS

[helmut72]

https://www.apple.com/macos/catalina-preview/features/

Third-party cloud service integration
An all-new FileProvider API for cloud storage providers delivers a new way to seamlessly integrate their services into the Finder without requiring a kernel extension, helping to maintain the security of your Mac. Cloud storage providers can now deliver their apps through the Mac App Store.

[pgauret]

Now that Catalina is out may be a good opportunity to look into FileProvider. This would allow to modernise the macOS desktop client and bring equivalent functionality to the virtual drive for Windows client.

[theJian]

Catalina is out but I can't find a way to create FileProviderExtension on macOS. Does anyone notice that FileProvider is actually removed from Catalina features page?

[mikelynch]

It's strange that it's not listed as a feature in the release, but the Developer documentation does say it's available in macOS 10.15+. I'm afraid I don't know anything about macOS development, but there are some First Steps topics there. @theJian, do those provide any useful guidance for getting started?

It's also just occured to me that this is already implemented in the iOS app, and hopefully that component should also work on macOS. Would that give an easier route to using this API?

One consideration is that presumably the current logic will have to be kept for the moment, in order to support older versions of macOS – I imagine it wouldn't be desirable to require Catalina for a while yet. It'd be great to be support this API as an option though!

[theJian]

@mikelynch I didn't find those useful. Luckily once you created a FileProviderExtension, Xcode will generate template code for you. It's a working simple demo and full of comments. Getting started by reading those code is much easy in my opinion.

[pointlessone]

Documentation is a little inconsistent. On the File Provider Framework page it states availability to be macOS 10.15+ but on the individual protocol pages (e.g. NSFileProviderReplicatedExtension) it's macOS 11.0+. I guess this feature didn't make it into Catalina but since Big Sur is around the corner maybe it's time to revisit this issue?

[leiless]

FYI, File Provider - macOS Support seems available by now.

[aniqueta]

Is this something being worked on? With Big Sur changes, it seems safe to assume that in the next macOS version using the FileProvider API will become mandatory and the Finder extension approach will be deprecated. Thanks.

[leiless]

Is this something being worked on?

Don't know.

It seems that FileProvider not ready yet, and Apple hasn't given any sample code on it.

[hemancuso]

@leiless we actually just shipped Nextcloud support in our FileProvider product (Strongsync). Definitely is ready. Happy to answer questions.

[leiless]

@leiless we actually just shipped Nextcloud support in our FileProvider product (Strongsync). Definitely is ready. Happy to answer questions.

Does there any sample code on FileProvider API? I've checked Apple Developer docs, haven't seeing it yet.

How you guys support FileProvider for Strongsync?

[hemancuso]

I believe the very latest XCode betas generate a template project

[helmut72]

@leiless we actually just shipped Nextcloud support in our FileProvider product (Strongsync). Definitely is ready. Happy to answer questions.

Target: common Apache2 WebDAV

Tasks:

1. Created test folder on macOS: ok, also on the server
2. Copied my Notes (a lot of Markdown files in private and work with subfolders): failed. Work and all subfolders are on the server, no subfolder in Private. On both no single file was uploaded!
3. Deleted test folder: still on the server (empty trashcan on macOS doesn't helped)
4. Result: uninstalled Strongsync

[andreasley]

Xcode 12.5, which was released today, contains a File Provider Extension target template to help getting started with macOS File Provider extensions.

[leiless]

Xcode 12.5, which was released today, contains a File Provider Extension target template to help getting started with macOS File Provider extensions.

FYI: https://developer.apple.com/documentation/xcode-release-notes/xcode-12_5-release-notes

[aniqueta]

Glad to see the activity here. Looks like the API is maturing in macOS Monterey. FYI: Box has adopted it now: https://blog.box.com/get-more-secure-seamless-box-drive-experience-across-macs

[solracsf]

See https://github.com/nextcloud/desktop/blob/master/doc/architecture.rst#virtual-files for how to enable experimental support on v3.4.0.

[pszypowicz]

See https://github.com/nextcloud/desktop/blob/master/doc/architecture.rst#virtual-files for how to enable experimental support on v3.4.0.

Hi, is this coded with a FileProviderAPI? Could you point to a commit/PR implementing this?

[aniqueta]

Note that OneDrive [1] and Dropbox [2] are stating there are changes coming in macOS 12.3 (March?) that is making them fully migrate over to the FileProvider API, with macOS 12.2 being the last version they support with their current virtual files / online files implementation. I can't find anything from Apple indicating what these changes are, but I suspect we'll find out soon...

[1] https://techcommunity.microsoft.com/t5/microsoft-onedrive-blog/inside-the-new-files-on-demand-experience-on-macos/bc-p/3066588
[2] https://help.dropbox.com/installs-integrations/desktop/macos-12-monterey-support

[jeffmancuso]

They both have exceptions to the no kernel extension policy allowing them to use their previous solutions. I imagine this exception is finally going away in a late-cycle point release rather than in macOS 13. This increases their pressure to actually migrate over to FPE.

[marcotrevisan]

Just for the records: https://developer.apple.com/documentation/fileprovider/macos_support/syncing_files_on_macos contains an example XCode project that uses NSFileProviderReplicatedExtension.

Would be great to have the system take care of most of the synchronization work for you, and you only have to focus on the client calls and the extra features.

[Saifudaullah]

Hey all, no need to give thanks, I wanted to use this as much as you did 🙂

Now that the relevant PR has been merged, the file provider module will be included in version 3.9. The module is now also already included in the unstable daily builds of the desktop client.

For the moment, the file provider module is hidden and disabled by default as we have some rough edges still to iron out. It can, however, be enabled by adding macFileProviderModuleEnabled = true to your client's nextcloud.cfg file. Be warned that this module is still not ready for use in production environments!

Finally, we are also providing a new version of the macOS VFS Beta version of the desktop client that can be installed alongside the normal desktop client. You can download it here:

https://cloud.nextcloud.com/s/8JnmYA8a7xDm9tT

Regarding the issues that you guys have brought up:

1. @marcotrevisan Unfortunately I haven't yet had the time to properly resolve several of the issues you were facing with borked change fetching after system shutdown, or the other issues. This is on the todo list for now and will need to be resolved before considering this feature stable
2. Initial sync is still very slow and this will need to also be resolved
3. There is no GUI management of the file provider sync module, this is -- you guessed it -- also on the todo list.

Hi @claucambra, I'm unable to access the provided link. Can you provide another link for me to test this beta client?

[BJKle]

@claucambra Any news or timeline?

[marcotrevisan]

I'm using the beta 2 client for a while now, under Mac OS Ventura.
I'd like to encourage the team to "do the last mile" asap, because even if it has its own glitches, it's also already showing some great advantages over the existing implememtation (not only the NC Client, but other webdav clients too).
Thanks!

[kklem0]

Can I test it with the daily build since it is still in the master branch?
https://download.nextcloud.com/desktop/daily/

[claucambra]

Can I test it with the daily build since it is still in the master branch? https://download.nextcloud.com/desktop/daily/

We had a major regression with the Finder integration caused by the File Provider module so we had to disable building of the File Provider module by default. So the dailies do not have the module enabled. If you are brave enough to try building the client yourself, however, you can toggle the compilation flag and build the client with the module enabled

[pete1019]

Hi, is the M1 support with virtual drive back in the daily?
Otherwise where to get one without need to build?
Thanks

[marcotrevisan]

I'm just guessing but I'd expect to see this being closed first: #5693
You might want to follow the activity on this feature branch: https://github.com/nextcloud/desktop/tree/feature/file-provider-config-ui

My 2 cents

[szaimen]

Reopening because fileprovider support was reverted for the time being iirc

[BJKle]

Is there a target release?

[claucambra]

Reopening because fileprovider support was reverted for the time being iirc

Reclosing, we are shipping working code in master at the moment ( we just are not enabling by default in normal releases as we are in the process of creating a specific version of the client with the file provider engine )

We also have the VFS betas for use at the moment

Is there a target release?

At the moment no, we are being cautious about deadlines considering we have had unexpected hiccups when it comes it existing integrations, for instance. At the moment we are working on getting the FIle Provider engine better integrated into our GUI and on improving initial sync times; once this is done, we will probably be ready to release something we are happy to call stable :)

[Saifudaullah]

@claucambra How do we test the beta? Is there any link that I can download the beta?

[marcotrevisan]

@claucambra How do we test the beta? Is there any link that I can download the beta?

I see the beta package here (not sure if it's the latest version of it):
(folder) https://download.nextcloud.com/desktop/prereleases/virtualdrive/
https://download.nextcloud.com/desktop/prereleases/virtualdrive/Nextcloud-VFS-beta-macOS.pkg

[Saifudaullah]

@marcotrevisan I'm afraid that is not the latest one. The timestamp of the installer is 2023-03-21.

[claucambra]

Hi all, we've built a new version of the File Provider based VFS beta based on the current stable branch of the desktop client.

You may download the macOS VFS Beta 3 from the link below:

https://download.nextcloud.com/desktop/prereleases/virtualdrive/Nextcloud-stable-3.10-fileprovider.pkg

Key changes in this beta include some small stability improvements, as well as all changes in the latest 3.10 desktop client :)

[kklem0]

Hi all, we've built a new version of the File Provider based VFS beta based on the current stable branch of the desktop client.

You may download the macOS VFS Beta 3 from the link below:

https://download.nextcloud.com/desktop/prereleases/virtualdrive/Nextcloud-stable-3.10-fileprovider.pkg

Key changes in this beta include some small stability improvements, as well as all changes in the latest 3.10 desktop client :)

Do you think the problematic issue you mentioned earlier is fixed?

[marcotrevisan]

Hi @claucambra, thanks very much for the new beta!
I noticed that now hitting the space bar on a not-already-materialized item will cause materialization and subsequent preview, nice!
I still can't get precisely under what conditions (maybe filename patterns?), but looks like the previews that available on the server aren't actually used for the icons of unmaterialized files, except for some of them in my private folders.
That's weird... is it a known issue?

Thanks!

[Saifudaullah]

Thank you so much for the new beta @claucambra. I've installed it, and I assume to enable the file provider, we need to add macFileProviderModuleEnabled = true to the client's nextcloud.cfg.

I can say that this beta seems promising and quite nice to be used. Now it really feels like a proper implementation of virtual file system.

Below are my feedback:

1. The initial sync is really slow.
2. The loading indication spinning forever when I open empty folder
3. Folders not showing size
4. Folders that are already synced (downloaded) still showing dotted cloud icon

I also has a question regarding Nextcloud folder. Is it normal that we have two Nextcloud folders? One under "/Users/"username"/Nextcloud" and another one under "/Users/"username"/Library/CloudStorage/Nextcloud-"nextcloud_username"@domain/"?

[marcotrevisan]

@Saifudaullah I didn't have to set macFileProviderModuleEnabled = true in my config.

This build is still including both sync engines, so I'd suggest you to uncheck the root folder to practically disable the legacy engine. The legacy engine is responsible for "/Users/"username"/Nextcloud" (it will still be there but it won't sync).

As you noticed the initial sync is slow and until it's finished you will experience delays in the UI.
Your point 2 is not showing up in my client so I'd suggest to wait a bit more before starting your test.
I can also confirm your points 3. and 4. Point 3 is a known issue and in to-do list if I remember correctly.

Regards,

[claucambra]

Thank you so much for the new beta @claucambra. I've installed it, and I assume to enable the file provider, we need to add macFileProviderModuleEnabled = true to the client's nextcloud.cfg.

I can say that this beta seems promising and quite nice to be used. Now it really feels like a proper implementation of virtual file system.

Below are my feedback:

1. The initial sync is really slow.

2. The loading indication spinning forever when I open empty folder

Points 1 and 2 are related. At the moment we require a full scan of the server folder hierarchy during initial sync. We are considering how to address this

3. Folders not showing size

Yeah, aware of this bug -- thanks for raising

4. Folders that are already synced (downloaded) still showing dotted cloud icon

Can reproduce, still not 100% sure why this is the case. Will investigate

[claucambra]

As an aside, I've opened a Discussions thread to continue any discussion on the VFS beta as discussing on a closed thread is not ideal :)

#6267

[claucambra]

Do you think the problematic issue you mentioned earlier is fixed?

I can confirm it has been fixed, but there is unfortunately no way we are going to be able to ship both the Finder Sync extension and the File Provider module alongside each other. 3.11 is going to be released imminently and I can confirm we will not be including the File Provider module with this release either.

When this module is considered stable we will be shipping a new client edition alongside the normal client which foregoes the Finder integration for the conventional sync engine in favour of the File Provider module. It is likely we will also strip out the traditional sync engine from the VFS-focused edition of the desktop client.

None of this is final and is subject to change, though! Thoughts welcome.

[marcotrevisan]

forgot to mention: that glich is happening to me since the first beta.
If I remember correctly (not 100% sure) beta2 made it partially work.

By the way, I noticed in my system (M1-Sonoma) the command “fileproviderctl ls domainname” seems not be implemented anymore, but the man page is unchanged… ??
Any clues on new ways to accomplish that are very welcome!
Thanks!

I still can't get precisely under what conditions (maybe filename patterns?), but looks like the previews that available on the server aren't actually used for the icons of unmaterialized files, except for some of them in my private folders. That's weird... is it a known issue?

[funkemunky]

When I download a file pressing "Download Now", the file does indeed download. However, the "Remove download" button is missing when I want to set it to cloud-only.