feat: federated calendar shares

[st3iny]

• Part of Federated calendar and contacts sharing #1440
• Resolves Read-only federated calendar shares #54682
• Requires feat: federated calendar sharing calendar#6998

Summary

The beast ...

Terminology

• Remote instance: Instance which is receiving a share.
• Host instance: Instance which is owning/hosting the shared calendar.

Architecture

Sharing works by leveraging our on-instance sharing via CalDavBackend (and appending the _shared_by_<uid> suffix). The remote user is authenticated in Sabre through a "virtual" principal in principals/remote-users/<base64-encoded-cloud-id> with whom an outgoing federated calendar is shared "locally". The cloud id needs to be encoded as it might contain slashes, for example, when the remote instance is hosted inside a web root. The outgoing shared calendars are hosted inside a separate calendar root in remote-calendars/<base64-encoded-cloud-id>/<calendar_name>_shared_by_<sharer> on the host instance. The remote instance will send all sync requests to this address and authenticate using the remote user's cloud id and a shared secret (token) via HTTP basic auth.

A federation provider was implemented to send (and receive) shares across federated instances. The remote instance will track the incoming federated calendar inside a new table oc_calendars_federated. The host instance will create a regular DAV share inside oc_dav_shares in case the remote user logs in and syncs the calendar. I had to add a new column token to oc_dav_shares in order to track the auth token (shared secret) for the remote user (so that the sharer is able to revoke individual shared calendars).

If a calendar object is changed on the host instance, a cloud notification is sent to all remote instances with which the calendar is shared with. Remote servers will then queue a sync job at the next cron interval to sync the affected calendar. This also has the benefit of nicely debouncing the sync requests. As a backup, there is an hourly background job which syncs all remaining calendars (starting with the least recently synced calendar).

Please let me know in case there are questions or further elaboration is needed.

How to test?

1. Set up two instances (host and remote).
2. Install calendar on both instances.
3. Calendar: Check out the linked PR above (at least on the remote instance).
4. Server: Check out this branch on both instances.
5. Federate both.
6. Optional: Sync the federated address books from remote to host (via occ federation:sync-addressbooks) to make finding the remote users easier in the sharing search.
7. Create a calendar with some events on the host instance.
8. Share the calendar with the remote instance (type in the domain of the remote instance to quickly find remote users to share with).

Run the following command on the remote instance: occ federation:sync-calendars

Now, you should be able to see all the events in the calendar on the remote instance.

TODO

• Remove a lot of weird leftover code
• Either trigger the sync job immediately or sync directly on incoming shares
• Fix existing tests
• Add new tests
• Chunk results (reuse Hamza's truncation?) -> deferred
• Consider multi-get to bundle ICS download -> deferred
• Resolve conflicts
• Fix integration tests
• Fix Psalm issues

Checklist

• Code is properly formatted
• Sign-off message is added to all commits
• Tests (unit, integration, api and/or acceptance) are included
• Documentation (manuals or wiki) has been updated or is not required

[tcitworld]

I get the architecture logic but having a one-hour sync interval between changes is quite a bumper, and in this case it's far more important for usability than the regular cached subscriptions sync interval.

Instead of going for a pull model, why not go for a push model? Queue a QueuedJob after each federated calendar change to propagate changes to every federated sharee, and only have pull as a fallback.

[st3iny]

Instead of going for a pull model, why not go for a push model? Queue a QueuedJob after each federated calendar change to propagate changes to every federated sharee, and only have pull as a fallback.

That was my idea initially but then I ran out of time. We could use ICloudFederationNotification to achieve that and write a simple calendar event listener.

[st3iny]

Done, I implemented a simple push sync to make the feature a bit more useful. The periodic job will pick up all other calendars.

[st3iny]

I'm sorry for creating this monster. Have fun reviewing ... 😅

[come-nc]

Still todo?

[st3iny]

Yeah, that is still a TODO left for the future. I didn't implement the events as there is no use case for them yet.

[come-nc]

Reviewed a tiny part for today, will continue next week, some random questions popping:

• How consistent is it with federated file sharing?
• It supports only users and not groups, right? Will it be "easy" to add groups later?
• What about circles?

[ChristophWurst]

Makes sense conceptually. Thanks a lot for the architecture overview

Code mostly makes sense. A bit much to review and process at a time 🙈

[SebastianKrupinski]

Some other thought and things,

Not sure what the reasons was to hard wire the federation in to the DAV stack (CalendarHome, Provider and Root), I think doing this as a DAV Calendar Plugin would have been much simpler. I think it would have been more modular, easier to maintain and control but this not a blocker.

One thing I did find in my testing was that after unsharing the calendar all the calendar object are still present in the database, of the user the calendar was shared with

[SebastianKrupinski]

This is not a blocker, but more of a this might become a issue when syncing large calendars. I noticed while testing with a large calendar (12k) that if the process is interrupted, for what ever reason, the sync start back at the beginning, even after it already syncing 2k of entries.

Since syncing large calendars can take a long time (12k = 3hours on my machine) and causes thousands of requests this can get interrupted but firewall rate limits, slow connection, etc.

[st3iny]

Yeah, that is a problem. However, this has to be solved on the CalDAV protocol level as I didn't implement a custom one.

AFAIK, there is no way to continue an interrupted initial sync according to the RFCs.

[st3iny]

* How consistent is it with federated file sharing?

Not at all because both behave very differently. Calendar federation will cause a local copy of all remote events to be synced. File federation will only request the data from the other instance if required. So I don't think they can be compared in a meaningful way.

* It supports only users and not groups, right? Will it be "easy" to add groups later?

Yes, for now it only supports users. Adding support for groups should be straight forward and not too much effort.

* What about circles?

I did not look into that. I have no idea about circle federation and can give no estimate on the work that would need to be done.

[SebastianKrupinski]

Tested again working good

[st3iny]

I rebased to fix conflicts. Should be all green now.