3rd party API: speed up synchronization

[korelstar]

Dear 3rd-party app developer for Nextcloud notes,

as far as I know these are @phedlund (iOS/MacOS), @stefan-niedermann and myself (Android). It seems that @steppenhahn doesn't work on his app anymore, but you (and everyone else) are also invited to participate in the discussion 😉 . If you know anybody else who uses the 3rd-party API, please give a hint. 😃

I would like to enhance the API in order to speed up synchronization and, therefore, discuss ideas on how to realize this.

Currently, on every synchronization, the whole list of notes with the content of every note has to be retrieved from the server to the mobile device. This has two disadvantages, especially if you have many and/or large notes:

1. On slow network connections, the transfer of the list of notes takes a long time.
2. On slow devices, the processing of the list of notes takes a long time.

Typically, only a small subset of notes have changed since the last synchronization. Therefore, it would be advantageous to limit the transferred notes to changed data only. However, in order to realize this, we have to change the server API -- and this could be done in different ways. Therefore, I'm introducing my idea and asking you to review and/or propose other ideas.

In order to check if a note was changed, we need to break the whole note down to a small attribute value that could be transferred and compared in place of the original note data. This value has to reflect changes to the text of the note (content) as well as to its meta-data (favorite, title, category (see #6), modified). Some possible approaches:

• use modified time
• use ETag from the note's file
• use own checksum

Since modified time and file ETag only reflect the content and not its meta-data, I propose to use an own checksum, i.e. the md5 sum of concatenated attributes content, favorite, title, category and modified. Attributes introduced in the future will have to be added to the checksum when they're introduced. I propose to call this checksum the ETag of the note (not file!) and add it as new attribute etag to db/note.php.

Now, on every synchronization, the client has to save this new ETag in its local database and send the list of known ETags on every synchronization request. In return, the server checks for every note, if the note's ETag has changed and sends only changed/new notes as full notes and unchanged notes as reduced entry with the note's ID only.

The server API changes would be fully backward compatible:

• old server app version will ignore the list of ETags sent by the client
• new server app does not require the client to send the list of ETags
• old clients will ignore the new ETag attribute sent by the server
• new clients should do not require the ETag to be sent by the server
• the ETag is calculated only on the server, so there will be no problem if it changes due to new attributes

I've implemented a prototype for this and did some benchmarks. In my experimental setup, I have 51 notes with a total length of 70KB.

• server: Interestingly, the checksum calculation does not noticeably slow down the server reaction: before I had ~810ms, now it's about 830ms.
• slow mobile phone: synchronization speed up from ~9000ms to ~3000ms (if no note was changed) => significant!
• fast mobile phone: synchronization speed up from ~1150ms to ~900ms (if no note was changed)
• message size:
  • request becomes 2KB larger (due to the list of ETags) => minimal
  • response when everything is new/changed: from 81KB to 83KB => minimal
  • response with no changes: from 81KB to 613 bytes => significant!

What do you think about this approach? Any doubts? Any other ideas? Any enhancements?

[korelstar]

@nextcloud/notes
Please have a look at my reference implementation and the discussion over there: #51.

[spaetz]

I am not sure, why you choose such a difficult approach. Using the if-modified-since header, a client could choose to only receive notes that were changed since the last check.
https://developer.mozilla.org/de/docs/Web/HTTP/Headers/If-Modified-Since

Why would you not use something like this, then the total number of notes does not matter (as most will not have changed betwen syncs) and you are not limited to a maximum of 400 notes in total which will come and bite you at some point in time.

[korelstar]

The main problem was, that Nextcloud doesn't maintain the "real" last modified time of a note. Please see #51 for a discussion about this problem.

However, based on that discussion, I went another way. With the realization in #95, we are not limited to a certain number of notes. Nevertheless, if you have any ideas how to improve that implementation, I would be very happy to discuss those.

But please note, that the description in this issue is totally outdated and doesn't correspond to the current implementation. Therefore, I'm closing this issue since it is solved by #95 (only the API documentation has to be extended in this aspect).