Modified date/time is not updated when a note is renamed.

[davidbannon]

Hi folks, love your work.
I have a beta (and only just beta) sync working with tomboy-ng ( https://github.com/tomboy-notes/tomboy-ng ) and I have found that when a note is renamed in nextCloud, its modified time (as returned under the sync api) is not updated. This means that the remote system is unaware of a change to the note and does not choose to download it.

Reproducing this bug will depend very much on how you personally choose to tickle the api ....

Davo

[stefan-niedermann]

Could you verify that the ETag changed?

[davidbannon]

I don't think I have access to an ETag via the api do I ? If not, I possibly cannot verify it.
Can you tell me what is / how to see the ETag ?
I can use a local, multipass based NextCloud server if necessary.
Davo

[stefan-niedermann]

ETags are used to verify that something has changed on the server side. You can imagine it as an hash over the http response. you send the hash in your next request as value of an If-None-Match-Header and the server will return 304 HTTP status code if the response is the exact same as last time. If anything changed (e.g. the title), you will get a complete response.

See the Notes API documentation and more information about ETags.

This is better as relying on the modified date, because in some situations one doesn't want to change the modified date. For example this would cause a order change when one ordered the notes by modified date, but this isn't actually wanted behavior.

Other scenarios might be "marking a note as favorite" - i wouldn't expect an updated modified date here, as it would cause a sort order change.

[davidbannon]

OK, I reckon I can modify my code to read the etag but will need a bit of time to do so. But that really won't help. It won't tell me which note/notes has changed. We need to have something that is a clear indicator that a particular server note is in need of being synced, if its not modified time, then what is it ?

If i am to rely on the eTag, and I get a non-match, I would need to download all the notes and compare them with local notes. Given that I have users with tens of thousands of notes, I really don't think thats a viable approach.

blue sky mode on :
If I could get a note by note eTag, when requesting a single note, that would help but would still be quite slow, too slow I expect. Better if you could add a hash of each note to the list of note attributes, i could save that locally and compare at sync time. It would also solve a very nasty problem where someone 're-establishes' a sync and has previously synced notes at both ends.

Davo

[stefan-niedermann]

You can combine the ETag with the If-Modified-Since header to solve this issue 🙂

[stefan-niedermann]

Maybe the maintainer @korelstar can provide some help?

[davidbannon]

You can combine the ETag with the If-Modified-Since header to solve this issue

Maybe ? The API docs do say that a GET can return 200, 400, 401, 404, an If-Modified-Since returns 304 so I'd tend to assume its not supported.

It would work but I'd need to repeat the GET for each and every note every time the ETag failed. And the API already seems pretty slow.

I assume korelstar watches this list ?

Davo

[korelstar]

Sorry for the late reply. Like @stefan-niedermann said, the modified time is not a technical time. It does not change if only meta data (like category, title, favorite) was changed. Therefore, you can't use this attribute for detecting changes.

Furthermore, it's true that we don't expose an ETag for single notes, currently. And we also did not implemented If-Modified-Since, since according to the HTTP standard, the result would be either 304 Not modified or the whole list of notes - and this is not what we need.

Instead you can use the parameter pruneBefore when requesting the list of notes (see API documentation for how to use this). Then, you will get the IDs only for notes that didn't change and the whole note for notes that where changed. Other third-party apps (like the Android app) use this approach, too.

[davidbannon]

OK, I guess that makes sense then. Are there plans to develop the API in the future ? if so, I could make a couple of constructive suggestions.

Firstly, use of pruneBefore, if there is a lot of notes (I have many users with 20K+ notes) and a large number of them change, its a huge download all at once. Difficult to manage errors. And it struggles when both local and NextCloud versions of a note have changed, I still need to present a last change date to to the user who has to decide how to handle the clash.

Secondly, identifying notes - if a note is assigned a unique ID it would make the system far more portable. At present, if a user, for example, moves their notes to a new install of NextCloud (or a new provider etc) a synced system cannot tell that the presented notes are the same ones it already has, this results in duplication, a major problem in any sync system. In the Tomboy world, we assign a UUID. In tomboy-ng, I use the last change date to determine that a pair of notes (with the same ID) are identical or not.

Anyway, thanks for the advice on pruneBefore, I will look at how that fits into the Tomboy model.

Davo

[korelstar]

Feedback is always welcome and of course, we can change the API if this is required. You are talking about three different points:

1. Limit list of notes

Indeed, this would be a nice feature, especially for the first synchronization where all notes have to be downloaded and saved locally. I can imagine that we could add a cursor-based pagination for this. However, we have to ensure that the client still detects removed notes immediately. I suggest that we create a new issue (feature-request) for this aspect after you've successfully tested/implemented the pruneBefore approach in your client.

2. Handling conflicts

This is a very important topic. I've written down some thoughts on this a long time ago, see #56. However, this is strongly related to #331, which could require a totally different solution for the API. Because of this, there is not much progress on this. But it's on our agenda, we just need some time to realize this.

3. static UUIDs for notes

I see the advantage of this approach. However, I'm not sure if it is possible to implement this in the Nextcloud eco-system. Notes are saved as files in Nextcloud. Therefore, if you migrate your notes to another Nextcloud server, you will move your files to that server. Hence, the UUID would must be saved in that file. This could be either the file name or the file content. Both is error-prone and unintentionally for users who use file synchronization and edit their notes manually (e.g. with the desktop system editor).

Currently, for other third-party apps, migration to another Nextcloud server works as follow: The client synchronizes with the server so that there are no local changes. Then, the client changes the URL of the Nextcloud server and synchronizes again. Now, the client removes all local notes (with obsolete ID) and downloads the notes from the server (with new ID). In your implementation, you can even do a heuristic to match old IDs to new IDs (e.g. by using title and category).

[davidbannon]

OK, sorry I have taken so long with this, life tends to get in the way ...

Yep, as you suggest, using pruneBefore solves my problem. I can turn off note content and get a reasonably fast response where is easy to see which have changed and which have not. Means my local last change date won't be accurate if all that has changed is the note title but I guess end users won't be too worried about that. Probably only important if we have a sync clash.

I will follow up on the other sub issues discussed as I get some free time.

Thanks for your advice.

Davo

[korelstar]

closing in favor of #627