Nextcloud sync: Very easy to cause data corruption with Nextcloud's rich text editor

[personalizedrefrigerator]

Operating system

Linux

Joplin version

2.14.12

Desktop version info

No response

Current behaviour

1. Open a file on the Nextcloud web interface (left-click)
2. Leave open for about 30 seconds (long enough to auto-save)
3. Try to sync Joplin

Currently,

id: 36dc5bb035a0433da224b8fcc0368081
parent_id: 
item_type: 1
item_id: ee829a38b5e84cd48d74aeee602b1e72
item_updated_time: 1707265701979
title_diff: "[{\"diffs\":[[1,\"Test NUL 2\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":10}]"
body_diff: "[{\"diffs\":[[1,\"Test NUL 2\\\n\\\n\\\n\\\n[js-draw.pdf](:/3529017ac0974c8a8fd463f60b3d16a8)\\\n\\\n\"]],\"start1\":0,\"start2\":0,\"length1\":0,\"length2\":65}]"
metadata_diff: {"new":{"id":"ee829a38b5e84cd48d74aeee602b1e72","parent_id":"248e9425a4e042cda71d51f42b6d498f","latitude":"123.45678","longitude":"9101112.13141","altitude":"0.0000","author":"","source_url":"","is_todo":0,"todo_due":0,"todo_completed":0,"source":"joplindev-desktop","source_application":"net.cozic.joplindev-desktop","application_data":"","order":0,"markup_language":1,"is_shared":0,"share_id":"","conflict_original_id":"","master_key_id":"","user_data":""},"deleted":[]}
encryption_cipher_text: 
encryption_applied: 0
updated_time: 2024-02-07T00:34:40.150Z
created_time: 2024-02-07T00:34:40.150Z
type_: 13

becomes

id: 36dc5bb035a0433da224b8fcc0368081
parent_id:
item_type: 1
item_id: ee829a38b5e84cd48d74aeee602b1e72
item_updated_time: 1707265701979
title_diff: "[{"diffs":\[\[1,"Test NUL 2"\]\],"start1":0,"start2":0,"length1":0,"length2":10}]"
body_diff: "[{"diffs":\[\[1,"Test NUL 2\\\\n\\\\n\\\\n\\\\n[js-draw.pdf](:/3529017ac0974c8a8fd463f60b3d16a8)\\\\n\\\\n"\]\],"start1":0,"start2":0,"length1":0,"length2":65}]"
metadata_diff: {"new":{"id":"ee829a38b5e84cd48d74aeee602b1e72","parent_id":"248e9425a4e042cda71d51f42b6d498f","latitude":"123.45678","longitude":"9101112.13141","altitude":"0.0000","author":"","source_url":"","is_todo":0,"todo_due":0,"todo_completed":0,"source":"joplindev-desktop","source_application":"net.cozic.joplindev-desktop","application_data":"","order":0,"markup_language":1,"is_shared":0,"share_id":"","conflict_original_id":"","master_key_id":"","user_data":""},"deleted":\[\]}
encryption_cipher_text:
encryption_applied: 0
updated_time: 2024-02-07T00:34:40.150Z
created_time: 2024-02-07T00:34:40.150Z
type\_: 13

Sync with Joplin then fails (key type_ not found).

Expected behaviour

Either

1. Joplin should be able to handle characters escaped by Nextcloud
   • Edit: This doesn't seem feasible — Nextcloud changes more than just type_ -> type\_ (compare body_diff and title_diff above).
2. It shouldn't be possible to edit Joplin sync files in Nextcloud's rich text editor UI (e.g. make them use a .yaml extension)
   • Edit: This isn't feasible either — see below.

Logs

No response

[laurent22]

I feel this bug should also be reported to Nextcloud - this is quite bad that they save a file that hasn't been changed, even worse that they actually auto-change it when saving.

[laurent22]

Joplin should be able to handle characters escaped by Nextcloud

Ideally we won't do this because we shouldn't have to because that's a lot of complexity to go around someone else's bug.

It shouldn't be possible to edit Joplin sync files in Nextcloud's rich text editor UI (e.g. make them use a .yaml extension)

Actually this is too big a change probably, so let's not look into this for now.

I feel it's a shame that Nextcloud is so popular and at the same time they do stuff like automatically breaking user's files. We shouldn't have to work to fix up their mistakes.

[laurent22]

Maybe something to investigate: Can we set a flag on WebDAV to make all these files hidden by default? Maybe we only leave a README.md in there that says "don't change any of these files - it will result in corruption".

Or maybe we add such file anyway named "_IMPORTANT_READ_FIRST!!.md" so that it shows up at the top of the list

[ghTravis]

I'm also experiencing this issue. Nearly every note is suffering from this type\_ problem. Has anyone created an issue on the nextcloud github or created a discussion there? Would be good to reference this issue in it also

[personalizedrefrigerator]

Has anyone created an issue on the nextcloud github or created a discussion there?

I don't think so — it could make sense to report the issue either in their server repository or their text repository (their forum seems to be down edit: it's working now!).

Related upstream issues:

• Detect changes to markdown syntax and open in code editing mode nextcloud/text#3477
• Markdown files from external editors lose formatting on save nextcloud/text#593
• Open editable files in read-only mode nextcloud/server#29783
• https://help.nextcloud.com/t/text-app-modifies-text/164155

[ghTravis]

My nextcloud usecase is pretty limited to just Joplin at the moment, and I have no intention of editing files in the web console, so I am testing out this particular feature in case something is being weird with the rich text editor.

I executed the two occ commands to disable rich workspaces and rich text editing to see if it would fix things. It definitely makes it easier to fix the files after they have been formatted with type\_ in the nextcloud web console. I'll observe editing and adding files through Joplin to see if this actually does anything.

https://docs.nextcloud.com/server/latest/admin_manual/configuration_server/text_configuration.html

[ghTravis]

Adding to my comment above, I haven't had any corruption issues since making the two changes to the config, and I am syncing across 3 different devices without issue.

[laurent22]

Related: #9997