Markdown files from external editors lose formatting on save

[inthreedee]

Summary
Markdown files that are created in external markdown and text apps will have their formatting completely overridden and/or lost when edited through the Nextcloud web interface. Newlines, tabs, and spaces are thrown out everywhere in the document--not just on the edited line--and lists will be changed to asterisks instead of dashes. See screenshots below.

Background
I use external text editors for my markdown files and then sync them to my Nexctloud. I primarly use Atom for its markdown support but will also use basic text editors like Gedit on Linux or Notepad on Windows. That's the great thing about markdown files, after all, they're just plain text. When I edit these files through my Nextcloud web interface, the formatting of the entire file is destroyed.

Related: #390

To Reproduce
Steps to reproduce the behavior:

1. Create a markdown file in an external editor such as Atom/Gedit or a basic text editor like Notepad
2. Add some spaces, tabs, newlines, lists, etc.
3. Open the markdown file in the Nextcloud web editor and notice the formatting differences
4. Make a minor edit to the file in the Nextcloud web interface
5. View the file again in your external editor and notice the formatted has been destroyed.

Expected behavior
I expect the Text app to not completely destroy any formatting created by external apps. If any special formatting is unsupported, it should be left alone for maximum compatibility with external apps. If any formatting changes must be made based on the current edits, those changes should apply ONLY to the current edits and not the entire file.

Screenshots

How it looks in Gedit under Linux:

How it looks when opened in my Nextcloud web interface:

Make an edit through the web interface and now the file's formatting has been destroyed when viewed in Gedit:

Client details:

• OS: Arch Linux
• Browser: Firefox
• Version: 72.0.1
• Device: desktop

Server details

Text app version: 1.1.1

Operating system: Ubuntu 18.04.3

Web server: Apache

Database: Mysql 5.7.29

PHP version: 7.3.13

Nextcloud version: 17.0.2

[nathanlesage]

Second that!

EDIT: Just found another extremely problematic behaviour: Automatic escaping of footnotes. Also not specified in CommonMark, of course, but it should not simply escape the square brackets, because it fully breaks support in those third-party editors that support those (besides, having links defined in-text and then the link at the bottom is in fact common, which uses exactly the same format)

[zerknorscht]

We use pico cms Nextcloud-Integration and need to be able to edit those .md files via Nextcloud.
However, this bug effects the ability to edit pico files, f.e. _meta.md looses all it's formatting and since it has a yaml header, it destroys parts of the website.
This is especially annoying, since there is no possibility to use the markdown editor or any other editor anymore in NC 18...

[nathanlesage]

May I suggest that, while WYSIWYG in Markdown is cool to have, a plain code view would be the easiest solution for the meantime until the mode works for everyone, so that we can work with the files easily without fearing to lose any data?

As the plain view was already implemented, it shouldn't prove too difficult to re-enable that and simply offer people a switch between the two!

[JoshuaPettus]

I've been having this issue as well. It disheartens me that we are now at NC19 and this still hasn't been addressed. The pico CMS is huge as it breaks the YAML code on the file.

[claell]

This seems to be a pretty prominent and relevant issue. Since this (and duplicates) have been in a limbo state for some time now, I'll ping some people (maybe you also know other people to ping who could help out or are responsible for this application).

@juliushaertl @jospoortvliet @rullzer

[szaimen]

To quote @juliushaertl from nextcloud/notes#652 (comment):

I feat that we might never be able to fully preserve the original formatting. The main reason is that the WYSIWYG editor does not preserve the original format but converts the markdown document to a different document format (a json tree with different node types like h1/li and annotation types like italic/bold) and then on save converts this json format back to markdown. Taking lists as an example we just don't know when saving if the original markdown file was written with * or - as list indicators. This was decided to be an acceptable downside of the approach since we mainly aimed for a rich content editor here rather than a markdown editor. However we should of course preserve as much as possible so the open enhancement requests about frontmatter or tables for example should definitely be implemented at some point.

[claell]

Either there have to be improvements to the currently used system (like @juliushaertl suggested in the comment) or the system has to be changed (not sure whether it is possible to create a WYSIWYG editor without the conversion to a different document format in between, but the nextcloud notes editor looks like it does just that). The current state is not suitable for productive usage, I basically agree with nextcloud/notes#652 (comment).

[bronson]

I'm glad the meeting went well. You covered a lot of ground.

As you might imagine, I'm surprised where you ended up. It sounds like Nextcloud will continue to be an unsafe palce for Markdown files while unkown developers fix some difficult problems. This feels like the sort of thing that could take a year or more to solve reliably, doesn't it?

Earlier in this thread, people proposed solutions that fix this 100% that can be implemented right now. I would be able to put together either of these patches myself if they would be welcomed:

1. Use a different file extension for the collaborative WYSIWYG editor (.mdc has been suggested) and open existing .md files in the code editor.
2. Set rich_editing_enabled default to false. (because file integrity should not be opt-in)

Then Nextcloud would be safe for Markdown files again while the WYSIWYG editor continues to improve. Is there any chance I could help here? I'd really like Nextcloud to be a safe place for user files again.

[max-nextcloud]

Earlier in this thread, people proposed solutions that fix this 100% that can be implemented right now.

Please don't repeat yourself. I read those proposals and I understood that you are volunteering to create a PR for them. I outlined requirements for a solution and i don't think these proposals meet them:

• People who have used text in Nextcloud up to 25 would be shown the source code of what they previously created in the editor. Most of them would consider this data loss.
• Setting rich_editing_enable by default would not allow users who never heard of markdown to use the editor to format text.

We have literally millions of users of Nextcloud who have never heard of markdown. Showing ![holidays.jpg](.attachments.123/holidays.jpg) instead of an image in one of the files they created with text is not an option. We could tell them that we managed to preserve the syntax - they won't care because it lost its meaning.

[jospoortvliet]

I agree with Max here. In the end, this seems the standard power user vs normal user dilemma. And at Nextcloud, we prioritize the use case for the majority (the typical users), while aiming to provide settings and options for advanced users.

The compromise solution of looking at the parsing result and disabling WYSIWYG is already going to inconvenience some people. I would not be surprised if the majority of markdown users will not be creating hugely complicated things and the minor changes Text does to their files are probably inconsequential. I know it is for me - I use dozens of md files for note taking, and as I use a flat text editor, a markdown editor and Text through each other, Text will start to drop WYSIWYG for me in quite some situations. I won't be happy ;-)

I think introducing a button to switch between the two will be quite helpful, but that can of course be done in a later stage.

I only mildly disagree with the decision to not have a user setting for this. I see the potential for confusion, certainly, but then people talk to each other in organizations, and Berta (from the example) will likely be quite aware of their choice. After all, they picked a power user setting here. An info screen will show Alex that wysiwyg is disabled and they will figure out why quick enough, I bet. Although again, a way to switch back to WYSIWYG would probably be extra nice in this scenario for Alex.

While, of course, having this user setting will help power users like @bronson immensely. Similarly, being able to configure extensions might help him and other power users. So I would personally rather not rule that out completely.

But honestly. This is an improvement. It will help power users with only a very mild inconvenience for non-techies. Once it's implemented, let's look what the next step is?

[brtptrs]

Just as a reminder:
In the drawio module there used to be a similar issue, with files being named *.xml
Converting to *.drawio for diagram files solved all problems.
It was just a one-time hassle to rename the files, done, and forgotten.
Why do we need to choose between annoying the one or the other group of users?

*.nctext is created by and opens in the text module by default
*.md has an option to use the text module if the user prefers to

[inthreedee]

*.nctext is created by and opens in the text module by default
*.md has an option to use the text module if the user prefers to

I outlined requirements for a solution and i don't think these proposals meet them:

• Avoid data loss / changing the .md syntax without consent.
• Allow users who never heard of markdown to use the editor to format text with the default settings.

Check and check. *.md files could continue to be opened in Text by default, with a user option to disable this. Are there any other requirements that this solution does not meet?

[bronson]

I'm not sure storing my Obsidian notes in Nextcloud makes me a power user.

@inthreedee's proposed fix sounds like the best of all. I volunteer to implement it and we can have this solved in a month. Are you interested? (this is time I do not have, but this is how strongly I feel about data loss in Nextcloud)

If you're not interested in that fix either, then could you please suggest a time frame that you expect to implement the fixes you suggest? This issue is 3 years old next month. Personally, I find corruption once or twice a year to be simply unacceptable, and it appears I need to plan a workaround until Nextcloud is a safe home for all my data again. Do you think before 2024? Or 2027?

[palmergw]

As a workaround for anyone running into this in the future, my solution was to install Markdown Editor on top of Plain Text Editor (per their instructions), and disabling the Text app. There are some drawbacks, but if you do not rely on rich editing of documents in the browser and wish to make Nextcloud safe for your MD files again, I recommend looking into it.

I'm using Nextcloud as a file store for Logseq, and whether it's that app or Obsidian or any other app using a markdown file system, this default functionality is a huge issue. In my mind this is a pretty blatant violation of Single Responsibility, opening a file should never under any circumstances cause an edit and an autosave to that file.

I would like to suggest that any time a plugin/app is interacting with a file in the browser, that app should have a file extension configuration option, or Nextcloud should have a centralized file extension configuration page (prevent multiple apps listing the same extension). Nextcloud can default those values to cater to their customer base, but it does away with the kludgey workarounds and is in line with how users on all platforms expect to be able to choose what functionality to associate with a file type.

[daffydock]

As a workaround for anyone running into this in the future, my solution was to install Markdown Editor on top of Plain Text Editor (per their instructions), and disabling the Text app. There are some drawbacks, but if you do not rely on rich editing of documents in the browser and wish to make Nextcloud safe for your MD files again, I recommend looking into it.

It is not a really a fix or a really, a workaround. It is akin to saying that the best way to fix the brakes in a car is to just not use the car.

Disabling the Text app had been brought up a while back and most Admins probably reached the conclusion that not using the app is a feasible workaround, if you could really call it that. Yet, the fact remains the Text is being used in the back-end for other apps. Drawbacks is putting it mildly. On my instance for example, we opted to use Collectives. Well, Collectives becomes unusable without the Text app. So, we are left with the option of looking at a different solution outside Nextcloud after implementation, or risk data loss. We also had a use case to incorporate Pico CMS into the instance but the risk of data loss, or having to constantly be restoring backups as part of the workflow made it a non-starter. I am sure I am not the only one who came to this conclusion.

I agree with the notion of perhaps having the Text app have a file configuration option, yet that too, was brought up in the past as well, or maybe even a warning. At this point, you can call it whatever you want. I think most of us are easy going. Hopefully, it is on the shorter side.

I too agree with both @brtptrs and @palmergw, if possible I would prefer a general timeline of what is going to be done, if possible. This data loss issue has been going on for at least 3 years, this very issue was opened on Jan. 19th 2020, and the same concern has been brought up via multiple other issues with other apps, like Pico, Collectives or just YML in general, just to name three. I have been keeping track of this for years now and if am honest, I think I see the Dev team dragging their feet on this issue. By now, my take is that in the end they do not see it as important or vital enough, given the workload and/or downstream implications? I do not know for sure, but that is what reading through the whole thing seems to point at. No judgement, at all guys, but that is what it looks like from the optics. So, I am at a loss.

Edit: Oh, I also meant to include @bronson, since he was willing to take the time to work on this. My mistake.
Nextcloud is a great piece of kit, but having to opt between not using apps that are being advertised as usable solutions or data loss, seems like an odd tradeoff the way things currently stand. "Power user," or not.

[max-nextcloud]

@inthreedee's proposed fix sounds like the best of all. I volunteer to implement it and we can have this solved in a month. Are you interested? (this is time I do not have, but this is how strongly I feel about data loss in Nextcloud)

Sorry for taking so long to reply. I'm pondering your ideas. We'll be meeting in person with the team in two weeks. I'll bring up the topic then.

[bronson]

I'm trying to understand something @jospoortvliet said:

In the end, this seems the standard power user vs normal user dilemma.

For the sake of argument, let's assume that this statement is true*. To solve this issue, we need to choose between:

• having features for normal users, but the featues come with the risk of occasionally corrupting power user files
• reliable file integrity for all users, but normal users might experience difficult/missing features or other inconvenience (maybe having file extensions renamed on an upgrade).

Jos then said that Nextcloud will "prioritize the use case for the majority".

Did he mean this? When deciding between features and file corruption, if the features are for regular users and the file corruption is restricted to minority/power users, then he chooses file corruption?

While this is a consistent position, and absolutley one the Nextcloud project can make, I am having a hard time understanding it. I'm hoping I'm mistaken somewhere? (though this would certainly help explain how this has been an issue for so long!)

*: I don't think making this choice is even necessary. People have suggested solutions that at least partially satisfy both types of users simultaneously. Even if it was necessary, I would disagree that the presence of Markdown files is a good indicator of whether someone is a power user or not.

[max-nextcloud]

I try to avoid the concept of power users. Some users use text a lot and know a lot of tricks to make it work well for them and don't know what markdown is. Others hardly use text at all and are impacted by changes to the markdown source because they also use the file in a different context where the changes matter. And of course there's a lot of other factors playing into this that "power users" vs. "normal users" glosses over.

At the same time I have not heard about this problem elsewhere then in this issue. I usually hear about issues affecting many users through various ways - from friends using nextcloud, from customers, and in forums. For example a lot of people accidentally created a Readme.md file by clicking the white area above the file list. I try to prioritize such issues - and still this took a long time to get fixed.

I currently prioritize making it easier to contribute to text and I work on #3543. There's only so much time we can spend on issues that do not have customer buy in.

In terms of this issue we're getting involved in the discussion to provide feedback on which proposals seem viable and which don't. We're reviewing and merging all improvements provided and we're breaking out individual issues such as #3516 and providing pointers how to get started. Right now the most promising way to fix the entire issue from my point of view still is #3477 - but as i said we'll discuss the file extension option.

That's the path... how long it will take us to walk it? No idea - it's a complex mix of issues.

[max-nextcloud]

Created #3630 to discuss the file extension there.

[bronson]

#3636 looks like a promising way to avoid the file renaming issues associated with #3630

[bronson]

I try to avoid the concept of power users.

I 100% agree. I apologize. From the way @jospoortvliet worded his message, I thought he was speaking for Nextcloud. I was getting worried!

[DonnerWolfBach]

@max-nextcloud This issue still exists for me, at least for the android app. Any updates?
Noticed it because I was syncing my Logseq notes through nextlcoud like @palmergw

[max-nextcloud]

@max-nextcloud This issue still exists for me, at least for the android app. Any updates? Noticed it because I was syncing my Logseq notes through nextlcoud like @palmergw

I think two issues might be mixed here.

Did you edit your logs in text? If you do this this issue would still affect you.

If you just looked at the file there are no changes to persist and thus the file should not get saved. However we had a bug that triggered saves if no changes were made when closing the file. This has been fixed in 26.0.4 and 27.0.1. So if the files changed by just opening them - could you check if that still is the case for you on the latest versions?

[DonnerWolfBach]

@max-nextcloud This issue still exists for me, at least for the android app. Any updates? Noticed it because I was syncing my Logseq notes through nextlcoud like @palmergw

I think two issues might be mixed here.

Did you edit your logs in text? If you do this this issue would still affect you.

If you just looked at the file there are no changes to persist and thus the file should not get saved. However we had a bug that triggered saves if no changes were made when closing the file. This has been fixed in 26.0.4 and 27.0.1. So if the files changed by just opening them - could you check if that still is the case for you on the latest versions?

I switched to logseq built in syncing since issues like these were to much of a hussle. So as of now I cannot test this.