Markdown files are corrupted

[bronson]

I stored my Markdown notes in Nextcloud. After making a few edits using the Nextcloud web interface, they've been ruined. I need to restore them from backup and try to remember the edits I made.

Is this behavior intentional?

For example, what used to be this:

Is now this:

Steps to Reproduce

1. In the Nextcloud browser, open any typical Markdown file that includes tables, references, line breaks, frontmatter, or related formatting
2. Make a one character edit (for example, remove the extra space after a period)
3. The entire file has been changed. Previously valid Markdown is no longer valid.
4. Restore from backup

Expected behaviour

These are text files. I'd expect my changes to be saved and the rest of the file to remain as it was.

Actual behaviour

The files are completely changed, top to bottom.

Server configuration detail

Operating system: Linux 5.4.10-x86_64-linode132 #1 SMP PREEMPT Thu Jan 9 21:17:12 UTC 2020 x86_64

Webserver: Apache/2.4.29 (Ubuntu) (apache2handler)

Database: mysql 5.7.29

PHP version:

7.2.24-0ubuntu0.18.04.3
Modules loaded: Core, date, libxml, openssl, pcre, zlib, filter, hash, Reflection, SPL, sodium, session, standard, apache2handler, mysqlnd, PDO, xml, apcu, calendar, ctype, curl, dom, mbstring, fileinfo, ftp, gd, gettext, iconv, igbinary, imagick, intl, json, exif, mysqli, pdo_mysql, Phar, posix, readline, redis, shmop, SimpleXML, sockets, sysvmsg, sysvsem, sysvshm, tokenizer, wddx, xmlreader, xmlwriter, xsl, zip, Zend OPcache

Nextcloud version: 18.0.3 - 18.0.3.0

Updated from an older Nextcloud/ownCloud or fresh install:

Where did you install Nextcloud from: unknown

Signing status

Array
(
)

List of activated apps

Enabled:
 - accessibility: 1.4.0
 - activity: 2.11.0
 - admin_audit: 1.8.0
 - cloud_federation_api: 1.1.0
 - comments: 1.8.0
 - contacts: 3.2.0
 - dav: 1.14.0
 - encryption: 2.6.0
 - federatedfilesharing: 1.8.0
 - federation: 1.8.0
 - files: 1.13.1
 - files_external: 1.9.0
 - files_pdfviewer: 1.7.0
 - files_rightclick: 0.15.2
 - files_sharing: 1.10.1
 - files_trashbin: 1.8.0
 - files_versions: 1.11.0
 - files_videoplayer: 1.7.0
 - firstrunwizard: 2.7.0
 - gpxpod: 4.2.1
 - issuetemplate: 0.6.0
 - logreader: 2.3.0
 - lookup_server_connector: 1.6.0
 - maps: 0.1.6
 - nextcloud_announcements: 1.7.0
 - notifications: 2.6.0
 - oauth2: 1.6.0
 - password_policy: 1.8.0
 - photos: 1.0.0
 - previewgenerator: 2.3.0
 - privacy: 1.2.0
 - provisioning_api: 1.8.0
 - recommendations: 0.6.0
 - serverinfo: 1.8.0
 - settings: 1.0.0
 - sharebymail: 1.8.0
 - spreed: 8.0.7
 - support: 1.1.0
 - survey_client: 1.6.0
 - systemtags: 1.8.0
 - theming: 1.9.0
 - twofactor_backupcodes: 1.7.0
 - updatenotification: 1.8.0
 - viewer: 1.2.0
 - workflowengine: 2.0.0
Disabled:
 - notes
 - text
 - user_ldap

Configuration (config/config.php)

{
    "instanceid": "***REMOVED SENSITIVE VALUE***",
    "passwordsalt": "***REMOVED SENSITIVE VALUE***",
    "secret": "***REMOVED SENSITIVE VALUE***",
    "trusted_domains": [
        "cloud.ifying.com"
    ],
    "datadirectory": "***REMOVED SENSITIVE VALUE***",
    "overwrite.cli.url": "https:\/\/cloud.ifying.com",
    "dbtype": "mysql",
    "version": "18.0.3.0",
    "dbname": "***REMOVED SENSITIVE VALUE***",
    "dbhost": "***REMOVED SENSITIVE VALUE***",
    "dbport": "",
    "dbtableprefix": "oc_",
    "mysql.utf8mb4": true,
    "dbuser": "***REMOVED SENSITIVE VALUE***",
    "dbpassword": "***REMOVED SENSITIVE VALUE***",
    "installed": true,
    "htaccess.RewriteBase": "\/",
    "memcache.local": "\\OC\\Memcache\\APCu",
    "memcache.locking": "\\OC\\Memcache\\Redis",
    "filelocking.enabled": "true",
    "redis": {
        "host": "***REMOVED SENSITIVE VALUE***",
        "port": 0,
        "timeout": 0
    },
    "maintenance": false,
    "theme": "",
    "loglevel": 2,
    "updater.release.channel": "stable",
    "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
    "enable_previews": true,
    "enabledPreviewProviders": [
        "OC\\Preview\\PNG",
        "OC\\Preview\\JPEG",
        "OC\\Preview\\GIF",
        "OC\\Preview\\HEIC",
        "OC\\Preview\\BMP",
        "OC\\Preview\\XBitmap",
        "OC\\Preview\\MP3",
        "OC\\Preview\\TXT",
        "OC\\Preview\\MarkDown",
        "OC\\Preview\\PDF",
        "OC\\Preview\\OpenDocument",
        "OC\\Preview\\MSOffice2003",
        "OC\\Preview\\MSOffice2007",
        "OC\\Preview\\MSOfficeDoc",
        "OC\\Preview\\PDF",
        "OC\\Preview\\Postscript",
        "OC\\Preview\\Image",
        "OC\\Preview\\Photoshop",
        "OC\\Preview\\TIFF",
        "OC\\Preview\\SVG",
        "OC\\Preview\\Font",
        "OC\\Preview\\Movie",
        "OC\\Preview\\MKV",
        "OC\\Preview\\MP4",
        "OC\\Preview\\AVI"
    ]
}

Are you using external storage, if yes which one: local/smb/sftp/...

Are you using encryption:

Are you using an external user-backend, if yes which one: LDAP/ActiveDirectory/Webdav/...

Client configuration

Browser: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_3) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15

Operating system: Macintosh

Logs

Web server error log

Insert your web server log here 

Nextcloud log

Insert your Nextcloud log here

Browser log

Insert your browser log here, this could for example include:

a) The javascript console log
b) The network log
c) ...

[bronson]

Just noticed in the README that the Text app advertises an:

Open format: Files are saved as Markdown,
so you can edit them from any other text app too.

This isn't true until the Text app preserves Markdown.

Right now it's only safe to either:

• edit Markdown 100% with the Text app, or
• avoid using it completely (so there's no risk of your files being reformatted)

[2q2q]

This problem makes the Text Editor completely unusable for editing any markdown files.

Example
Markdown Header before Editing:

---
Title: any
---

Markdown Header after Editing:

---

## Title: any

[mcgodwin]

Same issue here. I am trying to use Pico CMS app, which needs .md files to remain in a fixed layout, with clean and consistent YAML headers. The Text app messes with the YAML header's formatting, preventing Pico to render any page edited with Text. This is a big issue for me, as we will be a team of people collaborating on a website made with Pico CMS for Nextcloud, as developed in this issue: nextcloud/cms_pico#116

[almereyda]

To refrain from the negative impetus in this issue's title, can we safely rename it to YAML frontmatter support and assign another of the labels, like feature: integration?

It appears this is not an issue of the Text app alone, but stems from the wider ecosystem of unaligned Markdown implementations. Not even CommonMark 0.29 mentions frontmatter or YAML anywhere. It appears hard for me to consider this a bug, since the design documents nowhere state that the app supports a (YAML) frontmatter.

[mcgodwin]

Hi @almereyda and thanks for your comment.
Focusing back on usability, it appears that prior to v.18, the default text editor was not only preserving frontmatter data but could be plugged in with a Markdown editor. Since v.18, no available app allows to view a .md file's source, and the new default app alters frontmatter data.

So yes, from the usability point of view, it is a bug. A crucial feature (on which some apps rely) has been removed.

[lunar-debian]

I have been bitten by this bug too. My solution was to change the extension to .txt so I could still use Text as a collaborative editor while keeping the original formatting and all other characters.
Which now gives me an idea of a small step that could reduce user pain. Please hear me out:

1. Load the file.
2. Parse it as Markdown.
3. Make an export and compare it with the original file.
4. If the export and the original file are the same, start the rich-text editor ; otherwise keep the simple text editor with no extra parsing.

This should prevent any data loss.

[bronson]

@almereyda I'm happy to change the title. Do you have a suggestion? (edited to @ the right person, sorry!)

This issue isn't limited to frontmatter. I don't use frontmatter and it affected me (example above). It clobbers a number of markdown constructs.

[bronson]

Just to be clear: the issue isn't what markdown it supports. The issue is that it reformats markdown that it doesn't understand. No other Markdown editor (that I've used) does that.

[almereyda]

Indeed, I forgot to consider the case where decidedly formatted Markdown tables got distorted, as per the original posting.

Either these two are one case of general support for higher-order grammars of Markdown, then I suggest a title like "Support for extended Markdown features broken".

Else I would suggest to track the cases independently as:

• Frontmatter support gone
  which could include a mention of agnosticism against different formats that we find in the wild, like YAML, JSON or TOML
• Markdown table support rudimentary
  which could mention that we aim at recreating human-readable Markdown tables

[lunar-debian]

@almereyda I respectfully disagree with your classification. Opening a preexisting file, making a change, and losing information (it's not mere presentation, it's structure) should be seen as data loss. It's not only a regression from the previous editor in term of features.

[bronson]

@almereyda I'm not sure about your suggested title. I don't mind at all if this editor chooses not to support support any particular Markdown feature. That's A-OK. The problem is that it wrecks content that it doesn't understand.

If we were to track them separately, maybe we'd open issues for the following?

• frontmatter
• tables
• references
• line breaks
• setext headers
• line blocks
• definition lists

And probably find a few others used in data science and publishing: https://rmarkdown.rstudio.com/authoring_pandoc_markdown.html%23raw-tex

I'm happy to do it if you think that's the way to go.

[almereyda]

I'm unable to make a meaningful consideration at this point, and would leave a decision about classification to the maintainers.

…

On Wed, 29 Apr 2020 at 02:26, Scott Bronson ***@***.***> wrote: @almereyda <https://github.com/almereyda> I'm not sure about your suggested title. I don't mind at all if this editor chooses not to support support any particular Markdown feature. That's A-OK. The problem is that it wrecks content that it doesn't understand. If we were to track them separately, maybe we'd open issues for the following? - frontmatter - tables - references - line breaks - setext headers - line blocks - definition lists And probably find a few others used in data science and publishing: https://rmarkdown.rstudio.com/authoring_pandoc_markdown.html%23raw-tex I'm happy to do it if you think that's the way to go. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#772 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAMRV7FDCVWKHOLU67W6KZDRO5X2JANCNFSM4MIAKUVA> .

[e-alfred]

The old files_texteditor app is back on the App Store (https://apps.nextcloud.com/apps/files_texteditor), a rough workaround would be to use it (again) and files_markdown and not use Text for these type of files.

[mcgodwin]

@e-alfred thanks! 🙌

[bertrand-benoit]

@e-alfred Thanks, I find back something closer than what I need.

[vince2bir]

Have you tried the Markdown Editor ?
https://apps.nextcloud.com/apps/files_markdown

(You need to install the previously mentioned files_texteditor)

[bertrand-benoit]

Yes exactly, and so I find back something closer than what I need ;)

[cilex-ft]

Nice to see plain markdown edit back, this new editor was a pita.

[JoshuaPettus]

I agree lunar-debian. This is a data loss issue! As such, shouldn't this be marked critical?! Even if the problem lies with un-standardized markup implementations, there has to be a way to mitigate this, lunar-debian had a good suggestion, although I would suggest an error with a brief explanation why it wasn't opened in rtf mode with unsupported features. We can't have data loss.

[mdoggydog]

The fundamental problem is that the new Text app is not really a markdown editor. It is a WYSIWYG document editor that happens to use MD syntax to store its documents. It does not respect the MD semantics in which files are text files, where every byte belongs to the user, and the MD syntax provides rendering advice and a readable structure to the unrendered text file itself.

It seems like Text is being positioned as a Nextcloud analog to Google Docs. If that is the direction NC wants to take it, I think the most straightforward way to fix this is for Text documents to have their own mimetype/extension (e.g., .ncdoc), so that it is clear that Text is the master of the content of such files and the only thing guaranteed to be preserved by a save/load cycle is the on-screen presentation. There could still be an "Import from Markdown file" operation, but it would be an explicit operation, and it would create a new Text document (.ncdoc) and not reuse the input file. NC could still promote "You can even edit .ncdoc files in any text editor!" as a feature, but it would be clear that Text has total control over the on-disk contents of a document, subject to change the next time Text loads it.

As it stands, I am forced to disable the Text app in the two NC instances I administrate, because of the threat of corruption of actual .md files. The long-awaited and much-desired collaborative editing feature is trumped by the necessity of preserving data.

To be honest, I would love to just have collaborative editing in the good ol' Plain text editor, and along with it, Markdown editor (which supports LaTeX math rendering!). I'm curious why collaborative editing would be any harder on files that are just strings of characters presented as strings of characters. (It's been done before/elsewhere, right?)

[sparagi]

Looks like this is a duplicate of #593.

[claell]

Thanks @sparagi for the hint, I'll mark it accordingly.

[claell]

Duplicate of #593