Markdown plugin fails to parse metadata when given a file with CR or CRLF line endings

[StJohn3D]

Given a file _nav.md

---
menu: [
  {
    icon: faHome,
    page: Home
  }
]
---
Don't delete this text,
this file has to contain something besides metaData for it to be parsed by documentalist.

The metadata will only be parsed if the file has Unix LF line endings. If the file has Windows CR LF or Mac CR it will fail to parse the metadata.

[adidahiya]

Are you using Mac or Windows for these various files with different line endings? We simply use fs.readFileSync(path, "utf8") to read markdown files, so as long as you use the standard convention on the OS, it should work fine...

[StJohn3D]

@adidahiya
People using macs don't have this issue, but people on windows - if they haven't configured git to normalize line endings to LF on checkout, they will get different results. The generated json will just have empty meta-data entries instead of actually capturing the meta-data from the markdown file.

[StJohn3D]

@adidahiya to test, copy the example above (_nav.md) and open the file in notepad++
Once open in NotePad++

• click View/Show Symbol/Show All Characters
• then use Edit/EOL Conversion to test different line endings.
• save and run the file through documentalist to generate some json

On windows you should end up with a different/wrong json output when using anything besides LF
At least that has been my experience.

[adidahiya]

I doubt we're going to do anything special here for this edge case, but let me know if you have any clever ideas for addressing the problem in a simple way... for now just work around this by normalizing your file endings

[StJohn3D]

Sure thing, for now this is how I'm working around it.

npm i -D replace-in-file

normalizeLineEndings.js

const replace = require('replace-in-file');

module.exports = (filesGlob) => {
  const options = {
    files: filesGlob,
    from: /\r\n|\r/gm,
    to: '\n',
  };
  try {
    console.log("Normalizing line endings to LF...")
    const results = replace.sync(options);
    const filesChanged = results.filter(r => r.hasChanged).map(f => f.file)
    if (filesChanged.length > 0) console.log('    Normalized the following files:', filesChanged);
    console.log("done.")
  }
  catch (error) {
    console.error('Error occurred:', error);
  }
}

Then wherever you use documentalist

const normalizeLineEndings = require('./path/to/normalizeLineEndings')
// ...
normalizeLineEndings('src/**/*.md')
new Documentalist()
// ...

[egoroof]

Face the same problem

This is my solution:

const { Documentalist, MarkdownPlugin, TypescriptPlugin } = require('@documentalist/compiler');
const { writeFileSync } = require('fs');

class LineFormatter {
    compile(files) {
        for (const file of files) {
            writeFileSync(file.path, file.read().replace(/\r\n/g, '\n'));
        }
    }
}

new Documentalist()
    .use('.md', new LineFormatter())
    .use('.md', new MarkdownPlugin())
    .use(/\.ts$/, new TypescriptPlugin({ excludePaths: ['src'] }))
    .documentGlobs('sdk/**/*')
    .then((docs) => JSON.stringify(docs))
    .then((json) => writeFileSync('dist/docs.json', json))
    .catch((e) => {
        console.log(e);
        process.exit(1);
    });

But this should definitly do MarkdownPlugin itself

[bvandercar-vt]

Face the same problem

This is my solution:

const { Documentalist, MarkdownPlugin, TypescriptPlugin } = require('@documentalist/compiler');
const { writeFileSync } = require('fs');

class LineFormatter {
    compile(files) {
        for (const file of files) {
            writeFileSync(file.path, file.read().replace(/\r\n/g, '\n'));
        }
    }
}

new Documentalist()
    .use('.md', new LineFormatter())
    .use('.md', new MarkdownPlugin())
    .use(/\.ts$/, new TypescriptPlugin({ excludePaths: ['src'] }))
    .documentGlobs('sdk/**/*')
    .then((docs) => JSON.stringify(docs))
    .then((json) => writeFileSync('dist/docs.json', json))
    .catch((e) => {
        console.log(e);
        process.exit(1);
    });

But this should definitly do MarkdownPlugin itself

This worked for me! Except to avoid actually writing the files, did

class LineFormatter {
    compile(files) {
        return files.map(file => file.read().replace(/\r\n/g, "\n"));
    }
}

[bvandercar-vt]

Inline:

.use(".md", {
            compile: files =>
                process.platform === "win32" ? files.map(file => file.read().replace(/\r\n/g, "\n")) : files,
        })