Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

refactor(utils): use Remark to parse Markdown server-side #5670

Closed
wants to merge 5 commits into from
Closed

refactor(utils): use Remark to parse Markdown server-side #5670

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a01e8ec
fix(packages/docusaurus-utils): fix warning for unresolved link insid…
MattiaPrimavera Oct 8, 2021
e65254b
Update packages/docusaurus-utils/src/markdownLinks.ts
MattiaPrimavera Oct 9, 2021
46399b8
fix(packages/docusaurus-utils): test non-greedy regex
MattiaPrimavera Oct 11, 2021
cc89cef
Merge branch 'main' into unresolved-link-html-comment
Josh-Cena Dec 22, 2021
9c86c8a
Use Remark
Josh-Cena Dec 22, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions examples/facebook/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@
"eslint-plugin-jsx-a11y": "^6.5.1",
"eslint-plugin-react": "^7.27.0",
"eslint-plugin-react-hooks": "^4.3.0",
"prettier": "^2.5.0",
"prettier": "^2.5.1",
"stylelint": "^13.2.1"
},
"browserslist": {
Expand All @@ -52,4 +52,4 @@
]
},
"description": "Docusaurus example project (facebook template)"
}
}
2 changes: 1 addition & 1 deletion package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@
"netlify-cli": "^8.0.5",
"nodemon": "^2.0.13",
"npm-run-all": "^4.1.5",
"prettier": "^2.5.0",
"prettier": "^2.5.1",
"react": "^17.0.1",
"react-dom": "^17.0.1",
"react-test-renderer": "^17.0.2",
Expand Down
2 changes: 1 addition & 1 deletion packages/create-docusaurus/templates/facebook/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,7 +35,7 @@
"eslint-plugin-jsx-a11y": "^6.5.1",
"eslint-plugin-react": "^7.27.0",
"eslint-plugin-react-hooks": "^4.3.0",
"prettier": "^2.5.0",
"prettier": "^2.5.1",
"stylelint": "^13.2.1"
},
"browserslist": {
Expand Down
6 changes: 3 additions & 3 deletions packages/docusaurus-plugin-content-blog/src/__tests__/__snapshots__/feed.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ exports[`blogFeed atom shows feed item for each post 1`] = `
<id>/mdx-blog-post</id>
<link href=\\"https://docusaurus.io/myBaseUrl/blog/mdx-blog-post\\"/>
<updated>2021-03-05T00:00:00.000Z</updated>
<summary type=\\"html\\"><![CDATA[HTML Heading 1]]></summary>
<summary type=\\"html\\"><![CDATA[Heading 2]]></summary>
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like a different behavior here, don't know for sure what is best though 🤷‍♂️

image

<content type=\\"html\\"><![CDATA[<h1>HTML Heading 1</h1><h2>HTML Heading 2</h2><p>HTML Paragraph</p><div>Import DOM</div><h1>Heading 1</h1><h2>Heading 2</h2><h3>Heading 3</h3><h4>Heading 4</h4><h5>Heading 5</h5><ul><li>list1</li><li>list2</li><li>list3</li></ul><ul><li>list1</li><li>list2</li><li>list3</li></ul><p>Normal Text <em>Italics Text</em> <strong>Bold Text</strong></p><p><a href=\\"https://v2.docusaurus.io/\\">link</a>
<img src=\\"https://v2.docusaurus.io/\\" alt=\\"image\\"/></p>]]></content>
</entry>
Expand Down Expand Up @@ -105,7 +105,7 @@ exports[`blogFeed json shows feed item for each post 1`] = `
\\"content_html\\": \\"<h1>HTML Heading 1</h1><h2>HTML Heading 2</h2><p>HTML Paragraph</p><div>Import DOM</div><h1>Heading 1</h1><h2>Heading 2</h2><h3>Heading 3</h3><h4>Heading 4</h4><h5>Heading 5</h5><ul><li>list1</li><li>list2</li><li>list3</li></ul><ul><li>list1</li><li>list2</li><li>list3</li></ul><p>Normal Text <em>Italics Text</em> <strong>Bold Text</strong></p><p><a href=\\\\\\"https://v2.docusaurus.io/\\\\\\">link</a>\\\\n<img src=\\\\\\"https://v2.docusaurus.io/\\\\\\" alt=\\\\\\"image\\\\\\"/></p>\\",
\\"url\\": \\"https://docusaurus.io/myBaseUrl/blog/mdx-blog-post\\",
\\"title\\": \\"Full Blog Sample\\",
\\"summary\\": \\"HTML Heading 1\\",
\\"summary\\": \\"Heading 2\\",
\\"date_modified\\": \\"2021-03-05T00:00:00.000Z\\"
},
{
Expand Down Expand Up @@ -189,7 +189,7 @@ exports[`blogFeed rss shows feed item for each post 1`] = `
<link>https://docusaurus.io/myBaseUrl/blog/mdx-blog-post</link>
<guid>/mdx-blog-post</guid>
<pubDate>Fri, 05 Mar 2021 00:00:00 GMT</pubDate>
<description><![CDATA[HTML Heading 1]]></description>
<description><![CDATA[Heading 2]]></description>
<content:encoded><![CDATA[<h1>HTML Heading 1</h1><h2>HTML Heading 2</h2><p>HTML Paragraph</p><div>Import DOM</div><h1>Heading 1</h1><h2>Heading 2</h2><h3>Heading 3</h3><h4>Heading 4</h4><h5>Heading 5</h5><ul><li>list1</li><li>list2</li><li>list3</li></ul><ul><li>list1</li><li>list2</li><li>list3</li></ul><p>Normal Text <em>Italics Text</em> <strong>Bold Text</strong></p><p><a href=\\"https://v2.docusaurus.io/\\">link</a>
<img src=\\"https://v2.docusaurus.io/\\" alt=\\"image\\"/></p>]]></content:encoded>
</item>
Expand Down
5 changes: 5 additions & 0 deletions ...docusaurus-plugin-content-docs/src/markdown/__tests__/__fixtures__/docs/doc2.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,8 @@
- [doc2](./doc2.md)

- [doc-localized](./doc-localized.md)
<!-- [doc4](md-link-in-single-line-comment.md) -->
- [doc-localized](./doc-localized.md)
<!--
[doc5](md-link-in-multi-line-comment.md)
-->
MattiaPrimavera marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 2 additions & 0 deletions .../docusaurus-plugin-content-docs/src/markdown/__tests__/__snapshots__/linkify.test.ts.snap
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ exports[`transform to correct links 1`] = `
- [doc1](/docs/doc1)
- [doc2](/docs/doc2)

- [doc-localized](/fr/doc-localized)

- [doc-localized](/fr/doc-localized)
"
`;
Expand Down
2 changes: 2 additions & 0 deletions packages/docusaurus-plugin-content-docs/src/markdown/__tests__/linkify.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,6 +115,8 @@ test('transform to correct links', () => {
expect(transformedContent).not.toContain('](./doc2.md)');
expect(transformedContent).not.toContain('](subdir/doc3.md)');
expect(transformedContent).not.toContain('](/doc-localized');
expect(transformedContent).not.toContain('md-link-in-single-line-comment.md');
expect(transformedContent).not.toContain('md-link-in-multi-line-comment.md');
expect(content).not.toEqual(transformedContent);
});

Expand Down
3 changes: 3 additions & 0 deletions packages/docusaurus-utils/package.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,13 @@
"gray-matter": "^4.0.3",
"lodash": "^4.17.20",
"micromatch": "^4.0.4",
"remark": "^12.0.0",
"remark-mdx": "^1.6.22",
"remark-mdx-remove-exports": "^1.6.22",
"remark-mdx-remove-imports": "^1.6.22",
"resolve-pathname": "^3.0.0",
"tslib": "^2.3.1",
"unist-util-visit": "^2.0.2",
"url-loader": "^4.1.1"
},
"engines": {
Expand Down
41 changes: 32 additions & 9 deletions packages/docusaurus-utils/src/__tests__/markdownParser.test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -106,10 +106,10 @@ describe('createExcerpt', () => {
expect(
createExcerpt(dedent`
import Component from '@site/src/components/Component';
import Component from '@site/src/components/Component'
import Component2 from '@site/src/components/Component2'
import './styles.css';

export function ItemCol(props) { return <Item {...props} className={'col col--6 margin-bottom--lg'}/> }
export function ItemCol2(props) { return <Item {...props} className={'col col--6 margin-bottom--lg'}/> }

export function ItemCol(props) { return <Item {...props} className={'col col--6 margin-bottom--lg'}/> };

Expand All @@ -118,7 +118,7 @@ describe('createExcerpt', () => {
Nunc porttitor libero nec vulputate venenatis. Nam nec rhoncus mauris. Morbi tempus est et nibh maximus, tempus venenatis arcu lobortis.
`),
).toEqual(
'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum ex urna, molestie et sagittis ut, varius ac justo.',
'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum ex urna, ~molestie~ et sagittis ut, varius ac justo :wink:.',
);
});

Expand All @@ -142,6 +142,28 @@ describe('createExcerpt', () => {
`),
).toEqual('Lorem ipsum dolor sit amet, consectetur adipiscing elit.');
});

test('should ignore comments for excerpt', () => {
expect(
createExcerpt(dedent`
<!-- hey -->

boola boola
`),
).toEqual('boola boola');
});

test('should ignore multi-line comments for excerpt', () => {
expect(
createExcerpt(dedent`
<!--
hey
-->

boola boola
`),
).toEqual('boola boola');
});
});

describe('parseMarkdownContentTitle', () => {
Expand Down Expand Up @@ -401,7 +423,7 @@ Lorem Ipsum
expect(
parseMarkdownContentTitle(markdown, {removeContentTitle: true}),
).toEqual({
content: markdown.trim().replace('# Markdown Title', ''),
content: markdown.trim().replace('# Markdown Title\n', ''),
contentTitle: 'Markdown Title',
});
});
Expand All @@ -410,7 +432,7 @@ Lorem Ipsum
const markdown = dedent`

import Component from '@site/src/components/Component';
import Component from '@site/src/components/Component'
import Component2 from '@site/src/components/Component2'
import './styles.css';

Markdown Title
Expand All @@ -429,7 +451,7 @@ Lorem Ipsum
const markdown = dedent`

import Component from '@site/src/components/Component';
import Component from '@site/src/components/Component'
import Component2 from '@site/src/components/Component2'
import './styles.css';

Markdown Title
Expand All @@ -441,7 +463,7 @@ Lorem Ipsum
expect(
parseMarkdownContentTitle(markdown, {removeContentTitle: true}),
).toEqual({
content: markdown.replace('Markdown Title\n==============\n\n', ''),
content: markdown.replace('Markdown Title\n==============\n', ''),
contentTitle: 'Markdown Title',
});
});
Expand Down Expand Up @@ -544,7 +566,7 @@ Lorem Ipsum
expect(
parseMarkdownContentTitle(markdown, {removeContentTitle: true}),
).toEqual({
content: markdown.replace('# Markdown Title', ''),
content: markdown.replace('# Markdown Title\n', ''),
contentTitle: 'Markdown Title',
});
});
Expand Down Expand Up @@ -822,7 +844,8 @@ describe('parseMarkdownString', () => {
### test
test3",
"contentTitle": "test",
"excerpt": "test test test test test test",
"excerpt": "test test test test test test
test test test # test bar",
"frontMatter": Object {},
}
`);
Expand Down
2 changes: 2 additions & 0 deletions packages/docusaurus-utils/src/deps.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,5 +10,7 @@ declare module 'resolve-pathname' {
}

declare module '@mdx-js/runtime';
declare module 'mdast-util-mdx';
declare module 'remark-mdx';
declare module 'remark-mdx-remove-imports';
declare module 'remark-mdx-remove-exports';
18 changes: 0 additions & 18 deletions packages/docusaurus-utils/src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -420,21 +420,3 @@ export function updateTranslationFileMessages(
})),
};
}

// Input: ## Some heading {#some-heading}
// Output: {text: "## Some heading", id: "some-heading"}
export function parseMarkdownHeadingId(heading: string): {
text: string;
id?: string;
} {
const customHeadingIdRegex = /^(.*?)\s*\{#([\w-]+)\}$/;
const matches = customHeadingIdRegex.exec(heading);
if (matches) {
return {
text: matches[1],
id: matches[2],
};
} else {
return {text: heading, id: undefined};
}
}
7 changes: 6 additions & 1 deletion packages/docusaurus-utils/src/markdownLinks.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,20 +32,25 @@ export type ReplaceMarkdownLinksReturn<T extends ContentPaths> = {
brokenMarkdownLinks: BrokenMarkdownLink<T>[];
};

const stripHtmlComments = (fileString: string) => {
return fileString.replace(/<!--.*?-->/gs, '');
};
Copy link

@Jarod42 Jarod42 Oct 11, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Notice that it would also remove that text when it is inside another block as string or fenced codeblock.

Even worse when start-comment and end-comment are in separated blocks.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree

Even worse when start-comment and end-comment are in separated blocks.

Not sure what you mean here?

Copy link

@Jarod42 Jarod42 Oct 13, 2021 •
edited by Josh-Cena
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My first comment was for fenced codeblock (or any "block") for example:

```md
html comment have the following form <!-- comment with possibly [link_in_block_considered_in_comment](link.md) -->"
```

for which the removal for the tools might not be important.

but cases like 2 separates blocks:

```md
start of html comment is '<!--'
```
[link_falsely_considered_as_in_comment](link.md) 
```md
end of html comment is '-->'
```

would be more problematic.


export function replaceMarkdownLinks<T extends ContentPaths>({
siteDir,
fileString,
filePath,
contentPaths,
sourceToPermalink,
}: ReplaceMarkdownLinksParams<T>): ReplaceMarkdownLinksReturn<T> {
const fileStringWithoutHtmlComments = stripHtmlComments(fileString);
const {contentPath, contentPathLocalized} = contentPaths;

const brokenMarkdownLinks: BrokenMarkdownLink<T>[] = [];

// Replace internal markdown linking (except in fenced blocks).
let fencedBlock = false;
const lines = fileString.split('\n').map((line) => {
const lines = fileStringWithoutHtmlComments.split('\n').map((line) => {
if (line.trim().startsWith('```')) {
fencedBlock = !fencedBlock;
}
Expand Down
Loading