Rework documentation parser #446

MangelMaxime · 2019-07-28T20:00:28Z

⚠️ Do not merge yet, this is till a WIP ⚠️

This PR is related to ionide/ionide-vscode-fsharp#1012

Its goal is to rework the documentation parser.

Today, we have more or less 3 parsers:

Inside FsAutoComplete.LspHelpers.Markdown which takes a raw string an apply some regex against it
The same as 1. but copied inside Ionide Util.fs
The TipFormatter which use a combination of regex and XML parsing.

I proposed to @Krzysztof-Cieslak the possibility to only have one implementation and he was ok with that.

ionide/ionide-vscode-fsharp#1012 (comment)

Yes, markdown module need to go away, I think we use it only in Info Panel, but we should be able to apply same transformations on FSAC side

Demo of the current state

One parser to rule them all

So I am removing the parsers 1 and 2 and keeps parser 3.

Mutualize as much code as possible

All the following elements use the same abstraction:

code
c
a
para
block
see
xref
paramref
typeparamref
table

This abstraction support both "void and non-void" version of the elements and use the same regex to detect them.

Example:

<paramref name="c">c</paramref>
<paramref name="c" />

Implementation

We can them in the Formatter function, we decide how to retrieve the data depending on the version of the element.

Non standard features

Support `lang` inside `<code>`

ℹ️ I am "the creator" of this feature

You can add a lang attributes on a <code> block. This will alows us to provide coloration in the tooltip.

Support for `-or-` blocks.

ℹ️ Microsoft is using theses blocks

If the -or- block is on a single line Microsoft doesn't reformat it making it hard to read the different cases.

My implementation will always try to optimise the display if possible (for example, inside a table cell it will leave it as it is).

[...]  -or-  [...]  -or-  [...]

or

[...]
-or-
[...]
-or-
[...]

generates the same output:

>    [...]

*or*

>    [...]

*or*

>    [...]

Preview

Table support

ℹ️ Microsoft is using theses blocks

This is already supported in current version of FSAC.

`xref` block supports

ℹ️ Microsoft is using theses blocks instead of paramref or typeparamref

Fix invalid URLs

ℹ️ Microsoft is using theses blocks

Replace ~/docs/standard/cross-platform/cross-platform-development-with-the-portable-class-library.md by https://docs.microsoft.com/en-gb/dotnet/standard/cross-platform/cross-platform-development-with-the-portable-class-library

This is needed because of some links used inside of netstandard.xml

Questions

On twitter some people complained that the tooltip could be too verbose, so my question should we make the tooltip less verbose?

If yes, what info should we remove?

Here are the info we can choose from:

        "**Description**" + nl + nl
        + summary
        + Section.fromList "" remarks
        + Section.fromMap "Type parameters" typeParams
        + Section.fromMap "Parameters" parameters
        + Section.fromOption "Returns" returns
        + Section.fromMap "Exceptions" exceptions
        + Section.fromList "Examples" examples
        + Section.fromList "See also" seeAlso

TODO

Check that everything is still working after the or block addition
- Fix multilines code block, its seems I am removing their empty lines and significant EOL in it
Add missing bullet list support
- ~~When used to generates info panel, add bullet type support (this is not possible to do it in raw markdown so we need html to do it)~~
Use the parser to generate the info panel documentation [related to https://github.com/WIP: Rework documentation parser ionide-vscode-fsharp#1259)
~~Add configuration support to the parser in order to produce different outputs~~
- ~~When used to generates info panel, add command supports~~
Decide what to show in the tooltip

MangelMaxime · 2019-07-30T20:22:13Z

cc @Krzysztof-Cieslak @baronfel

Hello, I need your help to interpret the list tag documentation.

How should we represent the listheader thing? What the term thing and how to represent it?

Do you have any example usage of the list tag. I search for <list in netstandard.xml but it's not used in it...

baronfel · 2019-07-30T20:58:01Z

I found a more detailed example from an old blog post (not microsoft-affilated).

After reading the docs you linked and this post, I've come to the following sort of data-model for the list structures available to Xml Documentation:

type Term = string
type Definition = string

type ListStyle = Bulleted | Numbered

type ItemList =
/// a list where the items are just single-strings (ie a markdown list item that looks like: * Some text here)
| Simple of elements: string list * style : ListStyle
/// a list where the items are a term followed by a definition (ie in markdown: * <TERM> - <DEFINITION>)
| Definitions of elements: (Term * Definition) list * style: ListStyle

type ColumnHeader = string

// the length of the rows is equal the the length of the headers list
type ItemTable = Table of headers: ColumnHeader list * rows: (Term list) list

/// the overall xmldoc structure can be parsed into this
type XmlListStructure = 
| List of ItemList
| Table of ItemTable

That is, there are two kinds of 'list item' (single terms and full-definitions), two kinds of list (bulleted and numbered) and then a way to represent tables.

I think all of these could be translated into markdown equivalents fairly mechanically:

simple lists would translate to either: * <element> or 1. <element> depending on style
definitions lists would translate to either * **<term>** - <definition> or 1. **<term> - <definition> depending on style
tables would translate to markdown tables in a one-to-one fashion.

This is all assuming we get full-markdown in the tooltips/info panel.

MangelMaxime · 2019-07-31T06:22:36Z

@baronfel Thank you for your analysis.

I will try to implement the list parser now that I have examples and better comprehension of the block :)

Next step is to update the regex to support void & closing version of a tag

MangelMaxime · 2019-11-12T15:09:14Z

List support is slowly being added:

Simple bulleted list

Simple numbered list

Term & definition bulleted list

Mix (simple | term & definition) bulleted list

MangelMaxime · 2019-11-12T16:26:33Z

Here is the table support

MangelMaxime · 2019-11-18T14:56:43Z

Hello @Krzysztof-Cieslak @baronfel @7sharp9

I now have all the syntax covered by the new parser. And would like to tick the todo:

Decide what to show in the tooltip

Should we put everything in the tooltip or only the summary content?

7sharp9 · 2019-11-18T23:32:19Z

I think just type info and summary, the other stuff should go in the doc viewer.

MangelMaxime · 2019-11-19T10:41:03Z

To get the new parser out, I am postponing the commands generation. So now, we only need to decide what to include in the tooltip.

Adding commands for all the tags like cref, see, etc. will not be as easy as initially planned that's also why I am postponing it.

Krzysztof-Cieslak · 2019-11-25T11:30:22Z

Decide what to show in the tooltip

Maybe have single option? Something like full vs minimal? With Full we'd show everything, with minimal only summary?

MangelMaxime · 2019-11-25T12:55:07Z

Maybe have single option?

I am not sure to understand, do you want to add a setting so the user can configure it from his editor?

Krzysztof-Cieslak · 2019-12-05T16:52:13Z

I think there should be setting tooltipMode with possible values simple or full
For simple display just type info and summary, for full display everything.
Default... I think full would be good default in spirit of rich features that Ionide provides out of the box

baronfel · 2019-12-05T16:54:28Z

I'd agree with @Krzysztof-Cieslak. Defaulting to more info is how ionide works, and if a user prefers less information they can set one flag in their user profile settings to get that behavior across all Ionide instances.

7sharp9 · 2019-12-05T18:24:56Z

Biggest issue is not ionide but the tiny tot tooltip window in vscode. It's already too tiny for the simple info without hacking it larger.

…

On Thu, 5 Dec 2019 16:54 Chet Husk, ***@***.***> wrote: I'd agree with @Krzysztof-Cieslak <https://github.com/Krzysztof-Cieslak>. Defaulting to more info is how ionide works, and if a user prefers less information they can set one flag in their user profile settings to get that behavior across all Ionide instances. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#446>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEPXSRO6F772MI7U2FKPHLQXEW4JANCNFSM4IHNJXFA> .

MangelMaxime · 2020-01-06T13:16:40Z

Hello guys,

sorry for the long silence, I was taking a lot of vacations.

Thank you for the input I will update the PR to include the settings.

MangelMaxime · 2020-03-09T20:31:24Z

Ok, I added the option to control the tooltip mode.

…_formatter

MangelMaxime · 2020-03-09T20:43:18Z

@baronfel @Krzysztof-Cieslak

This finally is ready for review !!! 🎉 🎉

You should use Ionide from this PR ionide/ionide-vscode-fsharp#1259 in order to test it.

src/FsAutoComplete/FsAutoComplete.Lsp.fs

Krzysztof-Cieslak · 2020-03-10T13:21:38Z

Well, let's do this. Awesome work Maxime, thanks a lot!

MangelMaxime mentioned this pull request Jul 28, 2019

Can the format of the auto-generated doc comments be updated? ionide/ionide-vscode-fsharp#1012

Closed

MangelMaxime force-pushed the feature/doc_comment_formatter branch from 301e5d2 to 548d488 Compare November 8, 2019 15:47

Maxime Mangel added 9 commits November 12, 2019 11:19

Upgrade to LTS version

2bc831b

Rework the doc comment formatter code

f63ab05

WIP: Format works for XML from library minus bullet list

be226a4

More fault tolerant tagName / attributes parser

0dc2137

Next step is to update the regex to support void & closing version of a tag

Add void/non-void element support + or block parser + fix ~/ urls

bc7ea16

Support doc comment formatting for user code too

82469e8

Remove XML doc regex based parser

89872a2

Remove unneeded string replacement

41a4cf1

Add indentationSize detection for raw text

7e4570a

MangelMaxime force-pushed the feature/doc_comment_formatter branch from 548d488 to 7e4570a Compare November 12, 2019 11:02

WIP: Add list support to the TipFormatter

0490a97

WIP: Add List of type table support

811229f

MangelMaxime mentioned this pull request Nov 18, 2019

WIP: Rework documentation parser ionide/ionide-vscode-fsharp#1259

Merged

Maxime Mangel added 4 commits November 19, 2019 11:41

Generates cleaner markdown table output

2feb860

Remove log

064f4b6

Add a fallback for supporting keywords documentation

0c8eb10

Rework the format style chooser

5d28087

MangelMaxime added 3 commits March 6, 2020 06:17

Merge branch 'master' into feature/doc_comment_formatter

6b3a5f3

Use the new logger system

00dd8b6

Add option to configure tooltip mode

969bcb2

MangelMaxime added 2 commits March 9, 2020 21:39

Add TooltipMode value to test Helpers

22fef0b

Merge remote-tracking branch 'origin/master' into feature/doc_comment…

5c06730

…_formatter

MangelMaxime changed the title ~~WIP: Rework documentation parser~~ Rework documentation parser Mar 9, 2020

Krzysztof-Cieslak reviewed Mar 9, 2020

View reviewed changes

src/FsAutoComplete/FsAutoComplete.Lsp.fs Show resolved Hide resolved

Krzysztof-Cieslak merged commit 2d8b0ec into ionide:master Mar 10, 2020

MangelMaxime deleted the feature/doc_comment_formatter branch August 17, 2022 08:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Rework documentation parser #446

Rework documentation parser #446

MangelMaxime commented Jul 28, 2019 •

edited

Loading

MangelMaxime commented Jul 30, 2019

baronfel commented Jul 30, 2019

MangelMaxime commented Jul 31, 2019

MangelMaxime commented Nov 12, 2019

MangelMaxime commented Nov 12, 2019

MangelMaxime commented Nov 18, 2019

7sharp9 commented Nov 18, 2019

MangelMaxime commented Nov 19, 2019

Krzysztof-Cieslak commented Nov 25, 2019

MangelMaxime commented Nov 25, 2019

Krzysztof-Cieslak commented Dec 5, 2019

baronfel commented Dec 5, 2019

7sharp9 commented Dec 5, 2019 via email

MangelMaxime commented Jan 6, 2020

MangelMaxime commented Mar 9, 2020 •

edited

Loading

MangelMaxime commented Mar 9, 2020

Krzysztof-Cieslak commented Mar 10, 2020

Rework documentation parser #446

Rework documentation parser #446

Conversation

MangelMaxime commented Jul 28, 2019 • edited Loading

Demo of the current state

One parser to rule them all

Mutualize as much code as possible

Non standard features

Support lang inside <code>

Support for -or- blocks.

Table support

xref block supports

Fix invalid URLs

Questions

TODO

MangelMaxime commented Jul 30, 2019

baronfel commented Jul 30, 2019

MangelMaxime commented Jul 31, 2019

MangelMaxime commented Nov 12, 2019

MangelMaxime commented Nov 12, 2019

MangelMaxime commented Nov 18, 2019

7sharp9 commented Nov 18, 2019

MangelMaxime commented Nov 19, 2019

Krzysztof-Cieslak commented Nov 25, 2019

MangelMaxime commented Nov 25, 2019

Krzysztof-Cieslak commented Dec 5, 2019

baronfel commented Dec 5, 2019

7sharp9 commented Dec 5, 2019 via email

MangelMaxime commented Jan 6, 2020

MangelMaxime commented Mar 9, 2020 • edited Loading

MangelMaxime commented Mar 9, 2020

Krzysztof-Cieslak commented Mar 10, 2020

MangelMaxime commented Jul 28, 2019 •

edited

Loading

Support `lang` inside `<code>`

Support for `-or-` blocks.

`xref` block supports

MangelMaxime commented Mar 9, 2020 •

edited

Loading