Remove Razor doc template, replace by MiniScaffold/Waypoint/DotLiquid techniques

[dsyme]

FSharp.Formatting is a tool that takes four input formats (snippets, markdown, literate scripts, F#/.NET asssemblies) and produces two output formats (latex and html). Razor templating and a command line tool is added on top of this.

At the moment the package is split into multiple sub-packages along the lines of the input formats. This fractures the implementation, so yet another package (FSharp.Formatting.Common) is needed.

In practice FSharp.Formatting is built, tested and used as a single technology (with the exception of the Razor support which people may avoid). Using the technology is somewhat confusing because of the multiple bits involved.

Given this, I believe the split into multiple assemblies and packages is a waste of engineering effort (maintaining multiple assemblies and packages is not simple). We should have one package (FSharp.Formatting) and that's all.

This comes up in the context of #521, where the implementation would be simpler of everything was just one component

[dsyme]

At the moment the package is split into multiple sub-packages along the lines of the input formats. This fractures the implementation, so yet another package (FSharp.Formatting.Common) is needed.

I was wrong, there is actually only one pacakge including everything.

This makes this less important (it's the maintenance of multiple packages I find most annoying)

[dsyme]

@eiriktsarpalis @matthid Do you know of any reason why we make FSharp.Literate and FSharp.Formatting.Razor available as separate packages?

• FSharp.Literate is smaller since it doesn't include Razor support, but I can't find any projects using that alone

• FSharp.Formatting includes FSharp.Formatting.Razor in any case - we could separate these but there doesn't seem any compelling need to right now.

[eiriktsarpalis]

fssnip uses FSharp.Literate only.

[dsyme]

Hmmm would it matter if the dependency was FSharp.Formatting?

[dsyme]

Looking at things I think it wouldn't matter.

[thinkbeforecoding]

FSharp.Literate was split when ported to netstandard2.0 when FSharp.Formatting could not be ported due to Razor. As long as using FSharp.Formatting as a library is available, it's ok to stop FSharp.Literate. http://thinkbeforecoding.com

…

On Fri, Jul 3, 2020 at 1:30 PM Don Syme ***@***.***> wrote: Looking at things I think it wouldn't matter. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#523 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AABD2HJYYLTHDIQHIOND7NTRZW6L7ANCNFSM4ONNMHXA> .

[dsyme]

Thanks, yes, that appears to be the case

[eiriktsarpalis]

I'm mostly put off by the fact that it has the RazorEngine.NetCore dependency. That particular library depends on aspnetcore 2.x shared framework components, that I expect might not work in future .NET releases. Ideally the razor dependency should be replaced before the two packages are merged.

[eiriktsarpalis]

FSharp.Formatting is more of a glue package bundling FSharp.Literate, FSharp.Formatting.Razor and all their transitive dependencies for ease of consumption in F# scripts. The inclusion of #r nuget: in F# 5.0 will probably render it obsolete.

If anything, I'd be in favour of eventually deprecating FSharp.Formatting, at least in its current shape.

[dsyme]

I'm mostly put off by the fact that it has the RazorEngine.NetCore dependency. That particular library depends on aspnetcore 2.x shared framework components, that I expect might not work in future .NET releases. Ideally the razor dependency should be replaced before the two packages are merged.

Yes, I understand that concern, though FSharp.Formatting appears basically usable without Razor.

The Razor dependencies appear to be .NET Standard 2.0 packages - is there a concern that these will stop working? It doesn't feel like we're picking up any shared framework dependencies any more? Or how could I tell if we are?

    <PackageReference Include="RazorEngine.NetCore" Version="[2.2.6, 3.0)" />
    <PackageReference Include="Microsoft.AspNetCore.Razor" Version="[2.2, 3.0)" />
    <PackageReference Include="Microsoft.AspNetCore.Razor.Runtime" Version="[2.2, 3.0)" />
    <PackageReference Include="Microsoft.AspNetCore.Razor.Language" Version="[2.2, 3.0)" />
    <PackageReference Include="Microsoft.CodeAnalysis.CSharp" Version="[3.4, 4.0)" />

[eiriktsarpalis]

Based on what I've been told, the netstandard2.0 aspnetcore packages depend on .NET Core 2.x shared framework components, which is why they were discontinued in .NET Core 3.0 (aspnetcore is now bundled as a shared framework). I don't know whether this impacts the code paths used by FSharp.Formatting.Razor.

[dsyme]

I see

[eiriktsarpalis]

FWIW fssnip uses DotLiquid for templating. It looks like a well-maintained, zero-dependency alternative. It might be worth considering as a potential replacement in future releases.

[matthid]

I feel like part of this has been discussed here
And the linked issues. I still think that is a good path forward. This suggestion here feels like a step back?

[dsyme]

Yeah, I was under the impression we'd somehow broken free of the shared dependencies and Razor wasn't an issue now we're working on .NET COre, but we haven't. Ugh

Anyway we can keep the FSharp.Literate package, it's ok.

[dsyme]

BTW in v5.0 I will rename

FSharp.Literate --> FSharp.Formatting.Literate
FSharp.Markdown --> FSharp.Formatting.Markdown
FSharp.CodeFormat --> FSharp.Formatting.CodeFormat

We should never have been occupying so many of these 2nd tier namespaces with what is really one technology.

[dsyme]

I will close this for now. If someone could help us break free of Razor for the assembly generation docs it would really great. I don't really care how much damage is done along the way as long as assembly docs come out the other end.

This would allow us to have one package FSharp.Formatting plain and simple. We have to simplify this business if we're to maintain it going forward

[Krzysztof-Cieslak]

If someone could help us break free of Razor for the assembly generation docs it would really great

Well, both @TheAngryByrd's MiniScaffold and Ionide's Waypoint are not using Razor for API reference generation, and those are probably two most popular solution scaffolds at the moment

[dsyme]

If someone could help us break free of Razor for the assembly generation docs it would really great

Well, both @TheAngryByrd's MiniScaffold and Ionide's Waypoint are not using Razor for API reference generation, and those are probably two most popular solution scaffolds at the moment

Yes, it would be great if someone could just come in and nuke the API doc generation in this repo and replace it with something half decent not depending on Razor.

The literate programming is the part of this repo I really want to save (e.g. converting scripts to notebooks, html, latex etc.). But it seems right that this repo should also have API doc generation - they make a reasonable combination rather than trying to use two systems.

Do you think we could just remove the doc generation in this repo and replace it with the WayPoint doc generation? Merge the efforts?

[Krzysztof-Cieslak]

Jimmy can tell us details about his solution, but as far as I understand both Waypoint and MiniScaffold depends on MetadataFormat API from this repo.

Waypoint additionally depends on FSharp.Literate for XML comment -> nice HTML transformation

[dsyme]

Jimmy can tell us details about his solution, but as far as I understand both Waypoint and MiniScaffold depends on MetadataFormat API from this repo.

Right, exactly, it's just the templating part for the API docs that's garbage here and causing the complexity nightmare with Razor. It's not even a lot of code.

So I'd like to make it plan of record replace the templating here used for doc generation and put in whatever you're using for Waypoint and/or MiniScaffold and iterate/improve that here. So then the "fsformatting" tool just works out of the box (and we drop the Razor support in this repo). You will still be able to use the MatadataFormat API if you want.

This is what I will plan to use for DiffSharp. I will look into making this change later.

I will be working hard to make the DiffSharp docs look really good, so I hope to make the defaults work well.

[dsyme]

Waypoint additionally depends on FSharp.Literate for XML comment -> nice HTML transformation

Just to mention that WriteHtml overload will be renamed to ToHtml in v5.0.

[dsyme]

For those following this discussion, the main question is what to do about the Razor-dependent API doc generation templating in this repo going forward.

It seems there are currently three suer-facing API doc generation solutions around, which use some of the functionality in this repo but not all (there may be more I just don't know about yet). Notes to myself:

Current FSharp.Formatting.RazorMetadataFormat.Generate

• Uses FSharp.Formatting.MetadataFormat to generate basic information about types, namespaces etc. This includes important code to format strings for signatures of members etc.

• Uses Razor for templating to take this information and generate HTML pages

MiniScaffold

• Also uses FSharp.Formatting.MetadataFormat.Generate to generate basic information about types, namespaces etc.

• Uses Fable.React (an HTML DSL plus Fable.ReactServer.renderToString) for templating to take this information and generate HTML pages

• An example generated site is here https://www.jimmybyrd.me/miniscaffold-docs-test/

Waypoint

• Also uses FSharp.Formatting.MetadataFormat.Generate to generate basic information about types, namespaces etc.

• Uses Fornax static site generation (including an HTML DSL) for templating, and to take this information and generate HTML pages

• An example generated site is here http://kcieslak.io/SampleWaypoint/explanations/overview.html

Seems like options for this repo are

1. Rip out the use of Razor and offer no actual API doc generation. Let other solutions like Fornax/Waypoint and MiniScaffold blossom.

2. Use DotLiquid for HTML templating.

3. Offer HTML generation without such sophisticated templating

[dsyme]

OK I've finally decided the way forward here.

First, there will be no sophisticated templating customization in FSharp.Formatting.ApiDocs (previously called FSharp.Formatting.MetadataFormat). So there will be no dependency on any templating engine. I've decided that any dependency on any templating engine for such a foundational component is a nightmare and basically unnecessary, also getting our layering wrong.

Instead we will:

1. Continue to provide and improve the ApiDocs model so tools like Waypoint and MiniScaffold can continue to do high quality doc generation

2. We will bake in a default generation of ApiDocs ApiDocs.GenerateHtml that does a reasonable job of generating quality HTML docs. The implementation will be similar to Waypoint and I'll work with @Krzysztof-Cieslak on trying to get these aligned and getting searchable docs generated. There will be no customization of these docs except via an outer template (with a simple substitution of {document} and properties), plus CSS styling.

Also the ApiDocsModel (previously called GeneratorOutput provided by FSharp.Formatting.ApiDocs needs a lot of work. There are undefined type inference variables showing, for example.

[Krzysztof-Cieslak]

TBF, I believe that F#.Formatting shouldn't generate documentation on its own at all - we should instead work on making current Waypoint documentation template standalone (i.e. easily usable with just Fornax, without Waypoint solution scaffold) where ideal workflow would be:

1. dotnet tool install fornax
2. dotnet fornax new --docs that would generate current documentation template (using FSharp.Formatting.ApiDocs)
3. Edit some config .fsx file with paths to projects/solution name/etc
4. dotnet fornax build/watch

I'd rather have F#.Formatting focus on being a good library (i.e. providing good ApiDocs) rather than attempting to reimplement poorer version of what other solutions already provide.

EDIT:
This comment is about ApiDocs generation part, FSharp.Formatting.Literate looks good to me. I think the CLI tool provided by F#.Formatting should only handle Literate programming part.

[dsyme]

My comments from chat with @Krzysztof-Cieslak below

The problem is that I can't cope with a two or three layer dependency in this part of the toolchain. I feel like I need a one-stop shop, partly for simplicity as a user, and partly for simplicity in iteratively making improvements (the API docs model coming out of FSHarp.Formatting needs a huge amount of work, it's just not the basic for good lookin gsignatures etc.)

Hard to explain, but I'm not too thrilled by using customizable .fsx scripted site generation for project docs. I guess I want a single-shop doc generation solution that's entirely content based apart from an outer template. Nothing else checked in, no generator scripts, nothing. But yes, there's definite overlap between static site generation and project docs, I can see the issue in overlap for sure.

So my current holding pattern is to get a Razor-free static doc generation working in FSharp.Formatting, good enough to use for DiffSharp, then focus on improvements to the Api Docs model layer to get that looking really good. That puts me in a stable position and has some advantages for the whole ecosystem - both Waypoint and MiniScaffolf consume the ApiDocModel (prev called GeneratorOutput). And we needed to get rid of that Razor dependency really badly, so that's progress

SO right now my plan is:

• FSharp.Formatting and its CLI tool doing some kind of basic-but-complete content-based doc generation

• Joint work on improving the FSharp.Formatting ApiDocsModel consumed by everyone

• Fornax/Waypoint/MiniScaffold consuming the FSharp.Formatting ApiDocsModel to do better and more innovative docs generation e.g. the Lunr search index

However I can see I'm going to be tempted to put things like the Lunr search index generation into here.

[Krzysztof-Cieslak]

However I can see I'm going to be tempted to put things like the Lunr search index generation into here

Index generation itself is fairly straightforward - it's basically a list of type Entry = { uri: string; title: string; content: string }, with each entry representing single page. However, it requires some wiring-up on the client (web page) site - some js, some css, search box etc.

[dsyme]

Index generation itself is fairly straightforward - it's basically a list of type Entry = { uri: string; title: string; content: string }, with each entry representing single page. However, it requires some wiring-up on the client (web page) site - some js, some css, search box etc.

Yup, I was looking over the SampleWaypoint code for this. It seems reasonable to integrate in here into a "basic" story without things getting too out of hand.

[dsyme]

I'll close this issue since Razor has now been nuked. We can discuss in further issues

[dsyme]

@Krzysztof-Cieslak Just to say I guess #528 takes things a step further.

• Renames the fsformatting tool to fsdocs
• Has fsdocs build do something useful for a typical F# library repo, template driven.

It's far from perfect (and has a perf issue cracking project files) but it is like the entry level simplicity of project doc generation

[dsyme]

OK I'm really closing this one now, the work is most definitely done.

FSharp.Formatting 5.0.4 has been published and the documentation updated.