feat: add support for plantuml generated diagrams

[Silvanoc]

Adding an alternative UML generator. The advantage of this proposal over the existing one (YUML) is that it enables the usage of a service that can be self-hosted and doesn' have any license fees.

The source code is a modified version of the YUML generator.

[codecov-commenter]

Codecov Report

Merging #1509 (9fab2b9) into main (ed0c618) will increase coverage by 0.15%.
The diff coverage is 88.23%.

@@            Coverage Diff             @@
##             main    #1509      +/-   ##
==========================================
+ Coverage   79.95%   80.10%   +0.15%     
==========================================
  Files          93       94       +1     
  Lines        9711     9898     +187     
  Branches     2382     2439      +57     
==========================================
+ Hits         7764     7929     +165     
- Misses       1502     1515      +13     
- Partials      445      454       +9     

Files Changed	Coverage Δ
linkml/generators/__init__.py	100.00% <ø> (ø)
linkml/generators/plantumlgen.py	88.23% <88.23%> (ø)

[dalito]

Nice! Will it be possible to produce puml-files (or just final images)?

[Silvanoc]

Nice! Will it be possible to produce puml-files (or just final images)?

@dalito Thanks!

I've implemented it the same way as the yUML generator. If you don't pass the option -d/--directory, then you get the PlantUML code printed out.

[Silvanoc]

For backwards compatibility reasons I've kept the gen-markdown option --noyuml, although IMO it should be rather be called --nouml and apply to both PlantUML and yUML.

[sujaypatil96]

@Silvanoc: thank you for your work on this! I didn't know about the plantuml project before this, but it looks like another tool with UML diagram generation capabilities. Currently, the de facto tool that we're using within the linkml framework to generate/render UML diagrams is mermaid. The only kind of diagrams that we are rendering as part of the web documentation generator (gen-doc/docgen.py) at the moment are class diagrams (granted showing minimal relationship information, simply because we haven't prioritized it enough to improve the level of detail it shows).

I have a couple of questions for you -

1. Does the plantUML generator you wrote show more comprehensive linking and provide more valuable information than the current mermaid diagrams?
2. Why not focus on improving the detail in the mermaid diagrams rather than introducing another UML diagramming tool? Is there something lacking in mermaid that plantUML provides?

There's also an ER diagram generator in the framework: https://github.com/linkml/linkml/blob/main/linkml/generators/erdiagramgen.py

[sujaypatil96]

For backwards compatibility reasons I've kept the gen-markdown option --noyuml, although IMO it should be rather be called --nouml and apply to both PlantUML and yUML.

The gen-markdown generator has also been deprecated (it works, but it's not the default markdown documentation generator anymore) in favor of gen-doc which uses mermaid (for UML diagramming) and the jinja2 templating language. Apologies the markdown documentation guide hasn't been updated in a while: https://linkml.io/linkml/generators/markdown.html

[Silvanoc]

@Silvanoc: thank you for your work on this! I didn't know about the plantuml project before this, but it looks like another tool with UML diagram generation capabilities. Currently, the de facto tool that we're using within the linkml framework to generate/render UML diagrams is mermaid. The only kind of diagrams that we are rendering as part of the web documentation generator (gen-doc/docgen.py) at the moment are class diagrams (granted showing minimal relationship information, simply because we haven't prioritized it enough to improve the level of detail it shows).

I'm aware of it and I'm not trying to provide here a replacement for it, but offering an alternative with some advantages when talking about top-level diagrams for huge schemas.

I have a couple of questions for you -

1. Does the plantUML generator you wrote show more comprehensive linking and provide more valuable information than the current mermaid diagrams?

They are very similar WRT the diagrams that you can generate and their features. So WRT this question I wouldn't say that PlantUML has any remarkable advantage.

2. Why not focus on improving the detail in the mermaid diagrams rather than introducing another UML diagramming tool? Is there something lacking in mermaid that plantUML provides?

The main difference is that Mermaid is client generated on visualization (on the browser) and PlantUML is generating a diagram on build-time.

Build-time diagram generation has a couple of advantages:

• Less client-load
• Possibility to get an image that can be visualized with a different tool with zoom capabilities. That's relevant if you have huge and complex schemas, where the Mermaid approach IMO doesn't scale. If the individual elements are too small, bad luck.
• Easier to diagnose why a diagram is not showing as expected. The hereby contributed code let you generate PlantUML that you can review and use to diagnose issues by simply removing parts and feeding the rest to a PlantUML server.

There's also an ER diagram generator in the framework: https://github.com/linkml/linkml/blob/main/linkml/generators/erdiagramgen.py

[Silvanoc]

The gen-markdown generator has also been deprecated (it works, but it's not the default markdown documentation generator anymore) in favor of gen-doc which uses mermaid (for UML diagramming) and the jinja2 templating language. Apologies the markdown documentation guide hasn't been updated in a while: https://linkml.io/linkml/generators/markdown.html

Even better, I can remove the changes to that file from the patch 👍

[dalito]

@sujaypatil96 - FYI ,with sphinxcontrib.plantuml the plantuml diagrams can be rendered from *-puml files as svg (or other formats) and embedded in the docs. But there are probably no strong arguments for swtiching linkml-docs.

However, for users like myself that prefer plantuml this PR would be a nice addition.

[Silvanoc]

@sujaypatil96 WRT the deprecation of gen-markdown in favor of gen-doc, I think that the documentation should be updated accordingly. As of now I only find three references to gen-doc in the documentation and they cannot be considered proper documentation:

1. https://linkml.io/linkml/generators/markdown.html#alternatives
2. https://linkml.io/linkml/faq/tools.html#can-i-include-generated-documentation-in-a-sphinx-site
3. https://linkml.io/linkml/faq/tools.html#can-i-customize-the-markdown-generation-for-my-schema-site

Additionally, I'm miserably failing to the the top-level UML with --include-top-level-diagram --diagram-type uml_class_diagram. I'm getting the error NotImplementedError: This is currently handled in the jinja templates, and I don't know how to tackle it. But that might me only me not capable of getting what the tool is telling me.

Even with ER-Diagram I'm failing and I don't really know why. If I copy the result from the Markdown to an online editor, it's being properly rendered. But not on my generated pages.

[cmungall]

@sujaypatil96 can you give an update on @Silvanoc's comments above?

I think the option to have plantuml as an optional output is most welcome

[sujaypatil96]

@Silvanoc: yup, sorry the documentation's been so far behind. We've updated the linkml docs to capture the team's current recommendations on markdown documentation generation here: https://linkml.io/linkml/generators/markdown.html

That being said, based on @dalito and @cmungall's suggestions it would definitely be a nice option to have the markdown documentation generator render either Mermaid or plantuml diagrams in the generated web docs. The way you've developed it currently (as a separate module plantumlgen.py) is perfect. We can call this generator within docgen.

It would be nice if we could have unit tests for this generator. There's some documentation in our contributor guidelines: https://linkml.io/linkml/contributing/contributing.html#linkml-testing-framework, but I'm happy to work on this with you.

[Silvanoc]

It would be nice if we could have unit tests for this generator. There's some documentation in our contributor guidelines: https://linkml.io/linkml/contributing/contributing.html#linkml-testing-framework, but I'm happy to work on this with you.

I plan to add unit tests. But as of now I have higher prio tasks in an ongoing project.

[Silvanoc]

Marked as draft because adding unittests and fixing issues detected adding the tests.

[Silvanoc]

It would be nice if we could have unit tests for this generator. There's some documentation in our contributor guidelines: https://linkml.io/linkml/contributing/contributing.html#linkml-testing-framework, but I'm happy to work on this with you.

@sujaypatil96 now the unit tests are also in place. The GitHub pipelines are failing only due to the flaky codecov.

[sujaypatil96]

@Silvanoc: apologies for the late review on this PR, I/the team got caught up with other priorities and this sort of slipped through the cracks. We're bringing it back onto the front burner, and want to get this merged in at the earliest. Thank you for adding tests for your generator.

I cloned your fork of the repo, switched over to the branch where you have the plantUML generator and generated a UML visualization for the kitchen_sink.yaml schema, which is the test schema that most of our generators use. It did a fantastic job of generating a UML diagram. I checked the diagrams generated by both your generator, as well as the existing yUML generator for feature parity and I saw that everything checked out nicely.

The reason why GitHub Actions is complaining right now on this PR is because you have some upstream changes that you need to bring into your fork: Silvanoc#1, and doing so will get rid of the ruff linting error that Actions is complaining about.

[Silvanoc]

@Silvanoc: apologies for the late review on this PR, I/the team got caught up with other priorities and this sort of slipped through the cracks. We're bringing it back onto the front burner, and want to get this merged in at the earliest. Thank you for adding tests for your generator.

No problem, I appreciate very much your effort to process all PRs. It's normal that those that are not in any critical path are left behind for a while.

The reason why GitHub Actions is complaining right now on this PR is because you have some upstream changes that you need to bring into your fork: Silvanoc#1, and doing so will get rid of the ruff linting error that Actions is complaining about.

@sujaypatil96 I've noticed that your PR was equivalent to rebasing onto you upstream main. Therefore I've simply rebased instead of accepting your PR in order to have a cleaner git history. I don't like PRs to a PR-Branch simply to sync with the target branch. I very much prefer rebasing. I hope it's fine for you.

[sujaypatil96]

@sujaypatil96 I've noticed that your PR was equivalent to rebasing onto you upstream main. Therefore I've simply rebased instead of accepting your PR in order to have a cleaner git history. I don't like PRs to a PR-Branch simply to sync with the target branch. I very much prefer rebasing. I hope it's fine for you.

Great! rebasing works just as well, in fact it's better because it preserves linear git history. Merging your PR in 🚀

[VladimirAlexiev]

hi @Silvanoc and @sujaypatil96 !
I know a lot about PlantUML, so I can help to improve the look.
Eg https://linkml.io/linkml/generators/plantumlgen.html can be improved if you add this preamble and remove quotes around field names and types (like here):

hide circle
hide empty members

[Silvanoc]

hi @Silvanoc and @sujaypatil96 ! I know a lot about PlantUML, so I can help to improve the look. Eg https://linkml.io/linkml/generators/plantumlgen.html can be improved if you add this preamble (like here):

hide circle
hide empty members

@VladimirAlexiev I like your proposal, please send a PR with the proposal

[VladimirAlexiev]

@Silvanoc Why do you use Kroki and not https://www.plantuml.com/plantuml/ ? Unlike kroki, that one always has the very latest version of plantuml.

kroki_server: Optional[str] = "https://kroki.io"

[VladimirAlexiev]

send a PR with the proposal

#2060

[Silvanoc]

@Silvanoc Why do you use Kroki and not https://www.plantuml.com/plantuml/ ? Unlike kroki, that one always has the very latest version of plantuml.

kroki_server: Optional[str] = "https://kroki.io"

@VladimirAlexiev I use Kroki in order to support self-hosted diagram generators. That is in fact what we use in our company.
If I'm right, the APIs are slightly different and I didn't want to add support for both.

[dalito]

@VladimirAlexiev see also #1956 for plantuml-related problems/plans.