Generate reference documentation and XSD from metadata

[ppkarwasz]

Log4j plugins follow a slightly modified JavaBean model. As a first step of #1954 we should:

• research the available metadata formats that could be employed to document Log4j plugins,
• choose the most appropriate metadata format based on: maintenance state, available tools and features. An in-house solution should have lowest priority (cf. NIH),
• use an example metadata file to generate an XSD schema,
• use an example metadata file to generate reference documentation pages.

[ppkarwasz]

A loose list of metadata formats we might consider:

• an XML Schema itself,
• a Plexus component descriptor,
• a Maven plugin descriptor, which is used by the Maven Plugin plugin to generate documentation,
• a Modello model that can be converted in many other formats,
• Tomcat MBean descriptors,
• OSGI Declarative Services descriptors,
• maybe the Eclipse Modelling Framework can help, although it seems like an overkill.

[vy]

A loose list of metadata formats we might consider:

• an XML Schema itself,
• a Plexus component descriptor,
• a Maven plugin descriptor, which is used by the Maven Plugin plugin to generate documentation,
• a Modello model that can be converted in many other formats.

@ppkarwasz, I completely support the research phase you described. Though I will also appreciate some non-Maven-native solutions (besides an XSD) too.

[ppkarwasz]

@ppkarwasz, I completely support the research phase you described. Though I will also appreciate some non-Maven-native solutions (besides an XSD) too.

Sure thing, my research is not complete. Maven-related formats were just the easiest to find.

[jvz]

Neat, this is an idea I've had in the past. It would be great if we had proper XSDs that made it easier to edit a log4j configuration file. Might be useful to see if this schema can be represented as a JsonSchema, too, so it can be used for JSON and YAML configuration files.

One potential source of inspiration here would be whatever Spring Boot does for making your various @ConfigurationProperties classes autocomplete when modifying Spring properties configurations.

I should note that various old plugins still use strings for various plugin attributes that should be a different data type, but the initial plugin system only supported string values for attributes as it mimicked the DOM API here. Thus, if you use the type info of a plugin attribute, note that a string type is not necessarily authoritative.

[ppkarwasz]

The main plan here is to have a documentation like Maven plugins (e.g. Maven Compiler Plugin). These are generated without a single line of text (besides the *.java files). I would like something like this for each plugin:

Parameter	Type	Description
direct	boolean	Specifies whether to use System.out/err or the file descriptors
layout	Layout	The layout to use. Common layouts: list of links. (all layouts link)

The XSD will be a byproduct of this process, but the metadata must be a little bit larger: in February-March we plan to write an online configuration builder/validation tool. This tool might also spit up a list of Maven dependencies required by the configuration.

[jvz]

Oh that all sounds fantastic!

[ppkarwasz]

Our plugins can be approximately modeled as JavaBeans, i.e. as Java classes with a set of documented properties, so a descriptor of a Log4j plugin must allow:

• to describe a class and its hierarchy (both superclass and interfaces),
• to describe its properties, which can be primitive Java values, classes or interfaces: our current configuration visitor binds primitive values and Strings by name and other kinds of properties by type.

The second requirement limits the list of available read-to-use component descriptors: Plexus component descriptors do not allow properties of any type. An XSD schema might also not be enough to precisely document our components: there is no standard way to convey inheritance in an XSD. We should probably leave XSD as a product of our documentation process.

[ppkarwasz]

A second requirement from the metadata format is the availability of tools to generate various formats of documentation:

• as far as I can tell OSGi Declarative Services do not have such tools,
• the was an Eclipse EMF documentation generator, but it does not seem available at the moment,
• Tomcat Configuration Reference, which was the main inspiration for this task is actually generated manually (cf. git repository).

This leaves Modello that has an impressive number of generators and the custom Maven Plugin descriptor as contenders. The documentation of Maven Plugin descriptor is actually generated by Modello.

[ppkarwasz]

An experiment performed using Modello (cf. the Log4j Documentation Reference using Modello gist), shows that even if we use a Modello descriptor to describe Log4j plugins, we would need to write the documentation logic ourselves (even for "just" an XSD).

That is why I believe:

• we should still use Modello, but to generate our own descriptor that would not approximate our components as Java Beans, but use their natural representation (plugin + plugin builder with a separate hierarchy for each of them).

[vy]

How would you compare these solutions to simply rendering an XSD/JSON and mapping it to a Map<String, Object> passed to a [FreeMarker?] template where we can generate whatever plain-text (AsciiDoc, etc.) we want? I would rather have full control over model-to-AsciiDoc conversion, since we can tailor the emitted output as we see fit.

[ppkarwasz]

@vy,

These solutions are quite rigid. The one that gave the best result (Modello) would still need some code modifications in modello-plugin-xsd to give an XSD that describe our standard configurations. If you add to this plugins that can not appear in an XSD (arbiters and other lazily evaluated stuff), we need to write most of the generating code ourselves.

Therefore I think that the easiest solution is to:

• use Modello to generate our plugin model and XML parsers/writers,
• digest the model descriptors ourselves and output some maps for FreeMarker.

The maps we probably need are:

• a map that associates the name of each plugin with a plugin descriptor,
• a map that associates all the types used in properties (e.g. Layout, Appender, etc.) with the plugins that implement those.

[vy]

1. log4j-plugins parses Log4j plugins and generates some sort of DOM model already – correct me if I am wrong. How would you compare us generating an XSD out of this DOM model versus decorating plugins with Modello and using Modello to generate an XSD?
2. When Modello is employed, do we need to use both Log4j and Modello annotations for plugins?

Given XSD is an XML too, I would favor simply passing XSD model to FreeMarker (through a simple Map<String,Object>?) and deal with representation there.

[ppkarwasz]

1. log4j-plugins parses Log4j plugins and generates some sort of DOM model already – correct me if I am wrong. How would you compare us generating an XSD out of this DOM model versus decorating plugins with Modello and using Modello to generate an XSD?

I thought to use Modello only as an implementation detail. log4j-plugins will generate some XML (right now it generates Java code). We'll use Modello to parse these XML files and generate:

1. A common XSD for all the Log4j (and in the future third-party) modules,
2. A set of documentation pages.

2. When Modello is employed, do we need to use both Log4j and Modello annotations for plugins?

No, this will be transparent for the user.

Given XSD is an XML too, I would favor simply passing XSD model to FreeMarker (through a simple Map<String,Object>?) and deal with representation there.

The XSD schema will cover only part of our plugins (those in the "Core" namespace).

[vy]

log4j-plugins will generate some XML ... We'll use Modello to parse these XML files and generate ... XSD

This translates to

plugins --(Log4j)--> XML --(Modello)--> XSD

How would you compare this to the following?

plugins --(Log4j)--> XSD

[ppkarwasz]

This translates to

plugins --(Log4j)--> XML --(Modello)--> XSD

How would you compare this to the following?

plugins --(Log4j)--> XSD

Modello will only be used to generate code for XML serialization/deserialization and the model itself. Once that is done it will be:

Source code --(`log4j-plugins-processor`)--> XML --(`log4j-docgen`)--> XSD, Asciidoc et al.

[ppkarwasz]

With the submission of the PRs above, I consider this subtask complete.