[generator] Add generator --with-javadoc-xml=FILE support. (#687)

Fixes: #642

Context: dotnet/android#4789
Context: dotnet/android#5200

Commit 69e1b80 added `tools/java-source-utils`, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:

        # API-30 `sources` package contains both `Object.java` and `Object.annotated.java`;
        # Skip the `.annotated.java` files
        $ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \
                | grep -v '\.annotated\.' > sources.txt
        $ java -jar java-source-utils.jar -v \
                --source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
                --output-javadoc android-javadoc.xml \
                @sources.txt

What can we *do* with the generated `android-javadoc.xml`?

`android-javadoc.xml` contains parameter names, and thus can be used
with `class-parse --docspath`; see commit 806082f.

What we *really* want to do is make the Javadoc information *useful*
to consumers of the binding assembly.  This means that we want a
C# XML Documentation file for the binding assembly.

The most straightforward way to get a C# XML Documentation File is to
emit [C# XML Documentation Comments][0] into the binding source code!

Add a new `generator --with-javadoc-xml=FILE` option.  When specified,
`FILE` will be treated as an XML file containing the output from
`java-source-utils.jar --output-javadoc` (69e1b80), and all
`<javadoc/>` elements within the XML file will be associated with C#
types and members to emit, based on the `//@jni-signature` and
`//@name` attributes, as appropriate.

When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)

To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to `Java.Interop.Tools.JavaSource.dll`,
in the new `Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser`
type, which parses the Javadoc content and translates to XML.

In addition to transforming the Javadoc comments into C# XML
documentation comments, we *also* want to provide "upstream
information" in the form of:

 1. A URL to the corresponding online Javadoc HTML documentation.
 2. A copyright notice disclaimer.

Allow provision of this information by updating
`java-source-utils.jar` to support the new options:

        --doc-copyright FILE   Copyright information for Javadoc.  Should be in
                                 mdoc(5) XML, to be held within <remarks/>.
                                 Stored in //javadoc-metadata/copyright.
        --doc-url-prefix URL   Base URL for links to documentation.
                                 Stored in //javadoc-metadata/link/@Prefix.
        --doc-url-style STYLE  STYLE of URLs to generate for member links.
                                 Stored in //javadoc-metadata/link/@Style.
                                 Supported styles include:
                                 - developer.android.com/reference@2020-Nov

The new `/api/javadoc-metadata/link@prefix` and
`/api/javadoc-metadata/link@style` XML attributes, stored within
`java-source-utils.jar --output-javadoc` XML output, allow
construction of a URL to the Java member.

For example, given:

        java -jar java-source-utils.jar \
                --doc-url-prefix https://developer.android.com/reference \
                --doc-url-style developer.android.com/reference@2020-Nov

Then `generator` can emit the C# documentation comment for
`java.lang.Object.equals(Object)`:

        /// <format type="text/html">
        ///   <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
        /// </format>

The copyright notice disclaimer is supported by
`java-source-utils.jar --doc-copyright FILE`; the contents of `FILE`
are inserted into the `/api/javadoc-metadata/copyright` element, and
will be copied into the output of every C# XML documentation block.

Example output is at:

  * <https://gist.github.com/jonpryor/004f01f4cd5ff32299ff590ba7a2fe0e>

Unfortunately, converting Javadoc to C# XML Documentation Comments is
not a "zero cost" operation.  Add a new
`generator --doc-comment-verbosity=STYLE` option to control how
"complete" the generated documentation comments are:

        --doc-comment-verbosity=STYLE
                               STYLE of C# documentation comments to emit.
                                 Defaults to `full`.  STYLE may be:
                                   * `intellisense`: emit <summary>, <param>,
                                     <returns>, <exception>.
                                   * `full`: plus <remarks>, <altmember>, ...

Using `--doc-comment-verbosity=full` will *most* impact build times.

TODO:

  * `SourceJavadocToXmldocParser` doesn't support many constructs.

[0]: https://docs.microsoft.com/en-us/dotnet/csharp/codedoc

src/Java.Interop.Tools.JavaSource/Java.Interop.Tools.JavaSource/IronyExtensions.cs

@@ -0,0 +1,28 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Text;
+using Irony.Ast;
+using Irony.Parsing;
+
+namespace Java.Interop.Tools.JavaSource {
+
+	static class IronyExtensions {
+
+		public static void MakePlusRule (this NonTerminal star, Grammar grammar, BnfTerm delimiter)
+		{
+			star.Rule = grammar.MakePlusRule (star, delimiter);
+		}
+
+		public static void MakeStarRule (this NonTerminal star, Grammar grammar, BnfTerm delimiter, BnfTerm of)
+		{
+			star.Rule = grammar.MakeStarRule (star, delimiter, of);
+		}
+
+		public static void MakeStarRule (this NonTerminal star, Grammar grammar, BnfTerm of)
+		{
+			star.Rule = grammar.MakeStarRule (star, of);
+		}
+	}
+}

src/Java.Interop.Tools.JavaSource/Java.Interop.Tools.JavaSource/JavadocInfo.cs

@@ -0,0 +1,31 @@
+using System;
+using System.Collections.Generic;
+using System.Collections.ObjectModel;
+using System.IO;
+using System.Linq;
+using System.Xml.Linq;
+
+using Irony.Ast;
+using Irony.Parsing;
+
+namespace Java.Interop.Tools.JavaSource {
+
+	sealed class JavadocInfo {
+		public  readonly    ICollection<XNode>  Exceptions  = new Collection<XNode> ();
+		public  readonly    ICollection<XNode>  Extra       = new Collection<XNode> ();
+		public  readonly    ICollection<XNode>  Remarks     = new Collection<XNode> ();
+		public  readonly    ICollection<XNode>  Parameters  = new Collection<XNode> ();
+		public  readonly    ICollection<XNode>  Returns     = new Collection<XNode> ();
+
+		public override string ToString ()
+		{
+			return new XElement ("Javadoc",
+					new XElement (nameof (Parameters),  Parameters),
+					new XElement (nameof (Remarks),     Remarks),
+					new XElement (nameof (Returns),     Returns),
+					new XElement (nameof (Exceptions),  Exceptions),
+					new XElement (nameof (Extra),       Extra))
+			.ToString ();
+		}
+	}
+}

src/Java.Interop.Tools.JavaSource/Java.Interop.Tools.JavaSource/SourceJavadocToXmldocGrammar.BlockTagsBnfTerms.cs

@@ -0,0 +1,239 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Linq;
+using System.Xml.Linq;
+
+using Irony.Ast;
+using Irony.Parsing;
+
+namespace Java.Interop.Tools.JavaSource {
+
+	public partial class SourceJavadocToXmldocGrammar {
+
+		// https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javadoc.html#javadoctags
+		public class BlockTagsBnfTerms {
+
+			internal BlockTagsBnfTerms ()
+			{
+			}
+
+			internal void CreateRules (SourceJavadocToXmldocGrammar grammar)
+			{
+				AllBlockTerms.Rule = AuthorDeclaration
+					| ApiSinceDeclaration
+					| DeprecatedDeclaration
+					| DeprecatedSinceDeclaration
+					| ExceptionDeclaration
+					| ParamDeclaration
+					| ReturnDeclaration
+					| SeeDeclaration
+					| SerialDataDeclaration
+					| SerialFieldDeclaration
+					| SinceDeclaration
+					| ThrowsDeclaration
+					| UnknownTagDeclaration
+					| VersionDeclaration
+					;
+				BlockValue.Rule = grammar.HtmlTerms.ParsedCharacterData
+					| grammar.HtmlTerms.InlineDeclaration
+					;
+				BlockValues.MakePlusRule (grammar, BlockValue);
+
+				AuthorDeclaration.Rule = "@author" + BlockValues;
+				AuthorDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.AuthorTag))
+						return;
+					// Ignore; not sure how best to convert to Xmldoc
+					FinishParse (context, parseNode);
+				};
+
+				ApiSinceDeclaration.Rule = "@apiSince" + BlockValues;
+				ApiSinceDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.SinceTag)) {
+						return;
+					}
+					var p = new XElement ("para", "Added in API level ", AstNodeToXmlContent (parseNode.ChildNodes [1]), ".");
+					FinishParse (context, parseNode).Remarks.Add (p);
+					parseNode.AstNode   = p;
+				};
+
+				DeprecatedDeclaration.Rule = "@deprecated" + BlockValues;
+				DeprecatedDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.DeprecatedTag)) {
+						return;
+					}
+					var p = new XElement ("para", "This member is deprecated. ", AstNodeToXmlContent (parseNode.ChildNodes [1]));
+					FinishParse (context, parseNode).Remarks.Add (p);
+					parseNode.AstNode   = p;
+				};
+
+				DeprecatedSinceDeclaration.Rule = "@deprecatedSince" + BlockValues;
+				DeprecatedSinceDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.DeprecatedTag)) {
+						return;
+					}
+					var p = new XElement ("para", "This member was deprecated in API level ", AstNodeToXmlContent (parseNode.ChildNodes [1]), ".");
+					FinishParse (context, parseNode).Remarks.Add (p);
+					parseNode.AstNode   = p;
+				};
+
+				var nonSpaceTerm = new RegexBasedTerminal ("[^ ]", "[^ ]+") {
+					AstConfig = new AstNodeConfig {
+						NodeCreator = (context, parseNode) => parseNode.AstNode = parseNode.Token.Value,
+					},
+				};
+
+				ExceptionDeclaration.Rule = "@exception" + nonSpaceTerm + BlockValues;
+				ExceptionDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.ExceptionTag)) {
+						return;
+					}
+					// TODO: convert `nonSpaceTerm` into a proper CREF
+					var e = new XElement ("exception",
+							new XAttribute ("cref", string.Join ("", AstNodeToXmlContent (parseNode.ChildNodes [1]))),
+							AstNodeToXmlContent (parseNode.ChildNodes [2]));
+					FinishParse (context, parseNode).Exceptions.Add (e);
+					parseNode.AstNode   = e;
+				};
+
+				ParamDeclaration.Rule = "@param" + nonSpaceTerm + BlockValues;
+				ParamDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.ParamTag)) {
+						return;
+					}
+					var p = new XElement ("param",
+							new XAttribute ("name", string.Join ("", AstNodeToXmlContent (parseNode.ChildNodes [1]))),
+							AstNodeToXmlContent (parseNode.ChildNodes [2]));
+					FinishParse (context, parseNode).Parameters.Add (p);
+					parseNode.AstNode   = p;
+				};
+
+				ReturnDeclaration.Rule = "@return" + BlockValues;
+				ReturnDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.ReturnTag)) {
+						return;
+					}
+					var r = new XElement ("returns",
+							AstNodeToXmlContent (parseNode.ChildNodes [1]));
+					FinishParse (context, parseNode).Returns.Add (r);
+					parseNode.AstNode   = r;
+				};
+
+				SeeDeclaration.Rule = "@see" + BlockValues;
+				SeeDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.SeeTag)) {
+						return;
+					}
+					// TODO: @see supports multiple forms; see: https://docs.oracle.com/javase/7/docs/technotes/tools/windows/javadoc.html#see
+					var e = new XElement ("seealso",
+							new XAttribute ("cref", string.Join ("", AstNodeToXmlContent (parseNode.ChildNodes [1]))));
+					FinishParse (context, parseNode).Extra.Add (e);
+					parseNode.AstNode   = e;
+				};
+
+				SinceDeclaration.Rule = "@since" + BlockValues;
+				SinceDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.SinceTag)) {
+						return;
+					}
+					var p = new XElement ("para", "Added in ", AstNodeToXmlContent (parseNode.ChildNodes [1]), ".");
+					FinishParse (context, parseNode).Remarks.Add (p);
+					parseNode.AstNode   = p;
+				};
+
+				ThrowsDeclaration.Rule = "@throws" + nonSpaceTerm + BlockValues;
+				ThrowsDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.ExceptionTag)) {
+						return;
+					}
+					// TODO: convert `nonSpaceTerm` into a proper CREF
+					var e = new XElement ("exception",
+							new XAttribute ("cref", string.Join ("", AstNodeToXmlContent (parseNode.ChildNodes [1]))),
+							AstNodeToXmlContent (parseNode.ChildNodes [2]));
+					FinishParse (context, parseNode).Exceptions.Add (e);
+					parseNode.AstNode   = e;
+				};
+
+				// Ignore serialization informatino
+				SerialDeclaration.Rule = "@serial" + BlockValues;
+				SerialDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.SerialTag)) {
+						return;
+					}
+					FinishParse (context, parseNode);
+				};
+
+				SerialDataDeclaration.Rule = "@serialData" + BlockValues;
+				SerialDataDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.SerialTag)) {
+						return;
+					}
+					FinishParse (context, parseNode);
+				};
+
+				SerialFieldDeclaration.Rule = "@serialField" + BlockValues;
+				SerialFieldDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.SerialTag)) {
+						return;
+					}
+					FinishParse (context, parseNode);
+				};
+
+				var unknownTagTerminal = new RegexBasedTerminal ("@[unknown]", @"@\S+") {
+					Priority    = TerminalPriority.Low,
+				};
+				unknownTagTerminal.AstConfig.NodeCreator = (context, parseNode) =>
+					parseNode.AstNode = parseNode.Token.Value.ToString ();
+
+
+				UnknownTagDeclaration.Rule = unknownTagTerminal + BlockValues;
+				UnknownTagDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.Remarks)) {
+						return;
+					}
+					Console.WriteLine ($"# Unsupported @block-tag value: {parseNode.ChildNodes [0].AstNode}");
+					FinishParse (context, parseNode);
+				};
+
+				// Ignore Version
+				VersionDeclaration.Rule = "@version" + BlockValues;
+				VersionDeclaration.AstConfig.NodeCreator = (context, parseNode) => {
+					if (!grammar.ShouldImport (ImportJavadoc.VersionTag)) {
+						return;
+					}
+					FinishParse (context, parseNode);
+				};
+			}
+
+			public  readonly    NonTerminal AllBlockTerms              = new NonTerminal (nameof (AllBlockTerms), ConcatChildNodes);
+
+			public  readonly    Terminal    Cdata                      = new CharacterDataTerminal ("#CDATA", preserveLeadingWhitespace: true);
+/*
+			public  readonly    Terminal    Cdata                      = new RegexBasedTerminal (nameof (BlockValue), "[^<]*") {
+				AstConfig   = new AstNodeConfig {
+					NodeCreator = (context, parseNode) => parseNode.AstNode = parseNode.Token.Value.ToString (),
+				},
+			};
+			*/
+
+			public  readonly    NonTerminal BlockValue                 = new NonTerminal (nameof (BlockValue), ConcatChildNodes);
+			public  readonly    NonTerminal BlockValues                = new NonTerminal (nameof (BlockValues), ConcatChildNodes);
+			public  readonly    NonTerminal AuthorDeclaration          = new NonTerminal (nameof (AuthorDeclaration));
+			public  readonly    NonTerminal ApiSinceDeclaration        = new NonTerminal (nameof (ApiSinceDeclaration));
+			public  readonly    NonTerminal DeprecatedDeclaration      = new NonTerminal (nameof (DeprecatedDeclaration));
+			public  readonly    NonTerminal DeprecatedSinceDeclaration = new NonTerminal (nameof (DeprecatedSinceDeclaration));
+			public  readonly    NonTerminal ExceptionDeclaration       = new NonTerminal (nameof (ExceptionDeclaration));
+			public  readonly    NonTerminal ParamDeclaration           = new NonTerminal (nameof (ParamDeclaration));
+			public  readonly    NonTerminal ReturnDeclaration          = new NonTerminal (nameof (ReturnDeclaration));
+			public  readonly    NonTerminal SeeDeclaration             = new NonTerminal (nameof (SeeDeclaration));
+			public  readonly    NonTerminal SerialDeclaration          = new NonTerminal (nameof (SerialDeclaration));
+			public  readonly    NonTerminal SerialDataDeclaration      = new NonTerminal (nameof (SerialDataDeclaration));
+			public  readonly    NonTerminal SerialFieldDeclaration     = new NonTerminal (nameof (SerialFieldDeclaration));
+			public  readonly    NonTerminal SinceDeclaration           = new NonTerminal (nameof (SinceDeclaration));
+			public  readonly    NonTerminal ThrowsDeclaration          = new NonTerminal (nameof (ThrowsDeclaration));
+			public  readonly    NonTerminal UnknownTagDeclaration      = new NonTerminal (nameof (UnknownTagDeclaration));
+			public  readonly    NonTerminal VersionDeclaration         = new NonTerminal (nameof (VersionDeclaration));
+		}
+	}
+}