[Mono.Android] Generate Mono.Android.xml #5253
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
jonpryor
force-pushed
the
jonp-mono.android-xml-docs
branch
2 times, most recently
from
b886d61
to
52c9996
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
Current local build times… It takes ~3.5x longer for me to build I should look into disabling C# doc comments for (1) PR builds, and (2) "down-level" |
jonpryor
force-pushed
the
jonp-mono.android-xml-docs
branch
2 times, most recently
from
2afce7e
to
d649ce8
Compare
|
From the "Argh! Build Times!" department, the …Ouch. |
jonpryor
force-pushed
the
jonp-mono.android-xml-docs
branch
7 times, most recently
from
a060c7e
to
e90cd10
Compare
jonpryor
force-pushed
the
jonp-mono.android-xml-docs
branch
from
e90cd10
to
8070717
Compare
jonpryor
force-pushed
the
jonp-mono.android-xml-docs
branch
3 times, most recently
from
1759538
to
7dee454
Compare
Fixes: dotnet#4789 Context: dotnet/java-interop#687 Context: dotnet#5200 What do we want? Updated documentation! How do we get that? Uh… Historically, [Xamarin.Android API docs][0] were produced by parsing Android's HTML documentation from `docs-24_r01.zip` and converting it into [**mdoc**(5) documentation][1] via [`tools/javadoc2mdoc`]. The problem is that Google hasn't released an updated `docs*.zip` package since API-24 (~6 years ago), and thus our documentation is woefully out of date. We could have scraped developer.android.com/reference within that time frame, but web scraping is annoying, and we'd still have to deal with the pain of parsing HTML. There is an alternative, though: with API-30, there is a new `sources` package which contains the Java source code used for the API level, which in turn contains Javadoc source code comments. Previous API levels similarly contained an `android-stubs-src.jar` file which likewise contained Java source code containing Javadoc comments. The new approach is to: 1. Update `build-tools/xaprepare` to install the `sources` package. 2. Use [`java-source-tool.jar`][2] to parse the Java source code, creating an `android-javadoc.xml` file. 3. [Update `generator`][3] to consume `android-javadoc.xml` and convert the Javadocs into [C# XML documentation comments][4]. 4. Update `src/Mono.Android` so that `generator` will emit C# XML doc comments, and then produce a `Mono.Android.xml` file. The result is a `Mono.Android.xml` file which contains imported Android Javadoc documentation, e.g. <doc> <assembly><name>Mono.Android</name></assembly> <members> <member name="T:Java.Lang.Object"> <summary>Class <c>Object</c> is the root of the class hierarchy.</summary> <remarks> <para>Class <c>Object</c> is the root of the class hierarchy. … "Later", `Mono.Android.xml` can then be used alongside [`mdoc update --import=Mono.Android.xml`][5] to update our published API documentation. Finally, commit 380e95e, which added support for JDK 11, *broke* support for `@(JavaSourceJar)` when running under JDK 11, as JDK 11's Javadoc HTML output differs from what we expect. With the new `java-source-utils.jar` and `generator --with-javadoc-xml=FILE` infrastructure in place, we can reasonably address this breakage under JDK 11, fixing Issue dotnet#4789: Update `Xamarin.Android.Bindings.Core.targets` so that before invoking `generator`, we first run `java-source-utils.jar` on the `@(JavaSourceJar)` files, producing Javadoc XML files. The `<BindingsGenerator/>` task in turn is updated to accept a new `BindingsGenerator.JavadocXml` property, which is converted into a `generator --with-javadoc-xml=FILE` option. The `@(JavaSourceJar)` item group is "extended" to support the following item metadata: * `%(CopyrightFile)`: A path to a file that contains copyright information for the Javadoc contents, which will be appended to all imported documentation. * `%(UrlPrefix)`: A URL prefix to support linking to online documentation within imported documentation. * `%(UrlStyle)`: The "style" of URLs to generate when linking to online documentation. Only one style is currently supported: `developer.android.com/reference@2020-Nov`. For .NET 6 ("One .NET") integration purposes, provide a default item group of `@(JavaSourceJar)` which includes `**\*-source?.jar` and `**\*-src.jar`. This will allow `dotnet build` of an "Android Java Library Binding" project (`dotnet new android-bindinglib`) to automatically process `.java` source code for documentation translation purposes. Add `Java.Interop.Tools.Generator.dll` to the `Microsoft.Android.Sdk*.nupkg` file, as it is an assembly dependency of `generator.dll`. (Not sure why this didn't previously fail…) Finally, add a new `$(_UseLegacyJavadocImport)` MSBuild property which *disables* use of `java-source-utils.jar` for Javadoc importing, and instead use the previous, legacy, doesn't work on JDK 11, approach of using `javadoc` and HTML parsing. This shouldn't be needed, but just in case it *is* needed… [0]: https://github.com/xamarin/android-api-docs [1]: http://docs.go-mono.com/?link=man%3amdoc(5) [2]: dotnet/java-interop@69e1b80 [3]: dotnet/java-interop#687 [4]: https://docs.microsoft.com/dotnet/csharp/codedoc [5]: http://docs.go-mono.com/?link=man%3amdoc-update(1)
jonpryor
force-pushed
the
jonp-mono.android-xml-docs
branch
from
7dee454
to
3bd19e0
Compare
|
Back on the "OMG The Build Times!" front:
I disabled "down-level"
Ouch. Why? Looking at the then - 80745 ms _GenerateApiDescription 2 calls
+ 109191 ms _GenerateApiDescription 2 calls35% slower for - 126573 ms _BuildAndroidRuntimes 1 calls
+ 172634 ms _BuildAndroidRuntimes 1 calls~36% slower for This PR adds a new target, which isn't present in master: + 179719 ms _BuildAndroidJavadocXml 2 calls~3 minutes to process the API-30 sources and produce an Here we start seeing the cause of the increased build time:
Unexpected is Apparently Additionally, those Target times only amount to an increase of 21 minutes, which doesn't explain the 45 minute increase in |
I fixed what sets `$(DocumentationFile)`: * `$(OutputPath)` always ends with a trailing slash * Check `$(_ComputeFilesToPublishForRuntimeIdentifiers)` so it isn't set on the inner builds in .NET 6 * Fixed typo with `$(_UseLegacyJavadocImport)`
jonpryor
added a commit
to jonpryor/xamarin-android
that referenced
this pull request
Dec 11, 2020
Context: dotnet#5253 (comment) When Javadoc-to-Xmldoc conversion was enabled, `make jenkins` took 1h 19min 47sec, up from 34min 45sec (ouch!). Disable Javadoc-to-Xmldoc on PR builds. Hopefully `make jenkins` times will return to normal.
Context: dotnet#5253 (comment) When Javadoc-to-Xmldoc conversion was enabled, `make jenkins` took 1h 19min 47sec, up from 34min 45sec (ouch!). Disable Javadoc-to-Xmldoc on PR builds. Hopefully `make jenkins` times will return to normal.
jonpryor
force-pushed
the
jonp-mono.android-xml-docs
branch
from
6ec9ac0
to
587805a
Compare
|
Fortunately, commit [ci] Don't convert Javadoc to Xmldoc for PR builds did what I hoped: As an alternative approach, @jpobst suggested that instead of trying to convert all the docs, we just do |
jonpryor
added a commit
to jonpryor/xamarin-android
that referenced
this pull request
Dec 17, 2020
Context: dotnet#5253 (comment) Context: dotnet/java-interop#687 (comment) Context: dotnet/java-interop@a65b8ab It looks like only emitting `<summary/>`, `<param/>`, `<returns/>`, and `<exception/>` is *much* faster than trying to do full `<remarks/>`. Update the `generator.exe` invocation to use `generator --doc-comment-style=summary`, which will hopefully speed things up.
jonpryor
added a commit
to jonpryor/xamarin-android
that referenced
this pull request
Dec 17, 2020
Context: dotnet#5253 (comment) Context: dotnet/java-interop#687 (comment) Context: dotnet/java-interop@a65b8ab It looks like only emitting `<summary/>`, `<param/>`, `<returns/>`, and `<exception/>` is *much* faster than trying to do full `<remarks/>`. Update the `generator.exe` invocation to use `generator --doc-comment-style=summary`, which will hopefully speed things up.
jonpryor
force-pushed
the
jonp-mono.android-xml-docs
branch
from
d5e9f73
to
4f2b301
Compare
Context: dotnet#5253 (comment) Context: dotnet/java-interop#687 (comment) Context: dotnet/java-interop@a65b8ab It looks like only emitting `<summary/>`, `<param/>`, `<returns/>`, and `<exception/>` is *much* faster than trying to do full `<remarks/>`. Update the `generator.exe` invocation to use `generator --doc-comment-style=summary`, which will hopefully speed things up.
jonpryor
force-pushed
the
jonp-mono.android-xml-docs
branch
from
4f2b301
to
724cfac
Compare
…ml-docs Let's get that openssl fix!
src/Xamarin.Android.Build.Tasks/Microsoft.Android.Sdk/Sdk/AutoImport.props
Outdated
Show resolved
Hide resolved
Bump to jonpryor/java.interop@32d469bb, which addresses some feedback on the Java.Interop side. Document the new `$(AndroidJavadocVerbosity)` property. Document the existing `@(JavaSourceJar)` build action. Append to `$(NoWarn)` instead of *replacing* it in `src/Mono.Android`. Avoid `**\*-source?.jar`, as that doesn't mean what I thought it meant. Instead, add/use `$(_DefaultJavaSourceJarPattern)`, which explicitly specifies both `*-source.jar` and `*-sources.jar`. Re-introduce the "don't build XML docs on PR builds" behavior.
…ml-docs Fix merge conflict in `Generator.cs` task.
jonathanpeppers
approved these changes
Dec 18, 2020
|
Squash-and-merge: Summary: Message: Fixes: https://github.com/xamarin/xamarin-android/issues/4789
Context: https://github.com/xamarin/java.interop/commit/7574f166008bb45c0df97315aae7907ac25f8602
Context: https://github.com/xamarin/xamarin-android/issues/5200
What do we want? Updated documentation! Better Bindings!
How do we get that? Uh…
Historically, [Xamarin.Android API docs][0] were produced by parsing
Android's HTML documentation from `docs-24_r01.zip` and converting it
into [**mdoc**(5) documentation][1] via `tools/javadoc2mdoc`.
The problem is that Google hasn't released an updated `docs*.zip`
package since API-24 (~6 years ago), and thus our documentation is
woefully out of date.
We could have scraped developer.android.com/reference within that
time frame, but web scraping is annoying, and we'd still have to deal
with the pain of parsing HTML.
There is an alternative, though: with API-30, there is a new `sources`
package which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments. Previous API
levels similarly contained an `android-stubs-src.jar` file which
likewise contained Java source code containing Javadoc comments.
The new approach is to:
1. Update `build-tools/xaprepare` to install the `sources` package.
2. Use [`java-source-tool.jar`][2] to parse the Java source code,
creating an `android-javadoc.xml` file.
3. [Update `generator`][3] to consume `android-javadoc.xml` and
convert the Javadocs into [C# XML documentation comments][4].
4. Update `src/Mono.Android` so that `generator` will emit
C# XML doc comments, and then produce a `Mono.Android.xml` file.
The result is a `Mono.Android.xml` file which contains imported
Android Javadoc documentation, e.g.
<doc>
<assembly><name>Mono.Android</name></assembly>
<members>
<member name="T:Java.Lang.Object">
<summary>Class <c>Object</c> is the root of the class hierarchy.</summary>
<remarks>
<para>Class <c>Object</c> is the root of the class hierarchy.
…
"Later", `Mono.Android.xml` can then be used alongside
[`mdoc update --import=Mono.Android.xml`][5] to update our published
API documentation.
Generation of `Mono.Android.xml` is disabled by default on CI PR
builds, as generating `Mono.Android.xml` increases `src/Mono.Android`
build times by an unacceptable amount.
With the inclusion of `java-source-tools.jar`, we can now fix the
TODO mentioned in commit 380e95e3, and re-add support for
`@(JavaSourceJar)` when running under JDK 11.
Update `Xamarin.Android.Bindings.Core.targets` so that before
invoking `generator`, we first run `java-source-utils.jar` on the
`@(JavaSourceJar)` files, producing Javadoc XML files.
The `<BindingsGenerator/>` task in turn is updated to accept a new
`BindingsGenerator.JavadocXml` property, which is converted into a
`generator --with-javadoc-xml=FILE` option.
The `@(JavaSourceJar)` item group is "extended" to support the
following item metadata:
* `%(CopyrightFile)`: A path to a file that contains copyright
information for the Javadoc contents, which will be appended to
all imported documentation.
* `%(UrlPrefix)`: A URL prefix to support linking to online
documentation within imported documentation.
* `%(UrlStyle)`: The "style" of URLs to generate when linking to
online documentation. Only one style is currently supported:
`developer.android.com/reference@2020-Nov`.
For .NET 6 ("One .NET") integration purposes, provide a default item
group of `@(JavaSourceJar)` which includes `**\*-source*.jar` and
`**\*-src.jar`. This will allow `dotnet build` of an "Android Java
Library Binding" project (`dotnet new android-bindinglib`) to
automatically process `.java` source code for documentation
translation purposes.
Add a new `$(AndroidJavadocVerbosity)` MSBuild property, which
controls "how much" of the Javadoc comments are converted into C#
XML documentation. Supported values are:
* `full`: Convert as much Javadoc as possible.
* `intellisense`: Only emit XML documentation for IDE-useful
constructs: summary, parameters, returns, exceptions.
The difference between the two is *build time* impact: `full` takes
longer, and thus may not always be desirable.
Finally, add a new `$(_UseLegacyJavadocImport)` MSBuild property
which *disables* use of `java-source-utils.jar` for Javadoc importing,
and instead use the previous, legacy, doesn't work on JDK 11,
approach of using `javadoc` and HTML parsing. This shouldn't be
needed, but just in case it *is* needed…
[0]: https://github.com/xamarin/android-api-docs
[1]: http://docs.go-mono.com/?link=man%3amdoc(5)
[2]: https://github.com/xamarin/java.interop/commit/69e1b80afd81ac502494e4bc2a2de75f5171c73f
[3]: https://github.com/xamarin/java.interop/pull/687
[4]: https://docs.microsoft.com/dotnet/csharp/codedoc
[5]: http://docs.go-mono.com/?link=man%3amdoc-update(1) |
Changes: dotnet/java-interop@2f62ffd...7574f16 * dotnet/java-interop@7574f166: [generator] Add `generator --with-javadoc-xml=FILE` support. (dotnet#687) * dotnet/java-interop@7d197f17: Refactor Crc64 type (dotnet#769) * dotnet/java-interop@876442f4: [generator] Fix MSBuild warning/error format for Visual Studio (dotnet#765) * dotnet/java-interop@3f6cf72b: [.Localization, .Cecil, .Diagnostics, .Generator] $(Nullable)=enable (dotnet#746) dotnet/java-interop#687 was merged.
Context: dotnet/java-interop@7d197f1 Context: dotnet#5452 dotnet/java-interop@7d197f17 introduced build breakage: …/xamarin-android/external/Java.Interop/src/Java.Interop.Tools.JavaCallableWrappers/Java.Interop.Tools.JavaCallableWrappers/Crc64.cs(58,16): error CS0117: 'Crc64Helper' does not contain a definition for 'HashCore' …/xamarin-android/external/Java.Interop/src/Java.Interop.Tools.TypeNameMappings/Java.Interop.Tools.TypeNameMappings/JavaNativeTypeManager.cs(648,27): error CS0117: 'Crc64Helper' does not contain a definition for 'Compute' PR dotnet#5452 has the fix. Include the build fix so that PR dotnet#5253 can build.
dellis1972
reviewed
Jan 5, 2021
| } | ||
|
|
||
| // Arguments sent to java.exe | ||
| cmd.AppendSwitchIfNotNull ("-jar ", Path.Combine (MonoAndroidToolsDirectory, "java-source-utils.jar")); |
There was a problem hiding this comment.
Do we want to hard code this java-source-utils.jar ? Or have it as a property like we do for ToolName in other tasks?
There was a problem hiding this comment.
We should follow the other tasks.
Context: dotnet/java-interop#767 Context: dotnet#5253 (comment) dotnet/java-interop#767 suggests updating `generator` to use Javadoc output to determine parameter names. However, we don't need to do that, as `class-parse` *already* has (partial) support for that. Split the `<JavaSourceUtils/>` invocation out into a new `_ExtractJavadocsFromJavaSourceJars` target invocation, and give it a "proper" output item group. This allows it to be executed in an incremental manner, which wasn't previously the case. Additionally, the previous `<JavaSourceUtils/>` invocation "minimized" the number of Javadoc XML output files based on the "unique" values of `%(CopyrightFile)`/etc. Thus, if you had multiple `@(JavaSourceJar)`s with the same (or no) `%(CopyrightFile)` file, they'd all be present in the same output XML. While this was "nice" in that it reduced the number of files running around, it complicated coming up with a separate item group for incremental build purposes. Remove the "nicety" and go for "simplicity": one Javadoc XML per `@(JavaSourceJar)` file. Period. Add a `$(AndroidJavaSourceUtilsJar)` MSBuild property which controls where `java-source-utils.jar` is present. This is consistent with other tasks. However, *don't* allow `class-parse` to use `java-source-utils` output, as that doesn't actually work yet. Doh.
dellis1972
approved these changes
Jan 6, 2021
jonathanpeppers
approved these changes
Jan 6, 2021
|
Great work here! 👍 Is this change gonna be released next year with .NET 6 or with the next XA release? |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes: #4789
Context: dotnet/java-interop@7574f16
Context: #5200
What do we want? Updated documentation! Better Bindings!
How do we get that? Uh…
Historically, Xamarin.Android API docs were produced by parsing
Android's HTML documentation from
docs-24_r01.zipand converting itinto mdoc(5) documentation via
tools/javadoc2mdoc.The problem is that Google hasn't released an updated
docs*.zippackage since API-24 (~6 years ago), and thus our documentation is
woefully out of date.
We could have scraped developer.android.com/reference within that
time frame, but web scraping is annoying, and we'd still have to deal
with the pain of parsing HTML.
There is an alternative, though: with API-30, there is a new
sourcespackage which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments. Previous API
levels similarly contained an
android-stubs-src.jarfile whichlikewise contained Java source code containing Javadoc comments.
The new approach is to:
Update
build-tools/xaprepareto install thesourcespackage.Use
java-source-tool.jarto parse the Java source code,creating an
android-javadoc.xmlfile.Update
generatorto consumeandroid-javadoc.xmlandconvert the Javadocs into C# XML documentation comments.
Update
src/Mono.Androidso thatgeneratorwill emitC# XML doc comments, and then produce a
Mono.Android.xmlfile.The result is a
Mono.Android.xmlfile which contains importedAndroid Javadoc documentation, e.g.
"Later",
Mono.Android.xmlcan then be used alongsidemdoc update --import=Mono.Android.xmlto update our publishedAPI documentation.
Generation of
Mono.Android.xmlis disabled by default on CI PRbuilds, as generating
Mono.Android.xmlincreasessrc/Mono.Androidbuild times by an unacceptable amount.
With the inclusion of
java-source-tools.jar, we can now fix theTODO mentioned in commit 380e95e, and re-add support for
@(JavaSourceJar)when running under JDK 11.Update
Xamarin.Android.Bindings.Core.targetsso that beforeinvoking
generator, we first runjava-source-utils.jaron the@(JavaSourceJar)files, producing Javadoc XML files.The
<BindingsGenerator/>task in turn is updated to accept a newBindingsGenerator.JavadocXmlproperty, which is converted into agenerator --with-javadoc-xml=FILEoption.The
@(JavaSourceJar)item group is "extended" to support thefollowing item metadata:
%(CopyrightFile): A path to a file that contains copyrightinformation for the Javadoc contents, which will be appended to
all imported documentation.
%(UrlPrefix): A URL prefix to support linking to onlinedocumentation within imported documentation.
%(UrlStyle): The "style" of URLs to generate when linking toonline documentation. Only one style is currently supported:
developer.android.com/reference@2020-Nov.For .NET 6 ("One .NET") integration purposes, provide a default item
group of
@(JavaSourceJar)which includes**\*-source*.jarand**\*-src.jar. This will allowdotnet buildof an "Android JavaLibrary Binding" project (
dotnet new android-bindinglib) toautomatically process
.javasource code for documentationtranslation purposes.
Add a new
$(AndroidJavadocVerbosity)MSBuild property, whichcontrols "how much" of the Javadoc comments are converted into C#
XML documentation. Supported values are:
full: Convert as much Javadoc as possible.intellisense: Only emit XML documentation for IDE-usefulconstructs: summary, parameters, returns, exceptions.
The difference between the two is build time impact:
fulltakeslonger, and thus may not always be desirable.
Finally, add a new
$(_UseLegacyJavadocImport)MSBuild propertywhich disables use of
java-source-utils.jarfor Javadoc importing,and instead use the previous, legacy, doesn't work on JDK 11,
approach of using
javadocand HTML parsing. This shouldn't beneeded, but just in case it is needed…