Convert C# Reference Assembly Generation to Use GenAPI

[rladuca]

Fixes #932

What work is currently in this PR:

GitHub

• For all shipping C# reference assemblies built on GitHub generate their code from GenAPI
• Fixup any issues that result from GenAPI code gen (see ManualFixups.txt in this PR)
• Baseline all ApiCompat issues (there is a feedback loop here with fixups for a few commits)

dotnet-wpf-int

• Switch ProjectReference substitution for dotnet-wpf-int to use runtime assemblies
• For all shipping C# reference assemblies built on dotnet-wpf-int generate their code from GenAPI
• Fixup any issues that result from GenAPI code gen (see ManualFixups.txt in this PR)
• Baseline all ApiCompat issues (there is a feedback loop here with fixups for a few commits)

Testing Work:

• Build our internal test corpus against new ref assemblies
• Build our samples repo against new ref assemblies
• Build NuGet Package Explorer against new ref assemblies
• Verify there are no dangling references to assemblies not part of the reference pack

What you should center your review around

• Engineering system changes (in WpfArcadeSdk, ignore Arcade DARC flow)
• ApiCompat results and baselines
• Exclusion lists for GenApi (see GlobalApiExclusions.txt and GlobalAttrExclusions.txt)

These are what eventually feed into the reference assemblies and dictate the public surface area.

Links

dotnet-wpf-int PR: https://dev.azure.com/dnceng/internal/_git/dotnet-wpf-int/pullrequest/1772?_a=overview

Upstream notification issue: https://github.com/dotnet/core-setup/issues/6901

[rladuca]

.NET Framework reference assemblies contain resources (BAML/strings resx, etc). Should we also include these in .NET Core?

[stevenbrix]

In your description you mention a "ManualFixups.txt" file in the PR, but i don't see that anywhere. was that removed or changed to a different file name?

[stevenbrix]

is all of this extra work needed because we use the InternalsVisibleToAttribute?

[rladuca]

It's a combination of a few things, InternalsVisibleTo certainly complicates things between assemblies, but also XAML compilation will reflect and try to find internal API in order to take certain shortcuts and optimizations. (Look at IAddChildInternal for example).

There are also issues where we have public API that has internal abstract members, which confuses GenAPI. GenAPI itself just sometimes adds strange things, like private dummy int fields and properties. No idea why, there are just bugs. I intend to make small reproduction applications when I have time and file bugs against GenAPI, but that's future stuff.

[weltkante]

Are those bugs reported back to the Roslyn team so they can be fixed?

[rladuca]

GenAPI doesn't use Roslyn it uses Microsoft.CCI). Originally, we were using Roslyn to generate our reference assemblies, but the use of InternalsVisibleTo was creating dangling references. We filed dotnet/roslyn#36409 to address these issues. In the meantime, this was the fix.

I haven't created issues against GenAPI yet, I haven't had the time to create minimal reproductions of the various bugs.

[stevenbrix]

If i'm adding a new function to a class, what's the difference between adding the API to the reference assembly manually and using GenAPI? I might not fully understand the problem we are trying to solve here, but it seems like it would be much simpler to just manually modify the reference assembly. Is there some part of our workflow that makes things more complicated?

[rladuca]

I missed this. The C# reference assemblies didn't exist in a manual fashion before this (back in the old internal repo maybe), so we could not just add things. So GenAPI was used as an initial seed for the most part. For some changes, it's probably easy enough just to pop the change into the reference assembly. Other changes it might be more difficult and require basically mirrored code changes. Also if you make a small change sans GenAPI then someone makes a very large change and uses it, it'll likely generate a noisier diff since your code location might not be the same.

It's probably the simplest and most thorough to GenAPI and use that output to manually add to the reference assembly. It's pretty easy (mostly) to diff the generated output against the current surface and just pick and choose your new stuff.

Long term we'd rather just have Roslyn allow us to say, "All publics, but also these specific internals" in a nice configurable fashion and call it a day. But we need something in the short term.

[rladuca]

In your description you mention a "ManualFixups.txt" file in the PR, but i don't see that anywhere. was that removed or changed to a different file name?

I removed it from the docs and the repo. Basically, it would confuse more than help in light of some internals needing to be kept for XAML compilation.

I decided it's just best to tell developers to selectively add your change (as generated by GenAPI) and ignore the other fluff. We can work on improving the output from GenAPI in terms of the cleanup, but it's not there just yet.

[vatsan-madhavan]

.NET Framework reference assemblies contain resources (BAML/strings resx, etc). Should we also include these in .NET Core?

No, let's not include these.

[stevenbrix]

It doesn't sound like a good idea to have internal/private APIs required by the XAML compiler. I understand that IAddChildInternal exists today, but is this something we want to put in as being an ok thing to do in the future?

[rladuca]

This is more if you make a change to an existing class that uses something like that, the entire class will get re-generated, sans the IAddChildInternal. So you need to ensure that you're not messing with that piece.

[rladuca]

We can have a discussion about fixing the docs up later, at this point I don't want to spin another build just due to doc updates. Helix queues are killing me today.