Cross-Platform Support

[lordmilko]

ClrDebug is currently primarily targeted towards Windows users. As the COM marshalling infrastructure is not included in cross-platform builds of .NET Core, or programs compiled with NativeAOT, it is expected that users will use .NET's ComWrappers library for generating and managing COM marshalling interfaces themselves - an overlay onerous and unrealistic.

With the new COM source generators expected to ship in .NET 8, it will become possible to support both cross-platform and NativeAOT scenarios. This issue will list the tracking items that will need to be completed in order to support this.

While the quirks of the new source generator are not very well documented yet, we can see that many rules are the same as those found in LibraryImportGenerator

Prep

• Allow optional multi targeting netstandard2.0 and net8.0
• Specify explicit marshalling on bool types
• Marshal objects as UnmanagedType.Interface instead of UnmanagedType.IUnknown
• Convert StringBuilder / UnmanagedType.LPWStr to char[] / LPArray + U2
• Convert StringBuilder / UnmanagedType.LPStr to char[] / LPArray + U1

Multi Targeting

• Support passing Guid values by reference
• Conditionally include new overrides in derived interfaces
• Conditionally specify ComImport / GeneratedComInterface on interfaces
• Conditionally specify ComImport / GeneratedComClass on CoClasses
• Conditionally specify GeneratedComClass on callbacks/classes that implement COM interfaces
• Conditionally use LibraryImportAttribute instead of DllImportAttribute
• Update DbgShim to conditionally use NativeLibrary in .NET 8.0 builds, with different XmlDocs to state it does work cross-platform
• Stub out InAttribute / OutAttribute
• Automatically build netstandard2.0 / net8.0 except when building in VS2017
• Update NetCore sample to have separate launch targets for netcoreapp2.1 / net8.0
• Update NetCore README to clarify that the .NET 8.0 build does work cross-platform
• Only generate structs for known-bad structs + don't regenerate structs if input files haven't been modified (perhaps AddSource a text file listing the last modified time of each input file?)

Blockers

Resolved	Description	Issue	Workaround
TRUE	Cannot marshal structs containing bool	dotnet/runtime#88956	Custom source generator
TRUE	Cannot marshal structs containing string	dotnet/runtime#88284	Custom source generator
TRUE	Cannot marshal structs containing interfaces	dotnet/runtime#88284	Custom source generator
TRUE	Cannot marshal object for VARIANT	dotnet/runtime#66674	Custom source generator?
TRUE	Possibly cannot marshal enumerator results	dotnet/runtime#88253	N/A

Open Questions

• Does QI on types derived types decorated with [GeneratedComClass] (such as CorProfilerCallback) work automatically, or do derived types need [GeneratedComClass] too?
• Remove discussion around rd.xml in NetCore sample README if it isn't actually needed after all
• Will our current DbgEng implementation work with NativeAOT? (specifically all the delegates used). Do we need a "normal" wrapper implementation with custom IUnknown in .NET 8.0? Custom QueryInterface was mention in the NativeAOT QueryInterface source code
• Does the CLR translate a null VARIANT to VT_EMPTY?
• How to get function pointer for delegate that takes an interface as a parameter in a cross-platform way. We can use an ICustomMarshaller but ICustomMarshaller apparently isn't supported. There are plans for some other custom marshalling system to be supported however
• Can we use a custom string marshaller in both .NET Framework and .NET 8 to allow emitting out string instead of char[] or byte[]
• Can we make the NetCore sample use the right DbgShim automatically from the DbgShim NuGet package without having to publish a specific runtime?
• How to call SetEvent / WaitForSingleObject in a cross-platform way in NetCore sample?

Pre-Release

• Regenerate ComWrappers
• .NET 8.0 released
• Add profiler NativeAOT sample
• If DbgEng is compatible with NativeAOT, add NativeAOT DbgEng extension sample
• Update README to say ClrDebug is cross-platform and NativeAOT compatible + how to do NativeAOT

Final Tests (.NET Framework / .NET 8.0)

• rd.xml works from props in nupkg for net8.0 NativeAOT
• Marshalling
  • Bool
    • .NET Framework
    • .NET 8
  • IUnknown (object)
    • .NET Framework
    • .NET 8
  • Interface
    • .NET Framework
    • .NET 8
  • Array
    • .NET Framework
    • .NET 8
  • Struct
    • .NET Framework
    • .NET 8
  • Guid
    • .NET Framework
    • .NET 8
  • LPStr
    • .NET Framework
    • .NET 8
  • LPWStr
    • .NET Framework
    • .NET 8
  • BStr
    • .NET Framework
    • .NET 8
  • VARIANT
    • .NET Framework
    • .NET 8
  • CoClass
    • .NET Framework
    • .NET 8
• Get delegate for function pointer that has an interface parameter (Note: you can't do this. Delegates must emit raw pointers)