Enable compilation support in System.CodeDom

[stephentoub]

System.CodeDom in corefx provides the object model for describing code as well as code generators for both C# and VB. But attempts to compile source code (which in desktop works by shelling out to csc.exe/vbc.exe) isn't enabled and throws PlatformNotSupportedException. This should be fairly straightforward to build on top of Roslyn's APIs; I expect the main tricky part will be around whether we can take a dependency from this library to Roslyn. If we can't, we could explore indirect dependencies, e.g. via reflection.

[Priya91]

Most of the funtionality in this library is provided through Roslyn, and roslyn is constantly evolving in that space. We don't foresee innovating in codedom, and don't think it's worth the investment here, to light these up using roslyn..

[weshaggard]

I would be very reluctant to take the dependency directly in the System.CodeDom library itself because there is a decent chance that we may be required to pull that library into the shared framework because of other .NET Framework API compat issues and it would be a shame to pull in all of Roslyn because of that.

My suggestion would be to try and build a library on top of System.CodeDom that hooks up Roslyn and CodeDom together so we can always keep that optional. I also want to try and avoid corefx having a dependency on Roslyn packages as that will cause more build cycles, so this library would need to be outside of corefx.

We can also consider a reflection style approach but I suspect that maybe difficult because it isn't going to be one single API we need to call via reflection.

[terrajobst]

I like @stephentoub's suggestion and suggested something similar to @weshaggard.

@Priya91

We don't foresee innovating in codedom, and don't think it's worth the investment here, to light these up using roslyn.

It's not about innovating, it's about how much of the existing code we can make work as-is. Asking folks to change their code to compile with Roslyn isn't straight forward. Asking folks to install the package because we do the work via reflection is reasonable. We could even be more helpful by making sure we detect that case and throw an actionable error message.

[drvink]

@terrajobst

It's not about innovating, it's about how much of the existing code we can make work as-is.

I have a user in fsprojects/FSharp.Compiler.CodeDom#24 who is likely facing the problem that has always presented the greatest difficulty for people at the intersection of CodeDom and F#[1], which is the issue of having to register the provider with machine.config or web.config in order for it to be of any use at all--if all they want is to assemble an AST and get F# source from it, there are better tools than the F# CodeDom provider[2], and if they want a way of programmatically compiling F# source, there are also better ways to do it than via CodeDom.

Anyway, it looks like it's possible to register third-party providers with ASP.NET for .NET Core (see the web.config transform in Microsoft.CodeDom.Providers.DotNetCompilerPlatform), but it looks like machine.config doesn't even exist at all anymore in .NET Core. This means that the only way the F# provider can be used is by directly referencing the assembly, which is completely useless for users of the provider.

TL;DR: Is there any way to make the provider work "system-wide", as CodeDom used to allow via machine.config, or will the only useful purpose for third-party CodeDom providers under .NET Core be solely for use via ASP.NET? (edit: apparently ASP.NET Core doesn't use CodeDom)

[1] Or really, any language other than C# or VB.NET--even Microsoft has been guilty in a number of places of hard-coding the CodeDom "choices" to only these two, despite allowing for system-wide registration of new providers and for programs to trivially enable users to ask for any registered provider... :)
[2] As the joke goes, CodeDom is an API lets you write C# from any language.

[davidfowl]

ASP.NET Core doesn’t use code Dom, on top of that razor is c# only in asp.net core.

[drvink]

@davidfowl

What is the purpose of web.config for a NuGet-ized CodeDom provider package, then? (Sorry, I don't do web stuff at all; I simply assumed that it was related to ASP.NET)

[davidfowl]

It’s for ASP.NET (not core)

[drvink]

@davidfowl So this is entirely about providing NuGet packages for .NET, and .NET Core/Standard are completely unrelated?

[davidfowl]

Correct, if the CodeDom implementation for C# and VB are stubbed out using roslyn APIs then it will work for things that do use codedom (albeit with a large first time performance hit because using roslyn without ngen is awful). It just so happens that ASP.NET Core does not use the codedom for anything so there's no benefit there.

The API should be ported though, some other systems do rely on it.

[drvink]

@davidfowl So, if I were to provide a package similar to Microsoft.CodeDom.Providers.DotNetCompilerPlatform for the F# provider, it would help people who wanted to use NuGet/Paket/etc., but otherwise, it would have nothing to do with .NET Core?

[drvink]

@davidfowl I simply am 100% ignorant of this segment of the .NET world, so it would help if you can explain the situation here--if the F# provider is only going to be useful from this point on in the context of a compatibility shim for "old .NET", that's not a problem for me, but I need to alert people who have depended on it up until now that it won't work with .NET Core.

[davidfowl]

Ok let me summarize. There are 2 different concerns being discussed here, .NET Core and ASP.NET Core, they are not the same thing.

• CodeDom is made up of 2 pieces
  1. An object model (AST) for generating code in a language agnostic way (which is dubious in itself)
  2. A way to compile that code that produce and assembly
• Today on .NET Core System.CodeDom exists, it has code support for generating code but not for compiling code. That's what this issue is about. The old code dom providers in .NET Framework would shell out to vbc and csc that was installed with the framework folder. There's also a new codedom package for .NET Framework that uses roslyn to compile https://www.nuget.org/packages/Microsoft.CodeDom.Providers.DotNetCompilerPlatform/
• ASP.NET on .NET Framework used CodeDom in several ways:
  • For aspx pages and code generation for webforms controls
  • For compiling aspx and Razor pages (via the BuildManager)
  • For compiling App_Code and other App_* folders
  • In this world, adding more code dom providers would add more language support to various parts of ASP.NET
• ASP.NET Core does not use CodeDom in any way shape or form.

That said, having an F# provider that works on .NET Framework and .NET Core (.NET Standard really) is fine for things that are built on codedom.

Hope that helps

[drvink]

@davidfowl OK, thanks, that clarifies the ASP.NET/ASP.NET Core stuff for me. The question I now have is how to register CodeDom providers (i.e. the way one used to do so via machine.config) on .NET Core.

[trvsysadmin]

@davidfowl Thanks for the detailed explanation. Is there any update on this topic? We have a use case where we compile the VB code inside of excel files inside an app.

After some hours of researching, any clarification with these questions would be really appreciated:

• there is no way to compile VB code using the codedom providers in Core at the moment, correct?
• is it possible to "shell out" from Core by separately compiling the vb code as a project in another folder? Almost a bit like an RPC call to a VB server.
• can we use the Roslyn compiler (via nugetting Microsoft.CodeDom.Providers.DotNetCompilerPlatform) inside Core to achieve the same result? Asking this because @stephentoub said This should be fairly straightforward to build on top of Roslyn's API's and not sure if this was ever done.
• Can we use this: https://www.nuget.org/packages/CoreCompat.System.CodeDOM/ ?

We are using Core 2.1.400 .

Thank You!

[danmoseley]

Correct, it should be possible but it has not been done. It would be interesting if someone wanted to attempt a PR.

[trvsysadmin]

@danmosemsft Thank you.

Would it still be possible to use Roslyn like this in the meantime?

using System.Runtime.Loader;
using Microsoft.CodeAnalysis;
using Microsoft.CodeAnalysis.Emit;
using Microsoft.CodeAnalysis.VisualBasic;
using Microsoft.VisualBasic;

MetadataReference[] references = new MetadataReference[]
{
	MetadataReference.CreateFromFile(typeof(object).GetTypeInfo().Assembly.Location)
};

SyntaxTree syntaxTree = VisualBasicSyntaxTree.ParseText(VBSOURCECODESTRING);
VisualBasicCompilation compilation = VisualBasicCompilation.Create(
	"excelVbAssembly",
	syntaxTrees: new[] {syntaxTree},
	references: references,
	options: new VisualBasicCompilationOptions(OutputKind.DynamicallyLinkedLibrary));

MemoryStream ms = new MemoryStream();		
EmitResult result = compilation.Emit(ms);
ms.Seek(0, SeekOrigin.Begin);
Assembly assembly = AssemblyLoadContext.Default.LoadFromStream(ms);

I get this error from the compilation.Emit(ms); statement:

BC35000: Requested operation is not available because the runtime library function 'Microsoft.VisualBasic.CompilerServices.StandardModuleAttribute..ctor' is not defined.

[trvsysadmin]

I managed to fix my issue as it was just the usual problem of MetadataReferences. In the end, I hardcoded all my dlls. This is a well known topic and there is more information here:

https://github.com/dotnet/roslyn/wiki/Runtime-code-generation-using-Roslyn-compilations-in-.NET-Core-App

https://stackoverflow.com/questions/39257074/net-core-amd-roslyn-csharpcompilation-the-type-object-is-defined-in-an-assem

MetadataReference[] references = new MetadataReference[]
{
MetadataReference.CreateFromFile("/home/user/project/packages/microsoft.visualbasic/10.3.0/lib/netstandard2.0/Microsoft.VisualBasic.dll"),
// Where you have your system.runtime.dll etc.
};

[87Alex]

Hello,
Is there any news? We also use the CSharpCodeProvider class as follows:

public Assembly Compile(String sourceCode, String[] references, String assemblyFile) {
            CompilerParameters compilerParameters = new CompilerParameters();
            compilerParameters.ReferencedAssemblies.AddRange(references);
            compilerParameters.OutputAssembly = assemblyFile;
            CodeDomProvider codeProvider = new CSharpCodeProvider();
            CompilerResults compilerResults = codeProvider.CompileAssemblyFromSource(compilerParameters, new String[] { sourceCode });
            return compilerResults.CompiledAssembly;
        }

[Yamboy1]

Hi, Is there any movement on this?

https://stackoverflow.com/questions/63749712/dynamically-compile-multiple-files-into-assembly-using-rosyln

[w3lld0ne]

up.

[hemantsathe]

We recently migrated a .Net Framework app to .Net 5. We have observed that some of the VB code compiles differently in .Net 5. Specially when the data type is object. In .Net Framework the objects were getting cast properly from Object. However, in .Net 5 it becomes much complex. Dates get cast as integer and the functions for dates start throwing errors. This same code used to work in .Net Framework. I have logged detailed error in Roslyn project. In such case, I would prefer to start recompiling the code using CodeDom like we were doing before.

[dmanter]

Does it make sense to anyone else MS gives us a tool like CodeDOM and then takes it away. Our app uses codedom extensively for user defined fields. This decision to not include the compiler seems very unfair. I'd understand if there was a solution that was compatible to Framework but seems like that will not happen. Very disappointed in MS!

[jaredpar]

@dmanter

I coded up a hobby project that leverages Roslyn to provide CodeDOM support for .NET Core based applications.

https://github.com/jaredpar/roslyn-codedom

A number of customers that were using CodeDom on .NET Framework were able to migrate to this as they adopted .NET Core. The implementation doesn't cover the full breadth of CodeDom scenarios but it hit the most common ones I was aware of. Likely any missing ones could be added with not too much difficulty.

It may be a path forward for you.

[dmanter]

I like @stephentoub's suggestion and suggested something similar to @weshaggard.

@Priya91

We don't foresee innovating in codedom, and don't think it's worth the investment here, to light these up using roslyn.

It's not about innovating, it's about how much of the existing code we can make work as-is. Asking folks to change their code to compile with Roslyn isn't straight forward. Asking folks to install the package because we do the work via reflection is reasonable. We could even be more helpful by making sure we detect that case and throw an actionable error message.

I agree whole heartily with this! It shouldn't be innovation for something that was included in framework, the skeleton seems to still be offered System.Codedom but then to remove the actual engine csc.exe so it doesn't work and report an error is a confusing strategy to me. Then to tell me I have to use something I've never heard of before, Roslyn and comments about its difficulty and performance (slow) also is very confusing. Isn't this a step backwards and not forwards as is the whole idea of CORE is supposed to be. Faster, better, etc. If I had been told in the error message or compiler error I was missing a Nuget package and just install that and what worked in Framework now works in Core seems to be a proper way to better and move forward. I don't really care what the engine is underneath. I never even knew that csc.exe was being used as it just worked. I didn't need to see under the hood. So the "we do the work" statement and a solution that just works like it did in Framework, is my vote! BTW-Now I know about csc.exe, can it just be used again by copying the exe from a framework folder to a .Net 7 Core folder? Seems like and easy solution.

[stephentoub]

We've lived without this now for 8 years and external solutions exist, like the one linked above from Jared. I don't see us baking the reliance on Roslyn into the core library at this point, so I'm going to close the issue.