Scripting API: metadata references not backed by a file don't load

[tmat]

Scenario:

var s = await CSharpScript.EvaluateAsync("new MyLib.Class()", ScriptOptions.Default.AddReferences(
   MetadataReference.CreateFromImage(File.ReadAllBytes(@"file.dll"))))

fails since the in-memory assembly is not registered with InteractiveAssemblyLoader.

[ManishJayaswal]

@tmat moving to 1.2

[tmat]

Also affects usability of scripting API from within the REPL:

> public class Globals
. {
.     public int X;
.     public int Y;
. }
> var globals = new Globals { X = 1, Y = 2 }; 
> await CSharpScript.EvaluateAsync<int>("X+Y", globals: globals) 
Microsoft.CodeAnalysis.Scripting.CompilationErrorException: (1,3): error CS7012: The name 'Y' does not exist in the current context 

[hawkerm]

Any update on when this may be fixed?

[tmat]

@hawkerm We do not have any specific date. We review and re-prioritize open issues regularly for each release. Unfortunately, we have had other work that was deemed more important than this one.

Are you blocked or can you find a workaround for your scenario?

[hawkerm]

@tmat I'm testing out using Roslyn in a UWP application, but I think with all the security restrictions still it's not going to work out. That's why I was hoping this approach with the Globals would work, as then I could stick in my own object for externalization within the script code being run dynamically.

I think I'm going to have to use something like DynamicExpresso which is purely interpretive and doesn't need to write out a dll, but they don't have full language support yet.

[tmat]

@hawkerm Scripting is not gonna work at all in UWP as UWP doesn't support runtime code generation. Scripting is only supported on .NET Framework and .NET Core.

[jchable]

Same error but for a more global issue for developers working with Entity Framework (.NET Fx) and Roslyn. I got an object in a database I retrieves with EF, and I want to evaluate a condition based on the properties of the object in globals.

error CS7012: The name 'FirstName' does not exist in the current context (are you missing a reference to assembly 'EntityFrameworkDynamicProxies-Kelios.CAM.Engagement.Core, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'?)

The same code works well in my unit tests because I use the business models but it's not working with proxies generated by EF. Is there a way to do it without disabling EF proxy and make my code a nightmare to manage ?

[tmat]

@jchable Roslyn doesn't support compiling directly against assemblies generated by Reflection.Emit. I'm not sure if that's how EF produces the dynamic proxy assembly, but my guess would be so.

Could you share a snippet of code that that demonstrates the issue? Perhaps you could use "dynamic" to access the properties of the object.

[StevenRasmussen]

This is a bummer. I was really excited about using Roslyn as a scripting engine but this limits the use case drastically as I understand it. To only be able to use a predefined class to pass in variables to a script would mean that we would have to have some sort of understanding about what variables are required by the script itself. The whole idea around a scripting engine would be to have the ability to pass in dynamic variables at runtime. I've come up with a pretty ugly solution but I feel this should be handled OOB for sure.

[Diaskhan]

For fleunt implementation it must be dynamic object with variables or just wtih simple properpties

globals =new Object(){
x=null,
y=null};
await CSharpScript.EvaluateAsync("X+Y", globals: globals)

PHP is more fleunt is this case !
Roslyn lose to php in this use case !
to saaaaaddd

[pwhe23]

I can't believe the dynamic type isn't even supported :(

dynamic globals = new ExpandoObject();
globals.X = 1;
globals.Y = 2;
Console.WriteLine(await CSharpScript.EvaluateAsync<int>("X+Y", globals: globals));
// (1,1): error CS0103: The name 'X' does not exist in the current context

Anonymous types aren't supported either

var globals = new { X = 1, Y = 2 };
Console.WriteLine(await CSharpScript.EvaluateAsync<int>("X+Y", globals: globals));
//(1,1): error CS0122: '<>f__AnonymousType0<int, int>.X' is inaccessible due to its protection level

Has anyone found any kind of work-around to being required to use static predefined types to pass data to these dynamic scripts?

[Diaskhan]

the other solution is to use other experssion libraries.
David has provided some list of it !
https://github.com/davideicardi/DynamicExpresso#other-resources-or-similar-projects

[DeluxeAgency2020]

I can't believe the dynamic type isn't even supported :(

dynamic globals = new ExpandoObject();
globals.X = 1;
globals.Y = 2;
Console.WriteLine(await CSharpScript.EvaluateAsync<int>("X+Y", globals: globals));
// (1,1): error CS0103: The name 'X' does not exist in the current context

Anonymous types aren't supported either

var globals = new { X = 1, Y = 2 };
Console.WriteLine(await CSharpScript.EvaluateAsync<int>("X+Y", globals: globals));
//(1,1): error CS0122: '<>f__AnonymousType0<int, int>.X' is inaccessible due to its protection level

Has anyone found any kind of work-around to being required to use static predefined types to pass data to these dynamic scripts?

Is any progress on supporting ExpandoObject or Anonymous?

[Skyppid]

Kinda sad that this issue hasn't been touched in 7 years with no visible change on that matter to come. That's the first usage scenario that comes to my mind when using scripts: Passing in a globals object, that is not necessarily inside a physical assembly. Takes a lot of the "dynamic" out of scripting.

[MattMinke]

Related:
#3194
#2246

[MattMinke]

I found a solution to be able to use ExpandoObject and dynamic. It needs some cleanup before it will be production ready, but shows how to accomplish what is being asked.

using Microsoft.CodeAnalysis.CSharp.Scripting;
using Microsoft.CodeAnalysis.CSharp;
using Microsoft.CodeAnalysis.Scripting.Hosting;
using Microsoft.CodeAnalysis;
using System.Collections.Immutable;
using System.Reflection;
using Microsoft.Extensions.DependencyModel;
using System.Dynamic;

namespace MySample
{
    internal class Program
    {
        static async Task Main(string[] args)
        {
            try
            {
                await new Example().Execute();
            }
            catch (Exception ex)
            {
                throw;
            }
            
        }
    }
}



namespace MySample
{
    /// <summary>
    /// Using ReflectionEmit with Microsoft.CodeAnalysis.CSharp.Scripting is not supported. 
    /// The workaround is to use CSharpCompilation instead.
    /// ref: https://github.com/dotnet/roslyn/issues/2246
    /// ref: https://github.com/dotnet/roslyn/pull/6254
    /// </summary>
    public class CSharpCompilationScriptGlobalTypeBuilder
    {

        private const string TEMPLATE = @"
using System;
using System.Collections.Generic;
public class {0}
{{
    public {0}(
        IDictionary<string, Object> extensions)
    {{
        {1}
    }}
    {2}
}}";

        private int unique = 0;
        private readonly IDictionary<Guid, GlobalTypeInfo> _cache;

        public CSharpCompilationScriptGlobalTypeBuilder()
        {
            _cache = new Dictionary<Guid, GlobalTypeInfo>();
        }

        private static PortableExecutableReference GetMetadataReference(Type type)
        {
            var assemblyLocation = type.Assembly.Location;
            return MetadataReference.CreateFromFile(assemblyLocation);
        }

        public GlobalTypeInfo Create(Guid key, IDictionary<string, object> extensions)
        {
            // No locking. the worst that happens is we generate the type
            // multiple times and throw all but one away. 
            if (!_cache.TryGetValue(key, out var item))
            {
                item = CreateCore(key, extensions.ToDictionary(x => x.Key, x => x.Value.GetType()));
                _cache[key] = item;
            }
            return item;
        }

        private GlobalTypeInfo CreateCore(Guid key, IDictionary<string, Type> extensionDetails)
        {
            var count = Interlocked.Increment(ref unique);
            var typeName = $"DynamicType{count}";

            var code = String.Format(TEMPLATE,
                typeName,
                String.Join(System.Environment.NewLine, extensionDetails.Select(pair => $"{pair.Key} = ({pair.Value.FullName})extensions[\"{pair.Key}\"];")),
                String.Join(System.Environment.NewLine, extensionDetails.Select(pair => $"public {pair.Value.FullName} {pair.Key} {{ get; }}")));


            SyntaxTree syntaxTree = CSharpSyntaxTree.ParseText(code);

            //TODO: need to change the way references are being added.
            // See these links for how the preferred method:
            // ref: https://github.com/dotnet/roslyn/issues/49498
            // ref: https://github.com/dotnet/roslyn/issues/49498#issuecomment-776059232
            // ref: https://github.com/jaredpar/basic-reference-assemblies
            // ref: https://stackoverflow.com/q/32961592/2076531
            PortableExecutableReference[] references = Assembly.GetEntryAssembly().GetReferencedAssemblies()
                .Select(a => MetadataReference.CreateFromFile(Assembly.Load(a).Location))
                .Concat(extensionDetails.Values.Select(GetMetadataReference))
                .Append(GetRuntimeSpecificReference())
                .Append(GetMetadataReference(typeof(CSharpCompilationScriptGlobalTypeBuilder)))
                .Append(GetMetadataReference(typeof(System.Linq.Enumerable)))
                .Append(GetMetadataReference(typeof(System.Runtime.AssemblyTargetedPatchBandAttribute)))
                .Append(GetMetadataReference(typeof(Microsoft.CSharp.RuntimeBinder.CSharpArgumentInfo)))
                .Append(GetMetadataReference(typeof(System.Collections.Generic.IDictionary<string, object>)))
                .Append(GetMetadataReference(typeof(object)))
                .Append(GetMetadataReference(typeof(GlobalTypeInfo)))
                .ToArray();



            Compilation compilation = CSharpCompilation.Create(
                $"ScriptGlobalTypeBuilder{count}", new[] { syntaxTree }, references,
                new CSharpCompilationOptions(OutputKind.DynamicallyLinkedLibrary));

            ImmutableArray<byte> assemblyBytes = compilation.EmitToArray();
            PortableExecutableReference libRef = MetadataReference.CreateFromImage(assemblyBytes);
            Assembly assembly = Assembly.Load(assemblyBytes.ToArray());

            return new GlobalTypeInfo()
            {
                Key = key,
                Assembly = assembly,
                Reference = libRef,
                Type = assembly.GetType(typeName),
            };
        }


        private static PortableExecutableReference GetRuntimeSpecificReference()
        {
            var assemblyLocation = typeof(object).Assembly.Location;
            var runtimeDirectory = Path.GetDirectoryName(assemblyLocation);
            var libraryPath = Path.Join(runtimeDirectory, @"netstandard.dll");

            return MetadataReference.CreateFromFile(libraryPath);
        }
    }

    public class Example
    {
        public async Task Execute()
        {
            var _factory = new CSharpCompilationScriptGlobalTypeBuilder();
            string script = @"(Global1.Number+WhatEverNameIWant.Number).ToString() + Global1.Text + WhatEverNameIWant.Text";
            //TODO: need to generate an Id for this script. dealers choice
            Guid key = Guid.NewGuid();

            //TODO: populate this dictionary with the globals you want. 
            //Dictionary<string, object> globals = new Dictionary<string, object>
            //{
            //    { "Global1", new MyCoolClass() { Number = 100, Text = "Something" } },
            //    { "WhatEverNameIWant", new MyCoolClass() { Number = 500, Text = "Longer Text Value" } }
            //};

            dynamic globals = new ExpandoObject();
            globals.Global1 = new MyCoolClass() { Number = 100, Text = "Something" };
            globals.WhatEverNameIWant = new MyCoolClass() { Number = 500, Text = "Longer Text Value" };


            // Act: 
            GlobalTypeInfo typeInfo = _factory.Create(key, globals);

            // TODO: Ideally you would cache the runner for reuse instead of creating it each time.
            var runner = await CreateRunnerAsync(script, typeInfo);
            var instance = Activator.CreateInstance(typeInfo.Type, new object[] { globals });
            var result = await runner.Invoke(instance);
            Console.Write(result);
        }

        private async Task<Microsoft.CodeAnalysis.Scripting.ScriptRunner<string>> CreateRunnerAsync(
            string scriptContent,
            GlobalTypeInfo typeInfo)
        {

            //ref: https://github.com/dotnet/roslyn/blob/main/docs/wiki/Scripting-API-Samples.md
            var options = Microsoft.CodeAnalysis.Scripting.ScriptOptions.Default
                .AddImports("System")
                .AddImports("System.Text");

            //TODO: this is overkill find a better way to do this.
            var assemblies = Assemblies.ApplicationDependencies()
                .SelectMany(assembly => assembly.GetExportedTypes())
                .Select(type => type.Assembly)
                .Distinct();
            options.AddReferences(assemblies);

            using (var loader = new InteractiveAssemblyLoader())
            {
                loader.RegisterDependency(typeInfo.Assembly);

                var script = CSharpScript.Create<string>(
                    scriptContent,
                    options.WithReferences(typeInfo.Reference),
                    globalsType: typeInfo.Type,
                    assemblyLoader: loader);

                return script.CreateDelegate();
            }
        }
    }

    public class GlobalTypeInfo
    {
        public Assembly Assembly { get; init; }
        public MetadataReference Reference { get; init; }
        public Type Type { get; init; }
        public Guid Key { get; init; }
    }


    public class MyCoolClass
    {
        public string Text { get; set; }
        public int Number { get; set; }
    }


}


namespace System.Reflection
{
    public static class Assemblies
    {
        public static IEnumerable<Assembly> ApplicationDependencies(Func<AssemblyName, bool> predicate = null)
        {
            if (predicate == null)
            {
                predicate = _ => true;
            }
            try
            {
                return FromDependencyContext(DependencyContext.Default, predicate);
            }
            catch
            {
                // Something went wrong when loading the DependencyContext, fall
                // back to loading all referenced assemblies of the entry assembly...
                return FromAssemblyDependencies(Assembly.GetEntryAssembly(), predicate);
            }
        }

        private static IEnumerable<Assembly> FromDependencyContext(
            DependencyContext context, Func<AssemblyName, bool> predicate)
        {
            var assemblyNames = context.RuntimeLibraries
                .SelectMany(library => library.GetDefaultAssemblyNames(context));

            return LoadAssemblies(assemblyNames, predicate);
        }

        private static IEnumerable<Assembly> FromAssemblyDependencies(Assembly assembly, Func<AssemblyName, bool> predicate)
        {
            var dependencyNames = assembly.GetReferencedAssemblies();

            var results = LoadAssemblies(dependencyNames, predicate);

            if (predicate(assembly.GetName()))
            {
                results.Prepend(assembly);
            }

            return results;
        }

        private static IEnumerable<Assembly> LoadAssemblies(IEnumerable<AssemblyName> assemblyNames, Func<AssemblyName, bool> predicate)
        {
            var assemblies = new List<Assembly>();

            foreach (var assemblyName in assemblyNames.Where(predicate))
            {
                try
                {
                    // Try to load the referenced assembly...
                    assemblies.Add(Assembly.Load(assemblyName));
                }
                catch
                {
                    // Failed to load assembly. Skip it.
                }
            }

            return assemblies;
        }
    }
}

namespace Microsoft.CodeAnalysis
{
    public static class CompilationExtensions
    {
        public static ImmutableArray<byte> EmitToArray(this Compilation compilation)
        {
            using (MemoryStream assemblyStream = new MemoryStream())
            {

                Microsoft.CodeAnalysis.Emit.EmitResult emitResult = compilation.Emit(assemblyStream);

                if (emitResult.Success)
                {
                    return ImmutableArray.Create<byte>(assemblyStream.ToArray());
                }

                var errors = emitResult
                    .Diagnostics
                    .Select(diagnostic => diagnostic.GetMessage())
                    .Select(message => new Exception(message));

                throw new AggregateException(errors);
            }
        }
    }
}