Using AssemblyLoadContext for isolation, respecting transitive dependencies

[plioi]

I'm porting a test framework to .NET Core, so I need to avoid relying on AppDomains for netcoreapp test assemblies. I've attempted to use a custom AssemblyLoadContext to accomplish the same thing, but it's not clear how to have a custom AssemblyLoadContext that can successfully load dependencies like nuget packages now that such assemblies don't literally appear in a project's build output folder.

Background

Historically, like NUnit and xUnit, I've used an AppDomain to address assembly loading. The running executable is a console application living in some nuget-controlled folder, while the test assembly and its dependencies live in the test assembly's own build output folder. The primary AppDomain for the console application finds the runner's own dependencies right beside the exe, of course, and a secondary AppDomain is created with the ApplicationBase path set to the test assembly's folder (ie ...bin/Debug/MyProject.Tests/). This essentially tricks the test assembly into thinking it's the running application, as far as assembly loading goes.

First Attempt at a Custom AssemblyLoadContext

For a netcoreapp test assembly, we don't have AppDomains. It appears this is one of the things AssemblyLoadContext is intended to solve:

• https://blogs.msdn.microsoft.com/dotnet/2016/02/10/porting-to-net-core/ specifically calls out AssemblyLoadContext as the recommended alternative to AppDomains for resolving assemblies dynamically.
• @gkhanna79's helpful documentation at https://github.com/dotnet/coreclr/blob/master/Documentation/design-docs/assemblyloadcontext.md says these provide the "Ability to load assemblies explicitly in a context isolated from that of the application."
• @jkotas's examples in the comments at https://blogs.msdn.microsoft.com/dotnet/2016/05/27/making-it-easier-to-port-to-net-core/ got me far enough for basic scenarios to work:

    class TestAssemblyLoadContext : AssemblyLoadContext
    {
        readonly string testAssemblyDirectory;

        public TestAssemblyLoadContext(string testAssemblyDirectory)
        {
            this.testAssemblyDirectory = testAssemblyDirectory;
        }

        protected override Assembly Load(AssemblyName assemblyName)
        {
            return LoadFromFolderOrDefault(assemblyName);
        }

        Assembly LoadFromFolderOrDefault(AssemblyName assemblyName)
        {
            try
            {
                var path = Path.Combine(testAssemblyDirectory, assemblyName.Name);

                if (File.Exists(path + ".dll"))
                    return LoadFromAssemblyPath(path + ".dll");

                if (File.Exists(path + ".exe"))
                    return LoadFromAssemblyPath(path + ".exe");

                //TODO: Probably missing something here. What if it's
               //             a transitive nuget dependency, not literally in the
               //             test project's build output folder?

                return null;
            }
            catch (Exception ex)
            {
                  Console.WriteLine(ex);
            }

            return null;
        }
    }

Not too surprisingly, this would only help to load assemblies that literally appear beside the test assembly. Transitive nuget dependencies would only be implied by the runtimeconfig.json, deps.json, and runtimeconfig.dev.json files in that folder.

I had been hoping that the inherited behavior from AssemblyLoadContext would provide some assistance for traversing such dependencies and finding their location on disk, so that my TODO above could find them.

xUnit's Solution

Earlier, the xUnit team ran into a similar challenge, as discussed here: https://github.com/dotnet/core-setup/issues/1926

Instead of making a custom AssemblyLoadContext, their solution instead involves 2 parts:

• kicking off their console runner with "dotnet exec..." so that they can specify the test assembly's own json files as the --depsfile and --runtimeconfig as seen here: https://github.com/xunit/xunit/blob/091f2055c91357d0ac2322348f00e7f4ed7cc31c/src/dotnet-xunit/Program.cs#L388 That appears to solve loading of the test assembly, its physically-present dependencies in the same folder, and likely their transitive dependencies too.
• Since that alone would likely break their console runner from loading its own dependencies, they attach a handler to AssemblyLoadContext.Default.Resolving to load assemblies that live beside the console runner itself: https://github.com/xunit/xunit/blob/091f2055c91357d0ac2322348f00e7f4ed7cc31c/src/xunit.console/Utility/NetCoreAssemblyHelper.cs

Request for Guidance

I've attempted both approaches unsuccessfully, so this is a request for guidance.

With my custom AssemblyLoadContext above, I achieve the desired isolation you'd want in a test framework (it'd be awful if my own reference to Newtonsoft.Json for instance interfered with that of a test project!), but I fail to load transitive dependencies.

When I instead try to use dotnet exec and AssemblyLoadContext.Default.Resolving, my app crashes without even entering Main due to a failure to load its own dependency sitting right beside Main's assembly. I'm not sure why that's happening for me but not for xUnit. Even if I could get around that, I think this leaves the door wide open for mistakes at runtime when the test framework's own dependencies are a different version than those of the test assembly.

What's the right way to achieve AppDomain-like isolation, for a test assembly and its dependencies? VSTest must have had to encounter and resolve the same challenge for dotnet test, but I haven't been able to find it in their implementation.

[gkhanna79]

@kouvel should chime in on this.

[plioi]

I've gotten around this problem by making it so that the end user's test assembly is the console application. This means that it naturally finds any dependencies in the build output folder like any console application would. fixie/fixie#170

So, that resolves my immediate need, and you may decide this issue should be closed. However, the larger question still remains for people in general, "Without AppDomains, what can we really do to load dependencies from a given folder?"

[GiuseppePiscopo]

Is there any update on this "request for guidance"? I'd be interested as well to know how to isolate loading (user) test project dependencies from loading test runner own dependencies.

[kouvel]

Looks similar to https://github.com/dotnet/coreclr/issues/12707. CC @russellhadley

[jeffschwMSFT]

@plioi , @GiuseppePiscopo we are actively looking into isolation as part of AssemblyLoadContexts. Were y'all able to make any progress since the last update on this discussion?

[GiuseppePiscopo]

I struggled for some time then, without any real progress. At that time I think I came through the issue "by chance", because the user test project was referencing a version of Json.Net (what else? :-P ) while the test runner was depending on another version of that.

I thought about removing test runner dependency from Json.Net altogether, to reach a point of zero-dependencies apart from the system ones. But I wasn't sure whether system deps would cause the same issue or not. And I still had the dependency on actual testing framework left to think about.

At the end I dropped the ball, so that is still a blocking point for me.

[plioi]

@jeffschwMSFT No update, though in my case just setting AppDomain-like in-process isolation aside as a goal and instead shelling out to a separate process happened to be the right solution for my problem anyway.

[jeffschwMSFT]

@GiuseppePiscopo sorry to hear this is still blocking.
Thanks @plioi for the update. Out of process is a preferred option for isolation, if possible. What, if anything, are you using for cross process communication? (if you don't mind sharing)

[plioi]

I'm using named pipes. It was a little tricky to get going, but now I can essentially throw DTOs back and forth between the two process using json for serialization.

This is for a test framework. The VS "Test Explorer" adapter is the parent process. The test execution goes on in a child process.

Shared Infrastructure: https://github.com/fixie/fixie/blob/master/src/Fixie/Execution/Listeners/PipeStreamExtensions.cs

Usage (parent process): https://github.com/fixie/fixie/blob/master/src/Fixie.VisualStudio.TestAdapter/VsTestDiscoverer.cs#L37-L74

Usage (parent process): https://github.com/fixie/fixie/blob/master/src/Fixie.VisualStudio.TestAdapter/VsTestExecutor.cs#L98-L145

Usage (child process): https://github.com/fixie/fixie/blob/master/src/Fixie/Execution/Listeners/PipeListener.cs

[andrewLarsson]

I am working on a project that would benefit from AssemblyLoadContext isolation. We are using Assembly.LoadFrom and giving it the path to a .dll that has all of its dependencies in its same directory. Everything works great up until the point we try to use an assembly that references a newer version of Newtonsoft.Json. Looking at the loaded modules at runtime, Newtonsoft.Json version 10 is already loaded, but the dynamically-loaded .dll requires version 11 and an exception (System.IO.FileLoadException: 'Could not load file or assembly 'Newtonsoft.Json, Version=11.0.0.0...) is thrown when I start using types from the assembly that require version 11 (which is weird, the dynamically-loaded .dll loads just fine and Assembly.LoadFrom completes successfully, but I get an exception later on when trying to use those types) (I can use all the other types from the dynamically-loaded .dll just fine except for the ones that require Newtonsoft.Json version 11). I'm using .NET Core 2.0 if that makes any difference.

[jeffschwMSFT]

@andrewLarsson unfortunately you are hitting a known limitation of loadfrom where additional dependencies are not found. This is an area we are aware is lacking and exploring. In the meantime are you able to use AssemblyResolveEvent to find the missing dependency?

[andrewLarsson]

@jeffschwMSFT I tried what you suggested but it ends up throwing the same exception.

AssemblyLoadContext.Default.Resolving += (context, name) => {
	if (name.Name == "Newtonsoft.Json" && name.Version.ToString() == "11.0.0.0") {
		Assembly assembly = Assembly.LoadFrom("C:\\Users\\andre_000\\.nuget\\packages\\newtonsoft.json\\11.0.1\\lib\\netstandard2.0\\Newtonsoft.Json.dll");
		return assembly;
	}
	return null;
};

I can put a breakpoint in my event and see that it gets called, but Assembly.LoadFrom fails with:

System.IO.FileLoadException: 'Could not load file or assembly 'Newtonsoft.Json, Version=11.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'. Could not find or load a specific file. (Exception from HRESULT: 0x80131621)'

[jeffschwMSFT]

@andrewLarsson you will need to first setup an AssemblyLoadContext in order to load the same assembly with different versions. Once you have an assemblyloadcontext created, and set the resolve event for that assembly, you can a LoadAssemblyFromPath on the assembly that has a Newtonsoft.Json dependency of a different version. The future work we are doing will hopefully avoid the need for the additional assemblyresolve event.

[andrewLarsson]

@jeffschwMSFT Oh, I see now. I misunderstood how the AssemblyLoadContext worked. I finally found the documentation and now it makes sense. I implemented my own AssemblyLoadContext and it provides me exactly the level of "isolation" I need. Thank you!

[per-samuelsson]

@andrewLarsson

You still "only" support to load custom assemblies where all dependencies are in the same folder, right? Like a published app. And not supporting dependencies to be found in, and loaded from, packages?

We've been working on a solution for quite some time now, trying to achieve proper dependency resolving of the same sort done any standard application out-of-the-box, but for assemblies we do load in a custom AssemblyLoadContext. Still working on it, and I've found it's quite tricky to get it right.

Our solution currently use Microsoft.Extensions.DependencyModel with custom ICompilationAssemblyResolvers, but there seem to be quite a few corner cases - mostly with non-standard package structures - that is a challenge for sure.

I actually raised an issue about it some time ago, and @eerhardt helped me with some pointers in this area, for example this one: https://github.com/dotnet/core-setup/issues/3668#issuecomment-377033296. (That particular issue initially about published apps, but that was because I didn't understand what actually was failing at that point, so it's actually about the same thing: trying to load assemblies dynamically into some host, and have dependencies resolve properly).

If possible, I too would definitely appreciate some added "request for guidance", as is asked for in the OP of this issue.

[andrewLarsson]

@per-samuelsson Yes, my AssemblyLoadContext only supports dependencies in the same folder. I add <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies> to the PropertyGroup in the project file to make sure all the dependencies get copied over. I had found some information on how the DependencyModel works for inspecting/parsing .deps.json files, but I didn't go too far with it because, for my project, I was able to get away with just copying over the dependencies.

Edit: Now looking at that comment you linked, I see that dotnet publish does the same thing as CopyLocalLockFileAssemblies (but CopyLocalLockFileAssemblies works easier for me for now).

Edit: I would be more than happy to provide you with my AssemblyLoadContext if you would like. It supports sharing of dependencies, too, so it doesn't have to be totally isolated. Actually, it doesn't just support the same folder. It supports multiple folders (but not recursive, you could add that yourself if you want).

Edit: SharedAssemblyLoadContext

[per-samuelsson]

I had found some information on how the DependencyModel works for inspecting/parsing .deps.json files, but I didn't go too far with it because, for my project, I was able to get away with just copying over the dependencies.

I think that was a wise choice, and mainly b/c that sample is pretty naive and would not really work. I have worked my way up from something similar and learned the hard way how tricky this thing really is to get right. 😎 😓

Edit: SharedAssemblyLoadContext

Thanks for sharing! Our needs are unfortunately a bit more complex, so even extending it like you suggest would not be sufficient for us. What we aim at is a shared app host, where we can load any random set of applications in a single host, and have them run in isolated contexts, with all dependency resolving to happen just as they would be running in the standard host. Including resolving dependencies in even the most "advanced" packages (take a look at System.Data.SqlClient for example, or the roslyn code analysis "metapackage" - as far as I've found, there's no ready-made resolving for those scenarios, or at least I have not got it right after sweating it quite a bit), and unmanaged assets.

WTBS, at this point we are probably going back to the drawing table, trying to figure out some new plan for our product, and chose some other path. Simply because it seem to high a price to get this thing correct all together (if you're doing something as general as we first envisioned - something more niched).

[Lakritzator]

@per-samuelsson I just stumbled upon this issue, doesn't the AssemblyDependencyResolver which was introduced here: https://devblogs.microsoft.com/dotnet/announcing-net-core-3-preview-3/ help you?

Examples, and a description on the AssemblyDependencyResolver, can be found here: https://github.com/dotnet/samples/tree/master/core/extensions/AppWithPlugin

[BrainCrumbz]

@Lakritzator per this piece of example readme:

[...] an application can load plugins so that each plugin's dependencies are loaded from the correct location, and one plugin's dependencies will not conflict with another. This sample includes plugins that have conflicting dependencies and plugins that rely on satellite assemblies or native libraries.

it seems that could actually help.

[per-samuelsson]

@per-samuelsson I just stumbled upon this issue, doesn't the AssemblyDependencyResolver which was introduced here: https://devblogs.microsoft.com/dotnet/announcing-net-core-3-preview-3/ help you?

Examples, and a description on the AssemblyDependencyResolver, can be found here: https://github.com/dotnet/samples/tree/master/core/extensions/AppWithPlugin

At least look interesting! Thanks for the pointer. 👍

I wonder if it also solve traversal of references to shared framework assemblies.

[vitek-karas]

@per-samuelsson Can you please provide a bit more detail to "solve traversal of references to shared framework assemblies"? I'm not clear on what you're asking about.

The AssemblyDependencyResolver will intentionally not resolve any framework assemblies. If it is combined with a typical implementation of AssemblyLoadContext which would return null from its Load method if the resolver doesn't resolve an assembly, this should lead the binder to try to load the framework assembly from the AssemblyLoadContext.Default which should just work.

This design was intentional, since it's almost never desirable to load multiple copies of framework assemblies. This way all framework assemblies are loaded through the default context and thus end up loaded exactly once.

[per-samuelsson]

@vitek-karas

Sorry for the late response.

We paused what we were trying because we didn't get it to work properly at the time, and had to rethink some parts. We might need to go back to working on something similar later this year, so I'm still interested in checking out what progress you've made in this area though, just don't have the time for it right now.

Regarding

@per-samuelsson Can you please provide a bit more detail to "solve traversal of references to shared framework assemblies"? I'm not clear on what you're asking about.

you can read some background here:

natemcmaster/DotNetCorePlugins#19

Does it make any sense to you?

[vitek-karas]

I see - thanks for the pointer.

Unfortunately loaded frameworks through the AssemblyDependencyResolver (ADR) is currently not supported. I'll list just a couple of reasons why doing so is tricky (in no particular order):

• In general we don't think we want to support loading frameworks more than once into the process (using ALC with ADR which automatically loads frameworks could make this really easy to achieve).
  • Frameworks might have global state which should not be duplicated (ASP.NET is probably not like that, but WPF on the other hand is likely to have that problem).
  • Types from frameworks are typically used for sharing instances across ALC boundaries - duplicate loading would break this. Think what would happen in WPF scenario if the IFrameworkElement type from the plugin would not match the one from the host...
  • There's really no good way of allowing Microsoft.NETCore.App framework to exist twice in the process (that would create a complete mess, not counting that runtime is part of the framework as we can't load the runtime twice).
• The plugin would need .runtimeconfig.json to declare its framework dependencies
  • This would mean ADR would have to parse that and understand it. It would also have to reconcile those frameworks with those already loaded in the process - typically the Microsoft.NETCore.App framework.
  • We are sort of moving in the direction of component having .runtimeconfig.json see Add property for building a library as a component sdk#3305 for details, but that is mostly for components loaded through native hosts (COM and similar), the same could be applied to plugins like in this case probably.

So our current guidance is to "preload" all necessary frameworks to the host. I'm curious what is the real-world use case for this scenario: "Simple console app loading "in-proc" another application - possibly an ASP.NET app".
Right now I'm only aware of one such case which is essentially dotnet watch but done in-proc. We have some prototypes of it which already work and we don't think forcing the host to load the framework is a big problem.

One potentially related improvement in .NET Core 3.0 is the addition of rollForward:LatestMinor or rollForward:LatestMajor - see https://github.com/dotnet/designs/blob/master/accepted/runtime-binding.md for details. That would allow the host app to load latest available version of the framework, to make it "likely" that the plugin will work (basically fixing the problem where the plugin would require higher version than the host can provide).

[shvez]

I'm testing AssemblyDependencyResolver with dotnet core 3 preview 6 and encountered behavior which I can not explain and fix.

We have a host app, which loads plugins. The host app is built for netcoreapp3.0, plugins are built for netcoreapp2.1. The host app has its own project/solution for development. The plugins have their own solution

Now the issue:
if I publish the host app, and setup debugging for the plugin so that it starts published host application, then it fails to load assemblies which are in '.nuget' cache.

if I open the host app solution and setup debugging so that it loads plugin (still not published) it somehow finds stuff in the nuget cache and loads everything.

the only difference I found in my logs that in case of publishing libs are loaded from publish folder, in case of project debugging they are loaded from netcore3 install folder. Well, and this is expected.

So, I'm confused and not really understand how to fix this issue. It is really annoying because I would like to get rid of custom code for plugin loading and use AssemblyDependencyResolver, which seems to be working better

best,
ilya

[vitek-karas]

@shvez
Without repro I can only guess, but I'm pretty sure the below describes your situation:

• If you start with published app (host), there will only be app.runtimeconfig.json but no app.runtimeconfig.dev.json.
• If you start with the app solution and just "run" it, you will be using the dotnet build output which does contain app.runtimeconfig.dev.json.

In .NET Core 2.* the build (dotnet build) didn't copy all dependencies to the output, and instead it relied on the app.runtimeconfig.dev.json to include all the paths where the dependencies can be located (among other things this includes the NuGet cache). In .NET Core 3.0 the build will copy all dependencies to the output, but the app.runtimeconfig.dev.json is still produced. It's currently discussed if the build should not produce it (as it's effectively redundant).

In your case though it plays important role. Since your plugins are built using .NET 2.* they will not get all their dependencies copied to the output. So if the app has the .runtimeconfig.dev.json they will "accidentally" work because that file points to NuGet caches. But if the app doesn't have it, it will fail since the system doesn't know where to find the dependencies.

If you would build the plugins with .NET Core 3.0 this problem would go away since the build would copy dependencies to the plugins. You could also publish the plugins which would also do that.

This is all still an area we're trying to clean up - define the SDK experience and so on. But in a way, no matter what we'll do it's unlikely to be ported back to 2.*. So my current recommendation would be to port your plugins to 3.0 which should solve your immediate problem.

In general I would be very interested how is your entire "plugin system" experience. What are your thoughts on the solutions, what works, but is not ideal and so on. If you are willing to share that, please feel free to either respond here, file new issues or if you want send me an email (it should be on my profile).

[shvez]

@vitek-karas I've written an email to you. Later I will try to fix the issue according to your suggestions
thanks a lot

[shvez]

@vitek-karas so, I have installed VS preview and build everything with netcoreapp3.0 as target framework. but I still do not see that dependencies are copied. I'm talking about assembly build, not about console application build. It looks like in case of console application it works as you described - dependencies are there and runtime configs too.
for class libraries, it is not the case though
is there any compiler key for that?

[vitek-karas]

@shvez Sorry for the confusion - you're right, it doesn't work... "yet" :-)
The change which will make this work much better is dotnet/sdk#3305 - this will introduce some kind of MSBuild property (Currently called IsComponent) which you woudl set to true in the plugins and the rest will work.

For now, you might be able to set CopyLocalLockFileAssemblies to true in the plugin project ... but I haven't tried it.

[shvez]

@vitek-karas CopyLocalLockFileAssemblies seems to be working. It looks like I would do publishing

[shvez]

guys, one more question about AssemblyDependencyResolver. why in one case does it return assembly from 'runtimes' folder and in another case it takes it from assembly folder. One in assembly folder usually wrong one when there is platform-specific version.
As far as I may see there is some influence from app.runtimeconfig.dev.json side. But how I may get right one assembly if I do not have such file?

[vitek-karas]

@shvez Can you please create a new issue for this (this one is already getting out of hand anyway)?
It's really hard to tell what's wrong just from the description. Can you please:

• set COREHOST_TRACE=1 before you run the application
• Run the app and redirect its stderr output to a file (if you have 3.0 you can instead set COREHOST_TRACEFILE=path)

The log this produces will be large, so in it please search for the name of the plugin you're trying to load via the AssemblyDependencyResolver - there should be a section of that file which starts with something like this:

--- Invoked hostpolicy [commit hash: ... ] corehost_resolve_component_dependencies
...
corehost_resolve_component_dependencies results: {

The second line should contain the path to the plugin you're loading. Anything between the ---Invoked and the corehost_resolve_component_dependencies results is the log for the ADR operation. If you can include that in the new issue that will help a lot.

Please note that the log contains full file paths from your machine, so it may contain information like user name, project names and so on - if you're not comfortable posting such information online, then don't share the log.

[moh-hassan]

This custom ALC can load transitive dependencies

The ALC use the AssemblyDependencyResolver to resolve the dependencies.

I tested it with a complex plugin scenario that reference a class library and external OLEDB nuget package and it's working fine. Also it's working in dotnet samples.

[plioi]

I think this can be closed now that it's easy to implement an AssemblyLoadContext in combination with AssemblyDependencyResolver, and since there's some clear documentation on the ins-and-outs: https://docs.microsoft.com/en-us/dotnet/core/tutorials/creating-app-with-plugin-support