.NET Framework domains

[filmor]

No description provided.

[vadimkatsman]

I have a feeling that trying to solve it inside ClrLoader is too late - it is .Net managed code and already IN THE root domain.
That explains also why we cannot manage configuration file reference - the root domain is activated by CLR itself when a exported function is called upon.

I would explore trying to solve it in your Runtime - following the same code as .Net itself loads a selected domain (like inside IIS - where each deployed web site is loaded in its own domain).

Another thought: clr loader has 3 or 4 publicly exported functions - but each time you call them, they are executed inside the root domain. Even you are getting delegate pointers from another domain - that another domain is not actually running,

(NOTE: My thoughts are only based on master branch code; I will take a look at your branch some time later today).

I also may share my own experiments with managing multiple domains on few occasions (I had to load a bunch of assemblies to report on types inside them without loading them in the executing domain of the application; launching a side domain was the trick). Since I cannot attach a sample code to this comment, what would be the best way sharing?

[vadimkatsman]

One more thing after reading the code in the branch, maybe setting up the base directory as configurable will also help - if that alone could be done then keeping the config file name as python.exe.config would be OK IF that file is local per main executing script.

It did not occur to me yesterday to test using that general config file name but keeping it in the current directory (as opposed to in base directory, which is python installation folder). I will test it today.

[vadimkatsman]

One more thing after reading the code in the branch, maybe setting up the base directory as configurable will also help - if that alone could be done then keeping the config file name as python.exe.config would be OK IF that file is local per main executing script.

It did not occur to me yesterday to test using that general config file name but keeping it in the current directory (as opposed to in base directory, which is python installation folder). I will test it today.

Tested - did not work. The config file requires base directory not the current directory,

[filmor]

The way we currently host .NET Framework (which is the same as it was done in Python.NET 2) is by using the fact that .NET is loaded automatically by Windows when referencing a .NET-DLL. I'd be open to exploring a more direct way to do this with mscoree.dll (https://learn.microsoft.com/en-us/dotnet/framework/unmanaged-api/hosting/) but I have not looked into this at all, yet.

[vadimkatsman]

I've got it. BTW, here is the code I wanted to share if anything from it helps.
AssemblyAndDomains.zip

I am open to help but need some hints to setup debugging and testing environments for end to end validation. I know everything there is to know about .Net (aggravation but not too much) but did not figure how to do the same with C++ DLLs and Python (I am 10+ years away from last C++ project and still new to Python).

[vadimkatsman]

I just came across the top that allows setting up the configuration file (and I assume the rest of properties) to the loaded AppDomain: https://stackoverflow.com/questions/6150644/change-default-app-config-at-runtime

public ChangeAppConfig(string path) { AppDomain.CurrentDomain.SetData("APP_CONFIG_FILE", path); ResetConfigMechanism(); }

`private static void ResetConfigMechanism()
{
typeof(ConfigurationManager)
.GetField("s_initState", BindingFlags.NonPublic |
BindingFlags.Static)
.SetValue(null, 0);

        typeof(ConfigurationManager)
            .GetField("s_configSystem", BindingFlags.NonPublic | 
                                        BindingFlags.Static)
            .SetValue(null, null);

        typeof(ConfigurationManager)
            .Assembly.GetTypes()
            .Where(x => x.FullName == 
                        "System.Configuration.ClientConfigPaths")
            .First()
            .GetField("s_current", BindingFlags.NonPublic | 
                                   BindingFlags.Static)
            .SetValue(null, null);
    }`

Hope it helps. It may eliminate the need for non-root domains in the first place (even so having multiple domains or non-default domain sounds like a good capability - but with config files and private file path being set to existing app domain its priority could be way down).

[vadimkatsman]

Disclaimer: I hate using this PR as a discussion - but I don't see "Discussion" tab in the repo.

Digging into the subject, I realized non-root domains are simply impossible to have in the context of PythonNet environment - since ALL calls must be marshalled between app domains - the default domain in which Python de-facto is and the non-root domain, in which all managed code must execute. Even if the coding to support that could be simplified using tools such as jduv/AppDomainToolkit(https://github.com/jduv/AppDomainToolkit), it will require all .Net classes including delegates to be serializable with all quirks coming out of it - and I persoanlly operate under fundamental requirements that .Net code must be useful as-is - developed for the purpose of needs of .Net consumer solutions - meaning no code changes in .Net to make it runnable inside python - meaning if something is not serializable in the .Net code base it must remain such.

Which leaves configuring the root domain as close to what .Net expects to see as possible - meaningful configs, target framework, base and private path directories etc.

I understand you have invested on this subject significant time while I am just scratching the surface, but have you explored an option of having AppDomain manager configured in the pre-shipped config file (ptython.exe.config - section). That way even initial domain can be setup to some default settings - based on location of the script, target framework of entry assembly (which is mscorelib itself), initial private path (since these parameters cannot be changed) and then adjusting the location of the config file passed from the script.

Having such pre-shipped config file would resemble the approach IIS took with .Net-based Web applications - having the pre-shipped / global machine.config and allowing each application being configured using application specific config (web.config).

[vadimkatsman]

@filmor, I have a working solution that manages the root domain.

The way how I approached it involves an implementation of AppDomainManager (and without any hacky code around SetData, which did not work any way).

Since it involves AppDomainManager it is technically standalone to PythonNet, And companion Python code is largely independent from it as well - it must run before loading any CLR touching code, routes - in my testing - to a standalone assembly with app domain manager implementation.

I might be missing some .Net runtime features but from what I can see so far, I can setup and read configuration file (both inside assemblies loaded by PythonNet and from py script itself), configure application base directory and private bin path, set the target .Net framework name, verified AppContext switches, return proper entry assembly (at least not null), handle assembly loading. I did not test it yet, but since I handle assembly loading, it should solve assembly binding situation.

As far as configuration file, I can see ConfigurationManager functionality working correctly but did not test runtime sections related ones (such as diagnostics and WCF, for example). Will verify next day or so (or at least will be aware of limitations).

I would like to have some way of DM'ing with you (I can email you directly, for example) to converse on what I am suggesting and what is next - I can always just make my solution publicly available for anybody who face similar needs. However, integrating with your packages sounds like a better proposition.

[vadimkatsman]

@filmor, I created 2 public repos under my organization with the proposed solution.

https://github.com/sctaltrd/adm4p.net
https://github.com/sctaltrd/adm4p.py

There are few tests remaining - like testing switches aka gcAllowVeryLargeObjects and configuration (testing of configuration was successful), but otherwise it is close.

Worst case scenario, would have to be managed via python.exe.config hack (depending in which order the AppDomain's runtime is initialized).

Please note: the AppDomain manager assembly must reside in the same location as the executable of the process in which the runtime is loaded to - which is the installation location of Python (next to Python.exe). However, once app domain is loaded, python.exe directory does not play any role.

[vadimkatsman]

With the reference to pythonnet/pythonnet#2053,

it looks like switches defined in section of the config file are handled in the unmanaged code executed prior to creation of even first AppDomain instance, and those settings are one time only (even if we were able to get hold of for example IGCxxx COM interfaces of the unmanaged portion of the CLR hosting). Looking at the source code of mscorlib, I did not find a single reference to any gc configuration switches.

Without getting into hooking up with unmanaged CLR hosting, this section looks like off limits from what we can do inside app domain configuration (default or non-root alike) since managed code runs inside established runtime.

At the same time, switches defined in section are of global nature and maybe it is OK to have defined them in the python.exe.config file that would contain section and the rest will be handled by the script specific configuration files (per app domain setup).

The globally defined python.exe.config file could act similarly to machine.config or applicationHost.config files (normally present with IIS hosting). As an added benefit along these lines, GC setting switches are ignored inside machine.config but ARE considered inside our de-facto machine level setup, which python,exe.config may become - from the hack to be a valuable player in the maintaining working environment. So dedicating python.exe.config to global runtime switches sounds less of the hack for me (than using that file for script specific configuration).

I tested that the runtime section defined inside python.exe.config is considered and not overriden by the config file otherwise active after my app domain manager sets the script specified one.

Saying all that, I still have a bit of glimmer of hope. The IIS worker process that hosts web sites domains makes a new app domain for each web application. And I was setting the gcAllowVeryLargeObjects key routinely inside the site's configuration file (web.config). The reason why it works could be either IIS worker process is non-managed host - then it can do everything it wants - or there is the way to adjust runtime settings while configuring app domains from managed code. I will conduct more tests and research when time allows.

Yet to test seviceModel section.

[vadimkatsman]

Test for serviceModel section was successful.

I was able to configure the root domain using an appdomain manager setting up base directory, bin path and configuration file from the existing .Net application, which uses WCF to communicate with the remote server. Then, I loaded the application assembly of that application into the script (clr.AddReference()) and called main function from Python (Program.Main([])). The application behaved same as if I would launch it directly - was able to locate unmanaged files in proper places per relative path to binaries, loaded config file, used all features of the proper target framework, launched a subprocess (that hosted a remote functionality) and communicated to WCF end point hosted by that sub-process using WCF client configured from the configuration file setup for the AppDomain.

python.exe.config was not used by anything with exception of gcAllowVeryLargeObject setting.

[filmor]

Initialising a separate domain seems to work now, can you verify? The ADM code that you wrote is very interesting, but I don't have the time to integrate it properly. If you want to give it a go and build a PR from it, please do so! :)

[filmor]

I'll merge this in any case since it at least doesn't make since worse, we still need a test case for separate configuration files.

[vadimkatsman]

The ADM code that you wrote is very interesting, but I don't have the time to integrate it properly. If you want to give it a go and build a PR from it, please do so! :)

After implementing ADM, I used it for last 6 months. And I realized at some point, adding it to clr_loader is not going to work. It must be a standalone assembly not associated with any other assemblies loaded AFTER default domain is loaded (which is loaded by the mscorlib itself). I can move my implementation to your solution - for the purpose of integration.

But the bigger problem is how to integrate configuration. I am essentially ignoring the current netfx configuration (which is set too late in the domain pipeline) and providing an alternative pre-configuration - that sets configuration for AppDomain manager rather than for App domain itself. This is what is required - to allow external configuration for root domain and removing the need for non-root domain - but which is a bit against your initialization code.

I am bit hesitant injecting my thinking and style into your consistent flow. Hence, maintaining it as a separate companion solution.

[vadimkatsman]

Initialising a separate domain seems to work now, can you verify?

I will have to recover my old (pre-ADM) tests to give it a try to verify it can work all way through. I will also test if it takes a specified config file or not.