Adapt monotouch-test to work with NativeAOT

[ivanpovazan]

Description

During the startup, monotouch-test app collects the assemblies which represent test suites and calls NUnitTestAssemblyRunner to bundle each of them in a TestSuite object.
This happens in TouchRunner::GetViewController through LoadSync method.

The following code path is followed:

https://github.com/spouliot/Touch.Unit/blob/564433f35c8ab6b7bb0709f83e1b81a89c406956/NUnitLite/TouchRunner/TouchRunner.cs#L168-L170

https://github.com/spouliot/Touch.Unit/blob/564433f35c8ab6b7bb0709f83e1b81a89c406956/NUnitLite/TouchRunner/TouchRunner.cs#L608C4

https://github.com/nunit/nunit/blob/af8ca8bc779072b4fa5432e8620f06ca796d13a3/src/NUnitFramework/framework/Api/NUnitTestAssemblyRunner.cs#L170

https://github.com/nunit/nunit/blob/af8ca8bc779072b4fa5432e8620f06ca796d13a3/src/NUnitFramework/framework/Api/DefaultTestAssemblyBuilder.cs#L81

https://github.com/nunit/nunit/blob/af8ca8bc779072b4fa5432e8620f06ca796d13a3/src/NUnitFramework/framework/Internal/AssemblyHelper.cs#L48-L56

Finally, the final piece ends up calling CodeBase property on Assembly type which is not supported with NativeAOT:
https://github.com/dotnet/runtime/blob/588dcd7a29f109daae2d8c999322acd8e853a8c6/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/General/ThunkedApis.cs#L70-L78

Proposal

• Introduce a custom TestAssemblyBuilder implementing ITestAssemblyBuilder which would instead fetch Assembly.GetName().FullName for an assembly
• Try reusing NUnitLiteTestAssemblyBuilder from NUnitLite-1.0.0: https://github.com/spouliot/Touch.Unit/blob/564433f35c8ab6b7bb0709f83e1b81a89c406956/NUnitLite/TouchRunner/TouchRunner.cs#L610-L622

/cc: @rolfbjarne

[rolfbjarne]

Can you try just calling this overload instead?

https://github.com/nunit/nunit/blob/af8ca8bc779072b4fa5432e8620f06ca796d13a3/src/NUnitFramework/framework/Api/DefaultTestAssemblyBuilder.cs#L97

So here:

https://github.com/spouliot/Touch.Unit/blob/564433f35c8ab6b7bb0709f83e1b81a89c406956/NUnitLite/TouchRunner/TouchRunner.cs#L608

do this instead:

return AddSuite ((TestSuite) runner.Load (assembly.Location, CreateSettings (settings)));

[ivanpovazan]

I tried this:

return AddSuite ((TestSuite) runner.Load (assembly.Location, CreateSettings (settings)));

However, unfortunately, it also leads to a crash/unsupported features.

• The Location property on Assembly with NativeAOT is just an empty string:
  https://github.com/dotnet/runtime/blob/588dcd7a29f109daae2d8c999322acd8e853a8c6/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/Runtime/General/ThunkedApis.cs#L66

• The code path that is followed in this case, ends up trying to:
  https://github.com/nunit/nunit/blob/af8ca8bc779072b4fa5432e8620f06ca796d13a3/src/NUnitFramework/framework/Internal/AssemblyHelper.cs#L155
  which will again fail, as the AssemblyName.FullName will be an empty string
  Failure reached through: https://github.com/dotnet/runtime/blob/d9f453924f7c3cca9f02d920a57e1477293f216e/src/coreclr/nativeaot/System.Private.CoreLib/src/System/Reflection/RuntimeAssemblyName.cs#L172
  and finally:
  https://github.com/dotnet/runtime/blob/d9f453924f7c3cca9f02d920a57e1477293f216e/src/libraries/System.Private.CoreLib/src/System/Reflection/AssemblyNameFormatter.cs#L16

In general, notions of Location/CodeBase do not seem to make sense with NativeAOT as the assembly is not preserved in its original format.

PS I would be happy to contribute to solving this.

[rolfbjarne]

Next attempt: try calling this method using reflection (since it's private):

https://github.com/nunit/nunit/blob/af8ca8bc779072b4fa5432e8620f06ca796d13a3/src/NUnitFramework/framework/Api/DefaultTestAssemblyBuilder.cs#L117

NUnit's code flow is rather annoying, because the only reason it seems to want location/codebase is to figure out the assembly name, and then it also reloads the assembly when it's already been given the assembly. Calling that method might skip over that logic, and go straight to the actual important bits. If that doesn't work, maybe we'll have to implement a custom ITestAssemblyBuilder.

PS I would be happy to contribute to solving this.

I guess one way would be to just try to make NUnit work with NativeAOT, and get it fixed in NUnit instead...

[ivanpovazan]

According to your suggestion of using a different overload, would it make sense to do this instead of using reflection?

public bool Load (Assembly assembly, IDictionary<string, object> settings = null)
{
        var runner = new NUnitTestAssemblyRunner (builder);
        runners.Add (runner);
+#if NATIVEAOT
+       return AddSuite ((TestSuite) runner.Load (assembly.GetName().Name, CreateSettings (settings)));
+#else
        return AddSuite ((TestSuite) runner.Load (assembly, CreateSettings (settings)));
+#endif
}

I agree it is hacky, but seems to work as NUnit will recreate the FullName based on already available name...
If you think this is not acceptable, I can try to use reflection and/or look into how to contribute to NUnit framework instead.

[rolfbjarne]

According to your suggestion of using a different overload, would it make sense to do this instead of using reflection?

public bool Load (Assembly assembly, IDictionary<string, object> settings = null)
{
        var runner = new NUnitTestAssemblyRunner (builder);
        runners.Add (runner);
+#if NATIVEAOT
+       return AddSuite ((TestSuite) runner.Load (assembly.GetName().Name, CreateSettings (settings)));
+#else
        return AddSuite ((TestSuite) runner.Load (assembly, CreateSettings (settings)));
+#endif
}

I agree it is hacky, but seems to work as NUnit will recreate the FullName based on already available name... If you think this is not acceptable, I can try to use reflection and/or look into how to contribute to NUnit framework instead.

Won't that run into the same issue as you describe here: #17774 (comment)?

In any case, I'm fine with this solution if it works (even better if the NATIVEAOT codepath works when !NATIVEAOT, then we can just use that always).

[ivanpovazan]

Won't that run into the same issue as you describe here: #17774 (comment)?

The difference is that with the approach above:

return AddSuite ((TestSuite) runner.Load (assembly.Location, CreateSettings (settings)));

Location returns an empty string causing FullName generation to fail when NUnit tries to build the assembly name.
While on the other hand, we can trick NUnit by providing a simple name of the assembly.

[rolfbjarne]

Won't that run into the same issue as you describe here: #17774 (comment)?

The difference is that with the approach above, Location returns an empty string causing FullName generation to fail when NUnit tries to build the assembly name.

Ah ok - well, if that works, then great :)

[ivanpovazan]

In any case, I'm fine with this solution if it works (even better if the NATIVEAOT codepath works when !NATIVEAOT, then we can just use that always).

Regarding this, I was wondering whether some test (or the infrastructure around it) cares about assembly name in the "Assembly.dll" format.

[rolfbjarne]

In any case, I'm fine with this solution if it works (even better if the NATIVEAOT codepath works when !NATIVEAOT, then we can just use that always).

Regarding this, I was wondering whether some test (or the infrastructure around it) cares about assembly name in the "Assembly.dll" format.

There might be, but we can change those tests if need be.

[rolfbjarne]

PR with the Touch.Unit bump: #17819