[wasm] System.Runtime.Loader.Tests fail when run in randomized order

[radical]

Found on #65628 .

System.Runtime.Loader.Tests fail randomly, but deterministically with XUNIT_RANDOM_ORDER_SEED=2106784294.

[06:25:43] fail: [FAIL] System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(alc: "Empty", assemblyName: "System.Runtime.Loader.Tests", culture: "en")
[06:25:43] info: Assert.Same() Failure
[06:25:43] info: Expected: "Default" System.Runtime.Loader.DefaultAssemblyLoadContext #0
[06:25:43] info: Actual:   "Empty" System.Runtime.Loader.AssemblyLoadContext #4
[06:25:43] info:    at System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(String alc, String assemblyName, String culture)
[06:25:43] info:    at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters)

Maybe these tests have some unintended dependence on order?
To test it with that PR, make sure to false from the csproj.

[ghost]

Tagging subscribers to 'arch-wasm': @lewing
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Found on #65628 .

System.Runtime.Loader.Tests fail randomly, but deterministically with XUNIT_RANDOM_ORDER_SEED=2106784294.

[06:25:43] fail: [FAIL] System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(alc: "Empty", assemblyName: "System.Runtime.Loader.Tests", culture: "en")
[06:25:43] info: Assert.Same() Failure
[06:25:43] info: Expected: "Default" System.Runtime.Loader.DefaultAssemblyLoadContext #0
[06:25:43] info: Actual:   "Empty" System.Runtime.Loader.AssemblyLoadContext #4
[06:25:43] info:    at System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(String alc, String assemblyName, String culture)
[06:25:43] info:    at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters)

Maybe these tests have some unintended dependence on order?
To test it with that PR, make sure to false from the csproj.

Author:	radical
Assignees:	-
Labels:	arch-wasm
Milestone:	-

[dotnet-issue-labeler]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

Tagging subscribers to this area:
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Found on #65628 .

System.Runtime.Loader.Tests fail randomly, but deterministically with XUNIT_RANDOM_ORDER_SEED=2106784294.

[06:25:43] fail: [FAIL] System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(alc: "Empty", assemblyName: "System.Runtime.Loader.Tests", culture: "en")
[06:25:43] info: Assert.Same() Failure
[06:25:43] info: Expected: "Default" System.Runtime.Loader.DefaultAssemblyLoadContext #0
[06:25:43] info: Actual:   "Empty" System.Runtime.Loader.AssemblyLoadContext #4
[06:25:43] info:    at System.Runtime.Loader.Tests.SatelliteAssembliesTests.SatelliteLoadsCorrectly_FromName(String alc, String assemblyName, String culture)
[06:25:43] info:    at System.Reflection.RuntimeMethodInfo.InvokeWorker(Object obj, BindingFlags invokeAttr, Span`1 parameters)

Maybe these tests have some unintended dependence on order?
To test it with that PR, make sure to false from the csproj.

Author:	radical
Assignees:	-
Labels:	arch-wasm, untriaged, area-AssemblyLoader-mono
Milestone:	-

[ilonatommy]

Still failing.

[lewing]

@vargaz can you make sure we understand the issue here?

[vargaz]

Can't reproduce this. I ran with:

XUNIT_RANDOM_ORDER_SEED=2106784294 EMSDK_PATH=$REPO/src/mono/wasm/emsdk/ ./dotnet.sh build /t:Test /p:TargetOS=Browser /p:TargetArchitecture=wasm /p:Configuration=Release src/libraries/System.Runtime.Loader/tests/

[radical]

Did you disable this?

runtime/src/libraries/System.Runtime.Loader/tests/System.Runtime.Loader.Tests.csproj

Line 14 in ed2a5a1

<XUnitUseRandomizedTestOrderer>false</XUnitUseRandomizedTestOrderer>

[vargaz]

I can reproduce now.

[vargaz]

Fails without randomized order as well. Testcase:

using System;
using System.Reflection;
using System.Resources;
using System.Globalization;
using Xunit;

namespace System.Runtime.Loader.Tests
{
    public class SatelliteAssembliesTestsFixture
    {
            [Fact]
            public void FOO () {
                string assemblyName = "System.Runtime.Loader.Tests";

                AssemblyName satelliteAssemblyName = new AssemblyName(assemblyName + ".resources");
                satelliteAssemblyName.CultureInfo = new CultureInfo("en");

                var alc = new AssemblyLoadContext("Empty");

                Assembly satelliteAssembly = alc.LoadFromAssemblyName(satelliteAssemblyName);

                Assembly parentAssembly = alc.LoadFromAssemblyName(new AssemblyName(assemblyName));

                Console.WriteLine (parentAssembly == typeof (SatelliteAssembliesTestsFixture).Assembly);
                Assert.Equal (parentAssembly, typeof (SatelliteAssembliesTestsFixture).Assembly);
        }
    }
}

[vargaz]

Commenting out the:

                Assembly satelliteAssembly = alc.LoadFromAssemblyName(satelliteAssemblyName);

line makes the test pass.

[ghost]

This issue has been marked needs-author-action and may be missing some important information.

[ghost]

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

[ghost]

This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.

[ghost]

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

[ghost]

This issue has been automatically marked no-recent-activity because it has not had any activity for 14 days. It will be closed if no further activity occurs within 14 more days. Any new comment (by anyone, not necessarily the author) will remove no-recent-activity.

[ghost]

This issue will now be closed since it had been marked no-recent-activity but received no further activity in the past 14 days. It is still possible to reopen or comment on the issue, but please note that the issue will be locked if it remains inactive for another 30 days.