Some tests depend on machine specifics

[weltkante]

.NET Core Version:
5.0 master branch

Have you experienced this same bug with .NET Framework?:
not applicable

Problem description:
After removing culture-specific test failures (tracked by #2734) there are a few more tests failing for me which I believe happen due to relying on machine specific conditions.

list of failed tests

* System.Windows.Forms.Tests.CursorTests.Cursor_Clip_Set_GetReturnsExpected
Assert.Equal() Failure
Expected: {X=0,Y=0,Width=3456,Height=1080}
Actual:   {X=0,Y=0,Width=3840,Height=1080}
   at System.Windows.Forms.Tests.CursorTests.Cursor_Clip_Set_GetReturnsExpected() in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\CursorTests.cs:line 176

* System.Windows.Forms.Tests.ListViewTests.ListView_GetItemRect_InvokeWithHandle_ReturnsExpected
Assert.True() Failure
Expected: True
Actual:   False
   at System.Windows.Forms.Tests.ListViewTests.ListView_GetItemRect_InvokeWithHandle_ReturnsExpected() in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\ListViewTests.cs:line 3680

* System.Windows.Forms.Tests.ListViewTests.ListView_GetItemRect_InvokeWithoutHandle_ReturnsExpectedAndCreatedHandle
Assert.True() Failure
Expected: True
Actual:   False
   at System.Windows.Forms.Tests.ListViewTests.ListView_GetItemRect_InvokeWithoutHandle_ReturnsExpectedAndCreatedHandle() in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\ListViewTests.cs:line 3645

* System.Windows.Forms.Tests.MonthCalendarTests.MonthCalendar_CalendarDimensions_SetAreaOfTwelve_GetReturnsExpected(width: 12, height: 1)
Assert.True() Failure
Expected: True
Actual:   False
   at System.Windows.Forms.Tests.MonthCalendarTests.MonthCalendar_CalendarDimensions_SetAreaOfTwelve_GetReturnsExpected(Int32 width, Int32 height) in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\MonthCalendarTests.cs:line 586

* System.Windows.Forms.Tests.MonthCalendarTests.MonthCalendar_CalendarDimensions_SetWithHandleAreaOfTwelve_GetReturnsExpected(width: 12, height: 1)
Assert.True() Failure
Expected: True
Actual:   False
   at System.Windows.Forms.Tests.MonthCalendarTests.MonthCalendar_CalendarDimensions_SetWithHandleAreaOfTwelve_GetReturnsExpected(Int32 width, Int32 height) in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\MonthCalendarTests.cs:line 644

* System.Windows.Forms.Tests.MonthCalendarTests.MonthCalendar_SingleMonthSize_GetWithHandle_ReturnsExpected
Assert.True() Failure
Expected: True
Actual:   False
   at System.Windows.Forms.Tests.MonthCalendarTests.MonthCalendar_SingleMonthSize_GetWithHandle_ReturnsExpected() in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\MonthCalendarTests.cs:line 2338

* System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Ctor_Default
Assert.Equal() Failure
Expected: 2
Actual:   79
   at System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Ctor_Default() in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\ToolStripDropDownTests.cs:line 142

* System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Location_Set_GetReturnsExpected(value: {X=-1,Y=-2}, expected: {X=0,Y=0}, expectedLocationChangedCallCount: 0)
Assert.Equal() Failure
Expected: 0
Actual:   77
   at System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Location_Set_GetReturnsExpected(Point value, Point expected, Int32 expectedLocationChangedCallCount) in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\ToolStripDropDownTests.cs:line 1966

* System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Location_Set_GetReturnsExpected(value: {X=0,Y=2}, expected: {X=0,Y=2}, expectedLocationChangedCallCount: 1)
Assert.Equal() Failure
Expected: 0
Actual:   77
   at System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Location_Set_GetReturnsExpected(Point value, Point expected, Int32 expectedLocationChangedCallCount) in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\ToolStripDropDownTests.cs:line 1966

* System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Location_Set_GetReturnsExpected(value: {X=1,Y=0}, expected: {X=1,Y=0}, expectedLocationChangedCallCount: 1)
Assert.Equal() Failure
Expected: 1
Actual:   77
   at System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Location_Set_GetReturnsExpected(Point value, Point expected, Int32 expectedLocationChangedCallCount) in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\ToolStripDropDownTests.cs:line 1966

* System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Location_Set_GetReturnsExpected(value: {X=1,Y=2}, expected: {X=1,Y=2}, expectedLocationChangedCallCount: 1)
Assert.Equal() Failure
Expected: 1
Actual:   77
   at System.Windows.Forms.Tests.ToolStripDropDownTests.ToolStripDropDown_Location_Set_GetReturnsExpected(Point value, Point expected, Int32 expectedLocationChangedCallCount) in D:\projects\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\ToolStripDropDownTests.cs:line 1966

There is a compromise to make between developer convenience and how exactly you require tests matching their OS environment, if they are too strict developers can run tests only in a VM.

So far I could identify:

• Some MonthCalendar tests generate a huge control and rely on the OS cutting them off. If you actually have a big work area (e.g. two screens) this leads to test failures since the controls are no longer cut off and are larger than the test expects.
• Some tests mix up coordinate spaces and do not consider the fact that the taskbar is not necessarily at the bottom. When the OS ensures the taskbar is not covered some tests fail because the forms/controls are not at the location they expect them to be. This is easily worked around by considering the work area inside the test, not the virtual screen space.

I'll put together a PR, so you can then decide which of the changes are reasonable enough to take.

Expected behavior:
Tests are (within reason) coded to be independent of machine characteristics.

Minimal repro:
Having either of these characteristics will fail some tests:

• Set the taskbar to the left side of the monitor (instead of default bottom) and run the tests
• Use a multi-screen machine and run the tests (very wide or very high work area)

[hughbe]

I would emphasise getting tests passing on ANY machine over specific VMs as it reduces the contribution burden

[weltkante]

There are a lot of global knobs to turn in Windows, I would not expect tests to accomodate for all of them. Some things may even be impossible to test on some machines. Like theming may be turned off globally, it'd be hard to get those tests running sucessfully, the best you can do is to detect this and skip them. You might also get a lot of failures if you use high contrast themes globally (didn't test but its something which changes a lot of measurements). Neither of these happened to me, just saying this is a bottomless hole to dig into :-)

[RussKie]

Reopening since we still have few tests that appear to be machine specific:

    System.Windows.Forms.Tests.AxHostTests.AxHost_GetIFontDispFromFont_InvokeSimpleStyle_Roundtrips [FAIL]
      Assert.Equal() Failure
      Expected: 0
      Actual:   254
      Stack Trace:
        C:\Development\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\AxHostTests.cs(1423,0): at System.Windows.Forms.Tests.AxHostTests.AxHost_GetIFontDispFromFont_InvokeSimpleStyle_Roundtrips()
    System.Windows.Forms.Tests.AxHostTests.AxHost_GetIFontFromFont_InvokeSimpleStyle_Roundtrips [FAIL]
      Assert.Equal() Failure
      Expected: 0
      Actual:   254
      Stack Trace:
        C:\Development\winforms\src\System.Windows.Forms\tests\UnitTests\System\Windows\Forms\AxHostTests.cs(1515,0): at System.Windows.Forms.Tests.AxHostTests.AxHost_GetIFontFromFont_InvokeSimpleStyle_Roundtrips()
    System.Windows.Forms.Tests.MonthCalendarTests.MonthCalendar_Handle_GetWithDefaultFirstDayOfWeek_Success [FAIL]
      Assert.Equal() Failure
      Expected: 6
      Actual:   0

[weltkante]

On first glance both failures seem to be related to culture, which was not subject of the fixes here (have a separate issue for that).

MonthCalendar_Handle_GetWithDefaultFirstDayOfWeek_Success

First day of week is a machine wide culture setting, expected value should either come from CultureInfo or the test should run with the default en-US culture without user customizations.

AxHost_GetIFontDispFromFont_InvokeSimpleStyle_Roundtrips
AxHost_GetIFontFromFont_InvokeSimpleStyle_Roundtrips

The iFont.Charset depends on the machines locale, since you said somewhere else you had utf8/cyrillic setup thats probably the cause for the failure. I don't know if that goes away by switching to en-US culture, might require making the test more flexible.

[RussKie]

Debugged into MonthCalendar_Handle_GetWithDefaultFirstDayOfWeek_Success failure:

There is a misalignment between values in Day and DayOfWeek enums, which causes incorrect expectations in my case.

[weltkante]

There is a misalignment between values in Day and DayOfWeek enums, which causes incorrect expectations in my case.

So what did SendMessage actually return? Interop code uses LOCALE_IFIRSTDAYOFWEEK for Day.Default, doc says it will return 0-6 in this case (the high word BOOL being zero).

Maybe this test failure detected an actual mismatch between the locale MonthCalendar uses and the CurrentCulture locale? Could be an actual bug, so lets not be hasty considering the exceptions incorrect.

(Note the +6 mod 7 in the test code is to correct the misalignment between Day and DayOfWeek so the code is already aware of the difference between the enums.)

[RussKie]

So what did SendMessage actually return?

If I make the following change than the test pass:

-            int expected = (int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek + 6;
+            int expected = (int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek; 

[weltkante]

If I make the following change than the test pass:

-            int expected = (int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek + 6;
+            int expected = (int)CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek; 

Definitely wrong, the docs of MCM_GETFIRSTDAYOFWEEK and the Day enum have different numeric values than FirstDayOfWeek. Changing it this way would break it for everyone else.

I think the key here is figuring out why your locale and CurrentCulture differ, might be a bug in the .NET Core runtime? Maybe check with Desktop Framework if it differs there as well?

Normally (on Desktop Framework) CurrentCulture has the Windows locale settings applied over it (it doesn't roundtrip, if you were to ask for en-AU you don't get the same culture back, the initial CurrentCulture used to be special in that way)

[RussKie]

I tested further, I changed my Windows' first day of the week to be "Saturday".
CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek hasn't changed and remained Sunday.
User32.SendMessageW(control.Handle, (User32.WM)ComCtl32.MCM.GETFIRSTDAYOFWEEK, IntPtr.Zero, IntPtr.Zero) however returned 5, which stands for Saturday in our Day enum:

winforms/src/System.Windows.Forms/src/System/Windows/Forms/Day.cs

Lines 37 to 40 in 100762a

	/// <summary>
	/// The day Saturday.
	/// </summary>
	Saturday = 5,

So I can infer two things here:

1. We shouldn't use CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek, and/or
2. CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek is wrong.

@tarekgh @safern do you think you could demystify the purpose of CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek for me? I expect it (perhaps naively) to align with my Windows settings.

[tarekgh]

@tarekgh @safern do you think you could demystify the purpose of CultureInfo.CurrentCulture.DateTimeFormat.FirstDayOfWeek for me? I expect it (perhaps naively) to align with my Windows settings.

We have a bug in .NET 5.0 which currently @safern working on fixing it. dotnet/runtime#37121

[RussKie]

Some more locale-specific tests

    System.Windows.Forms.Tests.Interop.Kernel32.FILETIMETests.FILETIME_ToDateTime_Invoke_ReturnsExpected [FAIL]
      Assert.Equal() Failure
      Expected: 2020-05-13T13:03:12.0000000
      Actual:   2020-05-13T23:03:12.0000000+10:00
      Stack Trace:
        C:\Development\winforms\src\System.Windows.Forms.Primitives\tests\UnitTests\Interop\Kernel32\FILETIMETests.cs(46,0): at System.Windows.Forms.Tests.Interop.Kernel32.FILETIMETests.FILETIME_ToDateTime_Invoke_ReturnsExpected()
    System.Windows.Forms.Tests.Interop.Kernel32.FILETIMETests.FILETIME_Ctor_DateTime [FAIL]
      Assert.Equal() Failure
      Expected: 3680495616
      Actual:   162781184
      Stack Trace:
        C:\Development\winforms\src\System.Windows.Forms.Primitives\tests\UnitTests\Interop\Kernel32\FILETIMETests.cs(34,0): at System.Windows.Forms.Tests.Interop.Kernel32.FILETIMETests.FILETIME_Ctor_DateTime()
    System.Windows.Forms.Tests.Interop.Oleaut32.VARIANTTests.VARIANT_ToObject_DateFromFILETIME_Success [FAIL]
      Assert.Equal() Failure
      Expected: 2020-05-13T13:03:12.0000000
      Actual:   2020-05-13T03:03:12.0000000
      Stack Trace:
        C:\Development\winforms\src\System.Windows.Forms.Primitives\tests\UnitTests\Interop\Oleaut32\VARIANTTests.cs(6034,0): at System.Windows.Forms.Tests.Interop.Oleaut32.VARIANTTests.<>c__DisplayClass299_0.<AssertToObjectEqual>b__0(Object actual)
        C:\Development\winforms\src\System.Windows.Forms.Primitives\tests\UnitTests\Interop\Oleaut32\VARIANTTests.cs(6049,0): at System.Windows.Forms.Tests.Interop.Oleaut32.VARIANTTests.AssertToObject(VARIANT variant, Action`1 action)
        C:\Development\winforms\src\System.Windows.Forms.Primitives\tests\UnitTests\Interop\Oleaut32\VARIANTTests.cs(6034,0): at System.Windows.Forms.Tests.Interop.Oleaut32.VARIANTTests.AssertToObjectEqual(Object expected, VARIANT variant)
        C:\Development\winforms\src\System.Windows.Forms.Primitives\tests\UnitTests\Interop\Oleaut32\VARIANTTests.cs(884,0): at System.Windows.Forms.Tests.Interop.Oleaut32.VARIANTTests.VARIANT_ToObject_DateFromFILETIME_Success()

[RussKie]

I think this issue has resolved. I'm no longer getting any locale specific failures.
Holler if you still do, and we can reopen it.