Flaky System.Windows.Forms.UITests.ComboBoxTests.ComboBox_Select_Item_By_UpArrowKeyAsync

[runfoapp]

Runfo Tracking Issue: Flaky System.Windows.Forms.UITests.ComboBoxTests.ComboBox_Select_Item_By_UpArrowKeyAsync

Build	Definition	Kind	Run Name
1833634	dotnet-winforms CI	Rolling	Windows_x64-xunit
1833634	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1830774	dotnet-winforms CI	PR 7322	Windows_x64-xunit
1830434	dotnet-winforms CI	PR 7314	Windows_x64-xunit
1830426	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1830407	dotnet-winforms CI	Rolling	Windows_x86-xunit
1830374	dotnet-winforms CI	PR 7299	Windows_arm64-xunit
1830351	dotnet-winforms CI	PR 7256	Windows_x64-xunit
1830343	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1830343	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1830312	dotnet-winforms CI	Rolling	Windows_x86-xunit
1830305	dotnet-winforms CI	PR 7311	Windows_x64-xunit
1830210	dotnet-winforms CI	PR 7311	Windows_x64-xunit
1830207	dotnet-winforms CI	PR 7309	Windows_x86-xunit
1829407	dotnet-winforms CI	PR 7319	Windows_x64-xunit
1829234	dotnet-winforms CI	PR 7309	Windows_x86-xunit
1829234	dotnet-winforms CI	PR 7309	Windows_arm64-xunit
1829231	dotnet-winforms CI	PR 7311	Windows_x64-xunit
1829231	dotnet-winforms CI	PR 7311	Windows_x64-xunit
1829151	dotnet-winforms CI	PR 7316	Windows_x86-xunit
1829079	dotnet-winforms CI	PR 7309	Windows_x86-xunit
1828511	dotnet-winforms CI	PR 7312	Windows_x86-xunit
1828476	dotnet-winforms CI	PR 7298	Windows_x86-xunit
1828469	dotnet-winforms CI	PR 7309	Windows_x86-xunit
1828451	dotnet-winforms CI	PR 7310	Windows_x86-xunit
1828451	dotnet-winforms CI	PR 7310	Windows_arm64-xunit
1828451	dotnet-winforms CI	PR 7310	Windows_x64-xunit
1827818	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1827786	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1827432	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1827262	dotnet-winforms CI	PR 7306	Windows_x64-xunit
1826701	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1826701	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1826351	dotnet-winforms CI	PR 6638	Windows_arm64-xunit
1826351	dotnet-winforms CI	PR 6638	Windows_x86-xunit
1826193	dotnet-winforms CI	PR 7256	Windows_x86-xunit
1826117	dotnet-winforms CI	PR 7256	Windows_arm64-xunit
1826117	dotnet-winforms CI	PR 7256	Windows_x64-xunit
1825896	dotnet-winforms CI	PR 7304	Windows_x64-xunit
1825765	dotnet-winforms CI	PR 7242	Windows_arm64-xunit
1825483	dotnet-winforms CI	Rolling	Windows_x86-xunit
1825444	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1824228	dotnet-winforms CI	Rolling	Windows_arm64-xunit
1823515	dotnet-winforms CI	PR 7300	Windows_arm64-xunit
1823515	dotnet-winforms CI	PR 7300	Windows_x64-xunit
1823301	dotnet-winforms CI	PR 7290	Windows_x64-xunit
1823150	dotnet-winforms CI	PR 7294	Windows_arm64-xunit
1823150	dotnet-winforms CI	PR 7294	Windows_x64-xunit
1822922	dotnet-winforms CI	Rolling	Windows_x64-xunit

Build Result Summary

Day Hit Count	Week Hit Count	Month Hit Count
0	33	38

[RussKie]

I was able to replicate the failures locally, and the most likely reason for the failures is the loss of focus by the test form.

E.g., here's a script to automate the issue (make sure to have built the repo, i.e., run build.cmd):

pushd <path>\dotnet-winforms

$testProject = '.\src\System.Windows.Forms\tests\IntegrationTests\UIIntegrationTests\System.Windows.Forms.UI.IntegrationTests.csproj';
$testFilter = 'ComboBox_Select_Item_By_UpArrowKeyAsync';

try {
    for ($num = 0 ; $num -le 1 ; $num++) {
        dotnet test $testProject --nologo --no-build --no-restore --filter $testFilter
        if ($LastExitCode -ne 0) {
            exit -1;
        }
    }
}
finally {
    popd
}

While the test is running, bring other windows to the foreground (i.e., click on those). As a result, the test will fail:

[xUnit.net 00:00:03.94]     System.Windows.Forms.UITests.ComboBoxTests.ComboBox_Select_Item_By_UpArrowKeyAsync [FAIL]
  Failed System.Windows.Forms.UITests.ComboBoxTests.ComboBox_Select_Item_By_UpArrowKeyAsync [3 s]
  Error Message:
   Assert.Equal() Failure
Expected: 9
Actual:   8

  Standard Output Messages:
 i:1, eventsCount:1
 i:2, eventsCount:1
 i:3, eventsCount:2
 i:4, eventsCount:3
 i:5, eventsCount:4
 i:6, eventsCount:5
 i:7, eventsCount:6
 i:8, eventsCount:7
 i:9, eventsCount:8

[dkazennov]

@RussKie so test will be compromised if user moves the mouse and clicks it's buttons?

There are two integration tests with InputSimulator:
ComboBox_Select_Item_By_DownArrowKeyAsync and ComboBox_Select_Item_By_UpArrowKeyAsync.

There are also two unitTests: ComboBox_SelectedIndex_DoesNotChange_SelectedItem_AndDoesNotFire_OnSelectedIndexChanged and ComboBox_SelectedIndexChangesSelectedItemAndFiresOnSelectedIndexChanged

These 4 tests are very similar (they count change selection events).

I propose we recreate these two integration tests so it is not affected by user's actions (without InputSimulator). Turn them into unit tests. One possible way it seems is the unit test ListView_LeftRightArrow_DoesNotThrowException

[RussKie]

I've been staring (and not only) at ComboBox_Select_Item_By_DownArrowKeyAsync and ComboBox_Select_Item_By_UpArrowKeyAsync, and whilst they are similar, they don't behave the same...

winforms/src/System.Windows.Forms/tests/IntegrationTests/UIIntegrationTests/ComboBoxTests.cs

Lines 40 to 68 in 03f373f

	[WinFormsFact]
	public async Task ComboBox_Select_Item_By_UpArrowKeyAsync()
	{
	int expectedKeyPressesCount = 9;
	await RunComboBoxTestAsync(async (form, comboBox) =>
	{
	StringBuilder stringBuilder = new();
	// Reset EventsCount because the control gets selected right after the creation.
	comboBox.ResetEventsCount();
	for (int i = 0; i < expectedKeyPressesCount; i++)
	{
	await Task.Delay(100);
	await InputSimulator.SendAsync(
	form,
	inputSimulator => inputSimulator.Keyboard
	.Sleep(100)
	.KeyPress(WindowsInput.Native.VirtualKeyCode.UP));
	await Task.Delay(100);
	stringBuilder.AppendLine($"i:{i + 1}, eventsCount:{comboBox.EventsCount}");
	}
	TestOutputHelper.WriteLine($"{stringBuilder}");
	Assert.Equal(expectedKeyPressesCount, comboBox.EventsCount);
	},
	expectedKeyPressesCount,
	selectedIndex: 9);
	}

winforms/src/System.Windows.Forms/tests/IntegrationTests/UIIntegrationTests/ComboBoxTests.cs

Lines 70 to 91 in 03f373f

	[WinFormsFact]
	public async Task ComboBox_Select_Item_By_DownArrowKeyAsync()
	{
	int expectedKeyPressesCount = 9;
	await RunComboBoxTestAsync(async (form, comboBox) =>
	{
	// Reset EventsCount because the control gets selected right after the creation.
	comboBox.ResetEventsCount();
	for (int i = 0; i < expectedKeyPressesCount; i++)
	{
	await InputSimulator.SendAsync(
	form,
	inputSimulator => inputSimulator.Keyboard.KeyPress(WindowsInput.Native.VirtualKeyCode.DOWN));
	Assert.Equal(i + 1, comboBox.EventsCount);
	}
	Assert.Equal(expectedKeyPressesCount, comboBox.EventsCount);
	},
	expectedKeyPressesCount,
	selectedIndex: 0);
	}

Spot the difference 😉

I can't recall ComboBox_Select_Item_By_DownArrowKeyAsync failing, whilst the other one fails quite often, see https://runfo.azurewebsites.net/search/tests/?q=started%3A%7E60+definition%3A267+kind%3Arolling#build.
Perhaps start with updating ``ComboBox_Select_Item_By_UpArrowKeyAsync` to resemble it's counterpart by removing the "waits"?

[dkazennov]

@RussKie in fact I've already created a new unit test which tests ComboBox keyboard focus without input simulation and Async methods before I've seen your comment here.

I'll make the PR with both ways to solve this Issue, but I believe the test I've created is come convenient than previous ones.

Upd. #7323