Add AXE testing to test suite #3764

marcelwgn · 2020-12-04T11:31:08Z

Description

This PR enables AXE testing allowing us to essentially run Accessibility Insights on the controls as part of the CI.

Because of that we need two package changes:

Add Axe.Windows to the list of available Nuget packages (using 1.0.4 right now)
Upgrade Microsoft.Win32.Registry to the required version (4.7.0 would be required for 1.0.4)

Motivation and Context

Allows detecting of a11y issues during development using the a11y detection engine of Accessibility Insights

How Has This Been Tested?

Ran tests in question.

Screenshots (if appropriate):

@YuliKl @ranjesh @StephenLPeters FYI

marcelwgn · 2020-12-04T11:32:27Z

dev/NumberBox/TestUI/NumberBoxAxeTestPage.xaml

+        <!-- These ALL fail our axe scanning so we comment them out. -->
+        <!--
+            <muxc:NumberBox Header="Sample NumberBox"/>
+            <muxc:NumberBox AutomationProperties.Name="Sample NumberBox"/>
+            <muxc:NumberBox Header="Sample NumberBox" SpinButtonPlacementMode="Inline"/>
+            <muxc:NumberBox Header="Sample NumberBox" SpinButtonPlacementMode="Compact"/>
+        -->


NumberBox doesn't forward header or AutomationProperties.Name to the internal textbox, resulting in a missing UIA name for the internl "edit". @YuliKl @teaP FYI

I think this already has an issue... #3505

In reply to: 536034266 [](ancestors = 536034266)

Oh right, forgot about that issue. I'll remove the code then.

marcelwgn · 2020-12-04T11:33:35Z

test/TestAppUtils/Themes/Generic.xaml

@@ -39,7 +39,8 @@
                                <StackPanel Orientation="Horizontal">
                                    <Button x:Name="BackButton" Margin="0,0,4,0"
                                        Style="{ThemeResource AccentButtonStyle}"
-                                        AutomationProperties.AutomationId="__BackButton" 
+                                        AutomationProperties.AutomationId="__BackButton"
+                                        AutomationProperties.Name="Back"


Needed to add that since otherwise, Axe would always return at least one a11y issue.

dev/NumberBox/TestUI/NumberBoxAxeTestPage.xaml.cs

test/MUXControls.Test/AxeTestHelper.cs

docs/developer_guide.md

dev/TabView/TestUI/TabViewAxeTestPage.xaml

dev/TabView/TestUI/TabViewAxeTestPage.xaml.cs

test/MUXControls.Test/AxeTestHelper.cs

StephenLPeters · 2020-12-04T21:59:32Z

/azp run

azure-pipelines · 2020-12-04T21:59:48Z

Azure Pipelines successfully started running 1 pipeline(s).

StephenLPeters · 2020-12-04T23:38:58Z

/azp run

azure-pipelines · 2020-12-04T23:39:14Z

Azure Pipelines successfully started running 1 pipeline(s).

marcelwgn · 2020-12-05T00:06:45Z

@StephenLPeters CI failed since the required NuGet packages are not available in the WinUI dependency repository.

StephenLPeters · 2020-12-05T00:17:17Z

@kmahone Can you remind us the process of getting the nuget package into the dependencies?

kmahone · 2020-12-05T00:19:49Z

@kmahone Can you remind us the process of getting the nuget package into the dependencies?

nuget push mypackage.nupkg -Source https://pkgs.dev.azure.com/ms/microsoft-ui-xaml/_packaging/MUX-Dependencies/nuget/v3/index.json -ApiKey az

marcelwgn · 2021-01-31T08:55:19Z

@StephenLPeters @ranjeshj Is there anything on my side that needs to be changed for this PR?

ranjeshj · 2021-02-05T16:13:41Z

@chingucoding Apologies on the delay. Can you sync with master ? There have been quite a few changes since this PR.

marcelwgn · 2021-02-05T16:24:57Z

Merged with master now.

azure-pipelines · 2021-12-07T22:43:09Z

Azure Pipelines successfully started running 1 pipeline(s).

karkarl · 2021-12-09T19:18:08Z

/azp run

azure-pipelines · 2021-12-09T19:18:21Z

Azure Pipelines successfully started running 1 pipeline(s).

marcelwgn · 2021-12-09T21:58:44Z

Looks like the tests failed for a non build problem, I'll take a look tomorrow and see if and how I can fix it :)

marcelwgn · 2021-12-09T21:58:54Z

Thank you for all your help @karenbtlai !

karkarl · 2021-12-10T02:05:42Z

/azp run

azure-pipelines · 2021-12-10T02:05:54Z

Azure Pipelines successfully started running 1 pipeline(s).

marcelwgn · 2021-12-11T21:41:36Z

@karenbtlai Do you think that the code to retrieve the process ID could be broken with TAEF, i.e. if could retrieve the process ID of a process with the same name spawned by TAEF (and MSTest tests happen to work)?

karkarl · 2021-12-13T18:44:06Z

/azp run

azure-pipelines · 2021-12-13T18:44:18Z

Azure Pipelines successfully started running 1 pipeline(s).

karkarl · 2021-12-13T19:06:45Z

/azp run

azure-pipelines · 2021-12-13T19:06:56Z

Azure Pipelines successfully started running 1 pipeline(s).

karkarl · 2021-12-13T21:39:54Z

/azp run

azure-pipelines · 2021-12-13T21:40:06Z

Azure Pipelines successfully started running 1 pipeline(s).

karkarl · 2021-12-14T00:55:24Z

/azp run

azure-pipelines · 2021-12-14T00:55:35Z

Azure Pipelines successfully started running 1 pipeline(s).

karkarl · 2021-12-14T01:39:44Z

/azp run

azure-pipelines · 2021-12-14T01:39:56Z

Azure Pipelines successfully started running 1 pipeline(s).

marcelwgn added 3 commits December 4, 2020 10:06

Initial commit

ce4019b

Add docs, add more controls

c84f82a

Merge branch 'master' into user/chingucoding/axe-testing

0279222

ghost added the needs-triage Issue needs to be triaged by the area owners label Dec 4, 2020

marcelwgn commented Dec 4, 2020

View reviewed changes

Felix-Dev reviewed Dec 4, 2020

View reviewed changes

dev/NumberBox/TestUI/NumberBoxAxeTestPage.xaml.cs Outdated Show resolved Hide resolved

Felix-Dev reviewed Dec 4, 2020

View reviewed changes

test/MUXControls.Test/AxeTestHelper.cs Outdated Show resolved Hide resolved

Felix-Dev reviewed Dec 4, 2020

View reviewed changes

docs/developer_guide.md Outdated Show resolved Hide resolved

Felix-Dev reviewed Dec 4, 2020

View reviewed changes

dev/TabView/TestUI/TabViewAxeTestPage.xaml Outdated Show resolved Hide resolved

Felix-Dev reviewed Dec 4, 2020

View reviewed changes

dev/TabView/TestUI/TabViewAxeTestPage.xaml.cs Outdated Show resolved Hide resolved

Felix-Dev reviewed Dec 4, 2020

View reviewed changes

test/MUXControls.Test/AxeTestHelper.cs Outdated Show resolved Hide resolved

CR feedback 1

1440e88

StephenLPeters reviewed Dec 4, 2020

View reviewed changes

test/MUXControls.Test/AxeTestHelper.cs Show resolved Hide resolved

Small fixes

aba92ca

StephenLPeters added area-TestInfrastructure Issue in the test infrastructure (e.g. in Helix scripts) team-Controls Issue for the Controls team and removed needs-triage Issue needs to be triaged by the area owners labels Dec 5, 2020

ranjeshj requested a review from kmahone February 5, 2021 16:12

Merge branch 'master' into user/chingucoding/axe-testing

915b6b7

fix formatting

9f1c0f8

karkarl mentioned this pull request Dec 9, 2021

[BUG] ERROR Automation017: Fail to get the root element of a process(11548) microsoft/axe-windows#662

Open

karkarl added 3 commits December 9, 2021 15:45

add error message and output .a11yTest file

9fdec83

fix failing axe test

9055b9b

add missing dlls

7c106ae

karkarl force-pushed the user/chingucoding/axe-testing branch from d084ee1 to 7c106ae Compare December 13, 2021 21:39

add missing dll

7b14a2a

temp fix for axe test failures

3e857af

karkarl mentioned this pull request Dec 14, 2021

[Accessibility] AutoSuggestBox and NumberBox PlaceholderText fails Accessibility Insights Pass #6480

Closed

2 tasks

kmahone approved these changes Dec 15, 2021

View reviewed changes

karkarl merged commit fa28e62 into microsoft:main Dec 15, 2021

marcelwgn deleted the user/chingucoding/axe-testing branch January 19, 2022 23:44

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AXE testing to test suite #3764

Add AXE testing to test suite #3764

marcelwgn commented Dec 4, 2020

marcelwgn Dec 4, 2020

StephenLPeters Dec 4, 2020

marcelwgn Dec 4, 2020

marcelwgn Dec 4, 2020

StephenLPeters commented Dec 4, 2020

azure-pipelines bot commented Dec 4, 2020

StephenLPeters commented Dec 4, 2020

azure-pipelines bot commented Dec 4, 2020

marcelwgn commented Dec 5, 2020

StephenLPeters commented Dec 5, 2020

kmahone commented Dec 5, 2020

marcelwgn commented Jan 31, 2021

ranjeshj commented Feb 5, 2021

marcelwgn commented Feb 5, 2021

azure-pipelines bot commented Dec 7, 2021

karkarl commented Dec 9, 2021

azure-pipelines bot commented Dec 9, 2021

marcelwgn commented Dec 9, 2021

marcelwgn commented Dec 9, 2021

karkarl commented Dec 10, 2021

azure-pipelines bot commented Dec 10, 2021

marcelwgn commented Dec 11, 2021

karkarl commented Dec 13, 2021

azure-pipelines bot commented Dec 13, 2021

karkarl commented Dec 13, 2021

azure-pipelines bot commented Dec 13, 2021

karkarl commented Dec 13, 2021

azure-pipelines bot commented Dec 13, 2021

karkarl commented Dec 14, 2021

azure-pipelines bot commented Dec 14, 2021

karkarl commented Dec 14, 2021

azure-pipelines bot commented Dec 14, 2021

Add AXE testing to test suite #3764

Add AXE testing to test suite #3764

Conversation

marcelwgn commented Dec 4, 2020

Description

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate):

marcelwgn Dec 4, 2020

Choose a reason for hiding this comment

StephenLPeters Dec 4, 2020

Choose a reason for hiding this comment

marcelwgn Dec 4, 2020

Choose a reason for hiding this comment

marcelwgn Dec 4, 2020

Choose a reason for hiding this comment

StephenLPeters commented Dec 4, 2020

azure-pipelines bot commented Dec 4, 2020

StephenLPeters commented Dec 4, 2020

azure-pipelines bot commented Dec 4, 2020

marcelwgn commented Dec 5, 2020

StephenLPeters commented Dec 5, 2020

kmahone commented Dec 5, 2020

marcelwgn commented Jan 31, 2021

ranjeshj commented Feb 5, 2021

marcelwgn commented Feb 5, 2021

azure-pipelines bot commented Dec 7, 2021

karkarl commented Dec 9, 2021

azure-pipelines bot commented Dec 9, 2021

marcelwgn commented Dec 9, 2021

marcelwgn commented Dec 9, 2021

karkarl commented Dec 10, 2021

azure-pipelines bot commented Dec 10, 2021

marcelwgn commented Dec 11, 2021

karkarl commented Dec 13, 2021

azure-pipelines bot commented Dec 13, 2021

karkarl commented Dec 13, 2021

azure-pipelines bot commented Dec 13, 2021

karkarl commented Dec 13, 2021

azure-pipelines bot commented Dec 13, 2021

karkarl commented Dec 14, 2021

azure-pipelines bot commented Dec 14, 2021

karkarl commented Dec 14, 2021

azure-pipelines bot commented Dec 14, 2021