MFTF Process and Increase of Automated Test Coverage

Description of Process to Follow

1. HipTest is a source of truth for automated test coverage as well as regression testing.
2. HipTest scenarios should be extended for components with business cases which are not covered at the moment (this is the most challenging part).
3. All test cases in HipTest must be triaged by the Product Owner and the corresponding label which describes priority (S0/S1/S2/S3) should be assigned to them.
4. All the test cases which have S0/S1 label assigned is a matter of automation with MFTF.
5. For all the S0/S1 scenarios a corresponding GitHub issue should be created. All these issues should have "MFTF (Functional Test Coverage)" label assigned on GitHub.
6. Both HipTest scenario and a GitHub issue are interlinked so that HipTest scenario has a reference to GitHub issue and vice versa.
7. One GitHub issue corresponds to one and only HipTest scenario, so the relationship between HipTest and GitHub issues is 1-to-1.

It should look like:

GitHub issue

GitHub issue has a reference to HipTest scenario and the name of the issue on GitHub is identical to the use-case on HipTest.

HipTest Scenario

The Scenario on HipTest has linkage to GitHub as well

8. When a Pull Request created for a given GitHub issue and then the PR is merged we should add label "Automated" to the corresponding HipTest scenario.

Current Issues

Broken Linkage in GitHub issues referring to HipTest scenarios

It happened that some Folder IDs on HipTest have changed and that lead to the URLs pointing to HipTest scenarios got broken.

For example, let's look at the GitHub issue number: https://github.com/magento/inventory/issues/1360 It refers to HipTest scenario: https://app.hiptest.com/projects/69435/test-plan/folders/432042/scenarios/1446632 (following this URL client currently given an error 400).

What should be done to find matching test scenario:

1. Copy name of the Issue on Github. In this case: "Order with Virtual product on Single stock mode canceled from Admin"
2. Go to HipTest and make a search among all test scenarios by this name
3. Follow to the first result on the result page We see that link is broken because folder ID has changed - https://app.hiptest.com/projects/69435/test-plan/folders/735224/scenarios/1446632 So that we can update the link on GitHub now.

To verify that HipTest scenario exists on GitHub - the search on GitHub could be applied: