Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update test plans to include template-id instead of regex (infra) #1127

Closed
wants to merge 3 commits into from

Conversation

tang-mm
Copy link
Collaborator

@tang-mm tang-mm commented Mar 29, 2024

Description

  • replace most regex that are used in test plans' include fields with a corresponding template-id

Pending issues:

  • regex that includes only a subset of instantiated jobs are not updated, for example:
    • indexed jobs "audio/1_playback_thunderbolt_.*
    • wireless/wifi_ap_wpa_b_with_sta_.*_manual

Resolved issues

Resolves CHECKBOX-1295

Documentation

Tests

Copy link

codecov bot commented Mar 29, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 43.05%. Comparing base (4c58a14) to head (a16c90c).

Additional details and impacted files
@@           Coverage Diff           @@
##             main    #1127   +/-   ##
=======================================
  Coverage   43.05%   43.05%           
=======================================
  Files         355      355           
  Lines       38575    38575           
  Branches     6549     6549           
=======================================
  Hits        16607    16607           
  Misses      21302    21302           
  Partials      666      666           
Flag Coverage Δ
provider-base 15.94% <ø> (ø)
provider-certification-client 57.14% <ø> (ø)
provider-certification-server 57.14% <ø> (ø)
provider-sru 100.00% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@tang-mm tang-mm changed the title Infra: update test plans to include template-id instead of regex update test plans to include template-id instead of regex (infra) Mar 29, 2024
@tang-mm
Copy link
Collaborator Author

tang-mm commented Apr 2, 2024

Pending for discussion:
If a template is flagged with also-after-suspend, by referencing the template-id in a test plan, all generated jobs (including before- and after-suspend jobs) will be included in the test plan. The replacement of regex may need to be coupled with the retirement of after-suspend test plans.

@tang-mm tang-mm marked this pull request as ready for review April 8, 2024 07:50
wireless/wifi_ap_open_b_no_sta_interface_manual
wireless/wifi_ap_open_g_no_sta_interface_manual
wireless/wifi_ap_wpa_b_no_sta_interface_manual
wireless/wifi_ap_wpa_g_no_sta_interface_manual
wireless/wifi_ap_wpa_b_with_sta_.*_manual
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Need review: regex in this and the following line might not match any test jobs

@@ -300,7 +300,7 @@ unit: test plan
_name: Automated iperf3 tests
_description: Automated iperf3 performance test
include:
ethernet/iperf3_.*
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The original regex matches not only iperf3_interface but also subsets with longer segments, such as iperf3_reverse_interface. This results in missing jobs after switching to template-id.
Regex like this one need to be matched with more than one template

@tang-mm
Copy link
Collaborator Author

tang-mm commented Apr 16, 2024

Actions needed for this PR following the discussion:

  • break down the changes into smaller PRs for easier review (such as changes by test category)
  • compare and validate the changes in each test plan

@tang-mm tang-mm closed this Apr 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant