Extend the wwan connection tests to support multiple cycle connection tests (New)

[stanley31huang]

Description

Extend the wwan connection tests to support multiple cycle connection tests, we found an issue that the LTE connection could not be established at second round.

As discussed internally with the QA team, we have all agreed that the default number of test runs for WWAN connections tests is 2.

Resolved issues

Here is the support ticket: 00379830.

Our SWE has identified this issue is related to the modem firmware and modem-manager snap, and they are trying to add a patch to modem-manager snap as a workaround.

Documentation

N/A

Tests

https://certification.canonical.com/hardware/202302-31253/submission/386810/test-results/

[codecov]

Codecov Report

Attention: Patch coverage is 92.85714% with 14 lines in your changes missing coverage. Please review.

Project coverage is 47.75%. Comparing base (2d42b1e) to head (cf8e538).
Report is 15 commits behind head on main.

Files with missing lines	Patch %	Lines
providers/base/bin/reboot_check_test.py	92.15%	8 Missing and 4 partials ⚠️
checkbox-ng/checkbox_ng/launcher/subcommands.py	88.88%	1 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1309      +/-   ##
==========================================
+ Coverage   47.55%   47.75%   +0.19%     
==========================================
  Files         369      370       +1     
  Lines       39585    39739     +154     
  Branches     6685     6622      -63     
==========================================
+ Hits        18824    18976     +152     
- Misses      20050    20058       +8     
+ Partials      711      705       -6     

Flag	Coverage Δ
checkbox-ng	68.09% <88.88%> (-0.04%)	⬇️
provider-base	24.78% <93.25%> (+0.73%)	⬆️
provider-sru	97.97% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[pieqq]

Thanks for this contribution! This is a good idea to try to connect WWAN multiple times, this will for sure help with regression testing.

A few things:

Default number of iterations

The default number of iterations is something that should be discussed with other QA teams, especially PC QA. With your proposal, if the WWAN_CONNECTION_TEST_COUNT environment variable is not set, there will only be one connection test. Maybe PC QA and other IoT QA projects would benefit from having several connection attempts?

What if connection fails at first attempt?

Should we still try X times, adding X failures to the test report?

Splitting connection jobs Vs. connecting multiple times in the same job

Is it important to have multiple jobs (resulting in several lines in the test report), or could the number of iterations provided to the wwan_tests.py script, and the connection would be attempted X times instead?

Code duplication

I think you could combine the new wwan_resource_with_iterations.py with the existing wwan_tests.py. The latter already provides the functionality to print out the resource; you could just add the --iterations argument to the wwan_tests.py script and use the same mechanism you developed in wwan_resource_with_iterations.py (repeat the resource info for each iteration).

By doing this, you

• prevent code duplication
• reduce the number of scripts to maintain
• reduce the number of additional tests

Since you only want the connection test to be run multiple times, and not the other tests based on wwan_resource, an easy fix is to add

template-filter: wwan_resource.iteration == '1'

to these jobs. They will only be run for the first iteration.

I've tried it locally and it seems to work; here are the modifications I made to the WWAN jobs using wwan_resource:

diff --git a/providers/base/units/wwan/jobs.pxu b/providers/base/units/wwan/jobs.pxu
index 99cf4fb45..07d9fe901 100644
--- a/providers/base/units/wwan/jobs.pxu
+++ b/providers/base/units/wwan/jobs.pxu
@@ -27,7 +27,7 @@ requires:
   snap.name == 'modem-manager' or package.name == 'modemmanager'
 
 unit: template
-template-resource: wwan_resource_with_iteration
+template-resource: wwan_resource
 template-unit: job
 id: wwan/gsm-connection-{manufacturer}-{model}-{hw_id}-cycle{iteration}-auto
 template-id: wwan/gsm-connection-manufacturer-model-hw_id-auto
@@ -59,6 +59,7 @@ requires:
 unit: template
 template-resource: wwan_resource
 template-unit: job
+template-filter: wwan_resource.iteration == '1'
 id: wwan/check-sim-present-{manufacturer}-{model}-{hw_id}-auto
 template-id: wwan/check-sim-present-manufacturer-model-hw_id-auto
 _summary:  Check if a SIM card is present in a slot connected to the modem
@@ -79,6 +80,7 @@ requires:
 unit: template
 template-resource: wwan_resource
 template-unit: job
+template-filter: wwan_resource.iteration == '1'
 id: wwan/verify-sim-info-{manufacturer}-{model}-{hw_id}
 template-id: wwan/verify-sim-info-manufacturer-model-hw_id
 depends: wwan/check-sim-present-{manufacturer}-{model}-{hw_id}-auto

[stanley31huang]

@pieqq
Thanks for reviewing this PR and the suggestions. I like the idea to integrate the iteration into the wwan_resource job, to avoid the code duplication.

For the connections fail at the first attempt, I think it still worth to launch the rest of the connection tests, because of we could understand the failure rate is 100% or not. But it would be better to add a dependency in this job which is make sure the SIM card has been detected.

For the Splitting connection jobs Vs. connecting multiple times in the same job, I personally prefer the way to splitting connections jobs, because of it's easily to know the overall test results, no need to go through the details from the output of job.

[stanley31huang]

Converted this PR to draft, due to I am going to re-base this branch after another PR has been merged.
#1323

[pieqq]

Following today's Office Hours meeting, we discussed with QA and Checkbox team regarding this PR.

If the goal is "just" to run the jobs twice, then maybe we could just use a sibling, and add that new job to the test plan.

[stanley31huang]

close this one, due to the changes are bit messy. Use #1534 instead.