Longhaul: Tolerance adjustments

[and-rewsmith]

Overview

I split #5869 into two PRs. The first (#5902) has been merged. This is the second.

Currently longhaul pass results don't indicate the current release criteria. I am updating the tolerances for our tests in order to make it easier on the release manager to know what is releasable vs what requires further investigation. I have added tolerances for the broker-enabled case too, just so we know we aren't causing any big regressions.

A lot of these failures already have associated PBIs. For these cases I have linked the PBI/bugs below in the tolerance description section.

In order to do this there were these steps:

1. Logic specific to each test report type to determine pass result from report data (i.e. test events + test conditions).
2. Modified tests as needed. No new tests added as I was thinking it is fine to verify manually. The unit tests are super valuable for this test framework when dealing with report generators (i.e. the classes that have complex logic to deal with multiple queues of test events). But since I didn't touch those I didn't think it was worth the opportunity cost to add further testing.

I currently have longhaul and nested longhaul running and will wait to make sure the tolerances behave as expected before merge.

Tolerance Descriptions

Custom Mqtt Telemetry Tolerance
Overview:

• Sometimes we see message loss here so need a tolerance.

Tolerance description:

• [Nested-Edge] [Broker-Enabled]: We pass the tests as long as we get back 80% of messages. We sometimes have message loss, especially in middle->lower node telemetry.

PBI Required:

• No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses.

Notes:

• When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly.

IoTHub Message Tolerance
Overview:

• This only affects the broker. Sometimes we see message loss here so need a tolerance. When this problem occurs it affects < 10 messages.

Tolerance description:

• [Nested-Edge] [Broker-Enabled]: Pass the test if we receive 99% of messages.

PBI Required:

• No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses.

Notes:

• When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly.

Twin Test Tolerance
Overview:

• Sometimes we see some twin updates are missing in nested edge, but have still released in the past. This problem seems worse for non-broker vs broker-enabled, so it seems like a product issue we should followup on. I have only added a tolerance for the broker case so as not to mask this important issue.

Tolerance description:

• [Nested-Edge] [Broker-Enabled]: Two issues. 1) Missing some desired property updates coming through the sdk registered callback function. 2) Failing to update reported properties from the client. For both of these issues, we pass the tests if it happens less than 0.5% of the time.

PBI Required:

• No PBI needed to unwind this tolerance since it only affects the broker-enabled case. However it looks like there is a product issue for the non-broker case. I didn't add a tolerance here, as I think first we should do some investigation. It is worth running longhaul also on 1.2 to confirm behavior is the same. We logged a bug for a prior release here, but now the problem may be more severe. We will need logs to investigate further.

Direct Method Tolerance
Overview:

• A lot of these tolerances existed previously. For resource errors and transient errors, the tolerance seemed too high so I made the tests behave more strictly. For DeviceNotFound exceptions, we are looking at the test conditions to specify the exact tolerance.

Tolerance description:

• [All-Cases] Status code 0: Fail the tests if we get > 1 in 1000 direct methods with status code 0.
• [All-Cases] Resource error: Fail the tests if we get > 1 in 1000 direct methods with resource error.
• [All-Cases] Unauthorized: Fail the tests if we get > 1 in 1000 direct methods with resource error.
• [All-Cases] Transient error: Fail the tests if we get > 1 in 1000 direct methods with transient error.
• [All-Cases] NotImplemented: Fail the tests if we get > 1 in 1000 direct methods with NotImplemented error.
• [Nested-Edge] [Broker-Enabled] DeviceNotFound: Fail the tests if we get > 1 in 400 direct methods with DeviceNotFound
• [Nested-Edge] [Non-Broker] DeviceNotFound: Fail the tests if we get > 1 in 250 direct methods with DeviceNotFound
• [Single-Node] DeviceNotFound: Fail the tests if we get > 1 in 200 direct methods with DeviceNotFound

PBI Required:

• Yes, only for the fact that in nested edge, broker-enabled gets way less DeviceNotFound exceptions compared to non-broker. The bug we logged during a prior release exists here. We will need logs to investigate further.

Pipeline Confirmation

Here are the pipelines I have ran with these changes to confirm this will not regress any of our infra:

1. Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427019&view=results
2. Nested Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427258&view=results
3. Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431644&view=results
4. Nested Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431669&view=results
5. End to End: https://dev.azure.com/msazure/One/_build/results?buildId=49431614&view=results
6. Nested End-to-End : https://msazure.visualstudio.com/One/_build/results?buildId=49431628&view=results

Azure IoT Edge PR checklist:

This checklist is used to make sure that common guidelines for a pull request are followed.

General Guidelines and Best Practices

• I have read the contribution guidelines.
• Title of the pull request is clear and informative.
• Description of the pull request includes a concise summary of the enhancement or bug fix.

Testing Guidelines

• Pull request includes test coverage for the included changes.
• Description of the pull request includes
  • concise summary of tests added/modified
  • local testing done.

Draft PRs

• Open the PR in Draft mode if it is:
  • Work in progress or not intended to be merged.
  • Encountering multiple pipeline failures and working on fixes.

Note: We use the kodiakhq bot to merge PRs once the necessary checks and approvals are in place. When it merges a PR, kodiakhq converts the PR title to the commit title, PR description to the commit description, and squashes all the commits in the PR to a single commit. The net effect is that entire PR becomes a single commit. Please follow the best practices mentioned here for the PR title and description

[and-rewsmith]

I removed everything requiring a big tolerance, as the case that made me initially create this hasn't reproduced. But chose to leave it for now in case we need it later.

[pmzara]

Does this parse command always work? Maybe you can use TryParse here to guarantee that you don't get any exceptions.

[and-rewsmith]

I haven't seen this ever fail. Since I didn't change this in the PR and it isn't a problem, I'd rather leave it as-is.

[and-rewsmith]

To summarize the PBIs we need action on:

1. [Nested] [Non-Broker] Twin tests failing
2. [Nested] [Non-Broker] Excessive amount of DeviceNotFound exceptions

The behavior around 1 changed a lot across my multiple test runs, so I didn't add a tolerance. I think what we need to do is revive these bugs, and when we do the next 1.2 release followup with logs (or before if shiproom pushes out the release).

[and-rewsmith]

@varunpuranik Does this plan sound good?
#5916 (comment)