Longhaul: Plumbing and tolerance adjustments #5869

and-rewsmith · 2021-11-25T00:16:57Z

I split this into two PRs. The first (#5902) has been merged. The second is #5916 .

builds/e2e/connectivity.yaml

test/modules/TwinTester/StatusCode.cs

test/modules/TestResultCoordinator/Reports/LegacyTwin/LegacyTwinReportGenerator.cs

varunpuranik · 2021-12-02T14:46:46Z

The changes to plumb through additional information and flags are great.
But the key changes are to the test themselves and adjusting the tolerance - those might end up hiding underlying issues in the product, and need to be carefully considered. And if needed, we need to have PBIs to revert the changes after fixing underlying issues. Going forward, can we separate out these changes from the plumbing? It is hard to follow that in large PRs like these.
For this PR - can you please add details on what changes you made, related to tolerance adjustments?

…lh-tolerances

and-rewsmith · 2021-12-07T02:29:00Z

@varunpuranik Didn't expect as many changes from the plumbing as there were. I agree they are distracting. It was easy enough to isolate the plumbing into its own diff, so I have this draft PR here:
#5902

I'm waiting on test pass before opening #5902 up for review. Then we can handle the tolerance specific stuff, which I've cleaned up and documented in this PR separately. Converting this PR to draft.

…lh-tolerances

and-rewsmith · 2021-12-08T19:51:22Z

Closing in favor of #5916 for cleanup.

## Overview I split #5869 into two PRs. The first (#5902) has been merged. This is the second. Currently longhaul pass results don't indicate the current release criteria. I am updating the tolerances for our tests in order to make it easier on the release manager to know what is releasable vs what requires further investigation. I have added tolerances for the broker-enabled case too, just so we know we aren't causing any big regressions. A lot of these failures already have associated PBIs. For these cases I have linked the PBI/bugs below in the tolerance description section. In order to do this there were these steps: 1. Logic specific to each test report type to determine pass result from report data (i.e. test events + test conditions). 2. Modified tests as needed. No new tests added as I was thinking it is fine to verify manually. The unit tests are super valuable for this test framework when dealing with report generators (i.e. the classes that have complex logic to deal with multiple queues of test events). But since I didn't touch those I didn't think it was worth the opportunity cost to add further testing. I currently have longhaul and nested longhaul running and will wait to make sure the tolerances behave as expected before merge. ## Tolerance Descriptions **Custom Mqtt Telemetry Tolerance** Overview: - Sometimes we see message loss here so need a tolerance. Tolerance description: - [Nested-Edge] [Broker-Enabled]: We pass the tests as long as we get back 80% of messages. We sometimes have message loss, especially in middle->lower node telemetry. PBI Required: - No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses. Notes: - When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly. **IoTHub Message Tolerance** Overview: - This only affects the broker. Sometimes we see message loss here so need a tolerance. When this problem occurs it affects < 10 messages. Tolerance description: - [Nested-Edge] [Broker-Enabled]: Pass the test if we receive 99% of messages. PBI Required: - No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses. Notes: - When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly. **Twin Test Tolerance** Overview: - Sometimes we see some twin updates are missing in nested edge, but have still released in the past. This problem seems worse for non-broker vs broker-enabled, so it seems like a product issue we should followup on. I have only added a tolerance for the broker case so as not to mask this important issue. Tolerance description: - [Nested-Edge] [Broker-Enabled]: Two issues. 1) Missing some desired property updates coming through the sdk registered callback function. 2) Failing to update reported properties from the client. For both of these issues, we pass the tests if it happens less than 0.5% of the time. PBI Required: - No PBI needed to unwind this tolerance since it only affects the broker-enabled case. However it looks like there is a product issue for the non-broker case. I didn't add a tolerance here, as I think first we should do some investigation. It is worth running longhaul also on 1.2 to confirm behavior is the same. We logged a bug for a prior release [here](https://msazure.visualstudio.com/One/_workitems/edit/10039969), but now the problem may be more severe. We will need logs to investigate further. **Direct Method Tolerance** Overview: - A lot of these tolerances existed previously. For resource errors and transient errors, the tolerance seemed too high so I made the tests behave more strictly. For DeviceNotFound exceptions, we are looking at the test conditions to specify the exact tolerance. Tolerance description: - [All-Cases] Status code 0: Fail the tests if we get > 1 in 1000 direct methods with status code 0. - [All-Cases] Resource error: Fail the tests if we get > 1 in 1000 direct methods with resource error. - [All-Cases] Unauthorized: Fail the tests if we get > 1 in 1000 direct methods with resource error. - [All-Cases] Transient error: Fail the tests if we get > 1 in 1000 direct methods with transient error. - [All-Cases] NotImplemented: Fail the tests if we get > 1 in 1000 direct methods with NotImplemented error. - [Nested-Edge] [Broker-Enabled] DeviceNotFound: Fail the tests if we get > 1 in 400 direct methods with DeviceNotFound - [Nested-Edge] [Non-Broker] DeviceNotFound: Fail the tests if we get > 1 in 250 direct methods with DeviceNotFound - [Single-Node] DeviceNotFound: Fail the tests if we get > 1 in 200 direct methods with DeviceNotFound PBI Required: - Yes, only for the fact that in nested edge, broker-enabled gets way less DeviceNotFound exceptions compared to non-broker. The bug we logged during a prior release exists [here](https://msazure.visualstudio.com/One/_workitems/edit/10149346). We will need logs to investigate further. ## Pipeline Confirmation Here are the pipelines I have ran with these changes to confirm this will not regress any of our infra: 1. Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427019&view=results 2. Nested Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427258&view=results 3. Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431644&view=results 4. Nested Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431669&view=results 5. End to End: https://dev.azure.com/msazure/One/_build/results?buildId=49431614&view=results 6. Nested End-to-End : https://msazure.visualstudio.com/One/_build/results?buildId=49431628&view=results ## Azure IoT Edge PR checklist:

## Overview I split Azure#5869 into two PRs. The first (Azure#5902) has been merged. This is the second. Currently longhaul pass results don't indicate the current release criteria. I am updating the tolerances for our tests in order to make it easier on the release manager to know what is releasable vs what requires further investigation. I have added tolerances for the broker-enabled case too, just so we know we aren't causing any big regressions. A lot of these failures already have associated PBIs. For these cases I have linked the PBI/bugs below in the tolerance description section. In order to do this there were these steps: 1. Logic specific to each test report type to determine pass result from report data (i.e. test events + test conditions). 2. Modified tests as needed. No new tests added as I was thinking it is fine to verify manually. The unit tests are super valuable for this test framework when dealing with report generators (i.e. the classes that have complex logic to deal with multiple queues of test events). But since I didn't touch those I didn't think it was worth the opportunity cost to add further testing. I currently have longhaul and nested longhaul running and will wait to make sure the tolerances behave as expected before merge. ## Tolerance Descriptions **Custom Mqtt Telemetry Tolerance** Overview: - Sometimes we see message loss here so need a tolerance. Tolerance description: - [Nested-Edge] [Broker-Enabled]: We pass the tests as long as we get back 80% of messages. We sometimes have message loss, especially in middle->lower node telemetry. PBI Required: - No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses. Notes: - When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly. **IoTHub Message Tolerance** Overview: - This only affects the broker. Sometimes we see message loss here so need a tolerance. When this problem occurs it affects < 10 messages. Tolerance description: - [Nested-Edge] [Broker-Enabled]: Pass the test if we receive 99% of messages. PBI Required: - No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses. Notes: - When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly. **Twin Test Tolerance** Overview: - Sometimes we see some twin updates are missing in nested edge, but have still released in the past. This problem seems worse for non-broker vs broker-enabled, so it seems like a product issue we should followup on. I have only added a tolerance for the broker case so as not to mask this important issue. Tolerance description: - [Nested-Edge] [Broker-Enabled]: Two issues. 1) Missing some desired property updates coming through the sdk registered callback function. 2) Failing to update reported properties from the client. For both of these issues, we pass the tests if it happens less than 0.5% of the time. PBI Required: - No PBI needed to unwind this tolerance since it only affects the broker-enabled case. However it looks like there is a product issue for the non-broker case. I didn't add a tolerance here, as I think first we should do some investigation. It is worth running longhaul also on 1.2 to confirm behavior is the same. We logged a bug for a prior release [here](https://msazure.visualstudio.com/One/_workitems/edit/10039969), but now the problem may be more severe. We will need logs to investigate further. **Direct Method Tolerance** Overview: - A lot of these tolerances existed previously. For resource errors and transient errors, the tolerance seemed too high so I made the tests behave more strictly. For DeviceNotFound exceptions, we are looking at the test conditions to specify the exact tolerance. Tolerance description: - [All-Cases] Status code 0: Fail the tests if we get > 1 in 1000 direct methods with status code 0. - [All-Cases] Resource error: Fail the tests if we get > 1 in 1000 direct methods with resource error. - [All-Cases] Unauthorized: Fail the tests if we get > 1 in 1000 direct methods with resource error. - [All-Cases] Transient error: Fail the tests if we get > 1 in 1000 direct methods with transient error. - [All-Cases] NotImplemented: Fail the tests if we get > 1 in 1000 direct methods with NotImplemented error. - [Nested-Edge] [Broker-Enabled] DeviceNotFound: Fail the tests if we get > 1 in 400 direct methods with DeviceNotFound - [Nested-Edge] [Non-Broker] DeviceNotFound: Fail the tests if we get > 1 in 250 direct methods with DeviceNotFound - [Single-Node] DeviceNotFound: Fail the tests if we get > 1 in 200 direct methods with DeviceNotFound PBI Required: - Yes, only for the fact that in nested edge, broker-enabled gets way less DeviceNotFound exceptions compared to non-broker. The bug we logged during a prior release exists [here](https://msazure.visualstudio.com/One/_workitems/edit/10149346). We will need logs to investigate further. ## Pipeline Confirmation Here are the pipelines I have ran with these changes to confirm this will not regress any of our infra: 1. Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427019&view=results 2. Nested Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427258&view=results 3. Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431644&view=results 4. Nested Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431669&view=results 5. End to End: https://dev.azure.com/msazure/One/_build/results?buildId=49431614&view=results 6. Nested End-to-End : https://msazure.visualstudio.com/One/_build/results?buildId=49431628&view=results ## Azure IoT Edge PR checklist:

## Overview Cherry pick #5902 #5916. Tested via nested longhaul pipeline. I split #5869 into two PRs. The first (#5902) has been merged. This is the second. Currently longhaul pass results don't indicate the current release criteria. I am updating the tolerances for our tests in order to make it easier on the release manager to know what is releasable vs what requires further investigation. I have added tolerances for the broker-enabled case too, just so we know we aren't causing any big regressions. A lot of these failures already have associated PBIs. For these cases I have linked the PBI/bugs below in the tolerance description section. In order to do this there were these steps: 1. Logic specific to each test report type to determine pass result from report data (i.e. test events + test conditions). 2. Modified tests as needed. No new tests added as I was thinking it is fine to verify manually. The unit tests are super valuable for this test framework when dealing with report generators (i.e. the classes that have complex logic to deal with multiple queues of test events). But since I didn't touch those I didn't think it was worth the opportunity cost to add further testing. I currently have longhaul and nested longhaul running and will wait to make sure the tolerances behave as expected before merge. ## Tolerance Descriptions **Custom Mqtt Telemetry Tolerance** Overview: - Sometimes we see message loss here so need a tolerance. Tolerance description: - [Nested-Edge] [Broker-Enabled]: We pass the tests as long as we get back 80% of messages. We sometimes have message loss, especially in middle->lower node telemetry. PBI Required: - No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses. Notes: - When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly. **IoTHub Message Tolerance** Overview: - This only affects the broker. Sometimes we see message loss here so need a tolerance. When this problem occurs it affects < 10 messages. Tolerance description: - [Nested-Edge] [Broker-Enabled]: Pass the test if we receive 99% of messages. PBI Required: - No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses. Notes: - When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly. **Twin Test Tolerance** Overview: - Sometimes we see some twin updates are missing in nested edge, but have still released in the past. This problem seems worse for non-broker vs broker-enabled, so it seems like a product issue we should followup on. I have only added a tolerance for the broker case so as not to mask this important issue. Tolerance description: - [Nested-Edge] [Broker-Enabled]: Two issues. 1) Missing some desired property updates coming through the sdk registered callback function. 2) Failing to update reported properties from the client. For both of these issues, we pass the tests if it happens less than 0.5% of the time. PBI Required: - No PBI needed to unwind this tolerance since it only affects the broker-enabled case. However it looks like there is a product issue for the non-broker case. I didn't add a tolerance here, as I think first we should do some investigation. It is worth running longhaul also on 1.2 to confirm behavior is the same. We logged a bug for a prior release [here](https://msazure.visualstudio.com/One/_workitems/edit/10039969), but now the problem may be more severe. We will need logs to investigate further. **Direct Method Tolerance** Overview: - A lot of these tolerances existed previously. For resource errors and transient errors, the tolerance seemed too high so I made the tests behave more strictly. For DeviceNotFound exceptions, we are looking at the test conditions to specify the exact tolerance. Tolerance description: - [All-Cases] Status code 0: Fail the tests if we get > 1 in 1000 direct methods with status code 0. - [All-Cases] Resource error: Fail the tests if we get > 1 in 1000 direct methods with resource error. - [All-Cases] Unauthorized: Fail the tests if we get > 1 in 1000 direct methods with resource error. - [All-Cases] Transient error: Fail the tests if we get > 1 in 1000 direct methods with transient error. - [All-Cases] NotImplemented: Fail the tests if we get > 1 in 1000 direct methods with NotImplemented error. - [Nested-Edge] [Broker-Enabled] DeviceNotFound: Fail the tests if we get > 1 in 400 direct methods with DeviceNotFound - [Nested-Edge] [Non-Broker] DeviceNotFound: Fail the tests if we get > 1 in 250 direct methods with DeviceNotFound - [Single-Node] DeviceNotFound: Fail the tests if we get > 1 in 200 direct methods with DeviceNotFound PBI Required: - Yes, only for the fact that in nested edge, broker-enabled gets way less DeviceNotFound exceptions compared to non-broker. The bug we logged during a prior release exists [here](https://msazure.visualstudio.com/One/_workitems/edit/10149346). We will need logs to investigate further. ## Pipeline Confirmation Here are the pipelines I have ran with these changes to confirm this will not regress any of our infra: 1. Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427019&view=results 2. Nested Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427258&view=results 3. Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431644&view=results 4. Nested Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431669&view=results 5. End to End: https://dev.azure.com/msazure/One/_build/results?buildId=49431614&view=results 6. Nested End-to-End : https://msazure.visualstudio.com/One/_build/results?buildId=49431628&view=results ## Azure IoT Edge PR checklist: ***Please replace this line with your PR description and read PR checklist below*** ## Azure IoT Edge PR checklist: This checklist is used to make sure that common guidelines for a pull request are followed. ### General Guidelines and Best Practices - [x] I have read the [contribution guidelines](https://github.com/azure/iotedge#contributing). - [x] Title of the pull request is clear and informative. - [x] Description of the pull request includes a concise summary of the enhancement or bug fix. ### Testing Guidelines - [x] Pull request includes test coverage for the included changes. - Description of the pull request includes - [x] concise summary of tests added/modified - [x] local testing done. ### Draft PRs - Open the PR in `Draft` mode if it is: - Work in progress or not intended to be merged. - Encountering multiple pipeline failures and working on fixes. _Note: We use the kodiakhq bot to merge PRs once the necessary checks and approvals are in place. When it merges a PR, kodiakhq converts the PR title to the commit title, PR description to the commit description, and squashes all the commits in the PR to a single commit. The net effect is that entire PR becomes a single commit. Please follow the best practices mentioned [here](https://chris.beams.io/posts/git-commit/#:~:text=The%20seven%20rules%20of%20a%20great%20Git%20commit,what%20and%20why%20vs.%20how%20For%20example%3A%20) for the PR title and description_

## Overview I split Azure#5869 into two PRs. The first (Azure#5902) has been merged. This is the second. Currently longhaul pass results don't indicate the current release criteria. I am updating the tolerances for our tests in order to make it easier on the release manager to know what is releasable vs what requires further investigation. I have added tolerances for the broker-enabled case too, just so we know we aren't causing any big regressions. A lot of these failures already have associated PBIs. For these cases I have linked the PBI/bugs below in the tolerance description section. In order to do this there were these steps: 1. Logic specific to each test report type to determine pass result from report data (i.e. test events + test conditions). 2. Modified tests as needed. No new tests added as I was thinking it is fine to verify manually. The unit tests are super valuable for this test framework when dealing with report generators (i.e. the classes that have complex logic to deal with multiple queues of test events). But since I didn't touch those I didn't think it was worth the opportunity cost to add further testing. I currently have longhaul and nested longhaul running and will wait to make sure the tolerances behave as expected before merge. ## Tolerance Descriptions **Custom Mqtt Telemetry Tolerance** Overview: - Sometimes we see message loss here so need a tolerance. Tolerance description: - [Nested-Edge] [Broker-Enabled]: We pass the tests as long as we get back 80% of messages. We sometimes have message loss, especially in middle->lower node telemetry. PBI Required: - No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses. Notes: - When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly. **IoTHub Message Tolerance** Overview: - This only affects the broker. Sometimes we see message loss here so need a tolerance. When this problem occurs it affects < 10 messages. Tolerance description: - [Nested-Edge] [Broker-Enabled]: Pass the test if we receive 99% of messages. PBI Required: - No. We are rewriting the current version of the broker so not worth fixing, but the tolerance is good just to make sure nothing regresses. Notes: - When the new broker is running off of master branch, then this tolerance should be removed so we assure the new broker functions properly. **Twin Test Tolerance** Overview: - Sometimes we see some twin updates are missing in nested edge, but have still released in the past. This problem seems worse for non-broker vs broker-enabled, so it seems like a product issue we should followup on. I have only added a tolerance for the broker case so as not to mask this important issue. Tolerance description: - [Nested-Edge] [Broker-Enabled]: Two issues. 1) Missing some desired property updates coming through the sdk registered callback function. 2) Failing to update reported properties from the client. For both of these issues, we pass the tests if it happens less than 0.5% of the time. PBI Required: - No PBI needed to unwind this tolerance since it only affects the broker-enabled case. However it looks like there is a product issue for the non-broker case. I didn't add a tolerance here, as I think first we should do some investigation. It is worth running longhaul also on 1.2 to confirm behavior is the same. We logged a bug for a prior release [here](https://msazure.visualstudio.com/One/_workitems/edit/10039969), but now the problem may be more severe. We will need logs to investigate further. **Direct Method Tolerance** Overview: - A lot of these tolerances existed previously. For resource errors and transient errors, the tolerance seemed too high so I made the tests behave more strictly. For DeviceNotFound exceptions, we are looking at the test conditions to specify the exact tolerance. Tolerance description: - [All-Cases] Status code 0: Fail the tests if we get > 1 in 1000 direct methods with status code 0. - [All-Cases] Resource error: Fail the tests if we get > 1 in 1000 direct methods with resource error. - [All-Cases] Unauthorized: Fail the tests if we get > 1 in 1000 direct methods with resource error. - [All-Cases] Transient error: Fail the tests if we get > 1 in 1000 direct methods with transient error. - [All-Cases] NotImplemented: Fail the tests if we get > 1 in 1000 direct methods with NotImplemented error. - [Nested-Edge] [Broker-Enabled] DeviceNotFound: Fail the tests if we get > 1 in 400 direct methods with DeviceNotFound - [Nested-Edge] [Non-Broker] DeviceNotFound: Fail the tests if we get > 1 in 250 direct methods with DeviceNotFound - [Single-Node] DeviceNotFound: Fail the tests if we get > 1 in 200 direct methods with DeviceNotFound PBI Required: - Yes, only for the fact that in nested edge, broker-enabled gets way less DeviceNotFound exceptions compared to non-broker. The bug we logged during a prior release exists [here](https://msazure.visualstudio.com/One/_workitems/edit/10149346). We will need logs to investigate further. ## Pipeline Confirmation Here are the pipelines I have ran with these changes to confirm this will not regress any of our infra: 1. Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427019&view=results 2. Nested Longhaul: https://msazure.visualstudio.com/One/_build/results?buildId=49427258&view=results 3. Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431644&view=results 4. Nested Connectivity: https://msazure.visualstudio.com/One/_build/results?buildId=49431669&view=results 5. End to End: https://dev.azure.com/msazure/One/_build/results?buildId=49431614&view=results 6. Nested End-to-End : https://msazure.visualstudio.com/One/_build/results?buildId=49431628&view=results ## Azure IoT Edge PR checklist:

squash dev commits

33e61ea

and-rewsmith force-pushed the andsmi/master-lh-tolerances branch from 2e3a012 to 33e61ea Compare November 30, 2021 18:32

and-rewsmith commented Nov 30, 2021

View reviewed changes

builds/e2e/connectivity.yaml Outdated Show resolved Hide resolved

and-rewsmith commented Nov 30, 2021

View reviewed changes

test/modules/TwinTester/StatusCode.cs Outdated Show resolved Hide resolved

and-rewsmith marked this pull request as ready for review November 30, 2021 18:35

and-rewsmith requested review from nimanch and pmzara November 30, 2021 18:36

nimanch reviewed Nov 30, 2021

View reviewed changes

test/modules/TestResultCoordinator/Reports/LegacyTwin/LegacyTwinReportGenerator.cs Outdated Show resolved Hide resolved

and-rewsmith added 8 commits December 2, 2021 21:37

fix error in twin report generation

eea62fd

adjust single node dm tolerance

f6083f1

Merge branch 'master' of github.com:Azure/iotedge into andsmi/master-…

553df67

…lh-tolerances

adjust direct method tolerance for single node

e8fbe23

broaden tolerance for messages in nested broker case

68a3eeb

tolerance for notimpl

d8c6c50

bump devicenotfound tolerance for single node

f1f6ab0

adjust tolerances and self review

cadfd3d

and-rewsmith marked this pull request as draft December 7, 2021 02:29

Merge branch 'master' of github.com:Azure/iotedge into andsmi/master-…

fbde9fc

…lh-tolerances

and-rewsmith marked this pull request as ready for review December 8, 2021 19:39

and-rewsmith mentioned this pull request Dec 8, 2021

Longhaul: Tolerance adjustments #5916

Merged

6 tasks

and-rewsmith changed the title ~~Longhaul: Tolerance adjustments~~ Longhaul: Plumbing and tolerance adjustments Dec 8, 2021

and-rewsmith closed this Dec 8, 2021

and-rewsmith mentioned this pull request Jan 27, 2022

Longhaul: Tolerance adjustments (#5902) (#5916) #6052

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Longhaul: Plumbing and tolerance adjustments #5869

Longhaul: Plumbing and tolerance adjustments #5869

and-rewsmith commented Nov 25, 2021 •

edited

Loading

varunpuranik commented Dec 2, 2021 •

edited

Loading

and-rewsmith commented Dec 7, 2021

and-rewsmith commented Dec 8, 2021

Longhaul: Plumbing and tolerance adjustments #5869

Longhaul: Plumbing and tolerance adjustments #5869

Conversation

and-rewsmith commented Nov 25, 2021 • edited Loading

varunpuranik commented Dec 2, 2021 • edited Loading

and-rewsmith commented Dec 7, 2021

and-rewsmith commented Dec 8, 2021

and-rewsmith commented Nov 25, 2021 •

edited

Loading

varunpuranik commented Dec 2, 2021 •

edited

Loading