Reduce greentea socket tests failures related to network issues #10330
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Netsocket greentea tests are meant to test Socket API. However they highly depend on the quality of the underlying network connection, especially when it comes to using the echo server.
I tried to identify the most common and fixable failures, which seem to result from the underlying infrastructure rather than the socket operation itself. These failures obscure the overall view of the CI results and might in result hide some actual errors.
The first improvement is in UDPSOCKET_RECV_TIMEOUT. The test was checking that at least 5 out of 10 UDP packets were received from the echo server. I do not see how this is related to socket timeout. We have separate tests UDPSOCKET_ECHO* for UDP sockets to test echo responses and that test can handle WOULD_BLOCK responses in a more reasonable way and will also check if the data in echo responses match.
I suggest we remove the packet success ratio expectation from this test to avoid random failures due to network glitches.
The second potential improvement is in TLSSOCKET_RECV_TIMEOUT. Here we sometimes fail to receive a packet within 20 seconds, especially in the tests using wifi, which we know is rather unreliable. There is a separate mechanism in this test, which guarantees that we will not exceed half of the total time for the test suite, so increasing the sigio timeout is safe and might reduce the number of failures due to network issues.
Pull request type
Reviewers
@SeppoTakalo
@VeijoPesonen
@KariHaapalehto
@mtomczykmobica
@tymoteuszblochmobica