Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce greentea socket tests failures related to network issues #10330

Merged
merged 2 commits into from
Apr 9, 2019

Conversation

michalpasztamobica
Copy link
Contributor

Description

Netsocket greentea tests are meant to test Socket API. However they highly depend on the quality of the underlying network connection, especially when it comes to using the echo server.
I tried to identify the most common and fixable failures, which seem to result from the underlying infrastructure rather than the socket operation itself. These failures obscure the overall view of the CI results and might in result hide some actual errors.

The first improvement is in UDPSOCKET_RECV_TIMEOUT. The test was checking that at least 5 out of 10 UDP packets were received from the echo server. I do not see how this is related to socket timeout. We have separate tests UDPSOCKET_ECHO* for UDP sockets to test echo responses and that test can handle WOULD_BLOCK responses in a more reasonable way and will also check if the data in echo responses match.
I suggest we remove the packet success ratio expectation from this test to avoid random failures due to network glitches.

The second potential improvement is in TLSSOCKET_RECV_TIMEOUT. Here we sometimes fail to receive a packet within 20 seconds, especially in the tests using wifi, which we know is rather unreliable. There is a separate mechanism in this test, which guarantees that we will not exceed half of the total time for the test suite, so increasing the sigio timeout is safe and might reduce the number of failures due to network issues.

Pull request type

[ ] Fix
[ ] Refactor
[ ] Target update
[ ] Functionality change
[ ] Docs update
[x] Test update
[ ] Breaking change

Reviewers

@SeppoTakalo
@VeijoPesonen
@KariHaapalehto
@mtomczykmobica
@tymoteuszblochmobica

@ciarmcom
Copy link
Member

ciarmcom commented Apr 5, 2019

@michalpasztamobica, thank you for your changes.
@tymoteuszblochmobica @VeijoPesonen @SeppoTakalo @KariHaapalehto @mtomczykmobica @ARMmbed/mbed-os-ipcore @ARMmbed/mbed-os-test @ARMmbed/mbed-os-maintainers please review.

@@ -859,9 +859,6 @@ Within each loop, one `recvfrom()` may return the received packet size
When `NSAPI_ERROR_WOULD_BLOCK` is received, check that time consumed is
more that 100 milliseconds but less than 200 milliseconds.

After repeating for 10 times, at least 5 packets must have been
received.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

50 % packet loss is quite a big.
If there is 100 % packet loss, and we accept it, how is this going to test anything?

Copy link
Contributor

@SeppoTakalo SeppoTakalo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can approve, but if we drop the number of packets away, we might also drop the number of loops to two at least.
Would run much faster and still test what was the idea of the test case.

TESTS/netsocket/udp/udpsocket_recv_timeout.cpp Outdated Show resolved Hide resolved
@0xc0170
Copy link
Contributor

0xc0170 commented Apr 8, 2019

Ci started

@mbed-ci
Copy link

mbed-ci commented Apr 8, 2019

Test run: FAILED

Summary: 1 of 6 test jobs failed
Build number : 1
Build artifacts

Failed test jobs:

  • jenkins-ci/mbed-os-ci_greentea-test

@cmonr
Copy link
Contributor

cmonr commented Apr 8, 2019

CI job restarted: jenkins-ci/greentea-test

K66F issues don't appear to be related to PR.

@0xc0170 0xc0170 merged commit af3a765 into ARMmbed:master Apr 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants