lwip - Add more network tests #3265

geky · 2016-11-15T01:56:21Z

Adds more tests over network operations. Since these tests are currently dependent on the availability of a network interface, these live in FEATURE_LWIP.

Also cleaned up the existing network tests and removed the problematic nist test.

Available network tests:

connectivity - Tests connecting/disconnecting the network
tcp-hello-world - Simple http test using developer.mbed.org, also tests DNS resolution
tcp-echo - Tests simple TCP echo with host_test server
udp-echo - Tests simple UDP echo with host_test server, dropped packets are retried
tcp-packet-pressure - Heavy TCP test with an exponentially growing random sequence, provides runtime measurements
udp-packet-pressure - Heavy UDP test with an exponentially growing random sequence, dropped packets are retried with exponential backoff, provides runtime measurements
udp-dtls-handshake - Test of problematic UDP traffic present in DTLS handshake (5x ~300 byte packets simultaneously)
tcp-echo-parallel - Multithreaded TCP echo test with host_test server
udp-echo-parallel - Multithreaded UDP echo test with host_test server, dropped packets are retried
tcp-packet-pressure-parallel - Multithreaded TCP test with exponentially growing random sequence, provides runtime measurements
udp-packet-pressure-parallel - Multithreaded UDP test with exponentially growing random sequence, dropped packets are retried with exponential backoff, provides runtime measurements

Note: Not all tests may be passing due to issues with underlying drivers. This pr is for initial review, but needs additional work before merging.

cc @bridadan, @bulislaw, @c1728p9

geky · 2016-11-15T01:56:44Z

/morph test-nightly

mbed-bot · 2016-11-15T02:21:57Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test-nightly

Output

mbed Build Number: 1080

Build failed!

bulislaw · 2016-11-15T11:11:17Z

features/FEATURE_LWIP/TESTS/mbedmicro-net/tcp_packet_pressure/main.cpp

+int main() {
+    GREENTEA_SETUP(60, "tcp_echo");
+
+    EthernetInterface eth;


I think we shouldn't make it Ethernet specific, WiFi would use this kind of testing as well -> or any other IP network that we may have to be fair.
I think what we should do after we unify Ethernet and WiFi to use EMAC, scrape the DEVICE_EMAC flag and introduce DEVICE_WIFI and DEVICE_ETHERNET and run the test for each present interface, which shouldn't be difficult as only the connect function would be different.

I agree, this test being ethernet specific is only a makeshift solution until we have a consistent way to grab network interfaces.

We could duplicate these tests for now between the tests for WiFi and Ethernet, or do you think these tests should adopt the #ifdef checks for each interface?

It's all very theoretical right now, we don't test any WiFi. Depending when we are thinking of hooking Odin in the farm it may make sense to just hold of with implementing them for wifi till we have unified networking in place.

jupe · 2016-11-15T16:32:23Z

what about cellular, 6lowpan, thread, .. are you going to support those as well?

bulislaw · 2016-11-15T16:39:03Z

Not knowing anything about them (or the APIs), as long as their implement NetworkInterface they should be supported. That should be pretty much 'for free' if we get them supported in the 'built in network interface' class that would provide built in network interfaces.

geky · 2016-11-15T16:52:51Z

We don't currently have the testing infrastructure in place to choose interfaces with different names in a scalable manner.

We are planning on sorting out how to access the network without knowing the underlying network interface, but this is currently a work in progress and won't be ready for 5.3 (This API update will be slow since we want to avoid API churn and keep any updates transparent before being landed).

The goal of this pr is to get testing for at least one network interface type strongly covered and make sure the ethernet drivers are stable across all devices. Once we have a way of accessing the network generically, these tests will be moved up to cover all interfaces.

Unfortunately, with the current setup, the best way to have these tests on all devices is to duplicate them for each "FEATURE". I think that duplication may be better as a separate pr to keep this pr's scope limited (there is likely to be many bug fixes attached to this pr).

(unrelated morph test to avoid message spam)
/morph test-nightly

mbed-bot · 2016-11-15T17:25:39Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test-nightly

Output

mbed Build Number: 1084

Build failed!

geky · 2016-11-15T17:28:19Z

/morph test-nightly

mbed-bot · 2016-11-15T18:05:44Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test-nightly

Output

mbed Build Number: 1085

Build failed!

geky · 2016-11-15T18:24:48Z

/morph test-nightly

mbed-bot · 2016-11-16T00:13:36Z

Result: SUCCESS

Your command has finished executing! Here's what you wrote!

/morph test-nightly

Output

mbed Build Number: 1088

All builds and test passed!

0xc0170 · 2016-11-16T17:02:22Z

@geky Is this still in progress? Let us know once completed.

geky · 2016-11-16T18:17:46Z

@0xc0170, This is still in progress. Successfully passing all the tests was unexpected and a result of a restricted buffer being used across all devices. Working on a more flexible solution to scale the buffers per device.

I am still expecting board failures, so there's still quite a bit of work to do.

geky · 2016-11-17T00:18:00Z

Don't mind me, just giving the test bot stuff to do
/morph test-nightly

mbed-bot · 2016-11-17T07:04:29Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test-nightly

Output

mbed Build Number: 1111

Build failed!

geky · 2016-11-17T19:37:48Z

/morph test-nightly

mbed-bot · 2016-11-17T19:58:29Z

Result: ABORTED

Your command has finished executing! Here's what you wrote!

/morph test-nightly

Output

mbed Build Number: 1117

Build Prep failed!

geky · 2016-11-17T20:52:32Z

Local test results for K64F with GCC_ARM (This is the only board currently connected to network testing):

+--------------+---------------+------------------------------------------------------------------------+---------+--------------------+-------------+
| target       | platform_name | test suite                                                             | result  | elapsed_time (sec) | copy_method |
+--------------+---------------+------------------------------------------------------------------------+---------+--------------------+-------------+
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-connectivity                 | OK      | 25.99              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-gethostbyname                | OK      | 20.81              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-tcp_echo                     | OK      | 20.53              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-tcp_echo_parallel            | OK      | 20.93              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-tcp_hello_world              | OK      | 18.57              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-tcp_packet_pressure          | TIMEOUT | 76.44              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-tcp_packet_pressure_parallel | TIMEOUT | 136.61             | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-udp_dtls_handshake           | OK      | 19.45              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-udp_echo                     | OK      | 18.27              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-udp_echo_parallel            | OK      | 21.59              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-udp_packet_pressure          | OK      | 23.57              | shell       |
| K64F-GCC_ARM | K64F          | features-feature_lwip-tests-mbedmicro-net-udp_packet_pressure_parallel | OK      | 62.48              | shell       |
+--------------+---------------+------------------------------------------------------------------------+---------+--------------------+-------------+

SeppoTakalo · 2016-11-21T11:25:59Z

@mikaleppanen Are you aware of these?

We are working towards providing more IPv6 capable testcases also. Please synch efforts. Seems that these are using green-tea, and we are using clitest.

geky · 2016-12-19T18:06:29Z

/morph test-nightly

mbed-bot · 2016-12-19T19:39:33Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test-nightly

bridadan · 2016-12-19T20:03:22Z

Sorry for the lack explanation on that failure, found a bug in the CI script.

It was a build error for LPC1768 devices (LPC1768, ARCH_PRO, and UBLOX_C027), the test features-feature_lwip-tests-mbedmicro-net-udp_echo_parallel doesn't fit on the device:

13:30:48 Build failures:
13:30:48   * ARCH_PRO::IAR::FEATURES-FEATURE_LWIP-TESTS-MBEDMICRO-NET-UDP_ECHO_PARALLEL
13:30:48         Building project udp_echo_parallel (ARCH_PRO, IAR)
13:30:48         Scan: IAR
13:30:48         Scan: FEATURE_LWIP
13:30:48         Scan: udp_echo_parallel
13:30:48         Compile [100.0%]: main.cpp
13:30:48         [Warning] mbed_config.h@24,0: [Pe047]: incompatible redefinition of macro "UNITY_INCLUDE_CONFIG_H"
13:30:48         Link: udp_echo_parallel
13:30:48         Error[Lp011]: section placement failed
13:30:48                     unable to allocate space for sections/blocks with a total estimated
13:30:48                               minimum size of 0x7f8c bytes (max align 0x8) in
13:30:48                               <[0x100000c8-0x10007fdf]> (total uncommitted space
13:30:48                               0x7f18).

These tests could adopt the dynamically sized buffers used for the packet-pressure tests, however throughput is not an important feature of these tests.

Despite being able to buffer an arbitrary stream of data, TCP send is still limited by the available buffer space in the network stack. Errors from TCP send are perfectly reasonable and should be handled by reducing the buffer that is attempted.

geky · 2016-12-19T22:14:58Z

@bridadan, thanks! These tests have already gotten worrisomely small, but I've decreased the buffer size again to see if it will fit.

/morph test-nightly

mbed-bot · 2016-12-20T00:34:34Z

Result: FAILURE

Your command has finished executing! Here's what you wrote!

/morph test-nightly

Output

mbed Build Number: 1296

Test failed!

geky · 2016-12-20T00:41:30Z

	Test Result (21 failures / +21)

    UBLOX_EVK_ODIN_W2-ARM.features-feature_lwip-tests-mbedmicro-net-tcp_echo_parallel.features-feature_lwip-tests-mbedmicro-net-tcp_echo_parallel
    UBLOX_EVK_ODIN_W2-ARM.features-feature_lwip-tests-mbedmicro-net-connectivity.Testing bringing the network up and down
    UBLOX_EVK_ODIN_W2-ARM.features-feature_lwip-tests-mbedmicro-net-udp_dtls_handshake.features-feature_lwip-tests-mbedmicro-net-udp_dtls_handshake
    UBLOX_EVK_ODIN_W2-ARM.features-feature_lwip-tests-mbedmicro-net-udp_echo_parallel.features-feature_lwip-tests-mbedmicro-net-udp_echo_parallel
    UBLOX_EVK_ODIN_W2-ARM.features-feature_lwip-tests-mbedmicro-net-tcp_packet_pressure.features-feature_lwip-tests-mbedmicro-net-tcp_packet_pressure
    UBLOX_EVK_ODIN_W2-ARM.features-feature_lwip-tests-mbedmicro-net-udp_packet_pressure.features-feature_lwip-tests-mbedmicro-net-udp_packet_pressure
    NCS36510-IAR.tests-events-queue.Testing call_every
    K64F-ARM.features-feature_lwip-tests-mbedmicro-net-tcp_packet_pressure.features-feature_lwip-tests-mbedmicro-net-tcp_packet_pressure
    K64F-ARM.features-feature_lwip-tests-mbedmicro-net-udp_packet_pressure.features-feature_lwip-tests-mbedmicro-net-udp_packet_pressure
    UBLOX_EVK_ODIN_W2-IAR.features-feature_lwip-tests-mbedmicro-net-connectivity.Testing bringing the network up and down

UBLOX_EVK_ODIN_W2 - Looks like this is due to lwip being consumed by the wifi. I can add a ifdef on DEVICE_EMAC until Unify network interfaces #3269 is resolved.
NCS36510 - Sporadic semaphore timing issue on NCS36510 #3404
K64F - Not sure why this is only failing on ARM (line). I'll look into it locally.

*_packet_pressure_parallel tests are useful for checking for synchronization errors, but push the practical limitations of the network stack. Failing these tests is not unreasonable. *_packet_pressure tests are a little bit less unreasonable, but also push the practical limitations of the network stack. Hopefully these will become stable in the near future.

Currently DEVICE_EMAC is incompatible with the EthernetInterface

geky · 2017-01-05T17:18:14Z

/morph test-nightly

mbed-bot · 2017-01-05T19:30:21Z

Result: SUCCESS

Your command has finished executing! Here's what you wrote!

/morph test-nightly

Output

mbed Build Number: 1358

All builds and test passed!

geky · 2017-01-05T20:36:23Z

Better late than never, LGTM!

bridadan · 2017-01-05T20:53:18Z

Oops, sorry for the label craziness!

MarceloSalazar · 2017-01-11T17:17:34Z

Testing this on different K64F and multiple OSes (Windows, Mac), I see the case 'features-feature_lwip-tests-mbedmicro-net-udp_dtls_handshake' always fails (others are ok).

The result of the above CI tests seems to be fine, but are we aware of any instabilities around UDP/DTLS handshake?

If this requires further discussion, I'll happy to create an issue.

geky added the needs: work label Nov 15, 2016

bulislaw reviewed Nov 15, 2016

View reviewed changes

geky force-pushed the lwip-tests branch from 8ca45fe to f7ac183 Compare November 15, 2016 16:33

geky force-pushed the lwip-tests branch from f7ac183 to 1c09cec Compare November 15, 2016 17:26

geky force-pushed the lwip-tests branch from 1c09cec to 3678774 Compare November 15, 2016 18:24

geky force-pushed the lwip-tests branch 2 times, most recently from a5e63ca to 4db7183 Compare November 16, 2016 22:30

geky force-pushed the lwip-tests branch from c594c68 to b0e2721 Compare November 17, 2016 17:45

geky force-pushed the lwip-tests branch from 94ebf49 to 59aebf9 Compare December 7, 2016 17:22

This was referenced Dec 7, 2016

K66F: Enable LWIP feature #3378

Merged

nsapi - Fixed open/close issue in Socket #3402

Merged

geky force-pushed the lwip-tests branch from 59aebf9 to 69029be Compare December 8, 2016 22:26

geky force-pushed the lwip-tests branch from a0c9a07 to 7bf6d67 Compare December 19, 2016 18:06

geky force-pushed the lwip-tests branch from 7bf6d67 to ecf1ea0 Compare December 19, 2016 22:10

geky added 2 commits December 19, 2016 16:11

lwip - Reduced size of buffers in echo-parallel tests

2da71a7

These tests could adopt the dynamically sized buffers used for the packet-pressure tests, however throughput is not an important feature of these tests.

geky force-pushed the lwip-tests branch from ecf1ea0 to 17ce2cc Compare December 19, 2016 22:12

geky added the needs: work label Dec 20, 2016

geky added 2 commits January 5, 2017 10:58

lwip - Filtered network tests based on DEVICE_EMAC

bbcf8de

Currently DEVICE_EMAC is incompatible with the EthernetInterface

geky force-pushed the lwip-tests branch from 17ce2cc to bbcf8de Compare January 5, 2017 17:17

geky removed the needs: work label Jan 5, 2017

geky added ready for merge and removed needs: CI labels Jan 5, 2017

bridadan added needs: review ready for merge and removed ready for merge needs: review labels Jan 5, 2017

sg- merged commit 6af7b29 into ARMmbed:master Jan 9, 2017

sg- added release-version: 5.4.0 and removed ready for merge labels Jan 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

lwip - Add more network tests #3265

lwip - Add more network tests #3265

geky commented Nov 15, 2016 •

edited

Loading

geky commented Nov 15, 2016

mbed-bot commented Nov 15, 2016

bulislaw Nov 15, 2016

geky Nov 15, 2016

bulislaw Nov 15, 2016

jupe commented Nov 15, 2016

bulislaw commented Nov 15, 2016

geky commented Nov 15, 2016 •

edited

Loading

mbed-bot commented Nov 15, 2016

geky commented Nov 15, 2016

mbed-bot commented Nov 15, 2016

geky commented Nov 15, 2016

mbed-bot commented Nov 16, 2016

0xc0170 commented Nov 16, 2016

geky commented Nov 16, 2016

geky commented Nov 17, 2016

mbed-bot commented Nov 17, 2016

geky commented Nov 17, 2016

mbed-bot commented Nov 17, 2016

geky commented Nov 17, 2016

SeppoTakalo commented Nov 21, 2016

geky commented Dec 19, 2016

mbed-bot commented Dec 19, 2016

bridadan commented Dec 19, 2016 •

edited

Loading

geky commented Dec 19, 2016

mbed-bot commented Dec 20, 2016

geky commented Dec 20, 2016 •

edited

Loading

geky commented Jan 5, 2017

mbed-bot commented Jan 5, 2017

geky commented Jan 5, 2017

bridadan commented Jan 5, 2017

MarceloSalazar commented Jan 11, 2017

lwip - Add more network tests #3265

lwip - Add more network tests #3265

Conversation

geky commented Nov 15, 2016 • edited Loading

geky commented Nov 15, 2016

mbed-bot commented Nov 15, 2016

Output

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jupe commented Nov 15, 2016

bulislaw commented Nov 15, 2016

geky commented Nov 15, 2016 • edited Loading

mbed-bot commented Nov 15, 2016

Output

geky commented Nov 15, 2016

mbed-bot commented Nov 15, 2016

Output

geky commented Nov 15, 2016

mbed-bot commented Nov 16, 2016

Output

0xc0170 commented Nov 16, 2016

geky commented Nov 16, 2016

geky commented Nov 17, 2016

mbed-bot commented Nov 17, 2016

Output

geky commented Nov 17, 2016

mbed-bot commented Nov 17, 2016

Output

geky commented Nov 17, 2016

SeppoTakalo commented Nov 21, 2016

geky commented Dec 19, 2016

mbed-bot commented Dec 19, 2016

bridadan commented Dec 19, 2016 • edited Loading

geky commented Dec 19, 2016

mbed-bot commented Dec 20, 2016

Output

geky commented Dec 20, 2016 • edited Loading

geky commented Jan 5, 2017

mbed-bot commented Jan 5, 2017

Output

geky commented Jan 5, 2017

bridadan commented Jan 5, 2017

MarceloSalazar commented Jan 11, 2017

geky commented Nov 15, 2016 •

edited

Loading

geky commented Nov 15, 2016 •

edited

Loading

bridadan commented Dec 19, 2016 •

edited

Loading

geky commented Dec 20, 2016 •

edited

Loading