WebRTC peerdatachannels complicance Test Suite

[jbenet]

I think we should have a strong test suite to test the robustness of various implementations. I'm thinking with virtual networks and various topologies, including NATs, stun servers, etc. This way we could be certain libraries work.

Things to test:

• NAT traversal
• Double NATs
• with STUN/TURN servers
• direct connection found when in local network
• stress test data transmission
• various data transmission patterns (i.e. send for a while, stop for a while, continue)
• reliable and unreliable transports

If we can make it pluggable, needing only a single binary that can be fed a few arguments in each test case, we should be able to test everything, including a version using a build of chromium itself.

[jbenet]

Should use whatever we can from: http://www.webrtc.org/testing/w3c-conformance-tests

[mafintosh]

A shared benchmark would also be great

[jbenet]

Yeah! comprehensive benchmarks including:

• throughput
• resource usage (mem, cpu)
• bandwidth and latency overhead
• NAT traversal success rates, and latencies

[DamonOehlman]

+1 on this - let me know how I can help. I've had some pretty solid success using headless testing on travis to do a fair bit and I think we can probably leverage and while it's not a perfect example of what we will want to do in a more general WebRTC sense, this repo might be of interest:

https://github.com/rtc-io/rtc-signaller-testrun

which is a concrete use of:

https://github.com/rtc-io/webrtc-testing-on-travis

The main thing I haven't yet tackled, but would love to tackle is how to do interop tests, though that is most definitely a separate issue (which I might log in the future).

Anyway, ccing @diasdavid because I've recently seen some stuff that he's done recently and imagine this is something that he is interested in also.