test-dgram-pingpong flaky on parallel runs

[jbergstroem]

It's been very frequent on the ppc machines during -J runs as of late: https://ci.nodejs.org/job/node-test-commit-plinux/nodes=ppcbe-fedora20/ws/test.tap/*view*/

[Trott]

I doubt this is the problem, but just in case: Is there any special firewalling or throttling on those machines? It looks like this test fires 1500 UDP packets in (let's say) half a second or so.

[jbergstroem]

@Trott: iptables looks squeaky clean and the vm admin dashboard doesn't point to any firewalling.

[Trott]

At least one other "flaky on PPC when run in parallel" test is a test that is also network intensive. In #4527, you report that test-http-regr-gh-2928 is flaky on PPC. That test checks that the HTTP flood prevention mechanism kicks in.

Maybe there's something the provider is doing that we can't see from the host itself? (I think all this stuff happens on localhost, so it wouldn't be on the subnet or anything like that, I imagine...)

[jbergstroem]

Regarding provider: that's why i checked the dashboard (there's usually ways to access firewalling from there). I'll create a support ticket and ask.

[Trott]

Maybe this test should just be moved to sequential. Maybe it's just too network (or other resrouce) intensive for some platforms.

[Trott]

Probable fix: #4615

[Trott]

I think #4615 fixed this. Please re-open if I'm mistaken.

[Trott]

Sad to be re-opening this: https://ci.nodejs.org/job/node-test-binary-arm/934/RUN_SUBSET=0,nodes=pi2-raspbian-wheezy/tapTestReport/test.tap-138/

[jbergstroem]

:/

[Trott]

And another one: https://ci.nodejs.org/job/node-test-binary-arm/935/RUN_SUBSET=0,nodes=pi2-raspbian-wheezy/tapTestReport/test.tap-138/

[Trott]

The sudden-seeming spike in this failing on Raspberry Pi suggests either a regression in the code or else a change in CI devices or other CI infrastructure.

[Trott]

Stress test on master: https://ci.nodejs.org/job/node-stress-single-test/401/nodes=pi2-raspbian-wheezy/console

Stress test on 924cc6c: https://ci.nodejs.org/job/node-stress-single-test/402/nodes=pi2-raspbian-wheezy/console

Stress test on c3d5b2b: https://ci.nodejs.org/job/node-stress-single-test/403/nodes=pi2-raspbian-wheezy/console

[Trott]

All of them failing. :-(

[Trott]

I added some logging for each dgram message received. It looks like what happens is that a message is dropped from time to time. I interpret this as Not A Bug because there is no guarantee that UDP packets will be received. This test opens three UDP ports and sends 500 messages to them relatively quickly. No problem for a typical host, but the Raspberry Pi devices seem to have trouble from time to time.

So my inclination would be to alter the test to tolerate an occasional missed message.

Thoughts?

/cc @rvagg (who, as best as I can tell, is The Raspberry Pi Person around here to the extent that we have one)

Here's the stress test using the additional logging: https://ci.nodejs.org/job/node-stress-single-test/409/nodes=pi2-raspbian-wheezy/console

[rvagg]

sorry for the delay, I have nothing useful to add here sorry, let me know if I'm needed to dive deeper here cause I'm only passing a cursory glance over the issue right now