[CI] tests/async/tioselectors.nim flaky test for freebsd + OSX CI

[timotheecour]

CI broke in unrelated PR: #13165 https://builds.sr.ht/~araq/job/137694

CC: stdlib_json.nim
CC: stdlib_sets.nim
CC: stdlib_strformat.nim
CC: ci_testresults.nim
Hint: clang   -o /usr/home/build/Nim/tools/ci_testresults  /home/build/.cache/nim/ci_testresults_d/stdlib_assertions.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_io.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_system.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_parseutils.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_math.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_unicode.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_strutils.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_posix.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_times.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_os.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_hashes.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_tables.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_streams.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_lexbase.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_parsejson.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_json.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_sets.nim.c.o /home/build/.cache/nim/ci_testresults_d/stdlib_strformat.nim.c.o /home/build/.cache/nim/ci_testresults_d/@mci_testresults.nim.c.o  -lm   -Wl,-rpath=.:/usr/local/lib:/usr/pkg/lib:/usr/X11R6/lib [Link]
Hint: 57273 LOC; 1.350 sec; 72.137MiB peakmem; Debug build; proj: /usr/home/build/Nim/tools/ci_testresults.nim; out: /usr/home/build/Nim/tools/ci_testresults [SuccessX]
Hint: /usr/home/build/Nim/tools/ci_testresults  [Exec]
Category: async
Name: tests/async/tioselectors.nim C
Action: run
Result: reExitcodesDiffer
-------- Expected -------
exitcode: 0
--------- Given  --------
exitcode: 1

Output:
tioselectors.nim(466)    tioselectors
tioselectors.nim(158)    timer_notification_test
assertions.nim(27)       failedAssertImpl
assertions.nim(20)       raiseAssert
fatal.nim(55)            sysFatal
Error: unhandled exception: tioselectors.nim(158, 13) `len(rc4) == 1 and len(rc5) == 0`  [AssertionError]
-------------------------

+ exit 1

[timotheecour]

I got a different error this time on another unrelated PR:

line 152 instead of 158:
https://builds.sr.ht/~araq/job/166701

Hint: /usr/home/build/Nim/tools/ci_testresults  [Exec]
Category: async
Name: tests/async/tioselectors.nim C
Action: run
Result: reExitcodesDiffer
-------- Expected -------
exitcode: 0
--------- Given  --------
exitcode: 1

Output:
tioselectors.nim(466)    tioselectors
tioselectors.nim(152)    timer_notification_test
assertions.nim(29)       failedAssertImpl
assertions.nim(22)       raiseAssert
fatal.nim(55)            sysFatal
Error: unhandled exception: tioselectors.nim(152, 13) `len(rc1) == 1 and len(rc2) == 1`  [AssertionError]
-------------------------

+ exit 1

[timotheecour]

from experiments in #13648, tioselectors failure seems context dependent (dependent on what else is running in the system). eg, simply running the test 1000 times by itself will not trigger the bug. It only triggers in the context of other things running.

Obviously the easy way is to disable the test, but then BSD will just have a latent bug ready to bite at random in production

[timotheecour]

@euantorano since you've introduced freebsd CI (thanks for that!) and you're one of the most experienced with freebsd, I'd love your help with:

• instructions for how to get a freebsd VM up and running all the way to the point where you can build nim from devel (using docker would be ideal; I'm on OSX)

  • unfortunately using parallels on OSX didn't work (freebsd is not supported in parallels)
  • https://todo.sr.ht/~sircmpwn/builds.sr.ht/279 makes it hard to debug (builds.sr.ht sourcehut does not currently support github forks so I can't debug it in my fork); here's the latest I got on that issue from sircmpwn:

SourceHut does not generally encourage the use of GitHub-style forks, seeing them as an anti-feature which is designed more to encourage you to incorporate GitHub's centralization more deeply into your workflow than it is designed to speed up or simplify your workflow

• if possible, also if you have some insight on this tioselectors.nim issue, which is one of the flakiest and keeps popping up

[euantorano]

I can certainly write something up for that. I use macOS too and have used VirtualBox with some success. I’ll write something on the Nim forum later on today and link it here.

…

On Thu, 21 May 2020, at 04:37, Timothee Cour wrote: @euantorano <https://github.com/euantorano> since you've introduced freebsd CI (thanks for that!) and you're one of the most experienced with freebsd, I'd love your help with: * instructions for how to get a freebsd VM up and running all the way to the point where you can build nim from devel (using docker would be ideal; I'm on OSX) * unfortunately using parallels on OSX didn't work (no freebsd is not supported in parallels) * https://todo.sr.ht/~sircmpwn/builds.sr.ht/279 makes it hard to debug (builds.sr.ht sourcehut does not support github forks so I can't debug it in my fork) * if possible, also if you have some insight on this issue, which is one of the flakiest and keeps popping up — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#13166 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFW24NKOAVB4ERTJKW3UBTRSSOWVANCNFSM4KHLQLWQ>.

[timotheecour]

I’ll write something on the Nim forum later on today and link it here.

excellent thanks! even better would be in nim docs so it keeps updated and easier to be found, but that's less urgent and can be done at later time (either by you or anyone else)

[euantorano]

@timotheecour As requested: https://forum.nim-lang.org/t/6364#39239 - hopefully that helps. If there's anything missing, please let me know and I'll add some more details.