Some tests timeout non-deterministically

[sogartar]

🐛 Bug

On my machine often tests would hang and timeout.

To Reproduce

Steps to reproduce the behavior:

1. bazel test --cache_test_results=no //...

Expected behavior

All tests pass without timeout.

Environment

Please fill in this checklist:

• CompilerGym:
• How you installed CompilerGym (conda, pip, source): source
• OS: Ubuntu 20.04.3 LTS (x86_64)
• Python version: 3.8.12
• Build command you used (if compiling from source): make test
• GCC version: (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
• Clang version: 9.0.1-12
• Bazel version (if compiling from source): 4.2.2

Additional context

The timeout is non-deterministic. If I run then a few times, reusing cached results, they will eventually pass. It seems that the CI machines don't not have the same problem.
Attached is a list of tests that timed out when I was debugging and testing stuff.
timeout.txt

[ChrisCummins]

Hi @sogartar, I can reproduce the issue on my end, though not as severely as for you (judging by the attached timeout list). I think it's a combination of two things: (1) flaky timeout on an overloaded system, such as when running a bunch of tests in parallel, and (2) resource leaks preventing test scripts from shutting down. You can see this by checking the test logs to see if 100% of the tests completed. If they did, then its likely related to #459.

Thanks for the good report. I'll try and find some time to look into this soon.

BTW, I haven't had any timeout issues when using make install-test, which uses pytest's runner to coordinate test execution.

Cheers,
Chris