-
Notifications
You must be signed in to change notification settings - Fork 495
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix a number of flaky tests (.ordered expectation in multiple threads, and unstable output expectations) #736
Conversation
@@ -188,12 +188,6 @@ def setup_runtime_log | |||
|
|||
config.raise_errors_for_deprecations! | |||
|
|||
# sometimes stuff hangs -> do not hang everything | |||
config.include(Module.new {def test_timeout;30;end }) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
still need this though ?
... having tests just stop and hang is hard to debug :(
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sure I can re-add that! But I think the Timeout
block was actually swallowing some errors and making it super hard to figure out this test failure, and when I removed it the test started failing consistently, so it was easy to fix. So my theory is that those regular timeout errors might have actually been caused by this ordered
expectation.
But I'll run my delayed commit script again for a while and create 15 more builds, and will check to see if there's any other flaky tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I remember when I last worked on stabilizing CI here, I also temporarily removed this because it seemed to be indeed swallowing errors somehow. 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
bringing them back in #741 with a dedicated class and note on debugging
Heh I found one more flaky test. The test checked that the output doesn't include
So I've fixed that one too. Otherwise, the tests seem to be very stable now, and nothing is hanging or timing out. So I think the |
One more flaky test: https://ci.appveyor.com/project/grosser/parallel-tests/builds/29391856/job/8j2dqxd7duf10gra
On my machine, the
And then occasionally there is one fewer dot:
But for this CI build, the output only included 3 dots before
So it didn't match the regex: It seems like the value of After
|
614dae8
to
950777e
Compare
.travis.yml
Outdated
- ruby-head | ||
- jruby-head | ||
# - ruby-head | ||
# - jruby-head |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need that back ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oh yeah, I commented those out because they are consistently failing and adding 23 minutes to the build times, because Travis CI runs all of the builds in a sequence (not in parallel.)
See:
- https://travis-ci.org/grosser/parallel_tests/builds/621971355?utm_medium=notification&utm_source=github_status
- https://travis-ci.org/grosser/parallel_tests/builds/621971005?utm_medium=notification&utm_source=github_status
- https://travis-ci.org/grosser/parallel_tests/builds/621957792?utm_medium=notification&utm_source=github_status
I had a quick look to see if I could fix those builds as well, but it was a bit difficult. The ruby-head
one was particularly weird, because it's just getting stuck at the installation/gemset stage with RVM and timing out after 10m of inactivity. I found these related issues but couldn't find a solution:
- RVM install stuck while importing gemset global.gems rvm/rvm#3344
- Something to do with DNS timeouts?
- Installation of ruby stucks on importing gems rvm/rvm#2455
But I will uncomment these!
…are run in parallel threads and there are no guarantees about the execution order
950777e
to
e48578e
Compare
See the description and comments in #735
These tests are run in parallel threads, so there are no guarantees about the execution order.
I've also removed the
Timeout.timeout
block around each example, because that seemed to swallow the actual error and make it much harder to figure out why random tests were crashing. (And nothing seems to be hanging anymore.)EDIT: I'll also throw in the commit to cache the gems in
spec/fixtures/rails51/vendor/bundle
andspec/fixtures/rails52/vendor/bundle
, because that speeds up the CI builds.