Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ignore calls in other threads when counting ruby calls for a test #114

Merged
merged 1 commit into from
Nov 2, 2020

Conversation

dylanahsmith
Copy link
Contributor

Closes #113

Problem

As stated in #113 we have flaky test failures in a test that is counting ruby calls using a tracepoint.

I had trouble reproducing the test failure locally, so I commented out the test using TracePoint and started a tracepoint in test_helper.rb, which was able to capture calls on the Thread module we were seeing. After a bit of debugging, I found the source of the problem, that can be demonstrated using this in test/test_helper.rb

main_thread = Thread.current

c_call_trace = TracePoint.trace(:c_call) do |t|
  if t.defined_class == Thread
    thread_name = Thread.current == main_thread ? "main" : "other"
    raise "unexpected call to #{t.defined_class}##{t.method_id} on #{thread_name} thread"
  end
end

which outputs

Traceback (most recent call last):
	8: from /Users/dylants/.gem/ruby/2.5.8/gems/minitest-5.14.2/lib/minitest.rb:68:in `block in autorun'
	7: from /Users/dylants/.gem/ruby/2.5.8/gems/minitest-5.14.2/lib/minitest.rb:138:in `run'
	6: from /Users/dylants/.gem/ruby/2.5.8/gems/minitest-5.14.2/lib/minitest/parallel.rb:27:in `start'
	5: from /Users/dylants/.gem/ruby/2.5.8/gems/minitest-5.14.2/lib/minitest/parallel.rb:27:in `map'
	4: from /Users/dylants/.gem/ruby/2.5.8/gems/minitest-5.14.2/lib/minitest/parallel.rb:27:in `each'
	3: from /Users/dylants/.gem/ruby/2.5.8/gems/minitest-5.14.2/lib/minitest/parallel.rb:27:in `times'
	2: from /Users/dylants/.gem/ruby/2.5.8/gems/minitest-5.14.2/lib/minitest/parallel.rb:28:in `block in start'
	1: from /Users/dylants/.gem/ruby/2.5.8/gems/minitest-5.14.2/lib/minitest/parallel.rb:28:in `new'
/Users/dylants/src/liquid-c/test/test_helper.rb:10:in `block in <top (required)>': unexpected call to Thread#initialize on main thread (RuntimeError)

So these calls are coming from this line in minitest: https://github.com/seattlerb/minitest/blob/v5.14.2/lib/minitest/parallel.rb#L29

Solution

Filter calls to another thread in the test that was asserting on the number of ruby calls made.

@dylanahsmith dylanahsmith merged commit 9b720d6 into master Nov 2, 2020
@dylanahsmith dylanahsmith deleted the fix-flaky-tracing-test branch November 2, 2020 15:51
dylanahsmith added a commit that referenced this pull request Nov 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants