Skip to content

Conversation

@st0012
Copy link
Member

@st0012 st0012 commented Nov 15, 2022

There's a huge difference between how tests are run in this repo and after being synced to Ruby CI. And that causes Ruby CI failure like this not being detected in the original PR #441. I think the rule of thumb is to have every test file self-contained, which is hard to verify with the current rake test.

So in this PR I added test_in_isolation task, which runs each test file in isolation. With this task, we would be able to catch the above failure before the PR is merged.

With this new task, I also caught another undiscovered error:

=========================================================================================================================================================================================================================================================================================================================================================================================================================================
Error: test_raise_exception_with_different_encoding_containing_invalid_byte_sequence(TestIRB::TestRaiseNoBacktraceException):
  NoMethodError: undefined method `mktmpdir' for Dir:Class

        Dir.mktmpdir("test_irb_raise_no_backtrace_exception_#{$$}") do |tmpdir|
           ^^^^^^^^^
  Did you mean?  mkdir
/Users/hung-wulo/src/github.com/ruby/irb/test/irb/test_raise_no_backtrace_exception.rb:28:in `test_raise_exception_with_different_encoding_containing_invalid_byte_sequence'
     25:
     26:     def test_raise_exception_with_different_encoding_containing_invalid_byte_sequence
     27:       backup_home = ENV["HOME"]
  => 28:       Dir.mktmpdir("test_irb_raise_no_backtrace_exception_#{$$}") do |tmpdir|
     29:         ENV["HOME"] = tmpdir
     30:
     31:         bundle_exec = ENV.key?('BUNDLE_GEMFILE') ? ['-rbundler/setup'] : []
=========================================================================================================================================================================================================================================================================================================================================================================================================================================
.
Finished in 0.477298 seconds.
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
3 tests, 6 assertions, 0 failures, 1 errors, 0 pendings, 0 omissions, 0 notifications
66.6667% passed
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
6.29 tests/s, 12.57 assertions/s
=====================================================================================================================================================================================
Test 'test/irb/test_raise_no_backtrace_exception.rb' failed when being executed in isolation. Please make sure 'rake test TEST=test/irb/test_raise_no_backtrace_exception.rb' passes.
=====================================================================================================================================================================================

Which is now fixed in the last commit.

This simulates how tests are run in Ruby CI and can detect some issues
before they're merged and break Ruby CI later.
@st0012 st0012 force-pushed the improve-testing-infra branch from 8381bef to 195fa67 Compare November 15, 2022 17:18
@st0012 st0012 requested a review from k0kubun November 15, 2022 17:25
Copy link
Member

@k0kubun k0kubun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@k0kubun k0kubun merged commit 51f23c5 into ruby:master Nov 15, 2022
@st0012 st0012 deleted the improve-testing-infra branch November 15, 2022 17:26
matzbot pushed a commit to ruby/ruby that referenced this pull request Nov 15, 2022
(ruby/irb#442)

* Add test_in_isolation task to run tests in isolation

This simulates how tests are run in Ruby CI and can detect some issues
before they're merged and break Ruby CI later.

* Run test_in_isolation task on CI

* Fix TestRaiseNoBacktraceException's setup

ruby/irb@51f23c58b0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants