Replace Zeus with process forking strategy (#271)

Integration tests without Zeus are very slow because we run each example in a new process, which has to load all dependencies and frameworks – like RSpec, SuperDiff, ActiveSupport, etc – from scratch. Zeus improved on this, but has a client-server architecture and has been proving somewhat difficult to use. If we instead load the dependencies and then run RSpec in a new [_forked_](https://man7.org/linux/man-pages/man2/fork.2.html) subprocess, we don't have to reload anything loaded before fork time, and don't have to use Zeus. Light testing shows a ~5x speed-up over non-Zeus integration tests.
splitwise · Oct 24, 2024 · 88a123c · 88a123c
1 parent 9943ecc
commit 88a123c
Show file tree

Hide file tree

Showing 33 changed files with 586 additions and 273 deletions.
diff --git a/.gitignore b/.gitignore
@@ -6,8 +6,6 @@ pkg
 spec/examples.txt
 spec/internal
 tmp
-zeus.server.log
-zeus.server-start.log
 
 # Ignore Yarn stuff
 .pnp.*

diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -12,6 +12,7 @@
 
 - Switch from Prettier to Rubocop. [#269](https://github.com/splitwise/super_diff/pull/269)
 - Fix outdated reference in documentation. [#270](https://github.com/splitwise/super_diff/pull/270) by [@emmanuel-ferdman](https://github.com/emmanuel-ferdman)
+- Replace Zeus with forking strategy for tests. [#271](https://github.com/splitwise/super_diff/pull/271)
 
 ## 0.13.0 - 2024-09-22
 

diff --git a/Gemfile b/Gemfile
@@ -16,6 +16,5 @@ gem 'rubocop'
 gem 'syntax_tree'
 gem 'syntax_tree-haml'
 gem 'syntax_tree-rbs'
-gem 'warnings_logger'
 
 gemspec
diff --git a/Gemfile.lock b/Gemfile.lock
@@ -83,7 +83,6 @@ GEM
     thor (1.3.2)
     tilt (2.4.0)
     unicode-display_width (2.6.0)
-    warnings_logger (0.1.1)
 
 PLATFORMS
   arm64-darwin-21
@@ -105,7 +104,6 @@ DEPENDENCIES
   syntax_tree
   syntax_tree-haml
   syntax_tree-rbs
-  warnings_logger
 
 BUNDLED WITH
    2.5.20
diff --git a/bin/start-dev b/bin/start-dev
diff --git a/docs/contributors/architecture/structure.md b/docs/contributors/architecture/structure.md
@@ -244,28 +244,17 @@ They are divided into two kinds:
   holds tests for individual classes and methods.
 - **Integration tests**, located in `spec/integration/`,
   construct tiny test suites which import SuperDiff
-  and run them in isolated environments
+  and run them in forked processes
   to ensure that SuperDiff works as designed
-  in more real-world scenarios.
+  in more real-world scenarios. We fork instead of
+  running from scratch to avoid having to reload
+  SuperDiff and other libraries for every example.
 
 The files in `spec/support/` (imported via `spec_helper.rb`)
 contain helpers, matchers, and tests
 that are shared among unit and integration tests,
 at their respective levels.
 
-Beyond this, [Zeus][zeus], a way to speed up integration tests,
-and is run and configured via these files:
-
-- `bin/start-dev`
-- `config/zeus_plan.rb`
-- `support/test_plan.rb`
-- `zeus.json`
-
-The [official Zeus docs](https://github.com/burke/zeus/blob/master/docs/ruby/modifying.md)
-is helpful for understanding how these files work together.
-
-[zeus]: https://github.com/burke/zeus
-
 ## Ruby
 
 The following files are used to set up Ruby,

diff --git a/docs/contributors/how-to-contribute.md b/docs/contributors/how-to-contribute.md
@@ -62,23 +62,6 @@ bin/rspec spec/integration/...
 bin/rspec spec/unit/...
 ```
 
-Note that the integration tests
-can be quite slow to run.
-If you'd like to speed them up,
-run the following command in a separate terminal session:
-
-```
-zeus start
-```
-
-Now the next time you run an integration test by saying
-
-```
-bin/rspec spec/integration/...
-```
-
-it should run twice as fast.
-
 ## 5. Run the linter
 
 Code is linted and formatted using Rubocop.

diff --git a/gemfiles/no_rails_rspec_gte_3_10.gemfile b/gemfiles/no_rails_rspec_gte_3_10.gemfile
@@ -14,7 +14,6 @@ gem "rubocop"
 gem "syntax_tree"
 gem "syntax_tree-haml"
 gem "syntax_tree-rbs"
-gem "warnings_logger"
 gem "rspec", "3.12.0"
 gem "rspec-core", "3.12.0"
 gem "rspec-expectations", "3.12.3"

diff --git a/gemfiles/no_rails_rspec_lt_3_10.gemfile b/gemfiles/no_rails_rspec_lt_3_10.gemfile
@@ -14,7 +14,6 @@ gem "rubocop"
 gem "syntax_tree"
 gem "syntax_tree-haml"
 gem "syntax_tree-rbs"
-gem "warnings_logger"
 gem "rspec", "~> 3.9.0"
 
 gemspec path: "../"
diff --git a/gemfiles/rails_6_0_rspec_gte_3_10.gemfile b/gemfiles/rails_6_0_rspec_gte_3_10.gemfile
@@ -14,7 +14,6 @@ gem "rubocop"
 gem "syntax_tree"
 gem "syntax_tree-haml"
 gem "syntax_tree-rbs"
-gem "warnings_logger"
 gem "activerecord-jdbcsqlite3-adapter", platform: :jruby
 gem "jdbc-sqlite3", platform: :jruby
 gem "net-ftp"

diff --git a/gemfiles/rails_6_0_rspec_lt_3_10.gemfile b/gemfiles/rails_6_0_rspec_lt_3_10.gemfile
@@ -14,7 +14,6 @@ gem "rubocop"
 gem "syntax_tree"
 gem "syntax_tree-haml"
 gem "syntax_tree-rbs"
-gem "warnings_logger"
 gem "activerecord-jdbcsqlite3-adapter", platform: :jruby
 gem "jdbc-sqlite3", platform: :jruby
 gem "net-ftp"

diff --git a/gemfiles/rails_6_1_rspec_gte_3_10.gemfile b/gemfiles/rails_6_1_rspec_gte_3_10.gemfile
@@ -14,7 +14,6 @@ gem "rubocop"
 gem "syntax_tree"
 gem "syntax_tree-haml"
 gem "syntax_tree-rbs"
-gem "warnings_logger"
 gem "activerecord-jdbcsqlite3-adapter", platform: :jruby
 gem "jdbc-sqlite3", platform: :jruby
 gem "net-ftp"

diff --git a/gemfiles/rails_6_1_rspec_lt_3_10.gemfile b/gemfiles/rails_6_1_rspec_lt_3_10.gemfile
@@ -14,7 +14,6 @@ gem "rubocop"
 gem "syntax_tree"
 gem "syntax_tree-haml"
 gem "syntax_tree-rbs"
-gem "warnings_logger"
 gem "activerecord-jdbcsqlite3-adapter", platform: :jruby
 gem "jdbc-sqlite3", platform: :jruby
 gem "net-ftp"

diff --git a/gemfiles/rails_7_0_rspec_gte_3_10.gemfile b/gemfiles/rails_7_0_rspec_gte_3_10.gemfile
@@ -14,7 +14,6 @@ gem "rubocop"
 gem "syntax_tree"
 gem "syntax_tree-haml"
 gem "syntax_tree-rbs"
-gem "warnings_logger"
 gem "activerecord-jdbcsqlite3-adapter", platform: :jruby
 gem "jdbc-sqlite3", platform: :jruby
 gem "net-ftp"

diff --git a/gemfiles/rails_7_0_rspec_lt_3_10.gemfile b/gemfiles/rails_7_0_rspec_lt_3_10.gemfile
@@ -14,7 +14,6 @@ gem "rubocop"
 gem "syntax_tree"
 gem "syntax_tree-haml"
 gem "syntax_tree-rbs"
-gem "warnings_logger"
 gem "activerecord-jdbcsqlite3-adapter", platform: :jruby
 gem "jdbc-sqlite3", platform: :jruby
 gem "net-ftp"

diff --git a/lib/super_diff/rspec/operation_tree_builders/collection_containing_exactly.rb b/lib/super_diff/rspec/operation_tree_builders/collection_containing_exactly.rb
@@ -36,7 +36,7 @@ def unary_operations
         end
 
         def build_operation_tree
-          OperationTrees::Array.new([])
+          Basic::OperationTrees::Array.new([])
         end
 
         private

diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb
@@ -71,7 +71,6 @@
   config.color_mode = :on if ENV['CI'] == 'true'
 end
 
-require 'warnings_logger'
 $VERBOSE = true
 WarningsLogger.configure do |config|
   config.project_name = 'super_diff'

diff --git a/spec/support/command_runner.rb b/spec/support/command_runner.rb
@@ -77,6 +77,14 @@ def build_message
     end
   end
 
+  class Result
+    attr_reader :output
+
+    def initialize(output:)
+      @output = output
+    end
+  end
+
   def self.run(*args)
     new(*args).tap do |runner|
       yield runner if block_given?
@@ -137,7 +145,7 @@ def run
       fail! if run_successfully && !success?
     end
 
-    self
+    Result.new(output: output)
   end
 
   def stop

diff --git a/spec/support/integration/test_programs/base.rb b/spec/support/integration/test_programs/base.rb
@@ -51,45 +51,26 @@ def preserve_as_whole_file?
         end
 
         def result_of_command
-          @result_of_command ||=
-            if zeus_running?
-              Bundler.with_unbundled_env do
-                CommandRunner.run(Shellwords.join(command))
-              end
-            else
-              CommandRunner.run(
-                Shellwords.join(command),
-                env: {
-                  'DISABLE_PRY' => 'true'
-                }
-              )
-            end
-        end
-
-        def command
-          raise 'RAILS_ENV is being set somehow?!' if ENV['RAILS_ENV']
-
-          if zeus_running?
-            [
-              'zeus',
-              test_plan_command,
-              color_option,
-              '--no-pry',
-              tempfile.to_s,
-              '--super-diff-configuration',
-              JSON.generate(super_diff_configuration)
-            ]
-          else
-            ['rspec', '--options', '/tmp/dummy-rspec-config', tempfile.to_s]
-          end
+          @result_of_command ||= if Process.respond_to?(:fork)
+                                   result_of_command_with_fork
+                                 else
+                                   result_of_command_with_spawn
+                                 end
         end
 
-        def zeus_running?
-          PROJECT_DIRECTORY.join('.zeus.sock').exist?
+        def result_of_command_with_fork
+          RSpecForkedRunner.new(
+            rspec_options: ['--options', '/tmp/dummy-rspec-config', tempfile.to_s]
+          ).run
         end
 
-        def color_option
-          color_enabled? ? '--color' : '--no-color'
+        def result_of_command_with_spawn
+          CommandRunner.run(
+            Shellwords.join(['rspec', '--options', '/tmp/dummy-rspec-config', tempfile.to_s]),
+            env: {
+              'DISABLE_PRY' => true
+            }
+          )
         end
 
         def tempfile
@@ -103,23 +84,18 @@ def tempfile
         end
 
         def program
-          if zeus_running?
-            minimal_program
-          else
-            <<~PROGRAM
-              require "#{PROJECT_DIRECTORY.join('support/test_plan.rb')}"
-
-              test_plan = TestPlan.new(
-                using_outside_of_zeus: true,
-                color_enabled: #{color_enabled?.inspect},
-                super_diff_configuration: #{super_diff_configuration.inspect}
-              )
-              #{test_plan_prelude}
-              test_plan.#{test_plan_command}
-
-              #{minimal_program}
-            PROGRAM
-          end
+          <<~PROGRAM
+            require "#{PROJECT_DIRECTORY.join('support/test_plan.rb')}"
+
+            test_plan = TestPlan.new(
+              color_enabled: #{color_enabled?.inspect},
+              super_diff_configuration: #{super_diff_configuration.inspect}
+            )
+            #{test_plan_prelude}
+            test_plan.#{test_plan_command}
+
+            #{minimal_program}
+          PROGRAM
         end
 
         def minimal_program
@@ -129,7 +105,13 @@ def minimal_program
             <<~PROGRAM
               RSpec.describe "test" do
                 it "passes" do
-                  #{reindent(code, level: 2)}
+                  opt_before = RSpec::Expectations.configuration.on_potential_false_positives
+                  begin
+                    RSpec::Expectations.configuration.on_potential_false_positives = :nothing
+                #{reindent(code, level: 3)}
+                  ensure
+                    RSpec::Expectations.configuration.on_potential_false_positives = opt_before
+                  end
                 end
               end
             PROGRAM
-Original file line number
+Diff line change
@@ Expand Up / @@ -36,7 +36,7 @@ def unary_operations @@
             end
             def build_operation_tree
-              OperationTrees::Array.new([])
+              Basic::OperationTrees::Array.new([])
             end
             private
@@ Expand Down @@