Upgrade Rails 7.1.x to 7.2.x huge performance regression

[kimyu-ng]

Description:

After upgrading from Rails 7.1.5 to 7.2.2.1, along with upgrading other gems, I have observed significant performance degradation when running 8.3k+ tests in 12 parallel containerized environment. The same workload that previously took ~15 minutes now consistently takes ~40 minutes. Below are the details of my environment:

Environment Details:

Ruby Version: 3.3.6
Rails Versions Tested: 7.1.5, 7.2.2.1
Number of Containers: 12
Workload: Splitting 12 containers running Rspec tests

Observations:

1. Baseline Performance with Rails 7.1.5 (before any upgrade):

   • Rails Version: 7.1.5
   • Configuration: config.load_defaults 7.1
   • Time Taken: ~15 minutes

2. Upgraded Performance with Rails 7.2.2.1 and Configurations Tested:

   • Rails Version: 7.2.2.1
   • Configurations Tested:
     • config.load_defaults 7.2, YJIT enabled
     • config.load_defaults 7.2, YJIT disabled
     • config.load_defaults 7.1, YJIT disabled
   • Time Taken: ~40 minutes consistently across configurations

3. Isolating the Rails Upgrade:

   • Retested with Rails 7.1.5 and config.load_defaults 7.1, but with all other gem upgrades intact.
   • Time Taken: ~15 minutes

Resource Utilization:

Attached are graphs showing CPU and RAM usage across the test runs.

Expected Behavior:

The workload should not see a 2.5x increase in runtime when upgrading to Rails 7.2.2.1. Performance should be similar.

Actual Behavior:

The workload runtime increases from ~15 minutes to ~40 minutes post-upgrade to Rails 7.2.2.1, regardless of the configurations tested.

What additional debugging steps can I take to pinpoint the cause?

[ngan]

Just curious, did you happen to upgrade to Ruby 3.3.6 at the same time?

[kimyu-ng]

Just curious, did you happen to upgrade to Ruby 3.3.6 at the same time?

ruby version is kept the same. The only difference is rails 7.1 to 7.2

[byroot]

The way to debug this is to use a profiler, I recommend https://github.com/jhawthorn/vernier

See if you can reproduce the same perf difference in a single (or a few) spec files, and run them in both configurations with vernier.

Ideally, if you can, the vernier profiles can be uploaded from the Firefox Profiler viewer, that makes it easy to share them.

Comparing the two profiles should show where the difference comes from.

I suspect it might be the connection checkout refactoring, but that's just a guess, only profiling can make this bug report actionable.

[devansh-par]

We also notice performance degradation after upgrading from rails v7.1.3.4 to 7.1.5.1.
It is weird. Only changing rails version in Gemfile and running "bundle update rails" is responsible for performance degradation. Nothing else is done.

[rails-bot]

This issue has been automatically closed because there has been no follow-up response from the original author. We currently don't have enough information in order to take action. Please reach out if you have any additional information that will help us move this issue forward.

[devansh-par]

@kimyu92 Were you able to figure out the cause?

[kimyu-ng]

Finally I have gotten time to run vernier on a simple test 7.1.5 vs 7.2.2.1

7.1.5

7.2.2.1

@byroot Any pointer on figuring whether it's Rails performance bug or Rspec performance bug?

[byroot]

I can;t tell with what you posted, please you seh "Upload local profile" feature of Firefox FO xprofiler, you can see it at the top right coerner on your screenshots.

[kimyu-ng]

It's not open source. I pick some test cases out of thousands of tests, do you have any pointer how to move forward

7.1.5.1

7.2.2.1

[skipkayhil]

It's not open source. I pick some test cases out of thousands of tests, do you have any pointer how to move forward

The thing that stood out to me immediately is all the extra time spent in the RSpec methods in your Rails 7.2 profile. Was its version updated along with the Rails version? If so, you could compare Rails 7.1 with only the version of RSpec changed to see if you still see a slowdown.

[byroot]

I still can't tell that much from a screenshot. But:

• total time jumped from 5.9s to 7.3s, assuming the extra time is running test that can indeed be massive
• In both case there's 40% time spent just loading the app, so test run is somewhat dwarfed and that makes the profile hard to exploit. Not sure if this is running a single test or single test file, but running more tests would give better info.
• Not sure why lib/rubocop.rb is loaded, that seems wasteful, but it's the case in both versions.
• Also in both cases, 7% in Scrypt::Engine seems very wasteful, I'm unfamiliar with Scrypt but with BCrypt we lower the hasing cost in test to avoid that. Looks like SCrypt support the same thing.

If you can't share the actual profiles, I can't help you any more than that and you'll have to figure out where the problem comes from on your own.