Bundler seems to be invoked too late to get the correct versions of some built-in gems

[blowfishpro]

1) What is the problem?

Steps to Reproduce

• Have a ruby app with the gem io-wait included in its Gemfile.lock at a different version than the one built into ruby
  • e.g. io-wait 0.2.1 on ruby 3.0.2 (the included version is 0.1.0)
• Bundle install in a way that doesn't put gems in a place where ruby will find them by default
  • e.g. run bundler with the deployment flag so that it installs gems to your app's vendor/bundle
• Attempt to run the app through nginx
  • Standalone seems to reproduce this too but be sure not to invoke it with bundle exec

Observed behavior

The app fails to start:

Error: The application encountered the following error: You have already activated io-wait 0.1.0, but your Gemfile requires io-wait 0.2.1. Since io-wait is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports io-wait as a default gem. (Gem::LoadError)

Expected behavior

Passenger loads bundler before io-wait gets required so that the correct version is loaded

Workaround for now

Invoke nginx with bundle exec, this ensures that the correct versions of the gems are loaded when passenger starts executing ruby code

2) Passenger version and integration mode

Originally passenger 6.0.11/nginx, reproduced on passenger 6.0.12/standalone

3) OS or Linux distro

Originally RockyLinux 8.4 x86_64, reproduced on Debian 11.1 x86_64

4) Passenger installation method:

Originally source tarball, reproduced with just passenger gem

5) Your app's programming language (including any version managers) and framework (including versions)

Ruby 3.0.2 RVM, originally Rails 6.1.4.4, reproduced with no rails (pure rack 2.2.3)

6) Are you using a PaaS and/or containerization? If so which one?

We are using Docker (local version 20.10.11, this also happened in CircleCI's docker environment whose version I don't know)

7) Anything else about your setup that we should know?

Here's the minimal Gemfile and config.ru I used to reproduce, this was done in the official ruby 3.0.2 docker image:

Gemfile:

source 'https://rubygems.org'

gem 'rack'
gem 'io-wait'

(again be sure to run bundle install with the deployment option)

config.ru:

run lambda { |_env| [200, {'Content-Type'=>'text/plain'}, ["Hello Rack!"]] }

I ran gem install passenger to get passenger standalone installed

One other thing to note, io-wait ended up getting managed by bundler because ruby 3.1.0+ no longer bundles the net-smtp gem, so that has to be included in the Gemfile (net-smtp depends on net-protocol which depends on io-wait). Everything works at present on ruby 3.1.0 as the latest version of io-wait is bundled with ruby 3.1.0, however once that gem updates the versions will be mismatched again and things will break.

Additionally, here's the full error including the stack trace:

App 1589 output: Error: The application encountered the following error: You have already activated io-wait 0.1.0, but your Gemfile requires io-wait 0.2.1. Since io-wait is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports io-wait as a default gem. (Gem::LoadError)
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/runtime.rb:302:in `check_for_activated_spec!'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/runtime.rb:31:in `block in setup'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/spec_set.rb:159:in `each'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/spec_set.rb:159:in `each'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/runtime.rb:26:in `map'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/runtime.rb:26:in `setup'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler.rb:149:in `setup'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/setup.rb:20:in `block in <top (required)>'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/ui/shell.rb:136:in `with_level'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/ui/shell.rb:88:in `silence'
App 1589 output:     /usr/local/lib/ruby/3.0.0/bundler/setup.rb:20:in `<top (required)>'
App 1589 output:     <internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 1589 output:     <internal:/usr/local/lib/ruby/3.0.0/rubygems/core_ext/kernel_require.rb>:85:in `require'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb:380:in `activate_gem'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb:221:in `block in run_load_path_setup_code'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb:544:in `running_bundler'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb:220:in `run_load_path_setup_code'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/helper-scripts/rack-preloader.rb:91:in `preload_app'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/helper-scripts/rack-preloader.rb:189:in `block in <module:App>'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb:397:in `run_block_and_record_step_progress'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/helper-scripts/rack-preloader.rb:188:in `<module:App>'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
App 1589 output:     /usr/local/bundle/gems/passenger-6.0.12/src/helper-scripts/rack-preloader.rb:29:in `<main>'
[ E 2022-01-11 02:10:00.0021 1544/Tr age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /app: The application encountered the following error: You have already activated io-wait 0.1.0, but your Gemfile requires io-wait 0.2.1. Since io-wait is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports io-wait as a default gem. (Gem::LoadError)

[blowfishpro]

One more workaround I found: include the following in the nginx config:

passenger_env_var RUBYOPT '-r bundler/setup';

[CamJN]

I'm working on an option to require bundler when ruby starts up, which should help in these cases. It won't be on by default because it can ironically cause a conflict with two bundlers being loaded (an old one and a new one, in that order, doesn't work well).

[CamJN]

6.0.13 is released with the new option.

[joshuapinter]

FYI, we were receiving the following error:

Error: The application encountered the following error: You have already activated strscan 3.0.7, but your Gemfile requires strscan 3.1.0. Since strscan is a default gem, you can either remove your dependency on it or try updating to a newer version of bundler that supports strscan as a default gem. (Gem::LoadError)

What worked was manually install strscan at 3.1.0 in the global gem space:

sudo gem install strscan:3.1.0

This ensures that one is used instead of the default 3.0.7:

gem list strscan

*** LOCAL GEMS ***

strscan (3.1.0, default: 3.0.7)

I tried other methods before doing this that I saw elsewhere, including:

1. Upgrading rubygems.
2. Setting this NGINX config: passenger_env_var RUBYOPT '-r bundler/setup';

Neither worked.

I did get it working by pinning my version of strscan to 3.0.7 in our Gemfile but that wasn't a long term solution because we needed to upgrade rexml due to a security risk and 3.0.7 depended on an older version of rexml, so we needed to upgrade strscan.

Hope that helps others.

[KieranP]

@joshuapinter (and anyone else affected by this issue), we just encountered the same issue after upgrading Ruby, and fixed the issue by using the passenger_preload_bundler nginx option.

passenger_preload_bundler on;

This forces passenger to load bundler as soon as possible, which then prevents system default gems from being loaded but uses the Gemfile defined versions instead.

[joshuapinter]

@KieranP We just tested this in our Staging environment and everything appears to be working perfectly. Thanks!

To be clear, we removed the strscan 3.1.0 gem from our global gems and redeployed first to get the error back and then made the nginx change, redeployed and then confirmed the error went away. Brilliant.