intermittant "out of char range (RangeError) running ruby1.8 rails 2.3.18 when reloading apache

[conathan]

Issue report

After upgrading from Passenger 6.0.14 to 6.0.15 on AlmaLinux 8.6, very intermittantly we would get a stacktrace as follows:

[ E 2022-09-23 03:41:29.8051 463160/Tc age/Cor/Con/CheckoutSession.cpp:283 ]: [Client 3-390] Cannot checkout session because a spawning error occurred. The identifier of the error is 57585c0f. Please see earlier logs for details about the error.
[ E 2022-09-23 03:41:29.8052 463160/Tg age/Cor/Con/CheckoutSession.cpp:283 ]: [Client 5-390] Cannot checkout session because a spawning error occurred. The identifier of the error is 57585c0f. Please see earlier logs for details about the error.
[ E 2022-09-23 03:41:29.8052 463160/Te age/Cor/Con/CheckoutSession.cpp:283 ]: [Client 4-390] Cannot checkout session because a spawning error occurred. The identifier of the error is 57585c0f. Please see earlier logs for details about the error.
[Fri Sep 23 03:41:30.002050 2022] [mpm_event:notice] [pid 132718:tid 140003701397824] AH00492: caught SIGWINCH, shutting down gracefully
App 537418 output: Error: The application encountered the following error: 49688 out of char range (RangeError)
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:119:in `chr'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:119:in `string'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:78:in `value'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:92:in `hash'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:140:in `repeat_until'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:88:in `hash'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:74:in `object'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:78:in `value'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:92:in `hash'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:140:in `repeat_until'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:88:in `hash'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:74:in `object'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:64:in `parse'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:43:in `parse'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:74:in `read_startup_arguments'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:73:in `open'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:73:in `read_startup_arguments'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:35:in `init'
App 537418 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/preloader_shared_helpers.rb:38:in `init'
App 537418 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:184
[ E 2022-09-23 03:41:30.2797 463160/T6c7 age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /home/webapps/asterion/current: The application encountered the following error: 49688 out of char range (RangeError)
  Error ID: 1f8a4440
  Error details saved to: /tmp/passenger-error-HtBKWX.html

Out of about 30 servers, seems like every 1-2 days 1 (or 2) servers would act up when doing a logrotate. Also triggerable via systemctl reload httpd.

Rolling back to 6.0.14 appears to have stopped this from occuring

Are you sure this is a bug in Passenger?

This was working with no known issues in 6.0.14

Please try with the newest version of Passenger to avoid issues that have already been fixed

This started happening in 6.0.15

Question 1: What is the problem?

• What is the expected behavior?

Continue working after a systemctl reload httpd

• What is the actual behavior?

Very intermittantly has the above stacktrace.

• How can we reproduce it? Please try to provide a sample application (or Virtual Machine) demonstrating the issue. Otherwise, if we can't reproduce it, we might have to ask you a number of followup questions or run certain commands to try and figure out the problem.

Question 2: Passenger version and integration mode:

mod_passenger 6.0.14 running Apache 2.4.37 (Stock AlmaLinux)

Question 3: OS or Linux distro, platform (including version):

AlmaLinux 8.6, x86_64

Question 4: Passenger installation method:

Your answer:
[ ] RubyGems + Gemfile
[ ] RubyGems, no Gemfile
[ ] Phusion APT repo
[X] Phusion YUM repo
[ ] OS X Homebrew
[ ] source tarball
[X] Other, please specify:

Also a Ruby 1.8.7 native library compiled as follows: (Adapted from upstream spec file)
rake1.8 fakeroot
NATIVE_PACKAGING_METHOD=rpm
FS_PREFIX=%{_prefix}
FS_BINDIR=%{_bindir}
FS_SBINDIR=%{_sbindir}
FS_DATADIR=%{_datadir}
FS_LIBDIR=%{_libdir}
FS_DOCDIR=%{_docdir}
RUBYLIBDIR=%{passenger_libdir}
RUBYARCHDIR=%{passenger_ruby187_libdir}
APACHE2_MODULE_PATH=%{_httpd_moddir}/mod_passenger.so

Question 5: Your app's programming language (including any version managers) and framework (including versions):

• For example: Ruby 2.3.0, RVM, Rails 5.0.0; Node.js 4.0.0 with Express 4.13.4

Your answer:

Question 6: Are you using a PaaS and/or containerization? If so which one?

No

[CamJN]

Does this reproduce with an even remotely recent ruby version?

[conathan]

So far, no. Most of our servers are Ruby 2.5 based (AlmaLinux 8.6 system version), and no issues there. Had a couple servers on Ruby 2.0, but 2 servers (vs 30) so if there are issues, may not have noticed yet.

Ruby 1.8 is the only place I have seen issues with this so far.

[conathan]

As of today, this has also impacted a Ruby 2.0.0p648 server running Rails 3.2.21, and AlmaLinux 8.6.

(As of the last ticket I just reverted to 6.0.14 for the ruby1.8 systems, as was not sure where to start in diagnosing this). Did notice that the latest passenger 6.0.15 has a glibc long long support patch though that may not be in the rpm's we generated for ruby 1.8/2.5 native library support (Our environment uses passenger 6.0.15 rpm from the passenger repo + our own rpm's for passenger_native_library.so)

App 2725846 output: Error: The application encountered the following error: 1688 out of char range (RangeError)
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:119:in `chr'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:119:in `string'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:78:in `value'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:92:in `block in hash'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:140:in `repeat_until'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:88:in `hash'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:74:in `object'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:78:in `value'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:92:in `block in hash'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:140:in `repeat_until'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:88:in `hash'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:74:in `object'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:64:in `parse'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/utils/json.rb:43:in `parse'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:74:in `block in read_startup_arguments'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:73:in `open'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:73:in `read_startup_arguments'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:35:in `init'
App 2725846 output:     /usr/share/ruby/vendor_ruby/phusion_passenger/preloader_shared_helpers.rb:38:in `init'
App 2725846 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:184:in `'
App 2725846 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `'
App 2725846 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `'
[ E 2022-09-29 04:56:49.9160 2522748/T3c7 age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /home/webapps/jigsy/current: The application encountered the following error: 1688 out of char range (RangeError)
  Error ID: 383e4c1b
  Error details saved to: /tmp/passenger-error-qa6D3F.html

[CamJN]

Got a report of ruby 3, i'm starting to suspect that the change to the jsoncpp lib was to blame.

[theirix]

Same problem with Passenger 6.0.15 (Apache integration) with Ruby 2.5 on 64-bit Debian 10.

apache 2.4.38-3
ruby 2.5.5p157 (2019-03-15 revision 67260) [x86_64-linux-gnu]

[CamJN]

I've had issues trying to reproduce the bug myself so could you test if this issue is fixed in 6.0.16, you can get the pre-release packages from our testing servers by adjusting your repo definition file to use:

for debian10 replace the contents of passenger's apt source list file with: deb https://oss-binaries.phusionpassenger.com/apt/passenger-testing buster main
for alma 8 just grab the new repo definition file from: https://oss-binaries.phusionpassenger.com/yum/definitions/el-passenger-testing.repo

[conathan]

I'll run it out to our servers, and see if it happens again. (could take a couple days, but plan to hit it with a bunch of systemctl reload httpd's as that and startup are the trigger points).

Most likely tomorrow morning will be when we do this (to give more coverage time during normal business hours if things don't go well).

[conathan]

I triggered 10 systemctl reload httpd's across around 39 servers. (390 reload attempts)

When I ran into this originally, seemed like 1-2 servers would die a night (every 1-2 days), triggered on a systemctl reload httpd (so a 1 in 39 ish odds). I couldn't reproduce this on 6.0.16 though (Possibly resolved).

AlmaLinux 8.7

[CamJN]

@conathan thanks for checking.

[CamJN]

6.0.16 is out now and should have this fixed, thanks so much for your patience.