Crash when starting Ruby 3.3.0 Rails 7 application

[mitchellhenke]

Issue report

Fill in as much as possible so that we can understand, find and fix the problem.

Are you sure this is a bug in Passenger?
Please read our troubleshooting and debugging guides first:
https://www.phusionpassenger.com/library/admin/troubleshooting/
https://www.phusionpassenger.com/library/admin/

Please try with the newest version of Passenger to avoid issues that have already been fixed

Question 1: What is the problem?

• What is the expected behavior?

Booting a Ruby 3.3 application without errors

• What is the actual behavior?

Infrequently, passenger will crash when booting the application.

• How can we reproduce it? Please try to provide a sample application (or Virtual Machine) demonstrating the issue. Otherwise, if we can't reproduce it, we might have to ask you a number of followup questions or run certain commands to try and figure out the problem.

It does not happen every time which makes it a little difficult to track down. We didn't see the error prior to upgrading our application to Ruby 3.3.

This is the stacktrace in passenger.log:

Error: The application encountered the following error: Interrupted system call @ rb_sysopen - /tmp/passenger.spawn.XXXX0DHlFB/response/finish (Errno::EINTR)
/passenger-6.0.19/src/helper-scripts/rack-preloader.rb:29:in `<main>'
/passenger-6.0.19/src/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
/passenger-6.0.19/src/helper-scripts/rack-preloader.rb:211:in `<module:App>'
/passenger-6.0.19/src/helper-scripts/rack-preloader.rb:169:in `negotiate_spawn_command'
/passenger-6.0.19/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb:288:in `advertise_readiness'
/passenger-6.0.19/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb:288:in `open'
/passenger-6.0.19/src/ruby_supportlib/phusion_passenger/loader_shared_helpers.rb:288:in `initialize'

[ E 2024-01-23 17:12:25.0103 10559/T23 age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /app/application: The application encountered the following error: Interrupted system call @ rb_sysopen - /tmp/passenger.spawn.XXXX0DHlFB/response/finish (Errno::EINTR)
[ E 2024-01-23 17:12:25.0126 10559/T8 age/Cor/Con/CheckoutSession.cpp:281 ]: [Client 1-1] Cannot checkout session because a spawning error occurred. The identifier of the error is 874b2452. Please see earlier logs for details about the error.

We see the Interrupted system call @ rb_sysopen - /tmp/passenger.spawn.XXXX0DHlFB/response/finish (Errno::EINTR) more frequently, but sometimes the application boots successfully after that. In rare cases, it results in Could not spawn process for application and the application boot fails.

Question 2: Passenger version and integration mode:

open source 6.0.19/nginx

Question 3: OS or Linux distro, platform (including version):

Ubuntu 20.04

Question 4: Passenger installation method:

Your answer:
[ ] RubyGems + Gemfile
[x] RubyGems, no Gemfile
[ ] Phusion APT repo
[ ] Phusion YUM repo
[ ] OS X Homebrew
[ ] source tarball
[ ] Other, please specify:

Question 5: Your app's programming language (including any version managers) and framework (including versions):

Ruby 3.3.0, Rails 7.1.2

Question 6: Are you using a PaaS and/or containerization? If so which one?

No.

Question 7: Anything else about your setup that we should know?

Your answer:

---

We strive for quality and appreciate you taking the time to submit a report! Please note that if you want guaranteed response times and priority issue support we encourage you to join our enterprise customer base. They also provide us with the means to continue our high level of open source support!

[CamJN]

Your app is receiving an unhandled signal during a system call to open the file used to indicate that the app finished spawning. It is possible that Passenger sent that signal if the app took a long time to boot, but it is also possible that the signal came from elsewhere such as the oom killer or some other watchdog type process. It would be helpful if you could determine which signal number it is receiving, or what process is sending it.

[herrbuerger]

@mitchellhenke I was wondering if you figured out what was causing this issue as we are also running into it currently.

[mitchellhenke]

Sorry, no, have started migrating away from Passenger :/

[CamJN]

This should be fixed in 6.0.23

[jcrisp]

This issue does not seem to be fixed in Passenger 6.0.26 or has regressed. I am seeing it in my logs. Eg,

App 2970524 output: Error: The application encountered the following error: Interrupted system call @ rb_sysopen - /tmp/passenger.spawn.XXXXMjtufd/response/finish (Errno::EINTR)
App 2970524 output:     /usr/lib/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:288:in `initialize'
App 2970524 output:     /usr/lib/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:288:in `open'
App 2970524 output:     /usr/lib/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:288:in `advertise_readiness'
App 2970524 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:169:in `negotiate_spawn_command'
App 2970524 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:211:in `<module:App>'
App 2970524 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
App 2970524 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<main>'
[ E 2025-02-24 17:16:15.2313 2592699/Tpc age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/myapp/current: The application encountered the following error: Interrupted system call @ rb_sysopen - /tmp/passenger.spawn.XXXXMjtufd/response/finish (Errno::EINTR)
  Error ID: 16eef5df
  Error details saved to: /tmp/passenger-error-Qq70et.html

App 2971150 output: Error: The application encountered the following error: Interrupted system call @ rb_sysopen - /tmp/passenger.spawn.XXXXNSjOey/response/finish (Errno::EINTR)
App 2971150 output:     /usr/lib/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:288:in `initialize'
App 2971150 output:     /usr/lib/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:288:in `open'
App 2971150 output:     /usr/lib/ruby/vendor_ruby/phusion_passenger/loader_shared_helpers.rb:288:in `advertise_readiness'
App 2971150 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:169:in `negotiate_spawn_command'
App 2971150 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:211:in `<module:App>'
App 2971150 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:30:in `<module:PhusionPassenger>'
App 2971150 output:     /usr/share/passenger/helper-scripts/rack-preloader.rb:29:in `<main>'
[ E 2025-02-24 17:16:20.6768 2592699/Tpv age/Cor/App/Implementation.cpp:221 ]: Could not spawn process for application /var/www/myapp/current: The application encountered the following error: Interrupted system call @ rb_sysopen - /tmp/passenger.spawn.XXXXNSjOey/response/finish (Errno::EINTR)
  Error ID: 7303d011
  Error details saved to: /tmp/passenger-error-ho8WLE.html

Please investigate if it is a regression.

Are users receiving error pages if Passenger logs this error?

Thanks
James

[CamJN]

This is remarkably similar to https://bugs.ruby-lang.org/issues/20708. But the fix from that bug (https://github.com/ruby/ruby/pull/11537/files) is still present in ruby 3.4 afaict: https://github.com/ruby/ruby/blob/master/io.c#L7031

@jcrisp can you tell me a bit more about your system where this is happening? Especially anything that affects the way syscalls would behave such as distro, kernel, security software, containerization, etc.

[jcrisp]

@CamJN thanks for looking into this. I'll send you details via email, using the one listed on your silver.. domain.

[jcrisp]

Hi @CamJN, just wanted to check you got the details? I can email it to a phusionpassenger address if that would be better? Thanks.

[CamJN]

I got it, sorry for not responding yet, I've been busy.