Ruby 2.5 stream closed in another thread

[vholer]

Description
On Ruby 2.5.x systems (Ubuntu 18.04 + 18.10, Debian 10), we might experience a strange behaviour when running several commands in parallel inside different threads.

For example:

$ onehost sync -f
#<Thread:0x00007f32c400bda8@/usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:348 run> terminated with exception (report_on_exception is true): 
Traceback (most recent call last): 
3: from /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:349:in `block (2 levels) in sync' 
2: from /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:349:in `loop' 
1: from /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:372:in `block (3 levels) in sync' 
/usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:372:in ``': stream closed in another thread (IOError) 
#<Thread:0x00007f32c400bb78@/usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:348 run> terminated with exception (report_on_exception is true): 
Traceback (most recent call last): 
3: from /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:349:in `block (2 levels) in sync' 
2: from /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:349:in `loop' 
1: from /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:372:in `block (3 levels) in sync' 
/usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:372:in ``': stream closed in another thread (IOError) 
#<Thread:0x000055f12fdb2358@/usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:348 run> terminated with exception (report_on_exception is true): 
Traceback (most recent call last): 
3: from /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:349:in `block (2 levels) in sync' 
2: from /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:349:in `loop' 
1: from /usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:372:in `block (3 levels) in sync' 
/usr/lib/one/ruby/cli/one_helper/onehost_helper.rb:372:in ``': stream closed in another thread (IOError) 
...

Might be related to this bugfix, which is not applied to Ruby 2.5.x:

• https://svn.ruby-lang.org/cgi-bin/viewvc.cgi?view=revision&revision=63216

Details

• Affected Component:
• Version: 5.8.1, master

Progress Status

• Branch created
• Code committed to development branch
• Testing - QA
• Documentation
• Release notes - resolved issues, compatibility, known issues
• Code committed to upstream release/hotfix branches
• Documentation committed to upstream release/hotfix branches

[vholer]

A dirty workaround for this particular command action is:

--- a/src/cli/one_helper/onehost_helper.rb
+++ b/src/cli/one_helper/onehost_helper.rb
@@ -369,9 +369,20 @@ class OneHostHelper < OpenNebulaHelper::OneHelper
                                    " #{host['NAME']}:#{remote_dir} 2> /dev/null"
                     end

-                    `#{sync_cmd} 2>/dev/null`
+                    retries = 3
+
+                    begin
+                        `#{sync_cmd} 2>/dev/null`
+                    rescue IOError => err
+                        # workaround for unpatched Ruby 2.5
+                        # https://bugs.ruby-lang.org/projects/ruby-trunk/repository/trunk/revision?utf8=%E2%9C%93&rev=63216
+                        if (retries -= 1) > 0
+                            sleep 1
+                            retry
+                        end
+                    end

-                    if !$CHILD_STATUS.success?
+                    if $CHILD_STATUS.nil? || !$CHILD_STATUS.success?
                         error_lock.synchronize do
                             host_errors << host['NAME']
                         end

(More parts could be affected. And, also it's not possible to do a command retry in most cases.)

[vholer]

This must be addressed by 5.8.2 for sure, the patch above was provided to the customer.

[vholer]

PRs to merge:

• code - B #3229: Retry onehost sync on IOError #3309 into one-5.8, cherry-pick into master
• docs - B OpenNebula/one#3229: Include in 5.8.2 release notes docs#612 into one-5.8-maintenance