Catch EBADF when watch_stream fails

[zeari]

Solves https://bugzilla.redhat.com/show_bug.cgi?id=1507528
TBH I think we can catch any exception there and not IOError, EBADF specifically.

Explanation:
until finish is called, the code is looping in response.body.each do |chunk|.
finish closes @http_client while the other thread is reading from it so there could be any exception there depending where exactly the other thread is.

This isnt a great solution but the bug is just an error in the log and isnt an actual usability issue
@cben @agrare @moolitayer Please review

[moolitayer]

@zeari please add a test for the fix done here

[zeari]

@zeari please add a test for the fix done here

I have no clue how to test that

[agrare]

I have no clue how to test that

Could you add a test here https://github.com/abonas/kubeclient/blob/master/test/test_watch.rb which stubs the request and raises EBADF?

[agrare]

the bug is just an error in the log and isnt an actual usability issue

@zeari I thought you said this was preventing your event catcher worker from shutting down and causing it to get hung up?

[zeari]

@zeari I thought you said this was preventing your event catcher worker from shutting down and causing it to get hung up?

I thought so but now i see its just miq taking time shutting down workers.

[agrare]

Ah okay, still good to catch this cleanly 👍

[zeari]

@moolitayer @agrare Added specs

[agrare]

LGTM

[moolitayer]

nice trick LGTM 👍

[cben]

LGTM

[zeari]

TBH I think we can catch any exception there and not IOError, EBADF specifically.

This issue reared its ugly head again but this time as HTTP::ConnectionError instead of Errno::EBADF:

[----] E, [2018-04-09T14:52:33.790859 #21326:2245430] ERROR -- : MIQ(ManageIQ::Providers::Openshift::ContainerManager::EventCatcher::Runner#start_event_monitor) EMS [vm-48-38.eng.lab.tlv.redhat.com] as [] Event Monitor Thread aborted because [error reading from socket: Bad file descriptor]
[----] E, [2018-04-09T14:52:33.791242 #21326:2245430] ERROR -- : [HTTP::ConnectionError]: error reading from socket: Bad file descriptor  Method:[block in method_missing]
[----] E, [2018-04-09T14:52:33.791371 #21326:2245430] ERROR -- : .../.rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/openssl/buffering.rb:125:in `sysread'
.../.rvm/rubies/ruby-2.3.3/lib/ruby/2.3.0/openssl/buffering.rb:125:in `readpartial'
.../.rvm/gems/ruby-2.3.3/gems/http-2.2.2/lib/http/timeout/null.rb:44:in `readpartial'
.../.rvm/gems/ruby-2.3.3/gems/http-2.2.2/lib/http/connection.rb:212:in `read_more'
.../.rvm/gems/ruby-2.3.3/gems/http-2.2.2/lib/http/connection.rb:87:in `readpartial'
.../.rvm/gems/ruby-2.3.3/gems/http-2.2.2/lib/http/response/body.rb:29:in `readpartial'
.../.rvm/gems/ruby-2.3.3/gems/http-2.2.2/lib/http/response/body.rb:34:in `each'
.../.rvm/gems/ruby-2.3.3/gems/kubeclient-2.5.2/lib/kubeclient/watch_stream.rb:24:in `each'
.../.rvm/gems/ruby-2.3.3/bundler/gems/manageiq-providers-kubernetes-24f1e8999fa7/app/models/manageiq/providers/kubernetes/container_manager/kubernetes_event_monitor.rb:32:in `each'
.../.rvm/gems/ruby-2.3.3/bundler/gems/manageiq-providers-kubernetes-24f1e8999fa7/app/models/manageiq/providers/kubernetes/container_manager/event_catcher_mixin.rb:40:in `monitor_events'
.../Projects/manageiq/app/models/manageiq/providers/base_manager/event_catcher/runner.rb:161:in `block in start_event_monitor'

Can we just ignore exceptions when @finished == true ? @cben @moolitayer @agrare

[cben]

Yeah, I think

rescue StandardError
  raise unless @finished

would be good.
(all errors we saw till now fall under StandardError, and exceptions outside it are mostly there for good reason, e.g. Interrupt raised on Ctrl+C we shouldn't mask. see hierarchy: https://ruby-doc.org/core/Exception.html)