Various bugs have been found in 'OpenShiftConnector' while fixing "Error while pumping stream" issue

[ibuziuk]

See https://issues.jboss.org/browse/CHE-180
NOTE: The original "Error while pumping stream" issue was fixed, but @snjeza found a few other issues during investigation - https://issues.jboss.org/browse/CHE-180?focusedCommentId=13396715&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-13396715

• getEvents does nothing. I have replaced it with openShiftClient.events().inNamespace().list().getItems()
• I have replaced LogWatch with getLog in getContainerLogs. The openshift connector completely loads the workspace's log into memory, but, since those logs are currently empty, that isn't a problem. We will be able to use LogWatch when Issue with LogWatch  fabric8io/kubernetes-client#735 is fixed
  the Kubernetes client sets the default ping interval to 1s. See https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/Config.java#L97
  The ping interval can cause issues when the listener's onMessage method takes longer. InputStreamPumper throws "Pipe not exception" in that case
• The solution is to set the ping interval to 0 that will disable pinging which will also, I think, enhance performance.
  See Problem with ping interval square/okhttp#3297
• there is a bug in https://github.com/fabric8io/kubernetes-client/blob/master/kubernetes-client/src/main/java/io/fabric8/kubernetes/client/dsl/base/OperationSupport.java#L287. If a response is null, the Kubernetes client will throw an NPE and ExecWebSocketListener.onFailure won't close its listener properly. See OperationSupport.createStatus() can throw a NPE occasionally fabric8io/kubernetes-client#733
• currently, OpenShiftClient creates a new okhttpclient whenever it calls a method which is not recommended. See https://github.com/square/okhttp/blob/master/okhttp/src/main/java/okhttp3/OkHttpClient.java#L57. I think we should create OpenShiftClient only once.

[amisevsk]

Regarding

getEvents does nothing. I have replaced it with openShiftClient.events().inNamespace().list().getItems()

OpenShiftConnector#getEvents() is used only by DockerInstanceStopDetector. I've looked into making this work numerous times, and I can't really see a good, elegant way to adapt OpenShift events to the Docker events Che expects. There has been talk of improving this via the SPI (e.g. not relying on a getEvents method, and instead letting the impl side of the SPI decide how workspace health is detected), and I think postponing this issue until there is a more clear picture of how that will work is a good idea.

The issues I encountered with getEvents() are

• It seems like pod health events are only reported when the liveness probe pings the container -- this means that often, the relevant events don't happen quickly enough to be useful
• The oom event we're trying to detect is only one of various possible ways Che can crash due to oom. More often agents are killed, and getEvents() won't help us here.
• The way oom detection is done in DockerInstanceStopDetector is a bit of a workaround specific to Docker. It's hard to replicate this in OS without ugly hacks.
• The entire issue of Che oom stuff is kind of moot at this point. Che currently pings workspaces for health, and even when I managed to hack together a version of getEvents() that does what we expect, the wsagent ping request would often detect issues before the oom event ever fired. Since setting termination grace period on workspace pods to 0, the prompt that appears as a result of the wsagent ping (workspace is no longer responding, with close/restart buttons) does work, although it could be communicated better.

[l0rd]

We won't fix that in the openshiftconnector but with SPI OpenShift implementation

[ibuziuk]

@l0rd after talking with @skabashnyuk today we might have performance problems with multi-tenant che-server due to connection leaks [1] (looks like mt che-server could become unusable even for a few concurrent users)

[1] fabric8io/kubernetes-client#741

[l0rd]

@skabashnyuk can you help to reproduce that with rh-che? How many concurrent users? Is there any particular actions (like creating workspaces, executing commands...) that the concurrent users should do before we observe this behavior?

[l0rd]

Looks related to eclipse-che/che#7418

[garagatyi]

Seems that this issue is outdated.