'Application not available' when starting a Che workspace

[sunix]

Describe the bug

Starting a workspace it happens randomly about 3 times a day in che.openshift.io

I guess there is a synch issue, che-theia may be available internally but not externally maybe?

[ibuziuk]

This is a known issue that is supposed to be addressed in the upstream #17002

[skabashnyuk]

I have some doubts about #17002 and the current issue. Theia is a special server that is tested before reported that workspace is Running.

[sunix]

I am wondering if there is a global latency between the time the server is available internally (from the same cluster) and the time when it is available externally. I have also noticed that issue when opening an app preview in Che. it is happening randomly but still quite frequently. Could we do a wait for loop from the dashboard ?

[sunix]

or is readiness probe missing?

[ibuziuk]

I have some doubts about #17002 and the current issue. Theia is a special server that is tested before reported that workspace is Running.

@skabashnyuk this might be related to route flakiness in this case and we might need to enable retries (e.g. mark theia as ready only after 3 successful checks). As I recall the retry functionality should be already in place and available for configuration

[ibuziuk]

The issue should be fixed on Hosted Che end by increasing the success threshold - redhat-developer/rh-che#2009

[sunix]

@ibuziuk would that make sense to have that value the default one in Che ?

[ibuziuk]

@sunix it depends how easy it to repro against upstream + some default infra

[ibuziuk]

TL;DR this issue is infra related and the optimal config may vary

[sunix]

I frequently have that issue on openshift 4.4

[sunix]

frequently => maybe 1/10 workspace start, but it is very random obviously.
it is just that it doesn't look like very robust when you live demo Che and have that issue.

[sunix]

it just happened to me on CodeReady Workspaces

[ibuziuk]

yeah, so we can increase it in the success threshold in the upstream, but it would result in slower performance - so it is a tradeoff.
Also, note that there will be no such an issue when single-host become a default route exposure strategy

[ibuziuk]

This is a central outage issue and if can not affect it. Closing since there was no issue reported during the periodic test summary.

[sunix]

I understand but I don't think changing threshold is the fix. The dashboard javascript client should check if the route works or not and retry.

[Katka92]

The periodic tests caught that again today.
Job and screen

[sunix]

@ibuziuk ok to reopen that one ?

[ibuziuk]

@sunix the real solution is single-host, we just need to wait till it becomes the default option. You can reopen, but I doubt that anything apart from what has been already done will be improved

[Katka92]

This issue was caught again by our periodic tests. Job and screen

[sunix]

Let's reopen it and close it when we have a proper fix (single host or anything else)

[Katka92]

I've encountered that issue again when testing CRW.

[sleshchenko]

I faced a bit different but similar issue #19059
I assume Dashboard itself can test IDE URL before opening iframe.

[ibuziuk]

@sleshchenko I put the UD label

[ibuziuk]

I've encountered that issue again when testing CRW.

@Katka92 @sleshchenko folks, isn't this problem multi-host specific (should not be reproducible against single-host configuration)?

[Katka92]

IDK, I'm not load testing with single-host config. Maybe it would be worth adding that test case to load tests, but with lower priority than multi-host. Now I'm testing config with multi-host (~100 workspace startups for now) and I haven't seen that yet, but it was seen for testing CRW 2.9 (based on Che 7.30.x). I can try with single-host if I have time and let you know.

[ibuziuk]

@skabashnyuk do you have any input on when we will have a single-host enabled by default?

[skabashnyuk]

@skabashnyuk do you have any input on when we will have a single-host enabled by default?

I don't know

[sleshchenko]

I think now single host is the default suggested exposure strategy, so it should not happen and from dashboard point of view it would be overhead. Closing it.