Question: debug failed backends checks

[NOmri]

Lately we've upgrade our orchestrator instance to ubuntu18 with version 3.1.4
since then we experience a lot of failed checked to the backends which getting sorted out in one of the next checks.
the backends are up and the failures of the checks seems random.
this creates a-lot of confusion because checking the ui makes it impossible to know what is the status of the clusters

How can we debug it further to understand why the checks are failing?
attached orchestrator conf (user and pass removed)
orc.conf.txt

Thanks!

[shlomi-noach]

First, please run with --debug --stack and see if you spot any interesting error messages.

Next, do you perhaps have a low setting for open file limit (ulimit -n)? If you have many servers in your topologies, then consider increasing nofile to some higher vlaue, e.g. 8192

Last, let's try increasing some connection timeouts? Possibly your network times out. The default values are actually pretty permissive, but worth testing.

• MySQLConnectTimeoutSeconds: default 2s
• MySQLDiscoveryReadTimeoutSeconds: default 10s

[NOmri]

Thanks! i will check it out

[NOmri]

Thanks Shlomi, although ulimit -n is 63536
The service itself had only 1024-4096 increasing it to 8192 seems to solve the issue.
Thanks!