Attempting to resolve TLS certificate chain on OpenShift repeatedly hangs for minutes if cluster uses proxy

[amisevsk]

Describe the bug

Running Che on an OpenShift cluster with proxy configured results in the che-operator repeatedly getting stuck on resolving TLS certificates. This appears to be related to calling doRequestForTLSCrtChain in this section.

The initial check, which bypasses the proxy configuration, can end up waiting minutes for a timeout to expire.

Che version

7.42@latest

Steps to reproduce

1. Start OpenShift cluster with proxy configured (if you have access to cluster-bot: launch 4.9 aws,proxy)
2. Install Che with devworkspace enabled
3. Check che-operator logs

Expected behavior

Timeout should at least be decreased to a reasonable duration (in the first 15 minutes of running che-operator, around 15 minutes are spent waiting on this timeout)

Runtime

OpenShift

Screenshots

No response

Installation method

chectl/latest

Environment

Linux

Eclipse Che Logs

(note the timestamps)

2022-01-28T16:34:26.256Z	INFO	controller-runtime.manager.controller.checluster	Starting workers	{"reconciler group": "org.eclipse.che", "reconciler kind": "CheCluster", "worker count": 1}
time="2022-01-28T16:38:48Z" level=error msg="An error occurred when reaching test TLS route: Get \"https://test-eclipse-che.apps.<snip>.com\": dial tcp 18.219.65.50:443: connect: connection timed out"
time="2022-01-28T16:38:48Z" level=warning msg="Failed to get certificate chain of trust of the OpenShift Ingress bypassing the proxy"
time="2022-01-28T16:38:48Z" level=info msg="Configuring proxy with http://<snip>.compute.amazonaws.com:3128 to extract certificate chain from the following URL: https://test-eclipse-che.apps.<snip>.com"
time="2022-01-28T16:38:48Z" level=info msg="Using proxy: http://<snip>.compute.amazonaws.com:3128 to access TLS endpoint URL: https://test-eclipse-che.apps.<snip>.com"
time="2022-01-28T16:43:14Z" level=error msg="An error occurred when reaching test TLS route: Get \"https://test-eclipse-che.apps.<snip>.com\": dial tcp 3.19.207.178:443: connect: connection timed out"
time="2022-01-28T16:43:14Z" level=warning msg="Failed to get certificate chain of trust of the OpenShift Ingress bypassing the proxy"
time="2022-01-28T16:43:14Z" level=info msg="Configuring proxy with http://<snip>.compute.amazonaws.com:3128 to extract certificate chain from the following URL: https://test-eclipse-che.apps.<snip>.com"
time="2022-01-28T16:43:14Z" level=info msg="Using proxy: http://<snip>.compute.amazonaws.com:3128 to access TLS endpoint URL: https://test-eclipse-che.apps.<snip>.com"

Additional context

Originally detected while verifying Che with an airgap cluster; reproduced with regular cluster with proxy configured.

[amisevsk]

This issue also results in chectl server:deploy taking a total of 16m35s to deploy Che server.