Add grpc client-side reconnection

[aledbf]

In case of connection errors (pod restart), reconnect.

Example error:

{"@type":"type.googleapis.com/google.devtools.clouderrorreporting.v1beta1.ReportedErrorEvent","error":"invalid ref: rpc error: code = Unavailable desc = error reading from server: read tcp 10.0.128.131:55298-\u003e172.20.175.142:8080: read: no route to host:\n    github.com/gitpod-io/gitpod/registry-facade/pkg/registry.(*RemoteSpecProvider).GetSpec\n        github.com/gitpod-io/gitpod/registry-facade/pkg/registry/imagecfg.go:55\n  - invalid 

How to test:

• start a workspace and close it.
• kill ws-manager pod
• open a new workspace.

Right now we only see errors in the log (and "Pulling container image …")

[codecov]

Codecov Report

Merging #4598 (633b4ab) into main (b6eaadb) will decrease coverage by 20.47%.
The diff coverage is 0.00%.

❗ Current head 633b4ab differs from pull request most recent head 0b6923c. Consider uploading reports for the commit 0b6923c to get more accurate results

@@             Coverage Diff             @@
##             main    #4598       +/-   ##
===========================================
- Coverage   36.15%   15.68%   -20.48%     
===========================================
  Files          14        9        -5     
  Lines        3864     1760     -2104     
===========================================
- Hits         1397      276     -1121     
+ Misses       2349     1459      -890     
+ Partials      118       25       -93     

Flag	Coverage Δ
components-blobserve-app	28.40% <ø> (?)
components-registry-facade-app	11.52% <0.00%> (?)
components-registry-facade-lib	11.52% <0.00%> (?)
components-ws-manager-app	?

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
components/registry-facade/pkg/registry/blob.go	0.00% <ø> (ø)
...omponents/registry-facade/pkg/registry/imagecfg.go	0.00% <ø> (ø)
...onents/registry-facade/pkg/registry/layersource.go	28.80% <ø> (ø)
...omponents/registry-facade/pkg/registry/manifest.go	9.85% <ø> (ø)
...omponents/registry-facade/pkg/registry/registry.go	0.00% <0.00%> (ø)
components/ws-manager/pkg/manager/annotations.go
components/ws-manager/pkg/manager/probe.go
components/ws-manager/pkg/manager/create.go
...-manager/pkg/manager/internal/workpool/workpool.go
components/ws-manager/pkg/manager/manager_ee.go
... and 18 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update b6eaadb...0b6923c. Read the comment docs.

[csweichel]

Removing this is fine because

• we never had GRPC_GO_RETRY set, or because
• some other option is taking over the functionality this config provided?

[aledbf]

some other option is taking over the functionality this config provided?

The changes in the PR introduce three changes:

• grpc.KeepaliveEnforcementPolicy

Allow more frequent pings. Default is 5 minutes https://github.com/grpc/grpc-go/blob/87eb5b7502493f758e76c4d09430c0049a81a557/internal/transport/defaults.go#L40
This implies that more frequent pings result in an error HTTP/2 error code: ENHANCE_YOUR_CALM Received Goaway too_many_pings

• grpc.WithKeepaliveParams
  Adjust keepalive ping interval to avoid the default https://github.com/grpc/grpc-go/blob/87eb5b7502493f758e76c4d09430c0049a81a557/internal/transport/defaults.go#L35
  and quickly to restart of pods. Also, enables PermitWithoutStream keepalive pings even with no active RPCs

[aledbf]

we never had GRPC_GO_RETRY set, or because

Good point. This env variable was configured only in ws-manager. Now is present in all components with grpc