v1.1.0 - Incorrect response from /healthz endpoint

[ben-staked]

🐞 Bug Report

Description

When accessing the /healthz monitoring endpoint of a healthy beacon node with TLS enabled, it will always report an error for the gateway, e.g.

*powchain.Service: OK
*sync.Service: OK
*gateway.Gateway: ERROR grpc server is TRANSIENT_FAILURE
*prometheus.Service: OK
*p2p.Service: OK
*attestations.Service: OK
*blockchain.Service: OK
*initialsync.Service: OK
*rpc.Service: OK

This only occurs when TLS is enabled on a beacon node; if TLS is not enabled on the beacon node, it will correctly report on the status of the gateway.

Has this worked before in a previous version?

Have not tested in a prior version.

🔬 Minimal Reproduction

Run two different beacon nodes - one with TLS enabled and one with TLS disabled:

beacon-chain --accept-terms-of-use --tls-cert $TLS_CERT --tls-key $TLS_KEY --http-web3provider=$WEB3_PROVIDER --pyrmont

and

beacon-chain --accept-terms-of-use --http-web3provider=$WEB3_PROVIDER --pyrmont

The node running with TLS enabled should fail to return a passing status for the gateway even when otherwise healthy, while the node without TLS will be able to return a passing status once healthy.

🔥 Error

*gateway.Gateway: ERROR grpc server is TRANSIENT_FAILURE

🌍 Your Environment

Operating System:

  
Ubuntu 18.04
  

What version of Prysm are you running? (Which release)

  
v1.1.0
  

Anything else relevant (validator index / public key)?
There doesn't appear to be any connection to validator or public keys.

[prestonvanloon]

Looks like gRPC gateway does not dial the gRPC service with the TLS specified. It seems to always use the insecure protocol:

prysm/beacon-chain/gateway/gateway.go

Lines 162 to 165 in 04615cb

	opts := []grpc.DialOption{
	grpc.WithInsecure(),
	grpc.WithDefaultCallOptions(grpc.MaxCallRecvMsgSize(int(g.maxCallRecvMsgSize))),
	}

[prestonvanloon]

This is fixed in #8418. If you aren't using the gRPC gateway for JSON requests, you may disable it entirely to unblock yourself until that PR can be included in the next release. --disable-grpc-gateway