gRPC query waits indefinitely while execution giving no output

[TechDevCode]

Hi,
I am trying to run example from https://github.com/SeldonIO/seldon-core/blob/master/notebooks/kubectl_demo_gcp.ipynb for gRPC on google cloud kubernetes cluster. I am able to get token but gRPC query waits indefinitely. Finally I get following error :

{"access_token":"60b9ad42-90b9-4bb8-b90a-12308e35e584","token_type":"bearer","expires_in":42569,"scope":"read write"}
Traceback (most recent call last):
File "test_request.py", line 53, in
grpc_request()
File "test_request.py", line 50, in grpc_request
response = stub.Predict(request=request,metadata=metadata)
File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 487, in call
return _end_unary_response_blocking(state, call, False, deadline)
File "/usr/local/lib/python2.7/dist-packages/grpc/_channel.py", line 437, in _end_unary_response_blocking
raise _Rendezvous(state, None, None, deadline)
grpc._channel._Rendezvous: <_Rendezvous of RPC that terminated with (StatusCode.UNAVAILABLE, OS Error)>

Whats the issue?

[ukclivecox]

Are you sure you waited until seldon-apiserver is running before running the python code that setups up grpc and REST function. It runs:

SELDON_API_IP=getoutput("kubectl get svc -n "+NAMESPACE+" seldon-apiserver -o jsonpath='{.status.loadBalancer.ingress[0].ip}'")

which will only work ones an IP address is provided for a load balancer on GCP.

[TechDevCode]

I get the external IP for seldon-apiserver load balancer. Then only I run gRPC query, but still I am only able to get only token and gRPC query waits indefinitely. I have also tried with https://github.com/SeldonIO/seldon-core/tree/master/notebooks/resources/model_grpc.json example deployment file. Still I get same results. I think the problem occurs at authentication step in kubernetes, else grpc works fine while doing docker test.

[ukclivecox]

Closing as old issue. Please reopen if still applies to latest Seldon Core.