kafka server with seldon protocol and grpc transport sends requests to REST endpoint, not gRPC

[aberndsen]

Describe the bug

The SeldonDeployment is routing kafka messages to the REST endpoint even though the transport defined as gRPC.

We were following the various Seldon + Kafka examples, but notably none are using the protocol: seldon, transport: grpc and serverType: kafka combination.

Our SeldonDeployment with manifest-snippet:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
...
spec:
  name: object-detector
  protocol: seldon
  transport: grpc
  serverType: kafka
  predictors:
    - name: default
      componentSpecs:
        - spec:
            containers:
              - name: default
...

deploys okay and we can send gRPC requests directly to the service through port 5000.

Tailing the seldon-container-engine logs gives messages like:

2022-05-17T11:50:03.783Z        INFO    entrypoint.maxprocs     maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
2022-05-17T11:50:03.783Z        INFO    entrypoint      Hostname unset will use localhost
...
2022-05-17T11:50:03.786Z        INFO    entrypoint      Starting        {"worker": 10}
2022-05-17T11:50:03.787Z        INFO    entrypoint      Starting kafka server
2022-05-17T11:50:03.787Z        INFO    entrypoint      Start http kafka graph
...
2022-05-17T11:56:27.344Z        DEBUG   JSONRestClient  Calling HTTP    {"URL": "http://localhost:9000/predict"}
2022-05-17T11:56:27.347Z        INFO    JSONRestClient  httpPost failed {"response code": 500}
2022-05-17T11:56:27.347Z        ERROR   entrypoint.KafkaServer  Failed prediction       {"error": "Internal service call from executor failed calling http://localhost:9000/predict status code 500"}

The Start http kafka graph INFO log suggests the executor/api/kafka/server.go is creating a rest.NewJSONRestclient instead of a seldon.NewSeldonGrpcClient`, as on server.go#L71.

The seldon-container-engine log entry
2022-05-17T11:56:27.344Z DEBUG JSONRestClient Calling HTTP {"URL": "http://localhost:9000/predict"}
confirms that the kafka consumer is using the JSONRestClient instead of the Grpc one.

Looking through the seldon-core code, I think this can be traced to the operator/controllers/seldondeployment_engine.go#L187.

The createExecutorContainer extracts the protocol and serverType from the deployment spec, but not the transport.

Finally, the main executor uses the default transport, set to rest instead of having the spec passed-through.

Expected behaviour

gRPC messages serialized to a kafka topic should be forwarded to a gRPC client, not the JSON REST one.

Environment

• Cloud Provider: AWS and k3d

• Kubernetes Cluster Version

   > kubectl version
   Client Version: version.Info{Major:"1", Minor:"23", GitVersion:"v1.23.2", GitCommit:"9d142434e3af351a628bffee3939e64c681afa4d", 
   GitTreeState:"clean", BuildDate:"2022-01-19T17:35:46Z", GoVersion:"go1.17.5", Compiler:"gc", Platform:"linux/amd64"}
   Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.7+k3s1", 
   GitCommit:"8432d7f239676dfe8f748c0c2a3fabf8cf40a826", GitTreeState:"clean", BuildDate:"2022-02-24T23:03:47Z", 
   GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}

• Deployed Seldon System Images:

  > kubectl get --namespace seldon-system deploy seldon-controller-manager -o yaml  | grep seldonio
  value: docker.io/seldonio/seldon-core-executor:1.13.1
  image: docker.io/seldonio/seldon-core-operator:1.13.1

• Logs of your model (tail, upon publishing a gRPC-encoded message):

  2022-05-17 16:42:13,047 - seldon_core.wrapper:handle_generic_exception:53 - ERROR:  {'status': {'status': 1, 'info': "400 Bad   Request: Failed to decode JSON object: 'utf-8' codec can't decode byte 0xc2 in position 1: invalid continuation byte", 'code': -1, 'reason': 'MICROSERVICE_INTERNAL_ERROR'}}
  2022-05-17 16:42:13,047 - werkzeug:_log:225 - INFO:  127.0.0.1 - - [17/May/2022 16:42:13] "POST /predict HTTP/1.1" 500 -