Custom timeouts might not be used with REST api

[Villux]

This is my seldon configuration:

apiVersion: machinelearning.seldon.io/v1alpha2
kind: SeldonDeployment
metadata:
    labels:
        app: seldon
    name: "model-{{workflow.parameters.model-id}}"
spec:
    annotations:
        deployment_version: v1
        project_name: Cat Example Pytorch
        seldon.io/rest-read-timeout: '10000000000'
        seldon.io/rest-connection-timeout: '10000000000'
        seldon.io/rest-connect-retries: "1"
        seldon.io/grpc-read-timeout: "10000000000"
...

With these super long timeouts I still get '{\n "code": 203,\n "info": "Failed to retrieve predictions after 3 attempts",\n "reason": "Microservice error",\n "status": "FAILURE"\n}' from my client.

Pod containing the rest api has this logging

i.s.engine.config.AnnotationsConfig      : Annotations {kubernetes.io/config.source=api, kubernetes.io/config.seen=2019-09-18T14:26:47.596964209Z, prometheus.io/path=prometheus, prometheus.io/port=8000, prometheus.io/scrape=true}
i.seldon.engine.tracing.TracingProvider  : Not activating tracing
i.s.e.service.InternalPredictionService  : REST Connection timeout set to 200
i.s.e.service.InternalPredictionService  : REST read timeout set to 5000
i.s.e.service.InternalPredictionService  : gRPC max message size set to 4194304
i.s.e.service.InternalPredictionService  : gRPC read timeout set to 5000
i.s.e.service.InternalPredictionService  : REST retries set to 3
io.seldon.engine.grpc.SeldonGrpcServer   : FOUND env var [ENGINE_SERVER_GRPC_PORT], will use engine server port 5001

spring outputs this error: org.springframework.web.client.ResourceAccessException: I/O error on POST request for "http://url.kubeflow.svc.cluster.local.:9000/predict": Read timed out; nested exception is java.net.SocketTimeoutException: Read timed out

If I read it correctly it's not using the timeout values that I have specified. Do you have any idea why it might be so? I'm pretty sure that I have a configuration mistake, but I can't find anything from the document that would differ from this.

[ukclivecox]

Should be fixed in 1.1 with new executor.