openvino_imagenet_ensemble example - prediction1 and prediction2 containers error

[nareshganesan]

Hi There,

I'm trying to create the following example in our k8s cluster setup on google cloud (Not the managed k8s cluster).

https://docs.seldon.io/projects/seldon-core/en/stable/examples/openvino_ensemble.html

I get error for prediction1 & prediction2 containers.

Individual container status

  imagenet-itransformer:
      Ready:          True
  imagenet-otransformer:
      Ready:          True
  imagenet-combiner:
      Ready:          True
  prediction1:
    State:          Waiting
      Reason:       CrashLoopBackOff
      Ready:          False
  prediction2:
    State:          Waiting
      Reason:       CrashLoopBackOff
    Ready:          False
  seldon-container-engine:
    State:          Running
      Started:      Thu, 23 May 2019 04:49:13 +0000
    Ready:          False        

Prediction1 container logs

kubectl -n seldon logs -f pod/openvino-openvino-b6fd84f-85bdc798db-jjdff -c prediction1
starting microservice
2019-05-23 05:05:34.764794: F tensorflow/python/lib/core/bfloat16.cc:675] Check failed: PyBfloat16_Type.tp_base != nullptr

Prediction2 container logs

kubectl -n seldon logs -f pod/openvino-openvino-b6fd84f-85bdc798db-jjdff -c prediction2
starting microservice
2019-05-23 05:10:39.112946: F tensorflow/python/lib/core/bfloat16.cc:675] Check failed: PyBfloat16_Type.tp_base != nullptr 

seldon-container-engine logs

...
2019-05-23 04:49:28.579  INFO 1 --- [           main] i.seldon.engine.tracing.TracingProvider  : Activating tracing
2019-05-23 04:49:28.753  INFO 1 --- [           main] i.s.e.service.InternalPredictionService  : REST Connection timeout set to 200
2019-05-23 04:49:28.753  INFO 1 --- [           main] i.s.e.service.InternalPredictionService  : REST read timeout set to 5000
2019-05-23 04:49:29.597  INFO 1 --- [           main] i.s.e.service.InternalPredictionService  : gRPC max message size set to 4194304
2019-05-23 04:49:29.597  INFO 1 --- [           main] i.s.e.service.InternalPredictionService  : Setting grpc read timeout to 100000ms
2019-05-23 04:49:29.597  INFO 1 --- [           main] i.s.e.service.InternalPredictionService  : gRPC read timeout set to 100000
2019-05-23 04:49:29.597  INFO 1 --- [           main] i.s.e.service.InternalPredictionService  : REST retries set to 3
2019-05-23 04:49:29.744  INFO 1 --- [           main] io.seldon.engine.grpc.SeldonGrpcServer   : FOUND env var [ENGINE_SERVER_GRPC_PORT], will use engine server port 5001
2019-05-23 04:49:30.194  INFO 1 --- [cTaskExecutor-1] io.seldon.engine.grpc.SeldonGrpcServer   : Starting grpc server
2019-05-23 04:49:30.794  INFO 1 --- [cTaskExecutor-1] io.seldon.engine.grpc.SeldonGrpcServer   : Server started, listening on 5001
...
...
2019-05-23 04:49:37.117  WARN 1 --- [nio-8082-exec-6] i.s.e.api.rest.RestClientController      : Not ready graph checker false, controller true
2019-05-23 04:49:38.116  WARN 1 --- [nio-8082-exec-7] i.s.e.api.rest.RestClientController      : Not ready graph checker false, controller true
2019-05-23 04:49:39.121  WARN 1 --- [nio-8082-exec-8] i.s.e.api.rest.RestClientController      : Not ready graph checker false, controller true
2019-05-23 04:49:40.115  WARN 1 --- [io-8082-exec-10] i.s.e.api.rest.RestClientController      : Not ready graph checker false, controller true
2019-05-23 04:49:41.116  WARN 1 --- [nio-8082-exec-1] i.s.e.api.rest.RestClientController      : Not ready graph checker false, controller true
2019-05-23 04:49:41.211  WARN 1 --- [pool-2-thread-1] i.s.e.api.rest.SeldonGraphReadyChecker   : Failed to connect to localhost:9003
2019-05-23 04:49:41.211  WARN 1 --- [pool-2-thread-1] i.s.e.api.rest.SeldonGraphReadyChecker   : Failed to connect to localhost:9003
2019-05-23 04:49:41.211  WARN 1 --- [pool-2-thread-1] i.s.e.api.rest.SeldonGraphReadyChecker   : Failed to connect to localhost:9003
2019-05-23 04:49:41.212  WARN 1 --- [pool-2-thread-1] i.s.e.api.rest.SeldonGraphReadyChecker   : Failing localhost:9003
2019-05-23 04:49:41.212  INFO 1 --- [pool-2-thread-1] i.s.e.api.rest.SeldonGraphReadyChecker   : prediction1 not ready!

Any suggestions will really helpful. Thanks!

[ukclivecox]

This looks like a version issue. They reference here a numpy upgrade. The example was upgraded recently but maybe a versioning bug was introduced.

[ukclivecox]

Any thought @dtrawins

[nareshganesan]

Update: I was able to get the same example running, in the same cluster. I'm using the same seldon_ov_predict_ensemble.json for the seldon deployment given in the example. here

[nareshganesan]

Closing the issue. Please feel free to reopen if the issue occurs again.