Seldon Azure Deep Mnist tutorial CrashLoopBackOff while creating pods

[amanagarwal97]

Describe the bug

When following the tutorial and deploying everything to Azure Kubernetes Cluster the pod fails continuously

To reproduce

Just follow the tutorial here https://docs.seldon.io/projects/seldon-core/en/v1.1.0/examples/azure_aks_deep_mnist.html

Expected behaviour

Pods get created and service is deployed.

Environment

• Cloud Provider: Azure
• Kubernetes Cluster Version:

Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.0", GitCommit:"9e991415386e4cf155a24b1da15becaa390438d8", GitTreeState:"clean", BuildDate:"2020-03-25T14:58:59Z", GoVersion:"go1.13.8", Compiler:"gc", Platform:"windows/amd64"}  Server Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.11", GitCommit:"ec831747a3a5896dbdf53f259eafea2a2595217c", GitTreeState:"clean", BuildDate:"2020-05-29T19:56:10Z", GoVersion:"go1.12.17", Compiler:"gc", Platform:"linux/amd64"}

• Deployed Seldon System Images:

value: docker.io/seldonio/engine:1.2.0
value: docker.io/seldonio/seldon-core-executor:1.2.0
image: docker.io/seldonio/seldon-core-operator:1.2.0

Model Details

• Images of your model:

{"apiVersion":"machinelearning.seldon.io/v1alpha2","kind":"SeldonDeployment","metadata":{"annotations":{},"labels":{"app":"seldon"},"name":"deep-mnist","namespace":"seldon"},"spec":{"annotations":{"deployment_version":"v1","project_name":"Tensorflow MNIST"},"name":"deep-mnist","oauth_key":"oauth-key","oauth_secret":"oauth-secret","predictors":[{"annotations":{"predictor_version":"v1"},"componentSpecs":[{"spec":{"containers":[{"image":"seldonacrdemo.azurecr.io/deep-mnist:0.1","imagePullPolicy":"IfNotPresent","name":"classifier","resources":{"requests":{"memory":"1Mi"}}}],"terminationGracePeriodSeconds":20}}],"graph":{"children":[],"endpoint":{"type":"REST"},"name":"classifier","type":"MODEL"},"name":"single-model","replicas":1}]}}
  name: deep-mnist
  selfLink: /apis/machinelearning.seldon.io/v1/namespaces/seldon/seldondeployments/deep-mnist
  name: deep-mnist
        - image: seldonacrdemo.azurecr.io/deep-mnist:0.1
    url: http://deep-mnist-single-model.seldon.svc.cluster.local:8000/api/v1.0/predictions
    deep-mnist-single-model-0-classifier:
    deep-mnist-single-model-classifier:
      httpEndpoint: deep-mnist-single-model-classifier.seldon:9000
      svcName: deep-mnist-single-model-classifier

• Logs of your model:

starting microservice
2020-06-29 12:21:27,522 - seldon_core.microservice:main:190 - INFO:  Starting microservice.py:main
2020-06-29 12:21:27,523 - seldon_core.microservice:main:246 - INFO:  Parse JAEGER_EXTRA_TAGS []
2020-06-29 12:21:27,523 - seldon_core.microservice:load_annotations:114 - INFO:  Found annotation deployment_version:"v1"
2020-06-29 12:21:27,524 - seldon_core.microservice:load_annotations:114 - INFO:  Found annotation kubernetes.io/config.seen:"2020-06-29T12:15:08.66879713Z"
2020-06-29 12:21:27,524 - seldon_core.microservice:load_annotations:114 - INFO:  Found annotation kubernetes.io/config.source:"api"
2020-06-29 12:21:27,524 - seldon_core.microservice:load_annotations:114 - INFO:  Found annotation predictor_version:"v1"
2020-06-29 12:21:27,524 - seldon_core.microservice:load_annotations:114 - INFO:  Found annotation project_name:"Tensorflow MNIST"
2020-06-29 12:21:27,524 - seldon_core.microservice:load_annotations:114 - INFO:  Found annotation prometheus.io/path:"/prometheus"
2020-06-29 12:21:27,524 - seldon_core.microservice:load_annotations:114 - INFO:  Found annotation prometheus.io/scrape:"true"
2020-06-29 12:21:27,524 - seldon_core.microservice:load_annotations:114 - INFO:  Found annotation v1:""
2020-06-29 12:21:27,524 - seldon_core.microservice:main:257 - INFO:  Annotations: {'deployment_version': '"v1"', 'kubernetes.io/config.seen': '"2020-06-29T12:15:08.66879713Z"', 'kubernetes.io/config.source': '"api"', 'predictor_version': '"v1"', 'project_name': '"Tensorflow MNIST"', 'prometheus.io/path': '"/prometheus"', 'prometheus.io/scrape': '"true"', 'v1': '""'}
2020-06-29 12:21:27,524 - seldon_core.microservice:main:261 - INFO:  Importing DeepMnist
Traceback (most recent call last):
  File "/opt/conda/bin/seldon-core-microservice", line 11, in <module>
    load_entry_point('seldon-core', 'console_scripts', 'seldon-core-microservice')()
  File "/microservice/python/seldon_core/microservice.py", line 262, in main
    interface_file = importlib.import_module(args.interface_name)
  File "/opt/conda/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'DeepMnist'

[ukclivecox]

It just not finding DeepMnist.py
Can you run the docker image locally interactively with bash and check the DeepMnist.py exists at the top level?

[amanagarwal97]

Thanks. Issue got fixed