Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cannot run through the example. #677

Closed
NetWilliam opened this issue Jul 9, 2019 · 19 comments
Closed

cannot run through the example. #677

NetWilliam opened this issue Jul 9, 2019 · 19 comments

Comments

@NetWilliam
Copy link

NetWilliam commented Jul 9, 2019

I am following the guidance of:
https://docs.seldon.io/projects/seldon-core/en/latest/workflow/README.html
https://medium.com/analytics-vidhya/manage-ml-deployments-like-a-boss-deploy-your-first-ab-test-with-sklearn-kubernetes-and-b10ae0819dfe
and
https://docs.seldon.io/projects/seldon-core/en/latest/reference/seldon-deployment.html#single-model

I have intalled the Seldon-Core in the local k8s cluster via helm, now run kubectl get pods return:

image

then I have the file seldon-dep.yaml

{
    "apiVersion": "machinelearning.seldon.io/v1alpha2",
    "kind": "SeldonDeployment",
    "metadata": {
        "labels": {
            "app": "seldon"
        },
        "name": "seldon-deployment-example"
    },
    "spec": {
        "annotations": {
            "project_name": "FX Market Prediction",
            "deployment_version": "v1"
        },
        "name": "test-deployment",
        "oauth_key": "oauth-key",
        "oauth_secret": "oauth-secret",
        "predictors": [
            {
                "componentSpecs": [{
                    "spec": {
                        "containers": [
                            {
                                "image": "seldonio/mock_classifier:1.0",
                                "imagePullPolicy": "IfNotPresent",
                                "name": "classifier",
                                "resources": {
                                    "requests": {
                                        "memory": "1Mi"
                                    }
                                }
                            }
                        ],
                        "terminationGracePeriodSeconds": 20
                    }
                }],
                "graph": {
                    "children": [],
                    "name": "classifier",
                    "endpoint": {
            "type" : "REST"
            },
                    "type": "MODEL"
                },
                "name": "fx-market-predictor",
                "replicas": 1,
        "annotations": {
            "predictor_version" : "v1"
        }
            }
        ]
    }
}

Then I run kubectl apply -f seldon-dep.yaml.

I am expecting to see some results after I type:
kubectl get seldondeployments -o jsonpath='{.status}' but I see nothing.
and kubectl logs --tail=10 seldon-operator-controller-manager-0
return some logs like that:

{"level":"error","ts":1562651651.7709749,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562652651.7726917,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562652651.7736046,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562652651.7798111,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562653651.7812436,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562653651.7822802,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562653651.7854571,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562654651.787024,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562654651.7877965,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562654651.7910545,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

the formatted stack trace is like:

>>> print(s['stacktrace'])
github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error
	/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128
github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217
github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1
	/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158
github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
	/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133
github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134
github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until
	/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88

error:

>>> print(s['error'])
Deployment.apps "test-deployment-fx-market-predictor-7cd068f" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: "podinfo"

any suggestion?

@NetWilliam NetWilliam changed the title cannot run the example. cannot run through the example. Jul 9, 2019
@ukclivecox
Copy link
Contributor

I tried your json on latest master image and this seems to work. Can you do: k get sdep -o yaml

I get:

apiVersion: v1
items:
- apiVersion: machinelearning.seldon.io/v1alpha2
  kind: SeldonDeployment
  metadata:
    creationTimestamp: "2019-07-09T07:24:09Z"
    generation: 1
    labels:
      app: seldon
    name: seldon-deployment-example
    namespace: seldon
    resourceVersion: "2042"
    selfLink: /apis/machinelearning.seldon.io/v1alpha2/namespaces/seldon/seldondeployments/seldon-deployment-example
    uid: 8b0eac7a-a21a-11e9-a3a1-44e76eeb1d3c
  spec:
    annotations:
      deployment_version: v1
      project_name: FX Market Prediction
    name: test-deployment
    oauth_key: oauth-key
    oauth_secret: oauth-secret
    predictors:
    - annotations:
        predictor_version: v1
      componentSpecs:
      - spec:
          containers:
          - env:
            - name: PREDICTIVE_UNIT_SERVICE_PORT
              value: "9000"
            - name: PREDICTIVE_UNIT_ID
              value: classifier
            - name: PREDICTOR_ID
              value: fx-market-predictor
            - name: SELDON_DEPLOYMENT_ID
              value: seldon-deployment-example
            image: seldonio/mock_classifier:1.0
            imagePullPolicy: IfNotPresent
            lifecycle:
              preStop:
                exec:
                  command:
                  - /bin/sh
                  - -c
                  - /bin/sleep 10
            livenessProbe:
              failureThreshold: 3
              initialDelaySeconds: 60
              periodSeconds: 5
              successThreshold: 1
              tcpSocket:
                port: http
              timeoutSeconds: 1
            name: classifier
            ports:
            - containerPort: 9000
              name: http
              protocol: TCP
            readinessProbe:
              failureThreshold: 3
              initialDelaySeconds: 20
              periodSeconds: 5
              successThreshold: 1
              tcpSocket:
                port: http
              timeoutSeconds: 1
            resources:
              requests:
                memory: 1Mi
            terminationMessagePath: /dev/termination-log
            terminationMessagePolicy: File
            volumeMounts:
            - mountPath: /etc/podinfo
              name: podinfo
          dnsPolicy: ClusterFirst
          restartPolicy: Always
          schedulerName: default-scheduler
          securityContext: {}
          terminationGracePeriodSeconds: 20
          volumes:
          - downwardAPI:
              defaultMode: 420
              items:
              - fieldRef:
                  apiVersion: v1
                  fieldPath: metadata.annotations
                path: annotations
            name: podinfo
      graph:
        children: []
        endpoint:
          service_host: localhost
          service_port: 9000
          type: REST
        implementation: UNKNOWN_IMPLEMENTATION
        name: classifier
        type: MODEL
      name: fx-market-predictor
      replicas: 1
  status:
    deploymentStatus:
      test-deployment-fx-market-predictor-7cd068f:
        availableReplicas: 1
        replicas: 1
    serviceStatus:
      seldon-6d4c74de245da87a4b14caa5bb28a45a:
        httpEndpoint: seldon-6d4c74de245da87a4b14caa5bb28a45a.seldon:9000
        svcName: seldon-6d4c74de245da87a4b14caa5bb28a45a
      test-deployment-fx-market-predictor:
        grpcEndpoint: test-deployment-fx-market-predictor.seldon:5001
        httpEndpoint: test-deployment-fx-market-predictor.seldon:8000
        svcName: test-deployment-fx-market-predictor
    state: Available
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Note the podinfo volume which is added automatically by the admission controller. It looks like it is not being added in your case.

Which version of the controller image is running? And which version of the Helm chart?

@NetWilliam
Copy link
Author

$ helm version
Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}

$ docker version
Client:
Version: 18.06.1-ce
API version: 1.38
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:23:03 2018
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 18.06.1-ce
API version: 1.38 (minimum version 1.12)
Go version: go1.10.3
Git commit: e68fc7a
Built: Tue Aug 21 17:25:29 2018
OS/Arch: linux/amd64
Experimental: false

$ cat /etc/centos-release
CentOS Linux release 7.4.1708 (Core)

$ docker images | grep seldon
seldonio/seldon-core-operator 0.3.1 26b3082c3751 10 days ago 113MB
seldonio/mock_classifier 1.0 3e00af2ebf30 17 months ago 1.16GB

I get the result:

$ kubectl get sdep -o yaml
apiVersion: v1
items:
- apiVersion: machinelearning.seldon.io/v1alpha3
  kind: SeldonDeployment
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"machinelearning.seldon.io/v1alpha2","kind":"SeldonDeployment","metadata":{"annotations":{},"labels":{"app":"seldon"},"name":"seldon-deployment-example","namespace":"jhub"},"spec":{"annotations":{"deployment_version":"v1","project_name":"FX Market Prediction"},"name":"test-deployment","oauth_key":"oauth-key","oauth_secret":"oauth-secret","predictors":[{"annotations":{"predictor_version":"v1"},"componentSpecs":[{"spec":{"containers":[{"image":"seldonio/mock_classifier:1.0","imagePullPolicy":"IfNotPresent","name":"classifier","resources":{"requests":{"memory":"1Mi"}}}],"terminationGracePeriodSeconds":20}}],"graph":{"children":[],"endpoint":{"type":"REST"},"name":"classifier","type":"MODEL"},"name":"fx-market-predictor","replicas":1}]}}
    creationTimestamp: 2019-07-09T03:15:57Z
    generation: 1
    labels:
      app: seldon
    name: seldon-deployment-example
    namespace: jhub
    resourceVersion: "38791806"
    selfLink: /apis/machinelearning.seldon.io/v1alpha3/namespaces/jhub/seldondeployments/seldon-deployment-example
    uid: deaf6dfa-a1f7-11e9-9566-fa163ea46448
  spec:
    annotations:
      deployment_version: v1
      project_name: FX Market Prediction
    name: test-deployment
    oauth_key: oauth-key
    oauth_secret: oauth-secret
    predictors:
    - annotations:
        predictor_version: v1
      componentSpecs:
      - spec:
          containers:
          - image: seldonio/mock_classifier:1.0
            imagePullPolicy: IfNotPresent
            name: classifier
            resources:
              requests:
                memory: 1Mi
          terminationGracePeriodSeconds: 20
      graph:
        children: []
        endpoint:
          type: REST
        name: classifier
        type: MODEL
      name: fx-market-predictor
      replicas: 1
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

@NetWilliam
Copy link
Author

NetWilliam commented Jul 9, 2019

I do find no pods info related to the seldon in kubectl get pods. it seems like that something has happened even before the creating of the k8s pods.

@ukclivecox
Copy link
Contributor

Can you get the services running in seldon-system namespace and check they are ok. e.g.:

kubectl get svc -n seldon-system

NAME                                         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
seldon-operator-controller-manager-service   ClusterIP   10.97.32.112    <none>        443/TCP   46m
webhook-server-service                       ClusterIP   10.105.69.124   <none>        443/TCP   46m

kubectl describe svc -n seldon-system webhook-server-service

Name:              webhook-server-service
Namespace:         seldon-system
Labels:            app.kubernetes.io/instance=seldon-core
                   app.kubernetes.io/managed-by=Tiller
                   app.kubernetes.io/name=seldon-core-operator
                   control-plane=seldon-controller-manager
                   controller-tools.k8s.io=1.0
                   helm.sh/chart=seldon-core-operator-0.3.2-SNAPSHOT
Annotations:       <none>
Selector:          control-plane=seldon-controller-manager
Type:              ClusterIP
IP:                10.105.69.124
Port:              <unset>  443/TCP
TargetPort:        9876/TCP
Endpoints:         172.17.0.10:9876
Session Affinity:  None
Events:            <none>

@NetWilliam
Copy link
Author

$ kubectl get svc -n seldon-system
No resources found.

something is wrong.

@ukclivecox
Copy link
Contributor

It could be in another namespace - maybe default

@NetWilliam
Copy link
Author

kubectl get svc
NAME                                                TYPE           CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
hub                                                 ClusterIP      10.110.217.110   <none>        8081/TCP                     216d
proxy-api                                           ClusterIP      10.103.170.182   <none>        8001/TCP                     216d
proxy-public                                        LoadBalancer   10.107.112.165   <pending>     80:32190/TCP                 216d
seldon-operator-controller-manager-service          ClusterIP      10.102.243.215   <none>        443/TCP                      5h39m
wandering-albatross-nginx-ingress-controller        LoadBalancer   10.102.97.119    <pending>     80:31185/TCP,443:32158/TCP   231d
wandering-albatross-nginx-ingress-default-backend   ClusterIP      10.100.40.203    <none>        80/TCP                       231d
webhook-server-service                              ClusterIP      10.101.28.62     <none>        443/TCP                      4h56m
$ kubectl describe svc webhook-server-service
Name:              webhook-server-service
Namespace:         jhub
Labels:            <none>
Annotations:       <none>
Selector:          control-plane=seldon-controller-manager
Type:              ClusterIP
IP:                10.101.28.62
Port:              <unset>  443/TCP
TargetPort:        9876/TCP
Endpoints:         10.244.0.18:9876
Session Affinity:  None
Events:            <none>

@ukclivecox
Copy link
Contributor

That looks ok. I see there is a big AGE time difference between the controller 5h39 and webhook service 4h56m which seems strange.
Can you check if there are any earlier errors in the controller logs?
Maybe uninstall and reinstall after that.

@NetWilliam
Copy link
Author

NetWilliam commented Jul 9, 2019

kubectl logs seldon-operator-controller-manager-0

{"level":"info","ts":1562642332.5556116,"logger":"entrypoint","msg":"setting up client for manager"}
{"level":"info","ts":1562642332.5565143,"logger":"entrypoint","msg":"setting up manager"}
{"level":"info","ts":1562642332.627091,"logger":"entrypoint","msg":"Registering Components."}
{"level":"info","ts":1562642332.6271582,"logger":"entrypoint","msg":"setting up scheme"}
{"level":"info","ts":1562642332.627326,"logger":"entrypoint","msg":"setting up istio scheme"}
{"level":"info","ts":1562642332.6274457,"logger":"entrypoint","msg":"Setting up controller"}
{"level":"info","ts":1562642332.6275158,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"seldondeployment-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1562642332.627713,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"seldondeployment-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1562642332.6287649,"logger":"kubebuilder.controller","msg":"Starting EventSource","controller":"seldondeployment-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1562642332.6288753,"logger":"entrypoint","msg":"setting up webhooks"}
{"level":"info","ts":1562642332.629074,"logger":"entrypoint","msg":"Starting the Cmd."}
{"level":"info","ts":1562642332.7295353,"logger":"kubebuilder.webhook","msg":"installing webhook configuration in cluster"}
{"level":"info","ts":1562642332.7295687,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"seldondeployment-controller"}
{"level":"info","ts":1562642332.829909,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"seldondeployment-controller","worker count":1}
{"level":"info","ts":1562642332.8383832,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642332.8388371,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642332.841794,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642333.8424072,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642333.8426774,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642333.8446887,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642334.845231,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642334.84551,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642334.8475673,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642335.8480978,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642335.8484507,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642335.8508863,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642336.8515813,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642336.851944,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642336.8541443,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642337.8546705,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642337.8550425,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642337.8573532,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642338.8578815,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642338.8582475,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642338.8629363,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642339.8636196,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642339.86405,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642339.866288,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642340.86676,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642340.8671248,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642340.8692374,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642342.1496987,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642342.150125,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642342.1522117,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642344.7126524,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642344.7129047,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642346.1839566,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642351.304407,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642351.3047824,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642351.307194,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642361.547683,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642361.548101,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642361.554608,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642382.0350912,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642382.0353718,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642382.0411904,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642423.0017347,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642423.002124,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642423.0097036,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642504.9316072,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642504.9321063,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642504.9393418,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642668.781465,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642668.7822173,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642668.7883682,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562642996.4694283,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562642996.470369,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562642996.476801,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562643651.8389049,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562643651.8398492,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562643651.8474116,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562644651.8496325,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562644651.850718,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562644651.8569255,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562645651.8586783,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562645651.8595583,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562645651.8666818,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562646651.8687384,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562646651.8695376,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562646651.8765428,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562647651.7265232,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562647651.727471,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562647651.7343926,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562648651.7360392,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562648651.7369657,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562648651.7434444,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562649651.744893,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562649651.7456067,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562649651.7513704,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562650651.7531652,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562650651.7542176,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562650651.7605321,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562651651.7620592,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562651651.7627995,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562651651.7709749,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562652651.7726917,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562652651.7736046,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562652651.7798111,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562653651.7812436,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562653651.7822802,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562653651.7854571,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562654651.787024,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562654651.7877965,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562654651.7910545,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562655651.7928617,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562655651.7935412,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562655651.799617,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562656651.8013148,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562656651.8020988,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562656651.8079338,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562657651.8099496,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562657651.8109279,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562657651.8171625,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562658651.8186922,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562658651.8194423,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562658651.8258567,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}
{"level":"info","ts":1562659651.828337,"logger":"seldon-controller","msg":"Creating default Ambassador config"}
{"level":"info","ts":1562659651.8294075,"logger":"seldon-controller","msg":"Creating Deployment","namespace":"jhub","name":"test-deployment-fx-market-predictor-7cd068f"}
{"level":"error","ts":1562659651.8374796,"logger":"kubebuilder.controller","msg":"Reconciler error","controller":"seldondeployment-controller","request":"jhub/seldon-deployment-example","error":"Deployment.apps \"test-deployment-fx-market-predictor-7cd068f\" is invalid: spec.template.spec.containers[1].volumeMounts[0].name: Not found: \"podinfo\"","stacktrace":"github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr.(*zapLogger).Error\n\t/go/src/github.com/seldonio/seldon-operator/vendor/github.com/go-logr/zapr/zapr.go:128\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:217\ngithub.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:158\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:134\ngithub.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait.Until\n\t/go/src/github.com/seldonio/seldon-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:88"}

@NetWilliam
Copy link
Author

I haven't installed any ingress gateway yet. Is that nessiccary?

@ukclivecox
Copy link
Contributor

No should not be necessary. Are you running with any other specific settings for the helm chart? Can you posy how you installed it?
Has anything changed on the cluster - like the controller was restarted or the webhook service recreated for some reason?

@NetWilliam
Copy link
Author

I install seldon via the cmd:
$ helm install seldon-core-operator --name seldon-core --repo https://storage.googleapis.com/seldon-charts --set usageMetrics.enabled=true

@NetWilliam
Copy link
Author

It seems like something changed the networking setting of k8s. I can't re-install after I del the seldon. some TLS handshaking step of k8s cluster-info complains:

helm install seldon-core-operator --name seldon-core --repo https://storage.googleapis.com/seldon-charts
Error: Get https://10.202.235.223:6443/api/v1/namespaces/kube-system/pods?labelSelector=app%3Dhelm%2Cname%3Dtiller: net/http: TLS handshake timeout

@NetWilliam
Copy link
Author

$ helm install seldon-core-operator --name seldon-core --repo https://storage.googleapis.com/seldon-charts
NAME:   seldon-core
LAST DEPLOYED: Tue Jul  9 16:52:11 2019
NAMESPACE: jhub
STATUS: DEPLOYED

RESOURCES:
==> v1/ServiceAccount
NAME                              AGE
seldon-core-seldon-core-operator  0s

==> v1beta1/CustomResourceDefinition
seldondeployments.machinelearning.seldon.io  0s

==> v1/ClusterRole
seldon-operator-manager-role  0s

==> v1/ClusterRoleBinding
seldon-operator-manager-rolebinding  0s

==> v1/Service
seldon-operator-controller-manager-service  0s

==> v1/StatefulSet
seldon-operator-controller-manager  0s

==> v1/Pod(related)

NAME                                  READY  STATUS   RESTARTS  AGE
seldon-operator-controller-manager-0  0/1    Pending  0         0s

==> v1/Secret

NAME                                   AGE
seldon-operator-webhook-server-secret  0s


NOTES:
NOTES: TODO

it seems to work after I re-try several times.

@NetWilliam
Copy link
Author

and re-create the service:

$ kubectl apply -f seldon-dep.yaml
seldondeployment.machinelearning.seldon.io/seldon-deployment-example created
$ echo $?
0

@NetWilliam
Copy link
Author

image

don't know why, but it just worked. however, the status stuck at "ContainerCreating"

$ kubectl describe pod test-deployment-fx-market-predictor-7cd068f-686477cbbc-vpp7m
Name:               test-deployment-fx-market-predictor-7cd068f-686477cbbc-vpp7m
Namespace:          jhub
Priority:           0
PriorityClassName:  <none>
Node:               k3711v.add.bjyt.qihoo.net/10.209.1.150
Start Time:         Tue, 09 Jul 2019 17:02:57 +0800
Labels:             app=test-deployment-fx-market-predictor-7cd068f
                    fluentd=true
                    pod-template-hash=686477cbbc
                    seldon-app=test-deployment-seldon-deployment-example
                    seldon-app-classifier=seldon-6d4c74de245da87a4b14caa5bb28a45a
                    seldon-deployment-id=test-deployment-seldon-deployment-example
Annotations:        deployment_version: v1
                    project_name: FX Market Prediction
                    prometheus.io/path: prometheus
                    prometheus.io/port: 8000
                    prometheus.io/scrape: true
                    v1:
Status:             Pending
IP:
Controlled By:      ReplicaSet/test-deployment-fx-market-predictor-7cd068f-686477cbbc
Containers:
  classifier:
    Container ID:
    Image:          seldonio/mock_classifier:1.0
    Image ID:
    Port:           9000/TCP
    Host Port:      0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      memory:   1Mi
    Liveness:   tcp-socket :http delay=60s timeout=1s period=5s #success=1 #failure=3
    Readiness:  tcp-socket :http delay=20s timeout=1s period=5s #success=1 #failure=3
    Environment:
      PREDICTIVE_UNIT_SERVICE_PORT:  9000
      PREDICTIVE_UNIT_ID:            classifier
      PREDICTOR_ID:                  fx-market-predictor
      SELDON_DEPLOYMENT_ID:          seldon-deployment-example
    Mounts:
      /etc/podinfo from podinfo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qklk6 (ro)
  seldon-container-engine:
    Container ID:
    Image:          docker.io/seldonio/engine:0.3.1
    Image ID:
    Ports:          8000/TCP, 5001/TCP, 8082/TCP, 9090/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      100m
    Liveness:   http-get http://:admin/ready delay=20s timeout=2s period=5s #success=1 #failure=3
    Readiness:  http-get http://:admin/ready delay=20s timeout=2s period=1s #success=1 #failure=1
    Environment:
      ENGINE_PREDICTOR:         eyJuYW1lIjoiZngtbWFya2V0LXByZWRpY3RvciIsImdyYXBoIjp7Im5hbWUiOiJjbGFzc2lmaWVyIiwidHlwZSI6Ik1PREVMIiwiaW1wbGVtZW50YXRpb24iOiJVTktOT1dOX0lNUExFTUVOVEFUSU9OIiwiZW5kcG9pbnQiOnsic2VydmljZV9ob3N0IjoibG9jYWxob3N0Iiwic2VydmljZV9wb3J0Ijo5MDAwLCJ0eXBlIjoiUkVTVCJ9fSwiY29tcG9uZW50U3BlY3MiOlt7Im1ldGFkYXRhIjp7ImNyZWF0aW9uVGltZXN0YW1wIjpudWxsfSwic3BlYyI6eyJ2b2x1bWVzIjpbeyJuYW1lIjoicG9kaW5mbyIsImRvd253YXJkQVBJIjp7Iml0ZW1zIjpbeyJwYXRoIjoiYW5ub3RhdGlvbnMiLCJmaWVsZFJlZiI6eyJhcGlWZXJzaW9uIjoidjEiLCJmaWVsZFBhdGgiOiJtZXRhZGF0YS5hbm5vdGF0aW9ucyJ9fV0sImRlZmF1bHRNb2RlIjo0MjB9fV0sImNvbnRhaW5lcnMiOlt7Im5hbWUiOiJjbGFzc2lmaWVyIiwiaW1hZ2UiOiJzZWxkb25pby9tb2NrX2NsYXNzaWZpZXI6MS4wIiwicG9ydHMiOlt7Im5hbWUiOiJodHRwIiwiY29udGFpbmVyUG9ydCI6OTAwMCwicHJvdG9jb2wiOiJUQ1AifV0sImVudiI6W3sibmFtZSI6IlBSRURJQ1RJVkVfVU5JVF9TRVJWSUNFX1BPUlQiLCJ2YWx1ZSI6IjkwMDAifSx7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfSUQiLCJ2YWx1ZSI6ImNsYXNzaWZpZXIifSx7Im5hbWUiOiJQUkVESUNUT1JfSUQiLCJ2YWx1ZSI6ImZ4LW1hcmtldC1wcmVkaWN0b3IifSx7Im5hbWUiOiJTRUxET05fREVQTE9ZTUVOVF9JRCIsInZhbHVlIjoic2VsZG9uLWRlcGxveW1lbnQtZXhhbXBsZSJ9XSwicmVzb3VyY2VzIjp7InJlcXVlc3RzIjp7Im1lbW9yeSI6IjFNaSJ9fSwidm9sdW1lTW91bnRzIjpbeyJuYW1lIjoicG9kaW5mbyIsIm1vdW50UGF0aCI6Ii9ldGMvcG9kaW5mbyJ9XSwibGl2ZW5lc3NQcm9iZSI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifSwiaW5pdGlhbERlbGF5U2Vjb25kcyI6NjAsInRpbWVvdXRTZWNvbmRzIjoxLCJwZXJpb2RTZWNvbmRzIjo1LCJzdWNjZXNzVGhyZXNob2xkIjoxLCJmYWlsdXJlVGhyZXNob2xkIjozfSwicmVhZGluZXNzUHJvYmUiOnsidGNwU29ja2V0Ijp7InBvcnQiOiJodHRwIn0sImluaXRpYWxEZWxheVNlY29uZHMiOjIwLCJ0aW1lb3V0U2Vjb25kcyI6MSwicGVyaW9kU2Vjb25kcyI6NSwic3VjY2Vzc1RocmVzaG9sZCI6MSwiZmFpbHVyZVRocmVzaG9sZCI6M30sImxpZmVjeWNsZSI6eyJwcmVTdG9wIjp7ImV4ZWMiOnsiY29tbWFuZCI6WyIvYmluL3NoIiwiLWMiLCIvYmluL3NsZWVwIDEwIl19fX0sInRlcm1pbmF0aW9uTWVzc2FnZVBhdGgiOiIvZGV2L3Rlcm1pbmF0aW9uLWxvZyIsInRlcm1pbmF0aW9uTWVzc2FnZVBvbGljeSI6IkZpbGUiLCJpbWFnZVB1bGxQb2xpY3kiOiJJZk5vdFByZXNlbnQifV0sInJlc3RhcnRQb2xpY3kiOiJBbHdheXMiLCJ0ZXJtaW5hdGlvbkdyYWNlUGVyaW9kU2Vjb25kcyI6MjAsImRuc1BvbGljeSI6IkNsdXN0ZXJGaXJzdCIsInNlY3VyaXR5Q29udGV4dCI6e30sInNjaGVkdWxlck5hbWUiOiJkZWZhdWx0LXNjaGVkdWxlciJ9fV0sInJlcGxpY2FzIjoxLCJhbm5vdGF0aW9ucyI6eyJwcmVkaWN0b3JfdmVyc2lvbiI6InYxIn0sImVuZ2luZVJlc291cmNlcyI6e30sInN2Y09yY2hTcGVjIjp7fX0=
      DEPLOYMENT_NAME:          test-deployment
      DEPLOYMENT_NAMESPACE:     jhub
      ENGINE_SERVER_PORT:       8000
      ENGINE_SERVER_GRPC_PORT:  5001
      JAVA_OPTS:                -Dcom.sun.management.jmxremote.rmi.port=9090 -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9090 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname=127.0.0.1
    Mounts:
      /etc/podinfo from podinfo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-qklk6 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  default-token-qklk6:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-qklk6
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type    Reason     Age    From                                Message
  ----    ------     ----   ----                                -------
  Normal  Scheduled  2m19s  default-scheduler                   Successfully assigned jhub/test-deployment-fx-market-predictor-7cd068f-686477cbbc-vpp7m to k3711v.add.bjyt.qihoo.net
  Normal  Pulled     2m18s  kubelet, k3711v.add.bjyt.qihoo.net  Container image "seldonio/mock_classifier:1.0" already present on machine
  Normal  Created    2m17s  kubelet, k3711v.add.bjyt.qihoo.net  Created container
  Normal  Started    2m17s  kubelet, k3711v.add.bjyt.qihoo.net  Started container
  Normal  Pulling    2m17s  kubelet, k3711v.add.bjyt.qihoo.net  pulling image "docker.io/seldonio/engine:0.3.1"
$ kubectl get pods
NAME                                                              READY   STATUS              RESTARTS   AGE
cephfs-provisioner-db754f759-l4m8k                                1/1     Running             1          216d
cz-demo                                                           1/1     Running             1          144d
hub-64689b54b7-txvsm                                              1/1     Running             1          200d
jupyter-hejun                                                     1/1     Running             1          169d
nfs-client-provisioner-76b6bb84f8-rrh8t                           1/1     Running             61         225d
proxy-5f648bd7d9-cgxkq                                            1/1     Running             1          216d
seldon-operator-controller-manager-0                              1/1     Running             0          13m
test-deployment-fx-market-predictor-7cd068f-686477cbbc-vpp7m      0/2     ContainerCreating   0          2m24s
wandering-albatross-nginx-ingress-controller-58dcd9f588-77fnz     1/1     Running             17         231d
wandering-albatross-nginx-ingress-default-backend-6d466d74xklb6   1/1     Running             13         229d

@NetWilliam
Copy link
Author

image

creating succeed finally!

@ukclivecox
Copy link
Contributor

ukclivecox commented Jul 9, 2019 via email

@NetWilliam
Copy link
Author

it works. thank you @cliveseldon

$ curl 10.98.177.103:8000/api/v0.1/predictions -XPOST -d '{"data":{"names":["a","b"],"tensor":{"shape":[2,2],"values":[0,0,1,1]}}}' -H "Content-Type: application/json"
{
  "meta": {
    "puid": "d66pgfj1jv041big9tkrqqd1fd",
    "tags": {
    },
    "routing": {
    },
    "requestPath": {
      "classifier": "seldonio/mock_classifier:1.0"
    },
    "metrics": []
  },
  "data": {
    "names": ["proba"],
    "tensor": {
      "shape": [2, 1],
      "values": [0.05133579311531625, 0.12823373759251927]
    }
  }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants