Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Seldon Core Deployment Istio virtualservice is only created until after the pod is ready resulting in service unavailable until it propagates #1461

Closed
axsaucedo opened this issue Feb 18, 2020 · 2 comments
Closed

Seldon Core Deployment Istio virtualservice is only created until after the pod is ready resulting in service unavailable until it propagates #1461

axsaucedo opened this issue Feb 18, 2020 · 2 comments
Assignees
@axsaucedo @ukclivecox
Labels
bug
Milestone
1.1

Comments

@axsaucedo
Copy link
Contributor

It seems that when a Seldon Core model is created, the Istio Virtualservice is created until after the container is up and running. This sometimes results in the service being unavailable even though the container is actually running. When the Istio virtualservice is created, it seems that there is also a set of reconcile errors with the message Operation cannot be fulfilled on seldondeployments.machinelearning.seldon.io \"sw\": the object has been modified; please apply your changes to the latest version and try again.

Another seeming problem that may be a challenge is that when the Seldon Deployment is spinning up and it is available, the Seldon Engine pod seems to crash once as the reconcile error appears, however this may be due to another issue.

This was run on a DigitalOcean cluster Major:"1", Minor:"15", and can be recreated by creating any new deployment, for example the following:

apiVersion: machinelearning.seldon.io/v1
kind: SeldonDeployment
metadata:
  creationTimestamp: "2020-02-18T21:47:45Z"
  generation: 1
  name: income-classifier-istio-test
  namespace: default
  resourceVersion: "22571"
  selfLink: /apis/machinelearning.seldon.io/v1/namespaces/default/seldondeployments/income-classifier-istio-test
  uid: 00fac986-dee8-4575-8db7-bba558ebfe6a
spec:
  annotations:
    seldon.io/engine-seldon-log-messages-externally: "true"
  name: income-classifier-istio-test
  predictors:
  - componentSpecs:
    - metadata:
        creationTimestamp: "2020-02-18T21:47:46Z"
      spec:
        containers:
        - image: seldonio/sklearnserver_rest:0.2
          name: income-classifier-istio-test-default
          resources: {}
          volumeMounts:
          - mountPath: /etc/podinfo
            name: podinfo
    engineResources: {}
    explainer:
      containerSpec:
        name: ""
        resources: {}
    graph:
      endpoint:
        service_host: localhost
        service_port: 9000
        type: REST
      implementation: SKLEARN_SERVER
      modelUri: gs://seldon-models/sklearn/income/model
      name: income-classifier-istio-test-default
      type: UNKNOWN_TYPE
    labels:
      version: default
    name: default
    replicas: 1
    svcOrchSpec: {}
    traffic: 100

Logs from operator

Below are the logs of the operator, together with the sequence in which they appear.

First when the deployment is created, the initial reconcile logs are the following:

2020-02-18T21:47:45.707Z        INFO    seldondeployment        Defaulting v1 web hook called   {"name": "income-classifier-istio-test"}
2020-02-18T21:47:45.714Z        INFO    seldondeployment        Validating v1 Webhook called for CREATE {"name": "income-classifier-istio-test"}
2020-02-18T21:47:45.723Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.723Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:45.738Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:47:45.738Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:47:45.738Z        INFO    controllers.SeldonDeployment    Creating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:47:45.745Z        INFO    controllers.SeldonDeployment    Creating Service        {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7"}
2020-02-18T21:47:45.768Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:45.784Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.784Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.784Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:45.804Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:47:45.804Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:47:45.805Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:47:45.811Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.811Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetCreated","message":"Created new replica set \"income-classifier-istio-test-default-3cd8f75-84b76688d6\""},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:47:45.811Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetCreated","message":"Created new replica set \"income-classifier-istio-test-default-3cd8f75-84b76688d6\""},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:47:45.811Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:47:45.825Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:45.833Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.833Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.833Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:45.844Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:47:45.844Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:47:45.844Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:47:45.849Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.849Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetCreated","message":"Created new replica set \"income-classifier-istio-test-default-3cd8f75-84b76688d6\""},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:47:45.849Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetCreated","message":"Created new replica set \"income-classifier-istio-test-default-3cd8f75-84b76688d6\""},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:47:45.849Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:47:45.849Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:45.855Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.974Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.974Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:45.992Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:47:45.992Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:47:45.993Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:47:45.997Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:47:45.997Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" is progressing."}]}}
2020-02-18T21:47:46.016Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" is progressing."}]}}
2020-02-18T21:47:46.016Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:47:46.017Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:46.021Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}
2020-02-18T21:47:46.021Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:47:46.021Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:46.037Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:47:46.037Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:47:46.037Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:47:46.043Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:47:46.043Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" is progressing."}]}}
2020-02-18T21:47:46.043Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:47:45Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"ReplicaSetUpdated","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" is progressing."}]}}
2020-02-18T21:47:46.043Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:47:46.043Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:47:46.048Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}

Then it seems that the container becomes ready / running, and that's when the next reconciliation and the first error comes in. It seems that this is a first attempt to add the virtual service (which doesn't work)

2020-02-18T21:48:39.766Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:39.766Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:39.783Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:48:39.787Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:48:39.787Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:48:39.792Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:39.792Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:48:39Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."}]}}
2020-02-18T21:48:39.798Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:48:39Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."}]}}
2020-02-18T21:48:39.798Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:39.798Z        INFO    controllers.SeldonDeployment    Creating Service        {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:39.810Z        INFO    controllers.SeldonDeployment    Creating Virtual Service        {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-http"}
2020-02-18T21:48:39.826Z        INFO    controllers.SeldonDeployment    Creating Virtual Service        {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-grpc"}
2020-02-18T21:48:39.836Z        INFO    controllers.SeldonDeployment    Creating Istio Destination Rule {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:39.929Z        INFO    controllers.SeldonDeployment    Removing unused services        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:39.932Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:39.935Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:39.944Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}
2020-02-18T21:48:39.944Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:39.944Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:39.955Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:48:39.955Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:48:39.956Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:48:39.962Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:39.962Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:48:39Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."}]}}
2020-02-18T21:48:39.962Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:48:39Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."}]}}
2020-02-18T21:48:39.962Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:39.962Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075", "status": {"loadBalancer":{}}}
2020-02-18T21:48:39.969Z        ERROR   controller-runtime.controller   Reconciler error        {"controller": "seldondeployment", "request": "default/income-classifier-istio-test", "error": "Operation cannot be fulfilled on seldondeployments.machinelearning.seldon.io \"income-classifier-istio-test\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:171
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88

2020-02-18T21:48:40.969Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:40.975Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:40.997Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:48:40.997Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:48:40.997Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:48:41.010Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:41.010Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:48:39Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."}]}}
2020-02-18T21:48:41.010Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:48:39Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."}]}}
2020-02-18T21:48:41.011Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:41.011Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075", "status": {"loadBalancer":{}}}

2020-02-18T21:48:41.016Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-http"}
2020-02-18T21:48:41.016Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-grpc"}
2020-02-18T21:48:41.016Z        INFO    controllers.SeldonDeployment    Found identical Istio Destination Rule  {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:41.016Z        INFO    controllers.SeldonDeployment    Removing unused services        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:41.016Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:41.016Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:41.020Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}
2020-02-18T21:48:41.021Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:41.021Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:41.040Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:48:41.040Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:48:41.041Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:48:39Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."}]}}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:48:39Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."},{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."}]}}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075", "status": {"loadBalancer":{}}}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-http"}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-grpc"}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Found identical Istio Destination Rule  {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Removing unused services        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:41.045Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:41.051Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}

THen it is attempted once again with another error:

2020-02-18T21:48:49.999Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:49.999Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:50.020Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:48:50.020Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:48:50.021Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:48:50.033Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:50.033Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:48:49Z","lastTransitionTime":"2020-02-18T21:48:49Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:48:50.038Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:48:49Z","lastTransitionTime":"2020-02-18T21:48:49Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:48:50.038Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:50.038Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:50.044Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}
2020-02-18T21:48:50.044Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:50.044Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:50.054Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:48:50.054Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:48:50.054Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:48:50.061Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:50.062Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:48:49Z","lastTransitionTime":"2020-02-18T21:48:49Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:48:50.062Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:48:49Z","lastTransitionTime":"2020-02-18T21:48:49Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:48:50.062Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:50.062Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:50.064Z        ERROR   controller-runtime.controller   Reconciler error        {"controller": "seldondeployment", "request": "default/income-classifier-istio-test", "error": "Operation cannot be fulfilled on seldondeployments.machinelearning.seldon.io \"income-classifier-istio-test\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:171
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88
2020-02-18T21:48:51.064Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:51.064Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:51.110Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:48:51.111Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:48:51.111Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:48:51.116Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:48:51.116Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:48:49Z","lastTransitionTime":"2020-02-18T21:48:49Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:48:51.116Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"unavailableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"False","lastUpdateTime":"2020-02-18T21:48:49Z","lastTransitionTime":"2020-02-18T21:48:49Z","reason":"MinimumReplicasUnavailable","message":"Deployment does not have minimum availability."}]}}
2020-02-18T21:48:51.116Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:48:51.116Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:48:51.127Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}

Finally there is a last attempt iwth another error, but at the end it seems that this time it does create the virual service without it crashing

2020-02-18T21:49:09.853Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:49:09.853Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:49:09.893Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:49:09.893Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:49:09.895Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:49:09.915Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:49:09.915Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:49:09Z","lastTransitionTime":"2020-02-18T21:49:09Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."}]}}
2020-02-18T21:49:09.922Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:49:09Z","lastTransitionTime":"2020-02-18T21:49:09Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."}]}}
2020-02-18T21:49:09.922Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:49:09.922Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075", "status": {"loadBalancer":{}}}
2020-02-18T21:49:09.922Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-http"}
2020-02-18T21:49:09.922Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-grpc"}
2020-02-18T21:49:09.922Z        INFO    controllers.SeldonDeployment    Found identical Istio Destination Rule  {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:49:09.922Z        INFO    controllers.SeldonDeployment    Removing unused services        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:49:09.922Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:49:09.922Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:49:09.928Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}
2020-02-18T21:49:09.928Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:49:09.928Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:49:09.971Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:49:09.971Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:49:09.971Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:49:09Z","lastTransitionTime":"2020-02-18T21:49:09Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."}]}}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:49:09Z","lastTransitionTime":"2020-02-18T21:49:09Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."}]}}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075", "status": {"loadBalancer":{}}}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-http"}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-grpc"}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Found identical Istio Destination Rule  {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Removing unused services        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:49:09.983Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:49:09.986Z        ERROR   controller-runtime.controller   Reconciler error        {"controller": "seldondeployment", "request": "default/income-classifier-istio-test", "error": "Operation cannot be fulfilled on seldondeployments.machinelearning.seldon.io \"income-classifier-istio-test\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
        /go/pkg/mod/github.com/go-logr/zapr@v0.1.0/zapr.go:128
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:192
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.2.0/pkg/internal/controller/controller.go:171
k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:152
k8s.io/apimachinery/pkg/util/wait.JitterUntil
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:153
k8s.io/apimachinery/pkg/util/wait.Until
        /go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190404173353-6a84e37a896d/pkg/util/wait/wait.go:88
2020-02-18T21:49:10.993Z        INFO    controllers.SeldonDeployment    Reconcile called        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:49:10.993Z        INFO    controllers.SeldonDeployment    pSvcName        {"SeldonDeployment": "default/income-classifier-istio-test", "val": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:49:11.030Z        INFO    controllers.SeldonDeployment    Scheme  {"SeldonDeployment": "default/income-classifier-istio-test", "r.scheme": {}}
2020-02-18T21:49:11.030Z        INFO    controllers.SeldonDeployment    createDeployments       {"SeldonDeployment": "default/income-classifier-istio-test", "deploy": {"namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}}
2020-02-18T21:49:11.030Z        INFO    controllers.SeldonDeployment    Updating Deployment     {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75"}
2020-02-18T21:49:11.053Z        INFO    controllers.SeldonDeployment    The deployments are the same - api server defaults ignored      {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:49:11.053Z        INFO    controllers.SeldonDeployment    Found identical deployment      {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:49:09Z","lastTransitionTime":"2020-02-18T21:49:09Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."}]}}
2020-02-18T21:49:11.054Z        INFO    controllers.SeldonDeployment    Deployment status       {"SeldonDeployment": "default/income-classifier-istio-test", "name": "income-classifier-istio-test-default-3cd8f75", "status": {"observedGeneration":1,"replicas":1,"updatedReplicas":1,"readyReplicas":1,"availableReplicas":1,"conditions":[{"type":"Progressing","status":"True","lastUpdateTime":"2020-02-18T21:48:39Z","lastTransitionTime":"2020-02-18T21:47:45Z","reason":"NewReplicaSetAvailable","message":"ReplicaSet \"income-classifier-istio-test-default-3cd8f75-84b76688d6\" has successfully progressed."},{"type":"Available","status":"True","lastUpdateTime":"2020-02-18T21:49:09Z","lastTransitionTime":"2020-02-18T21:49:09Z","reason":"MinimumReplicasAvailable","message":"Deployment has minimum availability."}]}}
2020-02-18T21:49:11.054Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:49:11.054Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": true, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075", "status": {"loadBalancer":{}}}
2020-02-18T21:49:11.054Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-http"}
2020-02-18T21:49:11.055Z        INFO    controllers.SeldonDeployment    Found identical Virtual Service {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "income-classifier-istio-test-income-classifier-istio-test-grpc"}
2020-02-18T21:49:11.055Z        INFO    controllers.SeldonDeployment    Found identical Istio Destination Rule  {"SeldonDeployment": "default/income-classifier-istio-test", "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:49:11.055Z        INFO    controllers.SeldonDeployment    Removing unused services        {"SeldonDeployment": "default/income-classifier-istio-test"}
2020-02-18T21:49:11.055Z        INFO    controllers.SeldonDeployment    Found identical Service {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-c11b848d9d15de54052d9e8a09730be7", "status": {"loadBalancer":{}}}
2020-02-18T21:49:11.055Z        INFO    controllers.SeldonDeployment    Skipping Ambassador Svc {"SeldonDeployment": "default/income-classifier-istio-test", "all": false, "namespace": "default", "name": "seldon-822a0a28cf5422a4eec7f5a1b5a6b075"}
2020-02-18T21:49:11.062Z        DEBUG   controller-runtime.controller   Successfully Reconciled {"controller": "seldondeployment", "request": "default/income-classifier-istio-test"}
@axsaucedo axsaucedo added bug triage Needs to be triaged and prioritised accordingly labels Feb 18, 2020
@axsaucedo axsaucedo changed the title Seldon Core Deployment Istio virtualservice is only created until after the service is ready (and any crash will cause it to be unreachable) Seldon Core Deployment Istio virtualservice is only created until after the service is ready Feb 18, 2020
@axsaucedo axsaucedo changed the title Seldon Core Deployment Istio virtualservice is only created until after the service is ready Seldon Core Deployment Istio virtualservice is only created until after the pod is ready resulting in service unavailable until it propagates Feb 18, 2020
@ukclivecox
Copy link
Contributor

Yes, Services are only created once the Deployments are ready.

This looks like the reconcile error is on the SeldonDeployment resource not the a deployment?
It might be trying to update the /status part but be good to understand why its changing.

@ukclivecox ukclivecox removed the triage Needs to be triaged and prioritised accordingly label Feb 20, 2020
@ukclivecox ukclivecox added this to the 1.1 milestone Feb 20, 2020
@ukclivecox
Copy link
Contributor

reopen if can be recreated across cloud providers other than DigitalOcean

@adriangonz adriangonz mentioned this issue Mar 24, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@axsaucedo axsaucedo

@ukclivecox ukclivecox

Labels
bug
Projects
None yet
Milestone
1.1
Development

No branches or pull requests
2 participants
@axsaucedo @ukclivecox