routing metadata missing in new executor #1823

RafalSkolasinski · 2020-05-13T08:44:12Z

Reporter on community slack by user.

As new orchestrator (executor) is not inspecting payloads passed around the graph it is not currently adding the routing metadata, e.g.

"meta" : {
    "puid" : "...",
    "tags" : {},
    "routing" : {
        "ab_test" : 1,
        "combiner1" : -1,
        "mypostprocess1" : -1,
        "router1" : 1
    }
}

This may be important in context of multi-armed bandits and AB testing.

The text was updated successfully, but these errors were encountered:

RafalSkolasinski · 2020-06-25T10:28:26Z

This may be possible to be done on the wrapper itself.
The name should be available under one of the environmental variables (need to check which one) and the route integer is actual output of the wrapper so it should also be known there.

amirclam · 2020-07-15T10:49:17Z

Hi @cliveseldon
when can we expect this to be added for now we are using seldon-core 1.1 with the old java engine
since we need the routing /tags information in our system

ukclivecox · 2020-07-15T10:57:45Z

We have not had a chance to look at it but plan to discuss to see if it can be in language wrapper and prioritized.

RafalSkolasinski · 2020-07-15T14:07:01Z

After reviewing it does not seem it is possible to reconstruct current behaviour on the wrapper only.

This is because tags set on ROUTER nodes are disregarded by the executor - it only uses the route information to send original payload down the graph.

Changing that behaviour would mean that executor would have to modify the original payloads and that could mean performance hit to all graphs with the ROUTER component.

Alternative solution may be to record route information using the request headers.

@amirclam Would love to hear more about how you are using these in your system. I am mainly interested if you need to know the integer representing the chosen route or for example plain information which nodes were visited would be sufficient -> knowing the graph one could workout the route.

theone4ever · 2020-09-30T09:15:33Z

@RafalSkolasinski @cliveseldon We are experimenting Seldon-core and mostly interested with multi arm bandit solution, currently we are on Seldon 1.1 and feedback method is available. What is your suggestion? Does the fix included in 1.3 or there is some other workaround on 1.1?

RafalSkolasinski · 2020-09-30T09:21:30Z

No, unfortunately not in 1.3.0.
This one is next one on my task list, though.

theone4ever · 2020-09-30T09:22:42Z

Ok, thx for quick feedback. And there is no workaround on 1.1, right?

RafalSkolasinski · 2020-09-30T10:37:22Z

You could always run with old orchestrator system, the engine, for time being. There is a doc entry on how to set it up.

theone4ever · 2020-10-01T20:24:07Z

@RafalSkolasinski thx for suggestion, I follow the doc you pointed above, re-installed with following command:
helm install seldon-core seldon-core-operator --repo https://storage.googleapis.com/seldon-charts --set istio.enabled=true --set usageMetrics.enabled=true --set executor.enabled=false --namespace seldon-system.
However, I can't even deploy a prediction service anymore, the Seldon-container-engine container is always in error state, see info below:

seldon-container-engine:
    Container ID:   docker://6c5abf378419f26335da3c361e2c459642613ccf3808dbb90654da9d3d7f0e1d
    Image:          docker.io/seldonio/engine:1.3.0
    Image ID:       docker-pullable://seldonio/engine@sha256:25dd6b71a641b5d18c7f36e9282c7e69322b6be0c17db43f27b55f4e596816e8
    Ports:          8000/TCP, 8000/TCP, 5001/TCP, 8082/TCP, 9090/TCP
    Host Ports:     0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    137
      Started:      Thu, 01 Oct 2020 22:18:19 +0200
      Finished:     Thu, 01 Oct 2020 22:19:13 +0200
    Ready:          False
    Restart Count:  5
    Requests:
      cpu:      100m
    Liveness:   http-get https://:admin/live delay=20s timeout=60s period=5s #success=1 #failure=3
    Readiness:  http-get https://:admin/ready delay=20s timeout=60s period=5s #success=1 #failure=3
    Environment:
      ENGINE_PREDICTOR:                eyJuYW1lIjoicGlwZWxpbmUtcHJlZGljdG9yIiwiZ3JhcGgiOnsibmFtZSI6Im91dHB1dC10cmFuc2Zvcm1lciIsImNoaWxkcmVuIjpbeyJuYW1lIjoiaW5wdXQtdHJhbnNmb3JtZXIiLCJjaGlsZHJlbiI6W3sibmFtZSI6InJvdXRlciIsImNoaWxkcmVuIjpbeyJuYW1lIjoibW9kZWwtYSIsInR5cGUiOiJNT0RFTCIsImltcGxlbWVudGF0aW9uIjoiVU5LTk9XTl9JTVBMRU1FTlRBVElPTiIsImVuZHBvaW50Ijp7InNlcnZpY2VfaG9zdCI6ImxvY2FsaG9zdCIsInNlcnZpY2VfcG9ydCI6ODA4MSwidHlwZSI6IlJFU1QifX0seyJuYW1lIjoibW9kZWwtYiIsInR5cGUiOiJNT0RFTCIsImltcGxlbWVudGF0aW9uIjoiVU5LTk9XTl9JTVBMRU1FTlRBVElPTiIsImVuZHBvaW50Ijp7InNlcnZpY2VfaG9zdCI6ImxvY2FsaG9zdCIsInNlcnZpY2VfcG9ydCI6ODA4MywidHlwZSI6IlJFU1QifX0seyJuYW1lIjoibW9kZWwtYyIsInR5cGUiOiJNT0RFTCIsImltcGxlbWVudGF0aW9uIjoiVU5LTk9XTl9JTVBMRU1FTlRBVElPTiIsImVuZHBvaW50Ijp7InNlcnZpY2VfaG9zdCI6ImxvY2FsaG9zdCIsInNlcnZpY2VfcG9ydCI6ODA4NCwidHlwZSI6IlJFU1QifX1dLCJ0eXBlIjoiUk9VVEVSIiwiaW1wbGVtZW50YXRpb24iOiJVTktOT1dOX0lNUExFTUVOVEFUSU9OIiwiZW5kcG9pbnQiOnsic2VydmljZV9ob3N0IjoibG9jYWxob3N0Iiwic2VydmljZV9wb3J0Ijo4MDgwLCJ0eXBlIjoiUkVTVCJ9fV0sInR5cGUiOiJUUkFOU0ZPUk1FUiIsImltcGxlbWVudGF0aW9uIjoiVU5LTk9XTl9JTVBMRU1FTlRBVElPTiIsImVuZHBvaW50Ijp7InNlcnZpY2VfaG9zdCI6ImxvY2FsaG9zdCIsInNlcnZpY2VfcG9ydCI6ODA4NSwidHlwZSI6IlJFU1QifX1dLCJ0eXBlIjoiT1VUUFVUX1RSQU5TRk9STUVSIiwiaW1wbGVtZW50YXRpb24iOiJVTktOT1dOX0lNUExFTUVOVEFUSU9OIiwiZW5kcG9pbnQiOnsic2VydmljZV9ob3N0IjoibG9jYWxob3N0Iiwic2VydmljZV9wb3J0Ijo4MDg2LCJ0eXBlIjoiUkVTVCJ9fSwiY29tcG9uZW50U3BlY3MiOlt7Im1ldGFkYXRhIjp7ImNyZWF0aW9uVGltZXN0YW1wIjpudWxsfSwic3BlYyI6eyJjb250YWluZXJzIjpbeyJuYW1lIjoicm91dGVyIiwiaW1hZ2UiOiJzZWxkb25hY3IuYXp1cmVjci5pby9kZW1vL3NlbGRvbl9wcmVkaWN0OmxhdGVzdCIsInBvcnRzIjpbeyJuYW1lIjoiaHR0cCIsImNvbnRhaW5lclBvcnQiOjgwODB9LHsibmFtZSI6Im1ldHJpY3MiLCJjb250YWluZXJQb3J0Ijo2MDAwLCJwcm90b2NvbCI6IlRDUCJ9XSwiZW52IjpbeyJuYW1lIjoiUFJFRElDVElWRV9VTklUX1NFUlZJQ0VfUE9SVCIsInZhbHVlIjoiODA4MCJ9LHsibmFtZSI6IlBSRURJQ1RJVkVfVU5JVF9JRCIsInZhbHVlIjoicm91dGVyIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX0lNQUdFIiwidmFsdWUiOiJzZWxkb25hY3IuYXp1cmVjci5pby9kZW1vL3NlbGRvbl9wcmVkaWN0OmxhdGVzdCJ9LHsibmFtZSI6IlBSRURJQ1RPUl9JRCIsInZhbHVlIjoicGlwZWxpbmUtcHJlZGljdG9yIn0seyJuYW1lIjoiUFJFRElDVE9SX0xBQkVMUyIsInZhbHVlIjoie1widmVyc2lvblwiOlwicGlwZWxpbmUtcHJlZGljdG9yXCJ9In0seyJuYW1lIjoiU0VMRE9OX0RFUExPWU1FTlRfSUQiLCJ2YWx1ZSI6InBpcGVsaW5lIn0seyJuYW1lIjoiU0VMRE9OX0VYRUNVVE9SX0VOQUJMRUQiLCJ2YWx1ZSI6ImZhbHNlIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfU0VSVklDRV9QT1JUIiwidmFsdWUiOiI2MDAwIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfRU5EUE9JTlQiLCJ2YWx1ZSI6Ii9wcm9tZXRoZXVzIn1dLCJyZXNvdXJjZXMiOnt9LCJ2b2x1bWVNb3VudHMiOlt7Im5hbWUiOiJzZWxkb24tcG9kaW5mbyIsIm1vdW50UGF0aCI6Ii9ldGMvcG9kaW5mbyJ9XSwibGl2ZW5lc3NQcm9iZSI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifSwiaW5pdGlhbERlbGF5U2Vjb25kcyI6NjAsInRpbWVvdXRTZWNvbmRzIjoxLCJwZXJpb2RTZWNvbmRzIjo1LCJzdWNjZXNzVGhyZXNob2xkIjoxLCJmYWlsdXJlVGhyZXNob2xkIjozfSwicmVhZGluZXNzUHJvYmUiOnsidGNwU29ja2V0Ijp7InBvcnQiOiJodHRwIn0sImluaXRpYWxEZWxheVNlY29uZHMiOjIwLCJ0aW1lb3V0U2Vjb25kcyI6MSwicGVyaW9kU2Vjb25kcyI6NSwic3VjY2Vzc1RocmVzaG9sZCI6MSwiZmFpbHVyZVRocmVzaG9sZCI6M30sImxpZmVjeWNsZSI6eyJwcmVTdG9wIjp7ImV4ZWMiOnsiY29tbWFuZCI6WyIvYmluL3NoIiwiLWMiLCIvYmluL3NsZWVwIDEwIl19fX0sImltYWdlUHVsbFBvbGljeSI6IkFsd2F5cyJ9LHsibmFtZSI6Im1vZGVsLWEiLCJpbWFnZSI6InNlbGRvbmFjci5henVyZWNyLmlvL2RlbW8vc2VsZG9uX21vZGVsX2E6bGF0ZXN0IiwicG9ydHMiOlt7Im5hbWUiOiJodHRwIiwiY29udGFpbmVyUG9ydCI6ODA4MX0seyJuYW1lIjoibWV0cmljcyIsImNvbnRhaW5lclBvcnQiOjYwMDEsInByb3RvY29sIjoiVENQIn1dLCJlbnYiOlt7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfU0VSVklDRV9QT1JUIiwidmFsdWUiOiI4MDgxIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX0lEIiwidmFsdWUiOiJtb2RlbC1hIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX0lNQUdFIiwidmFsdWUiOiJzZWxkb25hY3IuYXp1cmVjci5pby9kZW1vL3NlbGRvbl9tb2RlbF9hOmxhdGVzdCJ9LHsibmFtZSI6IlBSRURJQ1RPUl9JRCIsInZhbHVlIjoicGlwZWxpbmUtcHJlZGljdG9yIn0seyJuYW1lIjoiUFJFRElDVE9SX0xBQkVMUyIsInZhbHVlIjoie1widmVyc2lvblwiOlwicGlwZWxpbmUtcHJlZGljdG9yXCJ9In0seyJuYW1lIjoiU0VMRE9OX0RFUExPWU1FTlRfSUQiLCJ2YWx1ZSI6InBpcGVsaW5lIn0seyJuYW1lIjoiU0VMRE9OX0VYRUNVVE9SX0VOQUJMRUQiLCJ2YWx1ZSI6ImZhbHNlIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfU0VSVklDRV9QT1JUIiwidmFsdWUiOiI2MDAxIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfRU5EUE9JTlQiLCJ2YWx1ZSI6Ii9wcm9tZXRoZXVzIn1dLCJyZXNvdXJjZXMiOnt9LCJ2b2x1bWVNb3VudHMiOlt7Im5hbWUiOiJzZWxkb24tcG9kaW5mbyIsIm1vdW50UGF0aCI6Ii9ldGMvcG9kaW5mbyJ9XSwibGl2ZW5lc3NQcm9iZSI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifSwiaW5pdGlhbERlbGF5U2Vjb25kcyI6NjAsInRpbWVvdXRTZWNvbmRzIjoxLCJwZXJpb2RTZWNvbmRzIjo1LCJzdWNjZXNzVGhyZXNob2xkIjoxLCJmYWlsdXJlVGhyZXNob2xkIjozfSwicmVhZGluZXNzUHJvYmUiOnsidGNwU29ja2V0Ijp7InBvcnQiOiJodHRwIn0sImluaXRpYWxEZWxheVNlY29uZHMiOjIwLCJ0aW1lb3V0U2Vjb25kcyI6MSwicGVyaW9kU2Vjb25kcyI6NSwic3VjY2Vzc1RocmVzaG9sZCI6MSwiZmFpbHVyZVRocmVzaG9sZCI6M30sImxpZmVjeWNsZSI6eyJwcmVTdG9wIjp7ImV4ZWMiOnsiY29tbWFuZCI6WyIvYmluL3NoIiwiLWMiLCIvYmluL3NsZWVwIDEwIl19fX0sImltYWdlUHVsbFBvbGljeSI6IkFsd2F5cyJ9LHsibmFtZSI6Im1vZGVsLWIiLCJpbWFnZSI6InNlbGRvbmFjci5henVyZWNyLmlvL2RlbW8vc2VsZG9uX21vZGVsX2I6bGF0ZXN0IiwicG9ydHMiOlt7Im5hbWUiOiJodHRwIiwiY29udGFpbmVyUG9ydCI6ODA4M30seyJuYW1lIjoibWV0cmljcyIsImNvbnRhaW5lclBvcnQiOjYwMDIsInByb3RvY29sIjoiVENQIn1dLCJlbnYiOlt7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfU0VSVklDRV9QT1JUIiwidmFsdWUiOiI4MDgzIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX0lEIiwidmFsdWUiOiJtb2RlbC1iIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX0lNQUdFIiwidmFsdWUiOiJzZWxkb25hY3IuYXp1cmVjci5pby9kZW1vL3NlbGRvbl9tb2RlbF9iOmxhdGVzdCJ9LHsibmFtZSI6IlBSRURJQ1RPUl9JRCIsInZhbHVlIjoicGlwZWxpbmUtcHJlZGljdG9yIn0seyJuYW1lIjoiUFJFRElDVE9SX0xBQkVMUyIsInZhbHVlIjoie1widmVyc2lvblwiOlwicGlwZWxpbmUtcHJlZGljdG9yXCJ9In0seyJuYW1lIjoiU0VMRE9OX0RFUExPWU1FTlRfSUQiLCJ2YWx1ZSI6InBpcGVsaW5lIn0seyJuYW1lIjoiU0VMRE9OX0VYRUNVVE9SX0VOQUJMRUQiLCJ2YWx1ZSI6ImZhbHNlIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfU0VSVklDRV9QT1JUIiwidmFsdWUiOiI2MDAyIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfRU5EUE9JTlQiLCJ2YWx1ZSI6Ii9wcm9tZXRoZXVzIn1dLCJyZXNvdXJjZXMiOnt9LCJ2b2x1bWVNb3VudHMiOlt7Im5hbWUiOiJzZWxkb24tcG9kaW5mbyIsIm1vdW50UGF0aCI6Ii9ldGMvcG9kaW5mbyJ9XSwibGl2ZW5lc3NQcm9iZSI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifSwiaW5pdGlhbERlbGF5U2Vjb25kcyI6NjAsInRpbWVvdXRTZWNvbmRzIjoxLCJwZXJpb2RTZWNvbmRzIjo1LCJzdWNjZXNzVGhyZXNob2xkIjoxLCJmYWlsdXJlVGhyZXNob2xkIjozfSwicmVhZGluZXNzUHJvYmUiOnsidGNwU29ja2V0Ijp7InBvcnQiOiJodHRwIn0sImluaXRpYWxEZWxheVNlY29uZHMiOjIwLCJ0aW1lb3V0U2Vjb25kcyI6MSwicGVyaW9kU2Vjb25kcyI6NSwic3VjY2Vzc1RocmVzaG9sZCI6MSwiZmFpbHVyZVRocmVzaG9sZCI6M30sImxpZmVjeWNsZSI6eyJwcmVTdG9wIjp7ImV4ZWMiOnsiY29tbWFuZCI6WyIvYmluL3NoIiwiLWMiLCIvYmluL3NsZWVwIDEwIl19fX0sImltYWdlUHVsbFBvbGljeSI6IkFsd2F5cyJ9LHsibmFtZSI6Im1vZGVsLWMiLCJpbWFnZSI6InNlbGRvbmFjci5henVyZWNyLmlvL2RlbW8vc2VsZG9uX21vZGVsX2M6bGF0ZXN0IiwicG9ydHMiOlt7Im5hbWUiOiJodHRwIiwiY29udGFpbmVyUG9ydCI6ODA4NH0seyJuYW1lIjoibWV0cmljcyIsImNvbnRhaW5lclBvcnQiOjYwMDMsInByb3RvY29sIjoiVENQIn1dLCJlbnYiOlt7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfU0VSVklDRV9QT1JUIiwidmFsdWUiOiI4MDg0In0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX0lEIiwidmFsdWUiOiJtb2RlbC1jIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX0lNQUdFIiwidmFsdWUiOiJzZWxkb25hY3IuYXp1cmVjci5pby9kZW1vL3NlbGRvbl9tb2RlbF9jOmxhdGVzdCJ9LHsibmFtZSI6IlBSRURJQ1RPUl9JRCIsInZhbHVlIjoicGlwZWxpbmUtcHJlZGljdG9yIn0seyJuYW1lIjoiUFJFRElDVE9SX0xBQkVMUyIsInZhbHVlIjoie1widmVyc2lvblwiOlwicGlwZWxpbmUtcHJlZGljdG9yXCJ9In0seyJuYW1lIjoiU0VMRE9OX0RFUExPWU1FTlRfSUQiLCJ2YWx1ZSI6InBpcGVsaW5lIn0seyJuYW1lIjoiU0VMRE9OX0VYRUNVVE9SX0VOQUJMRUQiLCJ2YWx1ZSI6ImZhbHNlIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfU0VSVklDRV9QT1JUIiwidmFsdWUiOiI2MDAzIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfRU5EUE9JTlQiLCJ2YWx1ZSI6Ii9wcm9tZXRoZXVzIn1dLCJyZXNvdXJjZXMiOnt9LCJ2b2x1bWVNb3VudHMiOlt7Im5hbWUiOiJzZWxkb24tcG9kaW5mbyIsIm1vdW50UGF0aCI6Ii9ldGMvcG9kaW5mbyJ9XSwibGl2ZW5lc3NQcm9iZSI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifSwiaW5pdGlhbERlbGF5U2Vjb25kcyI6NjAsInRpbWVvdXRTZWNvbmRzIjoxLCJwZXJpb2RTZWNvbmRzIjo1LCJzdWNjZXNzVGhyZXNob2xkIjoxLCJmYWlsdXJlVGhyZXNob2xkIjozfSwicmVhZGluZXNzUHJvYmUiOnsidGNwU29ja2V0Ijp7InBvcnQiOiJodHRwIn0sImluaXRpYWxEZWxheVNlY29uZHMiOjIwLCJ0aW1lb3V0U2Vjb25kcyI6MSwicGVyaW9kU2Vjb25kcyI6NSwic3VjY2Vzc1RocmVzaG9sZCI6MSwiZmFpbHVyZVRocmVzaG9sZCI6M30sImxpZmVjeWNsZSI6eyJwcmVTdG9wIjp7ImV4ZWMiOnsiY29tbWFuZCI6WyIvYmluL3NoIiwiLWMiLCIvYmluL3NsZWVwIDEwIl19fX0sImltYWdlUHVsbFBvbGljeSI6IkFsd2F5cyJ9LHsibmFtZSI6ImlucHV0LXRyYW5zZm9ybWVyIiwiaW1hZ2UiOiJzZWxkb25hY3IuYXp1cmVjci5pby9kZW1vL3NlbGRvbl9wcmVkaWN0OmxhdGVzdCIsInBvcnRzIjpbeyJuYW1lIjoiaHR0cCIsImNvbnRhaW5lclBvcnQiOjgwODV9LHsibmFtZSI6Im1ldHJpY3MiLCJjb250YWluZXJQb3J0Ijo2MDA0LCJwcm90b2NvbCI6IlRDUCJ9XSwiZW52IjpbeyJuYW1lIjoiUFJFRElDVElWRV9VTklUX1NFUlZJQ0VfUE9SVCIsInZhbHVlIjoiODA4NSJ9LHsibmFtZSI6IlBSRURJQ1RJVkVfVU5JVF9JRCIsInZhbHVlIjoiaW5wdXQtdHJhbnNmb3JtZXIifSx7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfSU1BR0UiLCJ2YWx1ZSI6InNlbGRvbmFjci5henVyZWNyLmlvL2RlbW8vc2VsZG9uX3ByZWRpY3Q6bGF0ZXN0In0seyJuYW1lIjoiUFJFRElDVE9SX0lEIiwidmFsdWUiOiJwaXBlbGluZS1wcmVkaWN0b3IifSx7Im5hbWUiOiJQUkVESUNUT1JfTEFCRUxTIiwidmFsdWUiOiJ7XCJ2ZXJzaW9uXCI6XCJwaXBlbGluZS1wcmVkaWN0b3JcIn0ifSx7Im5hbWUiOiJTRUxET05fREVQTE9ZTUVOVF9JRCIsInZhbHVlIjoicGlwZWxpbmUifSx7Im5hbWUiOiJTRUxET05fRVhFQ1VUT1JfRU5BQkxFRCIsInZhbHVlIjoiZmFsc2UifSx7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfTUVUUklDU19TRVJWSUNFX1BPUlQiLCJ2YWx1ZSI6IjYwMDQifSx7Im5hbWUiOiJQUkVESUNUSVZFX1VOSVRfTUVUUklDU19FTkRQT0lOVCIsInZhbHVlIjoiL3Byb21ldGhldXMifV0sInJlc291cmNlcyI6e30sInZvbHVtZU1vdW50cyI6W3sibmFtZSI6InNlbGRvbi1wb2RpbmZvIiwibW91bnRQYXRoIjoiL2V0Yy9wb2RpbmZvIn1dLCJsaXZlbmVzc1Byb2JlIjp7InRjcFNvY2tldCI6eyJwb3J0IjoiaHR0cCJ9LCJpbml0aWFsRGVsYXlTZWNvbmRzIjo2MCwidGltZW91dFNlY29uZHMiOjEsInBlcmlvZFNlY29uZHMiOjUsInN1Y2Nlc3NUaHJlc2hvbGQiOjEsImZhaWx1cmVUaHJlc2hvbGQiOjN9LCJyZWFkaW5lc3NQcm9iZSI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifSwiaW5pdGlhbERlbGF5U2Vjb25kcyI6MjAsInRpbWVvdXRTZWNvbmRzIjoxLCJwZXJpb2RTZWNvbmRzIjo1LCJzdWNjZXNzVGhyZXNob2xkIjoxLCJmYWlsdXJlVGhyZXNob2xkIjozfSwibGlmZWN5Y2xlIjp7InByZVN0b3AiOnsiZXhlYyI6eyJjb21tYW5kIjpbIi9iaW4vc2giLCItYyIsIi9iaW4vc2xlZXAgMTAiXX19fSwiaW1hZ2VQdWxsUG9saWN5IjoiQWx3YXlzIn0seyJuYW1lIjoib3V0cHV0LXRyYW5zZm9ybWVyIiwiaW1hZ2UiOiJzZWxkb25hY3IuYXp1cmVjci5pby9kZW1vL3NlbGRvbl9wcmVkaWN0OmxhdGVzdCIsInBvcnRzIjpbeyJuYW1lIjoiaHR0cCIsImNvbnRhaW5lclBvcnQiOjgwODZ9LHsibmFtZSI6Im1ldHJpY3MiLCJjb250YWluZXJQb3J0Ijo2MDA1LCJwcm90b2NvbCI6IlRDUCJ9XSwiZW52IjpbeyJuYW1lIjoiUFJFRElDVElWRV9VTklUX1NFUlZJQ0VfUE9SVCIsInZhbHVlIjoiODA4NiJ9LHsibmFtZSI6IlBSRURJQ1RJVkVfVU5JVF9JRCIsInZhbHVlIjoib3V0cHV0LXRyYW5zZm9ybWVyIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX0lNQUdFIiwidmFsdWUiOiJzZWxkb25hY3IuYXp1cmVjci5pby9kZW1vL3NlbGRvbl9wcmVkaWN0OmxhdGVzdCJ9LHsibmFtZSI6IlBSRURJQ1RPUl9JRCIsInZhbHVlIjoicGlwZWxpbmUtcHJlZGljdG9yIn0seyJuYW1lIjoiUFJFRElDVE9SX0xBQkVMUyIsInZhbHVlIjoie1widmVyc2lvblwiOlwicGlwZWxpbmUtcHJlZGljdG9yXCJ9In0seyJuYW1lIjoiU0VMRE9OX0RFUExPWU1FTlRfSUQiLCJ2YWx1ZSI6InBpcGVsaW5lIn0seyJuYW1lIjoiU0VMRE9OX0VYRUNVVE9SX0VOQUJMRUQiLCJ2YWx1ZSI6ImZhbHNlIn0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfU0VSVklDRV9QT1JUIiwidmFsdWUiOiI2MDA1In0seyJuYW1lIjoiUFJFRElDVElWRV9VTklUX01FVFJJQ1NfRU5EUE9JTlQiLCJ2YWx1ZSI6Ii9wcm9tZXRoZXVzIn1dLCJyZXNvdXJjZXMiOnt9LCJ2b2x1bWVNb3VudHMiOlt7Im5hbWUiOiJzZWxkb24tcG9kaW5mbyIsIm1vdW50UGF0aCI6Ii9ldGMvcG9kaW5mbyJ9XSwibGl2ZW5lc3NQcm9iZSI6eyJ0Y3BTb2NrZXQiOnsicG9ydCI6Imh0dHAifSwiaW5pdGlhbERlbGF5U2Vjb25kcyI6NjAsInRpbWVvdXRTZWNvbmRzIjoxLCJwZXJpb2RTZWNvbmRzIjo1LCJzdWNjZXNzVGhyZXNob2xkIjoxLCJmYWlsdXJlVGhyZXNob2xkIjozfSwicmVhZGluZXNzUHJvYmUiOnsidGNwU29ja2V0Ijp7InBvcnQiOiJodHRwIn0sImluaXRpYWxEZWxheVNlY29uZHMiOjIwLCJ0aW1lb3V0U2Vjb25kcyI6MSwicGVyaW9kU2Vjb25kcyI6NSwic3VjY2Vzc1RocmVzaG9sZCI6MSwiZmFpbHVyZVRocmVzaG9sZCI6M30sImxpZmVjeWNsZSI6eyJwcmVTdG9wIjp7ImV4ZWMiOnsiY29tbWFuZCI6WyIvYmluL3NoIiwiLWMiLCIvYmluL3NsZWVwIDEwIl19fX0sImltYWdlUHVsbFBvbGljeSI6IkFsd2F5cyJ9XX19XSwicmVwbGljYXMiOjAsImVuZ2luZVJlc291cmNlcyI6e30sImxhYmVscyI6eyJ2ZXJzaW9uIjoicGlwZWxpbmUtcHJlZGljdG9yIn0sInN2Y09yY2hTcGVjIjp7fX0=
      DEPLOYMENT_NAME:                 pipeline
      DEPLOYMENT_NAMESPACE:            seldon-system
      ENGINE_SERVER_PORT:              8000
      ENGINE_SERVER_GRPC_PORT:         5001
      JAVA_OPTS:                       -server
      SELDON_LOG_MESSAGES_EXTERNALLY:  false
    Mounts:
      /etc/podinfo from seldon-podinfo (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-5ztmq (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  seldon-podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.annotations -> annotations
  default-token-5ztmq:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-5ztmq
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason     Age                   From                               Message
  ----     ------     ----                  ----                               -------
  Normal   Scheduled  6m9s                  default-scheduler                  Successfully assigned seldon-system/seldon-fe3d02ba53e29f068d3499250bbcdd5f-7f85df48c7-ccpz2 to aks-nodepool1-12337826-1
  Normal   Pulling    6m8s                  kubelet, aks-nodepool1-12337826-1  Pulling image "seldonacr.azurecr.io/demo/seldon_predict:latest"
  Normal   Pulled     6m8s                  kubelet, aks-nodepool1-12337826-1  Successfully pulled image "seldonacr.azurecr.io/demo/seldon_predict:latest"
  Normal   Created    6m8s                  kubelet, aks-nodepool1-12337826-1  Created container router
  Normal   Created    6m7s                  kubelet, aks-nodepool1-12337826-1  Created container model-a
  Normal   Pulling    6m7s                  kubelet, aks-nodepool1-12337826-1  Pulling image "seldonacr.azurecr.io/demo/seldon_model_a:latest"
  Normal   Pulled     6m7s                  kubelet, aks-nodepool1-12337826-1  Successfully pulled image "seldonacr.azurecr.io/demo/seldon_model_a:latest"
  Normal   Started    6m7s                  kubelet, aks-nodepool1-12337826-1  Started container router
  Normal   Started    6m7s                  kubelet, aks-nodepool1-12337826-1  Started container model-a
  Normal   Pulling    6m7s                  kubelet, aks-nodepool1-12337826-1  Pulling image "seldonacr.azurecr.io/demo/seldon_model_b:latest"
  Normal   Pulled     6m6s                  kubelet, aks-nodepool1-12337826-1  Successfully pulled image "seldonacr.azurecr.io/demo/seldon_model_b:latest"
  Normal   Created    6m6s                  kubelet, aks-nodepool1-12337826-1  Created container model-b
  Normal   Started    6m6s                  kubelet, aks-nodepool1-12337826-1  Started container model-b
  Normal   Pulled     6m6s                  kubelet, aks-nodepool1-12337826-1  Successfully pulled image "seldonacr.azurecr.io/demo/seldon_model_c:latest"
  Normal   Pulling    6m6s                  kubelet, aks-nodepool1-12337826-1  Pulling image "seldonacr.azurecr.io/demo/seldon_model_c:latest"
  Normal   Created    6m5s                  kubelet, aks-nodepool1-12337826-1  Created container model-c
  Normal   Started    6m5s                  kubelet, aks-nodepool1-12337826-1  Started container model-c
  Normal   Pulling    6m5s                  kubelet, aks-nodepool1-12337826-1  Pulling image "seldonacr.azurecr.io/demo/seldon_predict:latest"
  Normal   Pulled     6m5s                  kubelet, aks-nodepool1-12337826-1  Successfully pulled image "seldonacr.azurecr.io/demo/seldon_predict:latest"
  Normal   Created    6m5s                  kubelet, aks-nodepool1-12337826-1  Created container input-transformer
  Normal   Started    6m5s                  kubelet, aks-nodepool1-12337826-1  Started container input-transformer
  Normal   Pulling    6m5s                  kubelet, aks-nodepool1-12337826-1  Pulling image "seldonacr.azurecr.io/demo/seldon_predict:latest"
  Normal   Pulled     6m5s                  kubelet, aks-nodepool1-12337826-1  Successfully pulled image "seldonacr.azurecr.io/demo/seldon_predict:latest"
  Normal   Created    6m4s                  kubelet, aks-nodepool1-12337826-1  Created container output-transformer
  Normal   Started    6m4s                  kubelet, aks-nodepool1-12337826-1  Started container output-transformer
  Normal   Pulled     6m4s                  kubelet, aks-nodepool1-12337826-1  Container image "docker.io/seldonio/engine:1.3.0" already present on machine
  Warning  Unhealthy  67s (x36 over 5m42s)  kubelet, aks-nodepool1-12337826-1  Readiness probe failed: Get https://10.244.1.8:8082/ready: http: server gave HTTP response to HTTPS client

Then container log of Seldon-container-engine looks like below:


  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v2.2.0.RELEASE)

2020-10-01 20:18:23.839  WARN 6 --- [           main] o.s.h.c.j.Jackson2ObjectMapperBuilder    : For Jackson Kotlin classes support please add "com.fasterxml.jackson.module:jackson-module-kotlin" to the classpath

Any tips?

RafalSkolasinski · 2020-10-01T20:35:39Z

Hi @theone4ever, thanks for reporting that. This is quite unfortunate - could you please open a separate issue about it?

theone4ever · 2020-10-01T20:38:03Z

@RafalSkolasinski sure, but before that just want to double confirm if java engine is still supported in 1.3? In doc here, it says:
You can continue to use the Java engine Service Orchestrator but this will be deprecated in release 1.2.: https://docs.seldon.io/projects/seldon-core/en/v1.1.0/graph/svcorch.html#using-the-java-engine

RafalSkolasinski · 2020-10-01T20:42:18Z

It's considered last resort fallback if for some reason one really, really cannot run with Executor. The above (this issue) seems the only missing piece and we believe only in context of multi-armed bandits...

Saying that we don't really develop java engine anymore and are seriously considering fully dropping it (probably with 1.4 though not decided)

theone4ever · 2020-10-01T20:50:34Z

ok, then in 1.3, at least java engine should still work, right @RafalSkolasinski ? In this case, I will report another issue then.

RafalSkolasinski · 2020-10-02T08:30:30Z

@theone4ever, yes, please - we briefly looked at your logs and believe that the problem is that your liveness and readiness probes are configured to use https endpoint when in reality they are http ones. But it's better to discuss this in separate issue and keep this one for routing metadata only

adriangonz · 2020-10-27T17:21:01Z

Hey @theone4ever @RafalSkolasinski, I've opened up a new issue with the Java engine issue you described. We can continue the discussion there: #2586

RafalSkolasinski added bug triage Needs to be triaged and prioritised accordingly labels May 13, 2020

ukclivecox added priority/p1 and removed triage Needs to be triaged and prioritised accordingly labels May 14, 2020

ukclivecox added priority/p0 and removed priority/p1 labels Jun 4, 2020

ukclivecox assigned RafalSkolasinski Jun 25, 2020

ukclivecox added this to the 1.2 milestone Jun 25, 2020

ukclivecox modified the milestones: 1.2, 1.3 Jul 9, 2020

axsaucedo modified the milestones: 1.3, 1.2 Jul 15, 2020

ukclivecox modified the milestones: 1.2, 1.3 Jul 16, 2020

ukclivecox added the Metrics label Aug 20, 2020

ukclivecox modified the milestones: 1.3, 1.4 Sep 17, 2020

axsaucedo modified the milestones: 1.4, 1.5 Oct 15, 2020

ukclivecox modified the milestones: 1.5, 1.6 Nov 30, 2020

axsaucedo mentioned this issue Nov 30, 2020

Routing metadata and fixed MAB case study #2723

Merged

seldondev closed this as completed in #2723 Dec 2, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

routing metadata missing in new executor #1823

routing metadata missing in new executor #1823

RafalSkolasinski commented May 13, 2020

RafalSkolasinski commented Jun 25, 2020

amirclam commented Jul 15, 2020

ukclivecox commented Jul 15, 2020

RafalSkolasinski commented Jul 15, 2020 •

edited

Loading

theone4ever commented Sep 30, 2020

RafalSkolasinski commented Sep 30, 2020

theone4ever commented Sep 30, 2020

RafalSkolasinski commented Sep 30, 2020 •

edited

Loading

theone4ever commented Oct 1, 2020

RafalSkolasinski commented Oct 1, 2020

theone4ever commented Oct 1, 2020

RafalSkolasinski commented Oct 1, 2020 •

edited

Loading

theone4ever commented Oct 1, 2020

RafalSkolasinski commented Oct 2, 2020

adriangonz commented Oct 27, 2020

routing metadata missing in new executor #1823

routing metadata missing in new executor #1823

Comments

RafalSkolasinski commented May 13, 2020

RafalSkolasinski commented Jun 25, 2020

amirclam commented Jul 15, 2020

ukclivecox commented Jul 15, 2020

RafalSkolasinski commented Jul 15, 2020 • edited Loading

theone4ever commented Sep 30, 2020

RafalSkolasinski commented Sep 30, 2020

theone4ever commented Sep 30, 2020

RafalSkolasinski commented Sep 30, 2020 • edited Loading

theone4ever commented Oct 1, 2020

RafalSkolasinski commented Oct 1, 2020

theone4ever commented Oct 1, 2020

RafalSkolasinski commented Oct 1, 2020 • edited Loading

theone4ever commented Oct 1, 2020

RafalSkolasinski commented Oct 2, 2020

adriangonz commented Oct 27, 2020

RafalSkolasinski commented Jul 15, 2020 •

edited

Loading

RafalSkolasinski commented Sep 30, 2020 •

edited

Loading

RafalSkolasinski commented Oct 1, 2020 •

edited

Loading