End-to-end tests for Pre-packaged model servers hang if name doesn't match exactly

[axsaucedo]

When running e2e tests the rollout deployment checks are done with the exact string of automatically generated deployment name - i.e.:

seldon-core/testing/scripts/test_prepackaged_servers.py

Line 35 in 60c9fd2

wait_for_rollout("iris-default-8bb3ef6")

If the model is created with a different name the deployment doesn't start. A fix could be to monitor the deployment through the label name as opposed to the generated name.

[ukclivecox]

I agree a better way to wait for rollout that is less brittle is needed.

[RafalSkolasinski]

Do we have cases when the deployment failed? How could I reproduce the failure?

Do I understand correctly that scope of the fix would be to modify wait_for_rollout function

seldon-core/testing/scripts/test_prepackaged_servers.py

Line 21 in 60c9fd2

def wait_for_rollout(deploymentName):

to monitor rollout status using labels and add appropriate labels to object's yaml definitions, e.g. here?

[RafalSkolasinski]

It seems that kubectl rollout status is expecting a deployment name, not labels.
I see two options:

1. Add labels to deployments and knowing it and the part of name (in context of example above: iris-default) find full name of the deployment using kubectl get ...
2. Write a python function that process yaml object definition and generate full name

I did try to do 2. Following function

def deployment_name(fname):
    with open(fname, 'r') as f:
        data = yaml.safe_load(f.read())
    
    sdep_name = data['metadata']['name']
    predictor_spec = data['spec']['predictors'][0]
    pod_spec = predictor_spec['componentSpecs'][0]['spec']

    s = []
    for container in pod_spec['containers']:
        s.append(container['name'])
        s.append(container['image'])

    s = ":".join(s) + ";"
    pod_hash = hashlib.md5(s.encode()).hexdigest()[:7]            
    
    sdep_name = "-".join([sdep_name, predictor_spec['graph']['name'], pod_hash])
    return sdep_name

seems to work properly on yaml's that define containers, .e.g this one but does not work on ones that do not define containers, e.g. iris.yml.

[RafalSkolasinski]

In case of iris.yml the name comes from pod having container classifier that uses seldonio/sklearnserver_rest:0.2 image:

>>> name = "classifier"
>>> image = "seldonio/sklearnserver_rest:0.2"
>>> s = f"{name}:{image};"
>>> hashlib.md5(s.encode()).hexdigest()[:7]    
4903e3c

As the yaml file does not contain information about which image will be used it may be better to indeed go with adding labels and filtering by them manually, a.k.a. option 1 in previous comment.

[RafalSkolasinski]

I think I may have found another option. I believe that SeldonDeployment names must be unique in within a namespace. If that is true I can get name of deployments with for example

>>> import yaml
>>> from subprocess import run
>>> ret = run('kubectl get -n seldon seldondeployment sklearn -o yaml', shell=True, capture_output=True)
>>> data = yaml.safe_load(ret.stdout.decode())
>>> list(data['status']['deploymentStatus'])
['iris-default-4903e3c']

@axsaucedo @adriangonz What do you think? It seems like simplest and shortest solution.

[RafalSkolasinski]

I pushed proof of concept fix. Check #1315.