Adding the capability of writing custom endpoints

[saeid93]

Hello Seldon team,

As part of a special usecase I need to access my custom server from an endpoint. In other words, I want to add some custom logic to my custom Seldon server other than the existing one that could be triggered an endpoint like the existing ones (POST /api/v1.0/predictions or POST /api/v1.0/feedback). E.g. I want to change the loaded model from a custom endpoint like POST /api/v1.0/switchmodel (just as an example usecase). So far I have come up with a hacky way to do that via some hardcoding in my custom server. But, I think it would be nice to add the capability of adding custom endpoints with a predefined format to the CRDs that automatically generate the endpoints for the new function defined in the custom server to the Seldon core engine. Becuase I think there might be other cases in production systems that Seldon core users need to add they own functions to the server. I had a quick look at the microservice.py which is called at the entry point of custom servers and then the imported wrapper.py file and it seems the routes of both Kserve and Seldon protocal are defined in the wrapper file

...
    @app.route("/predict", methods=["GET", "POST"])
    @app.route("/api/v1.0/predictions", methods=["POST"])
    @app.route("/api/v0.1/predictions", methods=["POST"])
    def Predict():
        requestJson = get_request(skip_decoding=PAYLOAD_PASSTHROUGH)
        logger.debug("REST Request: %s", request)
        response = seldon_core.seldon_methods.predict(
            user_model, requestJson, seldon_metrics
        )
...

I think the feature I mentioned should be some way to make Seldon cable add this routes dynamically (within an standard format) to the wrapper file from the CRDs and then the user defines a function in the custom python SDK. E.g.

class ModelWithCustomEndpoint(object):
    def __init__(self) -> None:
        self.mult = 2

    def predict(self, X, features_names=None):
        return X*self.mult

    def change_mult(self, input):
        self.mult = input

and change_mult function would be accessible via POST /api/v1.0/change_mult

[MattiaGallegati]

I have an identical use case to solve. How have you solved this in your custom server?
I will argue that from an "MLOps" perspective this will not be a "fair" use case, since we have to deliver new and indipendent service for each model we will deploy but practice is far from books :) .
Definetely there will be more use cases for this.
While Seldon team gives an official response you can have a look at MLServer (for V2 protocol), it has the undocumented ability to add custom endpoints, you can look at issue #96 where I was discussing this things.

[ukclivecox]

The simplest way is probably to extend the code we provide to add the endpoints you want. As @MattiaGallegati suggested there is also undocumented code in MLServer for custom endpoints that could be investigated.

[saeid93]

@MattiaGallegati Thank you for your feedback, I basically extended microservice.py file and added an extra endpoint for my work, however that required to copy my patch of Seldon core in the containers. The MLServer one looks interesting, I'll check that. Thank for suggesting the best practice, actually this is a research work and right now I'm mostly concerned to get it up and running and then try to apply best practices but I will definitely take a note of your comment for our later stages. If you are interested a related paper that is using the exact same on the fly model switching is Model-Switching: Dealing with Fluctuating Workloads in Machine-Learning-as-a-Service Systems.

@cliveseldon Thank you! I'll try MLServer and I think I am already using the first solution you mentioned.

[ukclivecox]

At present we support only v2 protocol.