requests per second from seldon_api_executor_cient_requests_count not right?

[chengcheng-pei]

Describe the bug

I used the following three to get RPS:

1, sum(rate(seldon_api_executor_client_requests_seconds_count{predictor_name=~"tf-rest.*"}[60s]))

2, sum(rate(seldon_api_executor_server_requests_seconds_count{predictor_name=~"tf-rest.*"}[60s]))

3, I used python codes to count the requests per second to model.

sum by (app) (rate(rest_requests_total{app=~"tf-.*bm"}[60s]))

python codes:

c = Counter('{}_requests'.format("rest"), 'REST requests')
c.inc()

But the first two are about HALF of the third

Any idea why? Thanks

To reproduce

Expected behaviour

Environment

• Kubernetes Version: [e.g. 1.15]
• Cloud Provider: [e.g. GKE, AWS, Bare Metal, Kind, Minikube]
• Deployed Seldon System Images: [Output of kubectl get --namespace seldon-system deploy seldon-controller-manager -o yaml | grep seldonio]
• Kubernetes Cluster Version [Output of kubectl version]

Model Details

• Images of your model: [Output of: kubectl get seldondeployment -n <yourmodelnamespace> <seldondepname> -o yaml | grep image: where <yourmodelnamespace>]
• Logs of your model: [You can get the logs of your model by running kubectl logs -n <yourmodelnamespace> <seldonpodname> <container>]

[ukclivecox]

So the third is a custom metric in your python class? Can you show how you create it?

[chengcheng-pei]

    c = Counter('{}_requests'.format("rest"), 'REST requests')
    while True:
        data_ = random_d() if deploy == 'xgboost-rest' else random_d2()
        start = time.time()
        r = requests.post('http://{}.my-stage.net/api/v1.0/predictions'.format(hostname), json={ "data": { "ndarray": data_ } })
        delta = time.time() - start
        h.observe(delta)
        c.inc()

[ukclivecox]

You show the counter value. Have you checked the histogram total as well?
Have you check in python code what the req/sec is to get the truth?