requests all failed during benchmarking with Locust

[amigniox]

I'm running a loadtest to see if we can reproduce the seldon REST benchmarking results here
https://docs.seldon.io/projects/seldon-core/en/latest/reference/benchmarking.html
but the unlike the REST results shown on the above webpage, in my test requests were all failed, I've tried the seldon-core-loadtesting helm chart version 0.1.1 and 0.1.2, image version seldonio/locust-core:0.6 and 0.8, the results are all the same.
some additional info:
when I check the statics .csv files in the locust master pod:

"Method","Name","# requests","# failures","Median response time","Average response time","Min response time","Max response time","Average Content Size","Requests/s"
"POST","predictions",375860,375860,45,110,7,15209,0,260.08
"None","Total",375860,375860,45,110,17,15209,0,260.0

and I checked the pod logs of the locust slave, the following error was found

[2019-06-20 14:22:55,078] locust-slave-1-6ql4p/INFO/stdout: Failed prediction request 0
[2019-06-20 14:22:55,078] locust-slave-1-6ql4p/INFO/stdout: 
[2019-06-20 14:22:55,078] locust-slave-1-6ql4p/INFO/stdout: {}
[2019-06-20 14:22:55,090] locust-slave-1-6ql4p/INFO/stdout: 
[2019-06-20 14:22:55,091] locust-slave-1-6ql4p/ERROR/stderr: Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/locustio-0.11.1-py2.7.egg/locust/core.py", line 361, in run
    self.execute_next_task()
  File "/usr/local/lib/python2.7/dist-packages/locustio-0.11.1-py2.7.egg/locust/core.py", line 387, in execute_next_task
    self.execute_task(task["callable"], *task["args"], **task["kwargs"])
  File "/usr/local/lib/python2.7/dist-packages/locustio-0.11.1-py2.7.egg/locust/core.py", line 399, in execute_task
    task(self, *args, **kwargs)
  File "/home/seldon/scripts/predict_rest_locust.py", line 147, in getPrediction
    r.raise_for_status()
  File "/usr/local/lib/python2.7/dist-packages/locustio-0.11.1-py2.7.egg/locust/clients.py", line 23, in raise_for_status
    raise self.error
ConnectionError: HTTPConnectionPool(host='loadtest', port=8000): Max retries exceeded with url: /api/v0.1/predictions (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efef416e210>: Failed to establish a new connection: [Errno -2] Name or service not known',))

Configs when deploying the loadtest:

helm install ../helm-charts/seldon-core-loadtesting --name loadtest  \
    --set locust.host=http://loadtest:8000 \
    --set oauth.enabled=false \
    --set oauth.key=oauth-key \
    --set oauth.secret=oauth-secret \
    --set locust.hatchRate=1 \
    --set locust.clients=256 \
    --set loadtest.sendFeedback=0 \
    --set locust.minWait=0 \
    --set locust.maxWait=0 \
    --set replicaCount=64

[ukclivecox]

#649 has a fix but you need to run 0.3.1-SNAPSHOT image of the seldon-core-operator. If you clone the project and use helm to install seldon-core-operator you will get this by default.