bad credentials error with get_token function

[paul-crease]

I am unable to run the deep_mnist demo, the oauth request returns
{"timestamp":1524558631450,"status":401,"error":"Unauthorized","message":"Bad credentials","path":"/oauth/token"}

I am trying to run this on :

K8s version 1.10.0 
minikube V0.26.0
s2i v1.1.9a-dirty
OSX v10.11.5.

Method to reproduce:

I cloned the repository 23/04/18 and followed https://github.com/SeldonIO/seldon-core/tree/master/examples/models/deep_mnist without making any changes to the code.

Expected Result:
I can run python ../../../util/api_tester/api-tester.py contract.json 0.0.0.0 8080 --oauth-key oauth-key --oauth-secret oauth-secret -p and it will return a prediction response

Actual Result:
CLI output is

{"timestamp":1524558631450,"status":401,"error":"Unauthorized","message":"Bad credentials","path":"/oauth/token"}
Traceback (most recent call last):
  File "../../../util/api_tester/api-tester.py", line 202, in <module>
    run(args)
  File "../../../util/api_tester/api-tester.py", line 159, in run
    token = get_token(args)
  File "../../../util/api_tester/api-tester.py", line 136, in get_token
    token =  response.json()["access_token"]
KeyError: 'access_token'

K8s Logs for the seldon-apiserver-container pod logs contain the following, and this repeats every time I run the api-tester.py request example.

io.kubernetes.client.ApiException: Forbidden
	at io.kubernetes.client.util.Watch.createWatch(Watch.java:81) ~[client-java-0.2.jar!/:na]
	at io.seldon.apife.k8s.DeploymentWatcher.watchSeldonSeldonDeployments(DeploymentWatcher.java:125) ~[classes!/:0.1.6]
	at io.seldon.apife.k8s.DeploymentWatcher.watch(DeploymentWatcher.java:181) ~[classes!/:0.1.6]
	at sun.reflect.GeneratedMethodAccessor66.invoke(Unknown Source) ~[na:na]
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_131]
	at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_131]
	at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65) ~[spring-context-4.3.11.RELEASE.jar!/:4.3.11.RELEASE]
	at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54) ~[spring-context-4.3.11.RELEASE.jar!/:4.3.11.RELEASE]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_131]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_131]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_131]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_131]
	at java.lang.Thread.run(Thread.java:748) [na:1.8.0_131]
 2018-04-24 08:51:18.803  INFO 6 --- [pool-3-thread-1] io.seldon.apife.k8s.DeploymentWatcher    : The time is now 08:51:18
2018-04-24 08:51:18.803  INFO 6 --- [pool-3-thread-1] io.seldon.apife.k8s.DeploymentWatcher    : Watching with rs null
2018-04-24 08:51:18.811 ERROR 6 --- [pool-3-thread-1] o.s.s.s.TaskUtils$LoggingErrorHandler    : Unexpected error occurred in scheduled task.

[gsunner]

The pod logs that you posted would suggest there is an authorisation issue.
Do you know if RBAC is enabled for your minikube cluster?

[paul-crease]

@gsunner thanks for the quick response. I ran minikube start --memory 4096.... I did not add RBAC to the extra configs. I will take a look in case it is somehow a default...

[paul-crease]

@gsunner - ok, I solved the problem. I removed my minikube installation and downgraded to v0.25.1 based on some research into problems with v0.26 and RBAC. I then ran minikube using K8s V1.9.4 (default K8s version for minikube v0.25.1), and it now works according to the instructions. So, it seems to be related to the most recent versions of Minikube and/or possibly K8s.