db-dumper fails to refresh expired access tokens

[gberche-orange]

Observed db-dumper to fail with the following user-facing symptoms

cf create-dump pad-mysql
Error: Server error, status code: 502, error code: 10001, message: Service broker error: Access token denied.
Tip: use --verbose to see what's going wrong with cf core command

The db-dumper broker logs seems to confirm this hypothesis

  2018-10-24T17:27:14.36+0200 [RTR/1] OUT db-dumper.redacted-domain- [2018-10-24T15:27:14.278+0000] "PATCH /v2/service_instances/ca6bfcb2-c84b-43d9-83b9-724ea980d1eb?accepts_incomplete=true HTTP/1.1" 500 1757 38 "-" "HTTPClient/1.0 (2.8.2.4, ruby 2.4.2 (2017-09-14))" "192.168.99.31:40130" "192.168.35.92:61028" x_forwarded_for:"192.168.99.31" x_forwarded_proto:"https" vcap_request_id:"a23f3c39-6a0f-4d3f-6cd5-dc86828b6184" response_time:0.090498547 app_id:"aff7eb2a-0107-4082-a257-32a56477010f" app_index:"1" x_b3_traceid:"b469f0a3abfb0565" x_b3_spanid:"b469f0a3abfb0565" x_b3_parentspanid:"-"
   2018-10-24T17:27:14.36+0200 [RTR/1] OUT 
   2018-10-24T17:27:14.36+0200 [APP/PROC/WEB/1] OUT 2018-10-24 15:27:14.367  WARN 14 --- [nio-8080-exec-2] .m.m.a.ExceptionHandlerExceptionResolver : Resolved exception caused by Handler execution: error="access_denied", error_description="Access token denied."
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT 2018-10-24 15:28:12.956  WARN 7 --- [nio-8080-exec-9] o.c.c.s.controller.BaseController        : Exception
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT org.springframework.security.oauth2.client.resource.OAuth2AccessDeniedException: Access token denied.
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.security.oauth2.client.token.OAuth2AccessTokenSupport.retrieveToken(OAuth2AccessTokenSupport.java:142)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.security.oauth2.client.token.grant.password.ResourceOwnerPasswordAccessTokenProvider.refreshAccessToken(ResourceOwnerPasswordAccessTokenProvider.java:40)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.cloudfoundry.client.lib.oauth2.OauthClient.refreshToken(OauthClient.java:126)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.cloudfoundry.client.lib.oauth2.OauthClient.getToken(OauthClient.java:88)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.cloudfoundry.client.lib.oauth2.OauthClient.getAuthorizationHeader(OauthClient.java:97)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.cloudfoundry.client.lib.rest.CloudControllerClientImpl$CloudFoundryClientHttpRequestFactory.createRequest(CloudControllerClientImpl.java:2789)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.http.client.support.HttpAccessor.createRequest(HttpAccessor.java:77)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:615)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.cloudfoundry.client.lib.rest.LoggingRestTemplate.doExecute(LoggingRestTemplate.java:62)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:580)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:380)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.cloudfoundry.client.lib.rest.CloudControllerClientImpl.createServiceKey(CloudControllerClientImpl.java:405)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.cloudfoundry.client.lib.rest.CloudControllerClientImpl.createServiceKey(CloudControllerClientImpl.java:382)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.cloudfoundry.client.lib.CloudFoundryClient.createServiceKey(CloudFoundryClient.java:296)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at com.orange.clara.cloud.servicedbdumper.service.servicekey.CloudFoundryServiceKeyManager.createServiceKey(CloudFoundryServiceKeyManager.java:49)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at com.orange.clara.cloud.servicedbdumper.dbdumper.DatabaseRefManager.updateDatabaseRef(DatabaseRefManager.java:76)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at com.orange.clara.cloud.servicedbdumper.service.DbDumperServiceInstanceService.createDumpFromdbDumperServiceInstance(DbDumperServiceInstanceService.java:219)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at com.orange.clara.cloud.servicedbdumper.service.DbDumperServiceInstanceService.updateServiceInstance(DbDumperServiceInstanceService.java:200)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at com.orange.clara.cloud.servicedbdumper.service.DbDumperServiceInstanceService$$FastClassBySpringCGLIB$$f90a4690.invoke(<generated>)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:651)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at com.orange.clara.cloud.servicedbdumper.service.DbDumperServiceInstanceService$$EnhancerBySpringCGLIB$$f56e2589.updateServiceInstance(<generated>)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.cloudfoundry.community.servicebroker.controller.ServiceInstanceController.updateServiceInstance(ServiceInstanceController.java:113)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at java.lang.reflect.Method.invoke(Method.java:498)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:114)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:963)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:897)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)
   2018-10-24T17:28:12.95+0200 [APP/PROC/WEB/0] OUT         at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)

Root cause seems to be in the deprecated cf-java client library
https://github.com/orange-cloudfoundry/db-dumper-service/blob/master/pom.xml#L37
which includes token refresh bugs