When getting OAuth client from redis, while redis is not responding, the oauth client is wrongly marked as deleted

[letzya]

Branch/Environment/Version

• Master, latest release 2.9.2
• Environment: On-prem

Describe the bug

Reproduction steps
Steps to reproduce the behavior: (haven't tried yet)

1. Add oauth api.
2. Use it with an access token and make sure it successfully returns response
3. Stope redis and make another call to the api
4. Get error
5. Start redis again and make sure the api successfully returns response without restarting the gateway

Actual behavior
The flow that discovered the bug was:

1. The gateway made a call to redis to get OAuth client details
2. Redis was having connectivity issues and as such returned
   Error trying to set value: read tcp xx.xx.xx.xx:454354 ... Attempted access for deleted OAuth client.
3. Following this the gateway kept on notifying for a few oauth clients that they are deleted:
   Attempted access for deleted OAuth client.
4. Restarting the gateways sorted the issue and calls of the oauth clients were respected.

My conclusion was that that the gateway is wrongfully marking the oauth client as deleted in its in-memory and as such only a restart would allow this oauth client to be used again.

Expected behavior
If there's an error when getting an oauth client either not found or a network error the gateway will temporarily mark the oauth client as deleted in its in-memory cache for a certain interval (1 sec)

Logs (debug mode or log file):

"level=error msg=\"Error trying to set value: read tcp 10.32.xxx.xxx:3xxx: i/o timeout\""
--
" level=error msg=\"Could not send notification: read tcp  i/o timeout\""
“level=error msg=\"Error trying to get TTL: read tcp : i/o timeout\""
"level=warning msg=\"Attempted access for deleted OAuth client.\" api_id=fdgddf api_name=quotation-external-orchestration key=\"****\" mw=Oauth2KeyExists oauthClientID={client-id} org_id={org-id} origin=fdsfdg path=\"/partner-quotation/\""
"level=error msg=\"Failure retreiving client ID key \\\"oauth-clientid.{client-id}\\\": key not found\""

Configuration (tyk config file):
Attach tyk configuration file

Additional context
Commit: 6647450
set interval so the OAuth client won't be marked as deleted till next restart

[christtyk]

@buger please let me know if the fix is ok, and how we can get it into testing for a 2.8.7 patch

[ilijabojanovic]

Verified on release-2.8 and release-2.9 branch