Token acquisition failure - develop approach and handling

[cd-rite]

Current behavior:
preflight request failure - leaves a log entry, does not retry on failure, watcher stops.
subsequent token failure has no effect on other processing (but will show lots of errors in the log)

subsequent token acquisition failure modes:

• Identity provider unavailable (timeout):
  • throw error, stop processing, retry
• client credentials no longer working
  • throw error, stop processing, retry

retry behavior:

• scan mode: suspend scanning and file parsing while retrying
• event mode: continue capturing file add/remove events, suspend file parsing
• retry every 60 seconds
  • Log initial failure
  • Log every (hour? day? thereafter)
  • Log success
  • log: every attempt at debug level
  • continue trying indefinitely