API-26059 Log durations of some sidekiq jobs

[caseywilliams]

Summary

• Since the move to EKS, sidekiq jobs should be running in less than 2 minutes, ideally within 1 minute.
• To facilitate refactoring for this requirement, this adds a new piece of sidekiq middleware which logs the duration of sidekiq jobs in the AppealsApi, VAForms, and VBADocuments APIs.
• This middlware will only be loaded if the appeals_api_sidekiq_job_duration_logging flipper flag is turned on.
  • This flipper flag (and the middleware itself) can be removed once we have an idea of how long each job generally takes

Related issue(s)

• API-26059

Testing done

• So far, I've just tested this in the rails console by running a few jobs via perform_async and then running the middleware manually.
• I put this behind a feature flag, since this manual testing may not be sufficient - I will be trying this in the lower environments before enabling it in production.

What areas of the site does it impact?

Loads middleware for sidekiq that only affects workers from the AppealsAPI module, the VAForms module, and the benefits intake (VBADocuments) module

Acceptance criteria

• I fixed|updated|added unit tests and integration tests for each feature (if applicable).
• No error nor warning in the console.
• Events are being sent to the appropriate logging solution
• Documentation has been updated (link to documentation)
• No sensitive information (i.e. PII/credentials/internal URLs/etc.) is captured in logging, hardcoded, or specs
• Feature/bug has a monitor built into Datadog or Grafana (if applicable)
• If app impacted requires authentication, did you login to a local build and verify all authenticated routes work as expected
• I added a screenshot of the developed feature

[mathisto]

Yo, this is slick, low friction, and efficient. NOICE.

[cilestin]

How are we to tell which job has which durations within datadog, if we're only grabbing the module name? Shouldn't we also grab the actual class that runs the job?

[caseywilliams]

As I was refactoring this, I started second guessing whether we should be adding to datadog at all yet - we already have datadog's sidekiq middleware set up here, and it seems like it reports very extensive information. Since none of us is able to actually see datadog yet, I was thinking it might make sense to wait to send data there until we've reviewed what's already being collected - any opinions on that?

[mathisto]

Do we know anyone with read access who could peek for us?

[cilestin]

manual datadog duration logging isn't necessary :)

[cilestin]

With the fact that we're getting automatic perform duration logging w/ DataDog - do we actually need this separate logging into rails logs?

[caseywilliams]

I'd much rather just look at datadog for this information - if we have a pathway to doing that and it's acceptable to wait to get this information until then, I'm happy to close this

[kristen-brown]

I think it was added as a stopgap since we don't have DataDog access yet.

[cilestin]

It also looks like not all of our jobs are being captured by DataDog, so let's keep this in for now until we know that all our jobs are registered.