Add instrumentation of driver_requests_* metrics

[saley89]

What this PR does / why we need it:
In version v0.50.0 of MCM new metrics were added to record driver request durations and failures. It is for the provider's to implement the logic of updating these metrics at the appropriate points in the code base based on their driver activity with the provider.

This PR is based on the machine-controller-manager-provider-azure implementation of these metrics and adds the duration of requests and records any failures during these operations:

• create machine
• delete machine
• list machines
• get machine status
• get volume ids

Which issue(s) this PR fixes:
Fixes #152

Special notes for your reviewer:

See the linked machine-controller-manager-provider-azure project and whether we are able to take their instrument.go function as is.

Release note:

Implements the driver metrics added to MCM in version `0.50.0` such that duration of calls to AWS and any failed requests are recorded:
* driver_request_duration_seconds
* driver_requests_failed_total

[gardener-robot]

@saley89 Thank you for your contribution.

[gardener-robot-ci-2]

Thank you @saley89 for your contribution. Before I can start building your PR, a member of the organization must set the required label(s) {'reviewed/ok-to-test'}. Once started, you can check the build status in the PR checks section below.

[rishabh-11]

Hey @saley89, with mcm v0.50 we also have introduced metrics for cloud provider API calls. I have raised an issue (#156) for recording these in provider-aws. Would you like to record these metrics as a part of this PR or wait for the MCM team to implement it when we pick it up?

[saley89]

Hi @rishabh-11 my team has a need for the metrics that are implemented by this change and I noticed it was relatively straight forward to add these in based on the implementation in the other provider.

If you are comfortable for this PR to go in for these metrics without the others that would be beneficial for us. The other metrics could go in as and when you have time/are able to do so.

[rishabh-11]

Can you move the instrument.go file in a separate directory called instrument inside pkg?

[saley89]

@rishabh-11 sure, that is done now.

[rishabh-11]

just a minor change. rest is good. Also did you test this? Are the metrics working as expected?

[saley89]

Regarding tests I think it can be done. Looks like the way to go about it would be to leverage the testutils package of the client_go/prometheus package we already have in this repo. Some of example of using it are in the prometheus repo.

I had a brief look this afternoon to see if we can introduce it to the existing core_test.go tests but would need more time to get something ready for the PR. In the meantime anyone can contribute to it if they have a solution.

My thinking would be ideally to get the metric, call methods like CreateMachine, DeleteMachine and see that the metric with the relevant labels has indeed been incremented. Unfortunately the Azure example hasn't tested it either for reference.

[rishabh-11]

Hey @saley89, While testing your PR I found a bug in the code related to metrics. Can you refer to gardener/machine-controller-manager-provider-azure#130 and fix this PR?

[rishabh-11]

/ping @saley89

[gardener-robot]

@saley89 ℹ️ please take some time to help rishabh-11 or redirect to someone else if you can't.

[gardener-robot]

@saley89 You need rebase this pull request with latest master branch. Please check.

[saley89]

@rishabh-11 I was able to pull in the changes needed for the updated driver API function. I needed to revendor due to go mod changes to bring in the latest prometheus golang client.

I will rebase with master to fix these conflicts.

[saley89]

The rebase has got things back in order and this is now ready for your review again. Thanks

[rishabh-11]

/lgtm

[unmarshall]

/lgtm