Azure metrics exporter

[nickbp]

Look into implementing an Azure exporter, similar to the current AWS/GCP ones.

Need to look into credential requirements, the exporter has different instructions based on whether "managed identities" are being used. The instructions seem to imply that metrics need to be fetched from within an Azure-hosted VM or "app"? Hopefully there's a way to use credentials to query into metrics from the outside... See also: https://github.com/RobustPerception/azure_metrics_exporter#azure-account-requirements

Exporter itself would be:

• https://github.com/RobustPerception/azure_metrics_exporter
• https://hub.docker.com/r/robustperception/azure_metrics_exporter/tags - only has a single latest tag, may want to pin to the exact SHA just in case.

[nickbp]

From reading the azure exporter readme linked above again, it sounds like getting a credential involves creating an "App" and give the App permission to access metrics. So in principle we should be good there.

See also:

• Azure docs: https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#create-a-new-application-secret
• Instructions for another monitoring service that appears to support extracting Azure metrics (and faces the same need for credentials to get in): https://www.appdynamics.com/community/exchange/extension/azure-monitoring-extension/

[nickbp]

FYI: There's also https://github.com/sylr/prometheus-azure-exporter which is recently active but only has one (human) contributor, whereas the RobustPerception one linked earlier has been less active recently but has had several contributors over the years. So it's not totally cut and dry but I think it makes sense to try the RobustPerception one first and see how it goes.

[nickbp]

One catch with RobustPerception exporter is that it wants Azure credentials directly in the exporter config. There isn't a route for supplying them via a separate secret channel: https://github.com/RobustPerception/azure_metrics_exporter/blob/40606b2ae091e57e08376c0214427e7ea8f71bdc/config/config.go#L17

A workaround for this could be to inject credentials into the yaml configuration when starting the exporter, via e.g. an initContainer. Not a huge deal but still not confidence inspiring.

I checked the sylr exporter and it does have built-in env secrets support so it may end up being the better route after all: https://github.com/sylr/prometheus-azure-exporter/blob/master/pkg/config/config.go#L46

[nickbp]

I'm not sure if this is the case, but from looking at the sylr exporter code a bit more, it may be limited to a predefined list of metrics, as opposed to letting the end user configure the metrics they want: https://github.com/sylr/prometheus-azure-exporter/blob/master/pkg/metrics/batch.go#L19 . The main repo README meanwhile has a list of about 35 metrics which also gives the impression that these are the only metrics supported by the sylr exporter - and that they were picked because the author needs them for a particular work project.

So now I'm leaning back towards the RobustPerception exporter despite the minor credential config hoops required, since it at least fits our model of having the user pick the metrics they want (and supporting more of them than the ones that the author exporter wants).

[nickbp]

Ended up creating a quick upstream patch to just support specifying credentials via envvars. Felt like it was overall going to be less complicated than trying to hack the yaml config directly, and they have a history of accepting small fixes like this: RobustPerception/azure_metrics_exporter#101

[nickbp]

Pushed an image containing a build of the above PR to opstrace/azure_metrics_exporter:4f85a01 on docker hub and made it public.

[nickbp]

I've gotten an initial implementation put together but need to test it. After a LOT of fiddling around in the Azure web UI I was finally able to set up permissions for a service account. When in "Role Assignments" you MUST attempt to text search for the app name or else it won't show up at all. Took me a long time to figure out this Azure UI issue.

[nickbp]

I've been able to get metrics out of Azure successfully. I just created an empty "Storage Account" against my personal freebie/PAYG azure account, and checked that the metrics exporter was able to fetch metrics (a bunch of zeroes since the storage is empty/idle - but still)

As of my latest updates, the config api/controller are also able to deploy the azure exporter and the resulting deployment was able to serve metrics into prometheus/cortex. So the exporter implementation itself seems to work, and I think I can merge what I've got once I've cleaned up the commits a bit.

Before marking the issue as done, I will also implement a test for the azure exporter in test_exporters.ts. One catch is that unlike the other exporters, the azure exporter will refuse to run if the credentials are invalid, so I'm thinking the way to test it is to just submit the config then check that the resulting exporter deployment logs an expected "unknown tenant" error message.

[nickbp]

Tests are in, so the implementation part is complete for now. #574 separately covers adding an Azure section to the exporters user guide.