Make it possible to support runtime metrics on a seperate path

[hsyed-dojo]

@SuperQ

This PR introduces a new flag to make it possible to serve the go runtime metrics on a seperate endpoint. The way we are orchestrating the deployment of the exporter (multiple k8s deployments), this flag will effectively halve our stackdriver API calls.

Manual testing done:

• With and without flag.
• Launch process locally with one of our GCP projects.
• Verify endpoint(s) contain what they should.

[SuperQ]

Interesting idea. What specifically about your deployments will reduce the number of calls?

[hsyed-dojo]

Interesting idea. What specifically about your deployments will reduce the number of calls?

@SuperQ we intend to orchestrate our scrapers to have a seperate scrape job for the runtime metrics -- possibly with a tighter interval.

As we have to orchestrate with multiple deployments due to quantity of metrics we need to ingest (divide and conquer) , this approach has the additional benefit of allowing us to model multiple stackdriver_exporter deployments ergonomically in our helm meta-chart.

[SuperQ]

I wonder if we should have this flag do the opposite. Have web.telemetry-path be where the runtime metrics live. But move allow moving the stackdriver metrics to a new path via a separate registry. For example --web.stackdriver-telemetry-path=/stackdriver.

I'm suggesting this because we're likely to move the web.telemetry-path feature to the exporter-toolkit. It would be easier to move the exported metrics, rather than the standard telemetry metrics. Similar to how some exporters have /scrape endpoints for the multi-target exporter pattern.

[hsyed-dojo]

@SuperQ yes changing the polarity of the flag would be the same for us. I'll update the PR.

---

We explored using query params to provide the filters and this might allow us to have a single deployment. The stackdriver API call metrics are specific to the registry created for the exposition and they reset on each call with the approach (IIRC). I think we would want those metrics to probably be cumulative/global (or with a use-case label --e.g., pubsub) and available with the runtime metrics. I hope this will be possible in the future.

[SuperQ]

Nice, thanks.