boskos: merge Prometheus resource metrics into boskos core

[ixdy]

What should be cleaned up or changed:
On k8s prow, the Boskos service runs inside the Prow builds cluster.
Back in 2017, we wanted to collect metrics on Boskos resources to show in Velodrome, so it was necessary to expose these metrics on an external IP, since the Velodrome collectors did not run inside the builds cluster.

To accomplish this, #3971 added the boskos-metrics component, which both proxied the /metric API on /, as well as exposing Prometheus-style metrics on /prometheus.

Since then, we've switched to serving Prometheus-style metrics on port 9090, and since #15356 have not even exposed the /metric proxy, but nobody has seemed to complain (so maybe it's not used?). We're now collecting metrics using the Prow monitoring stack, but since that runs in a separate cluster, we still need to expose the metrics via a public IP.

At the same time, the main Boskos deployment is also serving its own Prometheus metrics on port 9090. In #15990, I'm adding a new Service that exposes these metrics. Rather than running a separate boskos-metrics component, we could just move these resource metrics into the Prometheus metrics in boskos core, simplifying deployments slightly.

(I also intend to serve metrics on janitors. We might want to configure some sort of cluster-local aggregator for these, since they'll also be private to the cluster.)

/area boskos

cc @stevekuznetsov @krzyzacy is there something obvious I'm missing here?

[ixdy]

regarding metric "aggregation" - maybe federation is relevant?

[stevekuznetsov]

You could do that, yes, or configure for Prometheus in the build cluster to scrape everything there and remote_write to push out to something public./

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[ixdy]

I think this is actually done, though I haven't added metrics to the janitors yet.

/close

[k8s-ci-robot]

@ixdy: Closing this issue.

In response to this:

I think this is actually done, though I haven't added metrics to the janitors yet.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.