Expose and collect Boskos HTTP metrics

[ixdy]

The main Boskos service exposes useful Prometheus metrics about its HTTP API:

test-infra/boskos/boskos.go

Lines 60 to 62 in cd53865

	httpRequestDuration = metrics.HttpRequestDuration("boskos", 0.005, 360)
	httpResponseSize = metrics.HttpResponseSize("boskos", 128, 65536)
	traceHandler = metrics.TraceHandler(simplifier, httpRequestDuration, httpResponseSize)

We should collect these.

We'll need to create a new static IP to bind to the load balancer. Assuming we think this is a good idea, I'll do that and then update this PR.

I'm still figuring how to update the various grafana configurations to graph this data, but we may as well start by collecting it.

/area boskos
/area monitoring
/hold

/assign @cjwagner @stevekuznetsov
cc @Katharine as FYI

[k8s-ci-robot]

@ixdy: The label(s) area/monitoring cannot be applied, because the repository doesn't have them

In response to this:

The main Boskos service exposes useful Prometheus metrics about its HTTP API:

test-infra/boskos/boskos.go

Lines 60 to 62 in cd53865

	httpRequestDuration = metrics.HttpRequestDuration("boskos", 0.005, 360)
	httpResponseSize = metrics.HttpResponseSize("boskos", 128, 65536)
	traceHandler = metrics.TraceHandler(simplifier, httpRequestDuration, httpResponseSize)

We should collect these.

We'll need to create a new static IP to bind to the load balancer. Assuming we think this is a good idea, I'll do that and then update this PR.

I'm still figuring how to update the various grafana configurations to graph this data, but we may as well start by collecting it.

/area boskos
/area monitoring
/hold

/assign @cjwagner @stevekuznetsov
cc @Katharine as FYI

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ixdy]

This is intentional: the boskos-metrics service proxies the main /metric API on port 8080, and this was masked in #15356. Configuring the scraper to look at port 9090 (also included here) is a better fix.

[cjwagner]

Does port 8080 really need to be externally exposed? I don't know what that endpoint is for or what uses it so I'll leave that to your discretion.

[ixdy]

It used to be exposed, prior to #15356. it basically just proxies the /metric API:

test-infra/boskos/metrics/metrics.go

Lines 131 to 170 in 0a8437a

	// handleMetric: Handler for /
	// Method: GET
	func handleMetric(boskos *client.Client) http.HandlerFunc {
	return func(res http.ResponseWriter, req *http.Request) {
	log := logrus.WithField("handler", "handleMetric")
	log.Infof("From %v", req.RemoteAddr)
	if req.Method != "GET" {
	log.Warningf("[BadRequest]method %v, expect GET", req.Method)
	http.Error(res, "only accepts GET request", http.StatusMethodNotAllowed)
	return
	}
	rtype := req.URL.Query().Get("type")
	if rtype == "" {
	msg := "type must be set in the request."
	log.Warning(msg)
	http.Error(res, msg, http.StatusBadRequest)
	return
	}
	log.Infof("Request for metric %v", rtype)
	metric, err := boskos.Metric(rtype)
	if err != nil {
	log.WithError(err).Errorf("Fail to get metic for %v", rtype)
	http.Error(res, err.Error(), http.StatusNotFound)
	return
	}
	metricJSON, err := json.Marshal(metric)
	if err != nil {
	log.WithError(err).Errorf("json.Marshal failed: %v", metricJSON)
	http.Error(res, err.Error(), http.StatusInternalServerError)
	return
	}
	log.Infof("Metric query for %v: %v", rtype, string(metricJSON))
	fmt.Fprint(res, string(metricJSON))
	}
	}

Nobody has complained that this is missing, so maybe we don't need it.

[ixdy]

(removed)

[ixdy]

(related to #15557 - it'd be good to know how the k8s boskos is doing in comparison.)

[cjwagner]

/hold

[cjwagner]

Does port 8080 really need to be externally exposed? I don't know what that endpoint is for or what uses it so I'll leave that to your discretion.

[cjwagner]

Will this dummy value cause problems? (i.e. do we need to emit the field entirely until we have a valid value?)

[ixdy]

right, I don't intend to submit this until we've reserved an IP address. I can do that, assuming there's consensus that this is a good idea.

[cjwagner]

We should probably wait to add this until we have a valid value to avoid failing scrapes.

[cjwagner]

FYI @Katharine

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cjwagner, ixdy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• prow/cluster/OWNERS [cjwagner,ixdy]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[ixdy]

/hold cancel
👀