Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP: Rework metrics #715

Closed
wants to merge 13 commits into from
73 changes: 36 additions & 37 deletions Gopkg.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

6 changes: 0 additions & 6 deletions pkg/k8sutil/constants.go
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,4 @@ const (
// OperatorNameEnvVar is the constant for env variable OPERATOR_NAME
// wich is the name of the current operator
OperatorNameEnvVar = "OPERATOR_NAME"

// PrometheusMetricsPort defines the port which expose prometheus metrics
PrometheusMetricsPort = 60000

// PrometheusMetricsPortName define the port name used in kubernetes deployment and service
PrometheusMetricsPortName = "metrics"
)
8 changes: 4 additions & 4 deletions pkg/k8sutil/k8sutil.go
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ func GetOperatorName() (string, error) {
}

// InitOperatorService return the static service which expose operator metrics
func InitOperatorService() (*v1.Service, error) {
func InitOperatorService(port int32, portName string) (*v1.Service, error) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you say something about adding an OwnerReference to the service? Is this the service or am I getting that confused with something else?

operatorName, err := GetOperatorName()
if err != nil {
return nil, err
Expand All @@ -67,13 +67,13 @@ func InitOperatorService() (*v1.Service, error) {
Spec: v1.ServiceSpec{
Ports: []v1.ServicePort{
{
Port: PrometheusMetricsPort,
Port: port,
Protocol: v1.ProtocolTCP,
TargetPort: intstr.IntOrString{
Type: intstr.String,
StrVal: PrometheusMetricsPortName,
StrVal: portName,
},
Name: PrometheusMetricsPortName,
Name: portName,
},
},
Selector: map[string]string{"name": operatorName},
Expand Down
26 changes: 26 additions & 0 deletions pkg/metrics/constants.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
// Copyright 2018 The Operator-SDK Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package metrics

const (
// OperatorSDKPrometheusMetricsPortValue defines the port which expose prometheus metrics
OperatorSDKPrometheusMetricsPort = 60000

// OperatorSDKPrometheusMetricsPortName define the port name used in kubernetes deployment and service
OperatorSDKPrometheusMetricsPortName = "sdk-metrics"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any reason to rename the port to sdk-metrics?
I'm thinking metrics is simple and clear enough. The sdk prefix doesn't add much.

Copy link
Member Author

@lilic lilic Nov 8, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, the reason being port names need to be unique in the deployment files and metrics is a very generic name, so if a user defines a new one the metrics port would not make sense any more. Plus in the next few PRs when the controller-runtime metrics gets into a release we want to add a port to the deployment and service resources and 8080 where they expose their metrics and we want to name it differently. Hope that makes sense?

Note: This just means the name of the port not the actual path is called sdk-metrics

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So we are going to serve two metrics endpoints one for the SDK and one for the controller runtime metrics?

Could we just register the controller-runtime metrics as apart of the sdk metrics and then serve via a single endpoint?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes for now, well currently there are no metrics in controller-runtime release so there are no metrics there yet.
But see #715 (comment)

My plan is to adjust controller-runtime to not use global registry and we can then do a follow up PR to reuse that registry once that is merged.

Hope that clears things up?


// PrometheusMetricsPath
PrometheusMetricsPath = "/metrics"
)
74 changes: 74 additions & 0 deletions pkg/metrics/metrics.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
// Copyright 2018 The Operator-SDK Authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package metrics

import (
"fmt"
"net/http"
"os"
"strconv"

"github.com/operator-framework/operator-sdk/pkg/k8sutil"

"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
v1 "k8s.io/api/core/v1"
)

// Setup function registers go and process metrics, serves metrics and exposes Prometheus metrics port.
// It also generates and returns a Kubernetes Service to expose the metrics port.
func Setup() (*v1.Service, error) {
reg := NewPrometheusRegistery()
mux := http.NewServeMux()
HandlerFuncs(mux, reg)
go func() {
err := http.ListenAndServe(":"+strconv.Itoa(OperatorSDKPrometheusMetricsPort), mux)
if err != nil {
fmt.Printf("Serving metrics failed: %v", err)
}
}()

service, err := k8sutil.InitOperatorService(int32(OperatorSDKPrometheusMetricsPort), OperatorSDKPrometheusMetricsPortName)
if err != nil {
return nil, fmt.Errorf("failed to initialize service object for operator metrics: %v", err)
}
return service, nil
}

// NewPrometheusRegistery returns a newly created prometheus registry.
// It also registers go collector and process collector metrics.
func NewPrometheusRegistery() *prometheus.Registry {
r := prometheus.NewRegistry()
r.MustRegister(
prometheus.NewGoCollector(),
prometheus.NewProcessCollector(prometheus.ProcessCollectorOpts{
PidFn: func() (int, error) { return os.Getpid(), nil },
Namespace: "",
ReportErrors: false,
}),
)
return r
}

// HandlerFuncs registers the handles the /healthz and /metrics, to be able to serve the prometheus registry.
func HandlerFuncs(mux *http.ServeMux, reg *prometheus.Registry) {
mux.HandleFunc("/healthz", handlerHealthz)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to register the healthz endpoint? Right now handlerHealthz() would just ensure that the operator is always healthy. Which is the default even if we don't set this.

Liveness(healthz) and Readiness(readyz) endpoints/probes should be based on the operator logic or possibly related to leader election.

Does prometheus need to query the /healthz endpoint before scraping the actual metrics?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Liveness(healthz) and Readiness(readyz) endpoints/probes should be based on the operator logic or possibly related to leader election.

Good point will open an issue about the healtz endpoint, if we don't have one already.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we rename this to RegisterHandlerFuncs for added clarity? HandlerFuncs made me think it would return handler functions.

mux.Handle(PrometheusMetricsPath, promhttp.HandlerFor(reg, promhttp.HandlerOpts{ErrorHandling: promhttp.ContinueOnError}))
}

func handlerHealthz(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("ok"))
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.

package sdk
package metrics

import (
monitoringv1 "github.com/coreos/prometheus-operator/pkg/client/monitoring/v1"
Expand Down
Loading