Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pkg/{sdk,generator} Expose Prometheus metrics port #323

Merged
merged 1 commit into from
Jul 6, 2018

Conversation

etiennecoutaud
Copy link
Contributor

related to #222 discuss

This PR is similar to #241

\cc @hasbro17

@etiennecoutaud
Copy link
Contributor Author

etiennecoutaud commented Jun 25, 2018

CI fail because during memcached-operator generation, latest sdk code from github is vendored.
CI will always fail if we submit a change in pkg/sdk

@theishshah In travis.yml after cd memcached-operator we could manually vendor current code with : ln -sf ${TRAVIS_BUILD_DIR}/pkg vendor/github.com/operator-framework/operator-sdk/pkg
WDYT ?

@theishshah
Copy link
Member

The travis.yml has been updated on master, so you can pull down that version and re-trigger Travis and it should work as intended.

@etiennecoutaud
Copy link
Contributor Author

etiennecoutaud commented Jun 26, 2018

My bad, it's still failing

After investigation :

  • ln -sf ${TRAVIS_BUILD_DIR}/pkg vendor/github.com/operator-framework/operator-sdk/pkg is created a simlink under pkg so we have vendor/github.com/operator-framework/operator-sdk/pkg/pkg
  • We cannot overwrite directory using ln
  • We can :
    1. rm -r vendor/github.com/operator-framework/operator-sdk/pkg
    2. cp -r ${TRAVIS_BUILD_DIR}/pkg vendor/github.com/operator-framework/operator-sdk/pkg

It should finally work 😄

@theishshah
Copy link
Member

Hey, we noticed that last night as well, PR #329 should address this issue

@etiennecoutaud
Copy link
Contributor Author

Thanks @theishshah

@hasbro17
Copy link
Contributor

hasbro17 commented Jul 4, 2018

@etiennecoutaud Sorry for the delay. I've actually been trying to test your PR for a day now but I'm running into an issue.
The problem is that the prometheus dependencies are not being vendored for the generated project and so I can't build it.

When I do opeartor-sdk new app-operator --api-version=app.example.com/v1alpha1 --kind=App it will vendor the upstream operator-sdk master branch.
Then to test the changes in your PR I replace the vendored sdk pkg with the one from your PR, like we do it in the travis script

rm -r vendor/github.com/operator-framework/operator-sdk/pkg
cp -r ${PATH_TO_LOCAL_SDK}/pkg vendor/github.com/operator-framework/operator-sdk/pkg

However now the issue is that the updated vendored SDK depends on imports like github.com/prometheus/client_golang/prometheus/promhttp which were not vendored in app-opeartor/vendor on the first dep ensure run. And so I can't build the operator due to the missing imports.

$ operator-sdk build quay.io/coreos/operator-sdk-dev:app-operator
Error: failed to build: (building app-operator...
vendor/github.com/operator-framework/operator-sdk/pkg/sdk/metrics.go:8:2: cannot find package "github.com/prometheus/client_golang/prometheus/promhttp" in any of:
	/Users/haseeb/work/go-space/src/github.com/example-inc/app-operator/vendor/github.com/prometheus/client_golang/prometheus/promhttp (vendor tree)
	/usr/local/go/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOROOT)
	/Users/haseeb/work/go-space/src/github.com/prometheus/client_golang/prometheus/promhttp (from $GOPATH)
)

Since app-operator does not directly use that import you can't force the prometheus dependencies to be vendored by adding it as a constraint in app-operator/Gopkg.toml either.
This is one of those issues that get's fixed if your change gets merged into master since then everything is vendored correctly from the start but before that it's really hard to test.

So I was wondering if you had managed to do a local build and test of this with the README example?
And given my issue it's surprising is how the CI is able to build this successfully.

@etiennecoutaud
Copy link
Contributor Author

etiennecoutaud commented Jul 4, 2018

@hasbro17 no problem for the delay.

I reproduced your issue, It works and I'm able to build on my side because I have github.com/prometheus/client_golang in my GOPATH from older works.

ls ~/go/src/github.com/prometheus/
client_golang client_model  common        procfs
operator-sdk build quay.io/coreos/operator-sdk-dev:app-operator
building app-operator...

building container quay.io/coreos/operator-sdk-dev:app-operator...
Sending build context to Docker daemon  186.1MB
Step 1/4 : FROM alpine:3.6
 ---> 77144d8c6bdc
Step 2/4 : RUN adduser -D app-operator
 ---> Using cache
 ---> 2c9b1acc3402
Step 3/4 : USER app-operator
 ---> Using cache
 ---> b0a51d072c95
Step 4/4 : ADD tmp/_output/bin/app-operator /usr/local/bin/app-operator
 ---> d8e320875381
Successfully built d8e320875381
Successfully tagged quay.io/coreos/operator-sdk-dev:app-operator

Create deploy/operator.yaml
➜ (⎈ |minikube:default) ▶ kubectl create -f deploy/operator.yaml
deployment.apps "app-operator" created
➜ (⎈ |minikube:default)  ▶ kubectl get po
NAME                            READY     STATUS    RESTARTS   AGE
app-operator-5dd49b487d-2x55r   1/1       Running   0          5s
➜ (⎈ |minikube:default) ▶ kubectl logs app-operator-5dd49b487d-2x55r
time="2018-07-04T08:26:38Z" level=info msg="Go Version: go1.10"
time="2018-07-04T08:26:38Z" level=info msg="Go OS/Arch: linux/amd64"
time="2018-07-04T08:26:38Z" level=info msg="operator-sdk Version: 0.0.5+git"
time="2018-07-04T08:26:38Z" level=info msg="Metrics service app-operator created"
time="2018-07-04T08:26:38Z" level=info msg="Watching app.example.com/v1alpha1, App, default, 5"
➜ (⎈ |minikube:default) ▶ kubectl get svc
NAME           TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
app-operator   ClusterIP   10.106.156.196   <none>        60000/TCP   17s
kubernetes     ClusterIP   10.96.0.1        <none>        443/TCP     2m

Regarding this, why CI is green 🤔 ?

@@ -276,7 +282,7 @@ func TestGenDeploy(t *testing.T) {
}

buf = &bytes.Buffer{}
if err := renderFile(buf, operatorTmplName, operatorYamlTmpl, tmplData{ProjectName: appProjectName, Image: appImage}); err != nil {
if err := renderFile(buf, operatorTmplName, operatorYamlTmpl, tmplData{ProjectName: appProjectName, Image: appImage, MetricsPort: k8sutil.PrometheusMetricsPort, MetricsPortName: k8sutil.PrometheusMetricsPortName, OperatorNameEnv: k8sutil.OperatorNameEnvVar}); err != nil {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Define the tmplData before this since it's too long:

	td := tmplData{
		ProjectName:     appProjectName,
		Image:           appImage,
		MetricsPort:     k8sutil.PrometheusMetricsPort,
		MetricsPortName: k8sutil.PrometheusMetricsPortName,
		OperatorNameEnv: k8sutil.OperatorNameEnvVar,
	}
	if err := renderFile(buf, operatorTmplName, operatorYamlTmpl, td); err != nil {
		t.Error(err)
	}

@hasbro17
Copy link
Contributor

hasbro17 commented Jul 5, 2018

@etiennecoutaud I've just tested this out by pulling the dependencies locally and it works. We can figure out what's wrong with the CI later.

I'm going to push this change up as a separate branch on the repo and so I can test whether or not operator-sdk new will correctly vendor the prometheus dependencies for the generated operator when it's merged into the master branch. Once that works this should be good to go.

@etiennecoutaud
Copy link
Contributor Author

@hasbro17 okay !
Do not hesitate to ask if you need help.

"strconv"

k8sutil "github.com/operator-framework/operator-sdk/pkg/util/k8sutil"
"github.com/prometheus/client_golang/prometheus/promhttp"
Copy link
Contributor

@secat secat Jul 6, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a suggestion, the prometheus OpenCensus exporter library could be used.

Here is a code snippet to illustrate the usage:

import (
        "github.com/sirupsen/logrus"
        "github.com/prometheus/client_golang/prometheus"
        ocprometheus "go.opencensus.io/exporter/prometheus"
	ocview "go.opencensus.io/stats/view"
)

metricsRegistry := prometheus.NewRegistry()
metricsRegistry.MustRegister(prometheus.NewGoCollector())

prometheusExporter, err := ocprometheus.NewExporter(ocprometheus.Options{
	Namespace: "operator_sdk",
	Registry:  metricsRegistry,
})
if err != nil {
        logrus.Fatalf("Failed to create prometheus exporter: %v", err)
	return
}

ocview.RegisterExporter(prometheusExporter)

http.Handle("/"+k8sutil.PrometheusMetricsPortName, prometheusExporter)

IMHO, this project should integrate and use the OpenCensus library.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@secat I'm not too familiar with the OpenCensus library so I can't comment on the need for it yet. However this PR is only meant to be the starting point for exposing the Prometheus metrics so I would prefer to keep it simple right now and iterate on more changes after this.
It'll be helpful if you can create an issue so we can discuss the use cases for the OpenCensus library in the SDK.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@hasbro17 I have created the issue #343

@hasbro17
Copy link
Contributor

hasbro17 commented Jul 6, 2018

LGTM.
Thanks for working through all the iterations.

@hasbro17 hasbro17 merged commit 3c5667f into operator-framework:master Jul 6, 2018
m1kola pushed a commit to m1kola/operator-sdk that referenced this pull request Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants