⚠ Introduce Metrics Options struct & secure metrics serving #2407

sbueringer · 2023-07-15T13:17:08Z

This PR introduces metrics options on the manager options. Via the new metrics options it will now be possible to serve metrics via tls and to use authentication and authorization.

Metrics.FilterProvider allows to provide a metrics filter.
- filters.WithAuthenticationAndAuthorization is a builtin filter that implements authentication and authorization
  - Please note: consumers of controller-runtime will only get the dependency to k8s.io/apiserver if they actually import/use this filter. Here is an example repo to show that the go.mod dependencies don't change with this PR except if the filter is configured (https://github.com/sbueringer/controller-runtime-tests/tree/master/metrics-auth)
Metrics.SecureServing enables serving metrics on https
- This is necessary for the above filter because tokens for authentication should not be sent via http
Metrics.ExtraHandlers makes it possible to specify the extra handlers directly on the options. manager.AddMetricsExtraHandler has been removed.

Fixes #2073

sbueringer · 2023-07-15T13:18:02Z

@alvaroaleman @vincepri
I think this should be ready for a first round of review & discussion.

After that I'll add more documentation and add / fix unit tests.

pkg/manager/internal.go

pkg/manager/manager.go

pkg/manager/internal.go

sbueringer · 2023-07-17T20:28:02Z

@alvaroaleman Thx for the review. Changed the defaulting behavior and added test coverage

sbueringer · 2023-07-18T19:20:43Z

@vincepri PTAL when you have some time :)

sbueringer · 2023-07-22T14:16:30Z

Refactored the implementation to avoid introducing a dependency to k8s.io/apiserver for folks who are not using the authentication/authorization

pkg/manager/internal.go

pkg/metrics/filters/filters.go

pkg/metrics/listener.go

pkg/manager/manager.go

- Bumped golang to 1.22: - Update go.mod - Update Dockerfiles - Bumped k8s.io/* modules to 0.30.3 and OpenShift API to 20240812094746-86145edb40cf. - Bumped controller-runtime to 0.18.5: - Manager's `Port` option removed, now using dedicated webhook server field (kubernetes-sigs/controller-runtime#2422). - Manager's `MetricsBindAddress` option removed, now using dedicated metrics server field (kubernetes-sigs/controller-runtime#2407). - Cache's `Namespaces` option replaced by `DefaultNamespaces` (kubernetes-sigs/controller-runtime#2421). - Regenerated CRD and bundle manifests using `make bundle` command. - Bumped `kustomize` to v5 to fix a conflict caused by k8s.io bumps: - `kyaml` unable to use the bumped `github.com/google/gnostic-models/openapiv2` package.

- Bumped golang to 1.22: - Update go.mod - Update Dockerfiles - Bumped k8s.io/* modules to 0.30.3 and OpenShift API to 20240812094746-86145edb40cf. - Bumped controller-runtime to 0.18.5: - Manager's `Port` option removed, now using dedicated webhook server field (kubernetes-sigs/controller-runtime#2422). - Manager's `MetricsBindAddress` option removed, now using dedicated metrics server field (kubernetes-sigs/controller-runtime#2407). - Cache's `Namespaces` option replaced by `DefaultNamespaces` (kubernetes-sigs/controller-runtime#2421). - Regenerated CRD and bundle manifests using `make bundle` command. - Bumped `kustomize` to v5 to fix a conflict caused by k8s.io bumps: - `kyaml` unable to use the bumped `github.com/google/gnostic-models/openapiv2` package. - Removed `TypeMeta` from expected deployment object when it's compared to structured one retrieved from fake client (kubernetes-sigs/controller-runtime#2633).

- Bumped golang to 1.22: - Update go.mod - Update Dockerfiles - Bumped k8s.io/* modules to 0.30.3 and OpenShift API to 20240812094746-86145edb40cf. - Bumped controller-runtime to 0.18.5: - Manager's `Port` option removed, now using dedicated webhook server field (kubernetes-sigs/controller-runtime#2422). - Manager's `MetricsBindAddress` option removed, now using dedicated metrics server field (kubernetes-sigs/controller-runtime#2407). - Cache's `Namespaces` option replaced by `DefaultNamespaces` (kubernetes-sigs/controller-runtime#2421). - Bumped aws-load-balancer-controller to f39ae43121c3 to use latest CRD scheme in e2e tests. - Regenerated CRD and bundle manifests using `make bundle` command. - Bumped `kustomize` to v5 to fix a conflict caused by k8s.io bumps: - `kyaml` unable to use the bumped `github.com/google/gnostic-models/openapiv2` package. - Removed `TypeMeta` from expected deployment object when it's compared to structured one retrieved from fake client (kubernetes-sigs/controller-runtime#2633).

- Bumped golang to 1.22: - Update go.mod - Update Dockerfiles - Bumped k8s.io/* modules to 0.30.3 and OpenShift API to 20240812094746-86145edb40cf. - Bumped controller-runtime to 0.18.5: - Manager's `Port` option removed, now using dedicated webhook server field (kubernetes-sigs/controller-runtime#2422). - Manager's `MetricsBindAddress` option removed, now using dedicated metrics server field (kubernetes-sigs/controller-runtime#2407). - Cache's `Namespaces` option replaced by `DefaultNamespaces` (kubernetes-sigs/controller-runtime#2421). - Bumped aws-load-balancer-controller to f39ae43121c3 to use latest CRD scheme in e2e tests. - Regenerated CRD and bundle manifests using `make bundle` command. - Bumped `kustomize` to v5 to fix a conflict caused by k8s.io bumps: - `kyaml` unable to use the bumped `github.com/google/gnostic-models/openapiv2` package. - Removed `TypeMeta` from expected deployment object when it's compared to structured one retrieved from fake client (kubernetes-sigs/controller-runtime#2633). - Migrated infrastructure CRD retrieval to the new package `zz_generated.crd-manifests`: - Updated infrastructure CRD for unit tests using `make update-vendored-crds` command.

- Bumped golang to 1.22: - Update go.mod - Update Dockerfiles - Bumped k8s.io/* modules to 0.30.3 and OpenShift API to 20240812094746-86145edb40cf. - Bumped controller-runtime to 0.18.5: - Manager's `Port` option removed, now using dedicated webhook server field (kubernetes-sigs/controller-runtime#2422). - Manager's `MetricsBindAddress` option removed, now using dedicated metrics server field (kubernetes-sigs/controller-runtime#2407). - Cache's `Namespaces` option replaced by `DefaultNamespaces` (kubernetes-sigs/controller-runtime#2421). - Bumped aws-load-balancer-controller to f39ae43121c3 to use latest CRD scheme in e2e tests. - Regenerated CRD and bundle manifests using `make bundle` command. - Bumped `kustomize` to v5 to fix a conflict caused by k8s.io bumps: - `kyaml` unable to use the bumped `github.com/google/gnostic-models/openapiv2` package. - Removed `TypeMeta` from expected deployment object when it's compared to structured one retrieved from fake client (kubernetes-sigs/controller-runtime#2633). - Migrated infrastructure CRD retrieval to the new package `zz_generated.crd-manifests`: - Updated infrastructure CRD for unit tests using `make update-vendored-crds` command. - Updated envtest setup to use the downstream index. - Infrastructure CRD uses CEL functions backported from newer k8s API. - Upstream `envtest` is not ready to use newer CEL function. - Bumped `setup-envtest` to be able to use `--index` flag.

#143) - Bumped golang to 1.22: - Update go.mod - Update Dockerfiles - Bumped k8s.io/* modules to 0.30.3 and OpenShift API to 20240812094746-86145edb40cf. - Bumped controller-runtime to 0.18.5: - Manager's `Port` option removed, now using dedicated webhook server field (kubernetes-sigs/controller-runtime#2422). - Manager's `MetricsBindAddress` option removed, now using dedicated metrics server field (kubernetes-sigs/controller-runtime#2407). - Cache's `Namespaces` option replaced by `DefaultNamespaces` (kubernetes-sigs/controller-runtime#2421). - Bumped aws-load-balancer-controller to f39ae43121c3 to use latest CRD scheme in e2e tests. - Regenerated CRD and bundle manifests using `make bundle` command. - Bumped `kustomize` to v5 to fix a conflict caused by k8s.io bumps: - `kyaml` unable to use the bumped `github.com/google/gnostic-models/openapiv2` package. - Removed `TypeMeta` from expected deployment object when it's compared to structured one retrieved from fake client (kubernetes-sigs/controller-runtime#2633). - Migrated infrastructure CRD retrieval to the new package `zz_generated.crd-manifests`: - Updated infrastructure CRD for unit tests using `make update-vendored-crds` command. - Updated envtest setup to use the downstream index. - Infrastructure CRD uses CEL functions backported from newer k8s API. - Upstream `envtest` is not ready to use newer CEL function. - Bumped `setup-envtest` to be able to use `--index` flag.

As part of an effort to standardize metric exposure across KubeVirt components on port 8443, we are transitioning to HTTPS with TLS encryption for the metrics-server. To facilitate this, we leverage the controller-runtime's SecureServing option, which creates a self-signed certificate and configures it as the server certificate for the metrics endpoint when no external certificate is provided[1]. Subsequent PRs will replace this self-signed certificate with a CDI generated one to enable a fully trusted and secure connection between the Prometheus instance and the target metrics endpoints as specified by the CDI ServiceMonitor. Until that integration is complete, the ServiceMonitor will be configured with insecureSkipVerify to allow scraping despite the untrusted certificate. [1] kubernetes-sigs/controller-runtime#2407 Signed-off-by: Adi Aloni <aaloni@redhat.com>

As part of an effort to standardize metric exposure across KubeVirt components on port 8443, we are transitioning to HTTPS with TLS encryption for the metrics-server. To facilitate this, we leverage the controller-runtime's SecureServing option, which creates a self-signed certificate and configures it as the server certificate for the metrics endpoint when no external certificate is provided[1]. Subsequent PRs will replace this self-signed certificate with a CDI generated one to enable a fully trusted and secure connection between the Prometheus instance and the target metrics endpoints as specified by the CDI ServiceMonitor. Until that integration is complete, the ServiceMonitor will be configured with insecureSkipVerify to allow scraping despite the untrusted certificate. [1] kubernetes-sigs/controller-runtime#2407 Signed-off-by: Adi Aloni <aaloni@redhat.com> Co-authored-by: Adi Aloni <aaloni@redhat.com>

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jul 15, 2023

k8s-ci-robot requested review from FillZpp and varshaprasad96 July 15, 2023 13:17

k8s-ci-robot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 15, 2023

sbueringer changed the title ~~⚠ Implement secure metrics serving~~ [WIP] ⚠ Implement secure metrics serving Jul 15, 2023

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 15, 2023

sbueringer commented Jul 15, 2023

View reviewed changes

pkg/manager/internal.go Outdated Show resolved Hide resolved

sbueringer commented Jul 15, 2023

View reviewed changes

pkg/manager/manager.go Outdated Show resolved Hide resolved

sbueringer force-pushed the pr-auth-metrics branch from 96e77c5 to 0d05be8 Compare July 15, 2023 13:29

alvaroaleman reviewed Jul 16, 2023

View reviewed changes

pkg/manager/internal.go Outdated Show resolved Hide resolved

sbueringer changed the title ~~[WIP] ⚠ Implement secure metrics serving~~ [WIP] ✨ Implement secure metrics serving Jul 17, 2023

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/XL Denotes a PR that changes 500-999 lines, ignoring generated files. labels Jul 17, 2023

sbueringer force-pushed the pr-auth-metrics branch 2 times, most recently from 1504d8f to 66ae79b Compare July 17, 2023 20:20

sbueringer changed the title ~~[WIP] ✨ Implement secure metrics serving~~ ✨ Implement secure metrics serving Jul 17, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 17, 2023

sbueringer force-pushed the pr-auth-metrics branch from ef6074c to f764e41 Compare July 22, 2023 14:22

sbueringer commented Jul 22, 2023

View reviewed changes

pkg/manager/internal.go Outdated Show resolved Hide resolved

sbueringer force-pushed the pr-auth-metrics branch 3 times, most recently from 4b2e71f to 4aa927f Compare July 24, 2023 05:55

sbueringer changed the title ~~✨ Implement secure metrics serving~~ ⚠ Implement secure metrics serving Jul 24, 2023

alvaroaleman reviewed Jul 24, 2023

View reviewed changes

pkg/metrics/filters/filters.go Outdated Show resolved Hide resolved

pkg/metrics/listener.go Outdated Show resolved Hide resolved

pkg/manager/manager.go Outdated Show resolved Hide resolved

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 25, 2023

jiayiwang7 mentioned this pull request Mar 20, 2024

Introduce Metrics Serving in EKS Anywhere aws/eks-anywhere#7875

Closed

tenzen-y mentioned this pull request Apr 10, 2024

Drop Kubernetes v1.25, and Support Kubernetes v1.28 kubeflow/katib#2303

Merged

1 task

camilamacedo86 mentioned this pull request Apr 11, 2024

📖 : Proposal : Discontinue Kube RBAC Proxy in Default Kubebuilder Scaffolding kubernetes-sigs/kubebuilder#3860

Merged

tenzen-y mentioned this pull request Apr 12, 2024

bump k8s deps to v1.28.8 kubernetes-sigs/scheduler-plugins#711

Merged

camilamacedo86 mentioned this pull request Apr 16, 2024

Enhancement of Authentication and Authorization Filters to Allow Custom Configurations and Stability Improvements #2781

Open

camilamacedo86 mentioned this pull request Apr 23, 2024

[META - Phases]: Discontinue Kube RBAC Proxy in Default Kubebuilder Scaffolding kubernetes-sigs/kubebuilder#3871

Closed

5 tasks

regadas mentioned this pull request May 3, 2024

Update controller-runtime to v0.17.3 spotify/flink-on-k8s-operator#797

Merged

sbueringer mentioned this pull request May 7, 2024

🐛 Reintroduce AddMetricsServerExtraHandler on manager #2813

Merged

levan-m mentioned this pull request May 20, 2024

Upgrade controller-runtime, k8s dependencies and switch leader election lock to leases DataDog/extendeddaemonset#186

Merged

zhzhuang-zju mentioned this pull request Jul 17, 2024

Add missing command line arguments for karmada-operator karmada-io/karmada#5174

Merged

mansikulkarni96 mentioned this pull request Aug 22, 2024

WINC-1269: Expose WMCO /metrics endpoint via HTTPS openshift/windows-machine-config-operator#2388

Merged

pavolloffay mentioned this pull request Oct 18, 2024

Kubebuilder scaffolding moves away from kube-rbac-proxy open-telemetry/opentelemetry-operator#3369

Open

matthchr mentioned this pull request Nov 5, 2024

Initial dump of Compositions alpha implementation cloud-native-compositions/compositions#7

Merged

alebedev87 mentioned this pull request Dec 11, 2024

NE-1808: Bump k8s.io, openshift, controller-runtime modules and golang openshift/aws-load-balancer-operator#143

Merged

sujeet01 mentioned this pull request Feb 5, 2025

Remove dependency on kube-rbac-proxy by adopting controller-runtime's native authn/authz for metrics ironcore-dev/ironcore#1226

Merged

Rohit-0505 mentioned this pull request Feb 19, 2025

Drop kube-rbac-proxy container usage in config ironcore-dev/metal-operator#243

Merged

Acedus mentioned this pull request Jul 14, 2025

metrics-server: use SecureServing option kubevirt/containerized-data-importer#3837

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

⚠ Introduce Metrics Options struct & secure metrics serving #2407

⚠ Introduce Metrics Options struct & secure metrics serving #2407

Uh oh!

sbueringer commented Jul 15, 2023 •

edited

Loading

Uh oh!

sbueringer commented Jul 15, 2023 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbueringer commented Jul 17, 2023

Uh oh!

sbueringer commented Jul 18, 2023

Uh oh!

sbueringer commented Jul 22, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

⚠ Introduce Metrics Options struct & secure metrics serving #2407

⚠ Introduce Metrics Options struct & secure metrics serving #2407

Uh oh!

Conversation

sbueringer commented Jul 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbueringer commented Jul 15, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbueringer commented Jul 17, 2023

Uh oh!

sbueringer commented Jul 18, 2023

Uh oh!

sbueringer commented Jul 22, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbueringer commented Jul 15, 2023 •

edited

Loading

sbueringer commented Jul 15, 2023 •

edited

Loading