🚨 Action Required: Ensure that you no longer use gcr.io/kubebuilder images

[camilamacedo86]

⚠️ IMPORTANT ⚠️

If your project uses gcr.io/kubebuilder/kube-rbac-proxy it will be affected. Your project may fail to work if the image cannot be pulled. You must move as soon as possible, sometime from early 2025, the GCR will go away.

Unfortunately, we're unable to provide any guarantees regarding timelines or potential extensions at this time. Images provided under GRC will be unavailable from March 18, 2025, as per announcement. However, gcr.io/kubebuilder/ may be unavailable before this date due to efforts to deprecate infrastructure.

Key Update

kube-rbac-proxy was historically used to protect the metrics endpoint. However, its usage has been discontinued in Kubebuilder.
The default scaffold now leverages the WithAuthenticationAndAuthorization feature provided by Controller-Runtime.

This feature provides integrated support for securing metrics endpoints by embedding authentication (authn) and authorization (authz) mechanisms directly into the controller manager's metrics server, replacing the need for (https://github.com/brancz/kube-rbac-proxy) to secure metrics endpoints.

Why This Matters

• Unavailability: If your project depends on the image, it may no longer work when the image becomes unavailable.
• Security: Unprotected metrics endpoints may expose sensitive data like system performance and application behavior, creating security vulnerabilities.

What You Need to Do

Follow some options.

Option 1 - Upgrade the Project to the Latest Release (Recommended)

Upgrade your project to the latest release by re-scaffolding it and reintegrating your custom code.

• You can use kubebuilder alpha generate to regenerate the project from the PROJECT config.
• This ensures a default implementation (similar to kube-rbac-proxy) with options to improve production readiness, such as configuring certificates securely. This approach also allows you to take advantage of other improvements, bug fixes, and the latest updates.

Option 2 - Manually Modify Your Project

If you prefer not to fully upgrade, modify your project to use the built-in authn/authz protection via Controller-Runtime.

• Follow the instructions in the FAQ: "How can I manually change my project to switch to Controller-Runtime's built-in auth protection?".

Option 3 - Continue Using the kube-rbac-proxy Image (Not adopt or promoted by Kubebuilder)

If you want to continue using kube-rbac-proxy, source the image from an alternative location, at your own risk. Examples include:

• quay.io/brancz/kube-rbac-proxy
• Red Hat Registry (⚠️ If you are allowed to use it. Ensure that you check its terms & conditions. Example)

To mitigate risks, we manually mirror the images to registry.k8s.io/kubebuilder/kube-rbac-proxy. This registry is managed by the #sig-k8s-infra. However, we cannot promote the kube-rbac-proxy images on this registry or recommend their usage since they have been discontinued from the project.

FAQ

Why does the metrics endpoint need to be protected?

Unprotected metrics endpoints can expose sensitive data, such as system performance and application behavior, to unauthorized users. This can lead to security vulnerabilities where attackers gain insights into the system's operation and exploit weaknesses.

---

How can the metrics endpoint be protected?

The following are some options with details and info for those who were introduced to support and helpers in Kubebuilder.

1. (Protection enabled by default from release v4.1.0)
   Use Controller-Runtime's feature WithAuthenticationAndAuthorization to enable authn/authz for metrics endpoints. For reference, see the code from the release 4.3.1

2. (Optional helper introduced from release v4.2.0)
   Use NetworkPolicies to secure metrics endpoints.
   Example configuration: NetworkPolicy Example

   Also, feel free to check the (external code example)

3. Integrate cert-manager:
   Secure the metrics endpoint using TLS encryption with cert-manager. For example, ensure that you use valid certficates such as:

   • Reference for the ServiceMonitor to integrate it with Prometheus:

   kubebuilder/testdata/project-v4/config/prometheus/monitor_tls_patch.yaml

   Lines 10 to 22 in e07823e

   	- tlsConfig:
   	insecureSkipVerify: false
   	ca:
   	secret:
   	name: metrics-server-cert
   	key: ca.crt
   	cert:
   	secret:
   	name: metrics-server-cert
   	key: tls.crt
   	keySecret:
   	name: metrics-server-cert
   	key: tls.key

   Note that we plan add a feature/helper for it in the next release, see the PR: ✨ (go/v4): feat/fix: enhance cert-manager integration for metrics endpoints (follow-up to PR #4243) #4400

---

Why doesn't Network Policy provide the same level of protection as kube-rbac-proxy?

NetworkPolicy acts as a firewall for pods, controlling traffic flow at the IP or port level. However, it doesn’t handle authentication (authn), authorization (authz), or encryption like kube-rbac-proxy does.

---

How can I manually change my project to switch to Controller-Runtime's built-in auth protection?

Following these steps should help you resolve the issue. However, there may be a few caveats depending on how old the version used to create the project is, especially if it has never been upgraded to the latest versions by re-creating the project and adding your code changes on top.

1. Remove kube-rbac-proxy configurations:
   Ensure that you remove the container with the name: kube-rbac-proxy. Example from an old version

2. Ensure that your manager will have the args for the metrics service and binding at the same port

• For example, see the metrics service:

  kubebuilder/testdata/project-v4/config/default/metrics_service.yaml

  Lines 8 to 17 in e07823e

  	name: controller-manager-metrics-service
  	namespace: system
  	spec:
  	ports:
  	- name: https
  	port: 8443
  	protocol: TCP
  	targetPort: 8443
  	selector:
  	control-plane: controller-manager

• For example, see the arg to be patch to binding the metrics service:

  kubebuilder/testdata/project-v4/config/default/manager_metrics_patch.yaml

  Lines 2 to 4 in e07823e

  	- op: add
  	path: /spec/template/spec/containers/0/args/0
  	value: --metrics-bind-address=:8443

3. Enable built-in auth in main.go:

• Update your main.go to implement WithAuthenticationAndAuthorization. Here’s an example:

  // if the enable-http2 flag is false (the default), http/2 should be disabled
  // due to its vulnerabilities. More specifically, disabling http/2 will
  // prevent from being vulnerable to the HTTP/2 Stream Cancellation and
  // Rapid Reset CVEs. For more information see:
  // - https://github.com/advisories/GHSA-qppj-fm5r-hxr3
  // - https://github.com/advisories/GHSA-4374-p667-p6c8
  disableHTTP2 := func(c *tls.Config) {
   setupLog.Info("disabling http/2")
   c.NextProtos = []string{"http/1.1"}
  }
  
  if !enableHTTP2 {
   tlsOpts = append(tlsOpts, disableHTTP2)
  }
  
  metricsServerOptions := metricsserver.Options{
      BindAddress:   metricsAddr,
      SecureServing: secureMetrics,
      TLSOpts:       tlsOpts,
      FilterProvider: filters.WithAuthenticationAndAuthorization,
  }

• For a full implementation example, see the samples under testdata. For reference, see the code from the release 4.3.1:

kubebuilder/testdata/project-v4/cmd/main.go

Lines 89 to 152 in a9ee390

	// if the enable-http2 flag is false (the default), http/2 should be disabled
	// due to its vulnerabilities. More specifically, disabling http/2 will
	// prevent from being vulnerable to the HTTP/2 Stream Cancellation and
	// Rapid Reset CVEs. For more information see:
	// - https://github.com/advisories/GHSA-qppj-fm5r-hxr3
	// - https://github.com/advisories/GHSA-4374-p667-p6c8
	disableHTTP2 := func(c *tls.Config) {
	setupLog.Info("disabling http/2")
	c.NextProtos = []string{"http/1.1"}
	}
	if !enableHTTP2 {
	tlsOpts = append(tlsOpts, disableHTTP2)
	}
	webhookServer := webhook.NewServer(webhook.Options{
	TLSOpts: tlsOpts,
	})
	// Metrics endpoint is enabled in 'config/default/kustomization.yaml'. The Metrics options configure the server.
	// More info:
	// - https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.19.1/pkg/metrics/server
	// - https://book.kubebuilder.io/reference/metrics.html
	metricsServerOptions := metricsserver.Options{
	BindAddress: metricsAddr,
	SecureServing: secureMetrics,
	TLSOpts: tlsOpts,
	}
	if secureMetrics {
	// FilterProvider is used to protect the metrics endpoint with authn/authz.
	// These configurations ensure that only authorized users and service accounts
	// can access the metrics endpoint. The RBAC are configured in 'config/rbac/kustomization.yaml'. More info:
	// https://pkg.go.dev/sigs.k8s.io/controller-runtime@v0.19.1/pkg/metrics/filters#WithAuthenticationAndAuthorization
	metricsServerOptions.FilterProvider = filters.WithAuthenticationAndAuthorization
	// TODO(user): If CertDir, CertName, and KeyName are not specified, controller-runtime will automatically
	// generate self-signed certificates for the metrics server. While convenient for development and testing,
	// this setup is not recommended for production.
	}
	mgr, err := ctrl.NewManager(ctrl.GetConfigOrDie(), ctrl.Options{
	Scheme: scheme,
	Metrics: metricsServerOptions,
	WebhookServer: webhookServer,
	HealthProbeBindAddress: probeAddr,
	LeaderElection: enableLeaderElection,
	LeaderElectionID: "da1d9c86.testproject.org",
	// LeaderElectionReleaseOnCancel defines if the leader should step down voluntarily
	// when the Manager ends. This requires the binary to immediately end when the
	// Manager is stopped, otherwise, this setting is unsafe. Setting this significantly
	// speeds up voluntary leader transitions as the new leader don't have to wait
	// LeaseDuration time first.
	//
	// In the default scaffold provided, the program ends immediately after
	// the manager stops, so would be fine to enable this option. However,
	// if you are doing or is intended to do any operation such as perform cleanups
	// after the manager stops then its usage might be unsafe.
	// LeaderElectionReleaseOnCancel: true,
	})
	if err != nil {
	setupLog.Error(err, "unable to start manager")
	os.Exit(1)
	}

Note: Please ensure that you disable HTTP/2 by default, as you see in the above example. Disable HTTP/2 still required: kubernetes/kubernetes#121197

5 - Add e2e tests to validate the metrics endpoint.

Projects created with the latest versions are scaffolded with comprehensive E2E tests, including code to validate the metrics endpoint. Example. See: https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/test/e2e/e2e_test.go#L166-L235

Note that you can remove the Prometheus block if you are not providing this integration.

6. Build and Test Manually:

Build your project and ensure the metrics endpoint is working and protected with RBAC as expected. The RBAC permissions scaffolded under config/rbac should provide the required permissions. However, if you face issues, you might want to look at the last scaffolds to generate them properly. See that under testdata; we have examples.

Steps to Verify Metrics with curl manually

1. Create the role-binding

kubectl create clusterrolebinding <project-name>-metrics-binding \
  --clusterrole=<project-name>-metrics-reader \
  --serviceaccount=<project-name>-system:<project-name>-controller-manager

2. Generate the token to authenticate in the metrics

export TOKEN=$(kubectl create token operator-controller-controller-manager -n olmv1-system)
echo $TOKEN

3. Run the curl pod to perform the tests

kubectl run curl-metrics --rm -it --restart=Never \
  --image=curlimages/curl:7.87.0 -n <project>-system -- /bin/sh

4. Call the metrics enpoint using the TOKEN

curl -v -k -H "Authorization: Bearer $TOKEN" https://<my-project-name>-controller-manager-metrics-service.<my-project-name>-system.svc.cluster.local:8443/metrics

---

❓ Why is this happening?

The kube-rbac-proxy images have been rebuilt and re-tagged by Kubebuilder for an extended period. However, due to infrastructure changes within the Kubernetes ecosystem and the deprecation of Google Cloud Platform’s Container Registry (details here), continuing to maintain these images is no longer feasible.

Additionally, the project has been in the process of being donated to SIG-Auth for an extended period. Despite these efforts, significant requirements still need to be addressed before SIG-Auth can consider adopting the project. The latest review outlined several tasks being tracked here: kube-rbac-proxy issue #238.

For more details, refer to:

• Proposal for this change
• Efforts to deprecate infrastructure
• Meta Issue to Discontinue Kube RBAC Proxy in Default Scaffolding
• SIG-Auth Pre-Acceptance 2nd Review

---

Why can't we recommend just replacing the kube-rbac-proxy image with the latest version from another trusted source?

The following are some key reasons.

• Deprecated Features: The kube-rbac-proxy’s self-signed certificate generation is deprecated and unsuitable for production. Future updates will require actual certificates.
• Security Concerns: Updating the image alone doesn’t adequately address security needs for protecting metrics endpoints in production environments.
• Lack of SIG-Auth Support: The kube-rbac-proxy project is not formally part of SIG-Auth and has been in the process of donation for a long period but is not there yet. For details on what is needed after the second review round, please look at GitHub Issue #238.
• Better Alternatives Regarding Maintainability: Controller-runtime provides a better approach in the long-term. Note that when maintainers want to support new k8s versions in their solutions, they upgrade the controller runtime. This means projects using it can quickly adopt new enhancements, bug fixes, and metrics service protection improvements.
• Kubebuilder No Longer Adopts This Solution: The usage of kube-rbac-proxy was discontinued mainly for the reasons mentioned above and due to efforts to deprecate infrastructure. Furthermore, recommending its use or replacing it with a similar image introduces long-term risks, such as reliance on registries that may not remain available or supported indefinitely.

---

Please update your configurations accordingly to avoid disruptions. If you have any questions or need further assistance, feel free to ask in this discussion thread or the Kubebuilder Slack channel.

For further information, check the metrics section in the documentation: https://book.kubebuilder.io/reference/metrics.

[camilamacedo86]

Hi everyone,

We’ve been updating this discussion to provide the latest information, including details about the helpers implemented in Kubebuilder. Additionally, we’ve been enhancing it by, for example, adding more comprehensive steps to help you manually update your project if needed.

🖐️ Check out the FAQ section: "How can I manually change my project to switch to Controller-Runtime's built-in auth protection?" for detailed instructions.

We will continue to keep this discussion up to date.
If you face any issues, please don’t hesitate to reach out to us.

[camilamacedo86]

Hi everyone,

We’ve been updating this discussion to provide the latest information.

🖐️ Check out the FAQ section: Why can't we recommend just replacing the kube-rbac-proxy image with the latest version from another trusted source?

We will continue to keep this discussion up to date.
If you face any issues, please don’t hesitate to reach out to us.

[camilamacedo86]

Hey everybody,

For the next release (4.4.0, which will support Kubernetes 1.32), we are introducing a feature that will help users configure the metrics server with certificates, including those provided by cert-manager.

You might want to take a look at the related PR #4400 for more details!

Best regards,

[mattwelke]

@camilamacedo86 Thanks for the migration guide! We were able to migrate, but we have a question about k8s RBAC if you don't mind clearing things up for us.

We used the approach where you change main.go to use WithAuthenticationAndAuthorization from controller-runtime. We have things running where the metrics server appears to be starting correctly. Logs show this:

2024-12-05T19:57:49Z    INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8443", "secure": true}

However, we found that we can't make a successful request (it says it's unauthenticated) to the metrics endpoint unless we add a ClusterRoleBinding to bind the metrics-reader ClusterRole to a service account that we have access to that we can use to get a token. We used the controller manager service account to test this. Then, we were able to run a command like curl -H "Authorization: Bearer $TOKEN" https://10.96.99.213:8443/metrics -k to see the metrics.

It's worth noting that we don't make use of the metrics from our custom operator yet, we're just following this migration guide.

The example project has a ClusterRole (https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v4/config/rbac/metrics_reader_role.yaml) but not a ClusterRoleBinding that binds it to anything.

Is the idea that the example project isn't supposed to be a complete example that shows reading metrics, because everyone will have a different way they need to read metrics? And a different thing reading them, which would have its own service account that would be bound?

[camilamacedo86]

Hi @mattwelke,

Thank you for raising this question! I'm glad the migration guide helped you move forward. Let me address your points inline:

The example project has a ClusterRole (link) but not a ClusterRoleBinding that binds it to anything.

Both the ClusterRole and ClusterRoleBinding are provided in the example. The samples under testdata are continuously updated and tested via our CI pipeline. Here's the relevant file you may have missed:

metrics_auth_role_binding.yaml

Is the idea that the example project isn't supposed to be a complete example that shows reading metrics, because everyone will have a different way they need to read metrics? And a different thing reading them, which would have its own service account that would be bound?

The examples under testdata are generated by Kubebuilder's CLI and built as part of our development process. These files represent what the CLI scaffolds for users by default. You can find the commands that generate these samples here.

In the latest Kubebuilder releases, the CLI scaffolds a lot to support common use cases, including end-to-end (e2e) testing. For example, the scaffolded e2e tests include checks for the metrics endpoint. You can see this in the default e2e test file.

However, how the metrics endpoint is used depends on the specific requirements of each implementation. For instance:

• You might integrate the metrics endpoint with Prometheus. (Our default scaffold includes helpers for this integration.)
• You might integrate it with Grafana, using the files generated by our optional plugin for Grafana dashboards.
• You could use it for other monitoring solutions within your cluster.
• Either you can use it in your e2e tests, such as we use to validate the projects built with it as its features. Refer to how the metrics endpoint is tested in Kubebuilder's CLI itself. Here's an example: plugin_cluster_test.go

To answer your question directly: The example project doesn't dictate how to consume metrics because this varies across use cases. Instead, the scaffolds provide a solid foundation that you can extend based on your specific needs.

I hope this helps! Let me know if you have any other questions. 😊

[mattwelke]

That's super clear. Ty!

[camilamacedo86]

Hi @mattwelke

I also updated the steps to test to try to make it easier.
Thank you a lot for raising that.