Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⚠️ Discontinue Kube RBAC Proxy in Default Kubebuilder Scaffolding #3899

Merged

Conversation

@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. labels May 6, 2024
@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 6, 2024
- path: manager_auth_proxy_patch.yaml
# [METRICS] The following patch will enable the metrics endpoint. Ensure that you also protect this endpoint.
# If you want to expose the metric endpoint of your controller-manager uncomment the following line.
#- path: manager_metrics_patch.yaml
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Metrics should not be enable by default as it was before.

@camilamacedo86 camilamacedo86 requested review from erikgb and brancz May 6, 2024 16:04
@camilamacedo86 camilamacedo86 changed the title ⚠️ Discontinue Kube RBAC Proxy in Default Kubebuilder Scaffolding WIP: ⚠️ Discontinue Kube RBAC Proxy in Default Kubebuilder Scaffolding May 6, 2024
@k8s-ci-robot k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 6, 2024
@camilamacedo86 camilamacedo86 force-pushed the discontinue-rbac branch 2 times, most recently from 92b585f to 00f5ecf Compare May 6, 2024 17:56
@camilamacedo86 camilamacedo86 changed the title WIP: ⚠️ Discontinue Kube RBAC Proxy in Default Kubebuilder Scaffolding ⚠️ Discontinue Kube RBAC Proxy in Default Kubebuilder Scaffolding May 6, 2024
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label May 6, 2024
Copy link
Contributor

@erikgb erikgb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did a quick scan, and I think there is more to be cleaned up.

@camilamacedo86 camilamacedo86 force-pushed the discontinue-rbac branch 3 times, most recently from aba519b to cc85d34 Compare May 6, 2024 22:20
@camilamacedo86
Copy link
Member Author

Hi @erikgb

Thank you a lot for your time and help.
It was amazing !!!

All points are addressed and I supplement the docs to let the users know how to protected the endpoint.
I hope that is good enough to fly now !!!

@camilamacedo86 camilamacedo86 requested a review from erikgb May 6, 2024 22:22
containers:
- name: manager
args:
- "--metrics-bind-address=0.0.0.0:8080"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this patch is needed to enable metrics. We have our metrics scraped by Prometheus without this arg set. Ref. controller-runtime argument help.

-metrics-bind-address string
The address the metric endpoint binds to. (default ":8080")

So if you want to have a patch to enable the metrics endpoint, you have to disable it by default by setting it to 0, ref. https://github.com/kubernetes-sigs/controller-runtime/blob/479b723944e34ae42c9911fe01228ff34eb5ca81/pkg/metrics/server/server.go#L120-L122

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh. thank you a lot for check and share it.
I think ideally we should not enable by default.
Not everybody want to use it and by enable the metrics is required to protect the endpoint
So, it would be better if that is a conscious decision.

Copy link
Member Author

@camilamacedo86 camilamacedo86 May 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done the changes and also added a test to ensure that the metrics endpoint will not be exposed in this case

It("should generate a runnable project without metrics exposed", func() {
			kbc.IsRestricted = false
			GenerateV4WithoutMetrics(kbc)
			Run(kbc, true, false, false)
})

In this case, we are using the curl pod to ensure that the connection will be refused such as:

$ kubectl logs curl -n e2e-vbau-system
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 10.96.136.215:8080...
* connect to 10.96.136.215 port 8080 failed: Connection refused
* Failed to connect to e2e-vbau-controller-manager-metrics-service.e2e-vbau-system.svc.cluster.local port 8080 after 3 ms: Connection refused
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
curl: (7) Failed to connect to e2e-vbau-controller-manager-metrics-service.e2e-vbau-system.svc.cluster.local port 8080 after 3 ms: Connection refused

Thank you a lot.

- --leader-elect
- --leader-elect
- --health-probe-bind-address=:8081
- --metrics-bind-address=0
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

--metrics-bind-address=0 -> Used here to disable it by default

@camilamacedo86 camilamacedo86 force-pushed the discontinue-rbac branch 4 times, most recently from ab83bd8 to 5d38a92 Compare May 7, 2024 18:13
@camilamacedo86 camilamacedo86 force-pushed the discontinue-rbac branch 2 times, most recently from 9efea29 to b332e82 Compare May 14, 2024 23:56
Copy link
Member

@varshaprasad96 varshaprasad96 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for putting this through @camilamacedo86!

/lgtm
/approve

Just a few nits, can be handled later too!

Comment on lines 23 to 24
we now require the use of shared infrastructure. But as [`kube-rbac-proxy`](https://github.com/brancz/kube-rbac-proxy) is a process
to be part of but not yet, sadly we cannot build and promote these images using the new
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
we now require the use of shared infrastructure. But as [`kube-rbac-proxy`](https://github.com/brancz/kube-rbac-proxy) is a process
to be part of but not yet, sadly we cannot build and promote these images using the new
we now require the use of shared infrastructure. But as [`kube-rbac-proxy`](https://github.com/brancz/kube-rbac-proxy) is in a process
to be a part of it, but not yet, sadly we cannot build and promote these images using the new

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

fmt.Sprintf("\n- %[1]s_editor_role.yaml\n- %[1]s_viewer_role.yaml", crdName))
if err != nil {
log.Errorf("Unable to add Editor and Viewer roles in the file "+
"%s.", rbacKustomizeFilePath)
}
// Add an empty line at the end of the file
err = pluginutil.AppendCodeIfNotExist(rbacKustomizeFilePath,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just curious, why is appending an empty line at the end of this brace file necessary?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To avoid breaking changes. So, it can work with scaffolds that has the kube-rbac-proxy or not.

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 15, 2024
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: camilamacedo86, varshaprasad96

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:
  • OWNERS [camilamacedo86,varshaprasad96]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 15, 2024
@k8s-ci-robot
Copy link
Contributor

New changes are detected. LGTM label has been removed.

@camilamacedo86 camilamacedo86 force-pushed the discontinue-rbac branch 4 times, most recently from 8d39b62 to 6ddf84b Compare May 15, 2024 08:07
@camilamacedo86 camilamacedo86 merged commit c01af8f into kubernetes-sigs:master May 15, 2024
16 of 18 checks passed
@camilamacedo86 camilamacedo86 deleted the discontinue-rbac branch May 15, 2024 10:10
Len4i pushed a commit to lightrun-platform/lightrun-k8s-operator that referenced this pull request May 30, 2024
Len4i pushed a commit to lightrun-platform/lightrun-k8s-operator that referenced this pull request May 30, 2024
* fix eks/gke semver restriction by `kubeVersion`

* add pprof optional endpoint

* deprecate kube-proxy due to [PR](kubernetes-sigs/kubebuilder#3899)

* fix e2e pipeline
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. release-blocker size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files.
Projects
None yet
4 participants