Secure kube-scheduler #41285

liggitt · 2017-02-11T05:08:05Z

This PR:

Adds a bootstrap system:kube-scheduler clusterrole
Adds a bootstrap clusterrolebinding to the system:kube-scheduler user
Sets up a kubeconfig for kube-scheduler on GCE (following the controller-manager pattern)
Switches kube-scheduler to running with kubeconfig against secured port (salt changes, beware)
Removes superuser permissions from kube-scheduler in local-up-cluster.sh
Adds detailed RBAC deny logging

On kube-up.sh clusters on GCE, kube-scheduler now contacts the API on the secured port.

k8s-reviewable · 2017-02-11T05:08:12Z

This change is

smarterclayton · 2017-02-11T22:27:33Z

Audit and role changes lgtm, not qualified for salt

deads2k · 2017-02-13T14:12:37Z

plugin/pkg/auth/authorizer/rbac/bootstrappolicy/policy.go

+				rbac.NewRule("update").Groups(legacyGroup).Resources("pods/status").RuleOrDie(),
+				// things that select pods
+				rbac.NewRule(Read...).Groups(legacyGroup).Resources("services", "replicationcontrollers").RuleOrDie(),
+				rbac.NewRule(Read...).Groups(extensionsGroup).Resources("replicasets").RuleOrDie(),


This isn't a pattern that's going to scale out well people build other resources that result in pods. Something @kubernetes/sig-scheduling-misc is looking at? We can't bake them all in.

Can we just say that "system:kube-scheduler" is for the built-in scheduler, and custom schedulers may need to come packaged w/ ClusterRoles & ClusterRoleBindings?

yeah, we're saying this won't scale well for the built-in scheduler... it's currently trying to spread selected pods by watching all the resources that use label selectors over pods

😟 yeah that is concerning.

Maybe a subresource for selectors?

deads2k · 2017-02-13T14:13:25Z

permissions look fine.

@cjcullen @mikedanese The script changes seem to be working enough for e2e. Is there something else you'd like to check?

timothysc · 2017-02-13T17:05:31Z

cluster/gce/container-linux/configure-helper.sh

@@ -360,6 +363,30 @@ current-context: service-account-context
 EOF
 }

+function create-kubescheduler-kubeconfig {


Has this percolated to all other systems?

kops/kargo/kubeadm?

/cc @luxas

no. presumably most of those are still running the scheduler against the unsecured port as well, but that doesn't need to block this.

/cc @justinsb @kubernetes/kubeadm-maintainers

working on a guide for deployers setting up control plane components to work out of the box with RBAC at https://docs.google.com/document/d/1PqI--ql3LQsA69fEvRq1nQWgiIoE5Dyftja5Um9ML7Q/edit

that will be pulled into the doc repo as soon as we have a way to open PRs for 1.6 doc content

mikedanese · 2017-02-13T22:53:40Z

plugin/pkg/auth/authorizer/rbac/rbac.go

-				requestAttributes.GetResource(), requestAttributes.GetAPIGroup(), requestAttributes.GetSubresource(),
-			)
+			b := &bytes.Buffer{}
+			fmt.Fprint(b, `"`)


Replacing these Fprints with b.WriteString() would eliminate some unnecessary string scanning.

cjcullen · 2017-02-13T22:54:34Z

Changes LGTM. I don't think there is anything on the GKE-side that will need to change for this (the scripts should be self-contained for this token).

cjcullen · 2017-02-13T22:55:41Z

cluster/gce/trusty/configure-helper.sh

@@ -769,7 +769,7 @@ start_kube_scheduler() {
  if [ -n "${SCHEDULER_TEST_LOG_LEVEL:-}" ]; then
    log_level="${SCHEDULER_TEST_LOG_LEVEL}"
  fi
-  params="${log_level} ${SCHEDULER_TEST_ARGS:-}"
+  params="--master=127.0.0.1:8080 ${log_level} ${SCHEDULER_TEST_ARGS:-}"


Why does this one not use the secure port?

Do we even test or use this kube-up?

Not that I know of, so I was just maintaining status quo since the --master arg moved into the params

liggitt · 2017-02-14T02:59:31Z

@wojtek-t for kubemark review and approval

wojtek-t · 2017-02-14T18:25:31Z

changess to kubemark lgtm

smarterclayton · 2017-02-14T18:30:21Z

@mikedanese can you approve the salt changes?

mikedanese · 2017-02-14T22:19:27Z

/approve
/lgtm

smarterclayton · 2017-02-14T22:20:27Z

/approve

…

On Tue, Feb 14, 2017 at 5:19 PM, Kubernetes Submit Queue < ***@***.***> wrote: [APPROVALNOTIFIER] This PR is *NOT APPROVED* The following people have approved this PR: *liggitt, mikedanese* Needs approval from an approver in each of these OWNERS Files: - cluster/OWNERS <https://github.com/kubernetes/kubernetes/blob/master/cluster/OWNERS> [mikedanese] - cluster/gce/container-linux/OWNERS <https://github.com/kubernetes/kubernetes/blob/master/cluster/gce/container-linux/OWNERS> [mikedanese] - *hack/OWNERS <https://github.com/kubernetes/kubernetes/blob/master/hack/OWNERS>* - plugin/pkg/auth/OWNERS <https://github.com/kubernetes/kubernetes/blob/master/plugin/pkg/auth/OWNERS> [liggitt] - staging/src/k8s.io/apiserver/OWNERS <https://github.com/kubernetes/kubernetes/blob/master/staging/src/k8s.io/apiserver/OWNERS> [liggitt] - test/kubemark/OWNERS <https://github.com/kubernetes/kubernetes/blob/master/test/kubemark/OWNERS> [liggitt,mikedanese] We suggest the following people: cc @ethernetdan <https://github.com/ethernetdan> @jbeda <https://github.com/jbeda> @wojtek-t <https://github.com/wojtek-t> @deads2k <https://github.com/deads2k> You can indicate your approval by writing /approve in a comment You can cancel your approval by writing /approve cancel in a comment — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#41285 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABG_p0yzO9qZV6K3XXEAuve1BCjTeJQwks5rciiGgaJpZM4L-FM6> .

k8s-github-robot · 2017-02-15T06:06:11Z

[APPROVALNOTIFIER] This PR is APPROVED

The following people have approved this PR: liggitt, mikedanese, smarterclayton

Needs approval from an approver in each of these OWNERS Files:

~~cluster/OWNERS~~ [mikedanese,smarterclayton]
~~cluster/gce/container-linux/OWNERS~~ [mikedanese,smarterclayton]
~~hack/OWNERS~~ [smarterclayton]
~~plugin/pkg/auth/OWNERS~~ [liggitt,smarterclayton]
~~staging/src/k8s.io/apiserver/OWNERS~~ [liggitt,smarterclayton]
~~test/kubemark/OWNERS~~ [liggitt,mikedanese,smarterclayton]

You can indicate your approval by writing /approve in a comment
You can cancel your approval by writing /approve cancel in a comment

liggitt · 2017-02-15T06:06:26Z

rebased, changed apiVersion to apiGroup in bootstrap role testdata

k8s-github-robot · 2017-02-15T11:25:09Z

Automatic merge from submit-queue (batch tested with PRs 40297, 41285, 41211, 41243, 39735)

wojtek-t · 2017-02-15T14:53:46Z

This unfortunately broke kubemark - I'm going to revert it (to unblock submit queue).

deads2k · 2017-02-15T15:08:25Z

This unfortunately broke kubemark - I'm going to revert it (to unblock submit queue).

@liggitt

liggitt · 2017-02-15T15:09:53Z

fixed in #41480

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Feb 11, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 11, 2017

k8s-github-robot assigned davidopp Feb 11, 2017

k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Feb 11, 2017

liggitt assigned mikedanese, cjcullen, deads2k and davidopp and unassigned davidopp Feb 11, 2017

k8s-github-robot added release-note-label-needed and removed approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Feb 11, 2017

liggitt force-pushed the kube-scheduler-role branch from 07c2baf to 7b86472 Compare February 11, 2017 21:24

liggitt changed the title ~~Create bootstrap system:kube-scheduler role~~ Secure kube-scheduler Feb 11, 2017

deads2k reviewed Feb 13, 2017

View reviewed changes

timothysc reviewed Feb 13, 2017

View reviewed changes

mikedanese reviewed Feb 13, 2017

View reviewed changes

cjcullen reviewed Feb 13, 2017

View reviewed changes

liggitt force-pushed the kube-scheduler-role branch from 7b86472 to 1ce1430 Compare February 14, 2017 02:58

liggitt unassigned davidopp Feb 14, 2017

liggitt assigned wojtek-t Feb 14, 2017

liggitt added this to the v1.6 milestone Feb 14, 2017

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 14, 2017

k8s-github-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Feb 14, 2017

liggitt added 3 commits February 15, 2017 01:05

Create bootstrap system:kube-scheduler role

9f8b803

Switch kube-scheduler to secure API access

cc11d73

Add detailed RBAC deny logging

34782b2

liggitt force-pushed the kube-scheduler-role branch from 1ce1430 to 34782b2 Compare February 15, 2017 06:06

k8s-github-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 15, 2017

liggitt added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Feb 15, 2017

k8s-github-robot merged commit e4a4fe4 into kubernetes:master Feb 15, 2017

liggitt deleted the kube-scheduler-role branch February 15, 2017 14:08

This was referenced Feb 15, 2017

Revert "Secure kube-scheduler" #41478

Closed

Extend tests for kubemark presubmit #41481

Closed

shyamjvs mentioned this pull request Feb 15, 2017

RBAC roles for some components still missing #40021

Closed

3 tasks

luxas mentioned this pull request Feb 16, 2017

The scheduler and controller-mananager should talk securely to the apiserver kubernetes/kubeadm#172

Closed

fen4o mentioned this pull request Mar 2, 2017

RBAC bootstrap roles should not contain optional cluster addon roles and bindings #42401

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Secure kube-scheduler #41285

Secure kube-scheduler #41285

liggitt commented Feb 11, 2017 •

edited

Loading

k8s-reviewable commented Feb 11, 2017

smarterclayton commented Feb 11, 2017

deads2k Feb 13, 2017

cjcullen Feb 13, 2017

liggitt Feb 13, 2017

cjcullen Feb 13, 2017

0xmichalis Feb 15, 2017

deads2k commented Feb 13, 2017

timothysc Feb 13, 2017 •

edited

Loading

liggitt Feb 13, 2017 •

edited

Loading

timothysc Feb 14, 2017

liggitt Feb 14, 2017

mikedanese Feb 13, 2017

liggitt Feb 14, 2017

cjcullen commented Feb 13, 2017

cjcullen Feb 13, 2017

mikedanese Feb 13, 2017

liggitt Feb 14, 2017

liggitt commented Feb 14, 2017

wojtek-t commented Feb 14, 2017

smarterclayton commented Feb 14, 2017

mikedanese commented Feb 14, 2017

smarterclayton commented Feb 14, 2017 via email

k8s-github-robot commented Feb 15, 2017

liggitt commented Feb 15, 2017 •

edited

Loading

k8s-github-robot commented Feb 15, 2017

wojtek-t commented Feb 15, 2017

deads2k commented Feb 15, 2017

liggitt commented Feb 15, 2017

Secure kube-scheduler #41285

Secure kube-scheduler #41285

Conversation

liggitt commented Feb 11, 2017 • edited Loading

k8s-reviewable commented Feb 11, 2017

smarterclayton commented Feb 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

deads2k commented Feb 13, 2017

timothysc Feb 13, 2017 • edited Loading

Choose a reason for hiding this comment

liggitt Feb 13, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cjcullen commented Feb 13, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

liggitt commented Feb 14, 2017

wojtek-t commented Feb 14, 2017

smarterclayton commented Feb 14, 2017

mikedanese commented Feb 14, 2017

smarterclayton commented Feb 14, 2017 via email

k8s-github-robot commented Feb 15, 2017

liggitt commented Feb 15, 2017 • edited Loading

k8s-github-robot commented Feb 15, 2017

wojtek-t commented Feb 15, 2017

deads2k commented Feb 15, 2017

liggitt commented Feb 15, 2017

liggitt commented Feb 11, 2017 •

edited

Loading

timothysc Feb 13, 2017 •

edited

Loading

liggitt Feb 13, 2017 •

edited

Loading

liggitt commented Feb 15, 2017 •

edited

Loading