Removed awk from kubeadm reset #81494

Klaven · 2019-08-16T04:54:57Z

removed awk usage from kubeadm reset in favor of native golang calls
that are not vulnerable to expansion.

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespaces from that line:

/kind api-change

/kind bug

/kind cleanup
/kind design
/kind documentation
/kind failing-test
/kind feature
/kind flake

What this PR does / why we need it:
This PR was created based on results from the security assessment,

Which issue(s) this PR fixes:

Fixes #
xref#/kubernetes/kubeadm/issues/1715

Special notes for your reviewer:

@neolit123 :
Wanted to get feedback. in it's current state it's not very testable. But at the same time I did not want to pollute the core change with changes not related to removing awk.

also, I would not expect a function called absoluteKubeletRunDirectory to unmount directories. but again thought this change would pollute the core change. If it is acceptable I will make a new PR later this week to address this.

Does this PR introduce a user-facing change?:

kubeadm reset: unmount directories under "/var/lib/kubelet" for linux only

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

cmd/kubeadm/app/cmd/phases/reset/cleanupnode.go

Klaven · 2019-08-16T18:11:25Z

Added WIP due to testing needing to be run on at least one of the other platforms. I should be able to do this over the weekend.

Tested on Ubuntu 18.04 server vm and successfully unmounted correct directories.

Klaven · 2019-08-16T21:28:22Z

/test pull-kubernetes-e2e-gce

neolit123 · 2019-08-16T22:39:02Z

Added WIP due to testing needing to be run on at least one of the other platforms. I should be able to do this over the weekend.

i think it's fine to merge without testing !linux as its just a NOOP there.

Does this PR introduce a user-facing change?:
perhaps we should add a release note for this one:

kubeadm: unmount directories under "/var/lib/kubelet" for linux only

thanks.

/approve
/priority backlog
/assign @rosti
PTAL too.

k8s-ci-robot · 2019-08-16T22:39:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: Klaven, neolit123

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [neolit123]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

rosti

Thanks @Klaven !
Overall, this looks OK, but we may need to think on better platform split for reset (WRT the Windows side of things).

rosti · 2019-08-19T09:49:48Z

cmd/kubeadm/app/cmd/phases/reset/unmount_linux.go

+		if len(m) < 2 || !strings.HasPrefix(m[1], kubeadmconstants.KubeletRunDirectory) {
+			continue
+		}
+		if err := syscall.Unmount(m[1], syscall.MNT_FORCE); err != nil {


The original call did not imply MNT_FORCE. Also, that flag is somewhat misleading. It is actually used to force unmount a filesystem, which is a remote one and the connection to the server is lost. It does not force unmount a filesystem if it has opened handles.

I can remove this if you like. will have to be tonight as it will mean I need to retest it.

this has been removed and tested. (testing methodology posted.) I agree, it's misleading and really not helpful in this case. Also not used on the original so even if we wanted to add it would probably be better to add in new PR.

rosti · 2019-08-19T10:53:02Z

cmd/kubeadm/app/cmd/phases/reset/unmount.go

@@ -0,0 +1,29 @@
+// +build !linux


Let's not do premature platform splitting in reset at this point. It's almost all Linux dependent and we need to have the whole picture in mind to take the best approach.

but this function certainly should not run on _windows.go (NO-OP).

@rosti I guess I don't understand, if I don't have this file the code will not pass checks because it does not have the functions used in the unmount_linux.go file on windows and mac. For both windows and mac this is a no-op. How are you wanting me to handle compiling for windows and linux if not with this file?

Ok, we can leave it like this for now, but we may need to redesign and rename the files.
The question is if we should place absoluteKubeletRunDirectory in the platform specific portions of cleanupnode.go? So the idea is to have cleanupnode_unix.go and cleanupnode_windows.go with absoluteKubeletRunDirectory and a few other things in it.

But, certainly, that must be done in another PR.

Klaven · 2019-08-20T03:33:15Z

my test: (with removal of force)

>  sudo cat /proc/mounts | grep /var/lib/kubelet
/dev/loop5 /var/lib/kubelet ext4 rw,relatime,data=ordered 0 0

> sudo kubeadm reset
....

>  sudo cat /proc/mounts | grep /var/lib/kubelet

rosti · 2019-08-20T08:41:38Z

cmd/kubeadm/app/cmd/phases/reset/cleanupnode.go

-	umountDirsCmd := fmt.Sprintf("awk '$2 ~ path {print $2}' path=%s/ /proc/mounts | xargs -r umount", absoluteKubeletRunDirectory)
-	klog.V(1).Infof("[reset] Executing command %q", umountDirsCmd)
-	umountOutputBytes, err := exec.Command("sh", "-c", umountDirsCmd).Output()
+	err = unmountKubeletDirectory()


A thing, that I've missed on the first run here is, that we need absoluteKubeletRunDirectory to be passed to unmountKubeletDirectory (instead of using a const there).

rosti

Thanks @Klaven !
Let's merge these as they are, but first, please, squash your commits.

removed awk from kubeadm reset in favor of native go lang calls that are not vulnerable to expantion.

Klaven · 2019-08-20T18:48:45Z

/test pull-kubernetes-kubemark-e2e-gce-big

rosti

/lgtm

Klaven changed the title ~~removed awk from kubeadm reset~~ [WIP] removed awk from kubeadm reset Aug 16, 2019

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2019

k8s-ci-robot requested review from fabriziopandini and neolit123 August 16, 2019 04:55

k8s-ci-robot added area/kubeadm sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. and removed needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels Aug 16, 2019

Klaven changed the title ~~[WIP] removed awk from kubeadm reset~~ Removed awk from kubeadm reset Aug 16, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2019

neolit123 reviewed Aug 16, 2019

View reviewed changes

cmd/kubeadm/app/cmd/phases/reset/cleanupnode.go Outdated Show resolved Hide resolved

Klaven force-pushed the remove_awk branch from a0a005e to d0a5b45 Compare August 16, 2019 17:52

Klaven changed the title ~~Removed awk from kubeadm reset~~ [WIP] Removed awk from kubeadm reset Aug 16, 2019

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Aug 16, 2019

k8s-ci-robot assigned rosti Aug 16, 2019

k8s-ci-robot added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed needs-priority Indicates a PR lacks a `priority/foo` label and requires one. labels Aug 16, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 16, 2019

Klaven changed the title ~~[WIP] Removed awk from kubeadm reset~~ Removed awk from kubeadm reset Aug 17, 2019

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. and removed do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. labels Aug 17, 2019

k8s-ci-robot removed the release-note-none Denotes a PR that doesn't merit a release note. label Aug 17, 2019

rosti reviewed Aug 19, 2019

View reviewed changes

rosti reviewed Aug 20, 2019

View reviewed changes

Klaven force-pushed the remove_awk branch from 3d4cd2f to 4adf0c4 Compare August 20, 2019 12:59

rosti approved these changes Aug 20, 2019

View reviewed changes

removed awk from kubeadm reset

6845c66

removed awk from kubeadm reset in favor of native go lang calls that are not vulnerable to expantion.

Klaven force-pushed the remove_awk branch from 4adf0c4 to 6845c66 Compare August 20, 2019 15:54

rosti approved these changes Aug 21, 2019

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Aug 21, 2019

k8s-ci-robot merged commit 17a1859 into kubernetes:master Aug 21, 2019

k8s-ci-robot added this to the v1.16 milestone Aug 21, 2019

neolit123 mentioned this pull request Aug 28, 2019

TOB-K8S-014: Kubeadm performs potentially-dangerous reset operations #81120

Closed

cji mentioned this pull request Nov 10, 2019

Kubernetes 3rd Party Security Audit Findings #81146

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Removed awk from kubeadm reset #81494

Removed awk from kubeadm reset #81494

Klaven commented Aug 16, 2019 •

edited

Loading

Klaven commented Aug 16, 2019

Klaven commented Aug 16, 2019

neolit123 commented Aug 16, 2019

k8s-ci-robot commented Aug 16, 2019

rosti left a comment

rosti Aug 19, 2019

Klaven Aug 19, 2019

Klaven Aug 20, 2019

rosti Aug 19, 2019 •

edited

Loading

neolit123 Aug 19, 2019

Klaven Aug 19, 2019

rosti Aug 20, 2019

Klaven commented Aug 20, 2019 •

edited

Loading

rosti Aug 20, 2019

neolit123 Aug 20, 2019

rosti left a comment

Klaven commented Aug 20, 2019

rosti left a comment

Removed awk from kubeadm reset #81494

Removed awk from kubeadm reset #81494

Conversation

Klaven commented Aug 16, 2019 • edited Loading

Klaven commented Aug 16, 2019

Klaven commented Aug 16, 2019

neolit123 commented Aug 16, 2019

k8s-ci-robot commented Aug 16, 2019

rosti left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti Aug 19, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Klaven commented Aug 20, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rosti left a comment

Choose a reason for hiding this comment

Klaven commented Aug 20, 2019

rosti left a comment

Choose a reason for hiding this comment

Klaven commented Aug 16, 2019 •

edited

Loading

rosti Aug 19, 2019 •

edited

Loading

Klaven commented Aug 20, 2019 •

edited

Loading