Update OSFileWatcher to support symlinks #6172

timmysilv · 2020-09-15T19:35:06Z

This PR modifies the wrapper around fsnotify's file Watcher to watch symlinks in a custom manner such that if the symlink itself triggers an event and the target has been updated since the last event, the event should be called.

What this PR does / why we need it:

This PR is needed because fsnotify auto-resolves symlinks internally. fsnotify/fsnotify#199 provides a little bit of context on their internals/decisions. The problem that results from this is that any watcher on a file which is a kubernetes secret will not detect updates, due to the symlink nature of them. Here's a glimpse of an example directory:

bash-5.0$ pwd
/run/secrets/webhook-certs
bash-5.0$ ls -l
total 0
lrwxrwxrwx    1 root     root            13 Sep  8 20:00 ca.crt -> ..data/ca.crt
lrwxrwxrwx    1 root     root            14 Sep  8 20:00 tls.crt -> ..data/tls.crt
lrwxrwxrwx    1 root     root            14 Sep  8 20:00 tls.key -> ..data/tls.key
bash-5.0$ ls -l ..data
lrwxrwxrwx    1 root     root            31 Sep  8 20:36 ..data -> ..2020_09_08_20_36_52.438041338

This change will allow for libraries that auto-update secrets (such as cert-manager for webhook server certs in the above example) to work without interference.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)

How Has This Been Tested?

Added a unit test case that creates a symlink to file A, creates file B, deletes and replaces the symlink to point to file B instead, and ensures that the watcher raises the expected events.
Ran locally using the kind deploy manifest, deleted and recreated the ingress-validation-admission secret and confirmed that the validation webhook server reloaded the TLS files as expected.

Checklist:

My change requires a change to the documentation.
I have updated the documentation accordingly.
I've read the CONTRIBUTION guide
I have added tests to cover my changes.
All new and existing tests passed.

k8s-ci-robot · 2020-09-15T19:35:10Z

Thanks for your pull request. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please follow instructions at https://git.k8s.io/community/CLA.md#the-contributor-license-agreement to sign the CLA.

It may take a couple minutes for the CLA signature to be fully registered; after that, please reply here with a new comment and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please sign in with your organization's credentials at https://identity.linuxfoundation.org/projects/cncf to be authorized.
If you have done the above and are still having issues with the CLA being reported as unsigned, please log a ticket with the Linux Foundation Helpdesk: https://support.linuxfoundation.org/
Should you encounter any issues with the Linux Foundation Helpdesk, send a message to the backup e-mail support address at: login-issues@jira.linuxfoundation.org

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

k8s-ci-robot · 2020-09-15T19:35:15Z

Hi @timmysilv. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

aledbf · 2020-09-15T20:11:59Z

/ok-to-test

aledbf · 2020-09-15T20:25:54Z

/retest

ElvinEfendi · 2020-09-17T01:38:19Z

The new test being introduced passes without the patch:

> ingress-nginx (master)$ ./build/run-in-docker.sh go test -v ./internal/watch/...
go: downloading github.com/fsnotify/fsnotify v1.4.9
go: downloading golang.org/x/sys v0.0.0-20200803210538-64077c9b5642
go: downloading k8s.io/klog/v2 v2.2.0
go: downloading github.com/pkg/errors v0.9.1
go: downloading github.com/go-logr/logr v0.2.0
=== RUN   TestFileWatcher
--- PASS: TestFileWatcher (0.51s)
=== RUN   TestFileWatcherWithSymlink
--- PASS: TestFileWatcherWithSymlink (0.03s)
PASS
ok  	k8s.io/ingress-nginx/internal/watch	(cached)

Can you fix that first? The test should simulate the bug this PR is fixing.

timmysilv · 2020-09-17T16:12:01Z

Fixed the test to drill a little deeper, the same way k8s does with secrets. Thanks for catching that - maybe I should start writing tests before writing patches 🤔

ElvinEfendi · 2020-09-17T20:05:03Z

internal/watch/file_watcher_test.go

+
+	if targetName != target1.Name() {
+		t.Fatalf("expected symlink to point to %v, not %v", target1.Name(), targetName)
+	}


I'd move this assertion right below line 96. By reading the code I thought this assertion somehow depends on what is happening between line 97 and 113, but seems like they are unrelated.

I put it here because I wanted to wait until everything was set up (including the fileWatcher itself) to assert this, but I'm fine to move it to line 97. I should hope the creation of the watcher doesn't do anything weird :)

ElvinEfendi · 2020-09-17T20:17:11Z

internal/watch/file_watcher.go

+					} else if finfo.Mode()&os.ModeSymlink != 0 {
+						if currentRealFile, err := filepath.EvalSymlinks(f.file); err == nil &&
+							currentRealFile != realFile {
+							f.onEvent()


Do you not have to condition this on strings.HasSuffix(event.Name, file)? Why?

the FileWatcher watches the entire directory (that's not new behaviour). Using the example in the PR description, the watcher will watch $dir/ca.crt, which is a symlink to $dir/..data/ca.crt, which is a symlink to $dir/..<dated_dir>/ca.crt. The CREATE event in $dir is not for a file called ca.crt (which would have the correct suffix), but for a directory called ..data which symlinks to the dated directory. However, the watcher should detect that the top-level symlink ($dir/ca.crt) now points to a different file, and trigger an event.
tl;dr when any file is created in the directory being watched, this FileWatcher should confirm if the symlink it was told to monitor has been updated somewhere within the directory (maybe deeper than 1 level).

More details: this comment shows the internal order of events.

ElvinEfendi · 2020-09-18T01:24:55Z

/test pull-ingress-nginx-test

ElvinEfendi · 2020-09-18T01:37:10Z

/lgtm

@aledbf can you also take a look?

ElvinEfendi · 2020-09-18T01:37:46Z

/hold until the second review

aledbf · 2020-09-18T02:06:08Z

/lgtm
/hold cancel

k8s-ci-robot · 2020-09-18T02:06:24Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: aledbf, ElvinEfendi, timmysilv

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [ElvinEfendi,aledbf]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

This is cherry-picked from PR: kubernetes#6172 , to backport the feature to auto-reload certs on k8s secrets changes to 0.35.0 , upstream merged thin in 0.40.0.

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 15, 2020

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 15, 2020

k8s-ci-robot requested review from aledbf and ElvinEfendi September 15, 2020 19:35

timmysilv force-pushed the master branch from 7ed7c26 to 6479892 Compare September 17, 2020 15:23

ElvinEfendi reviewed Sep 17, 2020

View reviewed changes

inspect symlinks to auto-reload k8s secrets

d280a34

timmysilv force-pushed the master branch from 6479892 to d280a34 Compare September 17, 2020 20:13

ElvinEfendi reviewed Sep 17, 2020

View reviewed changes

k8s-ci-robot assigned ElvinEfendi Sep 18, 2020

k8s-ci-robot added lgtm "Looks good to me", indicates that a PR is ready to be merged. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels Sep 18, 2020

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 18, 2020

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Sep 18, 2020

k8s-ci-robot assigned aledbf Sep 18, 2020

k8s-ci-robot merged commit a6994be into kubernetes:master Sep 18, 2020

This was referenced Oct 29, 2020

AdmissionController is not reloading certificate on changes if they are symlinks as k8s mounted Secrets #6386

Closed

ensure admission controller certs are reloaded also if they are symli… #6387

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update OSFileWatcher to support symlinks #6172

Update OSFileWatcher to support symlinks #6172

timmysilv commented Sep 15, 2020

k8s-ci-robot commented Sep 15, 2020

k8s-ci-robot commented Sep 15, 2020

aledbf commented Sep 15, 2020

aledbf commented Sep 15, 2020

ElvinEfendi commented Sep 17, 2020

timmysilv commented Sep 17, 2020

ElvinEfendi Sep 17, 2020

timmysilv Sep 17, 2020

ElvinEfendi Sep 17, 2020 •

edited

Loading

timmysilv Sep 17, 2020

timmysilv Sep 17, 2020

ElvinEfendi commented Sep 18, 2020

ElvinEfendi commented Sep 18, 2020

ElvinEfendi commented Sep 18, 2020

aledbf commented Sep 18, 2020

k8s-ci-robot commented Sep 18, 2020

Update OSFileWatcher to support symlinks #6172

Update OSFileWatcher to support symlinks #6172

Conversation

timmysilv commented Sep 15, 2020

What this PR does / why we need it:

Types of changes

How Has This Been Tested?

Checklist:

k8s-ci-robot commented Sep 15, 2020

k8s-ci-robot commented Sep 15, 2020

aledbf commented Sep 15, 2020

aledbf commented Sep 15, 2020

ElvinEfendi commented Sep 17, 2020

timmysilv commented Sep 17, 2020

ElvinEfendi Sep 17, 2020

Choose a reason for hiding this comment

timmysilv Sep 17, 2020

Choose a reason for hiding this comment

ElvinEfendi Sep 17, 2020 • edited Loading

Choose a reason for hiding this comment

timmysilv Sep 17, 2020

Choose a reason for hiding this comment

timmysilv Sep 17, 2020

Choose a reason for hiding this comment

ElvinEfendi commented Sep 18, 2020

ElvinEfendi commented Sep 18, 2020

ElvinEfendi commented Sep 18, 2020

aledbf commented Sep 18, 2020

k8s-ci-robot commented Sep 18, 2020

ElvinEfendi Sep 17, 2020 •

edited

Loading