Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(addons/coredns.addons.k8s.io) Workaound to stop coredns crashing on 1.3.1 version #7492

Merged
merged 3 commits into from
Sep 4, 2019
Merged

fix(addons/coredns.addons.k8s.io) Workaound to stop coredns crashing on 1.3.1 version #7492

merged 3 commits into from
Sep 4, 2019

Conversation

phspagiari
Copy link
Contributor

@phspagiari phspagiari commented Aug 30, 2019

There is a known bug on CoreOS 1.3.1 wherein if the Kubernetes API shuts down while CoreDNS is connected, CoreDNS will crash.

The workaround is to create a /tmp volume See here

Logs:

E0827 00:44:53.344864       1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://172.20.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 172.20.0.1:443: connect: connection refused
E0827 00:44:53.344864       1 reflector.go:134] github.com/coredns/coredns/plugin/kubernetes/controller.go:322: Failed to list *v1.Namespace: Get https://172.20.0.1:443/api/v1/namespaces?limit=500&resourceVersion=0: dial tcp 172.20.0.1:443: connect: connection refused
log: exiting because of error: log: cannot create log: open /tmp/coredns.coredns-57f8cc697b-dnpkj.unknownuser.log.ERROR.20190827-004453.1: no such file or directory

After this patch, CoreDNS stopped crashing.

References

kubernetes/kubernetes#75414
coredns/coredns#2629
https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.14.md#known-issues
coredns/deployment#138

@k8s-ci-robot
Copy link
Contributor

Hi @phspagiari. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@k8s-ci-robot k8s-ci-robot added the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label Aug 30, 2019
@k8s-ci-robot k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Aug 30, 2019
@phspagiari phspagiari changed the title fix(addons/coredns.addons.k8s.io) Workaound to stop CoreOS crashing on 1.3.1 version fix(addons/coredns.addons.k8s.io) Workaound to stop coredns crashing on 1.3.1 version Aug 30, 2019
@mikesplain
Copy link
Contributor

/ok-to-test

@k8s-ci-robot k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Aug 30, 2019
@mikesplain
Copy link
Contributor

To make sure this gets picked up, can you bump the final number here too:

version := "1.3.0-kops.2"

Copy link
Member

@gjtempleton gjtempleton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it worth adding a note that this can be removed as soon as CoreDNS is upgraded to >=v1.4.0 as per coredns/coredns#2529 ?

@phspagiari
Copy link
Contributor Author

@mikesplain done!

@phspagiari
Copy link
Contributor Author

@gjtempleton makes sense, I just added two notes.

@@ -283,7 +283,7 @@ func (b *BootstrapChannelBuilder) buildAddons() *channelsapi.Addons {

{
key := "coredns.addons.k8s.io"
version := "1.3.0-kops.2"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just FYI, in the future you can leave -kops.0 on the end which just gives us an easy bump for kops specific changes to manifests

Copy link
Contributor Author

@phspagiari phspagiari Sep 4, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, nice to know. I will add -kops.0 in the next one. 👍

@mikesplain
Copy link
Contributor

Thanks @phspagiari!

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 4, 2019
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mikesplain, phspagiari

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 4, 2019
@k8s-ci-robot k8s-ci-robot merged commit d90b75c into kubernetes:master Sep 4, 2019
Copy link
Member

@gjtempleton gjtempleton left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, and a good catch!

@phspagiari phspagiari deleted the coredns_missing_dir_workaround branch January 15, 2020 15:12
oded7hoffman pushed a commit to spotinst/kubernetes-kops that referenced this pull request Jan 23, 2023
…pick-of-#7492-upstream-release-1.14

Automated cherry pick of kubernetes#7492: fix(addons/coredns.addons.k8s.io) Workaround to stop
oded7hoffman pushed a commit to spotinst/kubernetes-kops that referenced this pull request Jan 23, 2023
…pick-of-#7492-upstream-release-1.14

Automated cherry pick of kubernetes#7492: fix(addons/coredns.addons.k8s.io) Workaround to stop
oded7hoffman pushed a commit to spotinst/kubernetes-kops that referenced this pull request Jan 23, 2023
…pick-of-#7492-upstream-release-1.14

Automated cherry pick of kubernetes#7492: fix(addons/coredns.addons.k8s.io) Workaround to stop
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. ok-to-test Indicates a non-member PR verified by an org member that is safe to test. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants