Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add coredns proposal #1100

Merged
merged 5 commits into from
Oct 23, 2017
Merged

Add coredns proposal #1100

Changes from 1 commit
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
6d13333
Add coredns proposal for https://github.com/kubernetes/features/issue…
johnbelamaric Sep 19, 2017
57c5c9e
Add some configuration details, note about wildcard queries
johnbelamaric Sep 20, 2017
f8c9841
Add more operational details
johnbelamaric Sep 20, 2017
566693e
Add section on Reverse DNS
johnbelamaric Sep 21, 2017
7d3bbce
Add performance numbers and a note about upgrade
johnbelamaric Oct 20, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 72 additions & 0 deletions contributors/design-proposals/network/coredns.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
# Add CoreDNS for DNS-based Service Discovery

Status: Pending

Version: Alpha

Implementation Owner: @johnbelamaric

## Motivation

CoreDNS is another CNCF project and is the successor to SkyDNS, which kube-dns is based on. It is a flexible, extensible
authoritative DNS server and directly integrates with the Kubernetes API. It can serve as cluster DNS,
complying with the [dns spec](https://github.com/kubernetes/dns/blob/master/docs/specification.md).

CoreDNS has fewer moving parts than kube-dns, since it is a single executable and single process. It is written in Go so
it is memory-safe (kube-dns includes dnsmasq which is not). It supports a number of use cases that kube-dns does not
(see below). As a general-purpose authoritative DNS server it has a lot of functionality that kube-dns could not reasonably
be expected to add. See, for example, the [intro](https://docs.google.com/presentation/d/1v6Coq1JRlqZ8rQ6bv0Tg0usSictmnN9U80g8WKxiOjQ/edit#slide=id.g249092e088_0_181) or [coredns.io](https://coredns.io) or the [CNCF webinar](https://youtu.be/dz9S7R8r5gw).

## Proposal

The proposed solution is to enable the selection of CoreDNS as an alternate to Kube-DNS during cluster deployment, with the
intent to make it the default in the future.

## User Experience

### Use Cases

* Standard DNS-based service discovery
* Federation records
* Stub domain support
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the stub domain support interoperable with the current kube-dns configmap?

* Adding custom DNS entries
* Making an alias for an external name [#39792](https://github.com/kubernetes/kubernetes/issues/39792)
* Dynamically adding services to another domain, without running another server [#55](https://github.com/kubernetes/dns/issues/55)
* Adding an arbitrary entry inside the cluster domain (for example TXT entries [#38](https://github.com/kubernetes/dns/issues/38))
* Verified pod DNS entries (ensure pod exists in specified namespace)
* Experimental server-side search path to address latency issues [#33554](https://github.com/kubernetes/kubernetes/issues/33554)
* Limit PTR replies to the cluster CIDR [#125](https://github.com/kubernetes/dns/issues/125)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are PTR records for services implemented? I saw coredns/coredns#1074

Copy link
Member Author

@johnbelamaric johnbelamaric Sep 20, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they are. You have to configure the reverse zone to make it work. That means knowing the service CIDR and configuring that ahead of time (would love to have kubernetes/kubernetes#25533 implemented).

Since reverse DNS zones are on classful boundaries, if you have a classless CIDR for your service CIDR (say, a /12), then you have to widen that to the containing classful network. That leaves a subset of that network open to the spoofing described in kubernetes/dns#125, and so the issue you reference is to fix that.

We still have that issue (PTR hijacking) with CoreDNS for IPs in the pod CIDRs, but we are in a position to fix it if the operator is willing to enable pods verified.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We generally recommend production user disable pod IP DNS for this reason as well. I would prefer to let pod DNS get deprecated out since it was intentionally a stop gap.

Thanks for the clarity on the reverse CIDR. I think one part of this that would be good would be to include a sample Corefile in this proposal that implements conformance with the Kube DNS spec. To someone new to the core file syntax but deeply familiar with the kube dns spec I had to dig through the code to know what I had to set up. A core file would go a long way to assisting in understanding the implications.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I have added that. Let me know if I need any more examples (e.g., federation).

* Serve DNS for selected namespaces [#132](https://github.com/kubernetes/dns/issues/132)
* Serve DNS based on a label selector

By default, the user experience would be unchanged. For more advanced uses, existing users would need to modify the
ConfigMap that contains the CoreDNS configuration file.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discuss operational characteristics of coredns against existing kube-dns.

Have the existing e2e tests been run against a cluster configured with coredns?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I will add some to the proposal regarding the operational characteristics.

Existing e2e tests have not been run yet, see coredns/coredns#993 - we plan on this for sure.

## Implementation

Each distribution project (kubeadm, minikube, kubespray, and others) will implement CoreDNS as an optional
add-on as appropriate for that project.

### Client/Server Backwards/Forwards compatibility

No changes to other components are needed.

The method for configuring the DNS server will change. Thus, in cases where users have customized
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a good reason to break compatibility here?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since CoreDNS isn't solely a K8s product, it's configuration of these options is more general purpose and so falls outside of our Kubernetes plugin. Internally that would mean that the Kubernetes plugin has to reconfigure the overall server. I don't think that could be done in a deterministic way. That is, we don't know everything a user might configure, so we can't easily modify the overall configuration based on the additional ConfigMaps.

What we can do, is provide a tool to convert the existing kube-dns configuration on a cluster into a CoreDNS configuration. So, it would be a one-time operation when moving from kube-dns to CoreDNS.

(I think this answers your other config-related questions above too, let me know if not).

We may want to keep the service name kube-dns. If we do that, then the switch can be done without loss of service in a production environment. You just deploy CoreDNS, then modify the kube-dns service to point to it. Otherwise you have to delete that service and recreate one with the same IP. In the interim you will have DNS lookup failures.

Copy link
Member

@bowei bowei Sep 21, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The ideal would be to be able to seamlessly drop the CoreDNS Pod in as the replacement for the existing kube-dns. By CoreDNS pod, I mean the CoreDNS + the appropriate configuration for a Kubernetes cluster.

CoreDNS itself (the generic piece) does not have to change, but the ideal scenario would be to have the pod setup in such a way (e.g. via a very small shim sidecar) that preserve compatibility with the original kube-dns, especially wrt the parameters it consumes from the ConfigMap. This way we can switch to using coredns transparently with minimal user disruption. We would probably still need to maintain at least a two release deprecation of the existing ConfigMap and provide upgrade/downgrade automation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm somewhat skeptical of this approach - I understand the reasoning, but I also don't know that I consider it a requirement for a replacement implementation to emulate the previous config. I think that is something that might be desirable, but I also don't think I consider it a requirement for someone to use CoreDNS in a conformant way.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At the very least we need to support existing users of kube-dns to migrate to CoreDNS if CoreDNS is to replace kube-dns as the default provider. I know for sure that we have lots of users that utilize the stub domains via kube-system:kube-dns config map.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, we would need to have an upgrade/downgrade path for existing users. Otherwise changing coredns to be the default will break people's clusters...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alternatively, we could clean up kube-system:kube-dns to be more generic, say be portable across different DNS providers. E.g. make it part of the API for k8s cluster, rather than viewing it as a entirely bound to the existing implementation.

Copy link
Member

@liggitt liggitt Sep 24, 2017 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd expect the DNS API to be defined in terms of the DNS schema that is served, and the pod/service/endpoint inputs, not the config of the DNS server itself

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liggitt I can see this specific config being out of scope for k8s -- but we still need to support our users across versions in a transparent way.

If we make coredns the default "kube-dns" in say 1.10, it can't be the case that the users configs are not carried through. The proposal is replace kube-dns.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, those deployments that auto installed kube-dns would be responsible for adapting/shimming/converting the kube-dns config to cause the equivalent behavior via coredns if they wanted to switch on upgrade

the DNS configuration, they will need to modify their configuration if they move to CoreDNS.
For example, if users have configured stub domains, they would need to modify that configuration.

When serving SRV requests for headless services, some responses are different from kube-dns, though still within
the specification (see [#975](https://github.com/coredns/coredns/issues/975)). In summary, these are:

* kube-dns uses endpoint names that have an opaque identifier. CoreDNS instead uses the pod IP with dashes.
* kube-dns returns a bogus SRV record with port = 0 when no SRV prefix is present in the query.
coredns returns all SRV record for the service (see also [#140](https://github.com/kubernetes/dns/issues/140))

Additionally, federation may return records in a slightly different manner (see [#1034](https://github.com/coredns/coredns/issues/1034)),
though this may be changed prior to completing this proposal.

## Alternatives considered

Maintain existing kube-dns, add functionality to meet the currently unmet use cases above, and fix underlying issues.
Ensuring the use of memory-safe code would require replacing dnsmasq with another (memory-safe) caching DNS server,
or implementing caching within kube-dns.