Submariner does not add LightHouse DNS entry in corefile section of configmap in case of RKE2 cluster.

[manojgop]

What happened:

Submariner does not add LightHouse DNS entry in configmap "corefile" section in case of RKE2 cluster. For RKE2, I see "rke2-coredns" instead of core-dns. rke2-coredns is NOT configured to forward requests for domain clusterset.local to Lighthouse CoreDNS Server in the cluster making the query. I had to edit this config file manually in "corefile" section in all clusters to make it work.

The output of kubectl -n kube-system describe configmap rke2-coredns-rke2-coredns is as following. The forward rule is present in lighthouse.server section. But that didn't work in case of RKE2.

Data
====
Corefile:
----
.:53 {
    errors
    health  {
        lameduck 5s
    }
    ready
    kubernetes   cluster.local  cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus   0.0.0.0:9153
    forward   . /etc/resolv.conf
    cache   30
    loop
    reload
    loadbalance
}
lighthouse.server:
----
clusterset.local:53 {
    forward . 10.43.180.127
}

nslookup nginx.default.svc.clusterset.local returned server can't find nginx.default.svc.clusterset.local: NXDOMAIN

I had to manually edit config map and keep following section in the corefile section of the configmap

clusterset.local:53 {
    forward . 10.43.180.127
} 

Data
====
Corefile:
----
#lighthouse-start
clusterset.local:53 {
    forward . 10.43.180.127
}
#lighthouse-end
.:53 {
    errors
    health  {
        lameduck 5s
    }
    ready
    kubernetes   cluster.local  cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        fallthrough in-addr.arpa ip6.arpa
        ttl 30
    }
    prometheus   0.0.0.0:9153
    forward   . /etc/resolv.conf
    cache   30
    loop
    reload
    loadbalance
}

Adding the rules in lighthouse.server section seems to be a issue. Looks like RKE2 is expecting the rules in Corefile section

What you expected to happen:

Lighthouse DNS to work for exported services in RKE2 clusters

How to reproduce it (as minimally and precisely as possible):

Try using submariner with RKE2 cluster and export service

Anything else we need to know?: Check Slack for more details slack comments

Environment:

• Diagnose information (use subctl diagnose all):
• Gather information (use subctl gather):
• Cloud provider or hardware configuration:
• Install tools:
• Others:

[dfarrell07]

ACK, thanks for the report @manojgop. This does seem to be an issue. @vthapar can provide some details.

[vthapar]

CustomDNSCONfig was added at the time for an issue with Azure/AKS clusters where they required DNS configuration to be in a separate file and xyz.server format. That is why we use lighthouse.server. But rke2 expects it in the Corefile section itself. This will require some work, and potentially a new flag without breaking any existing AKS deployments.

[Kirubel-Fikru]

Do you happen to have any updates here? I have a similar issue where Submariner is incorrectly trying to update the non-existent dns-default ConfigMap. Where i get this error in the submariner operator pod.

Reconciler error error="error updating Openshift DNS operator: error trying to update microshift coredns configmap "openshift-dns" in namespace "openshift-dns": error updating DNS ConfigMap: error creating or updating resource: configmaps "dns-default" not found" ServiceDiscovery={"name":"service-discovery","namespace":"submariner-operator"} controller=servicediscovery-controller controllerGroup=submariner.io controllerKind=ServiceDiscovery name=service-discovery namespace=submariner-operator reconcileID=3e519719-b537-4219-9ece-4f106e15b05e.

It is attempting to interact with OpenShift-specific resources (like the openshift-dns namespace) that aren’t present in a default RKE2 environment?

[tpantelis]

This is scheduled for the 0.20 release and will be worked on based on priority. Of course we always welcome contributions.

[tpantelis]

Reconciler error error="error updating Openshift DNS operator: error trying to update microshift coredns configmap "openshift-dns" in namespace "openshift-dns": error updating DNS ConfigMap: error creating or updating resource: configmaps "dns-default" not found" ServiceDiscovery={"name":"service-discovery","namespace":"submariner-operator"} controller=servicediscovery-controller controllerGroup=submariner.io controllerKind=ServiceDiscovery name=service-discovery namespace=submariner-operator reconcileID=3e519719-b537-4219-9ece-4f106e15b05e.

It is attempting to interact with OpenShift-specific resources (like the openshift-dns namespace) that aren’t present in a default RKE2 environment?

That's a last resort b/c it didn't find the standard "coredns" ConfigMap and no custom config map was specified. @manojgop apparently did the latter but it didn't modify the "Corefile" section as explained here

[tpantelis]

So RKE2 uses the same CoreDNS ConfigMap as other providers but, for some reason, names it "rke-coredns" rather than the standard "coredns" that LH is expecting. I think a simple solution is to expand our search criteria - first search for "coredns" in the kube-system namespace as we do now and, if not found, search for a ConfigMap in the same namespace with the suffix "-coredns". This way RKE2 just works out-of-the-box w/o having to specify a custom ConfigMap. @vthapar @aswinsuryan

[Kirubel-Fikru]

I think this is a good solution. But one more question for clarification. Won't that be a problem if we have multiple ConfigMaps with the "-coredns" suffix?

[tpantelis]

I think this is a good solution. But one more question for clarification. Won't that be a problem if we have multiple ConfigMaps with the "-coredns" suffix?

Is that possible with RKE? I don't have an RKE setup - do you see multiple in your setup? We'd also look for the presence of the Corefile data entry. I'd think there would only be one of these.

Also, is the "rke-coredns" name well-known and hard-coded out-of-box or can the user change it to whatever they want? Is it possible Rancher could change the name to something completely different in the future?