Faulty DNS queries in musl-based containers due to "search ." in resolv.conf

[dghubble]

Describe the bug

Could we pull in openshift/okd-machine-os#159 to FCOS? Kubernetes v1.25.0 landed and the systemd behavior that adds "search ." on nodes with FQDNs now propagates into containers, which breaks musl-based DNS resolution apparently. I can see why OpenShift reverted it.

Reproduction steps

Originally I was preparing the Typhoon Kubernetes distro for v1.25.0 release and I did some sleuthing. The full write-up for Kubernetes is kubernetes/kubernetes#112135. The original bug in Kubelet was added to fix a problem RedHat customers saw, but I imagine the side effect wasn't noticed since OpenShift has openshift/okd-machine-os#159 to mitigate it.

Unfortunately, just starting a container with Podman (and mucking with /etc/resolv.conf) doesn't seem to demonstrate the issue. I'd like to try to get a single-node reproducer for you guys. But for now it seems to just be Typhoon and OpenShift

Expected behavior

Don't put "search ." at the end of /etc/resolv.conf on hosts.

Actual behavior

https://github.com/systemd/systemd/pull/17201/files adds "search ." to /run/systemd/resolve/stub-resolv.conf, which seems to confuse any musl-based DNS resolution when this propagates into Kubernetes containers. This didn't become apparent until Kubernetes v1.25.0, when Kubelet started propagating that "search ." directly into a container's /etc/resolv.conf in kubernetes/kubernetes#109441

System details

Bare Metal

NAME="Fedora Linux"
VERSION="36.20220806.3.0 (CoreOS)"

Ignition config

As a workaround, I can apply the same unit OpenShift is using. Removing the "." fixes the musl DNS problems.

variant: fcos
version: 1.4.0
systemd:
  units:
    - name: fix-resolv-conf-search.service
      enabled: true
      contents: |
        [Unit]
        Description=Remove search . from /etc/resolv.conf
        DefaultDependencies=no
        Requires=systemd-resolved.service
        After=systemd-resolved.service
        BindsTo=systemd-resolved.service
        [Service]
        Type=oneshot
        ExecStartPre=/usr/bin/sleep 5
        ExecStart=/usr/bin/sed -i -e "s/^search .$//" /run/systemd/resolve/resolv.conf
        [Install]
        WantedBy=multi-user.target

Related

Add any other information about the problem here.

• resolved: turn off that a search domain is derived from the host's fqdn systemd/systemd#17201
• use of cgroups v2 triggers SELinux denial when loading eBPF progams #881 (comment) cc @LorbusChris
• kubelet: parseResolvConf: Handle "search ." kubernetes/kubernetes#109441 @Miciah
• musl-based DNS resolution will break on v1.25.0 in certain configurations kubernetes/kubernetes#112135

[lucab]

Thanks for the report.

"search ." on nodes with FQDNs now propagates into containers, which breaks musl-based DNS resolution apparently.

For reference, which musl version is in use there? I know there have been recent changes in the resolver logic, it would be good to confirm that we are speaking about the latest stable release (1.2.3).

Additionally I'd wait to see how the upstream kubernetes bug discussion goes, as the in-container DNS resolution is really owned by the orchestrator.

Overall I feel this may be a musl bug. If so, the best outcome would be to get a minimal reproducer and then fix the faulty behavior there.

[lucab]

I think I can see what's going wrong with musl DNS resolution. Even outside of FCOS or kubernetes, just running a plain container with the docker.io/alpine:3.16.2 image:

# apk list | grep 'musl\|busybox'
musl-1.2.3-r0 x86_64 {musl} (MIT) [installed]
musl-utils-1.2.3-r0 x86_64 {musl} (MIT BSD GPL2+) [installed]
busybox-1.35.0-r17 x86_64 {busybox} (GPL-2.0-only) [installed]
ssl_client-1.35.0-r17 x86_64 {busybox} (GPL-2.0-only) [installed]

# cat /etc/resolv.conf 
search default.svc.cluster.local .
nameserver 8.8.8.8
options ndots:5

/ # wget google.foobarxyz
[... this lookup here takes way longer than expected to complete and fail ...]
wget: bad address 'google.foobarxyz'

The interesting part is what happens under the hood.
I took a capture of the DNS flow (which I'm attaching here) and it looks like musl is failing to properly wire-encode the trailing . for the DNS query:

In the screenshot, note the 0x2e at the end of the foobarxyz DNS label.

[lucab]

An even smaller reproducer is:

# cat /etc/resolv.conf 
search . customtld
nameserver 8.8.8.8

# nslookup -type=a foorbarxyz
Server:         8.8.8.8
Address:        8.8.8.8:53

** server can't find foorbarxyz.customtld: NXDOMAIN
** server can't find foorbarxyz..: FORMERR

For some reason, the bug seems to only happen if there are multiple search domains configured.
That is, just a plain search . seems to work fine and properly gets a response back:

# cat /etc/resolv.conf 
search .
nameserver 8.8.8.8

# nslookup -type=a foorbarxyz
Server:         8.8.8.8
Address:        8.8.8.8:53

** server can't find foorbarxyz: NXDOMAIN

[dghubble]

@lucab awesome digging!

the bug seems to only happen if there are multiple search domains configured

That or having the "options ndots:N" line (as happens in k8s) seem to be key for reproducing it. Now I can see this with a Podman container, adjusting /etc/resolv.conf, and using a musl-based resolver:

cat /etc/resolv.conf
search .
nameserver 8.8.8.8
options ndots:5

wget www.google.com
wget: bad address 'www.google.com'

I'd wait to see how the upstream kubernetes bug discussion goes... Overall I feel this may be a musl bug..

Fully agreed. This seems like a musl DNS issue, surfaced on certain hosts via the systemd change, and recently propagated into Kubernetes by kubernetes/kubernetes#109441. I think the workaround in openshift/okd-machine-os#159 makes sense to be applied at the k8s distro level for now. Not at the OS level, forget my original ask.

Hopefully Kubernetes can react or eventually musl has a fix.

I was also using docker.io/alpine:3.16.{0,1,2}. Its bugged in docker.io/busybox:1.34.1-musl too (which seems to build busybox using musl-dev from alpine:3.16 https://github.com/docker-library/busybox/blob/c2e72d6cf3cc382f25d7d71ae70bb2013a517f1f/stable/musl/Dockerfile.builder)

apk list | grep 'musl\|busybox'
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.16/main: No such file or directory
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.16/community: No such file or directory
musl-1.2.3-r0 x86_64 {musl} (MIT) [installed]
musl-utils-1.2.3-r0 x86_64 {musl} (MIT BSD GPL2+) [installed]
busybox-1.35.0-r17 x86_64 {busybox} (GPL-2.0-only) [installed]
ssl_client-1.35.0-r17 x86_64 {busybox} (GPL-2.0-only) [installed]

[lucab]

Not at the OS level, forget my original ask.
Hopefully Kubernetes can react or eventually musl has a fix.

I agree with this assessment.
Kubernetes can filter the trailing . search domain (it doesn't mean much in the container context) and musl should fix its search behavior.

Were you planning to report the musl bug to their ML too?

[dghubble]

I sent an email to musl at lists.openwall.com, EDIT: visible at https://www.openwall.com/lists/musl/2022/08/31/1

[LorbusChris]

fyi @vrutkovs

[dghubble]

Looks like Kubernetes will stop propagating search ., @lucab is cherry-picking to Kubernetes v1.25.1, and musl, CoreDNS, and systemd folks have been informed. I think we could close this

Thank you @lucab for the digging to help confirm and fix!

[dustymabe]

Thank all for the discussion!