k8s-dns-dnsmasq-nanny-amd64 broken 1.15.2 => 1.15.3

[hnoefer]

In Kubernetes I cannot start the container k8s-dns-dnsmasq-nanny-amd64.

I0605 09:33:12.503001 1 main.go:78] opts: {{/usr/sbin/dnsmasq [-k --cache-size=1000 --no-negcache --dns-loop-detect --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053] true} /etc/k8s/dns/dnsmasq-nanny 10000000000 127.0.0.1:10053}

I0605 09:33:12.503186 1 nanny.go:124] Starting dnsmasq [-k --cache-size=1000 --no-negcache --dns-loop-detect --log-facility=- --server=/cluster.local/127.0.0.1#10053 --server=/in-addr.arpa/127.0.0.1#10053 --server=/ip6.arpa/127.0.0.1#10053]

F0605 09:33:12.503691 1 nanny.go:211] Could not start dnsmasq with initial configuration: fork/exec /usr/sbin/dnsmasq: no such file or directory

1.15.2 was working fine.
root@debian:# docker pull k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.2
1.15.2: Pulling from k8s-dns-dnsmasq-nanny-amd64
c87736221ed0: Already exists
16368d27a0b1: Pull complete
21e2dcd607c0: Pull complete
4ab58a281344: Pull complete
1fc24f2358b5: Pull complete
Digest: sha256:f1530507260114cae5b5398deabbf2bda5cf6f24795e5496126fd11cb90bb117
Status: Downloaded newer image for k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.2
root@debian:# docker run --rm -it --entrypoint /usr/sbin/dnsmasq k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.2

error started with 1.15.3
root@debian:# docker pull k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.3
1.15.3: Pulling from k8s-dns-dnsmasq-nanny-amd64
39fafc05754f: Already exists
e230a3b4cf15: Pull complete
eb492f24cd8d: Pull complete
86e2cc41b518: Pull complete
51494bd77efb: Pull complete
Digest: sha256:4507fcda5713a16a5bd6b043af1a1c58e32826af3ca3729df5832186c0cdd1a1
Status: Downloaded newer image for k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.3
root@debian:# docker run --rm -it --entrypoint /usr/sbin/dnsmasq k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.3
standard_init_linux.go:207: exec user process caused "no such file or directory"

Binary is at the correct path:
root@debian:~# docker run --rm -it --entrypoint "/bin/sh" k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.3

ls -la /usr/sbin/dnsmasq

-rwxr-xr-x 1 root root 359016 May 30 18:23 /usr/sbin/dnsmasq

dependency missing?

wget https://github.com/kubernetes/dns/archive/1.15.3.zip
export VERSION=1.15.3
make build
make containers

root@debian:~# docker run --rm -it staging-k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.3
F0605 13:06:29.967038 1 nanny.go:211] Could not start dnsmasq with initial configuration: fork/exec /usr/sbin/dnsmasq: no such file or directory
goroutine 1 [running]:
k8s.io/dns/vendor/github.com/golang/glog.stacks(0xc0002a9700, 0xc0001141c0, 0x97, 0xdb)
/go/src/k8s.io/dns/vendor/github.com/golang/glog/glog.go:769 +0xd4
k8s.io/dns/vendor/github.com/golang/glog.(*loggingT).output(0x1d776e0, 0xc000000003, 0xc0000f62c0, 0x1d0d7b9, 0x8, 0xd3, 0x0)
/go/src/k8s.io/dns/vendor/github.com/golang/glog/glog.go:720 +0x329
k8s.io/dns/vendor/github.com/golang/glog.(*loggingT).printf(0x1d776e0, 0x3, 0x12b0ccc, 0x36, 0xc0005b7e28, 0x1, 0x1)
/go/src/k8s.io/dns/vendor/github.com/golang/glog/glog.go:655 +0x14b
k8s.io/dns/vendor/github.com/golang/glog.Fatalf(0x12b0ccc, 0x36, 0xc0005b7e28, 0x1, 0x1)
/go/src/k8s.io/dns/vendor/github.com/golang/glog/glog.go:1148 +0x67
k8s.io/dns/pkg/dnsmasq.RunNanny(0x13cd7e0, 0xc0002a97a0, 0x128cc67, 0x11, 0x1d98128, 0x0, 0x0, 0x0, 0x128ae81, 0xf)
/go/src/k8s.io/dns/pkg/dnsmasq/nanny.go:211 +0x295
main.main()
/go/src/k8s.io/dns/cmd/dnsmasq-nanny/main.go:82 +0x240

[MrHohn]

cc @prameshj

[MrHohn]

I can reproduce this on my test cluster as well with k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.3.

F0605 20:12:00.479362       1 nanny.go:211] Could not start dnsmasq with initial configuration: fork/exec /usr/sbin/dnsmasq: no such file or directory

[MrHohn]

Confirmed k8s.gcr.io/k8s-dns-dnsmasq-nanny-amd64:1.15.2 is working fine. Odd thing that there isn't any change WRT dnsmasq between 1.15.2 and 1.15.3.

[MrHohn]

Diff is here: 1eeaece...e7a6ef7

[bowei]

dnsmasq was built against Alpine (MUSL). It must be rebuilt against glibc.

$ ldd /tmp/dnsmasq
	linux-vdso.so.1 (0x00007ffd2cf7d000)
	libc.musl-x86_64.so.1 => not found

[MrHohn]

Humm, I thought we have moved away from Alpine since 1.15.2 (ref https://github.com/kubernetes/dns/pull/294/files, https://github.com/kubernetes/dns/pull/296/files).

[prameshj]

Yes, i am wondering the same. Perhaps it was something in the build environment. Trying out a new image right now.

[prameshj]

Seems like the 1.15.2 dnsmasq-nanny image was using alpine though we switched the base to debian. So the dnsmasq image was running fine. Things are failing in 1.15.3 since nanny image is now correctly using debian base. I will send a fix soon.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[prameshj]

/close
This has been fixed.

[k8s-ci-robot]

@prameshj: Closing this issue.

In response to this:

/close
This has been fixed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.