Using go1.11 breaks memcache DNS lookups

[gouthamve]

The net library in go1.11+ cannot read SRV responses from kube-dns v1.14.10 or below. This is because kube-dns prior to 1.14.11 used to send compressed SRV responses which is against the spec. See: golang/go#27546 for more details.

We use it here: https://github.com/cortexproject/cortex/blob/master/pkg/chunk/cache/memcached_client.go#L103

This means that we are right now stuck to go1.10. Upgrading cortex to go1.11+ would break memcache lookups among others. The fix for this would be to use https://github.com/miekg/dns like Prometheus does.

/cc @codesome

[codesome]

Found another place where it is used

cortex/pkg/alertmanager/peers.go

Line 65 in a888416

_, addrs, err := net.LookupSRV(s.service, "tcp", s.hostname)

[codesome]

I am trying to use https://github.com/miekg/dns in it's place.

[bboreham]

Doesn't the query-frontend also do an SRV lookup?
here
to here

[codesome]

Looks like I need to update in 1 more place. Thanks @bboreham!

[codesome]

I don't know about this part of Golang, but would this make this file to be compiled only with go1.8?

cortex/vendor/google.golang.org/grpc/naming/go18.go

Line 1 in a888416

// +build go1.8

[bboreham]

It means 1.8 and above. See https://golang.org/pkg/go/build/

[codesome]

Yes you are right. I should have searched before asking :)

[gouthamve]

This is only a problem when running kubernetes v1.11 or below and compiling Cortex with go v1.11 and above.

[gouthamve]

Having said that bumping up kube-dns is straight forward and will fix this. We're on k8s 1.11 but with kube-dns at 1.14.13

[bboreham]

So there is a more specific problem description involving kube-dns ?
EDIT: description at top has been updated, thanks

[bwplotka]

That's funny, we found similar issue on Thanos as well: thanos-io/thanos#1015 obviously same thing with go 1.12

We consider moving to https://github.com/miekg/dns as it allows much much more flexibility. Have you considered this?
EDIT: But it's quite low-level...

[bboreham]

@bwplotka see #1206 - in the end we decided that upgrading kube-dns would be a better route than changing the software.

[bwplotka]

Awesome, thanks for this link, super useful.

[bwplotka]

FYI: We added (basic) miekg's option to Thanos: thanos-io/thanos#1016

[tomwilkie]

We decided not to fix this, as we're all on k8s >1.14.