"no children to pick from" errors in v1.71.0 when using client-side load-balancing

[KnutZuidema]

What version of gRPC are you using?

v1.71.0

What version of Go are you using (go version)?

1.23.5

What operating system (Linux, Windows, …) and version?

Linux

What did you do?

updated from v1.70.0 to v1.71.0

What did you expect to see?

no functional change

What did you see instead?

gRPC calls started returning these errors:

rpc error: code = Unavailable desc = no children to pick from"

Further information

I am assuming that the error is connected to recent changes in client-side load-balancing, particularly this change.

I am using the round-robin load-balancing policy, using a service config like this one:

{
  "loadBalancingPolicy": "round_robin"
}

[arjan-bal]

Hi @KnutZuidema, the error message "no children to pick from" is due to #7966. It means that the resolver produced no endpoints for the specified target hostname. Before the change, the same situation would return the error message "produced zero addresses". Are you seeing new failures after updating or only a change in the error message?

[arjan-bal]

If you're seeing new failures after upgrading, can you please turn on gRPC debug logs and share the logs?

[KnutZuidema]

@arjan-bal the errors are new, there were none before the upgrade (i.e. no "produced zero addresses")

I'll try to get the debug logs for this soon.

Another supporting piece of information would potentially be that I am using kuberesolver for name resolution, but I am not sure whether that would be at all connected. However, I've been using kuberesolver before the update to the new version, and there was no update to kuberesolver itself either.

[arjan-bal]

I think I found the issue. kuberesolver v3 is using the deprecated cc.NewAddress API to update the resolver addresses. The master branch is using the new UpdateState API. Within gRPC Go, the address to endpoint conversion logic is present only for the new API:

grpc-go/resolver_wrapper.go

Lines 136 to 143 in f49c747

	if s.Endpoints == nil {
	s.Endpoints = make([]resolver.Endpoint, 0, len(s.Addresses))
	for _, a := range s.Addresses {
	ep := resolver.Endpoint{Addresses: []resolver.Address{a}, Attributes: a.BalancerAttributes}
	ep.Addresses[0].BalancerAttributes = nil
	s.Endpoints = append(s.Endpoints, ep)
	}
	}

The changes in #7966 made roundrobin use the resolver.State.Endpoints field instead of the resolver.State.Addresses field. We need to have the same address to endpoint conversion in NewAddress to handle users of the deprecated API.

[arjan-bal]

The fix is merged in master. A patch release will be made for 1.71 soon.