DaemonSet started before deployment can't get collector IPs

[jpkrohling]

Currently, we have daemonsets being created before services and deployments:

jaeger-operator/pkg/controller/jaeger/jaeger_controller.go

Lines 321 to 334 in 6451f06

	if err := r.applyDaemonSets(ctx, jaeger, str.DaemonSets()); err != nil {
	return jaeger, tracing.HandleError(err, span)
	}
	// seems counter intuitive to have services created *before* deployments,
	// but some resources used by deployments are created by services, such as TLS certs
	// for the oauth proxy, if one is used
	if err := r.applyServices(ctx, jaeger, str.Services()); err != nil {
	return jaeger, tracing.HandleError(err, span)
	}
	if err := r.applyDeployments(ctx, jaeger, str.Deployments()); err != nil {
	return jaeger, tracing.HandleError(err, span)
	}

Depending on the timing, the daemonset's pod might be up and running before the deployment gets even created, so that the headless service won't get a chance to get its IP for at least one backing collector. This causes the daemonset to not find any IPs upon bootstrap. The grpc-go client will not attempt to do any DNS refreshes at all (or only after 30m?), turning the agent unusable in this scenario.

This might be related to grpc/grpc-go#3353, so, it might be worth also bumping gRPC for Jaeger itself.

[jpkrohling]

cc @kevinearls