Always use auth_servers when proxying an auth connection

[espadolini]

As a followup to #12862, this PR makes it so that every instance of dialing an auth through a proxy (not just the reverse tunnel's @remote-auth-server) will select one random entry from the proxy's own auth_servers configuration rather than checking the auth server heartbeats in the backend. This is preferrable as the backend info could be out of date (10 minutes for expiration plus backend expiration and cache propagation delays), and doesn't allow for any flexibility in choosing a specific load balancer or running health checks.

note: backporting this requires backporting #12862 at the same time.

[codingllama]

LGTM.

This isn't a part of Teleport that I usually inhabit, so something might escape me, but I like that the change is both simpler and makes a clearly random choice.

[LKozlowski]

LGTM.

I'm not very familiar with this part of code base, but just one thing came to my mind.

In the previous version we were checking authorization servers for connection one by one. Now, we are picking up one server randomly and if it fails then we throws and error. There's no fallback or checking the other servers from the list. Isn't it going to affect the availability?

[espadolini]

@LKozlowski

There's no fallback or checking the other servers from the list. Isn't it going to affect the availability?

See #13310 (comment), the tl;dr is that it should be quite rare and only for tsh - if it turns out to be a problem, any retry logic should be in tsh itself rather than hidden away in the proxy. We never noticed problems because the usual setup in kubernetes will notify the proxy of a failure to connect to auth really quickly, and the retry logic masked the design flaw of using the backend info over the explicit user configuration.

[espadolini]

@codingllama

makes a clearly random choice

I specialcased length 1 in the utility function specifically because in almost every configuration (and in all supported ones) there will only be a single address in auth_servers; the random choice is just there to still guarantee the correct load balancing behavior for old and/or weird setups in which proxies have a hardcoded list of more than one auth, but we don't really expect it to happen.

For a DNS load balancer setup, net.Dial itself will connect to the resolved DNS entries (if there's more than one) in order, but that's because the entries from the DNS server are supposed to be in random order.

[LKozlowski]

@LKozlowski

There's no fallback or checking the other servers from the list. Isn't it going to affect the availability?

See #13310 (comment), the tl;dr is that it should be quite rare and only for tsh - if it turns out to be a problem, any retry logic should be in tsh itself rather than hidden away in the proxy. We never noticed problems because the usual setup in kubernetes will notify the proxy of a failure to connect to auth really quickly, and the retry logic masked the design flaw of using the backend info over the explicit user configuration.

Thanks for the explanation! Makes sense now. I must have written my comment when you were writing yours as I didn't notice it before sending my feedback 😄

[espadolini]

Holding off on the v7 backport.