distsql: high-level retry on connection failures

[jordanlewis]

Currently, DistSQL is not resilient to connection failures after flows have been setup - the failure will simply bubble all the way up through SQL to the user.

This is generally bad - our system shouldn't start failing queries when nodes die, or in the new circuit-breaker-y world, when a connection can't be established to a particular node because of a 500ms cooldown period.

We should add a high-level retry loop, above distsql physical planning, that activates if the failure that it sees is about connection failure or breaker tripped. I think it's only valid to do so for read-only queries, as well.

[github-actions]

We have marked this issue as stale because it has been inactive for
18 months. If this issue is still relevant, removing the stale label
or adding a comment will keep it active. Otherwise, we'll close it in
5 days to keep the issue queue tidy. Thank you for your contribution
to CockroachDB!

[jordanlewis]

I think this is the wrong way to do things.