Router sends requests to dead instances

[Serpentian]

This is the critical issue and must be fixed as soon as possible. When instance is dead for some reason (e.g. tx thread is 100% loaded, while true do end on instance), vshard continues to send requests to it, which fails with TimeOut error. This happens, as connection doesn't die, when tx thread is busy (wait_connected returns true). Manually disabling the instance always fixes the problem, erroneous requests disappear.

vshard/vshard/replicaset.lua

Lines 790 to 796 in 2211be2

	while r do
	if r:is_connected() and (not prefer_replica or r ~= master) and
	replica_check_backoff(r, now) then
	return r
	end
	r = r.next_by_priority
	end

Currently router requests fail way too often, which affects users in mission critical projects. Vshard must try to minimize the number of timed out errors (and probably any other errors). We should consider the following approaches:

• Failover must change priority based on ping #483
• Allow to make several requests with hanging connection #484

As a follow up we may introduce the following ticket, but only if smb asks, since it requires user intervention and writing triggers:

• Allow to backoff replica if request fails #485

[sergepetrenko]

There's also a ticket to make net.box track connection state: tarantool/tarantool#9563

But if we choose to fix that ticket, it'll only go to new tarantool releases, leaving vshard installations on old tarantool versions still affected by this bug.

So I vote for fixing it separately in vshard