Several vshard reconfiguration steps broke rebalancing

[yngvar-antonsson]

While trying to fix #1569 we've found that the test is flaky as a result of a bug.
After several vshard reconfiguration steps rebalancer breaks with some buckets stuck in state sending.

Repro:

local rebalancer = g.SA1.instance_uuid < g.SB1.instance_uuid and
        g.SA1 or g.SB1
local resp = g.cluster.main_server:graphql({query = [[
        mutation ($uuid: String!){
            edit_replicaset(
                uuid: $uuid
                weight: 2
            )
        }
    ]], variables = {uuid = g.SA1.replicaset_uuid}})

    t.assert_not(resp.errors)
    local resp = g.cluster.main_server:graphql({query = [[
        mutation ($uuid: String!){
            edit_replicaset(
                uuid: $uuid
                weight: 1
            )
        }
    ]], variables = {uuid = g.SA1.replicaset_uuid}})
    t.assert_not(resp.errors)

    local resp = g.cluster.main_server:graphql({query = [[
        mutation ($uuid: String!){
            edit_replicaset(
                uuid: $uuid
                weight: 2
            )
        }
    ]], variables = {uuid = g.SA1.replicaset_uuid}})
    t.assert_not(resp.errors)

    helpers.retrying({}, function()
        rebalancer:call('vshard.storage.rebalancer_wakeup')
        t.assert_equals(g.SA1:call('vshard.storage.buckets_count'), 2000)
        t.assert_equals(g.SB1:call('vshard.storage.buckets_count'), 1000)
    end)

Error:

./test/hotreload/vshard_test.lua:194: expected: 2000, actual: 1500

[R-omk]

Is it still actual?

[Gerold103]

Not blocked anymore.

[yngvar-antonsson]

Still reproducible.

[filonenko-mikhail]

#1898