RialtoParachain <> Millau bridge is broken

[svyatonik]

Our test deployments are alive after 2w break and there's an issue now: messages from Millau to RialtoParachain are delivered with huge lags. E.g. when we push a bunch of messages to the queue, I can see that the number of generated messages is~1.28k and number of delivered messages stays 490 for almost 30m.

[svyatonik]

Seems like it is related to #2030. I could see following in relay logs:

[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Received proof for nonces in range 1..=0 from Millau::MessagesDelivery
[Millau_to_RialtoParachain_MessageLane_00000000] 2023-04-24 07:41:04 +00 DEBUG bridge Asking Millau::MessagesDelivery to prove nonces in range 1..=0 at block HeaderId(636, 0xabd9d66f1840321dd6d6d8b5f82e52d2cb32cd40a3f8f91d5aa998b7c519451fc2489640dd40f73eebe9674893a2d6c17fb0755c6058529584e0ed92935d196f)

UPD1: selecting nonces 1..=0 seems valid, because:

[RialtoParachain_to_Millau_MessageLane_00000000] 2023-04-24 07:35:34 +00 DEBUG bridge Received best nonces from Millau::MessagesDelivery: TargetClientNonces { latest_nonce: 1265, nonces_data: DeliveryRaceTargetNoncesData { confirmed_nonce: 241, unrewarded_relayers: UnrewardedRelayersState { unrewarded_relayer_entries: 3, messages_in_oldest_entry: 449, total_messages: 1024, last_delivered_nonce: 1265 } } }

But:

• why we're constantly asking for proof and not submitting it?
• why we have so many unconfirmed rewards?

[svyatonik]

Should be fine now.