Out of order block announcements

[rahulksnv]

This behavior is observed when peers are undergoing reorg quickly. The announcements appear reordered, or in random order.

Example:

// Peers moving from 0xb296...ac0d1 -> 0x109e...0f09, block 0x109e...0f09 is downloaded
1623:2023-10-24T20:57:26.306524Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWMeWf1EGbjppDoEjwMXYhTZcQsVZEwDhPfc6ueuJBpyNX: AnnouncementSummary { block_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, number: 10525, parent_hash: 0x82b3471dc5f3dc8646b03af2719b1e3a1117b231c29bb0f362aa9f8e470cfd7d, state: Some(Best) }. Local best: true.
1675:2023-10-24T20:57:26.738876Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWMeWf1EGbjppDoEjwMXYhTZcQsVZEwDhPfc6ueuJBpyNX: AnnouncementSummary { block_hash: 0x109eb50e9dc4415a55e7f43bd8aeda7758b8469b8c8fdb6d8bd3ac14f7910f09, number: 10525, parent_hash: 0x82b3471dc5f3dc8646b03af2719b1e3a1117b231c29bb0f362aa9f8e470cfd7d, state: Some(Best) }. Local best: true.
1631:2023-10-24T20:57:26.545744Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWGU5VUqPyBUzg9kFKz7YGHNcYfupfCVJuHoaocVzMLN4B: AnnouncementSummary { block_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, number: 10525, parent_hash: 0x82b3471dc5f3dc8646b03af2719b1e3a1117b231c29bb0f362aa9f8e470cfd7d, state: Some(Best) }. Local best: true.
1680:2023-10-24T20:57:26.765149Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWGU5VUqPyBUzg9kFKz7YGHNcYfupfCVJuHoaocVzMLN4B: AnnouncementSummary { block_hash: 0x109eb50e9dc4415a55e7f43bd8aeda7758b8469b8c8fdb6d8bd3ac14f7910f09, number: 10525, parent_hash: 0x82b3471dc5f3dc8646b03af2719b1e3a1117b231c29bb0f362aa9f8e470cfd7d, state: Some(Best) }. Local best: true.
1632:2023-10-24T20:57:26.545884Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWFroqEkrSuW3XNqrHM5e3n6fdTMM43vKxXSAkBzkd8Bo9: AnnouncementSummary { block_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, number: 10525, parent_hash: 0x82b3471dc5f3dc8646b03af2719b1e3a1117b231c29bb0f362aa9f8e470cfd7d, state: Some(Best) }. Local best: true.
1677:2023-10-24T20:57:26.763354Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWFroqEkrSuW3XNqrHM5e3n6fdTMM43vKxXSAkBzkd8Bo9: AnnouncementSummary { block_hash: 0x109eb50e9dc4415a55e7f43bd8aeda7758b8469b8c8fdb6d8bd3ac14f7910f09, number: 10525, parent_hash: 0x82b3471dc5f3dc8646b03af2719b1e3a1117b231c29bb0f362aa9f8e470cfd7d, state: Some(Best) }. Local best: true.
1634:2023-10-24T20:57:26.546081Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWLdPnDksKX43hEgV4N9EeJzm6x8wPWkHCvRaFBZvj4GBg: AnnouncementSummary { block_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, number: 10525, parent_hash: 0x82b3471dc5f3dc8646b03af2719b1e3a1117b231c29bb0f362aa9f8e470cfd7d, state: Some(Best) }. Local best: true.
1681:2023-10-24T20:57:26.765176Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWLdPnDksKX43hEgV4N9EeJzm6x8wPWkHCvRaFBZvj4GBg: AnnouncementSummary { block_hash: 0x109eb50e9dc4415a55e7f43bd8aeda7758b8469b8c8fdb6d8bd3ac14f7910f09, number: 10525, parent_hash: 0x82b3471dc5f3dc8646b03af2719b1e3a1117b231c29bb0f362aa9f8e470cfd7d, state: Some(Best) }. Local best: true.


....
// Next block is against old branch 0xb296..ac0d1 as the parent, and gets downloaded
2061:2023-10-24T20:57:43.625977Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWMeWf1EGbjppDoEjwMXYhTZcQsVZEwDhPfc6ueuJBpyNX: AnnouncementSummary { block_hash: 0xe3f87f4d24a067c4b5c8856dc2cee0430f8007e87c55128f727e8fe8f393eec2, number: 10526, parent_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, state: Some(Best) }. Local best: true.
2064:2023-10-24T20:57:43.641413Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWGU5VUqPyBUzg9kFKz7YGHNcYfupfCVJuHoaocVzMLN4B: AnnouncementSummary { block_hash: 0xe3f87f4d24a067c4b5c8856dc2cee0430f8007e87c55128f727e8fe8f393eec2, number: 10526, parent_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, state: Some(Best) }. Local best: true.
2067:2023-10-24T20:57:43.641876Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWLdPnDksKX43hEgV4N9EeJzm6x8wPWkHCvRaFBZvj4GBg: AnnouncementSummary { block_hash: 0xe3f87f4d24a067c4b5c8856dc2cee0430f8007e87c55128f727e8fe8f393eec2, number: 10526, parent_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, state: Some(Best) }. Local best: true.
2070:2023-10-24T20:57:43.642883Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWFroqEkrSuW3XNqrHM5e3n6fdTMM43vKxXSAkBzkd8Bo9: AnnouncementSummary { block_hash: 0xe3f87f4d24a067c4b5c8856dc2cee0430f8007e87c55128f727e8fe8f393eec2, number: 10526, parent_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, state: Some(Best) }. Local best: true.
2073:2023-10-24T20:57:43.668640Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWJ216EP67meQU8fGJcYk1o4XcgUhyJMfDygaEjJstSQhJ: AnnouncementSummary { block_hash: 0xe3f87f4d24a067c4b5c8856dc2cee0430f8007e87c55128f727e8fe8f393eec2, number: 10526, parent_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, state: Some(Best) }. Local best: true.
2076:2023-10-24T20:57:43.681850Z TRACE tokio-runtime-worker sync: [Consensus] Block announcement validated successfully: from 12D3KooWLCmb61rkJafr7KVw8ba8X5Ko7vrqMV9vn8CoXHFdciTr: AnnouncementSummary { block_hash: 0xe3f87f4d24a067c4b5c8856dc2cee0430f8007e87c55128f727e8fe8f393eec2, number: 10526, parent_hash: 0xb29604c53f170c8db73e6764e2a295ab2c593702855ef98f84b044c0750ac0d1, state: Some(Best) }. Local best: true.

This results in the next block being downloaded against the old branch, and issues like discussed here: #493 (comment)

Guess there should not be any expectation wrt the the announcements showing up in any particular order from the network. But wanted to understand what could be causing this.

So blocks are announced when the import notification is received. I was wondering if the recent changes to verify blocks in parallel could cause this kind of behavior, when reorg happens in rapid succession (like in this case)?

[bkchr]

I was wondering if the recent changes to verify blocks in parallel could cause this kind of behavior,

We don't support this.