Panic: preselected index must be valid in peer_set

[teor2345]

Version

zebrad main branch, 19 October 2020.

Platform

Linux ... 5.4.68 #1-NixOS SMP Sat Sep 26 16:03:16 UTC 2020 x86_64 GNU/Linux

Description

While syncing testnet blocks, zebrad panicked with a "preselected index must be valid" error in peer_set::set, inside the span zebrad::components::sync::obtain_tips.

I tried this:

Syncing testnet from genesis.

I expected to see this happen:

A successful sync.

Instead, this happened:

A tip exhaustion cycle, followed by a zebra-network panic.

Error

preselected index must be valid

Metadata

key	value
version	3.0.0-alpha.0
location	/home/dev/zebra-revhex/zebra-network/src/peer_set/set.rs:381:22

SpanTrace

SpanTrace:
   0: zebrad::components::sync::obtain_tips
             at zebrad/src/components/sync.rs:314
   1: zebrad::components::sync::sync
             at zebrad/src/components/sync.rs:169

[teor2345]

@hdevalence I think you might understand the invariants in zebra-network?

[teor2345]

@hdevalence do you think this ticket is fixed now?

[hdevalence]

No, the error message indicates that there's an invariant being violated inside the peer set's selection logic. I gave it a cursory look when this ticket was filed and didn't see anything obvious, but I haven't seen the error recently, so I think we should deprioritize fixing it until later unless it comes up again.

[mpguerra]

removing from Alpha release milestone based on the above, we can review this if it happens again

[teor2345]

I've just seen this error on the main branch again

[teor2345]

The panic happened on a Testnet sync, but there's nothing stopping it from happening on Mainnet. (The sync and network code doesn't distinguish between Mainnet and Testnet.)

I suspect that we haven't seen this error recently because it's been hidden by other errors that we've just fixed.

Nov 23 21:12:22.110  INFO sync: zebrad::components::sync: waiting to restart sync timeout=45s                                                                                                  
Nov 23 21:13:07.111  INFO sync: zebrad::components::sync: starting sync, obtaining new tips                                                                                                    
Nov 23 21:13:07.111  INFO sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(551200) min_locator_height=551101 locators=[Height(551200), Height(551199), Height
(551198), Height(551196), Height(551192), Height(551184), Height(551168), Height(551136), Height(551101)]                                                                                      
Nov 23 21:13:23.363  INFO sync: zebrad::components::sync: extending tips tips.len=1 in_flight=140 lookahead_limit=2000                                                                         
Nov 23 21:13:24.783  INFO sync: zebrad::components::sync: exhausted prospective tip set                                                                                                        
Nov 23 21:13:24.783  INFO sync: zebrad::components::sync: waiting to restart sync timeout=45s                                                                                                  
Nov 23 21:14:09.784  INFO sync: zebrad::components::sync: starting sync, obtaining new tips                                                                                                    
Nov 23 21:14:09.784  INFO sync:obtain_tips:state: zebra_state::util: created block locator tip_height=Height(551200) min_locator_height=551101 locators=[Height(551200), Height(551199), Height
(551198), Height(551196), Height(551192), Height(551184), Height(551168), Height(551136), Height(551101)]                                                                                      
Nov 23 21:14:37.679  INFO sync: zebrad::components::sync: extending tips tips.len=1 in_flight=295 lookahead_limit=2000                                                                         
The application panicked (crashed).                                                                                                                                                            
Message:  preselected index must be valid
Location: /home/dev/zebra/zebra-network/src/peer_set/set.rs:381
  
  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ SPANTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   
   0: zebrad::components::sync::downloads::download_and_verify with hash=block::Hash("000005b0cf28b945393efed7d194f1ef48c1451bcdf2798faf596312e00a2908")
      at zebrad/src/components/sync/downloads.rs:113
   1: zebrad::components::sync::extend_tips
      at zebrad/src/components/sync.rs:381
   2: zebrad::components::sync::sync
      at zebrad/src/components/sync.rs:150