GRANDPA voter error: safety invariant has been violated: SelectChain returned a finality target higher than its best block

[nicolasochem]

System: ubuntu 22.04 vm with 16gi memory and 4 vcpu, running polkadot 0.9.37

When my node was in the Kusama active validator set, it rebooted several times due to some problem with the environment, and then was stuck in a reboot loop with the following message.

Note that the reboot look is from my orchestrator. The polkadot process just stopped and was restarted by the orchestration.

This eventually fixed itself and the node is able to restart now.

 2023-02-16 23:12:49 GRANDPA voter error: safety invariant has been violated: SelectChain returned a finality target higher than its best block                                                                   │
│ 2023-02-16 23:12:49 Essential task `grandpa-voter` failed. Shutting down service.                                                                                                                                │
│ Error:                                                                                                                                                                                                           │
│    0: Other: Essential task failed.                                                                                                                                                                              │
│                                                                                                                                                                                                                  │
│   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ BACKTRACE ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━                                                                                                                               │
│                                  ⋮ 1 frame hidden ⋮                                                                                                                                                              │
│    2: polkadot::main::h44a95f96d050c4db                                                                                                                                                                          │
│       at <unknown source file>:<unknown line>                                                                                                                                                                    │
│    3: std::sys_common::backtrace::__rust_begin_short_backtrace::h39b8c3c85e5d27d8                                                                                                                                │
│       at <unknown source file>:<unknown line>                                                                                                                                                                    │
│    4: main<unknown>                                                                                                                                                                                              │
│       at <unknown source file>:<unknown line>                                                                                                                                                                    │
│    5: __libc_start_main<unknown>                                                                                                                                                                                 │
│       at <unknown source file>:<unknown line>                                                                                                                                                                    │
│    6: _start<unknown>                                                                                                                                                                                            │
│       at <unknown source file>:<unknown line>                                                                                                                                                                    │
│                                                                                                                                                                                                                  │
│ Run with COLORBT_SHOW_HIDDEN=1 environment variable to disable frame filtering.                                                                                                                                  │
│ Run with RUST_BACKTRACE=full to include source snippets.                    

[bkchr]

Ty for the report. This was already fixed by: paritytech/substrate#13364

This fix will be released together with 0.9.38. If your node is stuck you can either downgrade to 0.9.36 or compile a prelease of 0.9.38.