Firehose outputs blocks with very old LIBNUM when lighthouse/beacon not in sync

[sduchesneau]

Description

In some conditions, the geth instance is running with 'optimistic' chain head (?) and will produce blocks with a bad libnum

• Example from Ropsten

  "libNum": "12660716",
  "headNum": "12747578",
  "headID": "b869c6f610588ed5618f24cb1e9d8beff6451d85bfec48950700f509092046e5",
  "headTime": "2022-08-08T21:45:12Z"
}````

* Logs that seem relevant
  * lighthouse
```Sep  7 00:53:54 ropsten-sfdm37 lighthouse[1950008]: Sep 07 00:53:54.002 WARN Head is optimistic                      execution_block_hash: 0x1ce8b5b3423323ec86bba8d77f591f8d17bf3cf9394a59081b4be8967ef37493, info: chain not fully verified, block and attestation production disabled until execution engine syncs, service: slot_notifier
Sep  7 00:53:54 ropsten-sfdm37 lighthouse[1950008]: Sep 07 00:53:54.002 INFO Synced                                  slot: 715769, block: 0x3019…4c78, epoch: 22367, finalized_epoch: 22365, finalized_root: 0x2185…4cd4, exec_hash: 0x1ce8…7493 (unverified), peers: 18, service: slot_notifier
Sep  7 00:53:54 ropsten-sfdm37 lighthouse[1950008]: Sep 07 00:53:54.002 WARN Syncing deposit contract block cache    est_blocks_remaining: 933, service: slot_notifier```
  * beacon
  ```7358 Sep  7 00:53:01 ropsten-sfdm37 sf[1953540]: 2022-09-07T00:53:01.293Z WARN (reader.geth) beacon chain gapped                      head=12,941,410 newHead=12,941,419```

This will cause lots of problem in firehose (merger, etc.)
especially if that new libnum is BELOW previously known libnum (we have merged files up to block 499, but then, geth with firehose produces block 550 with libnum=300 ... this breaks the merger...)





# System
Geth version:  `1.10.23-fh2`
OS & Version: Linux

[sduchesneau]

@maoueh @matthewdarwin ^

[sduchesneau]

The libnum (12660716) was the last finalized block when the mindreader was restarted:

Sep  7 00:52:42 ropsten-sfdm37 sf[1953540]: 2022-09-07T00:52:42.664Z INFO (reader.geth) loaded most recent local finalized block number=12,660,716 hash=44d789..f54956 td=50,353,550,149,371,329 age=1mo1w6d

More logs:

Sep  7 00:57:04 ropsten-sfdm37 sf[1953540]: 2022-09-07T00:57:04.080Z INFO (reader.geth) syncing beacon headers                   downloaded=251,392 left=12,690,092 eta=3h35m13.172s

Sep  7 00:57:33 ropsten-sfdm37 sf[1953540]: 2022-09-07T00:57:33.079Z WARN (reader.geth) beacon client online, but no consensus updates received in a while. Please fix your beacon client to follow the chain!

[sduchesneau]

THIS MAY NOT WORK
we are still investigating..

Proposed Workaround when syncing a chain with firehose:

1. stopping the merger...
2. deleting all the one-block-files that are stuck with the old LIBNUM in their name
3. restarting a reader (geth instance) from previous backup with --no-discover until the lighthouse/beacon are synced
4. restarting them without the --no-discover so they sync and produce blocks again
5. restart the merger

[sduchesneau]

example one-block-files produced from a snapshot of geth ropsten:

    16274 2022-09-07 17:23:53 0012660714-77570d9dc40c727f-7ec09df70f47ee5c-12660714-id.dbin.zst
    28921 2022-09-07 17:23:53 0012660715-49cc5918ab51c71d-77570d9dc40c727f-12660715-id.dbin.zst
    18763 2022-09-07 17:23:53 0012660716-0d8edd90b5f54956-49cc5918ab51c71d-12660716-id.dbin.zst
    18865 2022-09-07 17:23:53 0012660717-802db026cba41e25-0d8edd90b5f54956-12660716-id.dbin.zst

the libnum stays 12660716 "forever" passed that block (which happens to be the last Finalized block according to that geth instance when it started)

[sduchesneau]

should be fixed with last PR, relased as geth-v1.10.23-fh2.2