Properly handle restart edge cases between ExEx and new engine

[joshieDo]

With the previous engine, an exex might loose a chain notification, if the node crashes/stops before it has time to handle it. This means that on boot-up the exex should actually account for that since its tip will be behind the node tip.

With the new engine however, the exex will actually be ahead of the node in case of a crash or simple restart, since part of the chain will be purely in-memory

Example:

1. given a chain of blocks 1a(db) -> 2a(in-mem) -> 3a(in-mem)
2. exex processes 3a
3. node crashes/stops
4. node starts
5. node receives 2a from CL
6. node sends 2a to exex, even though its ready for 4a

This becomes more problematic if 2a -> 3a gets reorged during the crash and the exex depends solely on ChainReorged { old_chain } :

1. exex has a storage table with TxHash -> T
2. given a chain of blocks 1a(db) -> 2a(in-mem) -> 3a(in-mem)
3. exex processes 3a
4. node crashes/stops
5. chain gets reorged into 1a -> 2b -> 3b
6. node starts
7. node receives 2b from CL
8. node sends 2b to exex as a normal ChainCommitted notification. How is the exex supposed to clean-up the table on 1)?

With the roll-out of the new engine we should do:

• Document and highlight these edge cases (probably discourage relying solely on old_chain)
• Enforce some handling of these cases into the exex installation/start.

Proposal:

• persistence = 0 could mean that an exex notification is only sent after persisted. this would solve the ahead + old_chain dependency issue, but reintroduce the old engine case of potentially being behind the node, which can be solved by:
• exex passes its latest block on init/start. if behind the node (with the above suggestion), trigger backfill for missing range

Additional context

No response

[github-actions]

This issue is stale because it has been open for 21 days with no activity.