bug: group_manager crashes with unhandled exception

[vpavlin]

Problem

wakunode2 will crash in group_manager.nim if the RPC node fails to respond or timeouts

/app/apps/wakunode2/wakunode2.nim(144) wakunode2
/app/vendor/nim-chronos/chronos/asyncloop.nim(1539) runForever
/app/vendor/nim-chronos/chronos/asyncloop.nim(268) poll
/app/vendor/nim-chronos/chronos/asyncfutures2.nim(525) cb
Error: unhandled exception: Asynchronous task [getAndHandleEvents() at group_manager.nim:305] finished with an exception "ValueError"!
Message: {"code":-32603,"message":"request failed or timed out"}
Stack trace: /app/apps/wakunode2/wakunode2.nim(144) wakunode2
/app/vendor/nim-chronos/chronos/asyncloop.nim(1539) runForever
/app/vendor/nim-chronos/chronos/asyncloop.nim(268) poll
/app/vendor/nim-chronos/chronos/asyncfutures2.nim(323) futureContinue
/app/vendor/nim-chronos/chronos/asyncmacro2.nim(305) call
[[reraised from:
/app/apps/wakunode2/wakunode2.nim(144) wakunode2
/app/vendor/nim-chronos/chronos/asyncloop.nim(1539) runForever
/app/vendor/nim-chronos/chronos/asyncloop.nim(268) poll
/app/vendor/nim-chronos/chronos/asyncfutures2.nim(323) futureContinue
/app/vendor/nim-chronos/chronos/asyncmacro2.nim(305) eth_getLogs
]]
 [FutureDefect]

It should probably either fail gracefully, or rather retry a few times before failing

It seems the culprit is probably getJsonLogs (https://github.com/status-im/nim-web3/blob/428b931e7c4f1284b4272bc2c11fca2bd70991cd/web3.nim#L535) throwing without specifying Raises and thus us not handling the exception?

Impact

critical

To reproduce

Run nwaku-compose and observe the error if your Ethereum RPC node fails or timeouts

Expected behavior

Fail gracefully or retry without failing

Screenshots/logs

If applicable, add screenshots or logs to help explain your problem.

nwaku version/commit hash

State the version of nwaku where you've encountered the bug or, if built off a specific commit, the relevant commit hash. You can check the version by running ./wakunode2 --version.

docker.io/wakuorg/nwaku:v0.21.2-rc.0

Additional context

Add any other context about the problem here.

[vpavlin]

@alrevuelta @rymnc Any thoughts on this?

[rymnc]

yeah our error handling/retrying policy is not great. agree for graceful shutdown in case the eth node stops responding

[alrevuelta]

Unsure if related: #1958

retry without failing

I would retry yeap.

[chair28980]

@rymnc are you able to pick up this issue?

[rymnc]

Hi, this can be closed by #2250