Worker is up but not rewarded

[kalabukdima]

Worker with PeerID 12D3KooWHjRUmQVoS6555zeKqDi1nBDsVYbR3aFdezwqFZroBzy7 has stable uptime but doesn't get any rewards.
Find out why that happens, how to fix it, and how to avoid such problems in the future.

[kalabukdima]

Rewards are not granted because the worker didn't have any logs collected from it since April 15.
It's the date of dropping the Panthalassa support, and a lot of workers stopped sending logs at this time.

[kalabukdima]

Debugging the reason the logs were not collected gave the following:

• The last collected seq_no for this worker is 87870.
• Logs collector receives logs from it with seq_no near 86320 (but with fresh timestamp) and doesn't store them because seq_no is lower than expected.
• The worker has some logs in its logs.db but their seq_no are also less than 87870.
• The worker receives the LogsCollected message just fine, but the DB is already initialized so it just drops all the logs and retains its broken last_seq_no
  https://github.com/subsquid/worker-rs/blob/d59a1d95d47cc7181bc89bcf8f46b73b38927a02/src/logs_storage/mod.rs#L102-L104

The suggestion to manually remove the logs.db file and restart the worker helped, and now it's sending logs and receiving rewards just fine.

[kalabukdima]

I suppose that the local data was somehow corrupted during the migration to Tethys. Judging by numerous errors in the logs, it can be the case that multiple workers are sharing the same data dir.

Actions to prevent such issues in the future:

• Monitor that all the active workers get the rewards. @Igorgro please take a look
• Assert that the last collected log is not from the future on the worker side.
• Check that another worker is not already using the same data dir during the start up.

Files sent by the operator:

• 2024-05-18_logs.debug.txt
• 2024-05-17_logs.txt
• logs.db with the following contents:

  next_seq_no = 86434
  query_logs.seq_no = 86433
  query_logs.log_msg = 412D3KooWCwAsNpbB9umJ6vD5ApotdobCEDxsYZgibPxp93MjeHKg412D3KooWHjRUmQVoS6555zeKqDi1nBDsVYbR3aFdezwqFZroBzy7$862bb3f7-aede-4b37-ad7b-81ae62c08bc9czM6Ly9ldGhlcmV1bS1tYWlubmV0LTE{"fromBlock": 7299130, "toBlock": 7310017, "traces": [{"callTo": ["0xffdbe829d5a9afcd50482c23eb387b9050f1f825"], "subtraces": true}], "fields": {"block": {"timestamp": true}, "trace": {"type": true, "callFrom": true, "callTo": true, "callInput": true, "callResultOutput": true}}}
  

Logs extract from the logs collector filtered by peer id:

• logs-collector.trace.log