feat: lpmessage: Correct message sending

[magik6k]

Related Issues

Proposed Changes

This PR improves the lotus-provider message sender to handle failed message pushes correctly. Without this patch any failed message push (due to push error, network error between lotus and lotus-provider, etc..) would cause us to never send a message at the failed nonce, causing nonce gaps - essentially making it impossible to send more messages from that address without manual intervention to fix that issue.

Here we add a mechanism where:

• The message sender (lpmessage) now creates a Send task - harmonydb will make sure that this task will get executed
  • Sending can't fail in the middle now - harmony will retry
• There is a cluster-wide locking mechanism making sure that we don't ever try to do more than one send in parallel per address
  • This resembles what we do in raft-clustered lotus where all sends are sequenced by the leader - but here we don't need a leader
  • Locks don't automatically expire, they can only be dropped after we fully finish related send tasks - if a task runner fails in the middle, harmony will retry the task, and the task will be able to reclaim the send lock and finish the process.

Additional Info

Checklist

Before you mark the PR ready for review, please make sure that:

• Commits have a clear commit message.
• PR title is in the form of of <PR type>: <area>: <change being made>
  • example: fix: mempool: Introduce a cache for valid signatures
  • PR type: fix, feat, build, chore, ci, docs, perf, refactor, revert, style, test
  • area, e.g. api, chain, state, market, mempool, multisig, networking, paych, proving, sealing, wallet, deps
• If the PR affects users (e.g., new feature, bug fix, system requirements change), update the CHANGELOG.md and add details to the UNRELEASED section.
• New features have usage guidelines and / or documentation updates in
  • Lotus Documentation
  • Discussion Tutorials
• Tests exist for new functionality or change in behavior
• CI is green