-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
txmgr: Add closed
flag and behavior to SimpleTXManager
#8694
Conversation
WalkthroughWalkthroughThe recent update to the transaction manager codebase revolves around handling the state when the manager is closed. An Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good first simple step to a sane txmgr shutdown!
We should also check for closed
at the beginning of the retry.Do
loop of send
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I just realize that there might be another for
loop that we probably want to break out from on a closed txmgr, the one in publishTx
. It actually happened quite frequently in the past that we were stuck in a "tx underpriced" or "tx replacement underpriced" loop.
10a4d0f
to
4f52c00
Compare
bad76f4
to
888e466
Compare
issue fixed; rebased; canyon deployed
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM ✨
A good follow up is to check whether we call Close
on the txmgr from the batcher and proposer at the right point in time, and possibly adapt logging when returned errors are ErrClose
, since this is an expected returned error after the txmgr is closed.
closed
flag and behavior to SimpleTXManagerclosed
flag and behavior to SimpleTXManager
a1b913b
to
6a1cba6
Compare
What
Adds an internal flag
closed
to the SimpleTxManager, and uses that flag to stop new and ongoing workWhy
In situations where the BatchSubmitter is shutting down, there is ongoing work traveling through the Queue, and managed internally by the TXManager itself. This internally managed work delays shutdown, sometimes by a very long time if those transactions are being hopelessly retried.
The BatchSubmitter already
Close()
s the TXManager, but that close previously did not halt any work, it only closed connection to the backend. We would like to use this signal in order to more aggressively stop the transaction submission, allowing for the Batch Submitter to shut down more directly.How
A new
atomic.Bool
is introduced into the SimpleTxManager. OnClose()
, this value is set to true. At several points in transaction submission, this flag is checked:Send
methodsend
sendTx
publishTx
If this flag is set to true, the function returns early instead of continuing to wait or retry. In the case of the update polling in
sendTx
, waiting for confirmations takes precedence over checking the close signal, meaning transactions that should complete are awaited, while erroring or unmined transactions are abandoned.Tests
A new unit test has been included, demonstrating that in-flight work is cancelled early, and that new work is denied altogether.
An additional unit test has been included to show that transactions which are mined (but awaiting full confirmation) will hold the closing TxManager open until the work is complete.
Notes and Nuance