-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: sqlite: extract common init and migration utilities #12098
Conversation
77eea40
to
643f316
Compare
All checks have passed |
643f316
to
a779104
Compare
Reviewing this after landing #12080 today. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
some questions. Overall looking great. Would be nice to test this on a devnet/calibnet node once rebased on #12080.
a779104
to
891b251
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
OK, so on the use of the message index - it is being used by some large node operators, it's not just a hidden feature that nobody uses it. The change here, of switching the create+pragma order, means that if you have an existing message index, we are going to change the checkpoint size, which will impact how big the write-ahead-log will be before it tries to compact it:
There's also likely some minor performance impacts from increasing the block-size, but it depends on the exact usage and is likely a factor of how big the chunks are that we're writing. It probably makes the events db faster, but for the other 2 dbs they probably don't benefit from the larger block-size, but I doubt there'll be much cost from it. |
37b5a56
to
6bebb13
Compare
All of the versioning and migration logic is shared between the 3 sqlite databases we use the other two have the same pending-query problem as was resolved for _meta in #12090, so that's addressed here by making it common.
PRAGMA
set across all 3 - this may need to change in future but we can configure that later when we need it._meta
versioning and migration logic. Do it all in a migration loop where each migration is wrapped in a transaction, logging and error handling and the calling instance is responsible for feeding in functions that can perform a migration. Versioning and_meta
is entirely hidden from the calling instance.ethhashlookup.NewTransactionHashLookup
now takes a context as its first argument.index.NewMsgIndex
now takes a full path to its database as itspath
(notbasePath
) argument to make it consistent with the others.Needs a fresh set of eyes to make sure I haven't stumbled over something in one of these to make it inconsistent. I'll do another pass of this myself later too, before I try deploying this on my calibnet and then mainnet nodes. There should be no meaningful visible changes for a running instance from this, mostly they just get consistent
PRAGMA
s and possibly some changed logging.Also closes: #12081
Will need some rebase work after #12080 is landed so this should hold off till that's done.