-
Notifications
You must be signed in to change notification settings - Fork 1.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add new lotus-shed command for backfillling actor events #11088
Conversation
6fe2258
to
ed3a508
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks fine as-is, but I have a few performance nits you may want to look at.
(although I would wrap in a transaction)
currTs, err := api.ChainHead(ctx) | ||
if err != nil { | ||
return err | ||
} | ||
if cctx.IsSet("from") { | ||
// we need to fetch the tipset after the epoch being specified since we will need to advance currTs | ||
currTs, err = api.ChainGetTipSetByHeight(ctx, abi.ChainEpoch(cctx.Int("from")+1), currTs.Key()) | ||
if err != nil { | ||
return err | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: no need to call ChainHead
if we're just going to call ChainGetTipSetByHeight
.
|
||
// select the highest event id that exists in database, or null if none exists | ||
var entryID sql.NullInt64 | ||
err = stmtSelectEvent.QueryRow( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider "insert where not exists" instead of checking then inserting.
ed3a508
to
dafa837
Compare
dafa837
to
7192151
Compare
Related issue: #10807
Replaces previous PR: #10939
Context
When lotus is configured to enable indexing of actor events (when
EnableEthRPC=true
andDisableHistoricFilterAPI=false
) then Lotus will observe for new tipset and store all the actor events in a sqlite table stored atsqlite/events.db
.We however, don't have any way to backfill these actor events. This PR therefore implements this backfilling using a new lotus-shed` command.
Note that this implementation does not compute state (it only looks up the receipts so they need to exist on chain).
Test plan
Backfill active node that is already storing events
As expected, backfilling should not have any side effects on a node that is already saving actor events.
Backfill a node that had never stored any events
Verify backfill results
I compared the
events.db
that was generated from scratch via backfill to anotherevents.db
which was generated from by an active lotus node. I attached the second database asevents2
and used the following sql to verify there are no missing events: