Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new lotus-shed command for backfillling actor events #11088

Merged
merged 2 commits into from
Jul 25, 2023

Conversation

fridrik01
Copy link
Contributor

@fridrik01 fridrik01 commented Jul 19, 2023

Related issue: #10807
Replaces previous PR: #10939

Context

When lotus is configured to enable indexing of actor events (when EnableEthRPC=true and DisableHistoricFilterAPI=false) then Lotus will observe for new tipset and store all the actor events in a sqlite table stored at sqlite/events.db.

We however, don't have any way to backfill these actor events. This PR therefore implements this backfilling using a new lotus-shed` command.

Note that this implementation does not compute state (it only looks up the receipts so they need to exist on chain).

Test plan

Backfill active node that is already storing events

As expected, backfilling should not have any side effects on a node that is already saving actor events.

lotus-shed indexes backfill-events --epochs=1000
2023-07-20T10:45:13.018Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [0] backfilling actor events epoch:3051449, eventsAffected:0, entriesAffected:0
2023-07-20T10:45:13.029Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [1] backfilling actor events epoch:3051448, eventsAffected:0, entriesAffected:0
2023-07-20T10:45:13.045Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [2] backfilling actor events epoch:3051447, eventsAffected:0, entriesAffected:0
...
2023-07-20T10:46:14.769Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [996] backfilling actor events epoch:3050448, eventsAffected:0, entriesAffected:0
2023-07-20T10:46:14.782Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [997] backfilling actor events epoch:3050447, eventsAffected:0, entriesAffected:0
2023-07-20T10:46:14.814Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [998] backfilling actor events epoch:3050446, eventsAffected:0, entriesAffected:0
2023-07-20T10:46:14.825Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [999] backfilling actor events epoch:3050445, eventsAffected:0, entriesAffected:0
2023-07-20T10:46:14.843Z        INFO    lotus-shed      lotus-shed/indexes.go:318       backfilling events complete, eventsAffected:0, entriesAffected:0

Backfill a node that had never stored any events

lotus-shed indexes backfill-events  --epochs=100000
2023-07-20T10:29:37.535Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [0] backfilling actor events epoch:3051307, eventsAffected:0, entriesAffected:0
2023-07-20T10:29:37.548Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [1] backfilling actor events epoch:3051306, eventsAffected:7, entriesAffected:25
2023-07-20T10:29:37.559Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [2] backfilling actor events epoch:3051305, eventsAffected:11, entriesAffected:39
2023-07-20T10:29:37.565Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [3] backfilling actor events epoch:3051304, eventsAffected:11, entriesAffected:39
...
2023-07-20T10:32:31.736Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [7068] backfilling actor events epoch:3044289, eventsAffected:15491, entriesAffected:55988
2023-07-20T10:32:31.744Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [7069] backfilling actor events epoch:3044288, eventsAffected:15491, entriesAffected:55988
2023-07-20T10:32:31.777Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [7070] backfilling actor events epoch:3044287, eventsAffected:15491, entriesAffected:55988
2023-07-20T10:32:31.800Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [7071] backfilling actor events epoch:3044286, eventsAffected:15491, entriesAffected:55988
2023-07-20T10:32:31.832Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [7072] backfilling actor events epoch:3044285, eventsAffected:15491, entriesAffected:55988
2023-07-20T10:32:31.857Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [7073] backfilling actor events epoch:3044284, eventsAffected:15492, entriesAffected:55992
2023-07-20T10:32:31.884Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [7074] backfilling actor events epoch:3044283, eventsAffected:15492, entriesAffected:55992
2023-07-20T10:32:31.924Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [7075] backfilling actor events epoch:3044282, eventsAffected:15494, entriesAffected:56000
2023-07-20T10:32:31.966Z        INFO    lotus-shed      lotus-shed/indexes.go:168       [7076] backfilling actor events epoch:3044281, eventsAffected:15496, entriesAffected:56008

Verify backfill results

I compared the events.db that was generated from scratch via backfill to another events.db which was generated from by an active lotus node. I attached the second database as events2 and used the following sql to verify there are no missing events:

SELECT * FROM (
  SELECT e.height, e.tipset_key, e.emitter_addr, e.event_index,  e.message_index, e.reverted, ee.indexed, ee.flags, ee.key, ee.codec, ee.value 
  FROM event e, event_entry ee
  WHERE ee.event_id = e.id AND e.height > 3044282 AND e.height < 3050445
  EXCEPT
  SELECT e.height, e.tipset_key, e.emitter_addr, e.event_index,  e.message_index, e.reverted, ee.indexed, ee.flags, ee.key, ee.codec, ee.value 
  FROM events2.event e, events2.event_entry ee
  WHERE ee.event_id = e.id AND e.height > 3044282 AND e.height < 3050445
) WHERE reverted = 0

Result: 0 rows returned in 259ms

@fridrik01 fridrik01 force-pushed the backfill-events-cli branch 2 times, most recently from 6fe2258 to ed3a508 Compare July 20, 2023 10:42
@fridrik01 fridrik01 changed the title wip Add new lotus-shed command for backfillling actor events Jul 20, 2023
@fridrik01 fridrik01 marked this pull request as ready for review July 20, 2023 11:06
@fridrik01 fridrik01 requested a review from a team as a code owner July 20, 2023 11:07
Copy link
Member

@Stebalien Stebalien left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks fine as-is, but I have a few performance nits you may want to look at.

(although I would wrap in a transaction)

Comment on lines +70 to +81
currTs, err := api.ChainHead(ctx)
if err != nil {
return err
}
if cctx.IsSet("from") {
// we need to fetch the tipset after the epoch being specified since we will need to advance currTs
currTs, err = api.ChainGetTipSetByHeight(ctx, abi.ChainEpoch(cctx.Int("from")+1), currTs.Key())
if err != nil {
return err
}
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: no need to call ChainHead if we're just going to call ChainGetTipSetByHeight.


// select the highest event id that exists in database, or null if none exists
var entryID sql.NullInt64
err = stmtSelectEvent.QueryRow(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Consider "insert where not exists" instead of checking then inserting.

cmd/lotus-shed/indexes.go Show resolved Hide resolved
cmd/lotus-shed/indexes.go Outdated Show resolved Hide resolved
@fridrik01 fridrik01 merged commit 3f93b86 into master Jul 25, 2023
@fridrik01 fridrik01 deleted the backfill-events-cli branch July 25, 2023 10:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants