docs: finishing chain-indexer-overview-for-operators.md #12600

BigLep · 2024-10-14T23:33:09Z

This is being used to resolve the unresolved items in #12450 since that PR is unwieldy at this point.

It is part of the larger effort in #12453

Checklist

Before you mark the PR ready for review, please make sure that:

Commits have a clear commit message.
PR title conforms with contribution conventions
Update CHANGELOG.md or signal that this change does not need it per contribution conventions
New features have usage guidelines and / or documentation updates in
- Lotus Documentation
- Discussion Tutorials
Tests exist for new functionality or change in behavior
CI is green

This is being used to resolve the unresolved items in #12450 since that PR is unwieldly at this point.

…#12450

rvagg · 2024-10-16T00:12:12Z

From a chat with @jennijuju just now: we should call out the benefit of the message index that you get with chainindexer - this benefit exists even pre-fevm, StateSearchMsg and its various dependents have a shortcut to find the message with chainindex. It's worth pointing this out somewhere, that this benefit exists independently of events or eth APIs.

BigLep · 2024-10-16T05:27:26Z

From a chat with @jennijuju just now: we should call out the benefit of the message index that you get with chainindexer - this benefit exists even pre-fevm, StateSearchMsg and its various dependents have a shortcut to find the message with chainindex. It's worth pointing this out somewhere, that this benefit exists independently of events or eth APIs.

@rvagg : ACK - I made an update in #12453 (comment) . Can you please review? I am thinking for that to be the source of truth that we point to for the benefits of this project.

chain/index/chain-indexing-overview-for-rpc-providers.MD

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

BigLep · 2024-10-16T05:37:49Z

@rvagg @aarshkshah1992 : it could probably use one more read-through by me in its rendered form, but I am out of time tonight and wanted to get your feedback while there is still some overlap. No rush for review from me. Thanks for your time/help.

CHANGELOG.md

chain/index/chain-indexing-overview-for-rpc-providers.MD

Co-authored-by: Rod Vagg <rod@vagg.org>

github-actions

Please update the PR title to match https://github.com/filecoin-project/lotus/blob/master/CONTRIBUTING.md#pr-title-conventions

documentation/en/chain-indexer-overview-for-operators.md

Co-authored-by: Rod Vagg <rod@vagg.org>

PR title now matches the required format.

BigLep · 2024-10-17T18:34:10Z

This has all my currently intended changes plus @rvagg's "regular checks" addition.

CHANGELOG.md

documentation/en/chain-indexer-overview-for-operators.md

rvagg

just a couple more suggestions that come after further consideration of user personas
@aarshkshah1992 it would be really good for you to sanity check this text too

Co-authored-by: Rod Vagg <rod@vagg.org>

documentation/en/chain-indexer-overview-for-operators.md

aarshkshah1992 · 2024-10-21T08:48:12Z

CHANGELOG.md


 ## ☢️ Upgrade Warnings ☢️

 - Minimum go-version been updated to v1.22.7 ([filecoin-project/lotus#12459](https://github.com/filecoin-project/lotus/pull/12459))

 ## New features
+- New ChainIndexer subsystem to index Filecoin chain state such as tipsets, messages, events and ETH transactions for accurate and faster RPC responses. The `ChainIndexer` replaces the existing `MsgIndex`, `EthTxHashLookup` and `EventIndex` implementations in Lotus, which [suffer from a multitude of known problems](https://github.com/filecoin-project/lotus/issues/12293).  If you are an RPC provider or a node operator who uses or exposes Ethereum and/or events APIs, please refer to the [ChainIndexer documentation for operators](./documentation/en/chain-indexer-overview-for-operators.md) for information on how to enable, configure and use the new Indexer.  While there is no automated data migration and one can upgrade and downgrade without backups, there are manual steps that need to be taken to backfill data when upgrading to this Lotus version, or downgrading to the previous version without ChainIndexer. Please be aware that that this feature removes some options in the Lotus configuration file, if these have been set, Lotus will report an error when starting. See the documentation for more information.
+- Add `EthGetBlockReceipts` RPC method to retrieve transaction receipts for a specified block. This method allows users to obtain Ethereum format receipts of all transactions included in a given tipset as specified by its Ethereum block equivalent. ([filecoin-project/lotus#12478](https://github.com/filecoin-project/lotus/pull/12478))


This should go away after a rebase.

@aarshkshah1992 : I'm confused. I synced from the feature branch, and this change still seems relevant to land in the feature branch compared to what is there now: https://github.com/filecoin-project/lotus/blob/feat/msg-eth-tx-index/CHANGELOG.md

It looks like #12421 (comment) is awaiting this change to be merged.

documentation/en/chain-indexer-overview-for-operators.md

aarshkshah1992

@BigLep @rvagg

Have reviewed this and this looks great ! Just some comments. Please feel free to merge after addressing those.

BigLep · 2024-10-21T13:41:33Z

@BigLep @rvagg

Have reviewed this and this looks great ! Just some comments. Please feel free to merge after addressing those.

I can handle almost of these later today when I free up.
@aarshkshah1992 : I am not self-sufficient yet to do #12600 (comment) . If this is something you want me to merge today, then I'll need help on that one.

BigLep · 2024-10-21T16:47:11Z

@aarshkshah1992 @rvagg : I have incorporated feedback the best I can and responded to unresolved comments. I think you guys can see what is still unresolved and make the call on merging. I don't think I have anything extra of value to add at this point.

documentation/en/chain-indexer-overview-for-operators.md

Co-authored-by: Rod Vagg <rod@vagg.org>

aarshkshah1992 · 2024-10-22T02:59:09Z

Thanks @BigLep !

@jennijuju

* chain index complete for msgs and txns * dont need observer changes for now * changes * fix tests * fix tests * use th right context * index empty tipsets correctly * implement automated backfilling * add event indexing and remove all old indices * fix test * revert deployment test changes * revert test changes and better error handling for eth tx index lookups * fix sql statments naming convention * address review for Index GC * more changes as per review * changes as per review * fix config * mark events as reverted during reconciliation * better reconciliation; pens down and code complete; also reconcile events * fix tests * improve config and docs * improve docs and error handling * improve read logic * improve docs * better logging and handle ennable event storage * improve logs and index init proc * better logging * fix bugs based on calibnet testing * create sqliite Indices * gc should be based on epochs * fix event query * foreign keys should be enabled on the DB * reverted tipsets should be removed as part of GC * release read lock * make it easy to backfill an empty index using reconciliation * better docs for reconciliation * fix conflicts with master * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * fix go mod * fix formatting * revert config changes * address changes in observer * remove top level chainindex package * changes as per review * changes as per review * changes as per review * handle index with reverted tipsets during reconciliation * changes as per review * fix type of max reconcile epoch * changes to reconciliation as per review * log ipld error * better logging of progress * disable chain indexer hydrate from snapshot based on config * always populate index * make config easy to reason about * fix config * fix messaging * revert config changes * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * changes as per review * make error messages homogenous * fix indentation * changes as per review * feat: recompute tipset to generate missing events if event indexing is enabled (#12463) * auto repair events * make jen * fix leaky abstraction * better docs for gc retention epoch * imrpove DB handling (#12485) * fix conflict * fix lite node config for indexer * exclude reverted events from eth get logs if client queries by epoch * Simply addressing for event lookups in the index. simply addressing for event lookups * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * fix tests * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * feat: migration("re-indexing"), backfilling and diasgnostics tooling for the `ChainIndexer` (#12450) * fix conflicts with chain indexer * feat: chain indexer todos [skip changelog] (#12462) * feat: finish todos of validation api * feat: add indexed data verification with chain store * feat: address comments and finish TODO * fix: build issue * address comments * fix: ci issue * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * changes to Index Validation API based on Rodds first review * build chain indexer API * improve error handling * feat: lotus-shed tooling for chain indexer (#12474) * feat: add lotus-shed command for backfilling chain indexer * feat: add lotus-shed command for inspecting the chain indexer * feat: use single lotus-shed command to inspect and backfill * fix: remove the unused queries * small changes * add change log * backfilling improvements and fixes * finish chain index validation and backfill tooling * user documentation for the * validate from epoch * Apply suggestions from code review Suggestions from Steve's read of the user doc. Co-authored-by: Steve Loeppky <biglep@filoz.org> * changes to user doc as per review * Apply suggestions from code review Co-authored-by: Steve Loeppky <biglep@filoz.org> * changes to user doc as per review * Apply suggestions from code review Co-authored-by: Steve Loeppky <biglep@filoz.org> * changes as per review * feat: add event entries count in validation API (#12506) * feat: add event entry count in validation API * address comments * use sqllite defaults (#12504) * Apply suggestions from code review Co-authored-by: Steve Loeppky <biglep@filoz.org> * write chain index to a different dir * Apply suggestions from code review Co-authored-by: Steve Loeppky <biglep@filoz.org> * fix conflicts * UX improvements to backfilling * feat: tests for the chain indexer (#12521) * ddl tests * tests for the chain indexer * finish unit tests for chain indexer * fix formatting * cleanup reverted tipsets to avoid db bloat * fix logging * test for filter by address * test gc cascade delete * fix db locked error during backfilling * fix var name * increase db locked timeout * fix db locked issue * reduce db lock timeout * no lock in gc * reconcile does not need lock * improved error handling * Update chain-indexing-overview-for-rpc-providers.md Doc updates based on @jennijuju feedack. * Update chain-indexing-overview-for-rpc-providers.MD Fixes after reviewing 33c1ca1 * better metrics for backfilling * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <rod@vagg.org> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <rod@vagg.org> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <rod@vagg.org> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <rod@vagg.org> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <rod@vagg.org> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <rod@vagg.org> * Update chain/index/chain-indexing-overview-for-rpc-providers.MD Co-authored-by: Rod Vagg <rod@vagg.org> * tests for changes to event addressing * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * changes as per review -> round 1 * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * log tipset key cid * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * fix docs * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * fix tests * fix tests * make jen * fix conflicts --------- Co-authored-by: Aryan Tikarya <aryan.tikarya@dojima.network> Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Steve Loeppky <biglep@filoz.org> * fix lint * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * remove reverted flag from RPC * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * fix testing of events and dummy chain store * remove lotus shed commands for old Indices * change type of event counts to uint64 * only recompute events if theyre not found * short-circuit empty events path for older tipsets * chain indexer must be enabled if ETH RPC is enabled * change name of message_id column to id in tipset_message table * only expose SetRecomputeTipSetStateFunc * dont block on head indexing for reading messages * document why we're only checking for missing events for a single tipset * document when we query for reverted events * simplify event collection * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * fix test * change event_id to id in the event table * change head indexed timeout * remove deprecated config options * fail ETH RPC calls if ChainIndexer is disabled * fix docs * remove the tipset key cid func from lotus shed * address review comments * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * chore(events): remove unnecessary DisableRealTimeFilterAPI (#12610) * feat(cli): add --quiet to chainindex validate-backfill + cleanups (#12611) * fix tests * Apply suggestions from code review Co-authored-by: Rod Vagg <rod@vagg.org> * error type for disabled chainindexer * fix(chainindex): recompute tipset when we find no receipts * fix(chainindexer): backfilling should halt when chain state data is missing and not backfill parents (#12619) * fix backfilling UX * Update chain/index/api.go Co-authored-by: Rod Vagg <rod@vagg.org> * address review --------- Co-authored-by: Rod Vagg <rod@vagg.org> * reduce log noise * make jen * make jen * docs: finishing chain-indexer-overview-for-operators.md (#12600) * Followup to PR #12450 for doc updates This is being used to resolve the unresolved items in #12450 since that PR is unwieldly at this point. * Incorporated some items and added TODOs based on unresolved items from #12450 * Incorporating more feedback * Pointing to issue to learn about benefits * Formatting fixes * Apply most of the suggestions from @rvagg code review Co-authored-by: Rod Vagg <rod@vagg.org> * Incorporating feedback from #12600 (comment) * Addressing #12600 (comment) and more * Moved chain-indexer docs to documentation Renamed Added ToC We can move to lotus-docs later * Update documentation/en/chain-indexer-overview-for-operators.md Co-authored-by: Rod Vagg <rod@vagg.org> * Update documentation/en/chain-indexer-overview-for-operators.md Co-authored-by: Rod Vagg <rod@vagg.org> * Added upgrade path when importing chain state from a snapshot. * Typo fixes * Update documentation/en/chain-indexer-overview-for-operators.md Co-authored-by: Rod Vagg <rod@vagg.org> * chore(doc): "regular checks" section for chainindexer docs (#12612) * Apply suggestions from @rvagg code review Co-authored-by: Rod Vagg <rod@vagg.org> * Incorporating @aarshkshah1992 feedback * Update documentation/en/chain-indexer-overview-for-operators.md Co-authored-by: Rod Vagg <rod@vagg.org> --------- Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Aarsh Shah <aarshkshah1992@gmail.com> * remove go mod replace * remove unnecessary changes from CHANGELOG * fix test * compare events AMT root (#12632) * fix(chainindex): retry transaction if database connection is lost (#12657) * retry database lost connection * log context cancellation * address review * fix gateway itest: no chainindexer for lite nodes * fix changelog --------- Co-authored-by: Rod Vagg <rod@vagg.org> Co-authored-by: Aryan Tikarya <aryan.tikarya@dojima.network> Co-authored-by: Steve Loeppky <biglep@filoz.org>

Followup to PR #12450 for doc updates

6dc518f

This is being used to resolve the unresolved items in #12450 since that PR is unwieldly at this point.

BigLep changed the base branch from master to feat/msg-eth-tx-index October 14, 2024 23:33

BigLep self-assigned this Oct 14, 2024

BigLep changed the title ~~Followup to PR #12450 for doc updates~~ [WIP] Followup to PR #12450 for doc updates Oct 14, 2024

BigLep mentioned this pull request Oct 14, 2024

feat: migration("re-indexing"), backfilling and diasgnostics tooling for the ChainIndexer #12450

Merged

1 task

Incorporated some items and added TODOs based on unresolved items from …

8ede162

…#12450

BigLep mentioned this pull request Oct 15, 2024

A new ChainIndexer that subsumes that existing MsgIndex, EventIndex and TransactionIndex #12453

Open

Incorporating more feedback

d0cc462

Pointing to issue to learn about benefits

3a4d759

BigLep commented Oct 16, 2024

View reviewed changes

chain/index/chain-indexing-overview-for-rpc-providers.MD Outdated Show resolved Hide resolved

Formatting fixes

0e1773b

BigLep marked this pull request as ready for review October 16, 2024 05:36

github-actions bot previously requested changes Oct 16, 2024

View reviewed changes