Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feature: holder scanner refactoring #98

Open
lucasmenendez opened this issue Oct 5, 2023 · 1 comment
Open

feature: holder scanner refactoring #98

lucasmenendez opened this issue Oct 5, 2023 · 1 comment
Labels
enhancement New feature or request frozen This issue is currently being held up by external motivations and so on.

Comments

@lucasmenendez
Copy link
Collaborator

lucasmenendez commented Oct 5, 2023

Actually census3 scans token holders in a single step. It queries to the web3 endpoint for a batches of logs, filtering it by some contract events associated to the token type, and then calculates the resulting holders balances, adding and subtracting the logs amounts. When it completes a batch of logs, updates the database and continues to the next batch.

This design has some advantages about performance over other proposals, reducing the number of requests to the chain (through the web3 endpoint) and keeping the last holder list and balance updated.

But also has some disadvantages:

  • Requires a lot of complexity regarding partial balances calculation that requires a specific logic for every token type, making more difficult to support new token types.
  • Makes the token scanning process longer and heavier.
  • Other uses cases such as the creation of a census for a certain block number will require to scan again from the token creation block, because the past holders that current balance is zero are no longer in the database.

For these reasons, here is a proposal to redesign the holder scanner, making it simpler, more flexible and modular:

  • The holders scanner will work with the same principles, using batches of logs, filtered by some events associated to the token type, but it just maintain a list of holders addresses (pasts and current ones).
  • The balances calculation will be performed during census generation process, which is already performed in background.

Advantages of this design:

  • The scanner complexity and holders deviations will be reduced.
  • The balance calculation can be customised by the user, defining the method of the contract to get the balance of a holder (this is very useful to support tokens like the ERC1155).
  • The token scan will be faster.
  • The census calculation for strategy in a certain set of block numbers is trivial.
  • The number of database writes will be reduced and grouped in specific moments.
  • Reduce the complexity and required code to support new token types

Disadvantages:

  • The database will no longer contain and real time updated of valid holders. It could have the balances of the holders for the calculated census with latest block number, but not the latest balance at all.
  • The number of requests to the web3 endpoint will be increased. Quick estimation:
current requests += (census created * registered holders for the strategy census tokens of the created census)
@lucasmenendez lucasmenendez added the enhancement New feature or request label Oct 5, 2023
lucasmenendez added a commit that referenced this issue Feb 3, 2024
* New Feature: Added support for ERC20, ERC721, and ERC777 tokens in holders_provider.go.
* Bug Fix: Fixed a bug related to block number bounds iteration in the holder scanner.
* Refactor: Codebase has been restructured for better organization and readability. Unused database tables and queries have been removed.
* Documentation: Updated service reference file with component descriptions and improved instructions for running the service using Docker Compose.
* Chore: Database schema and queries have been updated, including modifications to the 'censuses' and 'tokens' tables.

Check the changelog in the PR.
It closes #122 #98 #85
@lucasmenendez
Copy link
Collaborator Author

This issue has been partially implemented on #137:

  • The scanner has been refactored and simplified.
  • The holder provider interface is now implemented also by the web3 token types.
  • The balance calculation remains equal, but several issues has been fixed, increasing the accuracy drastically.
  • The holders with negative or zero balance still stored in the database to allow to query about their balances in the past. The parts designed work on last block number has been updated to filter holders by balances.

@lucasmenendez lucasmenendez added the frozen This issue is currently being held up by external motivations and so on. label Feb 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request frozen This issue is currently being held up by external motivations and so on.
Projects
None yet
Development

No branches or pull requests

1 participant