[Custom Processors] Proposal for Enhancing Aptos Indexer: Subgraph-Inspired Improvements

[mshakeg]

Overview

The Aptos Indexer is a powerful tool. However, drawing inspiration from established systems like The Graph's subgraphs could significantly improve its usability, efficiency, and developer experience. This issue proposes several key enhancements to align the Aptos Indexer more closely with subgraph functionality and developer experience.

Proposed Improvements

1. Schema Definition and Code Generation

Current State: manually defined database schemas in code(e.g. using SQLAlchemy ORM in python or typeorm for typescript).

Proposal:

• Introduce a schema.graphql file for declarative schema definition, similar to subgraphs.
• Implement a code generation tool that supports multiple languages:

  aptos codegen --language [python|rust|typescript]

• This tool would likely wrap around existing codegen tools that generate language-specific code for ORM entities specified in the schema.graphql.

Benefits:

• Simplified, language-agnostic schema definition
• Reduced boilerplate code
• Improved consistency across different language implementations

2. Configuration and Event Filtering

Current State: Manual implementation of event filtering logic in processors.

Proposal:

• Extend config.yaml to specify:

  • Addresses of packages (to which the relevant Move modules have been published to)
  • Module events to index
  • Mapping of events to specific handler functions

• Example structure:

  dataSources:
    - address: "0x1234..."
      modules:
        - name: "MyModule"
          events:
            - name: "MyEvent"
              handler: handleMyEvent

• Modify the Transaction Stream Service to stream only the module events specified in this extended config.yaml. It currently seems like the Aptos Indexer receives a stream of ALL txs processed by the entirety of the chain, instead of ONLY the events relevant to it, which is quite inefficient. This may require changes to the Transaction Stream Service to facilitate a stream of only relevant/desired events for a given Aptos Indexer Processor.

Benefits:

• More declarative and easier-to-understand configuration
• Improved performance by processing only relevant events
• Reduced complexity in processor implementation

3. Deployment Pipeline and Hosting

Current State: Manual deployment and hosting of processors.

Proposal:

• Introduce an "Aptos Indexer Node":

  • Capable of hosting multiple "Aptos Indexer Processors"
  • Similar to how a Graph Node can host multiple subgraphs deployed to it.

• Develop a deployment pipeline:

  aptos indexer deploy --node [aptos-indexer-node-url]

Benefits:

• Simplified deployment process
• Improved resource utilization especially when hosting multiple processors.
• Potential for a decentralized network of indexer nodes

4. Testing Framework

Current State: Limited specialized testing tools for processors.

Proposal:

• Develop a testing framework for each supported language, inspired by the Matchstick framework for testing subgraphs. See uniswap/v3-subgraph for a good illustration of the ease of implementing tests for the Aptos Indexer should be.

  aptos indexer test --language [python|rust|typescript]
  

• Features:

  • developers can mock module events
  • developers can pass mocked events to module event handler functions
  • developers can easily query mock DB state post handler execution and add assertions for verifying correct data operations

Benefits:

• Improved testability of processors
• Faster development cycles
• Higher quality and more reliable indexers

5. Language-Specific API/SDK Libraries

Current State: Developers are required to fork the aptos-indexer-processors repo and implement custom processors within the relevant language folder(e.g. ./python, ./rust, ./typescript) in this repo.

Proposal:

Introduce language-specific API/SDK libraries (aptos-indexer-py, aptos-indexer-rs, aptos-indexer-ts) for Python, Rust, and TypeScript respectively, inspired by @graphprotocol/graph-ts used in subgraphs.

These libraries should provide:

1. Type-safe interfaces for Aptos blockchain data
2. Utility functions for common operations (e.g., cryptographic functions, big number handling)
3. Standardized logging and error handling
4. Helper methods for database operations
5. Wrappers for Aptos-specific data types and operations

Benefits:

• Cleaner custom aptos indexer repo that does not directly track any of the additional code present in the aptos-indexer-processors repo and reduced boilerplate code.
• Consistent developer experience across languages
• Improved type safety
• Easier implementation of best practices

For reference on functionality and API design, see The Graph's @graphprotocol/graph-ts documentation:
https://thegraph.com/docs/en/developing/graph-ts/api/

[mshakeg]

After further investigation it would seem like:

Developers are required to fork the aptos-indexer-processors repo and implement custom processors within the relevant language folder(e.g. ./python, ./rust, ./typescript) in this repo.

would only be applicable to python as a package has not yet been published like @aptos-labs/aptos-processor-sdk for typescript or aptos-indexer-processor-sdk for rust. However, the other suggestions would still be relevant.

[mshakeg]

I've hacked a work around using the @aptos-labs/aptos-processor-sdk to give a very rudimentary solution to some of the improvements suggested in the original issue, such as:

2. Configuration and Event Filtering - though all must be configured directly in code(not in a yaml file)
3. Deployment Pipeline and Hosting - though code has to be manually added to the super processor repo
4. Testing Framework - using jest with a GenericProcessorUoWTestHelper helper class to make testing event handlers for any GenericProcessorUoW extended coprocessor.

https://github.com/mshakeg/super-processor-example.git

[mshakeg]

@larry-aptos do you think I should add the GenericProcessorUoW I developed into the @aptos-labs/aptos-processor-sdk? It would require using mikro-orm instead of typeorm.

[mshakeg]

Hey @rtso @bowenyang007 not sure who to tag here, but I really think the ecosystem would benefit from this immensely.

[larry-aptos]

Thanks for your feedback! I'll get back to you when I have time. Also, feel free to chime in. @aptos-labs/ecosystem-infra