WIP on python substreams handler

[dkdocs]

PR Description: Abstracted gRPC Stream Event Processing and Enhanced Cursor Management

This PR refactors the existing gRPC streaming code to provide a reusable, abstracted event processing structure and introduces improved cursor management for efficient event handling. The following key changes and improvements have been made:

---

Key Changes

1. Abstracted Event Processing with EventProcessor Interface

   • Introduced an EventProcessor abstract base class to decouple the event processing logic from the core gRPC connection and cursor handling.
   • Added a default implementation, DefaultEventProcessor, which can be replaced or extended to customize event handling.
   • This design allows easy reuse of the streaming logic with different processing behaviors by subclassing EventProcessor.

2. Improved gRPC Connection Management

   • Encapsulated gRPC connection management in a start_stream function, making it easier to handle reconnections and manage metadata.
   • Added functionality to handle reconnections in case of RpcError, with a delay between retry attempts.

3. Cursor Management

   • Implemented functions save_cursor_to_file and get_cursor_from_file to save and retrieve the cursor position, allowing the stream to resume from where it left off.
   • This ensures continuity in data processing, especially in long-running or live stream scenarios.

Instructions for Testing

1. Run the Script:

   python main.py <package_url> <module_name> [start_block] [stop_block]

   • Replace <package_url> and <module_name> with the appropriate values.
   • Optional [start_block] and [stop_block] arguments specify the block range.

2. Customize Event Processing:

   • Create a custom event processor by subclassing EventProcessor and overriding the process_event method.
   • Pass the custom processor to start_stream.

3. Verify Cursor Handling:

   • Interrupt the stream, then restart the script to verify that it resumes from the saved cursor position.

This refactor improves the flexibility, reliability, and maintainability of the gRPC streaming code, making it well-suited for long-running Solana data processing tasks.