SYS-3560 Adds event listener feature to the node #418

thadouk · 2024-06-07T15:27:30Z

Proposed changes

Extends Ethereum bridge pallet to be bidirectional and listen to events emitted in the bridge contract.
The PR introduces:

avn-event-handler service: a thread running in client space that acts as an oracle, discovering information from Ethereum and submitting the information back to the runtime
The information is submitted in batches based on an Ethereum block range which is agreed initially from collators and then dictated by the runtime
New mechanism is configured to co-exist with ethereum-events and avoid double processing of events in the system.

Type of change/Merge

🚨What type of change is this PR?

Put an x in the boxes that apply

Checklist

Put an x in the boxes that apply. You can also fill these out after creating the PR.

You describe the purpose of the PR, e.g.:
- What does it do?
- Highlight what important points reviewers should know about;
- Indicates if there is something left for follow-up PRs.
Documentation updated
Business logic tested successfully
Verify First, Write Last: In Substrate development, it is important that you always ensure preconditions are met and return errors at the beginning. After these checks have completed, then you may begin the function's computation.

Further comments

Cargo.toml

Dockerfile.22_04

ivan-cholakov · 2024-06-10T13:28:24Z

node/src/command.rs

+ runner.async_run(|config| {
+ // we don't need any of the components of new_partial, just a runtime, or a task
+ // manager to do `async_run`.
+ let registry = config.prometheus_config.as_ref().map(|cfg| &cfg.registry);


you now create the task_manager inside the runner? why is this change needed?

it doesn't compile with the previous code when you enable try-runtime. This is copied from Substrate

pallets/eth-bridge/src/lib.rs

pallets/eth-bridge/src/tests/incoming_events_tests.rs

nahuseyoum · 2024-06-12T09:47:33Z

pallets/token-manager/src/lib.rs

@@ -1092,4 +1096,8 @@ impl<T: Config> BridgeInterfaceNotification for Pallet<T> {

 Ok(())
 }
+
+ fn on_incoming_event_processed(event: &EthEvent) -> DispatchResult {
+ Self::processed_event_handler(event)


Is it worth adding the feature guard here for Lift and Growth coming via this hook?

No, from the pallet point of view we either process events coming from eth-bridge or not.
The runtime defines the filter and the rest of the components are just executing.

primitives/avn-common/src/event_discovery.rs

runtime/avn/src/lib.rs

Introduces the data structures and the voting mechanism for accepting events ranges from ethereum. Ranges that are too big will be broken down into smaller partitions, voted upon individually, and accepted as one. Follow up commit will do the validate unsigned and processing of the partitions. Jira tickets: - SYS-3635 - SYS-3930

Updates the validate unsigned function to support submit_discovered_events extrinsic. Jira tickets: - SYS-3635 - SYS-3930

This PR introduces support for processing votes upon achieving consensus. It extends extrinsics and data structures accordingly. Once the final vote is cast, the voting round for the Ethereum range partition is finalized, and the partition is processed. Additionally, this PR extends the subscription mechanism of the eth-bridge pallet. By default, pallets ignore processed events, requiring the implementation of a handler to process them. Benchmarking for certain actions will be addressed in a separate PR. Jira Tasks: - SYS-3930 - SYS-3635

Signed-off-by: Ivan Cholakov <icholakov1@gmail.com> Co-authored-by: Thanos Doukoudakis <56822898+thadouk@users.noreply.github.com>

Adds a new endpoint that allows the chain to reach consenus over the initial ethereum range to be used. This allows to simplify the voting logic for active ranges, eliminating the special case of the first range. Jira ticket: - SYS-3930

Extends the runtime API for initial range consensus mechanism Jira task - SYS-3930

Adds basic unit tests that check the submit_ethereum_events interface Jira task: - SYS-3930 - SYS-3635

- General cleanup - Rename of storage items - Removal of mocks - Wire up with runtime, implementing different events filters for each runtime.

…397) This commit includes the integration of the initial range voting functionality with the client. It ports changes to integrate the runtime API for testing runtime behavior. Additionally, it refactors the runtime API to improve proof creation by relocating helper functions to primitive crates. Fixes genesis config for test runtime. Jira tasks: - SYS-3930 - SYS-3635

…398)

This commit addresses issues with extrinsic submission in the runtime: - Registers the extension to the instance submitting the extrinsic. - Updates signature validation. - Simplifies interface and error handling via the runtime API, following the report_equivocation style.

…o eth-bridge (#390) Co-authored-by: Thanos <56822898+thadouk@users.noreply.github.com> Co-authored-by: nahuseyoum <nahu.seyoum@aventus.io>

This commit allows the avn-event-hanlder to handle more graceful error. Also applies code formatting.

Signed-off-by: Ivan Cholakov <icholakov1@gmail.com> Co-authored-by: Ivan Cholakov <icholakov1@gmail.com> Co-authored-by: Thanos <56822898+thadouk@users.noreply.github.com> Co-authored-by: Michael <62653655+MBrozhko34@users.noreply.github.com>

Adds handlers in Token manager and wire up for Runtime to process the first type of events from eth-bridge. Jira ticket: - SYS-4072

Use checked fucntions to do the mod calculations. Updates tests, benchmarks and resto of the code accordingly Restores batch value to tested.

An event is marked as processed once the event processing logic is completed. The processing outcome of the pallet does not affect this.

Co-authored-by: Nahu <39748285+nahuseyoum@users.noreply.github.com> Signed-off-by: Thanos Doukoudakis <56822898+thadouk@users.noreply.github.com>

Deprecates ethereum-events entry points that add manually events in the system. Moving forward the responsibility will be shifted to eth-bridge event scanning mechanism on Ethereum.

- Fixes an issue where the last block of a range was polled twice and included to the next range as well. - Addresses feature PR feedback - cleanup

Adds AvtLowerClaimedData in the types the client scans. Fixes the way the data field is constructed ensuring the data field is explicitly set as None when no data is present, to align with the runtime implementation.

thadouk requested review from ivan-cholakov, MBrozhko34 and micaelffrancoAV June 7, 2024 15:27

thadouk requested review from nahuseyoum and fluorostani as code owners June 7, 2024 15:27

ivan-cholakov reviewed Jun 10, 2024

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

ivan-cholakov reviewed Jun 10, 2024

View reviewed changes

Dockerfile.22_04 Show resolved Hide resolved

ivan-cholakov approved these changes Jun 10, 2024

View reviewed changes

ivan-cholakov reviewed Jun 10, 2024

View reviewed changes

nahuseyoum approved these changes Jun 12, 2024

View reviewed changes

thadouk force-pushed the feat/SYS-3560-add-event-listener branch from 226e3ff to 58bf60c Compare June 26, 2024 11:09

ivan-cholakov and others added 19 commits June 26, 2024 12:36

Sys 3931 identify events (#368)

ee9eb2f

SYS-3930: Extends validate unsigned for new extrinsic (#370)

73fe963

Updates the validate unsigned function to support submit_discovered_events extrinsic. Jira tickets: - SYS-3635 - SYS-3930

feat: added runtime-api for eth-event polling (#376)

8b8a23d

Signed-off-by: Ivan Cholakov <icholakov1@gmail.com> Co-authored-by: Thanos Doukoudakis <56822898+thadouk@users.noreply.github.com>

feat: added runtime-api for eth-event polling (#376)

0b06265

Signed-off-by: Ivan Cholakov <icholakov1@gmail.com> Co-authored-by: Thanos Doukoudakis <56822898+thadouk@users.noreply.github.com>

SYS-3932 - implement initial version of polling mechanism (#378)

4031886

Exposes initial range voting via the runtime API (#393)

c48675f

Extends the runtime API for initial range consensus mechanism Jira task - SYS-3930

SYS-3930 Adds unit tests for event votes (#372)

1a9294d

Adds basic unit tests that check the submit_ethereum_events interface Jira task: - SYS-3930 - SYS-3635

SYS-3930 Cleanup and finalises interface with runtime (#396)

f3e52d9

- General cleanup - Rename of storage items - Removal of mocks - Wire up with runtime, implementing different events filters for each runtime.

fix: refactored getting the authors to verify current node's author (#…

ac70bd1

…398)

SYS-3965: added first stage migration of processed events mechanism t…

128ef19

…o eth-bridge (#390) Co-authored-by: Thanos <56822898+thadouk@users.noreply.github.com> Co-authored-by: nahuseyoum <nahu.seyoum@aventus.io>

fix: Update binary version

d928362

fix: Make avn-event-handler service more faullt tollerant

5ca8313

This commit allows the avn-event-hanlder to handle more graceful error. Also applies code formatting.

nahuseyoum and others added 15 commits June 26, 2024 12:37

SYS-4069 Wait for finality before fetching events (#409)

e3b96b0

SYS-4070 Fix keystore bug (#411)

68701f7

SYS-4072 fix handling of eth-bridge processed events (#410)

f2a0f03

Adds handlers in Token manager and wire up for Runtime to process the first type of events from eth-bridge. Jira ticket: - SYS-4072

SYS-3560 Set block range size via state migration (#413)

45140f2

fix: Runtime panic and upgrade migration issues (#414)

5766a88

Use checked fucntions to do the mod calculations. Updates tests, benchmarks and resto of the code accordingly Restores batch value to tested.

SYS-3560 add error on rejected event (#416)

97f5bc3

fix: Issues with event processing

1bc4ceb

An event is marked as processed once the event processing logic is completed. The processing outcome of the pallet does not affect this.

cargo package, spec and impl versions increased

a7b43c1

fix: missing dependencies on filter config trait

78dc31a

Apply suggestions from code review

9ab962d

Co-authored-by: Nahu <39748285+nahuseyoum@users.noreply.github.com> Signed-off-by: Thanos Doukoudakis <56822898+thadouk@users.noreply.github.com>

SYS-3635 deprecate ethreum events entrypoints (#419)

aa4335d

Deprecates ethereum-events entry points that add manually events in the system. Moving forward the responsibility will be shifted to eth-bridge event scanning mechanism on Ethereum.

SYS-3560 fix: ensures last block of a range is polled once (#420)

8ef2b05

- Fixes an issue where the last block of a range was polled twice and included to the next range as well. - Addresses feature PR feedback - cleanup

chore: Remove unused field

26f9881

fix: event parsing and adds test case

ffcbfad

Adds AvtLowerClaimedData in the types the client scans. Fixes the way the data field is constructed ensuring the data field is explicitly set as None when no data is present, to align with the runtime implementation.

fix: bump runtime version

3420b53

thadouk force-pushed the feat/SYS-3560-add-event-listener branch from 58bf60c to 3420b53 Compare June 26, 2024 11:38

thadouk merged commit 388193e into main Jun 26, 2024
5 checks passed

thadouk deleted the feat/SYS-3560-add-event-listener branch June 26, 2024 15:25

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SYS-3560 Adds event listener feature to the node #418

SYS-3560 Adds event listener feature to the node #418

thadouk commented Jun 7, 2024 •

edited

Loading

ivan-cholakov Jun 10, 2024

nahuseyoum Jun 12, 2024

nahuseyoum Jun 12, 2024

thadouk Jun 17, 2024 •

edited

Loading

SYS-3560 Adds event listener feature to the node #418

SYS-3560 Adds event listener feature to the node #418

Conversation

thadouk commented Jun 7, 2024 • edited Loading

Proposed changes

Type of change/Merge

Checklist

Further comments

ivan-cholakov Jun 10, 2024

Choose a reason for hiding this comment

nahuseyoum Jun 12, 2024

Choose a reason for hiding this comment

nahuseyoum Jun 12, 2024

Choose a reason for hiding this comment

thadouk Jun 17, 2024 • edited Loading

Choose a reason for hiding this comment

thadouk commented Jun 7, 2024 •

edited

Loading

thadouk Jun 17, 2024 •

edited

Loading