Hypersync source support #103

SozinM · 2024-09-16T06:08:19Z

This Pr adds hypersync integration to the rindexer.
Initially i made it with enum, but i think that dynamic dispatch fits better here. We need to choose instance during runtime via config.

This is p1 of the changes, would be nice to move streaming logic to another trait/extend current one, to be able to specialize streaming based on the upstream.

vercel · 2024-09-16T06:08:23Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
rindexer-documentation	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Jan 1, 2025 2:12am

SozinM · 2024-09-16T06:11:49Z

Hi @joshstevens19
I'm implementing hypersync support for rindexer and want to ask you opinion on using enum/dyn Trait for provider selection.
Now I've implemented a simple enum approach, but it would be quite easy to switch to dyn Trait

joshstevens19 · 2024-09-16T10:36:22Z

Hi @joshstevens19 I'm implementing hypersync support for rindexer and want to ask you opinion on using enum/dyn Trait for provider selection. Now I've implemented a simple enum approach, but it would be quite easy to switch to dyn Trait

Thanks for this i think how you done it is pretty good i was hoping to also extend this to support ExEx on reth later on (but that's for a later thought) but as dyn trait add runtime costs due to dynamic dispatcher vs enum which uses static dispatcher (which is faster) lets pick performance here so what you done IMHO is good - thoughts? if you are in agreement il do a PR review once its out of draft and rebased.

If you can also add to the docs for this once done as well to say we support hypersync endpoints

SozinM · 2024-10-08T12:57:14Z

Hi @joshstevens19
I think it's ready for review, had little fight with playground autogen :)
In this implementation dyn traits doesn't add much overhead (compared with enums as they will need vtable)
Example working, alingside with tests. Current implementation doesn't bring much benefit speedwise but would once we specialize streaming.
Also other providers could start implement additional traits.

joshstevens19 · 2024-10-08T23:17:06Z

thanks @SozinM i actually made a slight change today to provider to expose block timestamp do you want to rebase to resolve - d1683fb ? will take a look at this PR tomorrow good work

joshstevens19 · 2024-10-08T23:19:09Z

documentation/docs/pages/docs/start-building/add.mdx

@@ -26,11 +26,14 @@ it then ask you 2 questions:
 networks:
  - name: ethereum // [!code focus]
    chain_id: 1
+    kind: Rpc


i wouldnt change the rindexer yaml i would work this out under the hood basically if rpc .includes hypersync then you know its turned on

i could make default kind = Rpc, so we won't change default rindexer yaml. But considering other impls in the future (exex, sqd) it better to specify it trough param

yeah ok we can make it default RPC and then not add it to all examples but just have 1 example of how to enable it for hypersync?

joshstevens19 · 2024-10-08T23:19:42Z

core/src/provider.rs

+            .map_err(|err| ProviderError::CustomError(err.to_string()))
+    }
+
+    async fn get_logs(&self, filter: &RindexerEventFilter) -> Result<Vec<Log>, ProviderError> {


i did comment this but worth looking at this d1683fb as we exposed a new log type to expose the block timestamp

joshstevens19 · 2024-10-08T23:20:03Z

core/src/provider.rs

+
+    async fn get_block_number(&self) -> Result<U64, ProviderError>;
+
+    async fn get_logs(&self, filter: &RindexerEventFilter) -> Result<Vec<Log>, ProviderError>;


d1683fb#diff-871403679f46c8546ae048f666359f2bf092e8f9840e52172093327f95410e45R29

joshstevens19 · 2024-10-08T23:20:53Z

examples/rindexer_demo_cli/rindexer.yaml

@@ -15,8 +19,8 @@ contracts:
  details:
  - network: ethereum
    address: 0xae78736cd615f374d3085123a210448e74fc6393
-    start_block: '18600000'
-    end_block: '18718056'
+    start_block: '11600000'


joshstevens19 · 2024-10-08T23:21:11Z

rindexer_rust_playground/src/main.rs

-        "/Users/joshstevens/code/rindexer/rindexer_rust_playground/rindexer.yaml",
-    )
-    .expect("Invalid path");
+    let path = PathBuf::from_str("./rindexer.yaml").expect("Invalid path");


thx lol full path 😛

joshstevens19 · 2024-10-08T23:21:55Z

core/src/manifest/network.rs

@@ -11,6 +17,8 @@ pub struct Network {

    pub rpc: String,

+    pub kind: ProviderType,


yeah defo not keen on this approach would rather auto work it out by rpc URL IMHO

i could think only about 2 options:

Deduce type based on url

Implicitly specify it
First have the problem that it could fail for no apparent reason (change from using https://1.hypersync.xyz to localproxy/kubernetes service name will break it silently with no way to fix it unless code is changed)
Second is too verbose

Wdyt about combining approaches -> We deduce from url if no kind provided?

yeah ok we can make it default RPC and then not add it to all examples but just have 1 example of how to enable it for hypersync?

joshstevens19 · 2024-10-20T20:05:03Z

Thanks a lot I’m currently on annual leave till 28th but once back we can get this in 💪💪

SozinM · 2024-10-21T17:47:51Z

@joshstevens19
Sorry, i'm procrastinating to do final touches, but overall it should work now
Will push to complete this before you are back :)

joshstevens19 · 2024-10-29T16:50:35Z

hey is this complete @SozinM ?

SozinM · 2024-10-30T12:54:19Z

Hi @joshstevens19 yes, should be good now

joshstevens19 · 2024-11-02T00:34:39Z

documentation/docs/pages/docs/start-building/yaml-config/contracts.mdx

@@ -43,6 +43,7 @@ project_type: no-code
 networks:
  - name: ethereum
    chain_id: 1
+    kind: Rpc


no need for this if fallbacks to RPC (same for all below here)

joshstevens19 · 2024-11-02T00:35:51Z

documentation/docs/pages/docs/start-building/add.mdx

@@ -31,6 +31,16 @@ networks:

 This will skip for you if you only have one network setup.

+By setting kind to "Hypersync" you could use [hypersync](https://docs.envio.dev/docs/HyperSync/overview) as a data source.


hmm need to think about the best place for this callout

joshstevens19 · 2024-11-02T00:37:42Z

ok going to need to do some manual tests on this over weekend and make sure all good if so can deploy do want to fix some of the documentation call outs etc - nice job @SozinM

joshstevens19 · 2024-12-26T22:33:01Z

So sorry guys will review this tomorrow/Monday and test it out then merge it good fully forgot 🙏

joshstevens19 · 2024-12-28T10:26:43Z

hey when i build this locally i get

error: failed to run custom build command for `hypersync-net-types v0.9.0`

Caused by:
  process didn't exit successfully: `/Users/joshstevens/code/rindexer/target/debug/build/hypersync-net-types-b32cc79a337da369/build-script-build` (exit status: 101)
  --- stderr
  thread 'main' panicked at /Users/joshstevens/.cargo/registry/src/index.crates.io-6f17d22bba15001f/hypersync-net-types-0.9.0/build.rs:5:10:
  compiling schema: Error { kind: Failed, extra: "Failed to execute `capnp --version`: No such file or directory (os error 2). Please verify that version 0.5.2 or higher of the capnp executable is installed on your system. See https://capnproto.org/install.html" }
  note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
warning: build failed, waiting for other jobs to finish...
joshstevens@AMWM-G2136D03XG rindexer %

@SozinM

its trying to run a build.rs of:

fn main() {
    capnpc::CompilerCommand::new()
        .file("hypersync_net_types.capnp")
        .run()
        .expect("compiling schema");
}

we shouldnt have to install capnp locally or on machine to make work imho - how could we fix this?

SozinM marked this pull request as draft September 16, 2024 06:08

SozinM changed the title ~~Hypersync support~~ Hypersync source support Sep 16, 2024

vercel bot deployed to Preview September 16, 2024 06:09 View deployment

joshstevens19 mentioned this pull request Sep 16, 2024

Alternative data sources to RPC #43

Open

vercel bot deployed to Preview September 18, 2024 13:28 View deployment

vercel bot deployed to Preview September 18, 2024 15:38 View deployment

vercel bot deployed to Preview September 26, 2024 18:36 View deployment

vercel bot deployed to Preview September 30, 2024 11:01 View deployment

vercel bot deployed to Preview September 30, 2024 12:30 View deployment

vercel bot deployed to Preview October 8, 2024 11:01 View deployment

vercel bot deployed to Preview October 8, 2024 11:47 View deployment

SozinM force-pushed the feature/hypersync-support branch from 9c04744 to 02c9d7f Compare October 8, 2024 11:50

vercel bot deployed to Preview October 8, 2024 11:52 View deployment

SozinM force-pushed the feature/hypersync-support branch from 02c9d7f to a360138 Compare October 8, 2024 12:35

vercel bot deployed to Preview October 8, 2024 12:37 View deployment

vercel bot deployed to Preview October 8, 2024 12:49 View deployment

SozinM marked this pull request as ready for review October 8, 2024 12:52

vercel bot deployed to Preview October 8, 2024 12:53 View deployment

joshstevens19 reviewed Oct 8, 2024

View reviewed changes

SozinM force-pushed the feature/hypersync-support branch from fb388f9 to 5677c19 Compare October 15, 2024 17:07

vercel bot deployed to Preview October 15, 2024 17:09 View deployment

vercel bot deployed to Preview October 30, 2024 12:55 View deployment

joshstevens19 reviewed Nov 2, 2024

View reviewed changes

vercel bot deployed to Preview November 4, 2024 08:15 View deployment

joshstevens19 mentioned this pull request Dec 26, 2024

Support Reth nodes to index using ExExes #31

Open

Mikhail Sozin added 8 commits December 28, 2024 10:13

Hypersync support + docs

a108eab

Final touches

376350d

Playground bindings

6d1dd30

Playground bindings

96aa0be

Default provider without docs

765b999

Cleanup and fixes

e5ab953

Timestamp and cleanup

2b4ad51

Remove missed kind: Rpc docs entries

dbf7654

update docs

9c5d47a

joshstevens19 force-pushed the feature/hypersync-support branch from a45442d to 9c5d47a Compare December 28, 2024 10:27

vercel bot deployed to Preview December 28, 2024 10:29 View deployment

Merge branch 'master' into feature/hypersync-support

610384b

vercel bot deployed to Preview January 1, 2025 02:12 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Hypersync source support #103

Hypersync source support #103

SozinM commented Sep 16, 2024 •

edited

Loading

vercel bot commented Sep 16, 2024 •

edited

Loading

SozinM commented Sep 16, 2024

joshstevens19 commented Sep 16, 2024 •

edited

Loading

SozinM commented Oct 8, 2024

joshstevens19 commented Oct 8, 2024

joshstevens19 Oct 8, 2024

SozinM Oct 9, 2024

joshstevens19 Oct 15, 2024

joshstevens19 Oct 8, 2024

joshstevens19 Oct 8, 2024

joshstevens19 Oct 8, 2024

joshstevens19 Oct 8, 2024

joshstevens19 Oct 8, 2024

SozinM Oct 9, 2024

joshstevens19 Oct 15, 2024

joshstevens19 commented Oct 20, 2024

SozinM commented Oct 21, 2024

joshstevens19 commented Oct 29, 2024

SozinM commented Oct 30, 2024

joshstevens19 Nov 2, 2024

joshstevens19 Nov 2, 2024

joshstevens19 commented Nov 2, 2024

joshstevens19 commented Dec 26, 2024 •

edited

Loading

joshstevens19 commented Dec 28, 2024 •

edited

Loading


		async fn get_block_number(&self) -> Result<U64, ProviderError>;

		async fn get_logs(&self, filter: &RindexerEventFilter) -> Result<Vec<Log>, ProviderError>;

		@@ -11,6 +17,8 @@ pub struct Network {

		pub rpc: String,

		pub kind: ProviderType,

		@@ -31,6 +31,16 @@ networks:

		This will skip for you if you only have one network setup.

		By setting kind to "Hypersync" you could use [hypersync](https://docs.envio.dev/docs/HyperSync/overview) as a data source.

Hypersync source support #103

Are you sure you want to change the base?

Hypersync source support #103

Conversation

SozinM commented Sep 16, 2024 • edited Loading

vercel bot commented Sep 16, 2024 • edited Loading

SozinM commented Sep 16, 2024

joshstevens19 commented Sep 16, 2024 • edited Loading

SozinM commented Oct 8, 2024

joshstevens19 commented Oct 8, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshstevens19 commented Oct 20, 2024

SozinM commented Oct 21, 2024

joshstevens19 commented Oct 29, 2024

SozinM commented Oct 30, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joshstevens19 commented Nov 2, 2024

joshstevens19 commented Dec 26, 2024 • edited Loading

joshstevens19 commented Dec 28, 2024 • edited Loading

SozinM commented Sep 16, 2024 •

edited

Loading

vercel bot commented Sep 16, 2024 •

edited

Loading

joshstevens19 commented Sep 16, 2024 •

edited

Loading

joshstevens19 commented Dec 26, 2024 •

edited

Loading

joshstevens19 commented Dec 28, 2024 •

edited

Loading