feat(eigenda): implement eigenDA client remaining features #3243

juanbono · 2024-11-08T22:09:32Z

What ❔

This PR adds extra functionality to the eigen client:

Non authenticated blob dispersal feat(eigen-client-extra-features): non auth dispersal lambdaclass/zksync-era#318
RLP encoding for blobs feat(eigen-client-extra-feature): rlp encoding lambdaclass/zksync-era#319
Get inclusion data feat(eigen-client-extra-features): get inclusion data lambdaclass/zksync-era#324
Memstore feat(eigen-client-extra-features): Eigen client memstore lambdaclass/zksync-era#321
Blob size limit feat(eigen-client-extra-features): blob size limit lambdaclass/zksync-era#325
Soft confirmations feat(eigen-client-extra-features): Add soft confirmations lambdaclass/zksync-era#322
Verification logic feat(eigen-client-extra-features): Verifier lambdaclass/zksync-era#326
EigenDA Documentation feat(eigen-client-extra-features): Eigen client documentation lambdaclass/zksync-era#327

Why ❔

These features are needed to have a full featured client and to have the same features as when communicating with the EigenDA proxy.

Checklist

PR title corresponds to the body of PR (we generate changelog entries from PRs).
Tests for the changes have been added / updated.
Documentation comments have been added / updated.
Code has been formatted via zkstack dev fmt and zkstack dev lint.

impl rlp encoding

Implement non auth blob dispersal for eigen client --------- Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com>

feat(eigen-client-extra-features): get inclusion data

add test for remote disperser cfg

* initial commit * Add tests * Add memstore * Add assert to tests * Add rest of memstore * Address pr comments * Remove to retriable error * Fix conflicts * Document memstore * Fix typo --------- Co-authored-by: Juan Munoz <juanmunoz890@gmail.com>

blob size limit --------- Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com>

* initial commit * Add tests * Add memstore * Add assert to tests * Add rest of memstore * Add soft confirmations * Remove print * Address pr comments * Remove to retriable error * Fix conflicts * Add inclusion data * Change status query timeout to millis * Document memstore * Fix typo * Fix typo * Format --------- Co-authored-by: Juan Munoz <juanmunoz890@gmail.com>

* initial commit * Add tests * Add memstore * Add assert to tests * Add rest of memstore * Address pr comments * Remove to retriable error * Fix conflicts * Add verifier * Fix verifier * Add path to points to config * Fix typo * Fix eigenda env test * Fix verifier test * Move eigendaservicemanager to generated * Remove unneeded imports * Document verifier * Modify errors * Address comments * Fix conflicts --------- Co-authored-by: Juan Munoz <juanmunoz890@gmail.com>

* initial commit * move file location * fix step

* Add eth confirmation depth * Fix error

#329) * Remove unused custom quorum numbers * Add serial to tests

feat(eigen-client-extra-features): fix clippy and add doc

feat(eigen-client-extra-features): merge main

…dule-update feat(eigen-client-extra-features): update contracts submodule

core/lib/config/src/configs/da_client/eigen.rs

core/lib/protobuf_config/src/proto/config/da_client.proto

core/node/da_clients/src/eigen/sdk.rs

core/node/da_clients/src/eigen/verifier.rs

* Move inclusion logic * Remove unused import

* Move inclusion logic * Remove verify cert

* Change eth conf depth for settlement layer * Remove blob size limit * Move reserved * Change function name * Add constants to lib * Fix compilation

* initial commit * remove anyhow err * Format code * Fix compilation --------- Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com>

* Add option to download points * Fix test * Fix clippy * Change name to points source * Fix test * Fix test

…s with mock tests (#356) * initial commit * create verifier client trait * remove unwraps * Update core/node/da_clients/src/eigen/verifier.rs Ignore test Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com> * Update core/node/da_clients/src/eigen/sdk.rs Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com> * change ignore test comment * Format code --------- Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com>

…on docs (#360) * Remove steps to run from integration docs * Combine readme * Clarify readme

…and-main feat(eigen-client-extra-features): Merge Main

core/lib/config/src/configs/da_client/eigen.rs

core/lib/protobuf_config/src/proto/config/da_client.proto

core/node/da_clients/src/eigen/README.md

core/node/da_clients/src/eigen/verifier.rs

resources/g1.point

* Remove verify cert * Change names * Create aux functions * Remove milestones * Add url crate to save pointsz * Remove hex and rlp * Make get inclusion data not return entire blob info * Remove hardcoded function selectors * move tests to a different file * Remove retrys and add option to get inclusion data * Solve clippy * Get blob data from db * Add async trait * Change function to get blob data * Remove path to points in favor of links * Change link to url * Update core/lib/env_config/src/da_client.rs Co-authored-by: juan518munoz <62400508+juan518munoz@users.noreply.github.com> --------- Co-authored-by: juan518munoz <62400508+juan518munoz@users.noreply.github.com>

Eigen client extra merge main

dimazhornyk

The PR logic look great, left mostly some nits and style-related comments

core/lib/env_config/src/da_client.rs

core/node/da_clients/src/eigen/README.md

core/node/da_clients/src/eigen/sdk.rs

core/node/da_clients/src/eigen/verifier.rs

core/node/node_framework/src/implementations/layers/da_clients/eigen.rs

core/lib/config/src/configs/da_client/eigen.rs

* Add envy load * Readd proto reference * Rename blob id to request id * Make literals constants * Make point size constant * Get pool unique * Remaining comments * Fix comment * Add check for failed states * Change l1 name

Merge main to extra features

dimazhornyk

looks great! I will start the CI and ask for another reviewer to do a final look

core/lib/protobuf_config/src/da_client.rs

core/node/da_clients/src/eigen/sdk.rs

slowli

Some preliminary comments. Will try to do another review round later this week.

slowli · 2024-12-17T15:03:42Z

core/lib/config/src/configs/da_client/eigen.rs

+    pub disperser_rpc: String,
+    /// Block height needed to reach in order to consider the blob finalized
+    /// a value less or equal to 0 means that the disperser will not wait for finalization
+    pub settlement_layer_confirmation_depth: i32,


Do different negative values have semantic differences? If not, please use u32.

fixed here lambdaclass#375

slowli · 2024-12-17T15:03:50Z

core/lib/config/src/configs/da_client/eigen.rs

+    pub disperser_rpc: String,
+    /// Block height needed to reach in order to consider the blob finalized
+    /// a value less or equal to 0 means that the disperser will not wait for finalization
+    pub settlement_layer_confirmation_depth: i32,


Do different negative values have semantic differences? If not, please use u32.

fixed here lambdaclass#375

slowli · 2024-12-17T15:05:57Z

core/lib/config/src/configs/da_client/eigen.rs

-    pub rpc_node_url: String,
-    pub inclusion_polling_interval_ms: u64,
+    /// URL of the Disperser RPC server
+    pub disperser_rpc: String,


String::default() is not a reasonable default for URLs; please #[derive(Default)] (maybe replacing it with a manual impl, if the parameter does have a reasonable default value).

fixed here lambdaclass#375 , the testnet url was used as default

slowli · 2024-12-17T15:06:23Z

core/lib/config/src/configs/da_client/eigen.rs

+    /// URL of the Ethereum RPC server
+    pub eigenda_eth_rpc: Option<String>,
+    /// Address of the service manager contract
+    pub eigenda_svc_manager_address: String,


Why not Address? Also, again, addresses don't have meaningful defaults, and "" is not a valid address.

fixed here lambdaclass#375, the testnet URL was used as default

slowli · 2024-12-17T15:09:43Z

core/lib/config/src/configs/da_client/eigen.rs

+    /// Url to the file containing the G1 point used for KZG
+    pub g1_url: String,


Are there any integrity checks for the file contents? What is the outcome of receiving invalid points (e.g., is this an attack vector)?

slowli · 2024-12-17T16:10:46Z

core/node/node_framework/src/implementations/layers/da_clients/eigen.rs

+impl GetBlobData for GetBlobFromDB {
+    async fn call(&self, input: &'_ str) -> anyhow::Result<Option<Vec<u8>>> {
+        let pool = self.pool.clone();
+        let input = input.to_string();


This allocation is not required.

Fixed here lambdaclass#375

slowli · 2024-12-17T16:10:53Z

core/node/node_framework/src/implementations/layers/da_clients/eigen.rs

+            .data_availability_dal()
+            .get_blob_data_by_blob_id(&input)
+            .await?;
+        drop(conn);


This is useless.

fixed here lambdaclass#375

slowli · 2024-12-17T16:11:46Z

core/node/node_framework/src/implementations/layers/da_clients/eigen.rs

+
+#[async_trait::async_trait]
+impl GetBlobData for GetBlobFromDB {
+    async fn call(&self, input: &'_ str) -> anyhow::Result<Option<Vec<u8>>> {


This lifetime can be skipped, I think the only reason it's not flagged by Clippy is that it's inside the async_trait proc macro.

fixed here lambdaclass#375

slowli · 2024-12-17T16:16:12Z

core/node/da_clients/src/eigen/verifier.rs

+        self.verify_merkle_proof(cert.clone())?;
+        self.verify_security_params(cert.clone()).await?;


I'm reasonably sure cert can be passed by ref at least for the last two methods (verify_batch is tricky because of ethabi::Tokens created in hash_batch_metadata). In general, please pass values by ref where possible; there are too many examples in this module and adjacent modules to highlight.

fixed here lambdaclass#375

slowli · 2024-12-17T16:17:54Z

core/node/da_clients/Cargo.toml

+rust-kzg-bn254.workspace = true
+ark-bn254.workspace = true
+num-bigint.workspace = true
+serial_test.workspace = true


This looks like a dev dependency.

fixed here lambdaclass#375

* Change settlement layer for u32 * Change string to address * Remove unwraps * Remove error from name * Remove unused to bytes * Rename call for get blob data * Revert "Change string to address" This reverts commit 6dd94d4. * Change string for address * feat(eigen-client-extra-features): address PR comments (part 2) (#374) * initial commit * clippy suggestion * feat(eigen-client-extra-features): address PR comments (part 3) (#376) * use keccak256 fn * simplify get_context_block * use saturating sub * feat(eigen-client-extra-features): address PR comments (part 4) (#378) * Replace decode bytes for ethabi * Add default to eigenconfig * Change str to url * Add index to data availability table * Address comments * Change error to verificationerror * Format code * feat(eigen-client-extra-features): address PR comments (part 5) (#377) * use trait object * prevent blocking non async code * clippy suggestion --------- Co-authored-by: juan518munoz <62400508+juan518munoz@users.noreply.github.com> --------- Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com> --------- Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com> * Format code --------- Co-authored-by: juan518munoz <62400508+juan518munoz@users.noreply.github.com>

juan518munoz and others added 23 commits October 31, 2024 17:14

add integration doc

fcecc81

Reimplement eigen config (#320)

2bcf78f

feat(eigen-client-extra-feature): rlp encoding (#319)

537cf92

impl rlp encoding

feat(eigen-client-extra-features): non auth dispersal (#318)

f58e0bf

Implement non auth blob dispersal for eigen client --------- Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com>

initial commit

04b93e6

add spaces

c121eca

Revert wrong push

4a38ee2

initial commit

e64cc2e

Merge pull request #324 from lambdaclass/eigen-client-get-inclusion-data

5b8665b

feat(eigen-client-extra-features): get inclusion data

fix(eigen-client-extra-features): fix eigen config test (#323)

aa9f1b2

add test for remote disperser cfg

feat(eigen-client-extra-features): blob size limit (#325)

b164870

blob size limit --------- Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com>

feat(eigen-client-extra-features): Eigen client documentation (#327)

3b9e766

* initial commit * move file location * fix step

feat(eigen-client-extra-features): Eth confirmation depth (#328)

6668ab9

* Add eth confirmation depth * Fix error

feat(eigen-client-extra-features): Remove unused custom quorum numbers (

cc9e2be

#329) * Remove unused custom quorum numbers * Add serial to tests

doc & cargo check all

9769a17

clippy

91bb79b

Merge branch 'main' into eigen-client-merge-main

cbe124e

update contracts submodule

9c73b3f

Merge pull request #330 from lambdaclass/eigen-client-clippy-and-doc

94ecb52

feat(eigen-client-extra-features): fix clippy and add doc

Merge pull request #331 from lambdaclass/eigen-client-merge-main

342b476

feat(eigen-client-extra-features): merge main

github-actions bot added the external-contribution label Nov 8, 2024

juanbono changed the title ~~Eigen client extra features~~ Implement eigenDA client remaining features Nov 8, 2024

juanbono and others added 2 commits November 8, 2024 19:14

Merge branch 'main' into eigen-client-extra-features

077bc13

Update integration readme (#335)

c98726a

juan518munoz mentioned this pull request Nov 11, 2024

feat(main): add eigen client extra features lambdaclass/zksync-era#317

Closed

12 tasks

juan518munoz and others added 2 commits November 12, 2024 11:02

update contracts submodule

991c340

Merge pull request #337 from lambdaclass/eigen-client-contracts-submo…

f55a424

…dule-update feat(eigen-client-extra-features): update contracts submodule

dimazhornyk requested changes Nov 25, 2024

View reviewed changes

gianbelinche mentioned this pull request Nov 26, 2024

docs: add eigenda steps to run matter-labs/zksync-docs#271

Open

gianbelinche and others added 11 commits November 28, 2024 15:38

feat(eigen-client-extra-features): Move inclusion logic (#357)

01b50ed

* Move inclusion logic * Remove unused import

feat(eigen-client-extra-features): Remove verify cert (#359)

daace34

* Move inclusion logic * Remove verify cert

feat(eigen-client-extra-features): Small changes (#358)

99c6d42

* Change eth conf depth for settlement layer * Remove blob size limit * Move reserved * Change function name * Add constants to lib * Fix compilation

feat(eigen-client-extra-features): minimal client changes (#362)

163c41a

* initial commit * remove anyhow err * Format code * Fix compilation --------- Co-authored-by: Gianbelinche <39842759+gianbelinche@users.noreply.github.com>

feat(eigen-client-extra-features): Add option to download points (#361)

cef9a89

* Add option to download points * Fix test * Fix clippy * Change name to points source * Fix test * Fix test

feat(eigen-client-extra-features): Remove steps to run from integrati…

7a61574

…on docs (#360) * Remove steps to run from integration docs * Combine readme * Clarify readme

Merge branch 'main' into eigen-client-extra-features-and-main

bc90cc6

Update Cargo.lock

87d450f

Merge pull request #363 from lambdaclass/eigen-client-extra-features-…

e2d4dab

…and-main feat(eigen-client-extra-features): Merge Main

Merge branch 'main' into eigen-client-extra-features

4d1fadb

dimazhornyk requested changes Dec 4, 2024

View reviewed changes

gianbelinche and others added 4 commits December 6, 2024 15:03

Remove points (#366)

0d01df7

Merge branch 'main' into eigen-client-extra-features

2100256

Merge pull request #367 from lambdaclass/eigen-client-extra-main

83c3e13

Eigen client extra merge main

dimazhornyk reviewed Dec 10, 2024

View reviewed changes

gianbelinche added 4 commits December 11, 2024 16:31

feat(eigen-client-extra-features): Fix PR comments (#369)

fe8f68c

* Add envy load * Readd proto reference * Rename blob id to request id * Make literals constants * Make point size constant * Get pool unique * Remaining comments * Fix comment * Add check for failed states * Change l1 name

Merge branch 'main' into eigen-client-extra-features

64a08bc

Cargo lock conflicts

8395cca

Merge pull request #370 from lambdaclass/eigen-client-extra-main-merged

f2d7941

Merge main to extra features

dimazhornyk reviewed Dec 12, 2024

View reviewed changes

core/lib/protobuf_config/src/da_client.rs Outdated Show resolved Hide resolved

core/node/da_clients/src/eigen/sdk.rs Outdated Show resolved Hide resolved

Solve comments (#372)

4b3c059

juanbono changed the title ~~Implement eigenDA client remaining features~~ feat(eigenda): implement eigenDA client remaining features Dec 16, 2024

slowli requested changes Dec 17, 2024

View reviewed changes

juan518munoz mentioned this pull request Dec 18, 2024

feat(eigen-client-extra-features): address PR comments (part 5) lambdaclass/zksync-era#377

Merged

4 tasks

gianbelinche mentioned this pull request Dec 18, 2024

feat(eigen-client-extra-features): address PR comments (part 4) lambdaclass/zksync-era#378

Merged

4 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(eigenda): implement eigenDA client remaining features #3243

feat(eigenda): implement eigenDA client remaining features #3243

juanbono commented Nov 8, 2024 •

edited

Loading

dimazhornyk left a comment

dimazhornyk left a comment

slowli left a comment

slowli Dec 17, 2024

juanbono Dec 20, 2024

slowli Dec 17, 2024

juanbono Dec 20, 2024

slowli Dec 17, 2024

juanbono Dec 20, 2024

slowli Dec 17, 2024

juanbono Dec 20, 2024

slowli Dec 17, 2024

slowli Dec 17, 2024

juanbono Dec 20, 2024 •

edited

Loading

slowli Dec 17, 2024

juanbono Dec 20, 2024

slowli Dec 17, 2024

juanbono Dec 20, 2024

slowli Dec 17, 2024

juanbono Dec 20, 2024

slowli Dec 17, 2024

juanbono Dec 20, 2024

		/// Url to the file containing the G1 point used for KZG
		pub g1_url: String,

		self.verify_merkle_proof(cert.clone())?;
		self.verify_security_params(cert.clone()).await?;

feat(eigenda): implement eigenDA client remaining features #3243

Are you sure you want to change the base?

feat(eigenda): implement eigenDA client remaining features #3243

Conversation

juanbono commented Nov 8, 2024 • edited Loading

What ❔

Why ❔

Checklist

dimazhornyk left a comment

Choose a reason for hiding this comment

dimazhornyk left a comment

Choose a reason for hiding this comment

slowli left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juanbono Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

juanbono commented Nov 8, 2024 •

edited

Loading

juanbono Dec 20, 2024 •

edited

Loading