Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: EigenDA M0 data availability client #3041

Closed
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
be428a5
initial commit
juan518munoz Sep 17, 2024
0ad4ca9
fix retrieve-blobs in toml
juan518munoz Sep 18, 2024
6feb624
Add eigenda to toolbox
gianbelinche Sep 19, 2024
d3a95a0
feat: Eigen Blob ID in L1 (#275)
gianbelinche Sep 19, 2024
081c1b2
Merge branch 'da-eigen-implementation' into toolbox-eigenda
gianbelinche Sep 19, 2024
5c87565
Update submodule
gianbelinche Sep 19, 2024
d6f7f48
Merge pull request #277 from lambdaclass/toolbox-eigenda
juanbono Sep 20, 2024
869babb
feat: eigenDA more metrics (#281)
juan518munoz Sep 26, 2024
5021488
fix: Get blob from L1 (#282)
gianbelinche Sep 26, 2024
cc954c4
initial commit
juan518munoz Sep 26, 2024
3150e0d
use Notify for a more deterministic approach
juan518munoz Sep 27, 2024
b108adc
replace atomic for mutex
juan518munoz Oct 2, 2024
1286def
move const to config
juan518munoz Oct 2, 2024
3a21329
Merge branch 'main' into da-eigen-merged-main
gianbelinche Oct 4, 2024
d963287
Update contracts
gianbelinche Oct 4, 2024
2394dd5
Finalize merge
gianbelinche Oct 4, 2024
6b0b902
feat: concurrent da_dispatcher (#288)
juan518munoz Oct 7, 2024
e40c06b
Merge branch 'main' into da-eigen-merged-main
gianbelinche Oct 7, 2024
4726372
Update contracts
gianbelinche Oct 7, 2024
aa6ea7d
feat: Da eigen implementation docs & backup scripts (#289)
juan518munoz Oct 7, 2024
a5b7659
Update query jsons
gianbelinche Oct 7, 2024
df6d952
Merge branch 'da-eigen-implementation' into da-eigen-merged-main
gianbelinche Oct 7, 2024
04d2395
Format md
gianbelinche Oct 7, 2024
8a0ed94
Merge branch 'main' into da-eigen-merged-main
gianbelinche Oct 8, 2024
bdb3a82
Refactor secrets
gianbelinche Oct 8, 2024
d06b433
Merge pull request #290 from lambdaclass/da-eigen-merged-main
juan518munoz Oct 8, 2024
894ff20
update contracts submodule
juan518munoz Oct 8, 2024
b428234
remove eigen proxy from general setup, add it to doc steps
juan518munoz Oct 8, 2024
5bea05d
Remove formats (#293)
gianbelinche Oct 8, 2024
30fdc59
remove arguments from ecosystem init
juan518munoz Oct 8, 2024
27cebcc
Update queries (#294)
gianbelinche Oct 8, 2024
a9c0b3b
Update contracts
gianbelinche Oct 8, 2024
a6f496b
Format
gianbelinche Oct 8, 2024
9eb946b
Update contracts
gianbelinche Oct 8, 2024
c84e271
Add dashboard step
gianbelinche Oct 8, 2024
8806c09
Merge branch 'main' into m0-merge-main
gianbelinche Oct 8, 2024
6a573c5
Update queries
gianbelinche Oct 8, 2024
bf93f78
Format code
gianbelinche Oct 8, 2024
2e49a43
query changes
juan518munoz Oct 8, 2024
ebf7eca
Merge pull request #295 from lambdaclass/m0-merge-main
juan518munoz Oct 8, 2024
9c612a7
Remove formatting and configs
gianbelinche Oct 8, 2024
d6c4c26
Update contracts
gianbelinche Oct 8, 2024
22da332
add max blob size to doc
juan518munoz Oct 8, 2024
24256c8
Update eigenda-integration.md
juan518munoz Oct 8, 2024
16d499b
Add description and scope to eigenda-integration.md
juanbono Oct 14, 2024
75e8bbe
cargo new
juanbono Oct 15, 2024
e0d7ba2
initial impl with axum routes
juan518munoz Oct 15, 2024
713c85a
add component in node builder
juan518munoz Oct 15, 2024
098461c
Merge pull request #304 from lambdaclass/eigenda_proxy_setup
juanbono Oct 16, 2024
c99f2bd
Add new configs to eigenda
gianbelinche Oct 17, 2024
7f948de
Update comments
gianbelinche Oct 17, 2024
e859ca2
add eigenDA client protos (#306)
juanbono Oct 18, 2024
4a19f54
Merge pull request #307 from lambdaclass/eigenda-proxy-config
juanbono Oct 18, 2024
8c79c06
Eigenda memstore (#305)
gianbelinche Oct 18, 2024
2baa576
Add initial implementation disperser client
gianbelinche Oct 18, 2024
ac5f3f1
Add holesky tests
gianbelinche Oct 21, 2024
0ae04c3
Add error handling
gianbelinche Oct 21, 2024
1a9cf57
Remove proxy from name
gianbelinche Oct 21, 2024
017b544
Add new configs
gianbelinche Oct 21, 2024
3cb1c31
Update eigenda-integration.md
gianbelinche Oct 21, 2024
0b1727d
Address pr comments
gianbelinche Oct 21, 2024
7b4ced0
initial commit
juan518munoz Oct 21, 2024
790ab84
add conditional compilation attribute to test
juan518munoz Oct 21, 2024
924f788
remove unused imports
juan518munoz Oct 21, 2024
cdec06c
improve err
juan518munoz Oct 22, 2024
aff1048
remove unwraps
juan518munoz Oct 22, 2024
680a207
implement `IntoResponse` for `RequestProcessorError`
juan518munoz Oct 23, 2024
1054ca5
use a single `MemStoreConfig`
juan518munoz Oct 23, 2024
beb2c1d
add new step
juan518munoz Oct 23, 2024
4f204c4
change suggested api_node_url
juan518munoz Oct 23, 2024
89e1abf
memstore integration
juan518munoz Oct 24, 2024
ba7596c
remove comments & fix memstore test
juan518munoz Oct 24, 2024
d450836
EigenDA Disperser Store Module (#309)
gianbelinche Oct 24, 2024
69ca965
fix memstore config
juan518munoz Oct 24, 2024
de13f8a
initial commit
juan518munoz Oct 25, 2024
d3cdaaf
remove unused imports
juan518munoz Oct 25, 2024
b9dfcc4
modularize code
juan518munoz Oct 25, 2024
b2e9107
non auth: wait for dispersal
juan518munoz Oct 25, 2024
a553d8c
auth: wait for dispersal
juan518munoz Oct 28, 2024
89286ea
add config for auth dispersal
juan518munoz Oct 28, 2024
648db73
implement get blob data for remote disperser
juan518munoz Oct 28, 2024
c881a6d
remove unwraps, improve tests
juan518munoz Oct 28, 2024
90ab847
remove eigenda_proxy layer
juan518munoz Oct 28, 2024
e523d0d
remove field from cfg & update readme
juan518munoz Oct 28, 2024
2f3cca2
add padding before dispersal request
juan518munoz Oct 28, 2024
d89b13a
remove unwrap
juan518munoz Oct 29, 2024
0a2cfb5
merge dst branch
juan518munoz Oct 29, 2024
5d0479f
remove proxy mention from integration doc
juan518munoz Oct 29, 2024
dbdd161
feat:EigenDA - Layer combination (#314)
juan518munoz Oct 29, 2024
165d76b
merge main p1
juan518munoz Oct 29, 2024
e5cdb1d
merge branch m0 integration
juan518munoz Oct 29, 2024
d0a5cab
revert changes--no-verify
juan518munoz Oct 30, 2024
d71c1a8
Merge pull request #315 from lambdaclass/da-eigen-implementation-m0-r…
juan518munoz Oct 30, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
25 changes: 5 additions & 20 deletions .githooks/pre-push
Original file line number Diff line number Diff line change
Expand Up @@ -6,29 +6,14 @@
RED='\033[0;31m'
NC='\033[0m' # No Color

# Common prompts
INSTALL_PROPT="Please install ZK Stack CLI using zkstackup from https://github.com/matter-labs/zksync-era/tree/main/zkstack_cli/zkstackup"
FORMAT_PROMPT="Please format the code via 'zkstack dev fmt', cannot push unformatted code"

# Check that prettier formatting rules are not violated.
if which zkstack >/dev/null; then
if ! zkstack dev fmt --check; then
if which zk_supervisor >/dev/null; then
if ! zk_supervisor fmt --check; then
echo -e "${RED}Push error!${NC}"
echo -e "${FORMAT_PROMPT}"
echo "Please format the code via 'zks fmt', cannot push unformatted code"
exit 1
fi
else
if which zk_supervisor >/dev/null; then
echo -e "${RED}WARNING: zkup, zk_inception/zki, and zk_supervisor/zks are DEPRECATED.${NC}"
echo -e "${RED}${INSTALL_PROPT}${NC}"

if ! zk_supervisor fmt --check; then
echo -e "${RED}Push error!${NC}"
echo -e "${FORMAT_PROMPT}"
exit 1
fi
else
echo -e "${INSTALL_PROPT}"
exit 1
fi
echo "Please install zk_toolbox using zkup from https://github.com/matter-labs/zksync-era/tree/main/zk_toolbox/zkup, and then run ./bin/zkt from the zksync-era repository."
exit 1
fi
2 changes: 1 addition & 1 deletion .github/pull_request_template.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,4 +17,4 @@
- [ ] PR title corresponds to the body of PR (we generate changelog entries from PRs).
- [ ] Tests for the changes have been added / updated.
- [ ] Documentation comments have been added / updated.
- [ ] Code has been formatted via `zkstack dev fmt` and `zkstack dev lint`.
- [ ] Code has been formatted via `zk_supervisor fmt` and `zk_supervisor lint`.
4 changes: 2 additions & 2 deletions .github/release-please/config.json
Original file line number Diff line number Diff line change
Expand Up @@ -20,9 +20,9 @@
"release-type": "simple",
"component": "prover"
},
"zkstack_cli": {
"zk_toolbox": {
"release-type": "simple",
"component": "zkstack_cli",
"component": "zk_toolbox",
"plugins": [
"cargo-workspace"
]
Expand Down
4 changes: 2 additions & 2 deletions .github/release-please/manifest.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"core": "25.0.0",
"core": "24.28.0",
"prover": "16.5.0",
"zkstack_cli": "0.1.2"
"zk_toolbox": "0.1.2"
}
9 changes: 3 additions & 6 deletions .github/workflows/build-contract-verifier-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,7 @@ jobs:
- name: start-services
run: |
echo "IMAGE_TAG_SUFFIX=${{ env.IMAGE_TAG_SUFFIX }}" >> .env
mkdir -p ./volumes/postgres
run_retried docker compose pull zk postgres
docker compose up -d zk postgres
ci_run pre_download_compilers.sh
Expand All @@ -112,19 +113,15 @@ jobs:
ci_run git config --global --add safe.directory /usr/src/zksync/sdk/binaryen
ci_run git config --global --add safe.directory /usr/src/zksync/contracts/system-contracts
ci_run git config --global --add safe.directory /usr/src/zksync/contracts
ci_run ./bin/zkt || true
ci_run ./bin/zk || true
ci_run run_retried curl -LO https://storage.googleapis.com/matterlabs-setup-keys-us/setup-keys/setup_2\^26.key

- name: install zkstack
run: |
ci_run ./zkstack_cli/zkstackup/install -g --path ./zkstack_cli/zkstackup/zkstackup || true
ci_run zkstackup -g --local

- name: build contracts
if: env.BUILD_CONTRACTS == 'true'
run: |
ci_run cp etc/tokens/{test,localhost}.json
ci_run zkstack dev contracts
ci_run zk_supervisor contracts

- name: Login to Docker registries
if: ${{ inputs.action == 'push' }}
Expand Down
10 changes: 3 additions & 7 deletions .github/workflows/build-core-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -114,6 +114,7 @@ jobs:
- name: start-services
run: |
echo "IMAGE_TAG_SUFFIX=${{ env.IMAGE_TAG_SUFFIX }}" >> .env
mkdir -p ./volumes/postgres
run_retried docker compose pull zk postgres
docker compose up -d zk postgres
ci_run pre_download_compilers.sh
Expand All @@ -126,19 +127,14 @@ jobs:
ci_run git config --global --add safe.directory /usr/src/zksync/contracts/system-contracts
ci_run git config --global --add safe.directory /usr/src/zksync/contracts
ci_run ./bin/zk || true
ci_run ./bin/zkt || true
ci_run run_retried curl -LO https://storage.googleapis.com/matterlabs-setup-keys-us/setup-keys/setup_2\^26.key

- name: Install zkstack
if: env.BUILD_CONTRACTS == 'true'
run: |
ci_run ./zkstack_cli/zkstackup/install -g --path ./zkstack_cli/zkstackup/zkstackup || true
ci_run zkstackup -g --local

- name: build contracts
if: env.BUILD_CONTRACTS == 'true'
run: |
ci_run cp etc/tokens/{test,localhost}.json
ci_run zkstack dev contracts --system-contracts --l1-contracts --l2-contracts
ci_run zk_supervisor contracts

- name: Login to Docker registries
if: ${{ inputs.action == 'push' }}
Expand Down
12 changes: 3 additions & 9 deletions .github/workflows/build-local-node-docker.yml
Original file line number Diff line number Diff line change
Expand Up @@ -50,13 +50,9 @@ jobs:

- name: start-services
run: |
mkdir -p ./volumes/postgres
run_retried docker compose pull zk postgres
docker compose up -d zk postgres

- name: Install zkstack
run: |
ci_run ./zkstack_cli/zkstackup/install -g --path ./zkstack_cli/zkstackup/zkstackup || true
ci_run zkstackup -g

- name: init
run: |
Expand All @@ -65,11 +61,9 @@ jobs:
ci_run git config --global --add safe.directory /usr/src/zksync/contracts

ci_run zk
ci_run zkt
ci_run cp etc/tokens/{test,localhost}.json

- name: build contracts
run: |
ci_run zkstack dev contracts
ci_run zk_supervisor contracts

- name: update-image
run: |
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ jobs:
- name: Download Setup data
run: |
gsutil -m rsync -r gs://matterlabs-setup-data-us/${{ inputs.setup_keys_id }} docker/prover-gpu-fri-gar
cp -v docker/prover-gpu-fri-gar/*.bin docker/circuit-prover-gpu-gar/

- name: Login to us-central1 GAR
run: |
Expand Down Expand Up @@ -69,10 +70,6 @@ jobs:
--tag europe-docker.pkg.dev/matterlabs-infra/matterlabs-docker/prover-fri-gpu-gar:2.0-${{ inputs.protocol_version }}-${{ inputs.image_tag_suffix }} \
us-docker.pkg.dev/matterlabs-infra/matterlabs-docker/prover-fri-gpu-gar:2.0-${{ inputs.protocol_version }}-${{ inputs.image_tag_suffix }}

- name: Move Setup data from prover-gpu-fri-gar to circuit-prover-gpu-gar
run: |
mv -v docker/prover-gpu-fri-gar/*.bin docker/circuit-prover-gpu-gar/

- name: Build and push circuit-prover-gpu-gar
uses: docker/build-push-action@5cd11c3a4ced054e52742c5fd54dca954e0edd85 # v6.7.0
with:
Expand Down
5 changes: 3 additions & 2 deletions .github/workflows/build-prover-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ jobs:
RUNNER_COMPOSE_FILE: "docker-compose-runner-nightly.yml"
ERA_BELLMAN_CUDA_RELEASE: ${{ inputs.ERA_BELLMAN_CUDA_RELEASE }}
CUDA_ARCH: ${{ inputs.CUDA_ARCH }}
runs-on: [matterlabs-ci-runner-high-performance]
runs-on: [ matterlabs-ci-runner-high-performance ]
strategy:
matrix:
component:
Expand All @@ -56,7 +56,6 @@ jobs:
- prover-fri-gateway
- prover-job-monitor
- proof-fri-gpu-compressor
- prover-autoscaler
outputs:
protocol_version: ${{ steps.protocolversion.outputs.protocol_version }}
steps:
Expand All @@ -75,6 +74,7 @@ jobs:
- name: start-services
run: |
echo "IMAGE_TAG_SUFFIX=${{ env.IMAGE_TAG_SUFFIX }}" >> .env
mkdir -p ./volumes/postgres
run_retried docker compose pull zk postgres
docker compose up -d zk postgres
ci_run sccache --start-server
Expand All @@ -91,6 +91,7 @@ jobs:
run: |
ci_run run_retried curl -LO https://storage.googleapis.com/matterlabs-setup-keys-us/setup-keys/setup_2\^24.key


- name: login to Docker registries
if: github.event_name != 'pull_request' && (github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/'))
run: |
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/build-witness-generator-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,6 +75,7 @@ jobs:
- name: start-services
run: |
echo "IMAGE_TAG_SUFFIX=${{ env.IMAGE_TAG_SUFFIX }}" >> .env
mkdir -p ./volumes/postgres
run_retried docker compose pull zk postgres
docker compose up -d zk postgres
ci_run sccache --start-server
Expand Down
10 changes: 5 additions & 5 deletions .github/workflows/ci-common-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -27,15 +27,15 @@ jobs:
- name: Start services
run: |
run_retried docker-compose -f ${RUNNER_COMPOSE_FILE} pull
mkdir -p ./volumes/postgres
docker-compose -f ${RUNNER_COMPOSE_FILE} up --build -d zk postgres
- name: Install zkstack

- name: Init
run: |
ci_run ./zkstack_cli/zkstackup/install -g --path ./zkstack_cli/zkstackup/zkstackup
ci_run zkstackup -g --local
ci_run zkt

# This does both linting and "building". We're using `zk lint prover` as it's common practice within our repo
# `zk lint prover` = cargo clippy, which does cargo check behind the scenes, which is a lightweight version of cargo build
- name: Lints
run: ci_run zkstack dev lint -t rs --check
run: ci_run zk_supervisor lint -t rs --check

25 changes: 9 additions & 16 deletions .github/workflows/ci-core-lint-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -26,31 +26,24 @@ jobs:
- name: Start services
run: |
ci_localnet_up

- name: Install zkstack
run: |
ci_run ./zkstack_cli/zkstackup/install -g --path ./zkstack_cli/zkstackup/zkstackup
ci_run zkstackup -g --local

- name: Build
run: |
ci_run ./bin/zkt
ci_run yarn install
ci_run git config --global --add safe.directory /usr/src/zksync
ci_run zk_supervisor db setup --prover-url=${{ env.prover_url }} --core-url=${{ env.core_url }}

- name: Database setup
run: |
ci_run zkstack dev db setup --prover-url=${{ env.prover_url }} --core-url=${{ env.core_url }}

- name: Lints
run: |
ci_run zkstack dev fmt --check
ci_run zkstack dev lint -t md --check
ci_run zkstack dev lint -t sol --check
ci_run zkstack dev lint -t js --check
ci_run zkstack dev lint -t ts --check
ci_run zkstack dev lint -t rs --check
ci_run zkstack dev lint -t autocompletion --check
ci_run zk_supervisor fmt --check
ci_run zk_supervisor lint -t md --check
ci_run zk_supervisor lint -t sol --check
ci_run zk_supervisor lint -t js --check
ci_run zk_supervisor lint -t ts --check
ci_run zk_supervisor lint -t rs --check

- name: Check Database
run: |
ci_run zkstack dev database check-sqlx-data --prover-url=${{ env.prover_url }} --core-url=${{ env.core_url }}
ci_run zk_supervisor database check-sqlx-data --prover-url=${{ env.prover_url }} --core-url=${{ env.core_url }}
Loading