Skip to content

Commit

Permalink
chore(prover): Fixes broken non-GPU setup (#2953)
Browse files Browse the repository at this point in the history
Previously, this was done via a compile flag. This was deemed poor taste
and we've moved a specific env-var - `ZKSYNC_USE_CUDA_STUBS`. This is
wired deep within crypto dependencies.

This PR updates the dependencies, adds documentation on how to work with
the flag and updates all workflows (fixing the broken ones from
RUSTFLAGS time).

---------

Co-authored-by: Igor Aleksanov <popzxc@yandex.ru>
  • Loading branch information
EmilLuta and popzxc authored Sep 25, 2024
1 parent 6313c7d commit 2edd8a2
Show file tree
Hide file tree
Showing 13 changed files with 40 additions and 37 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/build-docker-from-tag.yml
Original file line number Diff line number Diff line change
Expand Up @@ -103,7 +103,7 @@ jobs:
image_tag_suffix: ${{ needs.setup.outputs.image_tag_suffix }}-avx512
ERA_BELLMAN_CUDA_RELEASE: ${{ vars.ERA_BELLMAN_CUDA_RELEASE }}
CUDA_ARCH: "60;70;75;80;89"
WITNESS_GENERATOR_RUST_FLAGS: "-Ctarget_feature=+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl --cfg=no_cuda"
WITNESS_GENERATOR_RUST_FLAGS: "-Ctarget_feature=+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl"
secrets:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down
1 change: 1 addition & 0 deletions .github/workflows/build-witness-generator-template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,7 @@ jobs:
ERA_BELLMAN_CUDA_RELEASE: ${{ inputs.ERA_BELLMAN_CUDA_RELEASE }}
CUDA_ARCH: ${{ inputs.CUDA_ARCH }}
WITNESS_GENERATOR_RUST_FLAGS: ${{ inputs.WITNESS_GENERATOR_RUST_FLAGS }}
ZKSYNC_USE_CUDA_STUBS: true
runs-on: [ matterlabs-ci-runner-c3d ]
strategy:
matrix:
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-common-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:
echo "SCCACHE_GCS_SERVICE_ACCOUNT=gha-ci-runners@matterlabs-infra.iam.gserviceaccount.com" >> .env
echo "SCCACHE_GCS_RW_MODE=READ_WRITE" >> .env
echo "RUSTC_WRAPPER=sccache" >> .env
echo "RUSTFLAGS=--cfg=no_cuda" >> .env
echo "ZKSYNC_USE_CUDA_STUBS=true" >> .env
- name: Start services
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-core-lint-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ jobs:
echo "SCCACHE_GCS_SERVICE_ACCOUNT=gha-ci-runners@matterlabs-infra.iam.gserviceaccount.com" >> .env
echo "SCCACHE_GCS_RW_MODE=READ_WRITE" >> .env
echo "RUSTC_WRAPPER=sccache" >> .env
echo "RUSTFLAGS=--cfg=no_cuda" >> .env
echo "ZKSYNC_USE_CUDA_STUBS=true" >> .env
echo "prover_url=postgres://postgres:notsecurepassword@localhost:5432/zksync_local_prover" >> $GITHUB_ENV
echo "core_url=postgres://postgres:notsecurepassword@localhost:5432/zksync_local" >> $GITHUB_ENV
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci-prover-reusable.yml
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,7 @@ jobs:
echo "SCCACHE_GCS_SERVICE_ACCOUNT=gha-ci-runners@matterlabs-infra.iam.gserviceaccount.com" >> .env
echo "SCCACHE_GCS_RW_MODE=READ_WRITE" >> .env
echo "RUSTC_WRAPPER=sccache" >> .env
echo "RUSTFLAGS=--cfg=no_cuda" >> .env
echo "ZKSYNC_USE_CUDA_STUBS=true" >> .env
- name: Start services
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -166,7 +166,7 @@ jobs:
with:
image_tag_suffix: ${{ needs.setup.outputs.image_tag_suffix }}-avx512
action: "build"
WITNESS_GENERATOR_RUST_FLAGS: "-Ctarget_feature=+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl --cfg=no_cuda"
WITNESS_GENERATOR_RUST_FLAGS: "-Ctarget_feature=+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl"
secrets:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/release-test-stage.yml
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@ jobs:
image_tag_suffix: ${{ needs.setup.outputs.image_tag_suffix }}-avx512
ERA_BELLMAN_CUDA_RELEASE: ${{ vars.ERA_BELLMAN_CUDA_RELEASE }}
CUDA_ARCH: "60;70;75;80;89"
WITNESS_GENERATOR_RUST_FLAGS: "-Ctarget_feature=+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl --cfg=no_cuda"
WITNESS_GENERATOR_RUST_FLAGS: "-Ctarget_feature=+avx512bw,+avx512cd,+avx512dq,+avx512f,+avx512vl "
secrets:
DOCKERHUB_USER: ${{ secrets.DOCKERHUB_USER }}
DOCKERHUB_TOKEN: ${{ secrets.DOCKERHUB_TOKEN }}
Expand Down
3 changes: 2 additions & 1 deletion docker/witness-generator/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
FROM ghcr.io/matter-labs/zksync-build-base:latest AS builder

ARG DEBIAN_FRONTEND=noninteractive
ARG RUST_FLAGS="--cfg=no_cuda"
ARG RUST_FLAGS=""
ENV RUSTFLAGS=${RUST_FLAGS}
ENV ZKSYNC_USE_CUDA_STUBS=true

# set of args for use of sccache
ARG SCCACHE_GCS_BUCKET=""
Expand Down
3 changes: 1 addition & 2 deletions docker/witness-vector-generator/Dockerfile
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
FROM ghcr.io/matter-labs/zksync-build-base:latest AS builder

ARG DEBIAN_FRONTEND=noninteractive
ARG RUST_FLAGS="--cfg=no_cuda"
ENV RUSTFLAGS=${RUST_FLAGS}
ENV ZKSYNC_USE_CUDA_STUBS=true

# set of args for use of sccache
ARG SCCACHE_GCS_BUCKET=""
Expand Down
3 changes: 1 addition & 2 deletions docs/guides/external-node/00_quick_start.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,7 @@ cd docker-compose-examples
docker compose --file testnet-external-node-docker-compose.yml down --volumes
```

You can see the status of the node (after recovery) in
[local grafana dashboard](http://localhost:3000/dashboards).
You can see the status of the node (after recovery) in [local grafana dashboard](http://localhost:3000/dashboards).

Those commands start ZKsync node locally inside docker.

Expand Down
23 changes: 13 additions & 10 deletions docs/guides/setup-dev.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,9 @@ cargo install sqlx-cli --version 0.8.1
curl -L https://foundry.paradigm.xyz | bash
foundryup --branch master

# Non GPU setup, can be skipped if the machine has a GPU configured for provers
echo "export RUSTFLAGS='--cfg=no_cuda'" >> ~/.bashrc

# Non CUDA (GPU) setup, can be skipped if the machine has a CUDA installed for provers
# Don't do that if you intend to run provers on your machine. Check the prover docs for a setup instead.
echo "export ZKSYNC_USE_CUDA_STUBS=true" >> ~/.bashrc
# You will need to reload your `*rc` file here

# Clone the repo to the desired location
Expand Down Expand Up @@ -243,20 +243,23 @@ commands related to deployment, you can pass flags for Foundry integration.

## Non-GPU setup

Circuit Prover requires a GPU (& CUDA bindings) to run. If you still want to be able to build everything locally on
non-GPU setup, you'll need to change your rustflags.
Circuit Prover requires a CUDA bindings to run. If you still want to be able to build everything locally on non-CUDA
setup, you'll need use CUDA stubs.

For a single run, it's enough to export it on the shell:

```
export RUSTFLAGS='--cfg=no_cuda'
export ZKSYNC_USE_CUDA_STUBS=true
```

For persistent runs, you can either echo it in your ~/.<shell>rc file (discouraged), or configure it for your taste in
`config.toml`.
For persistent runs, you can echo it in your ~/.<shell>rc file

```
echo "export ZKSYNC_USE_CUDA_STUBS=true" >> ~/.<SHELL>rc
```

For project level configuration, edit `/path/to/zksync/.cargo/config.toml`. For global cargo setup,
`~/.cargo/config.toml`. Add the following:
Note that the same can be achieved with RUSTFLAGS (discouraged). The flag is `--cfg=no_cuda`. You can either set
RUSTFLAGS as env var, or pass it in `config.toml` (either project level or global). The config would need the following:

```toml
[build]
Expand Down
28 changes: 14 additions & 14 deletions prover/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

4 changes: 2 additions & 2 deletions prover/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -63,8 +63,8 @@ circuit_sequencer_api = "=0.150.5"
zkevm_test_harness = "=0.150.5"

# GPU proving dependencies
wrapper_prover = { package = "zksync-wrapper-prover", version = "=0.150.7" }
shivini = "=0.150.7"
wrapper_prover = { package = "zksync-wrapper-prover", version = "=0.150.9" }
shivini = "=0.150.9"

# Core workspace dependencies
zksync_multivm = { path = "../core/lib/multivm", version = "0.1.0" }
Expand Down

0 comments on commit 2edd8a2

Please sign in to comment.