Panic for `cuda` prover running in docker container

[nuke-web3]

Component

Other (please describe), sp1-sdk

Have you ensured that all of these are up to date?

• SP1 SDK
• cargo prove CLI/sp1up

What version of SP1 SDK are you on?

4.0.1

What version of the cargo prove CLI are you on?

cargo-prove sp1 (a80c17c 2025-04-01T00:41:38.259051986Z)

Operating System

Linux (Other)

Describe the bug

I am struggling quite a bit to get a dockerized prover working, I think there is only one issue that blocks me:

[sp1] groth16 circuit artifacts already seem to exist at /root/.sp1/circuits/groth16/v4.0.0-rc.3. if you want to re-download them, delete the directory

thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sp1-recursion-gnark-ffi-4.1.7/src/ffi/docker.rs:92:61:
failed to prove with docker: Docker command failed
 stderr: "panic: open /circuit/groth16_circuit.bin: no such file or directory\n\ngoroutine 17 [running, locked to thread]:\ngithub.com/succinctlabs/sp1-recursion-gnark/sp1.ProveGroth16({0xc0002dee38, 0x8}, {0xc0002dee40, 0x8})\n\t/sp1/crates/recursion/gnark-ffi/go/sp1/prove.go:115 +0xce9\nmain.ProveGroth16Bn254(0x0?, 0x5baf5d79f4b0)\n\t/sp1/crates/recursion/gnark-ffi/go/main.go:115 +0x5a\n"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
^Croot@nuke-gpu-l4:/# ll /circuits/
total 6946624
drwxrwxr-x 2 ubuntu ubuntu       4096 Dec 25 19:56 ./
drwxr-xr-x 1 root   root         4096 Apr 22 16:08 ../
-rw-rw-r-- 1 ubuntu ubuntu      24978 Dec 26 05:16 Groth16Verifier.sol
-rw-rw-r-- 1 ubuntu ubuntu       2479 Dec 26 05:16 SP1VerifierGroth16.sol
-rw-rw-r-- 1 ubuntu ubuntu  172612971 Dec 25 19:53 constraints.json
-rw-rw-r-- 1 ubuntu ubuntu 1152193000 Dec 25 19:56 groth16_circuit.bin
-rw-rw-r-- 1 ubuntu ubuntu 2831171769 Dec 26 05:15 groth16_pk.bin
-rw-rw-r-- 1 ubuntu ubuntu        396 Dec 26 05:16 groth16_vk.bin
-rw-rw-r-- 1 ubuntu ubuntu     823158 Dec 25 19:53 groth16_witness.json
-rw-r--r-- 1 root   root   2956476403 Apr 22 16:05 pda-proxy.tar.gz
root@nuke-gpu-l4:~/.sp1/circuits/groth16/v4.0.0-rc.3# ls
Groth16Verifier.sol  SP1VerifierGroth16.sol  constraints.json  groth16_circuit.bin  groth16_pk.bin  groth16_vk.bin  groth16_witness.json



thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sp1-recursion-gnark-ffi-4.1.7/src/ffi/docker.rs:92:61:
failed to prove with docker: Docker command failed
 stderr: "panic: open /circuit/groth16_circuit.bin: no such file or directory\n\ngoroutine 17 [running, locked to thread]:\ngithub.com/succinctlabs/sp1-recursion-gnark/sp1.ProveGroth16({0xc00025ee38, 0x8}, {0xc00025ee40, 0x8})\n\t/sp1/crates/recursion/gnark-ffi/go/sp1/prove.go:115 +0xce9\nmain.ProveGroth16Bn254(0x0?, 0x653728ad64b0)\n\t/sp1/crates/recursion/gnark-ffi/go/main.go:115 +0x5a\n"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Note I build the container for my app, upload it to a cloud instancve, run the container after manual modifications to ensure permissions and paths are correct etc.

The panic: open /circuit/groth16_circuit.bin: no such file or directory seems to indicate that internally something cannot resolve even with ~.sp1 and /circuits both containing the correct files.

[nuke-web3]

relatedly I really would like a way to trigger

sp1/crates/sdk/src/install.rs

Line 24 in 51e3964

pub fn groth16_circuit_artifacts_dir() -> PathBuf {

manually and perhaps choose a location so I can do this inside a container (at build time perhaps) to ensure the artifacts I need are present & located where the must be

[jtguibas]

Can you try rm -rf ~/.sp1 and reinstalling the artifacts?

[nuke-web3]

For reference:

• Base = Stock ubuntu + nvidia drivers, .env files, static files needed for Proxy operation but not embedded in Proxy
• Proxy = pda-proxy app container built with https://github.com/celestiaorg/pda-proxy/blob/main/Dockerfile

So in Base I start Proxy with:

rm -rf ~/.sp1
mkdir ~/.sp1 # so we can link a volume & write to it

docker run -it\
 -v /app/static:/app/static\
 -v $PDA_DB_PATH:$PDA_DB_PATH\
 -v /var/run/docker.sock:/var/run/docker.sock\ # required for docker in docker
 -v /root/.sp1:/root/.sp1\ # required to persist 
 --env-file .env\
 --env RUST_LOG=pda_proxy=debug\
 --network=host\
 --entrypoint /bin/bash\
 pda-proxy

I get what looks to be a good download (persisted in Base) but a new panic:

[sp1] groth16 circuit artifacts for version v4.0.0-rc.3 do not exist at /root/.sp1/circuits/groth16/v4.0.0-rc.3. downloading...
  [00:01:15] [############] 2.75 GiB/2.75 GiB (37.12 MiB/s, 0s)[sp1] downloaded https://sp1-circuits.s3-us-east-2.amazonaws.com/v4.0.0-rc.3-groth16.tar.gz to "/root/.sp1/circuits/groth16/v4.0.0-rc.3"

thread 'tokio-runtime-worker' panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sp1-recursion-gnark-ffi-4.1.7/src/ffi/docker.rs:92:61:
failed to prove with docker: Docker command failed
 stderr: "panic: read /witness: is a directory\n\ngoroutine 17 [running, locked to thread]:\ngithub.com/succinctlabs/sp1-recursion-gnark/sp1.ProveGroth16({0xc000224e68, 0x8}, {0xc000224e70, 0x8})\n\t/sp1/crates/recursion/gnark-ffi/go/sp1/prove.go:145 +0xccb\nmain.ProveGroth16Bn254(0x0?, 0x5b60574864b0)\n\t/sp1/crates/recursion/gnark-ffi/go/main.go:115 +0x5a\n"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

same panic even with artifacts in place on subsequent calls

[nuke-web3]

Looks like the key issue here was needing to use the native-gnark feature, which requires:

• intalling go
• installing clang tooling

here is the gist of the changes:

• celestiaorg/pda-proxy@b5319c9
• celestiaorg/pda-proxy@3d2ed3f#diff-391cf9fb9377a413bbfc6ed99e02fcddce89e8474bd4b1c839cac8eb563d63f2R19

Here is my app's logs for reference:

[2025-04-24T01:42:15Z DEBUG pda_proxy::internal::runner] Starting proof: Job { anchor: Anchor { data: [212, 82, 120, 90, 44, 139, 71, 87, 183, 191, 100, 83, 237, 192, 22, 89, 28, 11, 38, 243, 88, 106, 143, 154, 124, 111, 48, 160, 93, 236, 81, 93] }, input: Input { data: [12, 64, 3, 7, 125] } }
[sp1] groth16 circuit artifacts already seem to exist at /root/.sp1/circuits/groth16/v4.0.0-rc.3. if you want to re-download them, delete the directory
Setting environment variables took 7.97µs
Reading witness file took 134.63µs
Deserializing JSON data took 8.271939ms
Generating witness took 308.424368ms
01:42:26 DBG constraint system solver done nbConstraints=8173052 took=2353.370621
01:42:45 DBG prover done acceleration=none backend=groth16 curve=bn254 nbConstraints=8173052 took=19011.933463
Generating proof took 21.365459855s
ignoring uninitialized slice: Vars []frontend.Variable
ignoring uninitialized slice: Vars []frontend.Variable
ignoring uninitialized slice: Vars []frontend.Variable
01:42:45 DBG verifier done backend=groth16 curve=bn254 took=0.8777
[2025-04-24T01:42:45Z DEBUG pda_proxy::internal::runner] COMPLETE proof: Job { anchor: Anchor { data: [212, 82, 120, 90, 44, 139, 71, 87, 183, 191, 100, 83, 237, 192, 22, 89, 28, 11, 38, 243, 88, 106, 143, 154, 124, 111, 48, 160, 93, 236, 81, 93] }, input: Input { data: [12, 64, 3, 7, 125] } }
[2025-04-24T01:42:45Z INFO  pda_proxy::internal::runner] 🎉 Job { anchor: Anchor { data: [212, 82, 120, 90, 44, 139, 71, 87, 183, 191, 100, 83, 237, 192, 22, 89, 28, 11, 38, 243, 88, 106, 143, 154, 124, 111, 48, 160, 93, 236, 81, 93] }, input: Input { data: [12, 64, 3, 7, 125] } } Finished!
[2025-04-24T01:42:45Z DEBUG pda_proxy::internal::runner] ZK request sent

Note that I didn't need go in the final runtime, only at compile time. 😁

I will need to do some testing to see if this changes performance at all, as I can build & run fine without the feature flag and go tooling 🤔 . I hope this native feature enabled is actually more performant at runtime? Would that be a reasonable expectation?

[leruaa]

Closing as the original issue seems solved