[ZK Stack] - Proof Generation for a Job Failing

[sagarparker]

Team or Project

No response

Module Affected

• System contracts (era-system-contracts)
• Smart contracts (era-contracts)
• zkEVM (era-zk_evm)
• Server (zksync-era)
• Prover (era-boojum)
• Fee withdrawer (era-fee-withdrawer)
• Block Explorer (clock-explorer)
• Portal - Wallet + Bridge (dapp-portal)
• Other (please specify below)

Rust Version

1.83.0

Operating System

• Ubuntu
• MacOS
• Windows
• Other (Please specify below)

Issue Description

Hi,

I’ve been following the guide below to run the proof generation process:
Proving Batch Guide

However, some jobs are failing with the following error in the logs from the witness vector generator:

2025-01-04T04:11:28.394220Z ERROR zksync_queued_job_processor: Error occurred while processing FriGpuProver job 32757: "get_setup_data()"

After 10 retries, the job is marked as failed, which causes proof generation for the entire batch to fail.

How can I resolve this issue?

OS : Ubuntu 22.04
rust : 1.83.0
CUDA : 12.4
Driver Version : 550.127.05

Expected Behavior

Proofs are generated for a job as usual.

Current Behavior

Proof generation fails for a job with error :

2025-01-04T04:11:28.394220Z ERROR zksync_queued_job_processor: Error occurred while processing FriGpuProver job 32757: "get_setup_data()"

Repository Link (if applicable)

No response

Additional Details

No response

Prior Research

• Yes, I searched for similar issues and did not find any matching ones.

[gleenai[bot]]

It seems like the issue you're encountering with the proof generation process in the ZKsync Prover (era-boojum) module might be related to the setup or configuration of your environment, specifically around the handling of setup data for the FriGpuProver job.

Here are a few steps you can take to troubleshoot and potentially resolve this issue:

1. Check CUDA and Driver Compatibility: Ensure that your CUDA version (12.4) is fully compatible with your GPU drivers (550.127.05). Mismatched versions can sometimes cause unexpected errors.

2. Review Configuration Files: Double-check the configuration files related to the FriGpuProver to ensure all paths and settings are correctly specified. This includes checking any environment variables or dependencies that might affect the get_setup_data() function.

3. Update and Rebuild: Make sure all your system packages and dependencies are up to date. After updating, rebuild your project to ensure all components are synchronized.

4. Consult Documentation and Community: Review the Proving documentation for any additional configuration steps or known issues. You can also seek further assistance or share your issue in the ZKsync developers discussions on Github for more specific guidance or similar experiences from other developers.

If these steps do not resolve the issue, it might be beneficial to provide more detailed logs or error messages to the community or support for more targeted help.

Your feedback helps us serve you better. If this fully addresses your question, please give it a thumbs up. If it doesn't, please give it a thumbs down. This will inform the community and team so we can improve our support.

[sagarparker]

I was able to resolve this issue by manually requeuing the stuck jobs using prover_cli.