-
Notifications
You must be signed in to change notification settings - Fork 31
Open
Labels
ClientSee https://miro.com/app/board/uXjVNZ03E-c=/ for detailsSee https://miro.com/app/board/uXjVNZ03E-c=/ for detailsbugSomething isn't workingSomething isn't working
Description
Found and captured in release tests + async profiler.
Found in bde9873 - fix(slotqueue): simplify slot queue workers (#1224)
Storage providing nodes exceed main thread blocking threshold when they receive a proof challenge.
The slot size in question is 8MB. The size does not seem to be causing trouble.
The proving system sample count is 5. We can see in the log below that the 5 samples are successfully collected from the block trees before the crash. The blocking operation must occur some time later.
TRC 2025-05-16 08:04:00.991+00:00 Received proof challenge topics="codex node" tid=1 challenge=0x81a6..b1c1 cid=zDvZRwzmCxeoQWhR9DcMWXcAERkdg1WvEfGoPe1RYeJxwCzk5GnV slot=0 count=6102
TRC 2025-05-16 08:04:00.991+00:00 Prover enabled topics="codex node" tid=1 challenge=0x81a6..b1c1 cid=zDvZRwzmCxeoQWhR9DcMWXcAERkdg1WvEfGoPe1RYeJxwCzk5GnV slot=0 count=6103
TRC 2025-05-16 08:04:00.991+00:00 Retrieving manifest for cid topics="codex node" tid=1 cid=zDv*zk5GnV count=6104
TRC 2025-05-16 08:04:00.991+00:00 Decoding manifest for cid topics="codex node" tid=1 cid=zDv*zk5GnV count=6105
TRC 2025-05-16 08:04:00.991+00:00 Decoded manifest topics="codex node" tid=1 cid=zDv*zk5GnV count=6106
TRC 2025-05-16 08:04:00.991+00:00 Received proof challenge topics="codex prover" tid=1 challenge=0x81a6..b1c1 cid=zDz*s7dUfp slot=0 count=6107
TRC 2025-05-16 08:04:01.001+00:00 Creating slots builder topics="codex prover" tid=1 count=6108
TRC 2025-05-16 08:04:01.001+00:00 Collecting input for proof topics="codex prover" tid=1 count=6109
TRC 2025-05-16 08:04:01.001+00:00 Retrieving sample from block tree topics="codex prover" tid=1 count=6110
TRC 2025-05-16 08:04:01.001+00:00 Building block tree topics="codex prover" tid=1 count=6111
TRC 2025-05-16 08:04:01.011+00:00 Retrieving sample from block tree topics="codex prover" tid=1 count=6112
TRC 2025-05-16 08:04:01.021+00:00 Building block tree topics="codex prover" tid=1 count=6113
TRC 2025-05-16 08:04:01.031+00:00 Retrieving sample from block tree topics="codex prover" tid=1 count=6114
TRC 2025-05-16 08:04:01.031+00:00 Building block tree topics="codex prover" tid=1 count=6115
TRC 2025-05-16 08:04:01.031+00:00 Returning empty digest tree for pad block topics="codex prover" tid=1 count=6116
TRC 2025-05-16 08:04:01.031+00:00 Retrieving sample from block tree topics="codex prover" tid=1 count=6117
TRC 2025-05-16 08:04:01.031+00:00 Building block tree topics="codex prover" tid=1 count=6118
TRC 2025-05-16 08:04:01.031+00:00 Returning empty digest tree for pad block topics="codex prover" tid=1 count=6119
TRC 2025-05-16 08:04:01.031+00:00 Retrieving sample from block tree topics="codex prover" tid=1 count=6120
TRC 2025-05-16 08:04:01.031+00:00 Building block tree topics="codex prover" tid=1 count=6121
ERR 2025-05-16 08:04:06.611+00:00 Unhandled exception in async proc, aborting topics="codex" tid=1 msg="Execution time of future is too high (prove[prover.nim:51]) time: 5s570ms795us834ns threshold: 200ms" count=6122
Warning: If blocking tree building calls are the cause of this, then this issue probably has the same root cause as #1226
Metadata
Metadata
Assignees
Labels
ClientSee https://miro.com/app/board/uXjVNZ03E-c=/ for detailsSee https://miro.com/app/board/uXjVNZ03E-c=/ for detailsbugSomething isn't workingSomething isn't working