prover exit (255)

[tony-armstrong]

1. After the full prover was started and send a tx, no tx occurs
2. After 3 hours, the prover exited, which also caused the sequencer to exit
3. There was an error level error before prover exited: {"level":"error","ts":"1704132171.499945","msg":"watchdogThread() timeout passed","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}

r).Start\n\t/src/sequencer/finalizer.go:194"}                                                                                                  │zkevm-prover  | 2024-01-01T18:02:51.498608540Z {"level":"info","ts":"1704132171.498553","msg":"ExecutorServiceImpl::ProcessBatch() returns result=1 old_state_root=a0c91eac11c11e4a812a98377bf6b606328a6a04314776872b521de
zkevm-sequencer  | 2024-01-01T18:03:33.509603670Z {"level":"error","ts":1704132213.509329,"caller":"state/batch.go:350","msg":"Error s.executor│746e60c19 new_state_root=0xa0c91eac11c11e4a812a98377bf6b606328a6a04314776872b521de746e60c19 new_acc_input_hash=0xaf499fcafad7587c6211bf771199ab357de5ce5baedefa8f2ec53349d79e034c new_local_exit_root=0x000000000000000000
Client.ProcessBatch: rpc error: code = Unavailable desc = error reading from server: read tcp 172.20.0.9:52522->172.20.0.6:50071: read: connect│0000000000000000000000000000000000000000000000 old_batch_num=27 steps=119 gasUsed=0 counters.keccakF=3 counters.poseidonG=28 counters.paddingPG=0 counters.memAlign=0 counters.arith=0 counters.binary=3 flush_id=585363 l
ion reset by peer","pid":1,"version":"v0.4.0-RC6-27-gca7a25af","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*State).sendBatchRequ│ast_sent_flush_id=585362 nTxs=0","pid":"a01ce77","tid":"27ed640","version":"v3.0.2","context_id":"f4f41682-a44c-400c-b272-32baff6f480d"}
estToExecutor\n\t/src/state/batch.go:350\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessBatch\n\t/src/state/batch.go:188\ngithub.│zkevm-prover  | 2024-01-01T18:02:51.500096025Z {"level":"error","ts":"1704132171.499945","msg":"watchdogThread() timeout passed","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).processTransaction\n\t/src/sequencer/finalizer.go:688\ngithub.com/0xPolygonHermez/zkevm-n│zkevm-prover  | 2024-01-01T18:02:51.500709793Z {"level":"info","ts":"1704132171.500650","msg":"ExecutorServiceImpl::ProcessBatch() done counter=585342 B=0 TX=0 gas=0 time=0.000747 TP=0.000000B/s=0.000000TX/s=0.000000ga
ode/sequencer.(*finalizer).newWIPBatch\n\t/src/sequencer/finalizer.go:500\ngithub.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).finaliz│s/s=-nangas/B totalTP(10s)=22.395022B/s=0.199956TX/s=4199.066548gas/s=187.500000gas/B totalTP(ever)=33.387400B/s=0.298102TX/s=6260.137427gas/s=187.500000gas/B totalTime=456.923254 filedesc=318","pid":"a01ce77","tid":"2
eBatch\n\t/src/sequencer/finalizer.go:429\ngithub.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).finalizeBatches\n\t/src/sequencer/final│7ed640","version":"v3.0.2","context_id":"f4f41682-a44c-400c-b272-32baff6f480d"}
izer.go:381\ngithub.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).Start\n\t/src/sequencer/finalizer.go:194"}                           │zkevm-prover  | 2024-01-01T18:02:51.501905430Z {"level":"info","ts":"1704132171.501843","msg":"CALL STACK","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
zkevm-sequencer  | 2024-01-01T18:03:33.509702730Z {"level":"error","ts":1704132213.509401,"caller":"state/batch.go:351","msg":"Error s.executor│zkevm-prover  | 2024-01-01T18:02:51.501933646Z {"level":"info","ts":"1704132171.501860","msg":"0: call=zkProver(_Z14printCallStackv+0x45) [0x561b705dd0c5]","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
Client.ProcessBatch response: <nil>","pid":1,"version":"v0.4.0-RC6-27-gca7a25af","stacktrace":"github.com/0xPolygonHermez/zkevm-node/state.(*St│zkevm-prover  | 2024-01-01T18:02:51.501942506Z {"level":"info","ts":"1704132171.501868","msg":"1: call=zkProver(_Z11exitProcessv+0x16) [0x561b70607626]","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
ate).sendBatchRequestToExecutor\n\t/src/state/batch.go:351\ngithub.com/0xPolygonHermez/zkevm-node/state.(*State).ProcessBatch\n\t/src/state/bat│zkevm-prover  | 2024-01-01T18:02:51.501950790Z {"level":"info","ts":"1704132171.501874","msg":"2: call=zkProver(_Z14watchdogThreadPv+0x118) [0x561b705f8738]","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
ch.go:188\ngithub.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).processTransaction\n\t/src/sequencer/finalizer.go:688\ngithub.com/0xPol│zkevm-prover  | 2024-01-01T18:02:51.501958840Z {"level":"info","ts":"1704132171.501879","msg":"3: call=/lib/x86_64-linux-gnu/libc.so.6(+0x94ac3) [0x7fdc6eb0cac3]","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
ygonHermez/zkevm-node/sequencer.(*finalizer).newWIPBatch\n\t/src/sequencer/finalizer.go:500\ngithub.com/0xPolygonHermez/zkevm-node/sequencer.(*│zkevm-prover  | 2024-01-01T18:02:51.501966655Z {"level":"info","ts":"1704132171.501884","msg":"4: call=/lib/x86_64-linux-gnu/libc.so.6(clone+0x44) [0x7fdc6eb9dbf4]","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
finalizer).finalizeBatch\n\t/src/sequencer/finalizer.go:429\ngithub.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).finalizeBatches\n\t/s│zkevm-prover  | 2024-01-01T18:02:51.502295458Z {"level":"info","ts":"1704132171.502244","msg":"MEMORY INFO
rc/sequencer/finalizer.go:381\ngithub.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).Start\n\t/src/sequencer/finalizer.go:194"}         │zkevm-prover  | 2024-01-01T18:02:51.502307561Z     MemTotal: 773382.167969 MB
zkevm-sequencer  | 2024-01-01T18:03:33.509730793Z {"level":"info","ts":1704132213.5095015,"caller":"state/batch.go:360","msg":"Batch: 441 took │zkevm-prover  | 2024-01-01T18:02:51.502314061Z     MemFree: 70961.128906 MB
31.473725952s to be processed by the executor ","pid":1,"version":"v0.4.0-RC6-27-gca7a25af"}                                                   │zkevm-prover  | 2024-01-01T18:02:51.502320328Z     MemAvailable: 305247.921875 MB
zkevm-sequencer  | 2024-01-01T18:03:33.518628266Z panic: runtime error: invalid memory address or nil pointer dereference                      │zkevm-prover  | 2024-01-01T18:02:51.502326380Z     Buffers: 1193.609375 MB
zkevm-sequencer  | 2024-01-01T18:03:33.518661872Z [signal SIGSEGV: segmentation violation code=0x1 addr=0xc1 pc=0x1305ead]                     │zkevm-prover  | 2024-01-01T18:02:51.502332350Z     Cached: 231007.414062 MB
zkevm-sequencer  | 2024-01-01T18:03:33.518671715Z                                                                                              │zkevm-prover  | 2024-01-01T18:02:51.502338304Z     SwapCached: 0.000000 MB
zkevm-sequencer  | 2024-01-01T18:03:33.518680031Z goroutine 325 [running]:                                                                     │zkevm-prover  | 2024-01-01T18:02:51.502344542Z     SwapTotal: 8191.996094 MB
zkevm-sequencer  | 2024-01-01T18:03:33.518688923Z github.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).processTransaction(0xc0006d2c00,│zkevm-prover  | 2024-01-01T18:02:51.502350891Z     SwapFree: 8191.996094 MB
 {0x1c1ca98, 0x2904820}, 0x0, 0x1)                                                                                                             │zkevm-prover  | 2024-01-01T18:02:51.502357316Z     VM: 481281.332031 MB
zkevm-sequencer  | 2024-01-01T18:03:33.518699537Z       /src/sequencer/finalizer.go:695 +0x15ed                                                │zkevm-prover  | 2024-01-01T18:02:51.502363670Z     RSS: 111.186749 MB","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
zkevm-sequencer  | 2024-01-01T18:03:33.518710006Z github.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).newWIPBatch(0xc0006d2c00, {0x1c1│zkevm-prover  | 2024-01-01T18:02:51.502407911Z {"level":"info","ts":"1704132171.502360","msg":"PROCESS INFO
ca98?, 0x2904820})                                                                                                                             │zkevm-prover  | 2024-01-01T18:02:51.502424165Z     Pid: 1
zkevm-sequencer  | 2024-01-01T18:03:33.518720998Z       /src/sequencer/finalizer.go:500 +0x229                                                 │zkevm-prover  | 2024-01-01T18:02:51.502430734Z     User time: 568298.200000 s
zkevm-sequencer  | 2024-01-01T18:03:33.518730960Z github.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).finalizeBatch(0xc0006d2c00, {0x1│zkevm-prover  | 2024-01-01T18:02:51.502437616Z     Kernel time: 47834.620000 s
c1ca98, 0x2904820})                                                                                                                            │zkevm-prover  | 2024-01-01T18:02:51.502443808Z     Total time: 616132.820000 s
zkevm-sequencer  | 2024-01-01T18:03:33.518741247Z       /src/sequencer/finalizer.go:429 +0x88                                                  │zkevm-prover  | 2024-01-01T18:02:51.502449769Z     Num threads: 207
zkevm-sequencer  | 2024-01-01T18:03:33.518751080Z github.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).finalizeBatches(0xc0006d2c00, {0│zkevm-prover  | 2024-01-01T18:02:51.502455802Z     Virtual mem: 481281 MB","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
x1c1ca98, 0x2904820})                                                                                                                          │zkevm-prover  | 2024-01-01T18:02:56.504209773Z {"level":"info","ts":"1704132176.504091","msg":"--> DATABASE_PROGRAM_CACHE_DESTRUCTOR starting...","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
zkevm-sequencer  | 2024-01-01T18:03:33.518759718Z       /src/sequencer/finalizer.go:381 +0x38b                                                 │zkevm-prover  | 2024-01-01T18:02:56.504289635Z {"level":"info","ts":"1704132176.504234","msg":"<-- DATABASE_PROGRAM_CACHE_DESTRUCTOR done: 0.000139 s","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
zkevm-sequencer  | 2024-01-01T18:03:33.518767686Z github.com/0xPolygonHermez/zkevm-node/sequencer.(*finalizer).Start(0xc0006d2c00, {0x1c1ca98?,│zkevm-prover  | 2024-01-01T18:02:56.504309495Z {"level":"info","ts":"1704132176.504244","msg":"--> DATABASE_CACHE_DESTRUCTOR starting...","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
 0x2904820}, 0x577b07?, 0xc000000d80)                                                                                                          │zkevm-prover  | 2024-01-01T18:02:56.504318522Z {"level":"info","ts":"1704132176.504253","msg":"<-- DATABASE_CACHE_DESTRUCTOR done: 0.000007 s","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
zkevm-sequencer  | 2024-01-01T18:03:33.518777221Z       /src/sequencer/finalizer.go:194 +0x2df                                                 │zkevm-prover  | 2024-01-01T18:02:56.504337600Z {"level":"info","ts":"1704132176.504259","msg":"--> DATABASE_MT_CACHE_DESTRUCTOR starting...","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
zkevm-sequencer  | 2024-01-01T18:03:33.518784975Z created by github.com/0xPolygonHermez/zkevm-node/sequencer.(*Sequencer).Start in goroutine 29│zkevm-prover  | 2024-01-01T18:02:56.504681156Z {"level":"info","ts":"1704132176.504623","msg":"<-- DATABASE_MT_CACHE_DESTRUCTOR done: 0.000359 s","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
8                                                                                                                                              │zkevm-prover  | 2024-01-01T18:02:56.504697434Z {"level":"info","ts":"1704132176.504634","msg":"--> DATABASE_CACHE_DESTRUCTOR starting...","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
zkevm-sequencer  | 2024-01-01T18:03:33.518793389Z       /src/sequencer/sequencer.go:113 +0x89e                                                 │zkevm-prover  | 2024-01-01T18:02:56.504706041Z {"level":"info","ts":"1704132176.504641","msg":"<-- DATABASE_CACHE_DESTRUCTOR done: 0.000006 s","pid":"a01ce77","tid":"c3fb640","version":"v3.0.2"}
b2-full-prover test git:(dev) ✗                                                                                                                │b2-full-prover test git:(dev) ✗
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
WARN[0000] The "ZKEVM_NODE_EXECUTOR_URI" variable is not set. Defaulting to a blank string.
NAME                    IMAGE                                                                     COMMAND                  SERVICE                 CREATED        STATUS                     PORTS
zkevm-aggregator        ghcr.io/b2network/b2-zkevm-node:dev-v0.4.0-RC6-20231222-091319-ca7a25af   "/bin/sh -c '/app/b2…"   zkevm-aggregator        11 hours ago   Up 11 hours                8123/tcp, 0.0.0.0:50081->50081/tcp, :::50081->50081/tcp, 0.0.0.0:9093->9091/tcp, :::9093->9091/tcp
zkevm-approve           ghcr.io/b2network/b2-zkevm-node:dev-v0.4.0-RC6-20231222-091319-ca7a25af   "/bin/sh -c '/app/b2…"   zkevm-approve           12 hours ago   Exited (0) 11 hours ago
zkevm-eth-tx-manager    ghcr.io/b2network/b2-zkevm-node:dev-v0.4.0-RC6-20231222-091319-ca7a25af   "/bin/sh -c '/app/b2…"   zkevm-eth-tx-manager    11 hours ago   Up 11 hours                8123/tcp, 0.0.0.0:9094->9091/tcp, :::9094->9091/tcp
zkevm-event-db          postgres:15-alpine                                                        "docker-entrypoint.s…"   zkevm-event-db          12 hours ago   Up 12 hours                0.0.0.0:5435->5432/tcp, :::5435->5432/tcp
zkevm-json-rpc          ghcr.io/b2network/b2-zkevm-node:dev-v0.4.0-RC6-20231222-091319-ca7a25af   "/bin/sh -c '/app/b2…"   zkevm-json-rpc          11 hours ago   Up 11 hours                0.0.0.0:8123->8123/tcp, :::8123->8123/tcp, 0.0.0.0:8133->8133/tcp, :::8133->8133/tcp, 0.0.0.0:9091->9091/tcp, :::9091->9091/tcp
zkevm-l2gaspricer       ghcr.io/b2network/b2-zkevm-node:dev-v0.4.0-RC6-20231222-091319-ca7a25af   "/bin/sh -c '/app/b2…"   zkevm-l2gaspricer       11 hours ago   Up 11 hours                8123/tcp
zkevm-metrics           prom/prometheus:v2.39.1                                                   "/bin/prometheus --c…"   zkevm-metrics           11 hours ago   Up 11 hours                0.0.0.0:9090->9090/tcp, :::9090->9090/tcp
zkevm-mock-l1-network   ghcr.io/b2network/b2-node:20231222-145531-18655e6-with-datadir-6baf597    "ethermintd start"       zkevm-mock-l1-network   12 hours ago   Up 12 hours                0.0.0.0:8545-8546->8545-8546/tcp, :::8545-8546->8545-8546/tcp, 0.0.0.0:8126->8125/tcp, :::8126->8125/tcp, 0.0.0.0:26658->26657/tcp, :::26658->26657/tcp
zkevm-pool-db           postgres:15-alpine                                                        "docker-entrypoint.s…"   zkevm-pool-db           12 hours ago   Up 12 hours                0.0.0.0:5433->5432/tcp, :::5433->5432/tcp
zkevm-prover            ghcr.io/b2network/b2-zkevm-prover:20231127-164129-7accb0b                 "zkProver -c /usr/sr…"   zkevm-prover            12 hours ago   Exited (255) 8 hours ago
zkevm-sequence-sender   ghcr.io/b2network/b2-zkevm-node:dev-v0.4.0-RC6-20231222-091319-ca7a25af   "/bin/sh -c '/app/b2…"   zkevm-sequence-sender   11 hours ago   Up 11 hours                8123/tcp
zkevm-sequencer         ghcr.io/b2network/b2-zkevm-node:dev-v0.4.0-RC6-20231222-091319-ca7a25af   "/bin/sh -c '/app/b2…"   zkevm-sequencer         11 hours ago   Exited (2) 8 hours ago
zkevm-state-db          postgres:15-alpine                                                        "docker-entrypoint.s…"   zkevm-state-db          12 hours ago   Up 12 hours                0.0.0.0:5432->5432/tcp, :::5432->5432/tcp
zkevm-sync              ghcr.io/b2network/b2-zkevm-node:dev-v0.4.0-RC6-20231222-091319-ca7a25af   "/bin/sh -c '/app/b2…"   zkevm-sync              11 hours ago   Up 11 hours                8123/tcp, 0.0.0.0:9095->9091/tcp, :::9095->9091/tcp