Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Skaled doesn't exit gracefully after rotation delay time #1415

Closed
DmytroNazarenko opened this issue Feb 14, 2023 · 9 comments · Fixed by #1427, #1440 or #1543
Closed

Skaled doesn't exit gracefully after rotation delay time #1415

DmytroNazarenko opened this issue Feb 14, 2023 · 9 comments · Fixed by #1427, #1440 or #1543
Assignees
@dimalit
Labels
bug Something isn't working epic:node-rotation release:2.2

Comments

@DmytroNazarenko
Copy link
Collaborator

DmytroNazarenko commented Feb 14, 2023 •
edited
Loading

Environment
20 active nodes on the network
8 schains up and running
Skaled: 3.16.0-beta.9
Admin: 2.4.0-beta.2
Skale Manager version: 1.9.3-beta.3

Steps to Reproduce/Description

  1. Start Node Rotation procedure
  2. Wait till Rotation Delay time
  3. Check skaled logs of staying nodes

Expected behavior
Skaled should stop mining blocks, exit gracefully and create new container

Actual result
Skaled stops mining blocks, but exits forcefully ATTENTION: Forcefully shutting down consensus!

2023-02-13 18:33:20.836270   Block stats:BN:20780:BTS:1676313186:TXS:0:HDRS:11:LOGS:40:SENGS:1:TXRS:0:BLCKS:3:ACCS:101:BQS:1:BDS:5892:TSS:0:UTX:0:VTX:0:CMM:6336053
2023-02-13 18:33:20.836354   Block 20781 DB usage is 1110. Piece DB usage is 23164859 bytes
[2023-02-13 18:33:31.612] [21:main] [info] 20781:CONSENSUS_STARTED:PROPOSING: 0101101111111100
[2023-02-13 18:33:32.012] [21:consensus] [info] 20781:CONSENSUS_COMPLETED:STATS::165501:83394:41833:20815:10495:5274:2564:1271:647:317:126:61:26:8:3:3
[2023-02-13 18:33:32.012] [21:consensus] [info] 20781:BLOCK_DECIDED:PROPOSER:2:BID:20782:STATS:|1|D0R2P49L109|2|D1R1P46L113|3|*|4|D1R1P40L113|5|D1R0P38L73|6|*|7|D1R0P33L81|8|D1R0P30L102|9|*|10|D1R0P27L83|11|D1R0P20L84|12|D1R1P18L112|13|D1R1P15L110|14|D1R0P12L108|15|D0R0P10L108|16|*|| Now signing block ...
[2023-02-13 18:33:32.181] [21:consensus] [info] 20781:BLOCK_DECIDED_AND_SIGNED:PRPSR:2:BID:20782:SIG:11845290796889670045903720720803104781332187429090716666696362278783394702912:6073141764117561600988161694955375182973633714439312178440327419506792644074:14291781364203370824295739155210477086393881210815244127591789292496254648638:0
[2023-02-13 18:33:32.190] [21:main] [info] 20781:BLOCK_COMMITED: PRPSR:2:BID: 20782:ROOT:68636725771493069109204820326378049879999328873399425990456736699998033979283:HASH:0b1fec5c:BLOCK_TXS:0:DMSG:26:MPRPS:3:RPRPS:45:TXS:46:TXLS:48:MGS:119:INSTS:325:BPS:0:HDRS:35:SOCK:0:FDS:409:PRT:10775:BTA:12206:BSA:0:TPS:0:LWT:0:LRT:0:LWC:2275223:LRC:3094161:KNWN:46:CONS:0:DSDS:0:SET:77:SBT:32:SEC:20780:SBC:61300:ZSC:1:EPT:3:STAMP:1676313210.994
[2023-02-13 18:33:32.199] [21:consensus] [info] 20781:Consensus engine exiting: blocking exit exitGracefully called by skaled
Here is exitGracefullyBlocking() stack trace for your information:
2023-02-13 18:33:32.198645   Block sealed #20782 (#581a9815…)
2023-02-13 18:33:32.198693   Block stats:BN:20781:BTS:1676313198:TXS:0:HDRS:11:LOGS:40:SENGS:1:TXRS:0:BLCKS:3:ACCS:101:BQS:1:BDS:5893:TSS:0:UTX:0:VTX:0:CMM:6336891
2023-02-13 18:33:32.198824   Block 20782 DB usage is 1110. Piece DB usage is 23165969 bytes
2023-02-13 18:33:32.199676   Setting ExitTimeReached = true
2023-02-13 18:33:32.199693   Skaled status: setExitState: ExitTimeReached to true
 0# 0x00005646A61E3CE0 in /skaled/skaled
 1# 0x00005646A5F2FD6E in /skaled/skaled
 2# 0x00005646A5F22069 in /skaled/skaled
 3# 0x00005646A62708BD in /skaled/skaled
 4# 0x00005646A627B051 in /skaled/skaled
 5# 0x00005646A627CFD9 in /skaled/skaled
 6# 0x00005646A627E318 in /skaled/skaled
 7# 0x00005646A62448E4 in /skaled/skaled
 8# 0x00005646A6249AE3 in /skaled/skaled
 9# 0x00005646A62717E6 in /skaled/skaled
10# 0x00007FF70E7064C0 in /usr/lib/x86_64-linux-gnu/libstdc++.so.6
11# start_thread at /build/glibc-CVJwZb/glibc-2.27/nptl/pthread_create.c:463
12# 0x00007FF70DDAC61F in /lib/x86_64-linux-gnu/libc.so.6

2023-02-13 18:33:32.275961   Skaled status: setExitState: StartAgain to false
[2023-02-13 18:33:32.276] [config] [info] Consensus engine exiting: exitGracefullyAsync called by skaled
[2023-02-13 18:33:32.276] [config] [info] exitGracefullyAsync running
[2023-02-13 18:33:32.276] [config] [info] Node exit called
[2023-02-13 18:33:32.276] [config] [info] Node::exit() requested
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: close all sockets called
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: server conditional vars notified
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: agent conditional vars notified
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: block proposal socket touched
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: catchup socket touched
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: SGXZMQClient exiting
[2023-02-13 18:33:32.276] [config] [info] Exiting SgxZmqClient
[2023-02-13 18:33:32.276] [config] [info] Shutting down SgxZmq context
[2023-02-13 18:33:32.276] [config] [info] Shut down SgxZmq context
[2023-02-13 18:33:32.276] [config] [info] Closing SgxZmq client sockets
[2023-02-13 18:33:32.276] [config] [info] Exited SgxZmqClient
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: SGXZMQClient exited
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: closeAndCleanupAll consensus zmq sockets
[2023-02-13 18:33:32.276] [config] [info] Cleaning up ZMQ sockets
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: closing ZMQ send sockets
[2023-02-13 18:33:32.276] [config] [info] consensus engine exiting: closing receive sockets
[2023-02-13 18:33:32.276] [config] [info] Closing ZMQ context
[2023-02-13 18:33:32.277] [21:main] [info] 20781:Consensus exiting: Node:Exit requested
[2023-02-13 18:33:32.277] [21:main] [info] 20781:Consensus exiting: Node:Exit requested
[2023-02-13 18:33:32.277] [21:main] [info] 20781:Consensus exiting: Node:Exit requested
2023-02-13 18:33:32.276147   Skaled status: setExitState: StartFromSnapshot to false
2023-02-13 18:33:32.276794   Skaled status: setExitState: ClearDataDir to false
[2023-02-13 18:33:32.280] [config] [info] Closed ZMQ
[2023-02-13 18:33:32.280] [config] [info] Node exit completed
2023-02-13 18:33:32.299720   Set rotation time to 1676313204
[2023-02-13 18:33:32.526] [21:main] [info] 20781:Exited Oracle worker thread 1
2023-02-13 18:33:34.277479   Rotation is completed. Instance is exiting
[2023-02-13 18:33:34.277] [21:main] [error] 20781:Exception in Schain::messageThreadProcessingLoop
[2023-02-13 18:33:34.277] [21:main] [info] 20781:Consensus exiting: Node:Exit requested
2023-02-13 18:33:46.072612   OK, server stopped on port 10450=0x000028d2
2023-02-13 18:33:55.854507   OK, server stopped on port 10455=0x000028d7
2023-02-13 18:33:55.855859   Skaled status: setSubsystemRunning: Rpc to false

2023-02-13 18:37:56.202054   ATTENTION: Forcefully shutting down consensus!
2023-02-13 18:37:56.202089   1 before exitGracefully()
[2023-02-13 18:37:56.202] [21:consensus] [info] 20781:Consensus engine exiting: blocking exit exitGracefully called by skaled
Here is exitGracefullyBlocking() stack trace for your information:
 0# 0x00005646A61E3CE0 in /skaled/skaled
 1# 0x00005646A5F254FD in /skaled/skaled
 2# 0x00005646A5EC1CE0 in /skaled/skaled
 3# 0x00005646A5D73E42 in /skaled/skaled
 4# 0x00007FF70DCACC87 in /lib/x86_64-linux-gnu/libc.so.6
 5# 0x00005646A5DCAE9A in /skaled/skaled

2023-02-13 18:37:56.202097   Exit requested with signal 15 and exit code 0
[2023-02-13 18:37:56.233] [config] [info] Consensus engine exiting: exitGracefullyAsync called by skaled
.
2023-02-13 18:37:56.233874   2 after exitGracefully()
2023-02-13 18:37:56.233902   3 after wait loop
2023-02-13 18:37:56.233946   4 before dtor
2023-02-13 18:37:56.250197   Blockchain is closed
2023-02-13 18:37:56.250352   Skaled status: setSubsystemRunning: Blockchain to false

2023-02-13 18:37:56.250682   Blockchain is closed

SIGNAL-HANDLER: Will wait for stop actions complete...


SIGNAL-HANDLER: Done waiting for stop actions] complete

[2023-02-13 18:37:56.617] [21:consensus] [info] 20781:Consensus engine exiting: exitGracefullyBlocking called by skaled
Here is exitGracefullyBlocking() stack trace for your information:
 0# 0x00005646A61E3521 in /skaled/skaled
 1# 0x00005646A61E3702 in /skaled/skaled
 2# 0x00005646A61E3AC9 in /skaled/skaled
 3# 0x00005646A5F234D9 in /skaled/skaled
 4# 0x00005646A5F1CED0 in /skaled/skaled
 5# 0x00007FF70DCCE031 in /lib/x86_64-linux-gnu/libc.so.6
 6# 0x00007FF70DCCE12A in /lib/x86_64-linux-gnu/libc.so.6
 7# 0x00007FF70DCACC8E in /lib/x86_64-linux-gnu/libc.so.6
 8# 0x00005646A5DCAE9A in /skaled/skaled

ConsensusEngine terminated.
@dimalit dimalit linked a pull request Feb 20, 2023 that will close this issue
IS-711 Release consensusWorkingMutex in createBlock #1427
Merged
@PolinaKiporenko PolinaKiporenko moved this from To Do to In Progress in SKALE Engineering 🚀 Feb 21, 2023
@PolinaKiporenko PolinaKiporenko moved this from In Progress to Ready For Release Candidate in SKALE Engineering 🚀 Feb 21, 2023
@DmytroNazarenko DmytroNazarenko mentioned this issue Feb 24, 2023
Merged
@DmytroNazarenko DmytroNazarenko linked a pull request Feb 26, 2023 that will close this issue
Revert "IS-711 Release consensusWorkingMutex in createBlock" #1440
Merged
@DmytroNazarenko DmytroNazarenko mentioned this issue Feb 27, 2023
Merged
@github-project-automation github-project-automation bot moved this from Ready For Release Candidate to Ready For Pickup in SKALE Engineering 🚀 Mar 24, 2023
@DmytroNazarenko DmytroNazarenko added epic:node-rotation bug Something isn't working labels Mar 24, 2023
@DmytroNazarenko DmytroNazarenko changed the title Internal issue #711 Skaled doesn't exit gracefully after rotation delay time Mar 24, 2023
@DmytroNazarenko
Copy link
Collaborator Author

DmytroNazarenko commented Mar 24, 2023 •
edited by dimalit
Loading

Needs new architecture from #1425 first
Likely, will be fixed by it

@dimalit dimalit mentioned this issue Apr 19, 2023
Closed
1 task
@dimalit dimalit moved this from Ready For Pickup to Blocked in SKALE Engineering 🚀 May 18, 2023
@DmytroNazarenko DmytroNazarenko moved this from Blocked to In Progress in SKALE Engineering 🚀 Jun 16, 2023
@olehnikolaiev olehnikolaiev mentioned this issue Jun 16, 2023
Merged
@olehnikolaiev olehnikolaiev moved this from In Progress to Code Review in SKALE Engineering 🚀 Jun 16, 2023
@github-project-automation github-project-automation bot moved this from Code Review to Ready For Release Candidate in SKALE Engineering 🚀 Jun 20, 2023
@DmytroNazarenko DmytroNazarenko mentioned this issue Jun 20, 2023
Merged
@DmytroNazarenko
Copy link
Collaborator Author

3.17.0-beta.1

@DmytroNazarenko DmytroNazarenko moved this from Ready For Release Candidate to Merged To Release Candidate in SKALE Engineering 🚀 Jun 21, 2023
@EvgeniyZZ EvgeniyZZ moved this from Merged To Release Candidate to QA in SKALE Engineering 🚀 Jun 21, 2023
@yatsunastya
Copy link

Blocked by https://github.com/skalenetwork/internal-support/issues/844

@yatsunastya
Copy link

checked ✅
Admin: 2.5.1-develop.4
Skaled: 3.17.0-beta.8

Skaled exited gracefully on all staying nodes

Here is stack trace for your info:

2023-08-30 16:22:04.278270   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
2023-08-30 16:22:04.278291   Setting ExitTimeReached = true
2023-08-30 16:22:04.278306   Skaled status: setExitState: ExitTimeReached to true
[2023-08-30 16:22:04.278] [9:consensus] [info] 2261225:Running exitGracefully method
[2023-08-30 16:22:04.278] [9:consensus] [info] 2261225:Iterating over nodes
[2023-08-30 16:22:04.278] [9:consensus] [info] 2261225:Processing a node exit
[2023-08-30 16:22:04.278] [9:main] [info] 2261225:Set exit on block boundary
[2023-08-30 16:22:04.278] [9:consensus] [info] 2261225:Consensus exiting: exitGracefully called by skaled
 0# 0x0000557BC1147E6E in /skaled/skaled
 1# 0x0000557BC0F111DE in /skaled/skaled
 2# 0x0000557BC0F0367D in /skaled/skaled
 3# 0x0000557BC11CE40E in /skaled/skaled
 4# 0x0000557BC11DA066 in /skaled/skaled
 5# 0x0000557BC11DCBD4 in /skaled/skaled
 6# 0x0000557BC11DE469 in /skaled/skaled
[2023-08-30 16:22:04.480] [config] [info] Consensus engine exiting: exitGracefullyAsync called by skaled
[2023-08-30 16:22:04.480] [config] [info] exitGracefullyAsync running
[2023-08-30 16:22:04.480] [config] [info] Node exit called
[2023-08-30 16:22:04.480] [config] [info] Node::exit() requested
 7# 0x0000557BC11A3C93 in /skaled/skaled
 8# 0x0000557BC11AA68B in /skaled/skaled
 9# 0x0000557BC11CF4DF in /skaled/skaled
10# 0x00007F0E2198DDF4 in /lib/x86_64-linux-gnu/libstdc++.so.6
11# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
12# 0x00007F0E2167A133 at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Caught (first) signal. Signal -1. Unknown signal type.


SELF-KILL: Will sleep 270 seconds before force exit...

[2023-08-30 16:22:04.480] [9:main] [info] 2261225:Exiting on block boundary for block 2261225
2023-08-30 16:22:04.480685   Rotation is completed. Instance is exiting
[2023-08-30 16:22:04.480] [9:main] [info] 2261225:exitImmediately called
2023-08-30 16:22:04.480705   THREADS timer started
[2023-08-30 16:22:04.480] [9:main] [info] 2261225:Status server stopped
[2023-08-30 16:22:04.480] [9:main] [info] 2261225:consensus engine exiting: close all sockets called
[2023-08-30 16:22:04.480] [9:main] [info] 2261225:consensus engine exiting: server conditional vars notified
[2023-08-30 16:22:04.480] [9:main] [info] 2261225:consensus engine exiting: agent conditional vars notified
[2023-08-30 16:22:04.480] [9:main] [info] 2261225:Consensus exiting: Node:Exit requested
[2023-08-30 16:22:04.481] [9:main] [info] 2261225:Consensus exiting: Node:Exit requested
[2023-08-30 16:22:04.481] [9:main] [info] 2261225:Consensus exiting: Node:Exit requested
[2023-08-30 16:22:04.481] [9:main] [info] 2261225:consensus engine exiting: catchup socket touched
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:consensus engine exiting: block proposal socket touched
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:consensus engine exiting: SGXZMQClient exiting
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:Exiting SgxZmqClient
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:Shutting down SgxZmq context
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:Shut down SgxZmq context
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:Closing SgxZmq client sockets
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:Exited SgxZmqClient
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:consensus engine exiting: SGXZMQClient exited
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:consensus engine exiting: exitZMQClient called
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:Cleaning up ZMQ sockets
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:consensus engine exiting: closing ZMQ send sockets
7.7361e-05 THREADS 65: +main +skaled +dPool0 +dPool1 +dPool2 +dPool3 +dPool4 +dPool5 +dPool6 +dPool7 +dPool8 +dPool9 +dPool10 +dPool11 +dPool12 +dPool13 +dPool14 +dPool15 +blockVerifier0 +skaled +MonitoringLoop +TimeoutLoop +StuckDetectionL +CatchupClient +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +skaled +skaled +Client +ZMQbg/Reaper +ZMQbg/IO/0 +ZmqBroadcaster +broadcastFunc +CatchupServer +BlockPropSrv +NtwkRdLoop +DeferMsgLoop +msgThreadProcLo +ZMQbg/Reaper +ZMQbg/IO/0 +ZMQbg/Reaper +ZMQbg/IO/0 +led/WS-listener +ed/WSS-listener +-0x557bc307ffb0 +HTTPSrvExec0 +HTTPSrvExec1 +HTTPSrvExec2 +HTTPSrvExec3 +HTTPSrvExec4 +HTTPSrvExec5 +HTTPSrvExec6 +HTTPSrvExec7 +HTTPSrvExec8 +HTTPSrvExec9 +IOThreadPool0 +-0x557bc25ff220 +msgThreadProcLo +msgThreadProcLo
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:consensus engine exiting: closing receive sockets
[2023-08-30 16:22:04.482] [9:main] [info] 2261225:Closing ZMQ context
[2023-08-30 16:22:04.483] [9:main] [info] 2261225:Closed ZMQ context
[2023-08-30 16:22:04.483] [9:main] [info] 2261225:consensus engine exiting: ZMQ sockets closeAndCleanupAll called
[2023-08-30 16:22:04.483] [9:main] [info] 2261225:exitImmediately completed
[2023-08-30 16:22:04.484] [9:main] [error] 2261225:Exception in Schain::messageThreadProcessingLoop
[2023-08-30 16:22:04.484] [9:main] [info] 2261225:Consensus exiting: Exit on block boundary successful:Exit requested
2023-08-30 16:22:04.503855   Skaled status: setExitState: StartAgain to true
2023-08-30 16:22:04.504035   Skaled status: setExitState: StartFromSnapshot to false
2023-08-30 16:22:04.504532   Skaled status: setExitState: ClearDataDir to false
2023-08-30 16:22:04.507591   Will stop WS server on address 0.0.0.0 and port 10002=0x00002712/nfo...
2023-08-30 16:22:04.507614   Will stop on port 10002=0x00002712...
[2023-08-30 16:22:04.580] [config] [info] Node::exit() will try to exit on block boundary for 30 seconds
0.101509 THREADS 37: -TimeoutLoop -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -CatchupServer -BlockPropSrv -NtwkRdLoop --0x557bc307ffb0 -HTTPSrvExec0 -HTTPSrvExec1 -HTTPSrvExec2 -HTTPSrvExec3 -HTTPSrvExec4 -HTTPSrvExec5 -HTTPSrvExec6 -HTTPSrvExec7 -HTTPSrvExec8 -HTTPSrvExec9 -IOThreadPool0
[2023-08-30 16:22:04.680] [config] [info] Node exit completed
0.605482 THREADS 35: -MonitoringLoop
0.706014 THREADS 34: -DeferMsgLoop
1.60973 THREADS 33: -StuckDetectionL
4.12191 THREADS 32: -led/WS-listener
2023-08-30 16:22:08.541701   OK, server stopped on port 10002=0x00002712
2023-08-30 16:22:08.541751   OK, stopped WS server on address 0.0.0.0 and port 10002=0x00002712/nfo
2023-08-30 16:22:08.541781   Will stop WSS server on address 0.0.0.0 and port 10007=0x00002717/nfo...
2023-08-30 16:22:08.541788   Will stop on port 10007=0x00002717...
4.22231 THREADS 31: -ed/WSS-listener
2023-08-30 16:22:08.653642   OK, server stopped on port 10007=0x00002717
2023-08-30 16:22:08.653691   OK, stopped WSS server on address 0.0.0.0 and port 10007=0x00002717/nfo
2023-08-30 16:22:08.653718   Will stop proxygen/HTTP server on address 0.0.0.0 and port 10003=0x00002713/nfo...
2023-08-30 16:22:08.653755   OK, stopped proxygen/HTTP server on address 0.0.0.0 and port 10003=0x00002713/nfo
2023-08-30 16:22:08.653773   Will stop proxygen/HTTPS server on address 0.0.0.0 and port 10008=0x00002718/nfo...
2023-08-30 16:22:08.653810   OK, stopped proxygen/HTTPS server on address 0.0.0.0 and port 10008=0x00002718/nfo
2023-08-30 16:22:08.653828   Will stop proxygen/HTTP server on address 0.0.0.0 and port 10009=0x00002719/std...
2023-08-30 16:22:08.653847   OK, stopped proxygen/HTTP server on address 0.0.0.0 and port 10009=0x00002719/std
2023-08-30 16:22:08.654450   Skaled status: setSubsystemRunning: Rpc to false

4.72473 THREADS 29: -CatchupClient
2023-08-30 16:22:09.278389   Stop worker 623 ms
2023-08-30 16:22:09.278440   1 before exitGracefully()
2023-08-30 16:22:09.278453   Exit requested internally with exit code 1
[2023-08-30 16:22:09.278] [9:consensus] [info] 2261225:Running exitGracefully method
2023-08-30 16:22:09.278506   2 after exitGracefully()
2023-08-30 16:22:09.278516   3 after wait loop
2023-08-30 16:22:09.290359   4 before dtor
2023-08-30 16:22:09.293534   Blockchain is closed
2023-08-30 16:22:09.293655   Skaled status: setSubsystemRunning: Blockchain to false

2023-08-30 16:22:09.293963   Blockchain is closed
.
4.82512 THREADS 23: -blockVerifier0 -Client -ZMQbg/Reaper -ZmqBroadcaster -broadcastFunc
2023-08-30 16:22:09.297145   Exiting main with code 1...

Dispatch: All threads stopped
Dispatch: All dispatch queues removed
[2023-08-30 16:22:10.029] [9:consensus] [info] 2261225:Running exitGracefully method

@yatsunastya
Copy link

skaled: 3.17.0-beta.10

Skaled exits gracefully with 0 exit code Exit requested internally with exit code 0 but not on all nodes. On some of the nodes it gets Caught (second) signal. Signal SIGABRT(6) and exits forcefully.

Gracefully ⬇️

2023-09-18 18:11:56.918075   Setting ExitTimeReached = true
2023-09-18 18:11:56.918083   Skaled status: setExitState: ExitTimeReached to true
[2023-09-18 18:11:56.918] [25:consensus] [info] 31275:Running exitGracefully method
[2023-09-18 18:11:56.918] [25:consensus] [info] 31275:Iterating over nodes
[2023-09-18 18:11:56.918] [25:consensus] [info] 31275:Processing a node exit
[2023-09-18 18:11:56.918] [25:main] [info] 31275:Set exit on block boundary
[2023-09-18 18:11:56.918] [25:consensus] [info] 31275:Consensus exiting: exitGracefully called by skaled
Here is stack trace for your info:
 0# 0x0000556F4071C71E in /skaled/skaled
 1# 0x0000556F404D122E in /skaled/skaled
 2# 0x0000556F404C36CD in /skaled/skaled
 3# 0x0000556F407A2CBE in /skaled/skaled
 4# 0x0000556F407AE906 in /skaled/skaled
 5# 0x0000556F407B1514 in /skaled/skaled
 6# 0x0000556F407B2DA9 in /skaled/skaled
 7# 0x0000556F40778543 in /skaled/skaled
 8# 0x0000556F4077EF3B in /skaled/skaled
 9# 0x0000556F407A3D8F in /skaled/skaled
10# 0x00007F76EE85FDF4 in /lib/x86_64-linux-gnu/libstdc++.so.6
11# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
12# 0x00007F76EE54C133 at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Caught (first) signal. Signal -1. Unknown signal type.

[2023-09-18 18:11:57.167] [config] [info] Consensus engine exiting: exitGracefullyAsync called by skaled
[2023-09-18 18:11:57.167] [config] [info] exitGracefullyAsync running
[2023-09-18 18:11:57.167] [config] [info] Node exit called
[2023-09-18 18:11:57.167] [config] [info] Node::exit() requested

SELF-KILL: Will sleep 270 seconds before force exit...

[2023-09-18 18:11:57.167] [25:main] [info] 31275:Exiting on block boundary for block 31275
[2023-09-18 18:11:57.167] [25:main] [info] 31275:exitImmediately called
[2023-09-18 18:11:57.167] [25:main] [info] 31275:Status server stopped
[2023-09-18 18:11:57.167] [25:main] [info] 31275:consensus engine exiting: close all sockets called
[2023-09-18 18:11:57.167] [25:main] [info] 31275:consensus engine exiting: server conditional vars notified
[2023-09-18 18:11:57.168] [25:main] [info] 31275:consensus engine exiting: agent conditional vars notified
[2023-09-18 18:11:57.168] [25:main] [info] 31275:consensus engine exiting: catchup socket touched
[2023-09-18 18:11:57.168] [25:main] [info] 31275:consensus engine exiting: block proposal socket touched
[2023-09-18 18:11:57.168] [25:main] [info] 31275:consensus engine exiting: SGXZMQClient exiting
[2023-09-18 18:11:57.168] [25:main] [info] 31275:Exiting SgxZmqClient
[2023-09-18 18:11:57.168] [25:main] [info] 31275:Shutting down SgxZmq context
[2023-09-18 18:11:57.168] [25:main] [info] 31275:Shut down SgxZmq context
[2023-09-18 18:11:57.168] [25:main] [info] 31275:Closing SgxZmq client sockets
[2023-09-18 18:11:57.168] [25:main] [info] 31275:Exited SgxZmqClient
[2023-09-18 18:11:57.168] [25:main] [info] 31275:consensus engine exiting: SGXZMQClient exited
[2023-09-18 18:11:57.168] [25:main] [info] 31275:consensus engine exiting: exitZMQClient called
[2023-09-18 18:11:57.168] [25:main] [info] 31275:Cleaning up ZMQ sockets
[2023-09-18 18:11:57.168] [25:main] [info] 31275:consensus engine exiting: closing ZMQ send sockets
[2023-09-18 18:11:57.168] [25:main] [info] 31275:consensus engine exiting: closing receive sockets
[2023-09-18 18:11:57.168] [25:main] [info] 31275:Closing ZMQ context
[2023-09-18 18:11:57.168] [25:main] [info] 31275:Consensus exiting: Node:Exit requested
2023-09-18 18:11:57.167772   Rotation is completed. Instance is exiting
2023-09-18 18:11:57.167904   THREADS timer started
[2023-09-18 18:11:57.168] [25:main] [info] 31275:Consensus exiting: Node:Exit requested
[2023-09-18 18:11:57.169] [25:main] [info] 31275:Consensus exiting: Node:Exit requested
0.000181863 THREADS 56: +main +skaled +dPool0 +dPool1 +dPool2 +dPool3 +dPool4 +dPool5 +dPool6 +dPool7 +dPool8 +dPool9 +dPool10 +dPool11 +dPool12 +dPool13 +dPool14 +dPool15 +blockVerifier0 +skaled +MonitoringLoop +TimeoutLoop +StuckDetectionL +CatchupClient +BlockPopClnt +BlockPopClnt +BlockPopClnt +skaled +skaled +Client +ZMQbg/Reaper +ZMQbg/IO/0 +ZmqBroadcaster +broadcastFunc +DeferMsgLoop +msgThreadProcLo +ZMQbg/Reaper +ZMQbg/IO/0 +ZMQbg/IO/0 +led/WS-listener +ed/WSS-listener +-0x556f437fbc30 +HTTPSrvExec0 +HTTPSrvExec1 +HTTPSrvExec2 +HTTPSrvExec3 +HTTPSrvExec4 +HTTPSrvExec5 +HTTPSrvExec6 +HTTPSrvExec7 +HTTPSrvExec8 +HTTPSrvExec9 +IOThreadPool0 +-0x556f41bc3240 +msgThreadProcLo +msgThreadProcLo
[2023-09-18 18:11:57.170] [25:main] [info] 31275:Closed ZMQ context
[2023-09-18 18:11:57.170] [25:main] [info] 31275:consensus engine exiting: ZMQ sockets closeAndCleanupAll called
[2023-09-18 18:11:57.170] [25:main] [info] 31275:exitImmediately completed
[2023-09-18 18:11:57.170] [25:main] [error] 31275:Exception in Schain::messageThreadProcessingLoop
[2023-09-18 18:11:57.170] [25:main] [info] 31275:Consensus exiting: Exit on block boundary successful:Exit requested
2023-09-18 18:11:57.178509   http://54.245.123.63:56268 >>> {"id":3149152748,"jsonrpc":"2.0","method":"skale_imaInfo","params":{"fromImaAgentIndex":13}}
2023-09-18 18:11:57.179263   http://54.245.123.63:56268 <<< {"id":3149152748,"jsonrpc":"2.0","result":{"BLSPublicKey0":"10737461108999331386357138679494079300104363867104246551897109738536964702547","BLSPublicKey1":"20502789143262619207267627008704233918491837596668224545253611414007619732699","BLSPublicKey2":"9571764645320646328308041043260482829748770664353619166800644989466449240457","BLSPublicKey3":"12811947376564262412939099055603616194404928993497948967201751475149292792668","commonBLSPublicKey0":"20290944374078261228220008861923589228740788750659612544284442570440305901317","commonBLSPublicKey1":"15958722212803524170537068116662090924445289543607213113362635635339508904086","commonBLSPublicKey2":"4766914959117568022526844528230118295233694675935148436224374342116783793577","commonBLSPublicKey3":"21132778593190322580164158198410288919826167158854973641739103829816740740443","n":16,"t":11,"thisNodeIndex":11}}
2023-09-18 18:11:57.204833   Skaled status: setExitState: StartAgain to false
2023-09-18 18:11:57.205063   Skaled status: setExitState: StartFromSnapshot to false
2023-09-18 18:11:57.205626   Skaled status: setExitState: ClearDataDir to false
2023-09-18 18:11:57.211331   Will stop WS server on address 0.0.0.0 and port 10130=0x00002792/nfo...
2023-09-18 18:11:57.211354   Will stop on port 10130=0x00002792...
[2023-09-18 18:11:57.268] [config] [info] Node::exit() will try to exit on block boundary for 30 seconds
0.102422 THREADS 38: -BlockPopClnt -BlockPopClnt -BlockPopClnt --0x556f437fbc30 -HTTPSrvExec0 -HTTPSrvExec1 -HTTPSrvExec2 -HTTPSrvExec3 -HTTPSrvExec4 -HTTPSrvExec5 -HTTPSrvExec6 -HTTPSrvExec7 -HTTPSrvExec8 -HTTPSrvExec9 -IOThreadPool0
[2023-09-18 18:11:57.368] [config] [info] Node exit completed
0.604512 THREADS 37: -DeferMsgLoop
0.805542 THREADS 34: -MonitoringLoop -TimeoutLoop
1.50892 THREADS 33: -CatchupClient
2.91696 THREADS 31: -StuckDetectionL
3.72023 THREADS 30: -ed/WSS-listener
2023-09-18 18:12:01.336618   OK, server stopped on port 10130=0x00002792
2023-09-18 18:12:01.336669   OK, stopped WS server on address 0.0.0.0 and port 10130=0x00002792/nfo
2023-09-18 18:12:01.336694   Will stop WSS server on address 0.0.0.0 and port 10135=0x00002797/nfo...
2023-09-18 18:12:01.336708   Will stop on port 10135=0x00002797...
2023-09-18 18:12:01.336835   OK, server stopped on port 10135=0x00002797
2023-09-18 18:12:01.336863   OK, stopped WSS server on address 0.0.0.0 and port 10135=0x00002797/nfo
2023-09-18 18:12:01.336889   Will stop proxygen/HTTP server on address 0.0.0.0 and port 10131=0x00002793/nfo...
2023-09-18 18:12:01.336917   OK, stopped proxygen/HTTP server on address 0.0.0.0 and port 10131=0x00002793/nfo
2023-09-18 18:12:01.336937   Will stop proxygen/HTTPS server on address 0.0.0.0 and port 10136=0x00002798/nfo...
2023-09-18 18:12:01.336958   OK, stopped proxygen/HTTPS server on address 0.0.0.0 and port 10136=0x00002798/nfo
2023-09-18 18:12:01.336979   Will stop proxygen/HTTP server on address 0.0.0.0 and port 10137=0x00002799/std...
2023-09-18 18:12:01.337001   OK, stopped proxygen/HTTP server on address 0.0.0.0 and port 10137=0x00002799/std
2023-09-18 18:12:01.337467   Skaled status: setSubsystemRunning: Rpc to false

4.222 THREADS 29: -led/WS-listener
[2023-09-18 18:12:01.917] [25:consensus] [info] 31275:Running exitGracefully method
2023-09-18 18:12:01.917424   Stop worker 579 ms
2023-09-18 18:12:01.917469   1 before exitGracefully()
2023-09-18 18:12:01.917481   Exit requested internally with exit code 0
2023-09-18 18:12:01.917522   2 after exitGracefully()
2023-09-18 18:12:01.917529   3 after wait loop
4.82506 THREADS 28: -ZmqBroadcaster
.
2023-09-18 18:12:01.994022   4 before dtor
2023-09-18 18:12:01.995668   Blockchain is closed
2023-09-18 18:12:01.995789   Skaled status: setSubsystemRunning: Blockchain to false

2023-09-18 18:12:01.996006   Blockchain is closed
4.92551 THREADS 23: -blockVerifier0 -Client -ZMQbg/Reaper -broadcastFunc
Dispatch: All threads stopped
Dispatch: All dispatch queues removed
[2023-09-18 18:12:02.995] [25:consensus] [info] 31275:Running exitGracefully method
5.83014 THREADS 5: -dPool0 -dPool1 -dPool2 -dPool3 -dPool4 -dPool5 -dPool6 -dPool7 -dPool8 -dPool9 -dPool10 -dPool11 -dPool12 -dPool13 -dPool14 -dPool15 --0x556f41bc3240

Forcefully ⬇️

2023-09-18 18:11:56.961182   Block sealed #31276 (#72314b69…)
2023-09-18 18:11:56.961211   Block stats:BN:31275:BTS:1695060704:TXS:0:HDRS:11:LOGS:40:SENGS:1:TXRS:0:BLCKS:3:ACCS:150:BQS:1:BDS:1082:TSS:0:UTX:0:VTX:0:CMM:1177172
2023-09-18 18:11:56.961473   Block 31276 DB usage is 1110. Piece DB usage is 183720943 bytes
2023-09-18 18:11:56.964103   noteChanged: {chain}
2023-09-18 18:11:56.964592   SWT:3:BFT:11478
2023-09-18 18:11:56.964611   Successfully imported 0 of 0 transactions

2023-09-18 18:11:56.964622   TQBYTES:CTQ:0:FTQ:0:TQSIZE:CTQ:0:FTQ:0
2023-09-18 18:11:56.964642   Setting ExitTimeReached = true
2023-09-18 18:11:56.964652   Skaled status: setExitState: ExitTimeReached to true
[2023-09-18 18:11:56.964] [9:consensus] [info] 31275:Running exitGracefully method
[2023-09-18 18:11:56.964] [9:consensus] [info] 31275:Iterating over nodes
[2023-09-18 18:11:56.964] [9:consensus] [info] 31275:Processing a node exit
[2023-09-18 18:11:56.964] [9:main] [info] 31275:Set exit on block boundary
[2023-09-18 18:11:56.964] [9:consensus] [info] 31275:Consensus exiting: exitGracefully called by skaled
Here is stack trace for your info:
 0# 0x000055CBE356B71E in /skaled/skaled
 1# 0x000055CBE332022E in /skaled/skaled
 2# 0x000055CBE33126CD in /skaled/skaled
 3# 0x000055CBE35F1CBE in /skaled/skaled
 4# 0x000055CBE35FD906 in /skaled/skaled
 5# 0x000055CBE3600514 in /skaled/skaled
 6# 0x000055CBE3601DA9 in /skaled/skaled
 7# 0x000055CBE35C7543 in /skaled/skaled
 8# 0x000055CBE35CDF3B in /skaled/skaled
 9# 0x000055CBE35F2D8F in /skaled/skaled
10# 0x00007F380A0B7DF4 in /lib/x86_64-linux-gnu/libstdc++.so.6
11# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
12# 0x00007F3809DA4133 at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Caught (first) signal. Signal -1. Unknown signal type.
[2023-09-18 18:11:57.154] [config] [info] Consensus engine exiting: exitGracefullyAsync called by skaled
[2023-09-18 18:11:57.154] [config] [info] exitGracefullyAsync running
[2023-09-18 18:11:57.154] [config] [info] Node exit called
[2023-09-18 18:11:57.154] [config] [info] Node::exit() requested
[2023-09-18 18:11:57.154] [9:main] [info] 31275:Exiting on block boundary for block 31275
[2023-09-18 18:11:57.154] [9:main] [info] 31275:exitImmediately called
[2023-09-18 18:11:57.154] [9:main] [info] 31275:Status server stopped
[2023-09-18 18:11:57.154] [9:main] [info] 31275:consensus engine exiting: close all sockets called
[2023-09-18 18:11:57.154] [9:main] [info] 31275:consensus engine exiting: server conditional vars notified


SELF-KILL: Will sleep 270 seconds before force exit...

2023-09-18 18:11:57.154716   Rotation is completed. Instance is exiting
2023-09-18 18:11:57.154746   THREADS timer started
[2023-09-18 18:11:57.154] [9:main] [info] 31275:consensus engine exiting: agent conditional vars notified
[2023-09-18 18:11:57.155] [9:main] [info] 31275:Consensus exiting: Node:Exit requested
[2023-09-18 18:11:57.155] [9:main] [info] 31275:consensus engine exiting: catchup socket touched
0.000104424 THREADS 71: +main +skaled +dPool0 +dPool1 +dPool2 +dPool3 +dPool4 +dPool5 +dPool6 +dPool7 +dPool8 +dPool9 +dPool10 +dPool11 +dPool12 +dPool13 +dPool14 +dPool15 +blockVerifier0 +skaled +MonitoringLoop +TimeoutLoop +StuckDetectionL +orclAssemblyLoo +CatchupClient +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +skaled +skaled +Client +ZMQbg/Reaper +ZMQbg/IO/0 +ZmqBroadcaster +broadcastFunc +CtchpSrvAg1 +CatchupServer +BlkPropSrvAg0 +BlockPropSrv +NtwkRdLoop +DeferMsgLoop +msgThreadProcLo +ZMQbg/Reaper +ZMQbg/IO/0 +ZMQbg/Reaper +ZMQbg/IO/0 +led/WS-listener +ed/WSS-listener +-0x55cbe70e1510 +HTTPSrvExec0 +HTTPSrvExec1 +HTTPSrvExec2 +HTTPSrvExec3 +HTTPSrvExec4 +HTTPSrvExec5 +HTTPSrvExec6 +HTTPSrvExec7 +HTTPSrvExec8 +HTTPSrvExec9 +IOThreadPool0 +-0x55cbe4a12240 +msgThreadProcLo +msgThreadProcLo
[2023-09-18 18:11:57.155] [9:main] [info] 31275:consensus engine exiting: block proposal socket touched
[2023-09-18 18:11:57.155] [9:main] [info] 31275:consensus engine exiting: SGXZMQClient exiting
[2023-09-18 18:11:57.155] [9:main] [info] 31275:Exiting SgxZmqClient
[2023-09-18 18:11:57.155] [9:main] [info] 31275:Shutting down SgxZmq context
[2023-09-18 18:11:57.156] [9:main] [info] 31275:Shut down SgxZmq context
[2023-09-18 18:11:57.156] [9:main] [info] 31275:Closing SgxZmq client sockets
[2023-09-18 18:11:57.156] [9:main] [info] 31275:Exited SgxZmqClient
[2023-09-18 18:11:57.156] [9:main] [info] 31275:consensus engine exiting: SGXZMQClient exited
[2023-09-18 18:11:57.156] [9:main] [info] 31275:consensus engine exiting: exitZMQClient called
[2023-09-18 18:11:57.156] [9:main] [info] 31275:Cleaning up ZMQ sockets
[2023-09-18 18:11:57.156] [9:main] [info] 31275:consensus engine exiting: closing ZMQ send sockets
[2023-09-18 18:11:57.156] [9:main] [info] 31275:consensus engine exiting: closing receive sockets
[2023-09-18 18:11:57.156] [9:main] [info] 31275:Closing ZMQ context
[2023-09-18 18:11:57.156] [9:main] [info] 31275:Consensus exiting: Node:Exit requested
[2023-09-18 18:11:57.156] [9:main] [info] 31275:Consensus exiting: Node:Exit requested
[2023-09-18 18:11:57.159] [9:main] [info] 31275:Closed ZMQ context
[2023-09-18 18:11:57.159] [9:main] [info] 31275:consensus engine exiting: ZMQ sockets closeAndCleanupAll called
[2023-09-18 18:11:57.159] [9:main] [info] 31275:exitImmediately completed
[2023-09-18 18:11:57.160] [9:main] [error] 31275:Exception in Schain::messageThreadProcessingLoop
[2023-09-18 18:11:57.160] [9:main] [info] 31275:Consensus exiting: Exit on block boundary successful:Exit requested
2023-09-18 18:11:57.178177   http://54.245.123.63:34134 >>> {"id":4123764856,"jsonrpc":"2.0","method":"skale_imaInfo","params":{"fromImaAgentIndex":13}}
2023-09-18 18:11:57.178937   http://54.245.123.63:34134 <<< {"id":4123764856,"jsonrpc":"2.0","result":{"BLSPublicKey0":"4947355822879923643861434925736985309562004851771444433487885674576413449120","BLSPublicKey1":"13431005726663074376148198779227691003101696105657904241840915959970292024407","BLSPublicKey2":"1011783082207142227405835236796091145131138616714686582722508809488452694011","BLSPublicKey3":"18259953076747238497101510927479891754697991809348223519473104780848254768166","commonBLSPublicKey0":"20290944374078261228220008861923589228740788750659612544284442570440305901317","commonBLSPublicKey1":"15958722212803524170537068116662090924445289543607213113362635635339508904086","commonBLSPublicKey2":"4766914959117568022526844528230118295233694675935148436224374342116783793577","commonBLSPublicKey3":"21132778593190322580164158198410288919826167158854973641739103829816740740443","n":16,"t":11,"thisNodeIndex":10}}
2023-09-18 18:11:57.210347   Skaled status: setExitState: StartAgain to false
2023-09-18 18:11:57.210575   Skaled status: setExitState: StartFromSnapshot to false
2023-09-18 18:11:57.211103   Skaled status: setExitState: ClearDataDir to false
2023-09-18 18:11:57.217396   Will stop WS server on address 0.0.0.0 and port 10258=0x00002812/nfo...
2023-09-18 18:11:57.217412   Will stop on port 10258=0x00002812...
[2023-09-18 18:11:57.254] [config] [info] Node::exit() will try to exit on block boundary for 30 seconds
0.101646 THREADS 36: -StuckDetectionL -orclAssemblyLoo -CatchupClient -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -CtchpSrvAg1 -CatchupServer -BlkPropSrvAg0 -BlockPropSrv -NtwkRdLoop --0x55cbe70e1510 -HTTPSrvExec0 -HTTPSrvExec1 -HTTPSrvExec2 -HTTPSrvExec3 -HTTPSrvExec4 -HTTPSrvExec5 -HTTPSrvExec6 -HTTPSrvExec7 -HTTPSrvExec8 -HTTPSrvExec9 -IOThreadPool0
[2023-09-18 18:11:57.354] [config] [info] Node exit completed
0.503647 THREADS 34: -DeferMsgLoop
0.604283 THREADS 33: -TimeoutLoop
0.805271 THREADS 31: -MonitoringLoop
8.94337 THREADS 30: -ed/WSS-listener
2023-09-18 18:12:06.122614   OK, server stopped on port 10258=0x00002812
2023-09-18 18:12:06.122663   OK, stopped WS server on address 0.0.0.0 and port 10258=0x00002812/nfo
2023-09-18 18:12:06.122683   Will stop WSS server on address 0.0.0.0 and port 10263=0x00002817/nfo...
2023-09-18 18:12:06.122693   Will stop on port 10263=0x00002817...
2023-09-18 18:12:06.123681   OK, server stopped on port 10263=0x00002817
2023-09-18 18:12:06.123718   OK, stopped WSS server on address 0.0.0.0 and port 10263=0x00002817/nfo
2023-09-18 18:12:06.123734   Will stop proxygen/HTTP server on address 0.0.0.0 and port 10259=0x00002813/nfo...
2023-09-18 18:12:06.123748   OK, stopped proxygen/HTTP server on address 0.0.0.0 and port 10259=0x00002813/nfo
2023-09-18 18:12:06.123757   Will stop proxygen/HTTPS server on address 0.0.0.0 and port 10264=0x00002818/nfo...
2023-09-18 18:12:06.123771   OK, stopped proxygen/HTTPS server on address 0.0.0.0 and port 10264=0x00002818/nfo
terminate called without an active exception

Stop flag was already raised on. WILL FORCE TERMINATE. Caught (second) signal. Signal SIGABRT(6). This is the abort signal. Typically, a process will initiate this kill signal on itself.

2023-09-18 18:12:06.123785   Will stop proxygen/HTTP server on address 0.0.0.0 and port 10265=0x00002819/std...
2023-09-18 18:12:06.123797   OK, stopped proxygen/HTTP server on address 0.0.0.0 and port 10265=0x00002819/std

  /skaled/skaled : dev::ExitHandler::exitHandler(int, dev::ExitHandler::exit_code_t)+0x730 [0x55cbe34031d0]
  /lib/x86_64-linux-gnu/libpthread.so.0 : ()+0x14420 [0x7f380a7ca420]
  /lib/x86_64-linux-gnu/libc.so.6 : gsignal()+0xcb [0x7f3809cc800b]
  /lib/x86_64-linux-gnu/libc.so.6 : abort()+0x12b [0x7f3809ca7859]
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0x9e8d1 [0x7f380a07f8d1]
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa37c [0x7f380a08b37c]
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa3e7 [0x7f380a08b3e7]
  /skaled/skaled : ()+0xb8d6c4 [0x55cbe33ec6c4]
  /skaled/skaled : ModularServer<dev::rpc::EthFace, dev::rpc::SkaleFace, dev::rpc::SkaleStats, dev::rpc::NetFace, dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x91 [0x55cbe31efaa1]
  /skaled/skaled : main()+0x13e41 [0x55cbe315cb21]
  /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main()+0xf3 [0x7f3809ca9083]
  /skaled/skaled : _start()+0x2e [0x55cbe31bcc4e]

55cbe285f000-55cbe48d7000 r-xp 00000000 00:ef 8312828                    /skaled/skaled
55cbe48d7000-55cbe499a000 r--p 02077000 00:ef 8312828                    /skaled/skaled
55cbe499a000-55cbe49a4000 rw-p 0213a000 00:ef 8312828                    /skaled/skaled
55cbe49a4000-55cbe4af9000 rw-p 00000000 00:00 0 
55cbe697e000-55cbe7de8000 rw-p 00000000 00:00 0                          [heap]
7f36d67ed000-7f36d67ee000 ---p 00000000 00:00 0 
7f36d67ee000-7f36d6fee000 rw-p 00000000 00:00 0 
7f36d6fee000-7f36d6fef000 ---p 00000000 00:00 0 
7f36d6fef000-7f36d77ef000 rw-p 00000000 00:00 0 
7f36d77ef000-7f36d77f0000 ---p 00000000 00:00 0 
7f36d77f0000-7f36d7ff0000 rw-p 00000000 00:00 0 
7f36d8ff2000-7f36d8ff3000 ---p 00000000 00:00 0 
7f36d8ff3000-7f36d97f3000 rw-p 00000000 00:00 0 
7f36dd8f9000-7f36de238000 rw-p 00000000 00:00 0 
7f36de41a000-7f36de48d000 rw-p 00000000 00:00 0 
7f36de497000-7f36de508000 rw-p 00000000 00:00 0 
7f36de508000-7f36de509000 r--p 00001000 00:ef 6489297                    /usr/lib/x86_64-linux-gnu/libresolv-2.31.so
7f36de509000-7f36de511000 rw-p 00000000 00:00 0 
7f36de511000-7f36de512000 r--p 00000000 00:ef 6489264                    /usr/lib/x86_64-linux-gnu/libnss_dns-2.31.so
7f36de512000-7f36de636000 rw-p 00000000 00:00 0 
7f36de636000-7f36de652000 rw-p 00000000 00:00 0 
7f36de652000-7f36de653000 rw-p 00000000 00:00 0 
7f36de653000-7f36de655000 r--p 00000000 00:ef 6489221                    /usr/lib/x86_64-linux-gnu/libffi.so.7.1.0
7f36de655000-7f36de656000 r--p 00001000 00:ef 6489266                    /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so
7f36de656000-7f36de659000 r--p 00002000 00:ef 6489236                    /usr/lib/x86_64-linux-gnu/libhogweed.so.5.0
7f36de659000-7f36de65c000 rw-p 00000000 00:00 0 
7f36de65c000-7f36de660000 r--p 00003000 00:ef 6489259                    /usr/lib/x86_64-linux-gnu/libnettle.so.7.0
7f36de660000-7f36de661000 rw-p 00000000 00:00 0 
7f36de661000-7f36de664000 r--p 00005000 00:ef 6489230                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.0
7f36de664000-7f36de666000 r--p 00000000 00:ef 6489315                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.0
7f36de666000-7f36de66b000 rw-p 00000000 00:00 0 
7f36de66b000-7f36de670000 r--p 00005000 00:ef 6489325                    /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
7f36de670000-7f36de671000 r--p 00002000 00:ef 6489234                    /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.28.0
7f36de671000-7f36de673000 r--p 00001000 00:ef 6489275                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
7f36de673000-7f36de676000 rw-p 00000000 00:00 0 
7f36de676000-7f36de6c1000 rw-p 00000000 00:00 0 
7f36de6c1000-7f36de6c2000 r--p 00000000 00:ef 6489179                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f36de6c2000-7f36de71d000 rw-p 00000000 00:00 0 
7f36de71d000-7f36de931000 r--s 00000000 00:f1 16973                      /data_dir/28e07f34/blocks_and_extras/3.db/000426.ldb
7f36de931000-7f36deb44000 r--s 00000000 00:f1 16972                      /data_dir/28e07f34/blocks_and_extras/3.db/000425.ldb
7f36deb44000-7f36ded47000 r--s 00000000 00:f1 96061                      /data_dir/da_proofs_9.db/db.70/000540.ldb
7f36ded47000-7f36def4a000 r--s 00000000 00:f1 96060                      /data_dir/da_proofs_9.db/db.70/000539.ldb
7f36def4a000-7f36df14d000 r--s 00000000 00:f1 96059                      /data_dir/da_proofs_9.db/db.70/000538.ldb
7f36df14d000-7f36df350000 r--s 00000000 00:f1 96058                      /data_dir/da_proofs_9.db/db.70/000537.ldb
7f36df350000-7f36df82d000 rw-p 00000000 00:00 0 
7f36df82d000-7f36dfbf9000 r--p 00138000 00:ef 8312828                    /skaled/skaled
7f36dfbf9000-7f36dfdfd000 r--s 00000000 00:f1 94837                      /data_dir/da_proofs_9.db/db.69/000657.ldb
7f36dfdfd000-7f36e0000000 r--s 00000000 00:f1 94836                      /data_dir/da_proofs_9.db/db.69/000656.ldb
7f36e0000000-7f36e0553000 rw-p 00000000 00:00 0 
7f36e0553000-7f36e4000000 ---p 00000000 00:00 0 
7f36e4000000-7f36e40da000 rw-p 00000000 00:00 0 
7f36e40da000-7f36e42dd000 r--s 00000000 00:f1 94835                      /data_dir/da_proofs_9.db/db.69/000655.ldb
7f36e42dd000-7f36e4322000 rw-p 00000000 00:00 0 
7f36e4322000-7f36e45a6000 r--s 00000000 00:f1 16971                      /data_dir/28e07f34/blocks_and_extras/3.db/000424.ldb
7f36e45a6000-7f36e47b0000 r--s 00000000 00:f1 16970                      /data_dir/28e07f34/blocks_and_extras/3.db/000423.ldb
7f36e47b0000-7f36e49c7000 r--s 00000000 00:f1 16969                      /data_dir/28e07f34/blocks_and_extras/3.db/000422.ldb
7f36e49c7000-7f36e4a28000 rw-p 00000000 00:00 0 
7f36e4a28000-7f36e4c8e000 r--p 00003000 00:ef 8312153                    /usr/lib/debug/.build-id/7b/4536f41cdaa5888408e82d0836e33dcf436466.debug
7f36e4c8e000-7f36e4ffa000 r--s 00000000 00:f1 1067                       /data_dir/blocks_9.db/db.4/000720.ldb
7f36e4ffa000-7f36e4ffb000 ---p 00000000 00:00 0 
7f36e4ffb000-7f36e57fb000 rw-p 00000000 00:00 0 
7f36e8000000-7f36ebed8000 rw-p 00000000 00:00 0 
7f36ebed8000-7f36ec000000 ---p 00000000 00:00 0 
7f36ec000000-7f36ec0df000 rw-p 00000000 00:00 0 
7f36ec0df000-7f36f0000000 ---p 00000000 00:00 0 
7f36f0000000-7f36f0546000 rw-p 00000000 00:00 0 
7f36f0546000-7f36f4000000 ---p 00000000 00:00 0 
7f36f4000000-7f36f406e000 rw-p 00000000 00:00 0 
7f36f406e000-7f36f8000000 ---p 00000000 00:00 0 
7f36f8000000-7f36f80ca000 rw-p 00000000 00:00 0 
7f36f80ca000-7f36fc000000 ---p 00000000 00:00 0 
7f36fc000000-7f36fc021000 rw-p 00000000 00:00 0 
7f36fc021000-7f3700000000 ---p 00000000 00:00 0 
7f3700000000-7f37000e2000 rw-p 00000000 00:00 0 
7f37000e2000-7f3704000000 ---p 00000000 00:00 0 
7f3704000000-7f37041ae000 rw-p 00000000 00:00 0 
7f37041ae000-7f37043b1000 r--s 00000000 00:f1 94834                      /data_dir/da_proofs_9.db/db.69/000654.ldb
7f37043b1000-7f370442f000 rw-p 00000000 00:00 0 
7f370442f000-7f37045f5000 r--s 00000000 00:f1 96293                      /data_dir/da_proofs_9.db/db.70/000636.ldb
7f37045f5000-7f37047f9000 r--s 00000000 00:f1 94154                      /data_dir/da_sigshares_9.db/db.14/000587.ldb
7f3708000000-7f3708021000 rw-p 00000000 00:00 0 
7f3708021000-7f370c000000 ---p 00000000 00:00 0 
7f370c000000-7f370c6e4000 rw-p 00000000 00:00 0 
7f370c6e4000-7f3710000000 ---p 00000000 00:00 0 
7f3710000000-7f371002f000 rw-p 00000000 00:00 0 
7f371002f000-7f371025e000 r--s 00000000 00:f1 16968                      /data_dir/28e07f34/blocks_and_extras/3.db/000421.ldb
7f371025e000-7f3710462000 r--s 00000000 00:f1 16967                      /data_dir/28e07f34/blocks_and_extras/3.db/000420.ldb
7f3710462000-7f3710489000 rw-p 00000000 00:00 0 
7f3710489000-7f37105f5000 r--s 00000000 00:f1 96675                      /data_dir/da_sigshares_9.db/db.14/000853.ldb
7f37105f5000-7f37107f9000 r--s 00000000 00:f1 93693                      /data_dir/da_proofs_9.db/db.69/000154.ldb
7f37107f9000-7f37107fa000 ---p 00000000 00:00 0 
7f37107fa000-7f3710ffa000 rw-p 00000000 00:00 0 
7f3711ffc000-7f3711ffd000 ---p 00000000 00:00 0 
7f3711ffd000-7f37127fd000 rw-p 00000000 00:00 0 
7f37137ff000-7f3713800000 ---p 00000000 00:00 0 
7f3713800000-7f3714000000 rw-p 00000000 00:00 0 
7f3714000000-7f3714315000 rw-p 00000000 00:00 0 
7f3714315000-7f3718000000 ---p 00000000 00:00 0 
7f3718000000-7f3718028000 rw-p 00000000 00:00 0 
7f3718028000-7f371c000000 ---p 00000000 00:00 0 
7f371c000000-7f371cc49000 rw-p 00000000 00:00 0 
7f371cc49000-7f3720000000 ---p 00000000 00:00 0 
7f3720000000-7f3720021000 rw-p 00000000 00:00 0 
7f3720021000-7f3724000000 ---p 00000000 00:00 0 
7f3724000000-7f37240e4000 rw-p 00000000 00:00 0 
7f37240e4000-7f3728000000 ---p 00000000 00:00 0 
7f3728000000-7f3728021000 rw-p 00000000 00:00 0 
7f3728021000-7f372c000000 ---p 00000000 00:00 0 
7f372c000000-7f372c010000 rw-p 00000000 00:00 0 
7f372c010000-7f372c1bb000 r--s 00000000 00:f1 16974                      /data_dir/28e07f34/blocks_and_extras/3.db/000427.ldb
7f372c1bb000-7f372c3c0000 r--s 00000000 00:f1 93410                      /data_dir/block_proposals_9.db/db.4/000585.ldb
7f372c3c0000-7f372c3c2000 rw-p 00000000 00:00 0 
7f372c3c2000-7f372c5c5000 r--s 00000000 00:f1 369                        /data_dir/blocks_9.db/db.4/000078.ldb
7f372c5c5000-7f372c7f9000 r--s 00000000 00:f1 93409                      /data_dir/block_proposals_9.db/db.4/000584.ldb
7f372c7f9000-7f372c7fa000 ---p 00000000 00:00 0 
7f372c7fa000-7f372cffa000 rw-p 00000000 00:00 0 
7f372dffc000-7f372dffd000 ---p 00000000 00:00 0 
7f372dffd000-7f372e7fd000 rw-p 00000000 00:00 0 
7f372f7ff000-7f372f800000 ---p 00000000 00:00 0 
7f372f800000-7f3730000000 rw-p 00000000 00:00 0 
7f3730000000-7f3730022000 rw-p 00000000 00:00 0 
7f3730022000-7f3734000000 ---p 00000000 00:00 0 
7f3734000000-7f3734021000 rw-p 00000000 00:00 0 
7f3734021000-7f3738000000 ---p 00000000 00:00 0 
7f3738000000-7f3738094000 rw-p 00000000 00:00 0 
7f3738094000-7f373c000000 ---p 00000000 00:00 0 
7f373c000000-7f373c041000 rw-p 00000000 00:00 0 
7f373c041000-7f3740000000 ---p 00000000 00:00 0 
7f3740000000-7f3740021000 rw-p 00000000 00:00 0 
7f3740021000-7f3744000000 ---p 00000000 00:00 0 
7f3744000000-7f3744178000 rw-p 00000000 00:00 0 
7f3744178000-7f3748000000 ---p 00000000 00:00 0 
7f3748000000-7f3748252000 rw-p 00000000 00:00 0 
7f3748252000-7f374836b000 r--s 00000000 00:f1 96678                      /data_dir/da_proofs_9.db/db.71/000170.ldb
7f374836b000-7f37485b2000 r--s 00000000 00:f1 93408                      /data_dir/block_proposals_9.db/db.4/000583.ldb
7f37485b2000-7f37485c2000 rw-p 00000000 00:00 0 
7f37485c2000-7f37485c9000 r--p 00011000 00:ef 6489201                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f37485c9000-7f37485d7000 rw-p 00000000 00:00 0 
7f37485d7000-7f37485d9000 r--p 00001000 00:ef 6489226                    /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f37485d9000-7f37485f6000 rw-p 00000000 00:00 0 
7f37485f6000-7f37487f9000 r--s 00000000 00:f1 96292                      /data_dir/da_proofs_9.db/db.70/000635.ldb
7f3748ffa000-7f3748ffb000 ---p 00000000 00:00 0 
7f3748ffb000-7f37497fb000 rw-p 00000000 00:00 0 
7f374a7fd000-7f374a7fe000 ---p 00000000 00:00 0 
7f374a7fe000-7f374affe000 rw-p 00000000 00:00 0 
7f374affe000-7f374afff000 ---p 00000000 00:00 0 
7f374afff000-7f374b7ff000 rw-p 00000000 00:00 0 
7f374b7ff000-7f374b800000 ---p 00000000 00:00 0 
7f374b800000-7f374c000000 rw-p 00000000 00:00 0 
7f374c000000-7f374c021000 rw-p 00000000 00:00 0 
7f374c021000-7f3750000000 ---p 00000000 00:00 0 
7f3750000000-7f3750021000 rw-p 00000000 00:00 0 
7f3750021000-7f3754000000 ---p 00000000 00:00 0 
7f3754000000-7f3754034000 rw-p 00000000 00:00 0 
7f3754034000-7f3758000000 ---p 00000000 00:00 0 
7f3758000000-7f375bf5d000 rw-p 00000000 00:00 0 
7f375bf5d000-7f375c000000 ---p 00000000 00:00 0 
7f375c000000-7f375c02d000 rw-p 00000000 00:00 0 
7f375c02d000-7f3760000000 ---p 00000000 00:00 0 
7f3760000000-7f3760037000 rw-p 00000000 00:00 0 
7f3760037000-7f3764000000 ---p 00000000 00:00 0 
7f3764000000-7f376402f000 rw-p 00000000 00:00 0 
7f376402f000-7f3768000000 ---p 00000000 00:00 0 
7f3768000000-7f3768043000 rw-p 00000000 00:00 0 
7f3768043000-7f376c000000 ---p 00000000 00:00 0 
7f376c000000-7f376c037000 rw-p 00000000 00:00 0 
7f376c037000-7f3770000000 ---p 00000000 00:00 0 
7f3770000000-7f37700e8000 rw-p 00000000 00:00 0 
7f37700e8000-7f37702eb000 r--s 00000000 00:f1 94833                      /data_dir/da_proofs_9.db/db.69/000653.ldb
7f37702eb000-7f37703fc000 rw-p 00000000 00:00 0 
7f37703fc000-7f3770600000 r--s 00000000 00:f1 95626                      /data_dir/da_sigshares_9.db/db.14/000744.ldb
7f3770600000-7f37706ff000 r--s 00000000 00:f1 14977                      /data_dir/28e07f34/blocks_and_extras/2.db/000098.ldb
7f37706ff000-7f37707f9000 r--s 00000000 00:f1 16543                      /data_dir/28e07f34/blocks_and_extras/1.db/000097.ldb
7f37707f9000-7f37707fa000 ---p 00000000 00:00 0 
7f37707fa000-7f3770ffa000 rw-p 00000000 00:00 0 
7f3770ffa000-7f3770ffb000 ---p 00000000 00:00 0 
7f3770ffb000-7f37717fb000 rw-p 00000000 00:00 0 
7f37717fb000-7f37717fc000 ---p 00000000 00:00 0 
7f37717fc000-7f3771ffc000 rw-p 00000000 00:00 0 
7f37727fd000-7f37727fe000 ---p 00000000 00:00 0 
7f37727fe000-7f3772ffe000 rw-p 00000000 00:00 0 
7f3774000000-7f3774030000 rw-p 00000000 00:00 0 
7f3774030000-7f3778000000 ---p 00000000 00:00 0 
7f3778000000-7f3778030000 rw-p 00000000 00:00 0 
7f3778030000-7f377c000000 ---p 00000000 00:00 0 
7f377c000000-7f377c021000 rw-p 00000000 00:00 0 
7f377c021000-7f3780000000 ---p 00000000 00:00 0 
7f3780000000-7f378003f000 rw-p 00000000 00:00 0 
7f378003f000-7f3784000000 ---p 00000000 00:00 0 
7f3784000000-7f3784036000 rw-p 00000000 00:00 0 
7f3784036000-7f3788000000 ---p 00000000 00:00 0 
7f3788000000-7f3788029000 rw-p 00000000 00:00 0 
7f3788029000-7f3788072000 r--p 0002c000 00:ef 6489311                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f3788072000-7f378816f000 r--s 00000000 00:f1 16563                      /data_dir/28e07f34/blocks_and_extras/0.db/000092.ldb
7f378816f000-7f3788373000 r--s 00000000 00:f1 368                        /data_dir/blocks_9.db/db.4/000077.ldb
7f3788373000-7f37884ee000 r--s 00000000 00:f1 90740                      /data_dir/da_sigshares_9.db/db.11/000910.ldb
7f37884ee000-7f378867e000 r--s 00000000 00:f1 90739                      /data_dir/da_sigshares_9.db/db.12/000915.ldb
7f378867e000-7f37887f9000 r--s 00000000 00:f1 90738                      /data_dir/da_sigshares_9.db/db.13/000910.ldb
7f378c000000-7f378c03d000 rw-p 00000000 00:00 0 
7f378c03d000-7f3790000000 ---p 00000000 00:00 0 
7f3790000000-7f3790035000 rw-p 00000000 00:00 0 
7f3790035000-7f3794000000 ---p 00000000 00:00 0 
7f3794000000-7f3794030000 rw-p 00000000 00:00 0 
7f3794030000-7f3798000000 ---p 00000000 00:00 0 
7f3798000000-7f3798038000 rw-p 00000000 00:00 0 
7f3798038000-7f379c000000 ---p 00000000 00:00 0 
7f379c000000-7f379c02f000 rw-p 00000000 00:00 0 
7f379c02f000-7f37a0000000 ---p 00000000 00:00 0 
7f37a0000000-7f37a0018000 rw-p 00000000 00:00 0 
7f37a0018000-7f37a00be000 r--s 00000000 00:f1 1045                       /data_dir/blocks_9.db/db.4/000714.ldb
7f37a00be000-7f37a020c000 r--s 00000000 00:f1 90468                      /data_dir/da_sigshares_9.db/db.14/000261.ldb
7f37a020c000-7f37a03f2000 r--s 00000000 00:f1 59953                      /data_dir/block_proposals_9.db/db.4/000548.ldb
7f37a03f2000-7f37a05f6000 r--s 00000000 00:f1 19903                      /data_dir/block_proposals_9.db/db.4/000009.ldb
7f37a05f6000-7f37a07f9000 r--s 00000000 00:f1 30213                      /data_dir/proposal_hashes_9.db/db.4/000553.ldb
7f37a4000000-7f37a40d9000 rw-p 00000000 00:00 0 
7f37a40d9000-7f37a8000000 ---p 00000000 00:00 0 
7f37a8000000-7f37abe82000 rw-p 00000000 00:00 0 
7f37abe82000-7f37ac000000 ---p 00000000 00:00 0 
7f37ac000000-7f37ac021000 rw-p 00000000 00:00 0 
7f37ac021000-7f37b0000000 ---p 00000000 00:00 0 
7f37b0000000-7f37b0021000 rw-p 00000000 00:00 0 
7f37b0021000-7f37b4000000 ---p 00000000 00:00 0 
7f37b4000000-7f37b4021000 rw-p 00000000 00:00 0 
7f37b4021000-7f37b8000000 ---p 00000000 00:00 0 
7f37b8000000-7f37bbe82000 rw-p 00000000 00:00 0 
7f37bbe82000-7f37bc000000 ---p 00000000 00:00 0 
7f37bc000000-7f37bc08a000 rw-p 00000000 00:00 0 
7f37bc08a000-7f37bc131000 r--s 00000000 00:f1 93414                      /data_dir/da_proofs_9.db/db.68/000693.ldb
7f37bc131000-7f37bc14c000 r--s 00000000 00:f1 93411                      /data_dir/block_proposals_9.db/db.4/000586.ldb
7f37bc14c000-7f37bc150000 rw-p 00000000 00:00 0 
7f37bc150000-7f37bc166000 r--p 00024000 00:ef 8304370                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f37bc166000-7f37bc1f0000 r--s 00000000 00:f1 93239                      /data_dir/proposal_hashes_9.db/db.6/000472.ldb
7f37bc1f0000-7f37bc45a000 r--s 00000000 00:f1 1062                       /data_dir/blocks_9.db/db.4/000717.ldb
7f37bec1c000-7f37bec5f000 rw-p 00000000 00:00 0 
7f37bec5f000-7f37bec60000 ---p 00000000 00:00 0 
7f37bec60000-7f37bf460000 rw-p 00000000 00:00 0 
7f37bf460000-7f37bf5ed000 r--s 00000000 00:f1 16514                      /data_dir/28e07f34/blocks_and_extras/4.db/000098.ldb
7f37bf5ed000-7f37bf5fb000 rw-p 00000000 00:00 0 
7f37bf5fb000-7f37bf7ff000 r--s 00000000 00:f1 96556                      /data_dir/da_proofs_9.db/db.71/000115.ldb
7f37bf7ff000-7f37bf800000 ---p 00000000 00:00 0 
7f37bf800000-7f37c0000000 rw-p 00000000 00:00 0 
7f37c0000000-7f37c0021000 rw-p 00000000 00:00 0 
7f37c0021000-7f37c4000000 ---p 00000000 00:00 0 
7f37c4000000-7f37c4021000 rw-p 00000000 00:00 0 
7f37c4021000-7f37c8000000 ---p 00000000 00:00 0 
7f37c8000000-7f37c8021000 rw-p 00000000 00:00 0 
7f37c8021000-7f37cc000000 ---p 00000000 00:00 0 
7f37cc000000-7f37cc021000 rw-p 00000000 00:00 0 
7f37cc021000-7f37d0000000 ---p 00000000 00:00 0 
7f37d0000000-7f37d0021000 rw-p 00000000 00:00 0 
7f37d0021000-7f37d4000000 ---p 00000000 00:00 0 
7f37d4000000-7f37d4021000 rw-p 00000000 00:00 0 
7f37d4021000-7f37d8000000 ---p 00000000 00:00 0 
7f37d8000000-7f37d8021000 rw-p 00000000 00:00 0 
7f37d8021000-7f37dc000000 ---p 00000000 00:00 0 
7f37dc000000-7f37dc021000 rw-p 00000000 00:00 0 
7f37dc021000-7f37e0000000 ---p 00000000 00:00 0 
7f37e0000000-7f37e0021000 rw-p 00000000 00:00 0 
7f37e0021000-7f37e4000000 ---p 00000000 00:00 0 
7f37e4000000-7f37e4021000 rw-p 00000000 00:00 0 
7f37e4021000-7f37e8000000 ---p 00000000 00:00 0 
7f37e8000000-7f37e8021000 rw-p 00000000 00:00 0 
7f37e8021000-7f37ec000000 ---p 00000000 00:00 0 
7f37ec000000-7f37ec021000 rw-p 00000000 00:00 0 
7f37ec021000-7f37f0000000 ---p 00000000 00:00 0 
7f37f0000000-7f37f3e8e000 rw-p 00000000 00:00 0 
7f37f3e8e000-7f37f4000000 ---p 00000000 00:00 0 
7f37f4000000-7f37f4021000 rw-p 00000000 00:00 0 
7f37f4021000-7f37f8000000 ---p 00000000 00:00 0 
7f37f8000000-7f37f8021000 rw-p 00000000 00:00 0 
7f37f8021000-7f37fc000000 ---p 00000000 00:00 0 
7f37fc000000-7f37fc011000 rw-p 00000000 00:00 0 
7f37fc011000-7f37fc01c000 r--s 00000000 00:f1 93518                      /data_dir/proposal_hashes_9.db/db.6/000474.ldb
7f37fc01c000-7f37fc049000 r--s 00000000 00:f1 93407                      /data_dir/block_proposals_9.db/db.4/000582.ldb
7f37fc049000-7f37fc04f000 rw-p 00000000 00:00 0 
7f37fc04f000-7f37fc064000 r--s 00000000 00:f1 330                        /data_dir/prices_9.db/db.4/000025.ldb
7f37fc064000-7f37fc095000 r--s 00000000 00:f1 60983                      /data_dir/proposal_vectors_9.db/db.4/000041.ldb
7f37fc095000-7f37fc0b5000 rw-p 00000000 00:00 0 
7f37fc0b5000-7f37fc0b8000 r--p 00009000 00:ef 6489243                    /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f37fc0b8000-7f37fc0d8000 rw-p 00000000 00:00 0 
7f37fc0d8000-7f37fc0e4000 r--p 0000d000 00:ef 6489232                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30.27.0
7f37fc0e4000-7f37fc0ef000 r--s 00000000 00:f1 93269                      /data_dir/outgoing_msgs_9.db/db.31/001427.ldb
7f37fc0ef000-7f37fc0f3000 r--p 00000000 00:ef 6489297                    /usr/lib/x86_64-linux-gnu/libresolv-2.31.so
7f37fc0f3000-7f37fc103000 r-xp 00004000 00:ef 6489297                    /usr/lib/x86_64-linux-gnu/libresolv-2.31.so
7f37fc103000-7f37fc107000 r--p 00014000 00:ef 6489297                    /usr/lib/x86_64-linux-gnu/libresolv-2.31.so
7f37fc107000-7f37fc108000 r--p 00017000 00:ef 6489297                    /usr/lib/x86_64-linux-gnu/libresolv-2.31.so
7f37fc108000-7f37fc109000 rw-p 00018000 00:ef 6489297                    /usr/lib/x86_64-linux-gnu/libresolv-2.31.so
7f37fc109000-7f37fc115000 rw-p 00000000 00:00 0 
7f37fc115000-7f37fc117000 r--p 00002000 00:ef 6489228                    /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.2.5
7f37fc117000-7f37fc119000 rw-p 00000000 00:00 0 
7f37fc119000-7f37fc11a000 r--p 00000000 00:ef 6489212                    /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f37fc11a000-7f37fc11b000 r--p 00000000 00:ef 6489238                    /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.6
7f37fc11b000-7f37fc128000 rw-p 00000000 00:00 0 
7f37fc128000-7f37fc15f000 r--s 00000000 00:f1 96587                      /data_dir/proposal_hashes_9.db/db.6/000529.ldb
7f37fc15f000-7f37fc1ba000 r--s 00000000 00:f1 96522                      /data_dir/outgoing_msgs_9.db/db.32/001384.ldb
7f37fc1ba000-7f37fc248000 r--s 00000000 00:f1 95122                      /data_dir/proposal_hashes_9.db/db.6/000503.ldb
7f37fc248000-7f37fc26c000 rw-p 00000000 00:00 0 
7f37fc26c000-7f37fc2bb000 r--s 00000000 00:f1 96110                      /data_dir/outgoing_msgs_9.db/db.32/001209.ldb
7f37fc2bb000-7f37fc2ff000 rw-p 00000000 00:00 0 
7f37fc2ff000-7f37fc31e000 r--s 00000000 00:f1 96683                      /data_dir/outgoing_msgs_9.db/db.33/000067.ldb
7f37fc31e000-7f37fc592000 r--s 00000000 00:f1 14974                      /data_dir/28e07f34/blocks_and_extras/2.db/000095.ldb
7f37fc592000-7f37fc7f9000 r--s 00000000 00:f1 16510                      /data_dir/28e07f34/blocks_and_extras/4.db/000094.ldb
7f37fc7f9000-7f37fc7fa000 ---p 00000000 00:00 0 
7f37fc7fa000-7f37fcffa000 rw-p 00000000 00:00 0 
7f37fcffa000-7f37fcffb000 ---p 00000000 00:00 0 
7f37fcffb000-7f37fd7fb000 rw-p 00000000 00:00 0 
7f37fd7fb000-7f37fd7fc000 ---p 00000000 00:00 0 
7f37fd7fc000-7f37fdffc000 rw-p 00000000 00:00 0 
7f37fdffc000-7f37fdffd000 ---p 00000000 00:00 0 
7f37fdffd000-7f37fe7fd000 rw-p 00000000 00:00 0 
7f37fe7fd000-7f37fe7fe000 ---p 00000000 00:00 0 
7f37fe7fe000-7f37feffe000 rw-p 00000000 00:00 0 
7f37feffe000-7f37fefff000 ---p 00000000 00:00 0 
7f37fefff000-7f37ff7ff000 rw-p 00000000 00:00 0 
7f37ff7ff000-7f37ff800000 ---p 00000000 00:00 0 
7f37ff800000-7f3800000000 rw-p 00000000 00:00 0 
7f3800000000-7f3800021000 rw-p 00000000 00:00 0 
7f3800021000-7f3804000000 ---p 00000000 00:00 0 
7f3804000000-7f3804010000 rw-p 00000000 00:00 0 
7f3804010000-7f3804013000 r--s 00000000 00:f1 93250                      /data_dir/proposal_vectors_9.db/db.4/000067.ldb
7f3804013000-7f3804052000 r--s 00000000 00:f1 16902                      /data_dir/28e07f34/12041/state/014847.ldb
7f3804052000-7f3804260000 r--s 00000000 00:f1 15802                      /data_dir/28e07f34/blocks_and_extras/0.db/000079.ldb
7f3804260000-7f380427f000 rw-p 00000000 00:00 0 
7f380427f000-7f38042e8000 r--s 00000000 00:f1 96428                      /data_dir/outgoing_msgs_9.db/db.32/001343.ldb
7f38042e8000-7f3804308000 r--s 00000000 00:f1 95967                      /data_dir/outgoing_msgs_9.db/db.32/001148.ldb
7f3804308000-7f3804313000 rw-p 00000000 00:00 0 
7f3804313000-7f380432e000 r--s 00000000 00:f1 96533                      /data_dir/outgoing_msgs_9.db/db.32/001387.ldb
7f380432e000-7f380432f000 rw-p 00000000 00:00 0 
7f380432f000-7f3804344000 r--s 00000000 00:f1 96580                      /data_dir/outgoing_msgs_9.db/db.33/000022.ldb
7f3804344000-7f3804360000 r--s 00000000 00:f1 95178                      /data_dir/outgoing_msgs_9.db/db.32/000801.ldb
7f3804360000-7f3804364000 rw-p 00000000 00:00 0 
7f3804364000-7f3804374000 r--s 00000000 00:f1 96640                      /data_dir/outgoing_msgs_9.db/db.33/000046.ldb
7f3804374000-7f3804384000 r--s 00000000 00:f1 96532                      /data_dir/outgoing_msgs_9.db/db.32/001386.ldb
7f3804384000-7f3804399000 r--s 00000000 00:f1 96305                      /data_dir/outgoing_msgs_9.db/db.32/001290.ldb
7f3804399000-7f380439a000 rw-p 00000000 00:00 0 
7f380439a000-7f38043a5000 r--s 00000000 00:f1 96695                      /data_dir/da_proofs_9.db/db.71/000176.ldb
7f38043a5000-7f38043d1000 r--s 00000000 00:f1 96206                      /data_dir/proposal_hashes_9.db/db.6/000522.ldb
7f38043d1000-7f38043dc000 r--s 00000000 00:f1 95847                      /data_dir/proposal_hashes_9.db/db.6/000515.ldb
7f38043dc000-7f38043e2000 rw-p 00000000 00:00 0 
7f38043e2000-7f38043fd000 r--s 00000000 00:f1 96191                      /data_dir/outgoing_msgs_9.db/db.32/001245.ldb
7f38043fd000-7f3804408000 r--s 00000000 00:f1 96185                      /data_dir/outgoing_msgs_9.db/db.32/001242.ldb
7f3804408000-7f3804413000 r--s 00000000 00:f1 95243                      /data_dir/proposal_hashes_9.db/db.6/000505.ldb
7f3804413000-7f3804418000 rw-p 00000000 00:00 0 
7f3804418000-7f380441e000 r--s 00000000 00:f1 96099                      /data_dir/outgoing_msgs_9.db/db.32/001204.ldb
7f380441e000-7f3804429000 r--s 00000000 00:f1 96691                      /data_dir/da_sigshares_9.db/db.14/000855.ldb
7f3804429000-7f3804439000 r--s 00000000 00:f1 96254                      /data_dir/outgoing_msgs_9.db/db.32/001270.ldb
7f3804439000-7f380443f000 r--s 00000000 00:f1 95671                      /data_dir/outgoing_msgs_9.db/db.32/001015.ldb
7f380443f000-7f3804449000 rw-p 00000000 00:00 0 
7f3804449000-7f3804455000 r--s 00000000 00:f1 95685                      /data_dir/outgoing_msgs_9.db/db.32/001023.ldb
7f3804455000-7f3804460000 r--s 00000000 00:f1 96287                      /data_dir/da_proofs_9.db/db.71/000005.ldb
7f3804460000-7f3804463000 rw-p 00000000 00:00 0 
7f3804463000-7f3804474000 r--s 00000000 00:f1 96606                      /data_dir/outgoing_msgs_9.db/db.33/000033.ldb
7f3804474000-7f380447f000 r--s 00000000 00:f1 95964                      /data_dir/proposal_hashes_9.db/db.6/000517.ldb
7f380447f000-7f3804487000 r--s 00000000 00:f1 94838                      /data_dir/da_proofs_9.db/db.69/000658.ldb
7f3804487000-7f3804489000 rw-p 00000000 00:00 0 
7f3804489000-7f38044a3000 r--s 00000000 00:f1 96126                      /data_dir/outgoing_msgs_9.db/db.32/001217.ldb
7f38044a3000-7f38044ae000 r--s 00000000 00:f1 96687                      /data_dir/da_proofs_9.db/db.71/000174.ldb
7f38044ae000-7f38044ba000 r--s 00000000 00:f1 96121                      /data_dir/outgoing_msgs_9.db/db.32/001214.ldb
7f38044ba000-7f38044be000 rw-p 00000000 00:00 0 
7f38044be000-7f38044c4000 r--s 00000000 00:f1 96146                      /data_dir/outgoing_msgs_9.db/db.32/001225.ldb
7f38044c4000-7f38044c5000 rw-p 00000000 00:00 0 
7f38044c5000-7f38044cb000 r--s 00000000 00:f1 96228                      /data_dir/outgoing_msgs_9.db/db.32/001259.ldb
7f38044cb000-7f38044cd000 rw-p 00000000 00:00 0 
7f38044cd000-7f38044dd000 r--s 00000000 00:f1 96281                      /data_dir/outgoing_msgs_9.db/db.32/001279.ldb
7f38044dd000-7f38044e3000 rw-p 00000000 00:00 0 
7f38044e3000-7f38044e9000 r--s 00000000 00:f1 95979                      /data_dir/outgoing_msgs_9.db/db.32/001154.ldb
7f38044e9000-7f38044eb000 rw-p 00000000 00:00 0 
7f38044eb000-7f38044f7000 r--s 00000000 00:f1 96452                      /data_dir/outgoing_msgs_9.db/db.32/001354.ldb
7f38044f7000-7f38044fd000 r--s 00000000 00:f1 96674                      /data_dir/outgoing_msgs_9.db/db.33/000062.ldb
7f38044fd000-7f3804503000 r--s 00000000 00:f1 96670                      /data_dir/outgoing_msgs_9.db/db.33/000060.ldb
7f3804503000-7f3804509000 r--s 00000000 00:f1 96142                      /data_dir/outgoing_msgs_9.db/db.32/001223.ldb
7f3804509000-7f380450f000 r--s 00000000 00:f1 95944                      /data_dir/outgoing_msgs_9.db/db.32/001139.ldb
7f380450f000-7f3804514000 rw-p 00000000 00:00 0 
7f3804514000-7f380451a000 r--s 00000000 00:f1 95977                      /data_dir/outgoing_msgs_9.db/db.32/001152.ldb
7f380451a000-7f3804525000 r--s 00000000 00:f1 96649                      /data_dir/outgoing_msgs_9.db/db.33/000051.ldb
7f3804525000-7f380452b000 r--s 00000000 00:f1 96568                      /data_dir/outgoing_msgs_9.db/db.33/000017.ldb
7f380452b000-7f3804530000 rw-p 00000000 00:00 0 
7f3804530000-7f3804535000 rw-p 00000000 00:00 0 
7f3804535000-7f3804540000 r--s 00000000 00:f1 96685                      /data_dir/da_proofs_9.db/db.71/000172.ldb
7f3804540000-7f3804546000 r--s 00000000 00:f1 96682                      /data_dir/outgoing_msgs_9.db/db.33/000066.ldb
7f3804546000-7f380454c000 r--s 00000000 00:f1 96222                      /data_dir/outgoing_msgs_9.db/db.32/001257.ldb
7f380454c000-7f3804550000 r--p 00022000 00:ef 6489295                    /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f3804550000-7f3804556000 r--s 00000000 00:f1 96411                      /data_dir/outgoing_msgs_9.db/db.32/001336.ldb
7f3804556000-7f380455c000 r--s 00000000 00:f1 96289                      /data_dir/outgoing_msgs_9.db/db.32/001283.ldb
7f380455c000-7f3804562000 r--s 00000000 00:f1 96233                      /data_dir/outgoing_msgs_9.db/db.32/001261.ldb
7f3804562000-7f3804568000 r--s 00000000 00:f1 96095                      /data_dir/outgoing_msgs_9.db/db.32/001202.ldb
7f3804568000-7f380456e000 r--s 00000000 00:f1 96089                      /data_dir/outgoing_msgs_9.db/db.32/001200.ldb
7f380456e000-7f3804571000 rw-p 00000000 00:00 0 
7f3804571000-7f380457c000 r--s 00000000 00:f1 96531                      /data_dir/outgoing_msgs_9.db/db.32/001385.ldb
7f380457c000-7f3804581000 r--s 00000000 00:f1 96693                      /data_dir/outgoing_msgs_9.db/db.33/000071.ldb
7f3804581000-7f3804587000 r--s 00000000 00:f1 96680                      /data_dir/outgoing_msgs_9.db/db.33/000064.ldb
7f3804587000-7f380458d000 r--s 00000000 00:f1 96407                      /data_dir/outgoing_msgs_9.db/db.32/001334.ldb
7f380458d000-7f3804593000 r--s 00000000 00:f1 96218                      /data_dir/outgoing_msgs_9.db/db.32/001255.ldb
7f3804593000-7f3804599000 r--s 00000000 00:f1 96403                      /data_dir/outgoing_msgs_9.db/db.32/001332.ldb
7f3804599000-7f380459f000 r--s 00000000 00:f1 96136                      /data_dir/outgoing_msgs_9.db/db.32/001221.ldb
7f380459f000-7f38045a5000 r--s 00000000 00:f1 96197                      /data_dir/outgoing_msgs_9.db/db.32/001247.ldb
7f38045a5000-7f38045aa000 r--s 00000000 00:f1 96626                      /data_dir/outgoing_msgs_9.db/db.33/000041.ldb
7f38045aa000-7f38045b0000 r--s 00000000 00:f1 95936                      /data_dir/outgoing_msgs_9.db/db.32/001135.ldb
7f38045b0000-7f38045b6000 r--s 00000000 00:f1 95697                      /data_dir/outgoing_msgs_9.db/db.32/001027.ldb
7f38045b6000-7f38045bc000 r--s 00000000 00:f1 96666                      /data_dir/outgoing_msgs_9.db/db.33/000058.ldb
7f38045bc000-7f38045c1000 r--s 00000000 00:f1 96164                      /data_dir/outgoing_msgs_9.db/db.32/001233.ldb
7f38045c1000-7f38045c7000 r--s 00000000 00:f1 96151                      /data_dir/outgoing_msgs_9.db/db.32/001227.ldb
7f38045c7000-7f38045cd000 r--s 00000000 00:f1 96563                      /data_dir/outgoing_msgs_9.db/db.33/000015.ldb
7f38045cd000-7f38045ce000 rw-p 00000000 00:00 0 
7f38045ce000-7f38045d3000 r--s 00000000 00:f1 96689                      /data_dir/outgoing_msgs_9.db/db.33/000069.ldb
7f38045d3000-7f38045d8000 r--s 00000000 00:f1 96417                      /data_dir/outgoing_msgs_9.db/db.32/001338.ldb
7f38045d8000-7f38045dd000 r--s 00000000 00:f1 96258                      /data_dir/outgoing_msgs_9.db/db.32/001272.ldb
7f38045dd000-7f38045de000 rw-p 00000000 00:00 0 
7f38045de000-7f38045e3000 r--s 00000000 00:f1 96591                      /data_dir/outgoing_msgs_9.db/db.33/000026.ldb
7f38045e3000-7f38045e8000 r--s 00000000 00:f1 95643                      /data_dir/outgoing_msgs_9.db/db.32/001003.ldb
7f38045e8000-7f38045e9000 rw-p 00000000 00:00 0 
7f38045e9000-7f38045ee000 r--s 00000000 00:f1 96612                      /data_dir/outgoing_msgs_9.db/db.33/000035.ldb
7f38045ee000-7f38045f4000 r--s 00000000 00:f1 96297                      /data_dir/outgoing_msgs_9.db/db.32/001285.ldb
7f38045f4000-7f38045fa000 r--s 00000000 00:f1 96539                      /data_dir/outgoing_msgs_9.db/db.33/000005.ldb
7f38045fa000-7f3804600000 r--s 00000000 00:f1 96132                      /data_dir/outgoing_msgs_9.db/db.32/001219.ldb
7f3804600000-7f3804605000 r--s 00000000 00:f1 96431                      /data_dir/outgoing_msgs_9.db/db.32/001345.ldb
7f3804605000-7f380460a000 r--s 00000000 00:f1 96237                      /data_dir/outgoing_msgs_9.db/db.32/001263.ldb
7f380460a000-7f380460b000 ---p 00000000 00:00 0 
7f380460b000-7f3804e0b000 rw-p 00000000 00:00 0 
7f3804e0b000-7f3804e0c000 ---p 00000000 00:00 0 
7f3804e0c000-7f380560c000 rw-p 00000000 00:00 0 
7f380560c000-7f380560d000 ---p 00000000 00:00 0 
7f380560d000-7f3805e0d000 rw-p 00000000 00:00 0 
7f3805e0d000-7f3805e0e000 ---p 00000000 00:00 0 
7f3805e0e000-7f380660e000 rw-p 00000000 00:00 0 
7f380660e000-7f380660f000 ---p 00000000 00:00 0 
7f380660f000-7f3806e0f000 rw-p 00000000 00:00 0 
7f3806e0f000-7f3806e10000 ---p 00000000 00:00 0 
7f3806e10000-7f3807610000 rw-p 00000000 00:00 0 
7f3807610000-7f3807611000 ---p 00000000 00:00 0 
7f3807611000-7f3807e11000 rw-p 00000000 00:00 0 
7f3807e11000-7f3807e12000 ---p 00000000 00:00 0 
7f3807e12000-7f3808612000 rw-p 00000000 00:00 0 
7f3808612000-7f3808613000 ---p 00000000 00:00 0 
7f3808613000-7f3808e13000 rw-p 00000000 00:00 0 
7f3808e13000-7f3808e14000 ---p 00000000 00:00 0 
7f3808e14000-7f3809760000 rw-p 00000000 00:00 0 
7f3809760000-7f3809761000 r--p 00000000 00:ef 6489264                    /usr/lib/x86_64-linux-gnu/libnss_dns-2.31.so
7f3809761000-7f3809765000 r-xp 00001000 00:ef 6489264                    /usr/lib/x86_64-linux-gnu/libnss_dns-2.31.so
7f3809765000-7f3809766000 r--p 00005000 00:ef 6489264                    /usr/lib/x86_64-linux-gnu/libnss_dns-2.31.so
7f3809766000-7f3809767000 r--p 00005000 00:ef 6489264                    /usr/lib/x86_64-linux-gnu/libnss_dns-2.31.so
7f3809767000-7f3809768000 rw-p 00006000 00:ef 6489264                    /usr/lib/x86_64-linux-gnu/libnss_dns-2.31.so
7f3809768000-7f3809769000 r--s 00000000 00:f1 19838                      /data_dir/internal_info_9.db/db.4/000005.ldb
7f3809769000-7f380976b000 r--s 00000000 00:f1 30227                      /data_dir/proposal_hashes_9.db/db.4/000557.ldb
7f380976b000-7f380976d000 r--s 00000000 00:f1 408                        /data_dir/prices_9.db/db.4/000063.ldb
7f380976d000-7f380986e000 rw-p 00000000 00:00 0 
7f380986e000-7f3809871000 r--p 00000000 00:ef 6489266                    /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so
7f3809871000-7f3809878000 r-xp 00003000 00:ef 6489266                    /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so
7f3809878000-7f380987a000 r--p 0000a000 00:ef 6489266                    /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so
7f380987a000-7f380987b000 r--p 0000b000 00:ef 6489266                    /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so
7f380987b000-7f380987c000 rw-p 0000c000 00:ef 6489266                    /usr/lib/x86_64-linux-gnu/libnss_files-2.31.so
7f380987c000-7f380988f000 rw-p 00000000 00:00 0 
7f380988f000-7f3809891000 r--p 00000000 00:ef 6489221                    /usr/lib/x86_64-linux-gnu/libffi.so.7.1.0
7f3809891000-7f3809897000 r-xp 00002000 00:ef 6489221                    /usr/lib/x86_64-linux-gnu/libffi.so.7.1.0
7f3809897000-7f3809898000 r--p 00008000 00:ef 6489221                    /usr/lib/x86_64-linux-gnu/libffi.so.7.1.0
7f3809898000-7f3809899000 ---p 00009000 00:ef 6489221                    /usr/lib/x86_64-linux-gnu/libffi.so.7.1.0
7f3809899000-7f380989a000 r--p 00009000 00:ef 6489221                    /usr/lib/x86_64-linux-gnu/libffi.so.7.1.0
7f380989a000-7f380989b000 rw-p 0000a000 00:ef 6489221                    /usr/lib/x86_64-linux-gnu/libffi.so.7.1.0
7f380989b000-7f380989d000 rw-p 00000000 00:00 0 
7f380989d000-7f38098a1000 r--p 00000000 00:ef 6489234                    /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.28.0
7f38098a1000-7f38098b4000 r-xp 00004000 00:ef 6489234                    /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.28.0
7f38098b4000-7f38098be000 r--p 00017000 00:ef 6489234                    /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.28.0
7f38098be000-7f38098bf000 r--p 00020000 00:ef 6489234                    /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.28.0
7f38098bf000-7f38098c0000 rw-p 00021000 00:ef 6489234                    /usr/lib/x86_64-linux-gnu/libgpg-error.so.0.28.0
7f38098c0000-7f38098ca000 r--p 00000000 00:ef 6489230                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.0
7f38098ca000-7f380992a000 r-xp 0000a000 00:ef 6489230                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.0
7f380992a000-7f3809941000 r--p 0006a000 00:ef 6489230                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.0
7f3809941000-7f3809942000 ---p 00081000 00:ef 6489230                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.0
7f3809942000-7f3809943000 r--p 00081000 00:ef 6489230                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.0
7f3809943000-7f3809944000 rw-p 00082000 00:ef 6489230                    /usr/lib/x86_64-linux-gnu/libgmp.so.10.4.0
7f3809944000-7f380994b000 r--p 00000000 00:ef 6489236                    /usr/lib/x86_64-linux-gnu/libhogweed.so.5.0
7f380994b000-7f380995b000 r-xp 00007000 00:ef 6489236                    /usr/lib/x86_64-linux-gnu/libhogweed.so.5.0
7f380995b000-7f3809979000 r--p 00017000 00:ef 6489236                    /usr/lib/x86_64-linux-gnu/libhogweed.so.5.0
7f3809979000-7f380997a000 r--p 00034000 00:ef 6489236                    /usr/lib/x86_64-linux-gnu/libhogweed.so.5.0
7f380997a000-7f380997b000 rw-p 00035000 00:ef 6489236                    /usr/lib/x86_64-linux-gnu/libhogweed.so.5.0
7f380997b000-7f3809984000 r--p 00000000 00:ef 6489259                    /usr/lib/x86_64-linux-gnu/libnettle.so.7.0
7f3809984000-7f38099a2000 r-xp 00009000 00:ef 6489259                    /usr/lib/x86_64-linux-gnu/libnettle.so.7.0
7f38099a2000-7f38099b2000 r--p 00027000 00:ef 6489259                    /usr/lib/x86_64-linux-gnu/libnettle.so.7.0
7f38099b2000-7f38099b4000 r--p 00036000 00:ef 6489259                    /usr/lib/x86_64-linux-gnu/libnettle.so.7.0
7f38099b4000-7f38099b5000 rw-p 00038000 00:ef 6489259                    /usr/lib/x86_64-linux-gnu/libnettle.so.7.0
7f38099b5000-7f38099b8000 r--p 00000000 00:ef 6489315                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.0
7f38099b8000-7f38099c4000 r-xp 00003000 00:ef 6489315                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.0
7f38099c4000-7f38099c8000 r--p 0000f000 00:ef 6489315                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.0
7f38099c8000-7f38099c9000 ---p 00013000 00:ef 6489315                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.0
7f38099c9000-7f38099ca000 r--p 00013000 00:ef 6489315                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.0
7f38099ca000-7f38099cb000 rw-p 00014000 00:ef 6489315                    /usr/lib/x86_64-linux-gnu/libtasn1.so.6.6.0
7f38099cb000-7f38099f6000 r--p 00000000 00:ef 6489275                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
7f38099f6000-7f3809a90000 r-xp 0002b000 00:ef 6489275                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
7f3809a90000-7f3809aec000 r--p 000c5000 00:ef 6489275                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
7f3809aec000-7f3809af7000 r--p 00120000 00:ef 6489275                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
7f3809af7000-7f3809b01000 rw-p 0012b000 00:ef 6489275                    /usr/lib/x86_64-linux-gnu/libp11-kit.so.0.3.0
7f3809b01000-7f3809b03000 rw-p 00000000 00:00 0 
7f3809b03000-7f3809b13000 r--p 00000000 00:ef 6489325                    /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
7f3809b13000-7f3809b49000 r-xp 00010000 00:ef 6489325                    /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
7f3809b49000-7f3809c80000 r--p 00046000 00:ef 6489325                    /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
7f3809c80000-7f3809c81000 ---p 0017d000 00:ef 6489325                    /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
7f3809c81000-7f3809c84000 r--p 0017d000 00:ef 6489325                    /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
7f3809c84000-7f3809c85000 rw-p 00180000 00:ef 6489325                    /usr/lib/x86_64-linux-gnu/libunistring.so.2.1.0
7f3809c85000-7f3809ca7000 r--p 00000000 00:ef 6489201                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f3809ca7000-7f3809e1f000 r-xp 00022000 00:ef 6489201                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f3809e1f000-7f3809e6d000 r--p 0019a000 00:ef 6489201                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f3809e6d000-7f3809e71000 r--p 001e7000 00:ef 6489201                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f3809e71000-7f3809e73000 rw-p 001eb000 00:ef 6489201                    /usr/lib/x86_64-linux-gnu/libc-2.31.so
7f3809e73000-7f3809e77000 rw-p 00000000 00:00 0 
7f3809e77000-7f3809e7a000 r--p 00000000 00:ef 6489226                    /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f3809e7a000-7f3809e8c000 r-xp 00003000 00:ef 6489226                    /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f3809e8c000-7f3809e90000 r--p 00015000 00:ef 6489226                    /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f3809e90000-7f3809e91000 r--p 00018000 00:ef 6489226                    /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f3809e91000-7f3809e92000 rw-p 00019000 00:ef 6489226                    /usr/lib/x86_64-linux-gnu/libgcc_s.so.1
7f3809e92000-7f3809e9f000 r--p 00000000 00:ef 6489243                    /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f3809e9f000-7f3809f46000 r-xp 0000d000 00:ef 6489243                    /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f3809f46000-7f3809fdf000 r--p 000b4000 00:ef 6489243                    /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f3809fdf000-7f3809fe0000 r--p 0014c000 00:ef 6489243                    /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f3809fe0000-7f3809fe1000 rw-p 0014d000 00:ef 6489243                    /usr/lib/x86_64-linux-gnu/libm-2.31.so
7f3809fe1000-7f380a077000 r--p 00000000 00:ef 6489311                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f380a077000-7f380a168000 r-xp 00096000 00:ef 6489311                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f380a168000-7f380a1b1000 r--p 00187000 00:ef 6489311                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f380a1b1000-7f380a1b2000 ---p 001d0000 00:ef 6489311                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f380a1b2000-7f380a1bd000 r--p 001d0000 00:ef 6489311                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f380a1bd000-7f380a1c0000 rw-p 001db000 00:ef 6489311                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28
7f380a1c0000-7f380a1c3000 rw-p 00000000 00:00 0 
7f380a1c3000-7f380a23b000 r--p 00000000 00:ef 8304370                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f380a23b000-7f380a3d6000 r-xp 00078000 00:ef 8304370                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f380a3d6000-7f380a467000 r--p 00213000 00:ef 8304370                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f380a467000-7f380a493000 r--p 002a3000 00:ef 8304370                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f380a493000-7f380a495000 rw-p 002cf000 00:ef 8304370                    /usr/lib/x86_64-linux-gnu/libcrypto.so.1.1
7f380a495000-7f380a49b000 rw-p 00000000 00:00 0 
7f380a49b000-7f380a49c000 r--p 00000000 00:ef 6489212                    /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f380a49c000-7f380a49e000 r-xp 00001000 00:ef 6489212                    /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f380a49e000-7f380a49f000 r--p 00003000 00:ef 6489212                    /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f380a49f000-7f380a4a0000 r--p 00003000 00:ef 6489212                    /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f380a4a0000-7f380a4a1000 rw-p 00004000 00:ef 6489212                    /usr/lib/x86_64-linux-gnu/libdl-2.31.so
7f380a4a1000-7f380a4ad000 r--p 00000000 00:ef 6489228                    /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.2.5
7f380a4ad000-7f380a57b000 r-xp 0000c000 00:ef 6489228                    /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.2.5
7f380a57b000-7f380a5b8000 r--p 000da000 00:ef 6489228                    /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.2.5
7f380a5b8000-7f380a5ba000 r--p 00116000 00:ef 6489228                    /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.2.5
7f380a5ba000-7f380a5bf000 rw-p 00118000 00:ef 6489228                    /usr/lib/x86_64-linux-gnu/libgcrypt.so.20.2.5
7f380a5bf000-7f380a5ee000 r--p 00000000 00:ef 6489232                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30.27.0
7f380a5ee000-7f380a710000 r-xp 0002f000 00:ef 6489232                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30.27.0
7f380a710000-7f380a781000 r--p 00151000 00:ef 6489232                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30.27.0
7f380a781000-7f380a782000 ---p 001c2000 00:ef 6489232                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30.27.0
7f380a782000-7f380a791000 r--p 001c2000 00:ef 6489232                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30.27.0
7f380a791000-7f380a793000 rw-p 001d1000 00:ef 6489232                    /usr/lib/x86_64-linux-gnu/libgnutls.so.30.27.0
7f380a793000-7f380a795000 rw-p 00000000 00:00 0 
7f380a795000-7f380a797000 r--p 00000000 00:ef 6489238                    /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.6
7f380a797000-7f380a79c000 r-xp 00002000 00:ef 6489238                    /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.6
7f380a79c000-7f380a7b3000 r--p 00007000 00:ef 6489238                    /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.6
7f380a7b3000-7f380a7b4000 ---p 0001e000 00:ef 6489238                    /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.6
7f380a7b4000-7f380a7b5000 r--p 0001e000 00:ef 6489238                    /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.6
7f380a7b5000-7f380a7b6000 rw-p 0001f000 00:ef 6489238                    /usr/lib/x86_64-linux-gnu/libidn2.so.0.3.6
7f380a7b6000-7f380a7bc000 r--p 00000000 00:ef 6489295                    /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f380a7bc000-7f380a7cd000 r-xp 00006000 00:ef 6489295                    /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f380a7cd000-7f380a7d3000 r--p 00017000 00:ef 6489295                    /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f380a7d3000-7f380a7d4000 r--p 0001c000 00:ef 6489295                    /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f380a7d4000-7f380a7d5000 rw-p 0001d000 00:ef 6489295                    /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
7f380a7d5000-7f380a7d9000 rw-p 00000000 00:00 0 
7f380a7d9000-7f380a7da000 r--s 00000000 00:f1 93263                      /data_dir/outgoing_msgs_9.db/db.31/001425.ldb
7f380a7da000-7f380a7db000 r--s 00000000 00:f1 16868                      /data_dir/28e07f34/blocks_and_extras/4.db/000100.ldb
7f380a7db000-7f380a7dc000 r--s 00000000 00:f1 16858                      /data_dir/28e07f34/blocks_and_extras/2.db/000100.ldb
7f380a7dc000-7f380a7dd000 r--s 00000000 00:f1 16853                      /data_dir/28e07f34/blocks_and_extras/1.db/000099.ldb
7f380a7dd000-7f380a7de000 r--s 00000000 00:f1 16848                      /data_dir/28e07f34/blocks_and_extras/0.db/000094.ldb
7f380a7de000-7f380a7e0000 rw-p 00000000 00:00 0 
7f380a7e0000-7f380a7e1000 r--p 00000000 00:ef 6489179                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f380a7e1000-7f380a804000 r-xp 00001000 00:ef 6489179                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f380a804000-7f380a80c000 r--p 00024000 00:ef 6489179                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f380a80c000-7f380a80d000 r--p 00000000 00:00 0 
7f380a80d000-7f380a80e000 r--p 0002c000 00:ef 6489179                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f380a80e000-7f380a80f000 rw-p 0002d000 00:ef 6489179                    /usr/lib/x86_64-linux-gnu/ld-2.31.so
7f380a80f000-7f380a810000 rw-p 00000000 00:00 0 
7ffcf4f08000-7ffcf4f29000 rw-p 00000000 00:00 0                          [stack]
7ffcf4f5d000-7ffcf4f60000 r--p 00000000 00:00 0                          [vvar]
7ffcf4f60000-7ffcf4f62000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

@PolinaKiporenko PolinaKiporenko moved this from QA to Ready For Pickup in SKALE Engineering 🚀 Sep 20, 2023
@PolinaKiporenko PolinaKiporenko moved this from Ready For Pickup to In Progress in SKALE Engineering 🚀 Sep 22, 2023
@dimalit
Copy link
Contributor

dimalit commented Sep 26, 2023

Debug stack trace!:+1:

  /skaled/skaled : dev::ExitHandler::exitHandler(int, dev::ExitHandler::exit_code_t)+0x44d [0x5577ca132bb9]                                                                                         
  /skaled/skaled : dev::ExitHandler::exitHandler(int)+0x1e [0x5577ca131aea]                                                                                                                         
  /lib/x86_64-linux-gnu/libpthread.so.0 : ()+0x14420 [0x7f5130910420]                                                                                                                               
  /lib/x86_64-linux-gnu/libc.so.6 : gsignal()+0xcb [0x7f512fe0e00b]                                                                                                                                 
  /lib/x86_64-linux-gnu/libc.so.6 : abort()+0x12b [0x7f512fded859]                                                                                                                                  
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xac52a [0x7f51301d352a]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa37c [0x7f51301d137c]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa3e7 [0x7f51301d13e7]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xab145 [0x7f51301d2145]                                                                                                                                
  /skaled/skaled : boost::system::error_code::message[abi:cxx11]() const+0x5b [0x5577c9cd62f3]                                                                                                      
  /skaled/skaled : boost::system::system_error::what() const+0x9f [0x5577c9cd692d]                                                                                                                  
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0x9e893 [0x7f51301c5893]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa37c [0x7f51301d137c]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xa9359 [0x7f51301d0359]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : __gxx_personality_v0()+0x2a1 [0x7f51301d0d11]                                                                                                              
  /lib/x86_64-linux-gnu/libgcc_s.so.1 : ()+0x10bff [0x7f512ffcdbff]                                                                                                                                 
  /lib/x86_64-linux-gnu/libgcc_s.so.1 : _Unwind_Resume()+0x12a [0x7f512ffce5ba]                                                                                                                     
  /skaled/skaled : SkaleRelayWS::stop()+0x53e [0x5577ca7352ce]                                                                                                                                      
  /skaled/skaled : SkaleRelayWS::~SkaleRelayWS()+0x8f [0x5577ca733eb1]                                                                                                                              
  /skaled/skaled : SkaleRelayWS::~SkaleRelayWS()+0x1c [0x5577ca733f98]                                                                                                                              
  /skaled/skaled : std::_Sp_counted_ptr<SkaleRelayWS*, (__gnu_cxx::_Lock_policy)2>::_M_dispose()+0x2c [0x5577ca76c488]                                                                              
  /skaled/skaled : std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release()+0x46 [0x5577c9d3ab30]                                                                                            
  /skaled/skaled : std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count()+0x2b [0x5577c9d2c341]                                                                                         
  /skaled/skaled : std::__shared_ptr<SkaleRelayWS, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr()+0x20 [0x5577ca759a8c]                                                                               
  /skaled/skaled : std::shared_ptr<SkaleRelayWS>::~shared_ptr()+0x1c [0x5577ca759ada]                                                                                                               
  /skaled/skaled : void __gnu_cxx::new_allocator<std::_List_node<std::shared_ptr<SkaleRelayWS> > >::destroy<std::shared_ptr<SkaleRelayWS> >(std::shared_ptr<SkaleRelayWS>*)+0x20 [0x5577ca7665f0]   
  /skaled/skaled : void std::allocator_traits<std::allocator<std::_List_node<std::shared_ptr<SkaleRelayWS> > > >::destroy<std::shared_ptr<SkaleRelayWS> >(std::allocator<std::_List_node<std::shared_ptr<SkaleRelayWS> > >&, std::shared_ptr<SkaleRelayWS>*)+0x27 [0x5577ca7633af]                                                                                                                      
  /skaled/skaled : std::__cxx11::_List_base<std::shared_ptr<SkaleRelayWS>, std::allocator<std::shared_ptr<SkaleRelayWS> > >::_M_clear()+0x66 [0x5577ca75f5ac]                                       
  /skaled/skaled : std::__cxx11::list<std::shared_ptr<SkaleRelayWS>, std::allocator<std::shared_ptr<SkaleRelayWS> > >::clear()+0x1c [0x5577ca75cb82]                                                
  /skaled/skaled : SkaleServerOverride::StopListening(e_server_mode_t)+0x175 [0x5577ca74881d]                                                                                                       
  /skaled/skaled : SkaleServerOverride::StopListening()+0x43 [0x5577ca749055]                                                                                                                       
  /skaled/skaled : ModularServer<>::StopListening()+0x8d [0x5577c9d39457]                                                                                                                           
  /skaled/skaled : ModularServer<>::~ModularServer()+0x2a [0x5577c9d4a03e]                                                                                                                          
  /skaled/skaled : ModularServer<dev::rpc::TestFace>::~ModularServer()+0x5c [0x5577c9d381a4]                                                                                                        
  /skaled/skaled : ModularServer<dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x5577c9d38278]                                                                     
  /skaled/skaled : ModularServer<dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x5577c9d3834c]                                                
  /skaled/skaled : ModularServer<dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x5577c9d38420]                        
  /skaled/skaled : ModularServer<dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x5577c9d384f4]
  /skaled/skaled : ModularServer<dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x5577c9d385c8]                                                                                                                                                                                
  /skaled/skaled : ModularServer<dev::rpc::NetFace, dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x5577c9d3869c]                                                                                                                                                             
  /skaled/skaled : ModularServer<dev::rpc::SkaleStats, dev::rpc::NetFace, dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x5577c9d38770]                                                                                                                                       
  /skaled/skaled : ModularServer<dev::rpc::SkaleFace, dev::rpc::SkaleStats, dev::rpc::NetFace, dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x5577c9d38844]                                                                                                                  
  /skaled/skaled : ModularServer<dev::rpc::EthFace, dev::rpc::SkaleFace, dev::rpc::SkaleStats, dev::rpc::NetFace, dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x5577c9dbc264]                                                                                               
  /skaled/skaled : ModularServer<dev::rpc::EthFace, dev::rpc::SkaleFace, dev::rpc::SkaleStats, dev::rpc::NetFace, dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x1c [0x5577c9dbc284]                                                                                               
  /skaled/skaled : std::default_delete<ModularServer<> >::operator()(ModularServer<>*) const+0x2c [0x5577c9d4647e]                                                                                  
  /skaled/skaled : std::unique_ptr<ModularServer<>, std::default_delete<ModularServer<> > >::~unique_ptr()+0x56 [0x5577c9d35b10]                                                                    
  /lib/x86_64-linux-gnu/libc.so.6 : ()+0x468a7 [0x7f512fe118a7]                                                                                                                                     
  /lib/x86_64-linux-gnu/libc.so.6 : on_exit()+0 [0x7f512fe11a60]                                                                                                                                    
  /skaled/skaled : Schain::healthCheck()+0x363 [0x5577ca4213af]                                                                                                                                     
  /skaled/skaled : Node::startClients()+0x3a [0x5577ca33a3a2]                                                                                                                                       
  /skaled/skaled : ConsensusEngine::startAll()+0x802 [0x5577ca2f3e02]                                                                                                                               
  /skaled/skaled : ()+0x1ee1e57 [0x5577c9f86e57]                                                                                                                                                    
  /skaled/skaled : ()+0x1ee6196 [0x5577c9f8b196]                                                                                                                                                    
  /skaled/skaled : ()+0x1ee614b [0x5577c9f8b14b]                                                                                                                                                    
  /skaled/skaled : ()+0x1ee60f8 [0x5577c9f8b0f8]                                                                                                                                                    
  /skaled/skaled : ()+0x1ee60ce [0x5577c9f8b0ce]                                                                                                                                                    
  /skaled/skaled : ()+0x1ee60b2 [0x5577c9f8b0b2]                                                                                                                                                    
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xd6df4 [0x7f51301fddf4]                                                                                                                                
  /lib/x86_64-linux-gnu/libpthread.so.0 : ()+0x8609 [0x7f5130904609]                                                                                                                                
  /lib/x86_64-linux-gnu/libc.so.6 : clone()+0x43 [0x7f512feea133]

5577c80a5000-5577cc319000 r-xp 00000000 00:43 544079 /skaled/skaled

@dimalit
Copy link
Contributor

dimalit commented Sep 26, 2023 

  /skaled/skaled : dev::ExitHandler::exitHandler(int, dev::ExitHandler::exit_code_t)+0x44d [0x55988177cbb9]                                                                                         
  /skaled/skaled : dev::ExitHandler::exitHandler(int)+0x1e [0x55988177baea]                                                                                                                         
  /lib/x86_64-linux-gnu/libpthread.so.0 : ()+0x14420 [0x7fce23032420]                                                                                                                               
  /lib/x86_64-linux-gnu/libc.so.6 : gsignal()+0xcb [0x7fce2253000b]                                                                                                                                 
  /lib/x86_64-linux-gnu/libc.so.6 : abort()+0x12b [0x7fce2250f859]                                                                                                                                  
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xac52a [0x7fce228f552a]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa37c [0x7fce228f337c]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa3e7 [0x7fce228f33e7]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xab145 [0x7fce228f4145]                                                                                                                                
  /skaled/skaled : boost::system::error_code::message[abi:cxx11]() const+0x5b [0x5598813202f3]                                                                                                      
  /skaled/skaled : boost::system::system_error::what() const+0x9f [0x55988132092d]                                                                                                                  
  /skaled/skaled : SkaleServerOverride::implHandleHttpRequest(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, e_server_mode_t)+0x1b83 [0x559881d884d5]                                                                                            
  /skaled/skaled : ()+0x26a30e2 [0x559881d920e2]                                                                                                                                                    
  /skaled/skaled : ()+0x26af403 [0x559881d9e403]                                                                                                                                                    
  /skaled/skaled : std::function<skutils::result_of_http_request (nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)>::operator()(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int) const+0xbd [0x5598818381ef]                                                                                  
  /skaled/skaled : skutils::http_pg::server::onRequest(nlohmann::basic_json<std::map, std::vector, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, unsigned long, double, std::allocator, nlohmann::adl_serializer> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int)+0x5f [0x55988182d249]                                                                                                                    
  /skaled/skaled : skutils::http_pg::request_site::onEOM()+0x890 [0x559881829c94]                                                                                                                   
  /skaled/skaled : proxygen::RequestHandlerAdaptor::onEOM()+0x3c [0x55988218e6ce]                                                                                                                   
  /skaled/skaled : proxygen::HTTPTransaction::processIngressEOM()+0x198 [0x5598821f966c]

55987f6ef000-559883963000 r-xp 00000000 00:43 774165 /skaled/skaled

@dimalit
Copy link
Contributor

dimalit commented Sep 26, 2023 

                                                                                                                                                                                                    
                                                                                                                                                                                                    
  /skaled/skaled : dev::ExitHandler::exitHandler(int, dev::ExitHandler::exit_code_t)+0x44d [0x55ce36514bb9]                                                                                         
  /skaled/skaled : dev::ExitHandler::exitHandler(int)+0x1e [0x55ce36513aea]                                                                                                                         
  /lib/x86_64-linux-gnu/libpthread.so.0 : ()+0x14420 [0x7ff7a14e4420]                                                                                                                               
  /lib/x86_64-linux-gnu/libc.so.6 : gsignal()+0xcb [0x7ff7a09e200b]                                                                                                                                 
  /lib/x86_64-linux-gnu/libc.so.6 : abort()+0x12b [0x7ff7a09c1859]                                                                                                                                  
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0x9e8d1 [0x7ff7a0d998d1]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa37c [0x7ff7a0da537c]                                                                                                                                
  /lib/x86_64-linux-gnu/libstdc++.so.6 : ()+0xaa3e7 [0x7ff7a0da53e7]                                                                                                                                
  /skaled/skaled : std::thread::~thread()+0x25 [0x55ce3624855f]                                                                                                                                     
  /skaled/skaled : std::default_delete<std::thread>::operator()(std::thread*) const+0x26 [0x55ce362d21ea]                                                                                           
  /skaled/skaled : std::unique_ptr<std::thread, std::default_delete<std::thread> >::~unique_ptr()+0x56 [0x55ce362ce744]                                                                             
  /skaled/skaled : dev::rpc::Skale::~Skale()+0x34 [0x55ce364f6446]                                                                                                                                  
  /skaled/skaled : dev::rpc::Skale::~Skale()+0x1c [0x55ce364f6492]                                                                                                                                  
  /skaled/skaled : std::default_delete<dev::rpc::SkaleFace>::operator()(dev::rpc::SkaleFace*) const+0x2c [0x55ce36141220]                                                                           
  /skaled/skaled : std::unique_ptr<dev::rpc::SkaleFace, std::default_delete<dev::rpc::SkaleFace> >::~unique_ptr()+0x56 [0x55ce3612ca34]                                                             
  /skaled/skaled : ModularServer<dev::rpc::SkaleFace, dev::rpc::SkaleStats, dev::rpc::NetFace, dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x54 [0x55ce3611a838]                                                                                                                  
  /skaled/skaled : ModularServer<dev::rpc::EthFace, dev::rpc::SkaleFace, dev::rpc::SkaleStats, dev::rpc::NetFace, dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x60 [0x55ce3619e264]                                                                                               
  /skaled/skaled : ModularServer<dev::rpc::EthFace, dev::rpc::SkaleFace, dev::rpc::SkaleStats, dev::rpc::NetFace, dev::rpc::Web3Face, dev::rpc::PersonalFace, dev::rpc::AdminEthFace, dev::rpc::DebugFace, dev::rpc::SkalePerformanceTracker, dev::rpc::TestFace>::~ModularServer()+0x1c [0x55ce3619e284]                                                                                               
  /skaled/skaled : std::default_delete<ModularServer<> >::operator()(ModularServer<>*) const+0x2c [0x55ce3612847e]                                                                                  
  /skaled/skaled : std::unique_ptr<ModularServer<>, std::default_delete<ModularServer<> > >::reset(ModularServer<>*)+0x65 [0x55ce3611aeed]                                                          
  /skaled/skaled : main()+0x18883 [0x55ce360ea8df]                                                                                                                                                  
  /lib/x86_64-linux-gnu/libc.so.6 : __libc_start_main()+0xf3 [0x7ff7a09c3083]                                                                                                                       
  /skaled/skaled : _start()+0x2e [0x55ce360b45ce]                                                                                                                                                   
                                                                                                                                                                                                    
55ce34487000-55ce386fb000 r-xp 00000000 00:43 781498                     /skaled/skaled```

@dimalit dimalit mentioned this issue Sep 27, 2023
Closed
@dimalit dimalit moved this from In Progress to Code Review in SKALE Engineering 🚀 Sep 28, 2023
@dimalit dimalit moved this from Code Review to QA in SKALE Engineering 🚀 Sep 28, 2023
@yatsunastya
Copy link

checked ✅
skaled: 3.17.0-beta.12

skaled container on all staying nodes exited gracefully with exit code 0

2023-10-03 12:19:00.571491   Setting ExitTimeReached = true
2023-10-03 12:19:00.571505   Skaled status: setExitState: ExitTimeReached to true
[2023-10-03 12:19:00.571] [77:consensus] [info] 6021:Running exitGracefully method
[2023-10-03 12:19:00.571] [77:consensus] [info] 6021:Iterating over nodes
[2023-10-03 12:19:00.571] [77:consensus] [info] 6021:Processing a node exit
[2023-10-03 12:19:00.571] [77:main] [info] 6021:Set exit on block boundary
[2023-10-03 12:19:00.571] [77:consensus] [info] 6021:Consensus exiting: exitGracefully called by skaled
Here is stack trace for your info:
 0# 0x00005631D031F6CE in /skaled/skaled
 1# 0x00005631D00D42CE in /skaled/skaled
 2# 0x00005631D00C67DD in /skaled/skaled
 3# 0x00005631D03A5C6E in /skaled/skaled
 4# 0x00005631D03B18B6 in /skaled/skaled
[2023-10-03 12:19:00.740] [config] [info] Consensus engine exiting: exitGracefullyAsync called by skaled
[2023-10-03 12:19:00.740] [config] [info] exitGracefullyAsync running
[2023-10-03 12:19:00.740] [config] [info] Node exit called
[2023-10-03 12:19:00.740] [config] [info] Node::exit() requested
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Exiting on block boundary for block 6021
[2023-10-03 12:19:00.740] [77:main] [info] 6021:exitImmediately called
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Status server stopped
[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: close all sockets called
[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: server conditional vars notified
2023-10-03 12:19:00.740060   Rotation is completed. Instance is exiting
2023-10-03 12:19:00.740083   THREADS timer started
[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: agent conditional vars notified
[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: catchup socket touched
 5# 0x00005631D03B44C4 in /skaled/skaled
 6# 0x00005631D03B5D59 in /skaled/skaled
 7# 0x00005631D037B4F3 in /skaled/skaled
 8# 0x00005631D0381EEB in /skaled/skaled
 9# 0x00005631D03A6D3F in /skaled/skaled
10# 0x00007F9334ECBDF4 in /lib/x86_64-linux-gnu/libstdc++.so.6
11# start_thread at /build/glibc-SzIz7B/glibc-2.31/nptl/pthread_create.c:478
12# 0x00007F9334BB8133 at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97


Caught (first) signal. Signal -1. Unknown signal type.


SELF-KILL: Will sleep 270 seconds before force exit...

[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: block proposal socket touched
[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: SGXZMQClient exiting
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Exiting SgxZmqClient
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Shutting down SgxZmq context
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Shut down SgxZmq context
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Closing SgxZmq client sockets
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Exited SgxZmqClient
[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: SGXZMQClient exited
[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: exitZMQClient called
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Cleaning up ZMQ sockets
[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: closing ZMQ send sockets
[2023-10-03 12:19:00.740] [77:main] [info] 6021:consensus engine exiting: closing receive sockets
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Closing ZMQ context
[2023-10-03 12:19:00.740] [77:main] [info] 6021:Consensus exiting: Node:Exit requested
7.1384e-05 THREADS 70: +main +skaled +dPool0 +dPool1 +dPool2 +dPool3 +dPool4 +dPool5 +dPool6 +dPool7 +dPool8 +dPool9 +dPool10 +dPool11 +dPool12 +dPool13 +dPool14 +dPool15 +blockVerifier0 +skaled +MonitoringLoop +TimeoutLoop +StuckDetectionL +CatchupClient +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +BlockPopClnt +skaled +Client +ZMQbg/Reaper +ZMQbg/IO/0 +ZmqBroadcaster +broadcastFunc +CtchpSrvAg0 +CtchpSrvAg1 +BlkPropSrvAg0 +BlockPropSrv +NtwkRdLoop +DeferMsgLoop +msgThreadProcLo +ZMQbg/Reaper +ZMQbg/IO/0 +ZMQbg/Reaper +ZMQbg/IO/0 +led/WS-listener +ed/WSS-listener +-0x5631d3dd1330 +HTTPSrvExec0 +HTTPSrvExec1 +HTTPSrvExec2 +HTTPSrvExec3 +HTTPSrvExec4 +HTTPSrvExec5 +HTTPSrvExec6 +HTTPSrvExec7 +HTTPSrvExec8 +HTTPSrvExec9 +IOThreadPool0 +-0x5631d17c8260 +msgThreadProcLo +msgThreadProcLo +msgThreadProcLo
[2023-10-03 12:19:00.741] [77:main] [info] 6021:Consensus exiting: Node:Exit requested
[2023-10-03 12:19:00.741] [77:main] [info] 6021:Consensus exiting: Node:Exit requested
[2023-10-03 12:19:00.747] [77:main] [info] 6021:Closed ZMQ context
[2023-10-03 12:19:00.747] [77:main] [info] 6021:consensus engine exiting: ZMQ sockets closeAndCleanupAll called
[2023-10-03 12:19:00.747] [77:main] [info] 6021:exitImmediately completed
[2023-10-03 12:19:00.747] [77:main] [error] 6021:Exception in Schain::messageThreadProcessingLoop
[2023-10-03 12:19:00.747] [77:main] [info] 6021:Consensus exiting: Exit on block boundary successful:Exit requested
2023-10-03 12:19:00.759311   Skaled status: setExitState: StartAgain to false
2023-10-03 12:19:00.759909   Skaled status: setExitState: StartFromSnapshot to false
2023-10-03 12:19:00.760558   Skaled status: setExitState: ClearDataDir to false
2023-10-03 12:19:00.763503   Will stop WS server on address 0.0.0.0 and port 10322=0x00002852/nfo...
2023-10-03 12:19:00.763537   Will stop on port 10322=0x00002852...
[2023-10-03 12:19:00.840] [config] [info] Node::exit() will try to exit on block boundary for 30 seconds
0.101035 THREADS 37: -MonitoringLoop -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -BlockPopClnt -CtchpSrvAg0 -CtchpSrvAg1 -BlkPropSrvAg0 -BlockPropSrv -NtwkRdLoop --0x5631d3dd1330 -HTTPSrvExec0 -HTTPSrvExec1 -HTTPSrvExec2 -HTTPSrvExec3 -HTTPSrvExec4 -HTTPSrvExec5 -HTTPSrvExec6 -HTTPSrvExec7 -HTTPSrvExec8 -HTTPSrvExec9 -IOThreadPool0
[2023-10-03 12:19:00.940] [config] [info] Node exit completed
0.703567 THREADS 34: -TimeoutLoop -DeferMsgLoop
2.10966 THREADS 33: -CatchupClient
2.41085 THREADS 31: -StuckDetectionL
2023-10-03 12:19:25.871776   OK, server stopped on port 10322=0x00002852
2023-10-03 12:19:25.871837   OK, stopped WS server on address 0.0.0.0 and port 10322=0x00002852/nfo
2023-10-03 12:19:25.871865   Will stop WSS server on address 0.0.0.0 and port 10327=0x00002857/nfo...
2023-10-03 12:19:25.871878   Will stop on port 10327=0x00002857...
25.1933 THREADS 30: -led/WS-listener
2023-10-03 12:19:26.004118   OK, server stopped on port 10327=0x00002857
2023-10-03 12:19:26.004165   OK, stopped WSS server on address 0.0.0.0 and port 10327=0x00002857/nfo
2023-10-03 12:19:26.004186   Will stop proxygen/HTTP server on address 0.0.0.0 and port 10323=0x00002853/nfo...
2023-10-03 12:19:26.004203   OK, stopped proxygen/HTTP server on address 0.0.0.0 and port 10323=0x00002853/nfo
2023-10-03 12:19:26.004210   Will stop proxygen/HTTPS server on address 0.0.0.0 and port 10328=0x00002858/nfo...
25.2938 THREADS 29: -ed/WSS-listener
2023-10-03 12:19:26.004219   OK, stopped proxygen/HTTPS server on address 0.0.0.0 and port 10328=0x00002858/nfo
2023-10-03 12:19:26.004228   Will stop proxygen/HTTP server on address 0.0.0.0 and port 10329=0x00002859/std...
2023-10-03 12:19:26.004237   OK, stopped proxygen/HTTP server on address 0.0.0.0 and port 10329=0x00002859/std
2023-10-03 12:19:26.005183   Skaled status: setSubsystemRunning: Rpc to false

2023-10-03 12:19:26.573744   Stop worker 568 ms
2023-10-03 12:19:26.573785   1 before exitGracefully()
2023-10-03 12:19:26.573796   Exit requested internally with exit code 0
[2023-10-03 12:19:26.573] [77:consensus] [info] 6021:Running exitGracefully method
2023-10-03 12:19:26.573830   2 after exitGracefully()
2023-10-03 12:19:26.573841   3 after wait loop
2023-10-03 12:19:26.619663   4 before dtor
.
2023-10-03 12:19:26.628173   Blockchain is closed
2023-10-03 12:19:26.628355   Skaled status: setSubsystemRunning: Blockchain to false

2023-10-03 12:19:26.628569   Blockchain is closed
25.8958 THREADS 23: -blockVerifier0 -Client -ZMQbg/Reaper -ZmqBroadcaster -broadcastFunc
Dispatch: All threads stopped
Dispatch: All dispatch queues removed
[2023-10-03 12:19:27.532] [77:consensus] [info] 6021:Running exitGracefully method
26.7989 THREADS 4: -dPool0 -dPool1 -dPool2 -dPool3 -dPool4 -dPool5 -dPool6 -dPool7 -dPool8 -dPool9 -dPool10 -dPool11 -dPool12 -dPool13 -dPool14 -dPool15 -ZMQbg/IO/0 --0x5631d17c8260

@github-project-automation github-project-automation bot moved this from QA to Ready For Release Candidate in SKALE Engineering 🚀 Oct 4, 2023
@EvgeniyZZ EvgeniyZZ moved this from Ready For Release Candidate to Done in SKALE Engineering 🚀 Oct 23, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dimalit dimalit

Labels
bug Something isn't working epic:node-rotation release:2.2
Projects
SKALE Engineering 🚀
Archived in project
Milestone
No milestone
5 participants
@dimalit @DmytroNazarenko @EvgeniyZZ @PolinaKiporenko @yatsunastya